Method of implementing two-phase commit (2PC) distributed transactions using relational database

Abstract

The present invention relates to the technical field of big data processing, and in particular, to a method for implementing a two-phase commit (2PC) distributed transaction using a relational database. The method comprises the following steps: S1: a metadata cluster, a storage cluster and a cluster management module are provided in a relational database; S2: a data sending unit on a server node sends a preparation message to each data receiving unit on the relational database, and waits for a response from the data receiving unit; S3: when the data receiving unit receives the preparation message, it executes a transaction operation and replies to the data sending unit according to the result of the operation. The method enhances the fault tolerance of the system, ensures the atomicity, consistency, isolation and persistence of distributed transactions, improves the transaction processing capability of the system, greatly reduces the synchronization blocking time of distributed transactions, and thus improves the transaction processing capability of the system.

Description

Technical Field

[0001] The present invention relates to the technical field of big data processing, and in particular to a method for implementing a two-phase commit (2PC) distributed transaction using a relational database. Background Art

[0002] In a distributed system, using a relational database to implement two-phase commit distributed transactions is a classic solution, and its background is mainly to ensure data consistency in a distributed environment. As the scale of business expands, a single database node is often difficult to meet the needs of high concurrent access and massive data storage, so enterprises tend to adopt distributed database systems. However, transaction management in a distributed environment is much more complicated than in a single-machine environment, because data is no longer stored centrally, but distributed on multiple nodes. In the prior art, it is still impossible to avoid the problem that if a failure such as GTM or RM crashes during the two-phase commit process, the two-phase commit process may be interrupted and cannot continue correctly. Therefore, a method for implementing two-phase commit 2PC distributed transactions using a relational database is designed. Summary of the invention

[0003] The purpose of the present invention is to provide a method for implementing two-phase commit (2PC) distributed transactions using a relational database, so as to solve the problem raised in the above background technology that if a GTM or RM crashes during the two-phase commit process, the two-phase commit process may be interrupted and cannot be continued correctly.

[0004] To achieve the above object, the present invention aims to provide a method for implementing a two-phase commit (2PC) distributed transaction using a relational database, comprising the following steps:

[0005] S1: A metadata cluster, a storage cluster, and a cluster management module are provided in a relational database;

[0006] S2: The data sending unit on the server node sends a preparation message to each data receiving unit on the relational database and waits for a response from the data receiving unit;

[0007] S3: When the data receiving unit receives the prepare message, it executes the transaction operation and replies to the data sending unit according to the result of the operation;

[0008] S4: the data sending unit waits for and collects the reply responses of all the data receiving units, and sends the corresponding message according to the reply responses of the data receiving units;

[0009] S5: The data receiving unit performs corresponding transaction operations according to the message from the data sending unit.

[0010] As a further improvement of the technical solution, the metadata cluster includes data distribution information, partition information and transaction global status.

[0011] As a further improvement of the present technical solution, a data receiving unit is provided on each node of the storage cluster, and each storage cluster uses a high availability mechanism to ensure that data is not lost when the node crashes. The data receiving unit is responsible for executing local transaction operations and responding to instructions from the data sending unit.

[0012] As a further improvement of the present technical solution, the cluster management module can be responsible for monitoring the operating status of the storage cluster and the metadata cluster.

[0013] As a further improvement of the technical solution, the cluster management module can decide whether to commit or roll back the transactions based on the information recorded in the metadata cluster when a failure occurs in the data sending unit or the data receiving unit.

[0014] As a further improvement of the present technical solution, each of the data receiving units includes an operation module, a data recording module and a message reply module; wherein the operation module is used to execute transaction-related operations in the database; the data recording module is used to record the undo and redo information of all operations in real time to the transaction log; and the message reply module is used to send a response message to the data sending unit according to the execution result of the operation module.

[0015] As a further improvement of the present technical solution, in S3, when the data receiving unit receives the prepare message, the operation module executes the transaction operation but does not commit it first; the data recording module records the undo and redo information in real time; the sending rule of the message reply module is: if the operation of the operation module is successful, the message reply module replies that it is ready; if the operation of the operation module fails, the message reply module replies that it is not ready.

[0016] As a further improvement of the present technical solution, the data sending unit includes a message response module, and the message response module cooperates with the message reply module to realize the interaction and response between the messages of the data sending module and the messages of the data receiving module.

[0017] As a further improvement of the present technical solution, in S4, a corresponding message is sent according to the reply response of the data receiving unit, and the rule for sending the message is: if all message reply modules reply that they are ready, the message response module sends a commit message to all data receiving units, otherwise the message response module sends a rollback message to all data receiving units.

[0018] As a further improvement of the present technical solution, in S5, the rules for executing corresponding transaction operations according to the message of the data sending unit are: if the data receiving unit receives a commit message, the operation module commits the transaction, and the message reply module sends a confirmation of the completion of the commit to the data sending unit; if the data receiving unit receives a rollback message, the operation module rolls back the transaction, and the message reply module sends a confirmation of the completion of the rollback to the data sending unit.

[0019] Compared with the prior art, the present invention has the following beneficial effects:

[0020] In the method of implementing two-phase commit 2PC distributed transactions using a relational database, the system's fault tolerance is enhanced by integrating a metadata cluster, a storage cluster and a cluster management module in the relational database; the metadata cluster records the global state of the transaction, and even in the case of partial node failure, the cluster management module can perform fault recovery based on this information to ensure the final consistency and data integrity of the transaction, and the cluster management module not only monitors the system state, but also quickly intervenes in processing the remaining prepared state transactions when a failure occurs, reducing the risk of data inconsistency, shortening the fault recovery time, and improving the availability of the system. In the first phase of 2PC, after each participating member of the distributed transaction completes the writing of the data log metadata, the data write lock will be released, and the writing of the new transaction log will be accepted. The new transaction can be processed without waiting for the submission of the second phase data, which greatly improves the throughput of the distributed transaction. If the submission fails in the second phase, the metadata of the transaction log will be checked for version when reading the data, and the transaction log records of the non-submitted version number will be cleared, thereby improving the fault tolerance caused by the failure of the second phase transaction submission. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 The figure is a flow chart of the overall method of the present invention. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] Example

[0024] See also Figure 1 As shown, a method for implementing a two-phase commit 2PC distributed transaction using a relational database is provided, including the following steps:

[0025] S1: A metadata cluster, a storage cluster, and a cluster management module are provided in a relational database;

[0026] S2: The data sending unit on the server node sends a preparation message to each data receiving unit on the relational database and waits for a response from the data receiving unit;

[0027] S3: When the data receiving unit receives the prepare message, it executes the transaction operation and replies to the data sending unit according to the result of the operation;

[0028] S4: the data sending unit waits for and collects the reply responses of all the data receiving units, and sends the corresponding message according to the reply responses of the data receiving units;

[0029] S5: The data receiving unit performs corresponding transaction operations according to the message from the data sending unit.

[0030] The metadata cluster includes data distribution information, partition information, and global transaction status.

[0031] A data receiving unit is provided on each node of the storage cluster, and each storage cluster uses a high availability mechanism to ensure that data is not lost when a node fails. The data receiving unit is responsible for executing local transaction operations and responding to instructions from the data sending unit.

[0032] The cluster management module is responsible for monitoring the operating status of the storage cluster and metadata cluster.

[0033] The cluster management module can decide whether to commit or roll back these transactions based on the information recorded in the metadata cluster when a data sending unit or a data receiving unit fails.

[0034] Each data receiving unit includes an operation module, a data recording module and a message reply module; wherein the operation module is used to execute transaction-related operations in the database; the data recording module is used to record the undo and redo information of all operations in real time to the transaction log; the message reply module is used to send a response message to the data sending unit according to the execution result of the operation module.

[0035] In S3, when the data receiving unit receives the prepare message, the operation module executes the transaction operation but does not commit it first; the data recording module records the undo and redo information in real time; the sending rule of the message reply module is: if the operation module succeeds, the message reply module replies ready; if the operation module fails, the message reply module replies not ready.

[0036] The data sending unit includes a message response module, which cooperates with the message reply module to realize the interaction and response between the messages of the data sending module and the messages of the data receiving module.

[0037] In S4, the corresponding message is sent according to the reply response of the data receiving unit. The rule for sending the message is: if all message reply modules reply that they are ready, the message response module sends a commit message to all data receiving units, otherwise the message response module sends a rollback message to all data receiving units.

[0038] In S5, according to the message of the data sending unit, the rules for executing the corresponding transaction operation are: if the data receiving unit receives a commit message, the operation module commits the transaction, and the message reply module sends a confirmation of the completion of the commit to the data sending unit; if the data receiving unit receives a rollback message, the operation module rolls back the transaction, and the message reply module sends a confirmation of the completion of the rollback to the data sending unit.

[0039] In this embodiment, a relational database is a data management system based on a relational model. Its main feature is to organize data into a series of interrelated tables and perform operations such as data storage, query, update and deletion through the structured query language SQL. Two-phase commit is an algorithm used to ensure data consistency between multiple nodes in a distributed system, especially in distributed transaction scenarios involving multiple databases or services. Its core goal is that the coordinator is responsible for the coordination of the entire distributed transaction, ensuring that all participants either commit all transactions or roll back all transactions, thereby maintaining global data consistency. In this embodiment, the coordinator is a data sending unit, and the participants are data receiving units. Before executing a transaction, S1: a metadata cluster, a storage cluster and a cluster management module are set in the relational database;

[0040] The metadata cluster includes data distribution information, partition information, and transaction global status. The distribution information records the specific distribution of data in the storage cluster, allowing the system to quickly locate the data, thereby improving the efficiency of query and transaction processing; partition information can ensure that data is reasonably divided between different nodes, improving parallel processing capabilities and system scalability; transaction global status: maintains the life cycle status of the transaction, including whether the transaction is prepared, committed, or rolled back, which is very important for handling abnormal situations and recovery operations in distributed transactions.

[0041] The storage cluster consists of multiple physical or virtual server nodes. Each node is equipped with a data receiving unit, and each storage cluster uses a high-availability mechanism to ensure that data is not lost when the node fails. The high-availability mechanism uses high-availability mechanisms such as replica sets, master-slave replication, or distributed consensus protocols for each storage cluster to ensure that even if a single node fails, the data can still be accessed and not lost, ensuring data persistence and service continuity. The data receiving unit is responsible for executing local transaction operations and responding to the instructions of the data sending unit, such as updating, inserting, or deleting, and feedback to the data sending unit based on the operation results.

[0042] The cluster management module is responsible for monitoring the operating status of the storage cluster and metadata cluster to ensure the health and performance of the cluster. When a data sending unit or a data receiving unit fails, the cluster management module can judge and process transactions in the intermediate state based on the information recorded by the metadata cluster, and decide whether to commit or roll back these transactions, so as to ensure the final consistency of the transactions.

[0043] When executing a transaction, S2: the data sending unit on the server node sends a prepare message to each data receiving unit on the relational database and waits for the response of the data receiving unit; each data receiving unit includes an operation module, a data recording module and a message reply module; wherein the operation module is used to perform transaction-related operations in the database. After receiving the prepare message, the module will start to perform local operations of the transaction, but will not immediately submit these changes to the persistent storage of the database, ensuring that all participating nodes can successfully prepare the transaction, thereby laying the foundation for subsequent global submission decisions; the data recording module is used to record the undo and redo information of all operations in real time to the transaction log. The undo information allows the transaction to be rolled back to the state before the transaction starts when necessary, while the redo information ensures that the transaction can be restored and submitted even after the system crashes. Real-time recording is an important mechanism to ensure the atomicity and persistence of transactions; the message reply module is used to send a response message to the data sending unit according to the execution result of the operation module. The data sending unit includes a message response module, which cooperates with the message reply module to realize the interaction and response between the messages of the data sending module and the messages of the data receiving module.

[0044] S3: When the data receiving unit receives the prepare message, it executes the transaction operation and replies to the data sending unit according to the result of the operation; when the data receiving unit receives the prepare message, the operation module executes the transaction operation but does not commit it first; the data recording module records the undo and redo information in real time; the sending rule of the message reply module is: if the operation module succeeds, the message reply module replies ready; if the operation module fails, the message reply module replies not ready.

[0045] S4: The data sending unit waits for and collects reply responses from all data receiving units, and sends corresponding messages according to the reply responses of the data receiving units; the corresponding messages are sent according to the reply responses of the data receiving units, and the rules for sending messages are: if all message reply modules reply that they are ready, the message response module sends a commit message to all data receiving units, otherwise the message response module sends a rollback message to all data receiving units.

[0046] S5: The data receiving unit performs corresponding transaction operations according to the message of the data sending unit. The rules for performing corresponding transaction operations according to the message of the data sending unit are as follows: if the data receiving unit receives a commit message, the operation module commits the transaction, and the message reply module sends a confirmation of the completion of the commit to the data sending unit; if the data receiving unit receives a rollback message, the operation module rolls back the transaction, and the message reply module sends a confirmation of the completion of the rollback to the data sending unit.

[0047] During the whole process, if any node failure occurs, the cluster management module intervenes and performs fault recovery based on the information recorded in the metadata to ensure the correct execution of transactions and data consistency. Specifically, the cluster management module continuously monitors all nodes involved in distributed transaction processing through heartbeat detection or other monitoring mechanisms, including data sending units, data receiving units, and nodes of the metadata cluster and storage cluster. Once a node is detected to be unresponsive or in an abnormal state, the cluster management module will first try to re-establish the connection or confirm that the node has indeed failed;

[0048] After confirming the failure, the cluster management module queries the metadata cluster to obtain information about the status of ongoing or recently completed distributed transactions. This includes identifying which transactions are in the preparation state, that is, those transactions that have received a successful preparation response from all participating nodes but have not yet received the final instruction to commit or rollback;

[0049] For transactions in the preparation state, the cluster management module needs to decide whether to commit or roll back these transactions based on the context and consistency requirements of the transactions. The decision is based on the distributed transaction number and the version number of the data. If all participating members of the distributed transaction have completed the first phase of the commit, the transaction manager will update the transaction status of the transaction, and the transaction logs of the participating members will delete the transaction logs except the latest version, and update the version number of the committed data in the transaction metadata. If a transaction participating member with the same distributed transaction number fails to complete the first phase of the commit, the transaction manager will execute a transaction rollback on the participating member, that is, delete the transaction logs except the committed version number in the transaction metadata;

[0050] Then, the cluster management module will send corresponding instructions to all relevant nodes, including those nodes that are operating normally and the original faulty nodes that may have been restored. If it is a transaction submission, it will send a commit message; if it is a transaction rollback, it will send a rollback message. The data receiving unit will perform corresponding operations according to the received instructions to ensure the consistency of the transaction;

[0051] After the transaction is executed, the cluster management module is also responsible for cleaning up temporary resources used in the transaction processing, updating the transaction status information in the metadata cluster, and ensuring that the metadata is consistent with the actual transaction status. In addition, it may also need to trigger some compensation operations or notify the upper-layer application about the final result of the transaction. After the fault is handled, the cluster management module will continue to monitor the system status to ensure that all services are restored to normal operation, and may perform some additional health check operations to verify the stability of the system and data consistency.

[0052] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments. The above embodiments and descriptions are only preferred examples of the present invention and are not intended to limit the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A method for implementing a two-phase commit (2PC) distributed transaction using a relational database, characterized in that: The steps include: S1: A metadata cluster, a storage cluster, and a cluster management module are provided in a relational database; S2: The data sending unit on the server node sends a preparation message to each data receiving unit on the relational database and waits for a response from the data receiving unit; S3: When the data receiving unit receives the prepare message, it executes the transaction operation and replies to the data sending unit according to the result of the operation; S4: the data sending unit waits for and collects the reply responses of all the data receiving units, and sends the corresponding message according to the reply responses of the data receiving units; S5: The data receiving unit performs corresponding transaction operations according to the message of the data sending unit; The cluster management module is responsible for monitoring the operating status of the storage cluster and metadata cluster; The cluster management module can decide whether to commit or roll back these transactions based on the information recorded in the metadata cluster when a data sending unit or a data receiving unit fails; Each data receiving unit includes an operation module, a data recording module and a message reply module; wherein the operation module is used to execute transaction-related operations in the database; the data recording module is used to record the undo and redo information of all operations in real time to the transaction log; the message reply module is used to send a response message to the data sending unit according to the execution result of the operation module; Each node in the storage cluster is equipped with a data receiving unit, and each storage cluster uses a high availability mechanism to ensure that data is not lost when a node crashes. The data receiving unit is responsible for executing local transaction operations and responding to instructions from the data sending unit. In S3, when the data receiving unit receives the preparation message, the operation module executes the transaction operation, but does not commit it first; the data recording module records the undo and redo information in real time; the sending rule of the message reply module is: if the operation module succeeds, the message reply module replies ready; if the operation module fails, the message reply module replies not ready; In S4, the corresponding message is sent according to the reply response of the data receiving unit, and the rule for sending the message is: if all the message reply modules reply that they are ready, the message response module sends a commit message to all the data receiving units, otherwise the message response module sends a rollback message to all the data receiving units; In S5, according to the message of the data sending unit, the rules for executing the corresponding transaction operation are as follows: if the data receiving unit receives a commit message, the operation module commits the transaction, and the message reply module sends a confirmation of the completion of the commit to the data sending unit; if the data receiving unit receives a rollback message, the operation module rolls back the transaction, and the message reply module sends a confirmation of the completion of the rollback to the data sending unit; In the first phase of 2PC, after each participating member of the distributed transaction completes the writing of the data log metadata, the cluster management module will release the data write lock and accept the writing of the new transaction log, and can process the new transaction without waiting for the submission of the second phase data. If the submission fails in the second phase, the metadata of the transaction log will be version checked when reading the data, and the transaction log records with non-submitted version numbers will be cleared.

2. The method for implementing two-phase commit (2PC) distributed transactions using a relational database according to claim 1, characterized in that: The metadata cluster includes data distribution information, partition information, and global transaction status.

3. The method for implementing two-phase commit (2PC) distributed transactions using a relational database according to claim 1, characterized in that: The data sending unit includes a message response module, which cooperates with the message reply module to realize the interaction and response between the messages of the data sending module and the messages of the data receiving module.

Images