Distributed transaction processing method based on message queue, computer equipment and medium

Abstract

The invention provides a distributed transaction processing method based on a message queue, computer equipment and a medium. The processing method comprises the steps that a producer sends a semi-transaction message to a Rocket MQ server; the semi-transaction message refers to a transaction message which is marked as a temporary non-delivery state by the Rocket MQ; the producer receives a semi-transaction confirmation message returned by the Rocket MQ server, and starts to execute a local transaction; if the local transaction is successfully executed, the producer immediately submits a secondary confirmation result of the semi-transaction message to the Rocket MQ server; if the secondary confirmation result of the semi-transactional message is submission, marking the semi-transactional message as a deliverable state, and delivering the semi-transactional message to the consumer; if rollback, the semi-transactional message will be deleted. By means of the distributed transaction processing method, the transaction processing efficiency can be remarkably improved, and operation and control are simplified.

Description

Technical Field

[0001] The present invention relates to the technical field of Internet application integration, and in particular to a distributed transaction processing method, computer equipment and medium based on a message queue. Background Art

[0002] Under the monolithic application architecture, the business volume can be well supported when it is small. However, as business demand continues to grow, the problems exposed by the monolithic application architecture are becoming increasingly prominent. On the one hand, as the number of visitors continues to increase, the monolithic application architecture cannot support the pressure brought by concurrency, and ultimately the pressure is placed on the database level. In the worst case, the database will crash, making the entire service unavailable. On the other hand, when updating a certain function, the monolithic application architecture needs to restart the entire project, which is not conducive to horizontal expansion and maintenance.

[0003] With the continuous development of technology in the Internet industry, distributed architecture is emerging. The modules in the monolithic application are separated according to the business, the coupling between modules is reduced, and each module is deployed, run and updated independently.

[0004] In a monolithic application architecture, local transactions can ensure the consistency of business data, but they have difficulty with high concurrency and read and write capabilities. In a distributed architecture, the business is split into modules and runs independently, which is friendly to high concurrency support, but cannot guarantee data consistency between modules. Therefore, there is an urgent need for a distributed transaction processing method that can overcome the defects of the above two architectures. Summary of the invention

[0005] The present invention aims to solve at least one of the above-mentioned technical problems existing in the prior art.

[0006] To this end, a first aspect of the present invention provides a distributed transaction processing method based on a message queue.

[0007] A second aspect of the present invention provides a computer device.

[0008] A second aspect of the present invention provides a computer-readable storage medium.

[0009] The present invention provides a distributed transaction processing method based on a message queue, wherein the transaction is a program execution unit for accessing or updating various data items in a database, and can transform the database from one consistency state to another consistency state; the transaction includes a local transaction and a distributed transaction;

[0010] The local transaction is a transaction based on a relational database;

[0011] The distributed transaction is a transaction completed by different services through remote collaboration over a network in a distributed system environment; the distributed system includes a number of business modules that can be combined into a single application, each business module has a corresponding relational database locally, wherein the business module that initiates the distributed transaction message is defined as a producer, and the business module that uses the distributed transaction message is defined as a consumer;

[0012] The processing method comprises:

[0013] The producer sends a semi-transactional message to the RocketMQ server; the semi-transactional message refers to a transactional message marked as temporarily undelivered by RocketMQ;

[0014] The producer receives the semi-transactional message confirmation message returned by the RocketMQ server and starts executing the local transaction;

[0015] If the local transaction is executed successfully, the producer will immediately submit the secondary confirmation result of the semi-transaction message to the RocketMQ server; if the secondary confirmation result of the semi-transaction message is committed, the semi-transaction message will be marked as deliverable and delivered to the consumer; if it is rolled back, the semi-transaction message will be deleted.

[0016] The message queue-based distributed transaction processing method according to the above technical solution of the present invention may also have the following additional technical features:

[0017] In the above technical solution, the transaction has atomicity, isolation and persistence;

[0018] Atomicity means that a transaction is an indivisible unit of work, and multiple operations in a transaction either all occur or none of them occur;

[0019] Isolation means that when multiple users access the database concurrently, the transaction opened by the database for each user will not be interfered by the operation data of other transactions, and multiple concurrent transactions are isolated from each other;

[0020] The persistence means that once a transaction is committed, the changes it makes to the data in the database are permanent and can be maintained even if the database fails.

[0021] In the above technical solution, the semi-transaction message has a set data format, and the data contains an ID field representing the current transaction.

[0022] In the above technical solution, the RocketMQ server provides a semi-transactional message review function.

[0023] In the above technical solution, the method for providing the semi-transaction message checkback function in the RocketMQ server includes:

[0024] In the producer's database, a transaction lookup table is established for transaction status lookup; the transaction lookup table records the execution result of each local transaction, and the transaction lookup table contains the ID field. If the local transaction is successfully executed, a transaction record is generated, otherwise no transaction record is generated;

[0025] The RocketMQ server implements the lookup of semi-transactional messages by calling the producer's transaction lookup table. The transaction lookup table is queried based on the ID field of the semi-transactional message. If there is a result in the query, the local transaction is considered to have been successfully executed and fed back to the RocketMQ server. The RocketMQ server marks the semi-transactional message as deliverable. If no result is found, it is fed back to the RocketMQ server and the RocketMQ server deletes the semi-transactional message.

[0026] In the above technical solution, when a transaction message cannot be delivered to a consumer, the RocketMQ server redelivers it until the maximum number of deliveries is reached.

[0027] In the above technical solution, a dead letter queue is set in the RocketMQ server, and the dead letter queue is used to store transaction messages that have not been delivered successfully and the number of repeated deliveries has reached the maximum number of delivery times. After the transaction message enters the dead letter queue, it waits for manual processing and data compensation.

[0028] In the above technical solution, in a distributed system, each business module runs and is deployed independently, and the corresponding relational database is independent of other databases.

[0029] A computer device provided by the present invention includes a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is loaded and executed by the processor, a distributed transaction processing method as described in any one of the above technical solutions is implemented.

[0030] The present invention provides a computer-readable storage medium storing a program, which, when loaded by a processor, implements a distributed transaction processing method as described in any one of the above technical solutions.

[0031] In summary, due to the adoption of the above technical features, the beneficial effects of the present invention are:

[0032] The distributed transaction processing method of the present invention can significantly improve transaction processing efficiency and simplify operation and control.

[0033] Specifically, traditional distributed transaction processing often relies on complex two-phase commit (2PC) or three-phase commit (3PC) protocols. These protocols require multiple network communications and lock resources when processing transactions across multiple databases or services, resulting in low transaction processing efficiency and prone to transaction failures due to network delays or single point failures. RocketMQ implements asynchronous processing of distributed transactions through its unique transaction message mechanism. The initiator of the transaction only needs to send a pre-commit message to the RocketMQ server and then asynchronously process the local transaction. When the local transaction succeeds, a confirmation message is sent to RocketMQ to ensure the final delivery of the message. This mechanism reduces the number of network communications and the occupation of lock resources, significantly improving the efficiency of transaction processing.

[0034] In addition, traditional distributed transaction processing requires developers to perform complex programming and control on transaction commit, rollback and other operations, which not only increases the difficulty of development, but may also cause transaction processing failure due to programming errors. RocketMQ provides a simple transaction message API and configuration options, allowing developers to easily implement distributed transaction processing. At the same time, RocketMQ's transaction message mechanism also provides automatic compensation and retry functions, further simplifying the control and monitoring of distributed transaction operations.

[0035] Additional aspects and advantages of the present invention will become apparent from the following description or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0037] Figure 1 The present invention is a flowchart of a distributed transaction processing method based on a message queue according to an embodiment of the present invention. DETAILED DESCRIPTION

[0038] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0039] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0040] Refer to the following Figure 1The following describes a message queue-based distributed transaction processing method, a computer device, and a medium according to some embodiments of the present invention.

[0041] Some embodiments of the present application provide a distributed transaction processing method based on a message queue.

[0042] First of all, we should make it clear that a transaction is a program execution unit used to access or update various data items in a database. Transactions have consistency, atomicity, isolation, and persistence.

[0043] The consistency refers to the ability to transform the database from one consistent state to another consistent state; for example, the total amount before and after the transfer remains unchanged; solving the data consistency problem between modules in a distributed architecture is also based on this property.

[0044] Atomicity means that a transaction is an indivisible unit of work, and multiple operations in a transaction either occur or none of them occur; isolation means that when multiple users access the database concurrently, the transaction opened by the database for each user is not interfered with by the operation data of other transactions, and multiple concurrent transactions are isolated from each other; persistence means that once a transaction is committed, the changes it makes to the data in the database are permanent, and can be maintained even if the database fails thereafter.

[0045] In the present disclosure, transactions are divided into local transactions and distributed transactions.

[0046] The local transaction is a transaction based on a relational database. Specifically, in a computer system, transactions are mostly controlled by a relational database, which is achieved by utilizing the transaction characteristics of the database itself, so it is called a database transaction. Since applications mainly rely on relational databases to control transactions, transactions based on relational databases are also called local transactions.

[0047] The distributed transaction is a transaction completed by remote collaboration between different services through the network in a distributed system environment. Specifically, a distributed system will split a single application system into multiple services that can run independently according to the business, so remote collaboration is required between services to complete transaction operations. In this distributed system environment, transactions completed by remote collaboration between different services through the network are called distributed transactions. Distributed transactions need to ensure that these collaborative operations are either all successful or all failed. In essence, distributed transactions are to ensure data consistency in different databases.

[0048] It should be noted that the distributed system described in the present disclosure includes several business modules that can be combined into a single application, each business module has a corresponding relational database locally, each business module runs and is deployed independently, and the corresponding relational database is independent of other databases.

[0049] Among them, the business module that initiates the distributed transaction message is defined as a producer, and the business module that uses the distributed transaction message is defined as a consumer.

[0050] The present disclosure adopts RocketMQ message middleware to realize the processing of distributed transactions. RocketMQ provides a transaction message mechanism. The transaction message supports the message initiator to decide whether to deliver or delete the message after executing the local transaction and according to the execution result of the local transaction.

[0051] In some embodiments, the processing method includes the following steps S1-S3.

[0052] S1. The initiator of the transaction, that is, the producer, sends a semi-transactional message to the RocketMQ server; the semi-transactional message refers to a transactional message marked by RocketMQ as temporarily not delivered, and the consumer cannot receive the message temporarily.

[0053] S2. The producer receives the semi-transactional message confirmation message returned by the RocketMQ server, confirming the success or failure of the semi-transactional message sending, and the producer starts executing the local transaction.

[0054] S3. If the producer's local transaction is executed successfully, the producer will immediately submit the secondary confirmation result of the semi-transaction message to the RocketMQ server (whether the semi-transaction message is committed or rolled back); if the secondary confirmation result of the semi-transaction message is committed, the semi-transaction message will be marked as deliverable and delivered to the consumer; if it is rolled back, the semi-transaction message will be deleted.

[0055] In some embodiments, the semi-transaction message has a set data format, such as a JSON format, and the data contains an ID field representing the current transaction, such as transactionId.

[0056] In some embodiments, when the producer submits the secondary confirmation result of a semi-transactional message, the RocketMQ server may be unable to process whether the message should be delivered or deleted due to network partitions, producer service downtime, and other issues. Therefore, the RocketMQ server provides a semi-transactional message review function.

[0057] Specifically, in the producer's database, a transaction lookup table transaction_log is established for transaction status lookup; the transaction lookup table records the execution result of each local transaction, and the transaction lookup table contains the ID field transactionId. If the local transaction is successfully executed, a transaction record is generated, otherwise no transaction record is generated;

[0058] The RocketMQ server checks the semi-transactional message by calling the producer's transaction checklist. The transaction_log transaction checklist is queried based on the ID field transactionId of the semi-transactional message. If there is a result, the local transaction is considered to have been successfully executed and fed back to the RocketMQ server. The RocketMQ server marks the semi-transactional message as deliverable. If no result is found, it is fed back to the RocketMQ server and the RocketMQ server deletes the semi-transactional message.

[0059] In some embodiments, when the RocketMQ server delivers a transaction message to a consumer, there may be no consumer or all consumers may fail, resulting in the inability to deliver the transaction message to the consumer. At this time, the RocketMQ server redelivers the message until the maximum number of deliveries is reached, for example, 16 times. In a specific embodiment, a dead letter queue is provided in the RocketMQ server, and the dead letter queue is used to store transaction messages that have not been delivered successfully and whose repeated deliveries have reached the maximum number of deliveries. After the transaction message enters the dead letter queue, it waits for manual processing and data compensation.

[0060] exist Figure 1 In the business process shown, the entire business process includes the producer (order system), database, RocketMQ server and consumer (points system). The producer and consumer are independently deployed applications, and the order database and the points database are also independently deployed and run. The dead letter queue is part of the RockerMQ server. The dead letter queue stores messages that have reached the maximum number of delivery times but have not been consumed. The order system and the points system interact with each other through the RocketMQ server. A Topic is created according to the business type. The order system sends messages to the topic, and the points system that subscribes to the topic will receive the sent messages.

[0061] Take a specific business process as an example to illustrate. Figure 1 In the distributed system shown in the figure, there is now a business in which when a user clicks on an order to receive the goods, the user is settled with points according to the order amount; because the monolithic architecture is upgraded to a distributed architecture, the order system and the points system are independently operated and deployed, and their corresponding databases are independent of each other.

[0062] When the user clicks on the receipt button on the interface, the order status change interface of the order system will be called. When the order system executes the business logic, two business operations will be involved. The first operation is to send a semi-transactional message with a custom topic to the RocketMQ server, and the second operation is to change the order status.

[0063] First, the first step is to send a semi-transactional message of a custom topic to the RocketMQ server (the status of the semi-transactional message is not delivered). The RocketMQ server immediately feeds back whether the semi-transactional message is successful to the order system. If successful, the second operation is executed to change the order status (if it fails, the user's delivery business execution fails, and the order system returns an exception to prompt the user). After the second operation successfully changes the order status (the status of the order will be changed to completed, and a record will be inserted into the transaction_log table), the local transaction execution result is immediately fed back to the RocketMQ server. The RocketMQ server will change the status of the semi-transactional message of the custom topic sent in the first step to the delivery status. Next, the points system (which has subscribed to the custom topic) will receive the semi-transactional message, and the points system will perform logical operations based on the data in the semi-transactional message. Finally, the user's delivery business execution is completed.

[0064] After the second operation successfully changes the order status, the local transaction execution result is immediately fed back to the RocketMQ server. During this gap, there may be network partitions, order system failures, and other reasons. The order system cannot immediately feed back the local transaction execution result to the RocketMQ server, resulting in the semi-transactional message sent in the first step to the custom topic being in the undelivered state. The RocketMQ server can check back the semi-transactional message, the default is 15 times, and the local transaction check method of the order system will be called according to the transactionId field in the semi-transactional message. If the check method query has results, it is considered that the local transaction of the order system is executed successfully, and the status of the semi-transactional message sent in the first step to the custom topic is immediately changed to the delivered state. If the query has no results, the semi-transactional message sent in the first step to the custom topic is immediately deleted.

[0065] Some other embodiments of the present invention provide a computer device, including a processor and a memory, wherein the memory stores a computer program, and when the computer program is loaded and executed by the processor, the distributed transaction processing method as described in any of the above embodiments is implemented.

[0066] Still other embodiments of the present invention provide a computer-readable storage medium storing a program, which, when loaded by a processor, implements the distributed transaction processing method as described in any of the above embodiments.

[0067] In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0068] Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A distributed transaction processing method based on message queue, characterized in that: The transaction is a program execution unit used to access or update various data items in the database, which can transform the database from one consistency state to another consistency state; the transaction includes local transactions and distributed transactions; The local transaction is a transaction based on a relational database; The distributed transaction is a transaction completed by different services through remote collaboration over a network in a distributed system environment; the distributed system includes a number of business modules that can be combined into a single application, each business module has a corresponding relational database locally, wherein the business module that initiates the distributed transaction message is defined as a producer, and the business module that uses the distributed transaction message is defined as a consumer; The processing method comprises: The producer sends a semi-transactional message to the RocketMQ server; the semi-transactional message refers to a transactional message marked as temporarily undelivered by RocketMQ; The producer receives the semi-transactional message confirmation message returned by the RocketMQ server and starts executing the local transaction; If the local transaction is executed successfully, the producer will immediately submit the secondary confirmation result of the semi-transaction message to the RocketMQ server; if the secondary confirmation result of the semi-transaction message is committed, the semi-transaction message will be marked as deliverable and delivered to the consumer; if it is rolled back, the semi-transaction message will be deleted.

2. The distributed transaction processing method based on message queue according to claim 1, characterized in that: The transaction is atomic, isolated and durable; Atomicity means that a transaction is an indivisible unit of work, and multiple operations in a transaction either all occur or none of them occur; Isolation means that when multiple users access the database concurrently, the transaction opened by the database for each user will not be interfered by the operation data of other transactions, and multiple concurrent transactions are isolated from each other; The persistence means that once a transaction is committed, the changes it makes to the data in the database are permanent and can be maintained even if the database fails.

3. The distributed transaction processing method based on message queue according to claim 1, characterized in that: The semi-transaction message has a set data format, and the data contains an ID field indicating the current transaction.

4. The distributed transaction processing method based on message queue according to claim 3 is characterized in that: The RocketMQ server provides a semi-transactional message review function.

5. The distributed transaction processing method based on message queue according to claim 4 is characterized in that: The method for providing a semi-transactional message checkback function in the RocketMQ server includes: In the producer's database, a transaction lookup table is established for transaction status lookup; the transaction lookup table records the execution result of each local transaction, and the transaction lookup table contains the ID field. If the local transaction is successfully executed, a transaction record is generated, otherwise no transaction record is generated; The RocketMQ server implements the lookup of semi-transactional messages by calling the producer's transaction lookup table. The transaction lookup table is queried based on the ID field of the semi-transactional message. If there is a result in the query, the local transaction is considered to have been successfully executed and fed back to the RocketMQ server. The RocketMQ server marks the semi-transactional message as deliverable. If no result is found, it is fed back to the RocketMQ server and the RocketMQ server deletes the semi-transactional message.

6. The distributed transaction processing method based on message queue according to claim 1, characterized in that: When a transaction message cannot be delivered to a consumer, the RocketMQ server redelivers it until the maximum number of deliveries is reached.

7. The distributed transaction processing method based on message queue according to claim 6 is characterized in that: The RocketMQ server is provided with a dead letter queue, which is used to store transaction messages that have not been successfully delivered and whose repeated delivery times have reached the maximum delivery times. After the transaction messages enter the dead letter queue, they wait for manual processing and data compensation.

8. The distributed transaction processing method based on message queue according to claim 1, characterized in that: In a distributed system, each business module runs and is deployed independently, and the corresponding relational database is independent of other databases.

9. A computer device, characterized in that: The invention comprises a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is loaded and executed by the processor, the distributed transaction processing method as claimed in any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium, characterized in that: A program is stored, and when the program is loaded by a processor, the distributed transaction processing method according to any one of claims 1 to 8 is implemented.

Images