Transaction processing system and method based on message queue
By introducing a transaction processing system based on message queue and compensation timing tasks in a distributed system, the reliability and efficiency of message delivery in a distributed environment are solved, the stability and fault tolerance of the system are improved, and the success rate of message delivery is ensured. and data consistency.
Patent Information
- Application Number
- CN202411208331.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-30
- Publication Date
- 2024-11-22
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
In distributed systems, traditional database management systems are difficult to ensure the characteristics of ACID transactions, especially in the event of network fluctuations and system failures, the reliability and efficiency of message delivery are prominent.
Adopt a transaction processing system based on message queues, and improve message delivery by introducing the 'MQ Maximum Notice' mode, setting up timed compensation tasks and building a message management system. reliability and system stability.
Improves the success rate of messaging and system scalability, flexibility and stability, ensuring that messages can eventually be processed correctly even in the event of network instability or system failure, improving the system's fault tolerance and business continuity.
Smart Images

Figure CN119003102A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of Internet technology, in particular to a message queue-based transaction processing system and method. Background Art
[0002] In the distributed system architecture, message queue (MQ) is an important component used for decoupling services, asynchronous processing, and improving system scalability. However, in the processing of distributed services, how to ensure the reliable delivery of messages has become a challenge, especially in the face of uncertain factors such as network fluctuations and system failures.
[0003] Traditional database management systems (DBMS) often use the ACID principle (atomicity, consistency, isolation, and durability) when processing transactions, but this is difficult to apply directly in a distributed system architecture. In a distributed system architecture, each node has various abnormal conditions, making it difficult to ensure ACID transaction characteristics.
[0004] Atomicity: All operations in a transaction are either completed or not completed, and will not end at some stage in the middle. If an error occurs during the execution of a transaction, it will be rolled back to the state before the transaction started, as if the transaction had never been executed.
[0005] Consistency, transaction consistency means that the execution of a transaction cannot destroy the integrity and consistency of the database data. The database must be in a consistent state before and after a transaction is executed. In other words, the result of the transaction execution must be to make the database change from one consistent state to another consistent state. Therefore, when the database only contains the results of successful transaction submissions, it can be said that the database is in a consistent state. If the database system fails during operation, some transactions are forced to be interrupted before completion, and some of the modifications made to the database by these unfinished transactions have been written to the physical database, then the database is in an incorrect state, or an inconsistent state.
[0006] Isolation, the ability of a database to allow multiple concurrent transactions to read, write and modify its data at the same time, can prevent data inconsistency caused by cross-execution when multiple transactions are executed concurrently. Transaction isolation is divided into different levels, including read uncommitted, read committed, repeatable read and serializable.
[0007] Persistence: After the transaction is completed, the changes to the data are permanent and will not be lost even if the system fails.
[0008] Therefore, it is necessary to optimize message delivery in a distributed environment to ensure the success rate of message delivery.
[0009] The information disclosed in this background technology section is only intended to deepen the understanding of the overall background technology of the present invention, and should not be regarded as acknowledging or suggesting in any form that the information constitutes the prior art already known to those skilled in the art. Summary of the invention
[0010] In view of the defects existing in the prior art, the purpose of the present invention is to provide a transaction processing system and method based on a message queue, which effectively solves the reliability and efficiency problems of message transmission in a distributed environment by introducing the "MQ maximum effort notification" mode, setting timed compensation tasks, and building a message management system, thereby improving the scalability, flexibility and stability of the system and providing strong technical support for a variety of application scenarios.
[0011] In order to achieve the above purpose, the technical solution adopted by the present invention is:
[0012] A transaction processing system based on a message queue, characterized by comprising:
[0013] The producer is the party responsible for sending messages. It stores messages in the database to ensure the consistency between message storage and business data storage. It includes producers and producer databases, which are responsible for generating and storing messages.
[0014] The consumer is the party responsible for receiving and processing messages. It first confirms whether the message is duplicated, then processes the business logic, and updates the message status based on the processing results. It includes consumers and consumer databases, which are responsible for receiving and processing messages.
[0015] Create a compensation timer task on the consumer side to handle the situation where message consumption fails. In the consumer service, regularly scan the producer database for messages that have failed to be consumed or have not been consumed, and re-consume them.
[0016] The message management system is used to maintain the message queues between producers and consumers, collect messages stored by producers, provide a message console for querying and filtering message lists, and resend messages that failed to be consumed.
[0017] Based on the above technical solution, messages are stored in the databases of each producer in a distributed manner. The producer service address is configured through the message console to centrally display the message push status.
[0018] Based on the above technical solution, for messages with an initialization or failure status, as well as messages older than 20 minutes, the compensation timer task will increase the number of retries and reprocess the business logic.
[0019] Based on the above technical solution, the producer is responsible for recording the message push trajectory and consumption status. After the consumer succeeds or fails in message consumption, the consumer calls the producer interface to update the message status.
[0020] Based on the above technical solution, a web management console is provided to allow users to view the message push status and manually resend the messages that failed to be consumed.
[0021] On the basis of the above technical solution, in the consumer service, a timed consumption compensation task is introduced by introducing a jar package to automatically scan and re-consume failed messages.
[0022] A transaction processing method based on a message queue, characterized in that it comprises the following steps:
[0023] The producer sends the message to the message queue, and the producer records the detailed information of the message in the producer database, including the message content and creation time;
[0024] The consumer receives the message from the message queue;
[0025] The consumer checks whether the received message is a duplicate message;
[0026] When it is not a duplicate message, the consumer will execute the relevant business logic;
[0027] Based on the execution results of the business logic, the consumer decides whether the message has been processed successfully;
[0028] The consumer updates the status of the message based on the execution results of the business logic.
[0029] Based on the above technical solution, if the message consumption fails, the consumer will increase the number of message retries and mark the message status as failed.
[0030] Based on the above technical solution, the consumer creates and starts a compensation timer task. For messages with an initialization or failure status, as well as messages older than 20 minutes, the compensation timer task will be executed regularly to query the producer's database for records about the message and try to reprocess these messages.
[0031] The message queue transaction processing system and method described in the present invention have the following features:
[0032] Beneficial effects:
[0033] The producer stores messages in the database in advance to ensure data consistency, the consumer promptly provides status feedback after message processing, and the compensation timer task periodically retries failed messages until manual intervention, which greatly improves the success rate of message delivery.
[0034] The introduction of the message management system makes it easier to monitor, filter and resend messages, thus ensuring that messages can be processed correctly even in the event of network instability or system failure.
[0035] The message queue-based transaction processing system and method described in the present invention not only improve the fault tolerance of the system, but also enhance the continuity and stability of the business, and are particularly suitable for application scenarios that do not require high immediacy of message transmission but emphasize data consistency. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] The present invention has the following accompanying drawings:
[0037] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.
[0038] Figure 1 A system architecture diagram of a first embodiment of a message queue transaction processing system according to the present invention.
[0039] Figure 2 A flowchart of embodiment 1 of a message queue transaction processing method according to the present invention. DETAILED DESCRIPTION
[0040] The present invention is further described in detail below in conjunction with the accompanying drawings. The detailed description is an explanation made in conjunction with the exemplary embodiments of the present invention, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those skilled in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.
[0041] like Figure 1 As shown, the present invention provides a transaction processing system based on a message queue, comprising:
[0042] The producer is the party responsible for sending messages. It stores messages in the database to ensure the consistency between message storage and business data storage. It includes producers and producer databases, which are responsible for generating and storing messages.
[0043] The consumer is the party responsible for receiving and processing messages. It first confirms whether the message is duplicated, then processes the business logic, and updates the message status based on the processing results. It includes consumers and consumer databases, which are responsible for receiving and processing messages.
[0044] Create a compensation timer task on the consumer side to handle the situation where message consumption fails. In the consumer service, regularly scan the producer database for messages that have failed to be consumed or have not been consumed, and re-consume them.
[0045] The message management system is used to maintain the message queues between producers and consumers, collect messages stored by producers, provide a message console for querying and filtering message lists, and resend messages that failed to be consumed.
[0046] In this embodiment, by compensating for the scheduled task, the system can automatically process the message that fails to be consumed, thus improving the reliability of message delivery. The system design allows messages to be retried until they succeed or reach the preset upper limit of the number of retries, which enhances the robustness of the system and enables the system to effectively process message queue transactions in a distributed environment, ensuring the reliable delivery of messages and the correct execution of business logic.
[0047] The system described in this embodiment uses its architecture and functional design. Before sending a message, the producer first stores the message in the producer's database to ensure the consistency of message storage. As long as the message is successfully saved, it will definitely be consumed, which effectively solves the reliability and efficiency problems of message transmission in a distributed environment. It is particularly suitable for application scenarios that have a certain tolerance for message immediacy but emphasize final consistency. Even in the case of network instability or system failure, it can maximize the delivery of messages and improve the overall stability and business continuity of the system. Its technical advantages are:
[0048] First, the producer service saves the message push track and message consumption status, and these data have a dedicated web management console to view the message push status, and click to resend in the console according to the push status.
[0049] Secondly, there is a special compensation task for messages that fail to be consumed. In the consumer service, for failed messages, the scheduled task will repeat the message until it succeeds. Even if a message fails to be sent, the compensation task will pull the message stored by the producer and then consume it successfully.
[0050] Based on the above technical solution, messages are stored in the databases of each producer in a distributed manner. The producer service address is configured through the message console to centrally display the message push status.
[0051] Based on the above technical solution, for messages with an initialization or failure status, as well as messages older than 20 minutes, the compensation timer task will increase the number of retries and reprocess the business logic.
[0052] For example, for messages that failed to be consumed, the compensation timer task will periodically read these messages in the producer database and try to consume them again until they succeed or the maximum number of retries (15 times) is reached. It is generally impossible to fail after 15 retries. If it exceeds 15 times, the R&D personnel need to check whether there are some unknown problems in the system.
[0053] Based on the above technical solution, the producer is responsible for recording the message push trajectory and consumption status. After the consumer succeeds or fails in message consumption, the consumer calls the producer interface to update the message status.
[0054] Based on the above technical solution, a web management console is provided to allow users to view the message push status and manually resend the messages that failed to be consumed.
[0055] After the maximum number of retries is reached, manual troubleshooting and compensation are allowed, and the message can be resent manually.
[0056] On the basis of the above technical solution, in the consumer service, a timed consumption compensation task is introduced by introducing a jar package to automatically scan and re-consume failed messages.
[0057] Based on the above system, Figure 2 As shown, the present invention provides a transaction processing method based on a message queue, comprising the following steps:
[0058] The producer sends the message to the message queue, and the producer records the detailed information of the message in the producer database, including the message content and creation time;
[0059] The consumer receives the message from the message queue;
[0060] The consumer checks whether the received message is a duplicate message;
[0061] When it is not a duplicate message, the consumer will execute the relevant business logic;
[0062] Based on the execution results of the business logic, the consumer decides whether the message has been processed successfully;
[0063] The consumer updates the status of the message based on the execution results of the business logic.
[0064] Based on the above technical solution, if the message consumption fails, the consumer will increase the number of message retries and mark the message status as failed.
[0065] Based on the above technical solution, the consumer creates and starts a compensation timer task. For messages with an initialization or failure status, as well as messages older than 20 minutes, the compensation timer task will be executed regularly to query the producer's database for records about the message and try to reprocess these messages.
[0066] The contents not described in detail in this specification belong to the prior art known to professional and technical personnel in this field.
[0067] The above description is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiment. Any equivalent modifications or changes made by those skilled in the art based on the contents disclosed by the present invention should be included in the protection scope recorded in the claims.
Claims
1. A transaction processing system based on message queue, It is characterized in that include: The producer is the party responsible for sending messages. It stores messages in the database to ensure the consistency between message storage and business data storage. It includes producers and producer databases, which are responsible for generating and storing messages. The consumer is the party responsible for receiving and processing messages. It first confirms whether the message is duplicated, then processes the business logic, and updates the message status based on the processing results. It includes consumers and consumer databases, which are responsible for receiving and processing messages. Create a compensation timer task on the consumer side to handle the situation where message consumption fails. In the consumer service, regularly scan the producer database for messages that have failed to be consumed or have not been consumed, and re-consume them. The message management system is used to maintain the message queues between producers and consumers, collect messages stored by producers, provide a message console for querying and filtering message lists, and resend messages that failed to be consumed.
2. A message queue-based transaction processing system as claimed in claim 1, It is characterized in that Messages are stored in distributed databases of various producers. The producer service address is configured through the message console to centrally display the message push status.
3. A message queue-based transaction processing system as claimed in claim 1, It is characterized in that For messages with an initialization or failure status, and messages older than 20 minutes, the compensation timer task increases the number of retries and reprocesses the business logic.
4. A message queue-based transaction processing system as claimed in claim 1, It is characterized in that The producer is responsible for recording the message push track and consumption status. After the message consumption succeeds or fails, the consumer calls the producer interface to update the message status.
5. A message queue-based transaction processing system as claimed in claim 1, It is characterized in that Provides a web management console that allows users to view message push status and manually resend messages that have failed to be consumed.
6. A message queue-based transaction processing system as claimed in claim 1, It is characterized in that In the consumer service, a scheduled consumption compensation task is introduced by introducing a jar package to automatically scan and re-consume failed messages.
7. A transaction processing method based on message queue, It is characterized in that The steps include: The producer sends the message to the message queue, and the producer records the detailed information of the message in the producer database, including the message content and creation time; The consumer receives the message from the message queue; The consumer checks whether the received message is a duplicate message; When it is not a duplicate message, the consumer will execute the relevant business logic; Based on the execution results of the business logic, the consumer decides whether the message has been processed successfully; The consumer updates the status of the message based on the execution results of the business logic.
8. A message queue-based transaction processing method as claimed in claim 7, It is characterized in that If message consumption fails, the consumer will increase the number of retries for the message and mark the message status as failed.
9. A message queue-based transaction processing method as claimed in claim 7, It is characterized in that The consumer creates and starts a compensation timer task. For messages with an initialization or failure status, as well as messages older than 20 minutes, the compensation timer task will be executed regularly, querying the producer's database for records about the message and trying to reprocess these messages.
Citation Information
Cited By
Log management tracking method based on message middleware
CN120763007A
