Idempotent distributed transaction processing method and system based on MQ message retry mechanism

Abstract

The invention provides an idempotent distributed transaction processing method and system based on an MQ message retry mechanism, and the method comprises the steps: S1, designing a message table, and carrying out the idempotent of message processing through combining a main key message id and a message state of the message table; s2, designing business logic of idempotent based on idempotent; and S3, designing a timed task to pull expired data in the message table at set intervals to perform deletion operation. According to the method disclosed by the invention, the idempotence during message processing is realized by newly adding the message table and combining the message id and the message state of the message table, and the message id is a unique primary key.

Description

Technical Field

[0001] The present invention relates to the field of computer information technology, and more specifically, to an idempotent distributed transaction processing method and system based on an MQ message retry mechanism. Background Art

[0002] We often use message middleware MQ in development. The advantage of using message middleware is that it can solve the problem of message transmission between distributed systems. It can shield the characteristics between various platforms and protocols and realize collaboration between applications.

[0003] When using message middleware, we may encounter the following two situations:

[0004] 1. Due to network reasons, the message MQ sent by the producer has been received, but the response to the service has timed out, causing the server to deliver the same message again. In this way, there are two identical messages in the message queue, and downstream consumers need to consume the same information twice.

[0005] 2. When the consumer consumes the message with id=1 for the first time, the consumption is successful, but the response to MQ times out, causing MQ to think that the current message consumer has not successfully consumed it. After a period of time, it is redelivered to the consumer for consumption, resulting in the consumer consuming the same message twice.

[0006] During the development process, we need to handle the above two situations to ensure the idempotence of consumer consumption, so that the consumer consumes the same message multiple times and the final result returned is the same. Summary of the invention

[0007] The present invention aims at the technical problems existing in the prior art and provides an idempotent distributed transaction processing method and system based on an MQ message retry mechanism, thereby ensuring the idempotence of message processing.

[0008] According to a first aspect of the present invention, an idempotent distributed transaction processing method based on an MQ message retry mechanism is provided, comprising the following steps:

[0009] Step S1, design a message table, and perform idempotence of message processing by combining the primary key message id and message status of the message table;

[0010] Step S2: Design idempotent business logic based on idempotence;

[0011] Step S3: Design a scheduled task to pull out expired data from the message table at regular intervals and delete it.

[0012] Based on the above technical solution, the present invention can also make the following improvements.

[0013] Optionally, the idempotence of processing a message in combination with the primary key message ID and the message status of the message table includes:

[0014] Design a message table with the primary key of message id. Before consumers pull messages from MQ for consumption, add a record to the table and set the message status to processing;

[0015] If the addition is successful, the local business of the message is completed first, and then the remote calling system is notified;

[0016] If the addition fails, query the existing records in the message table based on the message ID, obtain the message status and process it according to the status value.

[0017] Optionally, if the status is 0, it means that the message is being consumed. In this case, a message consumption failure is returned to MQ and the message is pushed again.

[0018] If the status is 1, it means that the message consumption is successful, and a message indicating successful consumption is returned to MQ.

[0019] Optionally, if the notification is successful, the message status in the message table is changed to 1, and finally MQ is notified that the message consumption is successful;

[0020] If the notification fails, the business is rolled back, the message record is deleted, and finally MQ is notified that the message consumption has failed, waiting for the next push to continue processing the business.

[0021] Optionally, the idempotence of performing message processing in combination with the primary key message ID and the message status of the message table includes:

[0022] When consumers pull messages from MQ for consumption, they first check whether there is a record with message id msg_id in the message table;

[0023] If there is no record, it means that the message is consumed for the first time, and the subsequent logic will proceed normally;

[0024] If there is a record, it means that the message has been consumed. At this time, the consumption status is judged based on the message status in the record.

[0025] Optionally, judging the consumption situation according to the message status in the record includes:

[0026] When the message status is 0, it means that the message is being consumed. In this case, the subsequent logic is skipped and the consumption failure is directly returned to MQ. When the message status is 1, it means that the message is consumed successfully. In this case, the subsequent logic is skipped and the consumption success is directly returned to MQ.

[0027] Optionally, the business logic of idempotency design based on idempotency includes:

[0028] Design a business flow table. Each time a request is processed, first check whether the same request serial number exists in the table. If it exists, return the result of the last processing. If not, perform normal business logic processing.

[0029] Optionally, the designing of a scheduled task to pull expired data from the message table at regular intervals for deletion includes:

[0030] Design a scheduled task to query all records in the message table with a message status of 1 at intervals, put the message id, message receiving time and message timeout time corresponding to each record into the list collection, traverse the list collection to calculate the difference between the message receiving time and the current system time of each message object, and when the time difference is greater than or equal to the timeout time of the message object, delete the record corresponding to the object in the message table.

[0031] According to a second aspect of the present invention, an idempotent distributed transaction processing system based on an MQ message retry mechanism is provided, comprising:

[0032] The producer module is used to send messages to the message queue;

[0033] The message middleware module acts as an intermediary between producers and consumers and is responsible for receiving, storing, and distributing messages to appropriate consumers.

[0034] The consumer module is used to read messages from the message queue and process them;

[0035] The database module is used to record the status flow of messages during normal consumption and clear message records in the case of dead letters.

[0036] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, an idempotent distributed transaction processing method based on the MQ message retry mechanism as described above is implemented.

[0037] Technical effects and advantages of the present invention:

[0038] The present invention provides an idempotent distributed transaction processing method and system based on an MQ message retry mechanism, which realizes idempotence in message processing by adding a message table with a message ID as a unique primary key and combining the message ID and message status of the message table.

[0039] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A flow chart of an idempotent distributed transaction processing method based on an MQ message retry mechanism provided in an embodiment of the present invention;

[0041] Figure 2 A schematic diagram of the hardware structure of a possible electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] It should be noted that the embodiments of the present invention address the problems in the prior art where after MQ receives a message, the response server times out, causing the server to repeatedly deliver the same message, and the response MQ times out after the consumer successfully consumes the message, causing MQ to re-deliver the same information to the consumer.

[0044] It is understandable that based on the defects in the background technology, the embodiment of the present invention proposes an idempotent distributed transaction processing method based on the MQ message retry mechanism, as shown in the following example: Figure 1 As shown, the following steps are included:

[0045] Step S1: Design a message table, and perform idempotence of message processing by combining the primary key message id and message status of the message table.

[0046] In this implementation step, the idempotence of message processing in combination with the primary key message ID and message status of the message table includes:

[0047] When consumers pull messages from MQ for consumption, they first insert a record with the message id msg_id into the message table (the primary key of the message table is the message id). If the insertion is successful, it means that the message is consumed for the first time, and the subsequent business logic will proceed normally. If the insertion fails (note that the failure here should be a database primary key conflict), that is, there is already a record with the message id msg_id in the message table, which means that the message has been consumed. At this time, the message table is queried according to msg_id to obtain the message status and process it according to the message status. When the message status is 0 (processing), it means that the message is being consumed. At this time, no business logic processing is performed and consumption failure is directly returned to MQ; when the message status is 1 (successful), it means that the message is consumed successfully and consumption success is directly returned to MQ without performing task business logic processing.

[0048] Therefore, first, design a message table whose primary key is the message id. Before consumers pull messages from MQ for consumption, they add a record to the table and set the message status to 0 (processing).

[0049] If the addition is successful, the local business of the message is completed first, and then the remote call system is notified. If the notification is successful, the message status in the message table is changed from being processed to successful, and finally the message consumption success is returned to MQ; if the notification fails, the business is rolled back, the message record is deleted, and finally the message consumption failure is returned to MQ, waiting for the next push to continue processing the business.

[0050] If the addition fails, query the existing records in the message table based on the message ID, obtain the message status and process it according to the status value;

[0051] If the status is 0, it means that the message is being consumed. At this time, the message consumption failure is returned to MQ and waits for the next push;

[0052] If the status is 1, it means that the message consumption is successful, and the message consumption success is returned to MQ.

[0053] Step S2: Design idempotent business logic;

[0054] It should be noted that when consumers pull messages from MQ for consumption, new records are added successfully and local business has been completed, and the remote call system is notified. The remote call system needs to design a set of logic to ensure the idempotence of business logic processing after receiving the notification, that is, the results returned by receiving the same request multiple times remain consistent.

[0055] Therefore, the business logic of designing idempotence specifically includes:

[0056] Design a business flow table. Each time a request is processed, first check whether there is the same request flow number in the table. If it exists, return the result of the last processing. If not, perform normal business logic processing. In this way, the idempotence of business logic can be achieved based on the query idempotence of the database.

[0057] Step S3: Design a scheduled task to pull out expired data from the message table at regular intervals and delete it.

[0058] It should be noted that when consumers pull messages from MQ for consumption, the first message is being processed, but for some reason it is processed too slowly and eventually fails. When processing the second message with the same id, the add operation continues, but fails, and the retry logic continues. After a certain number of retries, it will be added to the dead letter queue. In response to this situation, we need to design a scheduled task to pull expired data from the message table at regular intervals for deletion to prevent the occurrence of dead letters.

[0059] Therefore, the design of the scheduled task to pull the expired data in the message table at regular intervals for deletion specifically includes:

[0060] Design a scheduled task to query all records in the message table with a message status of 1 (processing) every three minutes, put the message ID, message receiving time and message timeout time corresponding to each record into the list collection, traverse the list collection to calculate the difference between the message receiving time and the current system time of each message object, and when the time difference is greater than or equal to the timeout time of the message object, delete the record corresponding to the object in the message table. This can prevent the problem of dead letters.

[0061] The following is a further description with reference to specific embodiments.

[0062] First, add a message table whose fields include message id, message description, message timeout, message consumption status and message receiving time.

[0063]

[0064]

[0065] Before a consumer pulls a message from the MQ queue and starts executing business logic, it first inserts a piece of data into the message table.

[0066] insert message(msg_id,msg_desc,msg_exprire_time,msg_status,msg_req_time)

[0067] values(1,'MQ message',5,0,sysdate)

[0068] Process the corresponding logic according to the message results entered into the message table.

[0069] The message table is successfully entered. First, complete the local business of the message, and then use RPC to notify other systems. When the notification to other systems is successful, change the data status of the message table to consumption success and then notify MQ that the message consumption is successful.

[0070] Update message set msg_status='1'where msg_id='5'

[0071] When notifying other systems fails, the local business of the message is rolled back and the record just added in the message table is deleted, and then the MQ message consumption is notified and the business processing is continued after the next push.

[0072] delete from message where msg_id='5'

[0073] Failure to enter a message into the message table indicates that the message id is msg_id and is being processed. At this time, the record corresponding to the message is found based on msg_id, and then subsequent processing is performed based on the value of the message status msg_status.

[0074] When the status value is 0, it means that the message is being consumed. At this time, it is necessary to return to MQ that the message consumption failed and wait for the next push; when the status value is 1, it means that the message consumption was successful, and at this time, it is returned to MQ that the message consumption was successful.

[0075] In actual applications, there may be a situation where the message with msg_id 1 is being processed. For some reason, the consistent processing is very slow, resulting in the failure of the message consumption, but the data in the message table is not deleted. Later, when the consumer receives the message with msg_id 1 re-pushed by MQ, it will go through the operation of adding data to the message table first. At this time, the message table fails to add data, and then the retry logic will be continuously executed. After a certain number of retries, it will be added to the dead letter queue. In this case, we need to clean up these expired data regularly to ensure that the message can be consumed normally. We can start a scheduled task to pull expired message data from the database every 3 minutes, and then delete it, which can prevent the problem of dead letters.

[0076] According to a second aspect of the present invention, an idempotent distributed transaction processing system based on an MQ message retry mechanism is provided, the system comprising a producer module, a message middleware module, a consumer module, a database module and other system modules; wherein:

[0077] The producer module is responsible for sending messages to the message queue;

[0078] The message middleware module acts as an intermediary between producers and consumers, responsible for receiving, storing, and distributing messages to appropriate consumers;

[0079] The consumer module is responsible for reading messages from the message queue and processing them;

[0080] The database module is responsible for recording the state flow of messages during normal consumption and clearing message records in the case of dead letters;

[0081] After receiving notifications from consumers, other system modules process specific business logic and return notification results.

[0082] It can be understood that the idempotent distributed transaction processing system based on the MQ message retry mechanism provided by the present invention corresponds to the idempotent distributed transaction processing method based on the MQ message retry mechanism provided by the aforementioned embodiments. The relevant technical features of the idempotent distributed transaction processing system based on the MQ message retry mechanism can refer to the relevant technical features of the idempotent distributed transaction processing method based on the MQ message retry mechanism, which will not be repeated here.

[0083] The embodiment of the present invention further provides an electronic device, such as Figure 2 As shown, the electronic device includes: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other through the communication bus 840. The processor 810 can call the logic instructions in the memory 830 to execute the implementation steps of the idempotent distributed transaction processing method based on the MQ message retry mechanism.

[0084] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0085] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0086] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

[0087] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An idempotent distributed transaction processing method based on MQ message retry mechanism, characterized in that: The following steps are involved: Step S1, design a message table, and perform idempotence of message processing by combining the primary key message id and message status of the message table; Step S2: Design idempotent business logic based on idempotence; Step S3: Design a scheduled task to pull out expired data from the message table at regular intervals and delete it.

2. The idempotent distributed transaction processing method based on the MQ message retry mechanism according to claim 1 is characterized in that: The idempotence of message processing in combination with the primary key message ID and message status of the message table includes: Design a message table with the primary key of message id. Before consumers pull messages from MQ for consumption, add a record to the table and set the message status to processing; If the addition is successful, the local business of the message is completed first, and then the remote calling system is notified; If the addition fails, query the existing records in the message table based on the message ID, obtain the message status and process it according to the status value.

3. The idempotent distributed transaction processing method based on the MQ message retry mechanism according to claim 2 is characterized in that: If the status is 0, it means that the message is being consumed. At this time, the message consumption failure is returned to MQ and waits for the next push; If the status is 1, it means that the message consumption is successful, and the message consumption success is returned to MQ.

4. The idempotent distributed transaction processing method based on the MQ message retry mechanism according to claim 3 is characterized in that: If the notification is successful, the message status in the message table is changed to 1, and finally MQ is notified that the message consumption is successful; If the notification fails, the business is rolled back, the message record is deleted, and finally MQ is notified that the message consumption has failed, waiting for the next push to continue processing the business.

5. The idempotent distributed transaction processing method based on the MQ message retry mechanism according to claim 1, characterized in that: The idempotence of message processing in combination with the primary key message ID and message status of the message table includes: When consumers pull messages from MQ for consumption, they first check whether there is a record with message id msg_id in the message table; If there is no record, it means that the message is consumed for the first time, and the subsequent logic will proceed normally; If there is a record, it means that the message has been consumed. At this time, the consumption status is judged based on the message status in the record.

6. The idempotent distributed transaction processing method based on the MQ message retry mechanism according to claim 5 is characterized in that: The determining of the consumption situation according to the message status in the record includes: When the message status is 0, it means that the message is being consumed. In this case, the subsequent logic is skipped and the consumption failure is directly returned to MQ. When the message status is 1, it means that the message is consumed successfully. In this case, the subsequent logic is skipped and the consumption success is directly returned to MQ.

7. The idempotent distributed transaction processing method based on the MQ message retry mechanism according to claim 1 is characterized in that: The business logic based on idempotent design includes: Design a business flow table. When processing each request, first check whether the same request serial number exists in the table. If it does, return the result of the last processing. If not, perform normal business logic processing.

8. The idempotent distributed transaction processing method based on the MQ message retry mechanism according to claim 1, characterized in that: The design of a scheduled task to pull expired data from the message table at regular intervals for deletion includes: Design a scheduled task to query all records in the message table with a message status of 1 at intervals, put the message id, message receiving time and message timeout time corresponding to each record into a list collection, traverse the list collection to calculate the difference between the message receiving time and the current system time of each message object, and when the time difference is greater than or equal to the timeout time of the message object, delete the record corresponding to the object in the message table.

9. An idempotent distributed transaction processing system based on MQ message retry mechanism, characterized in that: include: The producer module is used to send messages to the message queue; The message middleware module acts as an intermediary between producers and consumers and is responsible for receiving, storing, and distributing messages to appropriate consumers. The consumer module is used to read messages from the message queue and process them; The database module is used to record the status flow of messages during normal consumption and clear message records in the case of dead letters.

10. An electronic device, characterized in that: It comprises a memory and a processor, wherein the processor is used to implement the idempotent distributed transaction processing method based on the MQ message retry mechanism as described in any one of claims 1 to 7 when executing the computer program stored in the memory.

Images