Information synchronization and confirmation method and system and computer equipment

Abstract

The invention discloses an information synchronization and confirmation method and system and computer equipment, and relates to the technical field of distributed system message processing, and the method comprises the steps: defining a message sender; after persistently storing the message, sending the message through the MQ, and confirming to update a message state or trigger retry according to a monitoring result; the consumption end monitors and processes the message in the message queue, and determines whether to confirm the message and trigger an exception handling and retry mechanism according to a message processing result; according to a message processing result, determining to confirm successful consumption of the message or reputting the message back to the queue for retry; if the message is successfully consumed, the sender is informed of a processing result; and the message sender receives the processing result and updates the state, or actively queries the message state through a message query interface. According to the invention, the problems of repeated consumption of messages, disordered processing time sequence of lost messages and untimely and unreliable feedback of message processing states are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of distributed system message processing, and in particular to an information synchronization and confirmation method, system and computer equipment. Background Art

[0002] With the widespread use of distributed systems, different components need to interact and share data frequently. Message queues (MQ), as an important middleware, undertake key responsibilities such as decoupling applications, asynchronous processing, and peak-to-valley shifting. However, with the increase in business complexity and the expansion of user scale, MQ has exposed many problems in the information synchronization and confirmation links. The method of message processing in a distributed system is: on the sending end, use the MQ client library to create a message producer, configure the connection parameters, and then build a message containing the message body and attributes and send it to the specified topic or queue; on the receiving end, create a message consumer and set up a listener, and listen for messages after startup. Message confirmation methods include automatic confirmation and manual confirmation. However, in automatic confirmation mode, once a message is received, it will be confirmed even if a failure occurs during business processing. Although the message is confirmed, the actual processing may not be completed, resulting in data loss. In manual confirmation, although the risk of loss can be reduced, due to the complexity of the confirmation logic, if the confirmation logic is incorrect, such as successful processing but not confirmation, or repeated confirmation failure due to network delays, MQ may push messages repeatedly, seriously affecting data consistency and business accuracy.

[0003] The timing of message processing is also a prominent issue. When processing multiple messages from the same user, there is often a lack of effective sequence identifiers and processing status associations, which disrupts the message processing order. This can result in later messages being processed first, or messages that rely on the results of previous messages being processed incorrectly due to the previous messages not being completed, significantly disrupting the consistency of business logic. The message processing status feedback mechanism is also inadequate. Relying solely on MQ automatic notifications, if network fluctuations or MQ itself fails, it is difficult for the message sender to know whether the message has been processed correctly. The lack of a fallback solution leads to poor system robustness. Summary of the Invention

[0004] The purpose of the present invention is to provide an information synchronization and confirmation method, system and computer equipment for solving the problem in the existing technology of MQ information synchronization and confirmation that the automatic confirmation or manual confirmation mode of messages may lead to message loss, repeated consumption and performance impact due to improper timing coordination of message processing, transmission and confirmation.

[0005] The present invention solves the above problems through the following technical solutions:

[0006] An information synchronization and confirmation method, comprising:

[0007] Step S1, defining a message sender of a message, the message sender including a user ID, a message ID uniquely identifying the current message, a message generation time, a message ID of a previous message, and message content;

[0008] Step S2, persistently storing the message and then entering the MQ;

[0009] Step S3, sending the message through the MQ and performing listening, and confirming an update of a message state or triggering a retry according to a listening result;

[0010] Step S4, listening to the message in the message queue by a consumer, processing the message after receiving the message, and deciding whether to confirm the message and trigger an exception processing and a retry mechanism according to a message processing result, and deciding to confirm a successful message consumption or to put the message back into the queue for a retry according to the message processing result;

[0011] Step S5, if the message consumption is successful, notifying a sender of a processing result, the processing result including the message ID, a processing state of the message ID, and a previous processed message ID of the message;

[0012] Step S6, receiving the processing result by the message sender and updating a state, or actively querying the message state through a message query interface.

[0013] The application can effectively avoid message repeated consumption and guarantee data consistency by generating a unique ID for each message of each user, can solve a message loss problem by optimizing a confirmation process and combining a retry mechanism, can ensure message processing in a correct order, maintain coherence and accuracy of a business logic, and avoid a message processing timing disorder problem by using a message mark whether it is a first message and a previous message processing state, and can solve a problem that a message sender cannot know a processing result in time and accurately and a message processing state feedback is unreliable in an abnormal condition such as network failure by using a double feedback mechanism of the MQ automatic notification and the active query interface, and can improve robustness of the system.

[0014] Further, the step S4 specifically includes:

[0015] Message pulling and processing: connecting to a specified message queue, declaring the queue, listening to the message in the queue, processing the message after receiving the message, and deciding whether to confirm the message or retry according to a processing result;

[0016] Confirmation and callback processing: calling a confirmation method according to a message processing result after processing the message, and confirming successful message consumption or putting the message back into the queue for a retry;

[0017] Abnormality processing and retry mechanism: capture the abnormality occurred in the message processing, record the failure reason and trigger the retry mechanism, if the message exceeds the maximum retry times, put it into the dead letter queue or record the alarm.

[0018] Further, the processing of the message comprises:

[0019] 1) Check if the message has been consumed, if yes, discard the message, otherwise, store the message in the consumption table;

[0020] 2) Check the status of the previous message, if the previous message has not been processed, continue to process the current message, otherwise, wait for the previous message to be processed before execution;

[0021] 3) Perform business processing of the message, after successful processing, update the message status to completed, if the message processing fails, trigger the compensation mechanism and alarm.

[0022] Further, the triggering of the compensation mechanism comprises: automatic retry and manual intervention, automatic retry is to repeat steps 2) and 3) for several times; manual intervention is to manually repair when the automatic retry exceeds the maximum retry times.

[0023] Further, the alarm comprises recording the abnormal message and notifying through email and SMS.

[0024] Further, the method of actively querying the message status through the message query interface is to add a message query interface and provide an API for the message sender to query the message status.

[0025] Further, the persistent storage of the message in step S2 is to save the message to the message storage table.

[0026] Further, step S3 specifically comprises:

[0027] Send the message to the designated exchange and routing key;

[0028] Listen to whether the message reaches the exchange, listen to whether the message is successfully routed, update the message status or trigger the retry according to the callback result;

[0029] When an exception is captured, the failure reason is recorded, the retry mechanism is triggered, and after exceeding the maximum retry times, the failed message is persisted and an alarm is given.

[0030] An information synchronization and confirmation system comprises:

[0031] A message sender definition module for defining the message sender of the message, the message sender comprising a user ID, a message ID uniquely identifying the current information, a message generation time, a message ID of the previous message, and message content;

[0032] a storage module for persistently storing the message before the message enters the present;

[0033] a message sending and monitoring module for sending the message through the MQ and monitoring, and confirming to update the message state or triggering the retry according to the monitoring result;

[0034] a message processing and retry module for monitoring the message in the message queue through the consumption end, processing the message after receiving the message, and deciding whether to confirm the message and triggering the abnormal processing and the retry mechanism according to the message processing result; and deciding to confirm the message consumption success or putting back into the queue for retry according to the message processing result;

[0035] a processing result notification module for notifying the sender of the processing result when the message is consumed successfully, and the processing result includes the message ID, the processing state of the message ID and the last processed message ID of the message;

[0036] a message state module for updating the state after the message sender receives the processing result, or actively inquiring the message state through the message inquiry interface.

[0037] A computer device comprising a memory and a processor, the memory stores a computer program, and the processor implements the steps of the method when executing the computer program.

[0038] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0039] (1) The present application constructs a set of fine message management mechanism based on user dimension: on the one hand, a unique ID is generated for each message of each user, which is different from the conventional general message ID generation method, accurately avoids the repeated consumption caused by message confusion in multi-user scenario, ensures the accuracy and uniqueness of message processing, reduces the message loss situation combined with the retry mechanism; on the other hand, the message time sequence association mark is introduced, not only the first message is marked, but also the non-first message carries the processing state of the last message, which greatly enhances the logicality and continuity of message processing, which is different from the ordinary disordered processing mode. Ensure the message processing in order, maintain the logical continuity between messages, maintain the accuracy of business logic, solve the problem of message processing time sequence disorder.

[0040] (2) The present application combines the double insurance feedback mode of MQ automatic notification and active query interface, which not only guarantees real-time performance by means of MQ automatic notification, but also enhances the reliability of message processing state feedback and improves the robustness of the system by taking the active query interface as a bottom line, solves the problem that the message sender cannot know the processing result in time and accurately under abnormal conditions such as network failure, resulting in unreliable message processing state feedback. It not only guarantees real-time performance but also enhances fault tolerance, which is significantly different from the conventional scheme of single dependence on a certain notification method. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 The flowchart of the present application. DETAILED DESCRIPTION

[0042] The present application will be further described in detail below in conjunction with examples, but the embodiments of the present application are not limited thereto.

[0043] Example 1:

[0044] In conjunction with the accompanying Figure 1 The information synchronization and confirmation method comprises:

[0045] Step S1: Define the message sender. Before message transmission, a standardized message sender is defined to ensure traceability and unique identification of the message. The message body contains:

[0046] 1. User ID: Identifies the user to whom the message belongs.

[0047] 2. Message ID: Uniquely identifies the current message to prevent duplicate consumption.

[0048] 3. Message generation time: Records the creation time of the message to ensure time sequence consistency.

[0049] 4. Last message ID: Used to query the processing status of the last message to ensure orderly processing.

[0050] 5. Message content: Actual business data for parsing and processing by the consumer.

[0051] Step S2: Message persistent storage. Before the message enters the MQ, it is first stored in the message storage table to record the basic information of the message for subsequent query and recovery. In this way, when the MQ is abnormal or the service is restarted, the unfinished message can be recovered through the database to prevent message loss.

[0052] Step S3: After message storage, the message is sent through the MQ to realize asynchronous processing and decoupling. The sending module realizes as follows:

[0053] 1. Define the message sending body to the specified exchange and routing key. sendMessage(exchange, routingKey, message, UUID.randomUUID().toString());

[0054] 2. Confirm and callback processing: listen to whether the message arrives at the exchange, listen to whether the message is successfully routed, update the message state or trigger retry according to the callback result. onConfirmCallback(messageId, isSuccess); onReturnCallback(messageId, reason);

[0055] 3. Exception handling and retry mechanism: capture sending exceptions, record failure reasons, trigger retry mechanism, set the maximum number of retries, and after exceeding the maximum number of retries, persist the failed message and alarm. retryIfFailed(messageId, maxRetries);

[0056] Step S4: Create a message processing module, and the consumer pulls messages from the MQ and parses them. The processing module is implemented according to the following steps:

[0057] 1. Message pulling and processing: connect to the specified message queue, declare the queue, listen to the messages in the queue, process the received messages, and decide whether to confirm the message or retry according to the processing result. channel.basicConsume(queueName, false, new DefaultConsumer(channel) {...});

[0058] 2. Confirmation and callback processing: after processing the message, call the confirmation method according to the message processing result to confirm the successful consumption of the message or put it back into the queue for retry.

[0059] channel.basicAck(envelope.getDeliveryTag(), false);

[0060] channel.basicReject(envelope.getDeliveryTag(), true);

[0061] 3. Exception handling and retry mechanism: capture exceptions occurring during message processing, record failure reasons and trigger retry mechanism, and if the message exceeds the maximum number of retries, put it into the dead letter queue or record an alarm. retryIfFailed(messageId, maxRetries);

[0062] Step S5: Message idempotency check, after receiving the MQ message, first check if the message has been processed:

[0063] 1. Query the consumption record by message ID, determine if it has been processed.

[0064] 2. If consumed, discard the message to prevent repeated processing.

[0065] 3. If not consumed, store it in the consumption table and mark it as "processing", then proceed to the next step.

[0066] Step S6: Check the status of the previous message to ensure ordered processing, query the processing status of the previous message ID from the database:

[0067] 1. If the status of the previous message is "processed", continue processing the current message.

[0068] 2. If the status of the previous message is "unprocessed", wait until the previous message is completed before executing, to ensure sequentiality.

[0069] Step S7: Business processing and transaction management, start the actual business processing and ensure data consistency:

[0070] 1. Update the message status to "processing" and record the business log.

[0071] 2. Execute specific business logic, such as database updates, remote calls, etc.

[0072] 3. Based on Spring framework development, use Spring's transaction management mechanism (such as @Transactional annotation) in the executed business operations to ensure transaction consistency, ensure that business data and message state changes are atomic operations, prevent partial failure from causing data inconsistency.

[0073] 4. After successful processing, update the message status to "completed".

[0074] 5. When processing fails, mark it as "failed" and trigger the compensation mechanism.

[0075] Step S8: Exception handling and compensation mechanism, if message processing fails, trigger the compensation mechanism to avoid business interruption:

[0076] 1. Automatic retry: messages that fail processing support multiple reprocessing, repeat steps S6 and S7 to avoid temporary failure.

[0077] 2. Manual intervention: if the maximum number of retries is exceeded, notify the operation and maintenance personnel for manual repair.

[0078] 3. Alarm system: record abnormal messages and notify the responsible person by email, SMS, etc.

[0079] Step S9: The processing result is notified to the sender. After the message processing is completed, the processing result is notified to the sender, and the message body contains:

[0080] 1. Message ID: used to uniquely identify the current message.

[0081] 2. Processed message ID: identifies the last processed message of the message.

[0082] 3. Processing status: processed / not processed / processing failure.

[0083] Step S10: The message sender receives the processing result and updates the status. After the message sender receives the processing result, the following operations are performed:

[0084] 1. Update the local message status to ensure that the message status is consistent with the business processing.

[0085] 2. For messages with processing failure, manually mark and enter the exception processing process.

[0086] Step S11: Add a message query interface to provide an API that allows the message sender to actively query the message status, including:

[0087] 1. Query the current message status (unprocessed / processing / processing success / processing failure).

[0088] 2. Query the failure reason to facilitate remedial measures.

[0089] 3. Query the business processing result to ensure that the business status and message status are synchronized.

[0090] The present application has the following advantages:

[0091] Reliable message delivery: before entering the MQ, the message is first stored persistently to prevent message loss due to MQ abnormalities, and a MQ sending retry mechanism is introduced to ensure successful delivery;

[0092] Message consumption processing: the consumer uses concurrent processing to improve throughput, queries the consumption table to ensure idempotency, avoids duplicate consumption, checks the state of the last message to ensure business execution order, and combines transaction consistency mechanism to prevent data inconsistency problems;

[0093] Processing result feedback: the consumer notifies the sender in a timely manner after the message processing is completed, and feeds back the processing result status (processed, failed, not processed, etc.), to ensure that the sender can correctly track the message execution and perform subsequent processing;

[0094] Improve the reliability of message status feedback: MQ automatic notification can push the status immediately when the message processing is completed. The active query interface provides a backup guarantee in abnormal situations such as MQ notification failure, ensuring that the message sender can obtain accurate message processing status at any time, greatly reducing the lack of status feedback and enhancing system robustness.

[0095] Ensure business process continuity: The message sender is informed of the processing results in a timely manner and can proceed with subsequent business operations in an orderly manner based on the status, avoiding business stagnation or incorrect execution due to unclear status.

[0096] Enhanced system stability and scalability: The decoupling feature of MQ automatic notification enables independent operation of various system components, reducing coupling and facilitating expansion and maintenance. The active query interface is independent of MQ and can still function normally even if it fails, ensuring stable system operation. The combination of the two can flexibly adapt to different business scales and scenarios, improving the overall stability and scalability of the system.

[0097] Message query and monitoring: Provides a message query interface to support the sender's active query status. Combined with scheduled tasks to scan timed-out and unprocessed messages, it also achieves traceability of message processing through logging and real-time monitoring. It also triggers alarms in abnormal situations to ensure stable system operation.

[0098] Exception handling and compensation mechanism: The system supports automatic retry of failed messages to prevent transient exceptions from affecting business. Manual intervention is triggered after the maximum number of retries is exceeded. Unprocessed messages are scanned through scheduled tasks, and active compensation or alarm reminders are issued to ensure message reliability.

[0099] Example 2:

[0100] An information synchronization and confirmation system, comprising:

[0101] A message body definition module is used to define the message body of a message, wherein the message body includes a user ID, a message ID that uniquely identifies the current message, a message generation time, a message ID of a previous message, and message content;

[0102] The storage module is used to store messages persistently before they enter the current state.

[0103] The message sending and monitoring module is used to send messages through MQ and monitor them, confirm and update the message status or trigger a retry based on the monitoring results;

[0104] The message processing and retry module is used to monitor messages in the message queue through the consumer end, process the message after receiving it, and then decide whether to confirm the message and trigger the exception handling and retry mechanism based on the message processing result; and decide whether to confirm the successful consumption of the message or put it back into the queue for retry based on the message processing result;

[0105] a processing result notification module, configured to send a processing result to the sender when the message consumption is successful, the processing result comprising: a message ID, a processing status of the message ID, and a last processed message ID of the message;

[0106] a message status module, configured to update the status after the processing result is received by the message sender, or actively query the message status through a message query interface.

[0107] Embodiment 3

[0108] A computer device, comprising a memory and a processor, the memory stores a computer program, and the processor implements the steps of the method in embodiment 1 when executing the computer program.

[0109] Although the present application has been described with reference to explanatory embodiments thereof, the foregoing embodiments are merely meant to be the best mode of the present application, and the embodiments of the present application are not limited to the foregoing embodiments. It should be understood that those skilled in the art can design many other modifications and embodiments, which will fall within the scope and spirit of the principles disclosed in the present application.

Claims

1. A method for information synchronization and confirmation, characterized in that: include: Step S1: define the message body of the message, wherein the message body includes the user ID, the message ID that uniquely identifies the current message, the message generation time, the message ID of the previous message, and the message content; Step S2: store the message persistently and then enter MQ; Step S3: Send a message via MQ and monitor it. According to the monitoring result, update the message status or trigger a retry. Step S4: The consumer monitors the messages in the message queue, processes the messages after receiving them, and decides whether to confirm the messages and trigger the exception handling and retry mechanism based on the message processing results; and decides whether to confirm the message consumption is successful or put it back into the queue for retry based on the message processing results; Step S5: If the message is consumed successfully, the sender is notified of the processing result, which includes: the message ID, the processing status of the message ID, and the ID of the previous processed message; Step S6: The message sender receives the processing result and updates the status, or actively queries the message status through the message query interface.

2. The information synchronization and confirmation method according to claim 1, characterized in that: The step S4 specifically includes: Message pulling and processing: connect to the specified message queue, declare the queue, listen for messages in the queue, process the messages after receiving them, and decide whether to confirm the message or retry based on the processing results; Confirmation and callback processing: After processing the message, the confirmation method is called according to the message processing result to confirm that the message consumption is successful or put it back into the queue for retry; Exception handling and retry mechanism: Capture exceptions that occur during message processing, record the cause of failure and trigger the retry mechanism. If the message exceeds the maximum number of retries, it will be placed in the dead letter queue or record an alarm.

3. The information synchronization and confirmation method according to claim 2, characterized in that: The message processing includes: 1) Check whether the message has been consumed. If so, discard the message; otherwise, store the message in the consumption table. 2) Check the status of the previous message. If the previous message has not been processed, continue processing the current message. Otherwise, wait until the previous message is processed before executing again. 3) Perform business processing on the message. If the processing is successful, update the message status to completed. If the message processing fails, trigger the compensation mechanism and issue an alarm.

4. The information synchronization and confirmation method according to claim 3, characterized in that: The trigger compensation mechanism includes: automatic retry and manual intervention. Automatic retry is to repeat 2) and 3) several times; manual intervention is to perform manual repair when the automatic retry exceeds the maximum number of retries.

5. The information synchronization and confirmation method according to claim 3, characterized in that: The alarm includes recording abnormal messages and notifying via email or text message.

6. The information synchronization and confirmation method according to claim 1, characterized in that: The method for actively querying the message status through the message query interface is: adding a message query interface and providing an API for the message sender to query the message status.

7. The information synchronization and confirmation method according to claim 1, characterized in that: The persistent storage of the message in step S2 is to save the message to a message storage table.

8. The information synchronization and confirmation method according to claim 1, characterized in that: The step S3 is specifically as follows: Send the message to the specified exchange and routing key; Monitor whether the message reaches the switch, monitor whether the message is successfully routed, and update the message status or trigger a retry based on the callback result; When a sending exception is captured, the failure reason is recorded and the retry mechanism is triggered. After the maximum number of retries is exceeded, the failure message is persisted and an alarm is issued.

9. An information synchronization and confirmation system, characterized in that: include: A message body definition module is used to define the message body of a message, wherein the message body includes a user ID, a message ID that uniquely identifies the current message, a message generation time, a message ID of a previous message, and message content; The storage module is used to store messages persistently before they enter the current state. The message sending and monitoring module is used to send messages through MQ and monitor them, confirm and update the message status or trigger a retry based on the monitoring results; The message processing and retry module is used to monitor messages in the message queue through the consumer end, process the messages after receiving them, and then decide whether to confirm the messages and trigger the exception handling and retry mechanism based on the message processing results; And based on the message processing results, decide whether to confirm that the message consumption is successful or put it back into the queue for retry; The processing result notification module is used to notify the sender of the processing result when the message consumption is successful. The processing result includes: message ID, processing status of the message ID, and the ID of the previous processed message of the message; The message status module is used to update the status after the message sender receives the processing result, or actively query the message status through the message query interface.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images