Asynchronous message processing method and device and storage medium
By introducing asynchronous message processing methods into the message middleware, the problem of message failure, loss and imperfect idempotence retry mechanism in different business scenarios is solved, reliable message delivery and idempotence consumption are realized, and flexible configuration and downgrade solutions are provided.
Patent Information
- Application Number
- CN202510077741.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-17
- Publication Date
- 2025-05-16
AI Technical Summary
In the prior art, message middleware lacks a general asynchronous retry mechanism and heterogeneous downgrade mechanism in different business scenarios, resulting in incomplete message failure, loss and idempotent retry mechanisms.
A method of asynchronous message processing is proposed, including receiving and processing asynchronous messages, updating local asynchronous records, scanning records to determine whether to initiate retry or apply for asynchronous threads based on the availability of message middleware.
It realizes reliable delivery, unique consumption and idempotent consumption of messages, builds a common timeout and failure retry mechanism, supports personalized configuration parameters, and provides a local thread pool downgrade solution when message middleware exceptions.
Smart Images
Figure CN120011102A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of asynchronous communication technology, and in particular to an asynchronous message processing method, device and storage medium. Background Art
[0002] Message middleware is a commonly used software for message transmission between distributed systems. It provides a set of asynchronous communication mechanisms, which enables each service to run independently without direct calls, effectively helping to decouple systems, thereby improving the scalability and fault tolerance of microservice systems.
[0003] Microservice architecture has become the mainstream of current system construction, and the modular splitting of the system is becoming more refined. This architecture can largely ensure the high availability of the system, making each module independent of each other, and the failure of any one of them can minimize the impact on the overall availability of the system. Especially for important and complex systems, the division of labor of microservices is clearer. Based on this, communication between systems is essential. The emergence of message middleware reduces the communication cost between systems. It is widely used in scenarios such as asynchronous calls, decoupling and traffic peak shaving between microservices.
[0004] Under the current circumstances, there is a lack of universal asynchronous retry mechanism and heterogeneous degradation mechanism for common message failure, loss and idempotent retry mechanism in message middleware in different business scenarios.
[0005] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention
[0006] The main purpose of this application is to provide an asynchronous message processing method, device and storage medium, aiming to solve the common message failure, loss and idempotent retry mechanism of message middleware in different business scenarios, and the technical problems of lack of universal asynchronous retry mechanism and heterogeneous degradation mechanism.
[0007] To achieve the above object, the present application proposes an asynchronous message processing method, the method comprising:
[0008] Receiving asynchronous messages and local asynchronous records sent by the first business system;
[0009] Processing the asynchronous message, updating the local asynchronous record according to the processing result and returning it to the first business system;
[0010] Scan the local asynchronous records to determine whether there are timeout, unprocessed or failed records;
[0011] If there are timeouts or failure records, determine whether the message middleware is available;
[0012] If the message middleware is available, an asynchronous message retry is initiated; otherwise, an asynchronous thread of the second business system is requested.
[0013] In one embodiment, the step of processing the asynchronous message includes:
[0014] Transferring the asynchronous message from a pending state to a processing state, and locking the asynchronous message;
[0015] Waiting for the asynchronous message to complete the processing of the status, transferring the asynchronous message to a failure or success state according to the processing completion result, and locking the asynchronous message;
[0016] If it is detected that the processing completion result is a failure state, the asynchronous message is transferred to a pending state and retried;
[0017] If it is detected that there are other asynchronous messages when the asynchronous message is in the processing state, the concurrent consumption control strategy is triggered to discard the other asynchronous messages;
[0018] If it is detected that other asynchronous information exists when the asynchronous message is in a success or failure state, the idempotent consumption control strategy is triggered to discard the other asynchronous messages.
[0019] In one embodiment, the step of updating the local asynchronous record according to the processing result and returning it to the first business system includes:
[0020] Obtain the exception message according to the processing result, and update the local asynchronous record to the second business system;
[0021] Sending an abnormal message receipt confirmation to the first business system through the second business system;
[0022] Update local asynchronous records by calling the API provided by the first business system;
[0023] Sending an update request to the first business system and updating the local asynchronous record status of the first business system;
[0024] After updating the local asynchronous record of the first business system, the first business system returns a confirmation response to the second business system, indicating that the update operation is completed.
[0025] In one embodiment, the step of scanning the local asynchronous records and determining whether there are timeout, unprocessed or failed records includes:
[0026] Determine whether there is a failure record in the local asynchronous record;
[0027] If there is a failure record in the local asynchronous record, determine whether the task failure caused by the asynchronous message processing exception does not exceed the maximum number of failures. If it does not exceed the maximum number of failures, convert the asynchronous message to a pending state;
[0028] If there is no failed record in the local asynchronous record, further determine whether there is a timeout and unprocessed record;
[0029] If there are timeout records that are not processed, determine whether the timeout is not processed due to a large number of asynchronous messages, insufficient asynchronous thread resources, or asynchronous message loss. Otherwise, determine whether the asynchronous message processing exceeds the preset maximum allowed time. If it exceeds the preset maximum allowed time, scan them out through a backup job and resend the asynchronous message.
[0030] In one embodiment, initiating an asynchronous message retry includes:
[0031] When an asynchronous message times out and is not processed due to an exception in the local asynchronous scheduling process or insufficient thread resources, it is transferred to the pending state and waits for the asynchronous message to be re-issued;
[0032] After the asynchronous message is reissued, it is transferred from the pending state to the processing state, and the asynchronous message is locked;
[0033] If an asynchronous message in the processing state fails, it will be retried and transferred back to the pending state. If there are sufficient thread resources, it will continue to be processed and remain in the processing state.
[0034] Through the bottom-up operation, the timed-out pending asynchronous processing is uniformly scanned, and an asynchronous message is sent to restore the pending status;
[0035] After the asynchronous message is processed from the processing state, it is transferred to the failure state or the success state according to the processing result, and the asynchronous message is locked;
[0036] It is determined whether the number of final states in the processing result reaches an upper limit, if the number of final states reaches the upper limit, the process remains in a failed state, otherwise, the process remains in a successful state.
[0037] In one embodiment, initiating an asynchronous message retry further includes:
[0038] Personalized configuration of asynchronous message processing, the personalized configuration includes:
[0039] Adjust the interval time for failed retries, the upper limit of failed retries, the maximum waiting time during processing, and the maximum waiting time to be processed.
[0040] In one embodiment, the step of applying for a second business system asynchronous thread includes:
[0041] When the message middleware is unavailable, the first business system uniformly scans the timed-out asynchronous messages to be processed through a backup job, and applies for the local thread pool of the second business system to process the asynchronous messages;
[0042] The first business system locks the asynchronous message in the to-be-processed state through the thread in the local thread pool, and continues to process and converts it into the processing state;
[0043] The first business system waits for the asynchronous message to be processed in a state process, converts the state to a success state or a failure state according to the processing result, and locks the asynchronous message.
[0044] In one embodiment, the step of processing the asynchronous message by the local thread pool of the second business system includes:
[0045] Determine the relationship between the number of threads currently running in the local thread pool of the second business system and the number of core threads;
[0046] If the number of threads currently running in the local thread pool of the second business system is less than the number of core threads, a new thread is created to execute the task;
[0047] If the number of threads currently running in the local thread pool of the second business system reaches the number of core threads, the task is added to the message queue, a blocking queue is used, and a new thread is directly created to process the task;
[0048] If the local thread pool of the second business system creates a new thread so that the currently running thread exceeds the maximum number of threads, a rejection strategy is adopted to pop up a rejection execution exception and refuse to execute the task.
[0049] In addition, to achieve the above purpose, the present application also proposes an asynchronous message processing device, the asynchronous message processing device comprising:
[0050] An information receiving module, used for receiving asynchronous messages and local asynchronous records sent by the first business system;
[0051] A message processing module, used for processing the asynchronous message, updating the local asynchronous record according to the processing result and returning the result to the first business system;
[0052] A first judgment module is used to scan the local asynchronous records to determine whether there are timeout unprocessed or failed records;
[0053] The second judgment module determines whether the message middleware is available if there is a timeout or failure record; if the message middleware is available, an asynchronous message retry is initiated, otherwise an asynchronous thread of the second business system is applied.
[0054] In addition, to achieve the above-mentioned purpose, the present application also proposes an asynchronous message processing device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the asynchronous message processing method described above.
[0055] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the asynchronous message processing method described above are implemented.
[0056] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the asynchronous message processing method described above are implemented.
[0057] One or more technical solutions proposed in this application have at least the following technical effects:
[0058] The message middleware receives the asynchronous message and local asynchronous record sent by the first business system; processes the asynchronous message, updates the local asynchronous record according to the processing result and returns it to the first business system, at which time the local asynchronous record of the first business system is updated, and the local asynchronous record can be stored in the database of the first business system and the message middleware respectively; scans the local asynchronous record to determine whether there is a timeout or failure record; if there is a timeout or failure record, determine whether the message middleware is available; if the message middleware is available, initiate an asynchronous message retry, otherwise apply for the second business system asynchronous thread. The common message failure, loss and idempotent retry mechanism of the message middleware in different business scenarios are solved, and the technical problems of the lack of a universal asynchronous retry mechanism and a heterogeneous downgrade mechanism are solved. The asynchronous scheduling control is carried out by using the unique identification method of asynchronous messages, and reliable message delivery, unique consumption and idempotent consumption are realized. A universal timeout and failure retry mechanism is constructed, and personalized configuration parameters are supported to apply to a variety of application scenarios. When the message middleware is abnormal, a downgrade solution for the local thread pool is provided. BRIEF DESCRIPTION OF THE DRAWINGS
[0059] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0060] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0061] Figure 1 A flowchart of the first embodiment of the asynchronous message processing method of the present application is provided;
[0062] Figure 2 A flowchart of the second embodiment of the asynchronous message processing method of the present application is provided;
[0063] Figure 3 A flowchart of the third embodiment of the asynchronous message processing method of the present application is provided;
[0064] Figure 4 A flowchart of the fourth embodiment of the asynchronous message processing method of the present application is provided;
[0065] Figure 5 A flowchart of the fifth embodiment of the asynchronous message processing method of the present application is provided;
[0066] Figure 6 A flowchart of the sixth embodiment of the asynchronous message processing method of the present application is provided;
[0067] Figure 7 This is a schematic diagram of the module structure of the asynchronous message processing device according to an embodiment of the present application;
[0068] Figure 8 This is a schematic diagram of the device structure of the hardware operating environment involved in the asynchronous message processing method in the embodiment of the present application.
[0069] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0070] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.
[0071] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.
[0072] In this embodiment, for the convenience of description, the following description is made by taking the identification message middleware as the execution subject.
[0073] Due to the common message failure, loss and idempotent retry mechanism of message middleware in different business scenarios in the existing technology, there is a lack of universal asynchronous retry mechanism and heterogeneous degradation mechanism.
[0074] The present application provides a solution to receive asynchronous messages and local asynchronous records sent by the first business system; process the asynchronous messages, update the local asynchronous records according to the processing results and return them to the first business system; scan the local asynchronous records to determine whether there are timeout unprocessed or failed records; if there are timeout unprocessed or failed records, determine whether the message middleware is available; if the message middleware is available, initiate an asynchronous message retry, otherwise apply for an asynchronous thread of the second business system. By adopting the asynchronous message unique identification method for asynchronous scheduling control, reliable message delivery, unique consumption and idempotent consumption are achieved, a general timeout and failure retry mechanism is constructed, personalized configuration parameters are supported to apply to a variety of application scenarios, and a downgrade solution for the local thread pool is provided when the message middleware is abnormal.
[0075] Based on this, the embodiment of the present application provides an asynchronous message processing method, which is applied to message middleware, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the asynchronous message processing method of the present application.
[0076] In this embodiment, the asynchronous message processing method includes steps S10 to S50:
[0077] Step S10: receiving an asynchronous message and a local asynchronous record sent by the first business system;
[0078] It should be noted that the asynchronous message may include information such as the data content, message type, and message priority of the message; creating a local asynchronous record in the first business system refers to recording the metadata of a message in the database or memory of the first business system, including the message content, sending time, target queue or topic, etc.; before the message middleware receives the asynchronous message sent by the first business system, it should configure the connection information of the message middleware in the first business system, including the server address, port, user name, password, etc. After the configuration is completed, the prepared asynchronous message will be sent to the specified queue or topic of the message middleware.
[0079] Step S20: Process the asynchronous message, update the local asynchronous record according to the processing result and return it to the first business system;
[0080] It should be noted that when processing the asynchronous message, the corresponding business logic is executed according to the message content and business requirements, including data processing, status update, calculation operation, etc.; according to the result of the executed business logic, the local asynchronous record of the first business system is updated.
[0081] Step S30: Scan the local asynchronous records to determine whether there are any timeout, unprocessed or failed records;
[0082] It should be noted that both the message middleware and the first business system can retrieve and traverse local asynchronous records from their storage systems to obtain the status of all messages that have been sent to the message middleware but have not yet been processed or have failed.
[0083] Step S40: If there is a timeout or failure record, determine whether the message middleware is available;
[0084] It should be noted that when the first business system retrieves and traverses local asynchronous records and finds that there are timed-out, unprocessed or failed records, the status of the message middleware can be determined by checking the connection, response time, available resources, etc. of the message middleware; situations in which the message middleware is unavailable include message middleware connection failure or response timeout, insufficient resources, message middleware failure or exception, etc.
[0085] Step S50: If the message middleware is available, an asynchronous message retry is initiated; otherwise, an asynchronous thread of the second business system is requested.
[0086] It should be noted that if the message middleware is available, try to send the asynchronous message again, such as requeuing the message, reallocating the task, etc.; the retry operation should follow the preset retry strategy and number limit to avoid infinite retries; if the message middleware is not available, apply for an asynchronous thread from the second business system, and the first business system applies to the second business system for a local thread pool through API calls or message notifications to perform a bottom-up downgrade process.
[0087] This embodiment provides an asynchronous message processing method, which is applied to the message middleware, including: receiving an asynchronous message and a local asynchronous record sent by a first business system; processing the asynchronous message, updating the local asynchronous record according to the processing result and returning it to the first business system; scanning the local asynchronous record to determine whether there is a timeout or failure record; if there is a timeout or failure record, determine whether the message middleware is available; if the message middleware is available, initiate an asynchronous message retry, otherwise apply for an asynchronous thread of the second business system. This application uses an asynchronous message unique identification method for asynchronous scheduling control, realizes reliable message delivery, unique consumption and idempotent consumption, builds a general timeout and failure retry mechanism, supports personalized configuration parameters to apply to a variety of application scenarios, and provides a downgrade solution for the local thread pool when the message middleware is abnormal.
[0088] Further, refer to Figure 2 The second embodiment of the asynchronous message processing method of the present application provides a flow chart based on the above Figure 2 In the illustrated embodiment, the step of "processing the asynchronous message" in step S20 is further refined to include steps A201 to A205:
[0089] Step A201, transferring the asynchronous message from the pending state to the processing state, and locking the asynchronous message;
[0090] It should be noted that the asynchronous message locking refers to locking the messages in the message queue to ensure that in a multi-threaded or distributed system, only one process or thread can process the message at the same time, avoiding multiple processes processing the same message at the same time, resulting in data inconsistency or duplicate processing problems.
[0091] Step A202, waiting for the asynchronous message to be processed in the state processing to be completed, transferring the asynchronous message to a failure state or a success state according to the processing completion result, and locking the asynchronous message;
[0092] It should be noted that locking the message before the state transfer can prevent other processes or threads from accessing or modifying the message at the same time during the state transfer process, thereby causing data inconsistency.
[0093] Step A203, if it is detected that the processing completion result is a failure state, the asynchronous message is transferred to a pending state for retry;
[0094] It should be noted that when a message is detected to have a failed processing result, if the retry conditions are met, the system will transfer the status of the asynchronous message from the processing status to the pending status, re-queue it or reallocate it to the processing flow, and prepare to try processing again, thereby improving the success rate of message processing and reducing the possibility of message backlog in the system.
[0095] Step A204: If it is detected that other asynchronous messages exist when the asynchronous message is in the processing state, a concurrent consumption control strategy is triggered to discard the other asynchronous messages;
[0096] It should be noted that the concurrent consumption control strategy is a mechanism for controlling concurrent consumer messages in a multi-threaded or distributed environment. In a message queue system, multiple consumers may consume messages from the same queue at the same time. The concurrent consumption control strategy is used to avoid multiple consumers processing the same message at the same time.
[0097] Specifically, when a consumer starts processing a message, if it does not complete the processing within the specified time, the message is considered expired and the system will automatically retry; when the consumer has processed the message, it needs to send a confirmation message to other message queues. Only when the message is confirmed can other consumers read the message.
[0098] Step A205: If it is detected that other asynchronous information exists when the asynchronous message is in a success or failure state, the idempotent consumption control strategy is triggered to discard the other asynchronous messages.
[0099] It should be noted that the idempotent consumption control strategy is a mechanism that ensures that message processing operations produce the same results when executed multiple times; in a distributed system, when multiple consumers consume the same message queue at the same time, idempotent processing can prevent data inconsistency and repeated operations.
[0100] Specifically, in a simple transfer operation, a certain amount of funds is transferred from one account to another. The transfer operation is designed to be an idempotent operation. No matter how many times the transfer is performed, the result is always the same. When it is repeated, the idempotent consumption control strategy detects that there is other transfer information when the transfer operation is in a successful or failed state. The system will trigger idempotent processing and the transfer amount will always remain consistent. Even if multiple transfer requests arrive at the same time, the system can ensure that the total amount of funds in each account will not change due to repeated operations.
[0101] In this embodiment, by transferring the asynchronous message from the pending state to the processing state, and transferring the message to the failed or successful state according to the processing result, the system can clearly track the processing status of the message; the locking mechanism and concurrent consumption control strategy ensure that in a multi-threaded or distributed environment, only one process or thread can process a message at the same time, thereby preventing data inconsistency or repeated processing problems; when the processing fails, by transferring the message to the pending state and retrying, the system improves the reliability of message processing, and even if the initial processing fails, the message can be successfully processed in the end; the idempotent consumption control strategy ensures that even if the message is processed repeatedly, the result is consistent, which helps to maintain data consistency and system stability; using the unique identifier of the asynchronous message, the system can accurately identify and track the status of each message, ensuring the accuracy and consistency of message processing.
[0102] Further, refer to Figure 3 The third embodiment of the asynchronous message processing method of the present application provides a flow chart based on the above Figure 3 In the illustrated embodiment, the step of "updating the local asynchronous record according to the processing result and returning to the first business system" in step S20 is further refined, including steps A301 to A305:
[0103] Step A301: Obtain an exception message according to the processing result, and update the local asynchronous record to the second business system;
[0104] It should be noted that, when a timeout or failure occurs during message processing, the message middleware updates the local asynchronous record of the first business system and synchronizes the updated record status to the second business system.
[0105] Step A302: Sending an abnormal message receipt confirmation to the first business system through the second business system;
[0106] Specifically, the second business system sends an exception message reception confirmation to the first business system, and the confirmation message may include detailed information of the exception message, processing results, status update and other information.
[0107] Step A303: Update local asynchronous records by calling the API provided by the first business system;
[0108] It should be noted that the second business system updates the local asynchronous record of the first business system by calling the API provided by the first business system, and the API interface is used to transmit the abnormal message reception confirmation and related data between the two business systems.
[0109] Step A304: Send an update request to the first business system and update the local asynchronous record status of the first business system;
[0110] It should be noted that the second business system sends an update request to the first business system, and the update request may include updated data, status information, operation type, etc. The request is usually sent through network communication or message passing mechanism parameters to update the corresponding local asynchronous record status.
[0111] Step A305: After updating the local asynchronous record of the first business system, the first business system returns a confirmation response to the second business system, indicating that the update operation has been completed;
[0112] It should be noted that after successfully updating the local asynchronous record, the first business system returns a confirmation response to the second business system. The confirmation response may include detailed information of the update operation, such as updated data, operation timestamp, etc., indicating that the update operation has been completed.
[0113] In this embodiment, through the interaction between the message middleware and the second business system, the consistency between the status of the local asynchronous record of the first business system and the actual processing result is ensured; the second business system sends an abnormal message to the first business system to confirm receipt, providing asynchronous message processing feedback; by calling the API provided by the first business system, the timely update of the local asynchronous record of the first business system is ensured, and the confirmation response mechanism improves the system redundancy, reflecting how different business systems work together in a distributed system to maintain data consistency and stability.
[0114] Further, refer to Figure 4 The fourth embodiment of the asynchronous message processing method of the present application provides a flow chart based on the above Figure 4 In the illustrated embodiment, the step of "scanning the local asynchronous records to determine whether there are timeout unprocessed or failed records" in step S30 is further refined, including steps A401 to A404:
[0115] Step A401: Determine whether there is a failure record in the local asynchronous record;
[0116] It should be noted that the system can periodically scan local asynchronous records to check whether there are records marked as "failed". The "failed" records may be messages that were not successfully processed due to an exception or error in the asynchronous message processing process.
[0117] Step A402: if there is a failure record in the local asynchronous record, determine whether the task failure caused by the asynchronous message processing exception does not exceed the maximum failure number, and if it does not exceed the maximum failure number, convert the asynchronous message to a pending state;
[0118] It should be noted that the pending state indicates that the asynchronous message needs to re-enter the processing flow to prepare for the next processing attempt; the re-issued asynchronous message is processed according to the normal processing flow, including state transfer, locking, retry, etc.
[0119] Step A403: If there is no failed record in the local asynchronous record, further determine whether there is a timeout unprocessed record;
[0120] It should be noted that the timeout unprocessed record refers to an asynchronous message that has not been processed within a preset time.
[0121] Step A404: If there are timeout records that are not processed, determine whether the timeout is caused by a large number of asynchronous messages but insufficient asynchronous thread resources or asynchronous message loss. Otherwise, determine whether the asynchronous message processing exceeds the preset maximum allowed time. If it exceeds the preset maximum allowed time, scan them out through a backup job and reissue the asynchronous message.
[0122] It should be noted that when the number of asynchronous messages is large, the linear pool size can be increased to handle more asynchronous messages, or message queue partitioning or partition exchange can be implemented to disperse the message load; when the asynchronous thread resources are insufficient, a more efficient thread management strategy can be used, such as dynamic adjustment of the thread pool; the backup job is usually a backup mechanism for handling asynchronous messages in system failures or other abnormal situations.
[0123] In this embodiment, by monitoring the status of local asynchronous records, failed records or timed-out records are discovered in a timely manner. Through the retry mechanism, timeout processing and backup operations for failed records, the system can effectively manage various situations that may lead to failure or message timeouts to be processed, optimize the use of system resources, and form a stable and reliable asynchronous message processing mechanism, which reflects the system's ability to self-diagnose and recover.
[0124] In a possible embodiment, if Figure 5As shown, the fifth embodiment of the asynchronous message processing method of the present application provides a flow chart based on the above Figure 5 In the illustrated embodiment, initiating an asynchronous message retry includes:
[0125] When an asynchronous message times out and is not processed due to an exception in the local asynchronous scheduling process or insufficient thread resources, it is transferred to the pending state and waits for the asynchronous message to be re-issued;
[0126] It should be noted that the status of the affected asynchronous message is transferred from the processing status to the pending status. In the pending status, it means that the message needs to re-enter the processing flow and wait for a suitable time window or trigger condition. When the conditions for triggering the re-issuance of the asynchronous message are met, the system will re-issue the message and start a new round of processing flow.
[0127] After the asynchronous message is reissued, it is transferred from the pending state to the processing state, and the asynchronous message is locked;
[0128] It should be noted that message locking usually refers to locking messages in a message queue to ensure that in a multi-threaded or distributed system, only one process or thread can process the message at the same time. This mechanism can prevent multiple processes from processing the same message at the same time, resulting in inconsistent data or duplicate processing.
[0129] If an asynchronous message in the processing state fails, it will be retried and transferred back to the pending state. If there are sufficient thread resources, it will continue to be processed and remain in the processing state.
[0130] Through the bottom-up operation, the timed-out pending asynchronous processing is uniformly scanned, and an asynchronous message is sent to restore the pending status;
[0131] It should be noted that the unified scanning of the bottom-up operation is effective in the entire process of message processing. The system can handle timed-out and unprocessed asynchronous messages caused by various abnormal situations, restore these messages to the pending state through the bottom-up operation, and try to process them again.
[0132] After the asynchronous message is processed from the processing state, it is transferred to the failure state or the success state according to the processing result, and the asynchronous message is locked;
[0133] It is determined whether the number of final states in the processing result reaches an upper limit, if the number of final states reaches the upper limit, the process remains in a failed state, otherwise, the process remains in a successful state.
[0134] Specifically, the system determines whether the number of final states in the processing result reaches a preset upper limit, where the number of final states is the number of successes or failures. When the number of final states reaches the upper limit, the system will keep the message in the current state.
[0135] In this embodiment, when a message is processed abnormally or due to insufficient resources and timeout, the message is transferred to a pending state and retried to improve the success rate of message processing. During the message state transfer process, the asynchronous message is locked to ensure that only one process or thread can process the message at the same time to prevent data inconsistency. The timed-out pending asynchronous messages are uniformly scanned through a bottom-up operation, and an asynchronous message is sent to restore the pending state, thereby optimizing the use of system resources and reducing unnecessary repetitive operations.
[0136] In a feasible embodiment, the initiating asynchronous message retry further includes:
[0137] Personalized configuration of asynchronous message processing, the personalized configuration includes:
[0138] Adjust the interval time for failed retries, the upper limit of failed retries, the maximum waiting time during processing, and the maximum waiting time to be processed.
[0139] It should be noted that through personalized configuration of asynchronous message processing, it can flexibly adapt to different business scenarios and system requirements; different business scenarios may require different retry intervals to balance the response speed and stability of the system; by defining the maximum number of times an asynchronous message can be retried after failure, after exceeding this number, the system will process it according to the preset strategy, such as marking it as failure or transferring it to the fallback processing flow; define the maximum number of times an asynchronous message can be retried after failure, after exceeding this number, the system will process it according to the preset strategy, such as marking it as failure or transferring it to the fallback processing flow; define the maximum time that an asynchronous message can wait in the pending state, after exceeding this time, the system will re-evaluate the status of the message and may restart the processing flow.
[0140] In this embodiment, through personalized configuration of asynchronous message processing, the message processing strategy can be adjusted according to different business needs and system conditions, and a universal asynchronous scheduling, timeout and failure retry mechanism can be built. There is no need to develop separately for each scenario, which reduces the complexity and maintenance cost of the system.
[0141] In a possible embodiment, if Figure 6 As shown, the sixth embodiment of the asynchronous message processing method of the present application provides a flow chart based on the above Figure 6 In the illustrated embodiment, the step of applying for the second business system asynchronous thread includes steps A601 to A603:
[0142] Step A601: When the message middleware is unavailable, the first business system uniformly scans the timed-out asynchronous messages to be processed through a backup job, and applies for the local thread pool of the second business system to process the asynchronous messages;
[0143] It should be noted that the backup job uniformly scans all asynchronous messages in the system that are in a timeout and pending state. The asynchronous messages in the timeout and pending state may include those that cannot be processed in time due to the unavailability of the message middleware; the local thread pool of the second business system processes asynchronous messages, and according to the request of the first business system, allocates threads in the local thread pool to process asynchronous messages.
[0144] Step A602: The first business system locks the asynchronous message in the to-be-processed state through the thread in the local thread pool, and continues to process and convert it into the processing state;
[0145] It should be noted that the local thread pool manages the life cycle and allocation of threads. The first business system selects an available thread from the local thread pool of the second business system to process asynchronous messages.
[0146] Step A603: The first business system waits for the asynchronous message processing status to be completed, converts to a success state or a failure state according to the processing result, and locks the asynchronous message.
[0147] It should be noted that after the state switching is completed, the asynchronous message is unlocked. After unlocking, other threads or processes can read or process the message, provided that they have obtained the corresponding access rights.
[0148] In this embodiment, by uniformly scanning the timed-out asynchronous processing through a bottom-up operation, the system can handle abnormal situations when the message middleware is unavailable. At the same time, when the message middleware is unavailable, the first business system applies to the second business system for a local thread pool to process asynchronous messages, making full use of the resources of the second business system, helping to reduce the load on the first business system, and ensuring the high availability of the system and the continuity of data processing in abnormal situations.
[0149] In a feasible embodiment, the step of processing asynchronous messages by the local thread pool of the second business system includes:
[0150] Determine the relationship between the number of threads currently running in the local thread pool of the second business system and the number of core threads;
[0151] If the number of threads currently running in the local thread pool of the second business system is less than the number of core threads, a new thread is created to execute the task;
[0152] It should be noted that the local thread pool manages thread creation and task processing based on the relationship between the number of currently running threads and the number of core threads.
[0153] If the number of threads currently running in the local thread pool of the second business system reaches the number of core threads, the task is added to the message queue, a blocking queue is used, and a new thread is directly created to process the task;
[0154] It should be noted that a blocking queue blocks the addition of new tasks when the queue capacity is reached until there are available threads to process them.
[0155] If the local thread pool of the second business system creates a new thread so that the currently running thread exceeds the maximum number of threads, a rejection strategy is adopted to pop up a rejection execution exception and refuse to execute the task.
[0156] It should be noted that common rejection strategies include directly throwing exceptions, logging, and placing tasks in a failure queue.
[0157] Specifically, there is a local thread pool with 5 core threads and a maximum number of 20 threads. The rejection strategy uses AbortPolicy, that is, when a task is added to the thread pool and is rejected, an exception is directly thrown. The message queue uses SynchronousQueue, that is, when the capacity is 0, the elements are directly passed from the producer to the consumer; if the number of currently running threads is 3, which is less than the number of core threads 5, a new thread is created to execute the task; if the number of running threads is 5, which is equal to the number of core threads 5, the task is added to the message queue. Since the SynchronousQueue blocking queue is used, a new thread is directly created to process the task; if the currently running thread of the new thread zombie created exceeds the maximum number of threads 20, the task will be rejected. Since the AbortPolicy rejection strategy is used, a RejectedExecutionException exception is directly thrown.
[0158] In this embodiment, by managing the number of threads, comparing the number of core threads, and adopting blocking queues and rejection strategies, the local thread pool can efficiently manage thread resources, balance system load and resource usage, and ensure that the system does not crash due to too many threads when facing high load.
[0159] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the asynchronous message processing method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.
[0160] In addition, this application also provides an asynchronous message processing device, please refer to Figure 7 , the asynchronous message processing device comprises:
[0161] An information receiving module, used for receiving asynchronous messages and local asynchronous records sent by the first business system;
[0162] A message processing module, used for processing the asynchronous message, updating the local asynchronous record according to the processing result and returning the result to the first business system;
[0163] A first judgment module is used to scan the local asynchronous records to determine whether there are timeout unprocessed or failed records;
[0164] The second judgment module determines whether the message middleware is available if there is a timeout or failure record; if the message middleware is available, an asynchronous message retry is initiated, otherwise an asynchronous thread of the second business system is applied.
[0165] The asynchronous message processing device provided by the present application adopts the asynchronous message processing method in the above embodiment, which can solve the common message failure, loss and idempotent retry mechanism of message middleware in different business scenarios, and the technical problems of lack of universal asynchronous retry mechanism and heterogeneous degradation mechanism. Compared with the prior art, the beneficial effects of the asynchronous message processing device provided by the present application are the same as the beneficial effects of the asynchronous message processing method provided by the above embodiment, and the other technical features in the asynchronous message processing device are the same as the features disclosed in the above embodiment method, which will not be repeated here.
[0166] In addition, the present application provides an asynchronous message processing device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the asynchronous message processing method in the above-mentioned embodiment one.
[0167] Reference below Figure 8 , which shows a schematic diagram of the structure of an asynchronous message processing device suitable for implementing the embodiment of the present application. The asynchronous message processing device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 8 The asynchronous message processing device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0168] like Figure 8As shown, the asynchronous message processing device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the asynchronous message processing device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the asynchronous message processing device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows an asynchronous message processing device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or provided instead.
[0169] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.
[0170] The asynchronous message processing device provided by the present application adopts the asynchronous message processing method in the above embodiment, which can solve the common message failure, loss and idempotent retry mechanism of message middleware in different business scenarios, and the technical problems of lack of universal asynchronous retry mechanism and heterogeneous degradation mechanism. Compared with the prior art, the beneficial effects of the asynchronous message processing device provided by the present application are the same as the beneficial effects of the asynchronous message processing method provided by the above embodiment, and the other technical features in the asynchronous message processing device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.
[0171] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.
[0172] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
[0173] In addition, the present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the asynchronous message processing method in the above-mentioned embodiment.
[0174] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.
[0175] The computer-readable storage medium may be included in the asynchronous message processing device; or may exist independently without being assembled into the asynchronous message processing device.
[0176] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the asynchronous message processing device, the asynchronous message processing device: receives the asynchronous message and local asynchronous record sent by the first business system; processes the asynchronous message, updates the local asynchronous record according to the processing result and returns it to the first business system; scans the local asynchronous record to determine whether there is a timeout and unprocessed or failed record; if there is a timeout and unprocessed or failed record, determines whether the message middleware is available; if the message middleware is available, initiates an asynchronous message retry, otherwise applies for an asynchronous thread of the second business system.
[0177] Computer program code for performing the operations of the present application may be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0178] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0179] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.
[0180] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned asynchronous message processing method, and can solve the common message failure, loss and idempotent retry mechanism of message middleware in different business scenarios, and the technical problems of lack of universal asynchronous retry mechanism and heterogeneous degradation mechanism. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the asynchronous message processing method provided by the above-mentioned embodiment, which will not be repeated here.
[0181] In addition, the present application also provides a computer program product, including a computer program, which implements the steps of the asynchronous message processing method as described above when executed by a processor.
[0182] The computer program product provided by the present application can solve the common message failure, loss and idempotent retry mechanism of message middleware in different business scenarios, and the technical problems of lack of universal asynchronous retry mechanism and heterogeneous degradation mechanism. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as those of the asynchronous message processing method provided by the above embodiment, which will not be elaborated here.
[0183] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.
Claims
1. An asynchronous message processing method, characterized in that: Applied to message middleware, the method includes: Receiving asynchronous messages and local asynchronous records sent by the first business system; Processing the asynchronous message, updating the local asynchronous record according to the processing result and returning it to the first business system; Scan the local asynchronous records to determine whether there are timeout, unprocessed or failed records; If there are timeouts or failure records, determine whether the message middleware is available; If the message middleware is available, an asynchronous message retry is initiated; otherwise, an asynchronous thread of the second business system is requested.
2. The method according to claim 1, characterized in that The step of processing the asynchronous message comprises: Transferring the asynchronous message from a pending state to a processing state, and locking the asynchronous message; Waiting for the asynchronous message to complete the processing of the status, transferring the asynchronous message to a failure or success state according to the processing completion result, and locking the asynchronous message; If it is detected that the processing completion result is a failure state, the asynchronous message is transferred to a pending state and retried; If it is detected that there are other asynchronous messages when the asynchronous message is in the processing state, the concurrent consumption control strategy is triggered to discard the other asynchronous messages; If it is detected that other asynchronous information exists when the asynchronous message is in a success or failure state, the idempotent consumption control strategy is triggered to discard the other asynchronous messages.
3. The method according to claim 2, characterized in that The step of updating the local asynchronous record according to the processing result and returning it to the first business system comprises: Obtain the exception message according to the processing result, and update the local asynchronous record to the second business system; Sending an abnormal message receipt confirmation to the first business system through the second business system; Update local asynchronous records by calling the API provided by the first business system; Sending an update request to the first business system and updating the local asynchronous record status of the first business system; After updating the local asynchronous record of the first business system, the first business system returns a confirmation response to the second business system, indicating that the update operation is completed.
4. The method according to claim 1, characterized in that The step of scanning the local asynchronous records and determining whether there are timeout, unprocessed or failed records includes: Determine whether there is a failure record in the local asynchronous record; If there is a failure record in the local asynchronous record, determine whether the task failure caused by the asynchronous message processing exception does not exceed the maximum number of failures. If it does not exceed the maximum number of failures, convert the asynchronous message to a pending state; If there is no failed record in the local asynchronous record, further determine whether there is a timeout and unprocessed record; If there are timeout records that are not processed, determine whether the timeout is not processed due to a large number of asynchronous messages, insufficient asynchronous thread resources, or asynchronous message loss. Otherwise, determine whether the asynchronous message processing exceeds the preset maximum allowed time. If it exceeds the preset maximum allowed time, scan them out through a backup job and resend the asynchronous message.
5. The method according to claim 4, characterized in that The initiating asynchronous message retry comprises: When an asynchronous message times out and is not processed due to an exception in the local asynchronous scheduling process or insufficient thread resources, it is transferred to the pending state and waits for the asynchronous message to be re-issued; After the asynchronous message is reissued, it is transferred from the pending state to the processing state, and the asynchronous message is locked; If an asynchronous message in the processing state fails, it will be retried and transferred back to the pending state. If there are sufficient thread resources, it will continue to be processed and remain in the processing state. Through the bottom-up operation, the timed-out pending asynchronous processing is uniformly scanned, and an asynchronous message is sent to restore the pending status; After the asynchronous message is processed from the processing state, it is transferred to the failure state or the success state according to the processing result, and the asynchronous message is locked; It is determined whether the number of final states in the processing result reaches an upper limit, if the number of final states reaches the upper limit, the process remains in a failed state, otherwise, the process remains in a successful state.
6. The method according to claim 5, characterized in that The initiating asynchronous message retry also includes: Personalized configuration of asynchronous message processing, the personalized configuration includes: Adjust the interval time for failed retries, the upper limit of failed retries, the maximum waiting time during processing, and the maximum waiting time to be processed.
7. The method according to claim 1, characterized in that The step of applying for the second business system asynchronous thread includes: When the message middleware is unavailable, the first business system uniformly scans the timed-out asynchronous messages to be processed through a backup job, and applies for the local thread pool of the second business system to process the asynchronous messages; The first business system locks the asynchronous message in the to-be-processed state through the thread in the local thread pool, and continues to process and converts it into the processing state; The first business system waits for the asynchronous message to be processed in a state process, converts the state to a success state or a failure state according to the processing result, and locks the asynchronous message.
8. The method according to claim 7, characterized in that The step of the local thread pool of the second business system processing asynchronous messages includes: Determine the relationship between the number of threads currently running in the local thread pool of the second business system and the number of core threads; If the number of threads currently running in the local thread pool of the second business system is less than the number of core threads, a new thread is created to execute the task; If the number of threads currently running in the local thread pool of the second business system reaches the number of core threads, the task is added to the message queue, a blocking queue is used, and a new thread is directly created to process the task; If the local thread pool of the second business system creates a new thread so that the currently running thread exceeds the maximum number of threads, a rejection strategy is adopted to pop up a rejection execution exception and refuse to execute the task.
9. An asynchronous message processing device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the asynchronous message processing method according to any one of claims 1 to 8.
10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the asynchronous message processing method according to any one of claims 1 to 8 are implemented.