Retry method and system for message consumption failure and storage medium

Abstract

The invention relates to the technical field of information communication, in particular to a message consumption failure retry method and system and a storage medium, and the method comprises the steps: obtaining a to-be-consumed message; based on the message queue, delivering the to-be-consumed message; determining idempotent identification information of the to-be-consumed message based on the idempotent identifier of the to-be-consumed message under the condition that the to-be-consumed message fails to be delivered; based on the idempotent identification information and the idempotent auxiliary table, consuming the to-be-consumed message; and under the condition that the consumption of the to-be-consumed message fails, performing attenuation type consumption retry on the to-be-consumed message based on the retry parameter of the to-be-consumed message. According to the method, under the condition of consumption failure, attenuation type consumption retry is carried out to realize dynamic consumption retry, so that repeated development of codes can be reduced, system resources are saved, and the timeliness of a business process is improved; and the message queue is analyzed and processed, so that the extra work and the complexity of the system can be reduced.

Description

Technical Field

[0001] The present invention relates to the field of information communication technology, and in particular to a retry method, system and storage medium for message consumption failure in the field of information communication technology. Background Art

[0002] In the related art, although the idempotent consumption failure decay retry method based on message queue can effectively ensure the reliability of messages and the stability of the system in many scenarios, in most message queue systems, if the number of retries reaches the upper limit, the message will no longer be retried, but will be transferred to the dead letter queue, which may cause some messages that could have been processed to be lost. In addition, the setting of the retry interval has an important impact on system performance; if the interval is set too short, it may lead to excessive consumption of system resources; if it is set too long, it may affect the timeliness of the business process. Ensuring the idempotence of message consumption is crucial to preventing data errors, but in actual applications, the design of idempotence logic is often more complicated, and additional logic is required to ensure the consistency of message processing logic. Although the dead letter queue provides a storage place for messages that cannot be consumed normally, the management, monitoring and subsequent processing of the dead letter queue require additional work, which increases the complexity of the system. Summary of the invention

[0003] The purpose of the present invention is to provide a retry method, system and storage medium for message consumption failure, and the technical scheme adopted is as follows:

[0004] In a first aspect, an embodiment of the present invention provides a retry method for message consumption failure, the method comprising:

[0005] Get the message to be consumed;

[0006] Based on the message queue, the message to be consumed is delivered;

[0007] In the case where the delivery of the message to be consumed fails, determining the idempotence identification information of the message to be consumed based on the idempotence identifier of the message to be consumed;

[0008] Based on the idempotent identification information and the idempotent auxiliary table, the message to be consumed is consumed; wherein the idempotent auxiliary table is used to allow one of the multiple messages to be written successfully when multiple messages with the same idempotent identifier are written at the same time;

[0009] In the case that the consumption of the message to be consumed fails, based on the retry parameter of the message to be consumed, a decaying consumption retry is performed on the message to be consumed.

[0010] In combination with the first aspect above, in some possible implementations, the method further includes:

[0011] Determining the transaction level of the message to be consumed based on the importance and urgency of the message to be consumed;

[0012] Based on the transaction level, determine a retry time interval that matches a retry number threshold of the message to be consumed and a current retry number of the message to be consumed; wherein the retry time interval increases as the current retry number increases;

[0013] The retry times threshold and the retry time interval are determined as the retry parameters.

[0014] In combination with the first aspect above, in some possible implementations, when the delivery of the message to be consumed fails, determining the idempotence identification information of the message to be consumed based on the idempotence identifier of the message to be consumed includes:

[0015] In the event that the delivery of the message to be consumed fails, determining the service name of the producer end corresponding to the message to be consumed and the identifier of the producer end;

[0016] Based on the idempotence identifier, the service name and the identifier of the producer end, the idempotence identification information of the message to be consumed is determined.

[0017] In combination with the first aspect above, in some possible implementations, determining the idempotence identification information of the message to be consumed based on the idempotence identifier, the service name, and the identifier of the producer end includes:

[0018] Taking the idempotence identifier as a constraint, the service name and the identifier of the producer end are written into the idempotence auxiliary table to obtain the idempotence identification information of the message to be consumed; wherein the constraint is used to constrain the uniqueness of messages with the same idempotence identifier written into the idempotence auxiliary table.

[0019] In combination with the first aspect above, in some possible implementations, consuming the message to be consumed based on the idempotent identification information and the idempotent auxiliary table includes:

[0020] In the idempotent auxiliary table, determining whether there is a consumption record corresponding to the idempotent identification information;

[0021] If the consumption record does not exist in the idempotent auxiliary table, start the consumption transaction;

[0022] Based on the consumption transaction, inserting the idempotent identification information of the message to be consumed into the idempotent auxiliary table;

[0023] The message to be consumed is consumed based on the idempotent auxiliary table.

[0024] In combination with the first aspect above, in some possible implementations, the method further includes:

[0025] When multiple messages having the same idempotency identifier are simultaneously written into the idempotency auxiliary table, detecting the writing status of the multiple messages;

[0026] When it is detected that the write status of any message is write successful, based on the constraint conditions of the idempotent auxiliary table, a message write exception is triggered so that the write status of other messages among the multiple messages is write failed; wherein, the other messages are messages among the multiple messages except the any one message.

[0027] In combination with the first aspect above, in some possible implementations, when the consumption of the message to be consumed fails, performing a decaying consumption retry on the message to be consumed based on a retry parameter of the message to be consumed, including:

[0028] In the event that the consumption of the message to be consumed fails, obtaining and updating the record log corresponding to the message to be consumed;

[0029] Determine the current retry count of the message to be consumed based on the record log corresponding to the message to be consumed;

[0030] If the current number of retries does not reach the retry number threshold in the retry parameter, based on the retry time interval matched by the current number of retries, delivering the delayed message corresponding to the message to be consumed;

[0031] The delayed message is retried for consumption.

[0032] In combination with the first aspect above, in some possible implementations, if the current number of retries does not reach the retry number threshold in the retry parameter, based on the retry time interval matched by the current number of retries, delivering the delayed message corresponding to the message to be consumed includes:

[0033] If the current number of retries does not reach the retry number threshold in the retry parameter, based on the current number of retries, the retry time interval of the previous consumption retry is extended to obtain the retry time interval of the current consumption retry;

[0034] Based on the retry time interval of the current consumption retry, a delayed message corresponding to the message to be consumed is generated; wherein the message content of the delayed message is the same as the message content of the message to be consumed;

[0035] The delayed message is delivered to the message queue of the message to be consumed.

[0036] In combination with the first aspect above, in some possible implementations, the method further includes:

[0037] If the current number of retries reaches the retry number threshold in the retry parameter, the consumption retry is stopped and an alarm message is generated; wherein the alarm message is used to prompt the client to process the message to be consumed.

[0038] In combination with the first aspect above, in some possible implementations, obtaining the message to be consumed includes:

[0039] Get any message in the communication information;

[0040] Determine the association record between any one of the messages and the consumer end;

[0041] Based on the associated record, determining whether any of the messages needs to be consumed;

[0042] If any of the messages needs to be consumed, the any of the messages is determined to be the message to be consumed.

[0043] In a second aspect, a retry system for message consumption failure is provided, the system comprising:

[0044] The acquisition module is used to obtain the messages to be consumed;

[0045] A delivery module, used for delivering the to-be-consumed message based on a message queue;

[0046] A determination module, configured to determine, when delivery of the message to be consumed fails, idempotence identification information of the message to be consumed based on the idempotence identifier of the message to be consumed;

[0047] A consumption module, used for consuming the message to be consumed based on the idempotent identification information and the idempotent auxiliary table; wherein the idempotent auxiliary table is used for allowing one message among the multiple messages to be written successfully when multiple messages with the same idempotent identifier are written at the same time;

[0048] The consumption retry module is used to perform attenuated consumption retry on the message to be consumed based on the retry parameters of the message to be consumed when the consumption of the message to be consumed fails.

[0049] According to a third aspect, a computer program product is provided, comprising: a computer program code, which, when executed on a computer, enables the computer to execute the method in the first aspect or any possible implementation of the first aspect.

[0050] In a fourth aspect, a computer-readable storage medium is provided, which stores a computer program code. When the computer program code runs on a computer, the computer executes the method in the first aspect or any possible implementation manner described in the first aspect.

[0051] The present invention has the following beneficial effects: for the acquired information to be consumed, the message to be consumed is delivered by means of a message queue; and in the case where the delivery of the message to be consumed fails, the idempotency identification information of the message to be consumed is determined based on the idempotency identifier of the message to be consumed; thereby, the message to be consumed is consumed through the idempotency identification information and the idempotency auxiliary table; wherein the idempotency auxiliary table is used to allow one of the multiple messages to be written successfully when multiple messages with the same idempotency identifier are written at the same time; in this way, by introducing the idempotency auxiliary table and setting the idempotency identification information of the message to be consumed, the workload of the developer in implementing the idempotency logic can be simplified, and the consistency of the message processing logic can be ensured. Finally, in the case where the consumption of the message to be consumed fails, the consumption of the message to be consumed is retried in an attenuated manner based on the retry parameters of the message to be consumed. In this way. In the case where the consumption of the message to be consumed fails, by combining the message queue and the retry parameters of the message to be consumed, a more reasonable retry parameter is dynamically configured, which can not only save system resources but also improve the timeliness of the business process. By performing attenuated consumption retries on consumption messages according to retry parameters, dynamic consumption retries can be implemented, reducing repeated code development; and by analyzing and processing message queues, additional workload and system complexity can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0053] Figure 1 It is a schematic diagram of the implementation process of a retry method for message consumption failure provided by an embodiment of the present invention;

[0054] Figure 2 It is another implementation flow diagram of a retry method for message consumption failure provided by an embodiment of the present invention;

[0055] Figure 3 It is another implementation flow diagram of a retry method for message consumption failure provided by an embodiment of the present invention;

[0056] Figure 4 It is another implementation flow diagram of a retry method for message consumption failure provided by an embodiment of the present invention;

[0057] Figure 5 It is another implementation flow diagram of a retry method for message consumption failure provided by an embodiment of the present invention;

[0058] Figure 6 It is a schematic diagram of the composition structure of a retry system for message consumption failure provided by an embodiment of the present invention;

[0059] Figure 7 It is a structural schematic diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0060] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following is a detailed description of the specific implementation method, structure, features and effects of a retry method for message consumption failure proposed by the present invention in combination with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form as described.

[0061] Among them, in the description of the embodiments of the present invention, unless otherwise specified, " / " means or, for example, A / B can mean A or B: "and / or" in the text is only a way to describe the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present invention, "multiple" refers to two or more than two.

[0062] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features.

[0063] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0064] In some embodiments, in a message queue, processing message consumption failure and retrying is a common requirement. According to the search results, we can understand some key points of the method for implementing attenuation retry based on idempotent consumption failure of message queue in RocketMQ:

[0065] 1. Message retry mechanism: RocketMQ supports retry mechanisms on both the producer and consumer sides. After a consumer fails to consume a message, it can set the number of retries and the interval time. For example, the default retry interval is "10 seconds (s) 30s 1 minute (m) 2m 3m 4m 5m 6m 7m 8m 9m 10m 20m 30m 1 hour (h) 2h", and the maximum number of retries can be set, such as "setMaxReconsumeTimes(20)".

[0066] 2. Dead letter message processing: When the message retry reaches the set threshold, the message will be considered a dead letter message and will no longer be consumed normally, but will be sent to the dead letter queue. The dead letter queue is created for each consumer end group (Group ID), and the dead letter message will not be consumed again in it, and the validity period is 3 days.

[0067] 3. Consumer configuration: You can configure the consumer by implementing the "RocketMQ Push Consumer LifecycleListener" interface, such as setting the maximum number of retries and instance name. The maximum number of message retries is valid for all consumer instances under the same GroupID.

[0068] 4. Exception and timeout retry: Retry on the consumer side may be triggered by exception or timeout. RocketMQ provides options for configuring the number of retries and time intervals in the broker.conf file, such as the "messageDelayLevel" parameter.

[0069] 5. Idempotence guarantee: In order to ensure the idempotence of message consumption, idempotence check can be implemented in the consumer-side logic to ensure that even if the message is consumed repeatedly, it will not cause business logic errors.

[0070] Based on this, we can see that the method of implementing idempotent consumption failure decay retry based on message queues involves the configuration of message retry strategies, the processing mechanism of dead letter messages, and the logical implementation of ensuring the idempotence of message consumption. These mechanisms can help developers effectively handle message consumption failures in message queues and reduce the risks of repeated message consumption through reasonable strategies.

[0071] In related technologies, although the idempotent consumption failure decay retry method based on message queue can effectively ensure the reliability of messages and the stability of the system in many scenarios, it still has some technical deficiencies, mainly including:

[0072] 1. Retry limit: Most message queue systems, such as Rocket Message Queue (Rocket MQ), allow each message to be retried up to 16 times by default. If the retry limit is reached, the message will no longer be retried but transferred to the dead letter queue, which may cause some messages that could have been processed to be lost.

[0073] 2. Retry interval setting: The retry interval setting has an important impact on system performance. If the interval is set too short, it may lead to excessive consumption of system resources; if it is set too long, it may affect the timeliness of business processes.

[0074] 3. Idempotence design challenge: Ensuring the idempotence of message consumption is crucial to preventing data errors. However, in actual applications, designing idempotence logic is often complex and requires additional logic to ensure the consistency of message processing logic.

[0075] 4. Management of dead letter queues: Although dead letter queues provide a storage place for messages that cannot be consumed normally, the management, monitoring and subsequent processing of dead letter queues require additional work, which increases the complexity of the system.

[0076] 5. Consumer-side state management: After the consumer side restarts or crashes, how to correctly restore to the consumption state and ensure that messages are not consumed repeatedly or lost is a technical challenge.

[0077] In response to these technical deficiencies, developers need to comprehensively consider factors such as business requirements, system stability, and resource consumption when designing message queue consumption systems, and adopt appropriate strategies to optimize the retry mechanism and improve the robustness and reliability of the system.

[0078] Based on this, in view of the technical deficiencies of the idempotent consumption failure attenuation retry method based on the message queue, an embodiment of the present invention provides a message consumption failure retry method, which can achieve:

[0079] 1. Dynamically adjust the number of retries: According to the importance and urgency of the message, dynamically adjust the maximum number of retries for each message to provide more retry opportunities for critical messages.

[0080] 2. Enhanced idempotency design: Provide a general idempotency framework or tool to simplify the workload of developers in implementing idempotency logic and ensure the consistency of message processing logic.

[0081] 3. Diversification of retry strategies: Supports multiple retry strategies, such as fixed interval, increasing interval, exponential backoff, etc., and allows developers to customize retry strategies according to business needs.

[0082] 4. Automated processing of dead letter queues: Develop automated tools to analyze and process messages in dead letter queues, such as resending, notifying administrators, and recording logs.

[0083] 5. Transactional message support: Introduce a transactional message mechanism to ensure the atomicity of message processing and avoid inconsistencies caused by partial operation failures.

[0084] 6. Flexible message confirmation mechanism: Provide more flexible message confirmation methods, such as batch confirmation, delayed confirmation, etc., to adapt to different business scenarios.

[0085] In this way, a retry method for message consumption failure provided by an embodiment of the present invention can improve the reliability, stability and intelligence of the message queue consumption system and better meet the needs of different business scenarios.

[0086] The following is a detailed description of a specific scheme of a retry method for message consumption failure provided by the present invention in conjunction with the accompanying drawings. Figure 1 , which shows a schematic diagram of an implementation flow of a retry method for message consumption failure provided by an embodiment of the present invention, the method comprising:

[0087] 101. Obtain the message to be consumed.

[0088] Here, the message to be consumed is a message that is obtained from the communication information and needs to be consumed. The message to be consumed can be a non-transactional message or a transactional message.

[0089] In some possible implementations, the above step 101 may be implemented by the following steps 111 to 113 (not shown):

[0090] 111, obtain any message in the communication information.

[0091] Here, the communication information may be a plurality of messages of any type output by the producer end, and any one message may be selected from the plurality of messages.

[0092] 112. Determine the association record between any message and the consumer end.

[0093] Here, after obtaining any message from the communication information, an association record between the message and the consumer end is created, wherein the association record includes: whether the message needs to be consumed, whether it is consumed, whether the consumption is successful, etc.

[0094] 113. Based on the associated record, determine whether any of the messages needs to be consumed; if any of the messages needs to be consumed, determine that any of the messages is the message to be consumed.

[0095] Here, after obtaining the associated record, it is possible to quickly analyze from the associated record whether the message needs to be consumed; if the message needs to be consumed, then the message is treated as a message to be consumed. In this way, by analyzing whether any message in the communication information needs to be consumed through the associated record, the message that needs to be consumed is treated as a message to be consumed, and the message that needs to be consumed in the communication information can be accurately determined.

[0096] 102. Deliver the message to be consumed based on the message queue.

[0097] Here, the messages to be consumed are delivered in a message queue manner. For example, the messages to be consumed are added to the message queue, and the messages to be consumed are delivered according to the first-in-first-out principle, so that the consumer end consumes the messages to be consumed through the message queue.

[0098] In some possible implementations, before delivering the message to be consumed, the transaction level of the message to be consumed can also be determined based on the importance and urgency of the message to be consumed; for example, the higher the importance or the higher the urgency, the higher the transaction level. Afterwards, based on the transaction level, determine the retry count threshold of the message to be consumed and the retry time interval that matches the current retry count of the message to be consumed. For example, the higher the transaction level, the larger the retry count threshold of the message to be consumed is set, as well as the retry time interval that increases successively. Finally, the retry count threshold and the retry time interval are determined as the retry parameters. In this way, according to the transaction level of the message to be consumed, the retry count threshold of the message to be consumed is dynamically set to obtain the retry parameters of the message to be consumed, so that the maximum retry count of each message can be dynamically adjusted to provide more retry opportunities for key messages.

[0099] 103. When the delivery of the message to be consumed fails, determine the idempotence identification information of the message to be consumed based on the idempotence identifier of the message to be consumed.

[0100] Here, the idempotent identifier of the message to be consumed can uniquely represent the message to be consumed. For example, idempotent_key is used to represent the idempotent identifier of the message to be consumed. If the idempotent_key of two messages is the same, it means that the two messages are the same message.

[0101] In some possible implementations, a universal message format is defined, and the format is as follows: (producer, producerBusId). All messages sent by the producer must use this format; producer: the service name can be used; producerBusId: the unique identifier of the message output by the producer, for example, a Universally Unique Identifier (UUID) can be used.

[0102] In some possible implementations, the above step 103 can be performed by Figure 2 The steps described achieve:

[0103] 201. When the delivery of the message to be consumed fails, determine the service name of the producer end corresponding to the message to be consumed and the identifier of the producer end.

[0104] Here, if the delivery of the message to be consumed fails, the message sending compensation task is triggered, and the service name of the producer end corresponding to the message to be consumed and the identifier of the producer end are obtained at the same time. That is, the service name of the producer end and the universal unique identifier that can uniquely represent the producer end are obtained, such as UUID.

[0105] 202. Determine idempotence identification information of the message to be consumed based on the idempotence identifier, the service name, and the identifier of the producer end.

[0106] In some possible implementations, by introducing an idempotent auxiliary table, the service name and the identifier of the producer end are written into the idempotent auxiliary table with the idempotent identifier as a constraint condition, so as to obtain the idempotent identifier information of the message to be consumed; wherein the constraint condition is used to constrain the uniqueness of the messages with the same idempotent identifier written into the idempotent auxiliary table. In this way, the idempotent identifier is used as a constraint condition, and a unique constraint is added to the idempotent auxiliary table, so that when multiple threads write data to this table at the same time, if the idempotent_key is the same, only one will succeed, and the others will violate the unique constraint and trigger an exception, resulting in failure. In this way, by using the service name and the identifier of the producer end as the message format, and combining the idempotent identifier, the idempotent identifier information of the message to be consumed is generated, so that the idempotent identifier information of the message to be consumed can not only indicate the uniqueness of the message to be consumed, but also ensure the idempotence of the message to be consumed, that is, ensure that the same message is only successfully consumed once.

[0107] In some possible implementations, when multiple messages with the same idempotency identifier are simultaneously written into the idempotent auxiliary table, the write status of the multiple messages is detected; wherein the write status includes: write failure and write success. Afterwards, when it is detected that the write status of any message is write success, based on the constraint conditions of the idempotent auxiliary table, a message write exception is triggered, so that the write status of other messages in the multiple messages is write failure; wherein the other messages are messages in the multiple messages other than the any one message.

[0108] Here, if multiple threads write data to the idempotent auxiliary table at the same time, and the idempotent identifiers of these multiple threads are the same, that is, multiple messages have the same idempotent identifier; then since the constraint condition of the idempotent auxiliary table is to allow only one of the multiple messages with the same idempotent identifier to be written successfully, when the write status of any message is detected as being written successfully for the first time, a message write exception is triggered, causing the writing of other messages to fail, so as to achieve the purpose of allowing only one message to be written successfully, thereby ensuring that the same message is only successfully consumed once.

[0109] 104. Consume the message to be consumed based on the idempotent identification information and the idempotent auxiliary table.

[0110] The idempotent auxiliary table is used to allow one message among multiple messages to be written successfully when multiple messages with the same idempotent identifier are written at the same time.

[0111] In some possible implementations, the above step 104 can be performed by Figure 3 The steps shown achieve:

[0112] 301. In the idempotent auxiliary table, determine whether there is a consumption record corresponding to the idempotent identification information.

[0113] 302. If the consumption record does not exist in the idempotent auxiliary table, start the consumption transaction.

[0114] 303. Based on the consumption transaction, insert the idempotent identification information of the message to be consumed into the idempotent auxiliary table.

[0115] Here, after the consumption transaction is started, the idempotence identifier in the idempotence identification information is used as a constraint condition, and the service name and the identifier of the producer end in the idempotence identification information are used as threads and written into the idempotence auxiliary table; the following code can be used to implement "create table if not exists t_idempotent(id varchar(50)primary key comment'id,primary key',idempotent_key varchar(500)not null comment'key that needs to ensure idempotence',unique key uq_idempotent_key(idempotent_key))comment'idempotence auxiliary table';". In this way, the idempotence identification information is used as the only constraint condition, and the service name and the identifier of the producer end in the idempotence identification information are written into the idempotence auxiliary table, so that when multiple threads write data to this table at the same time, if the idempotence identification information is the same, only one will succeed, thereby ensuring that the same message to be consumed can only be consumed once by the same consumer end, so as to realize idempotent consumption at the consumer end.

[0116] 304. Consume the message to be consumed based on the idempotent auxiliary table.

[0117] In the above steps 301 to 304, idempotent consumption can be achieved through the idempotent auxiliary table, and the process is as follows:

[0118] Here, the idempotent identification information consists of the (producer, producerBusId) in the message to be consumed plus the complete class name of the consumer, which means that the same message can only be consumed once by the same consumer.

[0119] By judging whether there is a consumption record corresponding to the idempotency identification information in the idempotency auxiliary table, that is, judging whether there is a record of the message to be consumed being consumed in the idempotency auxiliary table. If there is a consumption record corresponding to the idempotency identification information in the idempotency auxiliary table, it means that the message to be consumed has been processed and success is directly returned.

[0120] If there is no consumption record corresponding to the idempotent identification information in the idempotent auxiliary table, the code "IdempotentPO idempotentPO = select * from t_idempotent where idempotent_key = #{idempotentKey}; if (idempotentPO! = null) {return

[0121] "SUCCESS";}" starts the Spring transaction, that is, starts the consumer transaction.

[0122] After the consumption transaction is started, the idempotent identification information of the message to be consumed is inserted into the idempotent auxiliary table. If there are concurrent threads sending it, only one thread will succeed, and the others will report an exception, causing the transaction to roll back. This can be achieved through the following code: "String idempotentId = ""; insert into t_idempotent(id, idempotent_key) values(#{idempotentId}, #{idempotentKey}); commit spring transaction". In this way, after the transaction is committed, the message to be consumed can be consumed from the idempotent auxiliary table. In this way, when there is no consumption record corresponding to the idempotent identification information in the idempotent auxiliary table, the consumption transaction is started, so that the idempotent identification information of the message to be consumed is inserted into the idempotent auxiliary table, so that the consumer end consumes the message to be consumed. Since when multiple threads are written to the idempotent auxiliary table at the same time, if the idempotent_key is the same, only one will succeed, so it can ensure that the consumer end only successfully consumes the same message once.

[0123] 105. When the consumption of the message to be consumed fails, a decaying consumption retry is performed on the message to be consumed based on a retry parameter of the message to be consumed.

[0124] Here, the retry parameters of the message to be consumed include: a retry count threshold and a decaying retry time interval. If the message to be consumed fails to be consumed, then the message to be consumed is retried in a decaying manner according to the retry count threshold in the retry parameters and the retry time interval that changes step by step.

[0125] In some possible implementations, the above step 105 can be performed by Figure 4 The steps shown achieve:

[0126] 401. When the consumption of the message to be consumed fails, obtain and update the record log corresponding to the message to be consumed.

[0127] Here, the record log corresponding to the message to be consumed includes: the number of retries of the message to be consumed, and the consumption status of each message, such as consumption success data and consumption failure data.

[0128] If the consumption of the message to be consumed fails, the log corresponding to the message to be consumed is read. At the same time, the message to be consumed is stored in the dead letter queue and the log is updated. If the message to be consumed is consumed successfully, the data of successful consumption is updated in the log.

[0129] 402. Determine the current retry count of the message to be consumed based on the record log corresponding to the message to be consumed.

[0130] Here, by reading the record log, the current retry count of the information to be consumed can be determined from the record log.

[0131] 403: If the current number of retries does not reach the retry number threshold in the retry parameter, based on the retry time interval that matches the current number of retries, deliver the delayed message corresponding to the message to be consumed.

[0132] Here, by judging whether the current number of retries reaches the retry threshold in the retry parameters, if the current number of retries does not reach the retry threshold, the delayed message corresponding to the message to be consumed is delivered to the message queue according to the retry time interval matched by the current number of retries.

[0133] In some possible implementations, the above step 403 may be implemented by the following steps 431 to 433 (not shown):

[0134] 431. If the current number of retries does not reach the retry number threshold in the retry parameters, based on the current number of retries, the retry time interval of the previous consumption retry is extended to obtain the retry time interval of the current consumption retry.

[0135] For example, after the consumption of the message fails, it will retry again after a period of time (for example, 10 seconds); if it still fails, it will retry again after a period of time (for example, 20 seconds). For example, if the current number of retries (that is, the current cumulative number of failures) is between 1 and 5, it will retry every 10 seconds after each failure; between 6 and 10, it will retry every 20 seconds; between 11 and 20, it will retry every 30 seconds; but when the number of retries reaches the upper limit, for example, the number of retries reaches 50, the machine reaches the maximum number, and it will no longer retry consumption, but output an alarm message to request manual intervention.

[0136] 432. Generate a delay message corresponding to the message to be consumed based on the retry time interval of the current consumption retry.

[0137] Here, according to the retry time interval of the current consumption retry, a delayed message that can delay the retry time interval is generated, and the message content of the delayed message is the same as the message to be consumed. After the consumption fails, a delayed message will be delivered, and the content of the delayed message is the same as the content of the message to be consumed.

[0138] 433. Deliver the delayed message to the message queue of the message to be consumed.

[0139] Here, after the delay time reaches the retry interval of the current consumption retry, the delayed message will enter the original message queue of the message to be consumed, triggering consumption again. In this way, if the current retry count does not reach the retry count threshold in the retry parameters, the retry interval of the previous consumption retry will be delayed, thereby achieving attenuated consumption retry.

[0140] In some embodiments, if the current number of retries reaches the retry number threshold in the retry parameter, consumption retries are stopped and an alarm message is generated; wherein the alarm message is used to prompt the client to process the message to be consumed.

[0141] Here, if the current number of retries reaches the retry number threshold in the retry parameters, it means that the number of retries is too large, so the consumption retry is stopped, an alarm message is generated and output, and the user end is requested to perform manual intervention.

[0142] 404: Retry consumption of the delayed message.

[0143] Here, the delayed message is delivered to the message queue corresponding to the message to be consumed, so that the content in the delayed message is consumed and retried in the message queue. In this way, after the consumption of the message to be consumed fails, the record log is updated, and if consumption retry is required, consumption will be performed. By obtaining the record log corresponding to the message to be consumed, according to whether a retry is required and the next retry time, if a retry is required, a retry message is delivered to continue the retry. In some possible implementations, in the case of consumption failure, the message to be consumed can be delivered to the queue again in a delayed manner to achieve consumption retry.

[0144] In an embodiment of the present invention, the acquired information to be consumed is delivered through a message queue; and in the case where the delivery of the message to be consumed fails, the idempotency identification information of the message to be consumed is determined based on the idempotency identifier of the message to be consumed; thereby, the message to be consumed is consumed through the idempotency identification information and the idempotency auxiliary table; wherein the idempotency auxiliary table is used to allow one of the multiple messages to be written successfully when multiple messages with the same idempotency identifier are written at the same time; in this way, by introducing the idempotency auxiliary table and setting the idempotency identification information of the message to be consumed, the workload of the developer in implementing the idempotency logic can be simplified, and the consistency of the message processing logic can be ensured. Afterwards, in the case where the consumption of the message to be consumed fails, the attenuated consumption retry is performed through the message queue and the retry parameters of the message to be consumed, thereby realizing dynamic consumption retry, reducing the repeated development of the code and the complexity of the system.

[0145] A retry method for message consumption failure provided by an embodiment of the present invention can also be used Figure 5 The steps shown achieve:

[0146] 501, get the message from the information (message).

[0147] Here, after receiving the message, whether a retry is required and the next retry interval are dynamically configured according to the transaction level of the message.

[0148] 502, delivering the message to the message queue.

[0149] 503: Use a message queue to store the message.

[0150] Here, if the delivery fails, a message sending compensation task is triggered to facilitate message delivery.

[0151] 504, idempotent consumption of messages in the message queue.

[0152] 505. If the consumption fails, the message is retried according to the decaying retry time interval.

[0153] The embodiment of the present invention abstracts the business scenarios of complex, multi-source data message delivery and consumption through dynamic message level templates, and provides general consumption delivery, delayed delivery, transaction delivery, and idempotent consumption; after consumption failure, a general solution of decaying retry is adopted, thereby reducing the repeated development of code, improving productivity, and avoiding waste of labor. Against the backdrop of a dramatic increase in massive data, the embodiment of the present invention enhances fault tolerance for business scenarios that require high-reliability message delivery, effectively avoids business errors to a certain extent, and improves the reliability of message processing and the stability of the system.

[0154] The embodiment of the present invention provides a retry system for message consumption failure, please refer to Figure 6 , which shows a schematic diagram of the composition structure of a retry system for message consumption failure provided by an embodiment of the present invention, the system 600 includes:

[0155] Acquisition module 601, used to acquire the message to be consumed;

[0156] A delivery module 602, used for delivering the to-be-consumed message based on a message queue;

[0157] Determination module 603, used for determining idempotence identification information of the message to be consumed based on the idempotence identifier of the message to be consumed when the delivery of the message to be consumed fails;

[0158] A consumption module 604 is used to consume the message to be consumed based on the idempotent identification information and the idempotent auxiliary table; wherein the idempotent auxiliary table is used to allow one of the multiple messages to be written successfully when multiple messages with the same idempotent identifier are written at the same time;

[0159] The consumption retry module 605 is used to perform a decaying consumption retry on the message to be consumed based on the retry parameters of the message to be consumed when the consumption of the message to be consumed fails.

[0160] In some possible implementations, the consumption retry module 605 is also used to determine the transaction level of the message to be consumed based on the importance and urgency of the message to be consumed; based on the transaction level, determine the retry number threshold of the message to be consumed and the retry time interval that matches the current retry number of the message to be consumed; wherein the retry time interval increases with the increase of the current retry number; and determine the retry number threshold and the retry time interval as the retry parameters.

[0161] In some possible implementations, the determination module 603 is also used to determine the service name of the producer side corresponding to the message to be consumed and the identifier of the producer side when the delivery of the message to be consumed fails; and determine the idempotence identification information of the message to be consumed based on the idempotence identifier, the service name and the identifier of the producer side.

[0162] In some possible implementations, the determination module 603 is also used to write the service name and the identifier of the producer end into the idempotency auxiliary table with the idempotency identifier as a constraint condition to obtain the idempotency identification information of the message to be consumed; wherein the constraint condition is used to constrain the uniqueness of messages with the same idempotency identifier written into the idempotency auxiliary table.

[0163] In some possible implementations, the consumption module 604 is also used to determine whether there is a consumption record corresponding to the idempotent identification information in the idempotent auxiliary table; if the consumption record does not exist in the idempotent auxiliary table, start a consumption transaction; based on the consumption transaction, insert the idempotent identification information of the message to be consumed into the idempotent auxiliary table; and consume the message to be consumed based on the idempotent auxiliary table.

[0164] In some possible implementations, the consumption module 604 is further used to

[0165] When multiple messages having the same idempotency identifier are simultaneously written into the idempotency auxiliary table, detecting the writing status of the multiple messages;

[0166] When it is detected for the first time that the write status of any message is write successful, based on the constraint conditions of the idempotent auxiliary table, a message write exception is triggered so that the write status of other messages among the multiple messages is write failed; wherein, the other messages are messages among the multiple messages other than the any one message.

[0167] In some possible implementations, the consumption retry module 605 is also used to obtain and update the record log corresponding to the message to be consumed when the consumption of the message to be consumed fails; determine the current retry number of the message to be consumed based on the record log corresponding to the message to be consumed; if the current retry number does not reach the retry number threshold in the retry parameters, deliver the delayed message corresponding to the message to be consumed based on the retry time interval matching the current retry number; and retry consumption of the delayed message.

[0168] In some possible implementations, the consumption retry module 605 is also used to, if the current number of retries does not reach the retry number threshold in the retry parameters, extend the retry time interval of the previous consumption retry based on the current number of retries to obtain the retry time interval of the current consumption retry; generate a delayed message corresponding to the message to be consumed based on the retry time interval of the current consumption retry; wherein the message content of the delayed message is the same as the message content of the message to be consumed; and deliver the delayed message to the message queue of the message to be consumed.

[0169] In some possible implementations, the consumption retry module 605 is also used to stop consumption retry and generate an alarm message if the current number of retries reaches a retry threshold in the retry parameters; wherein the alarm message is used to prompt the client to process the message to be consumed.

[0170] In some possible implementations, the acquisition module 601 is also used to obtain any message in the communication information; determine the association record between any message and the consumer end; based on the association record, determine whether any message needs to be consumed; if any message needs to be consumed, determine that any message is the message to be consumed.

[0171] Optionally, the transmission medium can be a wired link (for example, but not limited to, coaxial cable, optical fiber and digital subscriber line (DSL), etc.) or a wireless link (for example, but not limited to, wireless Fidelity (WIFI), Bluetooth and mobile device network, etc.). It should be noted that: the system provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device is divided into different functional modules to complete all or part of the functions described above. In addition, the method embodiments provided in the above embodiments belong to the same concept. The specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0172] Figure 7 is a schematic diagram of the structure of a computer device provided by an embodiment of the present invention. Figure 7 As shown, the computer device 700 includes: a memory 701, a processor 702, and a computer program 703 stored in the memory 701 and running on the processor 702, wherein when the processor 702 executes the computer program 703, the computer device can execute any of the message consumption failure retry methods introduced above.

[0173] In addition, an embodiment of the present invention also protects a system, which may include a memory and a processor, wherein an executable program code is stored in the memory, and the processor is used to call and execute the executable program code to perform a message consumption failure retry method provided by an embodiment of the present invention. This embodiment can divide the system into functional modules according to the above method example. For example, it can correspond to each functional module, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware. It should be noted that the division of modules in this embodiment is schematic, which is only a logical function division, and there may be other division methods in actual implementation. It should be noted that all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module, which will not be repeated here.

[0174] It should be understood that the system provided in this embodiment is used to execute the above-mentioned retry method for failed message consumption, so the same effect as the above-mentioned implementation method can be achieved. In the case of an integrated unit, the system may include a processing module and a storage module. Among them, when the system is applied to a device, the processing module can be used to control and manage the actions of the device. The storage module can be used to support the device to execute mutual program codes, etc. Among them, the processing module can be a processor or a controller, which can implement or execute various exemplary logical boxes, modules and circuits described in conjunction with the disclosure of the present invention. The processor can also be a combination that implements a computing function, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, etc., and the storage module can be a memory.

[0175] In addition, the system provided by the embodiment of the present invention may be specifically a chip, a component or a module, and the chip may include a connected processor and a memory; wherein the memory is used to store instructions, and when the processor calls and executes the instructions, the chip can execute a message consumption failure retry method provided in the above embodiment. This embodiment also provides a computer-readable storage medium, in which a computer program code is stored, and when the computer program code is run on a computer, the computer executes the above-mentioned related method steps to implement a message consumption failure retry method provided in the above embodiment.

[0176] This embodiment also provides a computer program product, when the computer program product is run on a computer, the computer executes the above-mentioned related steps to implement a retry method for message consumption failure provided in the above embodiment. Among them, the system, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding method provided above, so the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above, and will not be repeated here. Through the description of the above implementation mode, the technicians in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In practical applications, the above-mentioned function allocation can be completed by different functional modules as needed, that is, the internal structure of the system is divided into different functional modules to complete all or part of the functions described above. In the embodiments provided by the present invention, it should be understood that the disclosed system and method can be implemented in other ways. For example, the system embodiment described above is only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interface, system or unit, which may be electrical, mechanical or other forms.

[0177] It should be noted that the sequence of the above-mentioned embodiments of the present invention is only for description and does not represent the advantages and disadvantages of the embodiments. The process depicted in the accompanying drawings does not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multi-task processing and parallel processing are also possible or may be advantageous. The various embodiments in this specification are described in a progressive manner, and the same and similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments. The above content is only a specific implementation method of the present invention, but the protection scope of the present invention is not limited to this. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be covered within the protection scope of the present invention.

Claims

1. A retry method for message consumption failure, characterized in that: The retry method for message consumption failure includes: Get the message to be consumed; Based on the message queue, the message to be consumed is delivered; In the case where the delivery of the message to be consumed fails, determining the idempotence identification information of the message to be consumed based on the idempotence identifier of the message to be consumed; Based on the idempotent identification information and the idempotent auxiliary table, the message to be consumed is consumed; wherein the idempotent auxiliary table is used to allow one message among the multiple messages to be written successfully when multiple messages with the same idempotent identifier are written at the same time; In the case that the consumption of the message to be consumed fails, based on the retry parameter of the message to be consumed, a decaying consumption retry is performed on the message to be consumed.

2. A retry method for message consumption failure according to claim 1, characterized in that: The method further comprises: Determining the transaction level of the message to be consumed based on the importance and urgency of the message to be consumed; Based on the transaction level, determine a retry time interval that matches a retry number threshold of the message to be consumed and a current retry number of the message to be consumed; wherein the retry time interval increases as the current retry number increases; The retry times threshold and the retry time interval are determined as the retry parameters.

3. A retry method for message consumption failure according to claim 1, characterized in that: In the case where the delivery of the message to be consumed fails, determining the idempotence identification information of the message to be consumed based on the idempotence identifier of the message to be consumed includes: In the event that the delivery of the message to be consumed fails, determining the service name of the producer end corresponding to the message to be consumed and the identifier of the producer end; Based on the idempotence identifier, the service name and the identifier of the producer end, the idempotence identification information of the message to be consumed is determined.

4. A retry method for message consumption failure according to claim 3, characterized in that: The determining, based on the idempotence identifier, the service name, and the identifier of the producer end, the idempotence identification information of the message to be consumed includes: Taking the idempotence identifier as a constraint, the service name and the identifier of the producer end are written into the idempotence auxiliary table to obtain the idempotence identification information of the message to be consumed; wherein the constraint is used to constrain the uniqueness of messages with the same idempotence identifier written into the idempotence auxiliary table.

5. A retry method for message consumption failure according to claim 1, characterized in that: The consuming the message to be consumed based on the idempotent identification information and the idempotent auxiliary table includes: In the idempotent auxiliary table, determining whether there is a consumption record corresponding to the idempotent identification information; If the consumption record does not exist in the idempotent auxiliary table, start the consumption transaction; Based on the consumption transaction, inserting the idempotent identification information of the message to be consumed into the idempotent auxiliary table; The message to be consumed is consumed based on the idempotent auxiliary table.

6. A retry method for message consumption failure according to claim 1, characterized in that: The method further comprises: When multiple messages having the same idempotency identifier are simultaneously written into the idempotency auxiliary table, detecting the writing status of the multiple messages; When it is detected for the first time that the write status of any message is write successful, based on the constraint conditions of the idempotent auxiliary table, a message write exception is triggered so that the write status of other messages among the multiple messages is write failed; wherein, the other messages are messages among the multiple messages other than the any one message.

7. A retry method for message consumption failure according to claim 1, characterized in that: In the case where the consumption of the message to be consumed fails, performing a decaying consumption retry on the message to be consumed based on a retry parameter of the message to be consumed, including: In the event that the consumption of the message to be consumed fails, obtaining and updating the record log corresponding to the message to be consumed; Determine the current retry count of the message to be consumed based on the record log corresponding to the message to be consumed; If the current number of retries does not reach the retry number threshold in the retry parameter, based on the retry time interval matched by the current number of retries, delivering the delayed message corresponding to the message to be consumed; The delayed message is retried for consumption.

8. A retry method for message consumption failure according to claim 7, characterized in that: If the current number of retries does not reach the retry number threshold in the retry parameter, based on the retry time interval matched by the current number of retries, delivering the delayed message corresponding to the message to be consumed includes: If the current number of retries does not reach the retry number threshold in the retry parameter, based on the current number of retries, the retry time interval of the previous consumption retry is extended to obtain the retry time interval of the current consumption retry; Based on the retry time interval of the current consumption retry, a delayed message corresponding to the message to be consumed is generated; wherein the message content of the delayed message is the same as the message content of the message to be consumed; The delayed message is delivered to the message queue of the message to be consumed.

9. A retry system for message consumption failure, characterized in that: The system comprises: The acquisition module is used to obtain the messages to be consumed; A delivery module, used for delivering the to-be-consumed message based on a message queue; A determination module, configured to determine, when delivery of the message to be consumed fails, idempotence identification information of the message to be consumed based on the idempotence identifier of the message to be consumed; A consumption module, used for consuming the message to be consumed based on the idempotent identification information and the idempotent auxiliary table; wherein the idempotent auxiliary table is used for allowing one message among the multiple messages to be written successfully when multiple messages with the same idempotent identifier are written at the same time; The consumption retry module is used to perform attenuated consumption retry on the message to be consumed based on the retry parameters of the message to be consumed when the consumption of the message to be consumed fails.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program code, and when the computer program code is executed on a computer, the computer is enabled to execute the message consumption failure retry method according to any one of claims 1 to 8.

Images