Kafka-based message processing method, apparatus, device, and storage medium

Abstract

The application relates to the technical field of message queues, and provides a Kafka-based message processing method and device, equipment and a storage medium, the method comprising the following steps: when target message consumption fails, a delay message corresponding to the target message is generated, the delay message containing message body content of the target message; the delay message is sent to a target Kafka delay message queue configured in advance; when the delay message in the target Kafka delay message queue expires, the message body content in the delay message is extracted, and a retransmission message corresponding to the target message is generated according to the message body content; and the retransmission message is sent to a Kafka message queue, so that a consumer obtains the retransmission message from the Kafka message queue for consumption, thereby improving the message consumption success rate. The application also relates to the blockchain technology, and the Kafka message queue can be stored in a blockchain node.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of message queue, and particularly relates to a Kafka-based message processing method and device, equipment and a storage medium. BACKGROUND

[0002] Kafka is a distributed message queue system based on a publish / subscribe mode, which can process all action flow data of consumers in a website. Kafka can be commonly applied to access logs, message services and the like. At present, the Kafka system itself only provides basic message receiving and message sending functions, and does not provide a message processing function in a message consumption failure scenario.

[0003] Therefore, how to process a message that fails to be consumed based on Kafka to improve the message consumption success rate becomes a problem to be solved. SUMMARY

[0004] The present application provides a Kafka-based message processing method, device, equipment and a storage medium, which aims to process a message that fails to be consumed based on Kafka to improve the message consumption success rate.

[0005] To achieve the above purpose, the present application provides a Kafka-based message processing method, which comprises the following steps:

[0006] When a target message fails to be consumed, a delay message corresponding to the target message is generated, and the delay message contains the message body content of the target message;

[0007] The delay message is sent to a target Kafka delay message queue;

[0008] When the delay message in the target Kafka delay message queue expires, the message body content in the delay message is extracted, and a retransmission message corresponding to the target message is generated according to the message body content;

[0009] The retransmission message is sent to a Kafka message queue, so that a consumer obtains the retransmission message from the Kafka message queue for consumption.

[0010] In addition, to achieve the above purpose, the present application further provides a Kafka-based message processing device, which comprises the following steps:

[0011] A message generation module is configured to generate a delay message corresponding to a target message when the target message fails to be consumed, and the delay message contains the message body content of the target message;

[0012] The message sending module is configured to send the delay message to a target Kafka delay message queue configured in advance.

[0013] The message management module is configured to extract the message body content in the delay message when the delay time of the delay message in the target Kafka delay message queue expires, and generate a retransmission message corresponding to the target message according to the message body content.

[0014] The message consumption module is configured to send the retransmission message to a Kafka message queue, so that a consumer obtains the retransmission message from the Kafka message queue for consumption.

[0015] In addition, to achieve the above-mentioned purpose, the present application also provides a computer device, which comprises a memory and a processor.

[0016] The memory is configured to store a computer program.

[0017] The processor is configured to execute the computer program and realize the Kafka-based message processing method as described above when executing the computer program.

[0018] In addition, to achieve the above-mentioned purpose, the present application also provides a computer readable storage medium, which stores a computer program, and the computer program realizes the steps of the Kafka-based message processing method when executed by a processor.

[0019] The present application discloses a Kafka-based message processing method, device, equipment and storage medium. When target message consumption fails, a delay message corresponding to the target message is generated, the delay message contains the message body content of the target message, and the delay message is sent to a target Kafka delay message queue configured in advance. When the delay time of the delay message in the target Kafka delay message queue expires, the message body content in the delay message is extracted, a retransmission message corresponding to the target message is generated, and the retransmission message is sent to a Kafka message queue, so that a consumer obtains the retransmission message from the Kafka message queue for consumption. By generating a retransmission message corresponding to the target message, the retransmission message is consumed, that is, the target message is re-consumed, and thus the success rate of target message consumption is improved. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0021] Figure 1 is a step schematic flowchart of a message processing method based on Kafka provided by an embodiment of the present application;

[0022] Figure 2 is a step schematic flowchart of generating a delay message corresponding to the target message provided by an embodiment of the present application;

[0023] Figure 3 is a step schematic flowchart of sending the delay message to a preconfigured target Kafka delay message queue provided by an embodiment of the present application;

[0024] Figure 4 is a message processing flowchart based on Kafka provided by an embodiment of the present application;

[0025] Figure 5 is a schematic block diagram of a message processing device based on Kafka provided by an embodiment of the present application;

[0026] Figure 6 is a structural schematic block diagram of a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0027] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0028] The flowcharts shown in the drawings are only exemplary descriptions, not necessarily including all the contents and operations / steps, and not necessarily executed in the described order. For example, some operations / steps can be decomposed, combined or partially merged, so the actual execution order can be changed according to the actual situation.

[0029] It should be understood that the terms used in this specification of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the specification and the appended claims of the present application, unless otherwise clear from the context, the singular forms “a”, “an” and “the” are intended to include the plural forms.

[0030] It should also be understood that the term “and / or” used in the specification and the appended claims of the present application means any combination of one or more of the associated listed items and all possible combinations, and includes these combinations.

[0031] Embodiments of the present application provide a Kafka-based message processing method, device and equipment and a storage medium, which are used for monitoring a webpage at a user-perceptible level.

[0032] Please refer to Figure 1 , Figure 1 is a flowchart of a Kafka-based message processing method provided by an embodiment of the present application. The method can be applied to a computer device, and the application scenario of the method is not limited in the present application. In the following, the Kafka-based message processing method is taken as an example applied to a computer device, and the Kafka-based message processing method is introduced in detail.

[0033] As Figure 1 shown, the Kafka-based message processing method specifically includes steps S101 to S104.

[0034] S101, when target message consumption fails, generating a delay message corresponding to the target message, the delay message containing the message body content of the target message.

[0035] For example, the message production end sends each message to be consumed to a Kafka message queue, wherein the Kafka message queue includes at least one. For example, the Kafka message queue includes Topic A and Topic B, and each Kafka message queue buffers each message to be consumed.

[0036] The consumer end pulls the target message to be consumed from the Kafka message queue, and then performs business processing. For example, taking the target message C to be consumed in Topic B as an example, the target message C is pulled from Topic B, and the target message C is sent to the consumer end, and the consumer end consumes the target message C to perform corresponding business processing.

[0037] When the target message consumption fails, the processing of the target message ends, and other corresponding business processing can be continued.

[0038] When the target message consumption fails, a delay message corresponding to the target message is generated, wherein the delay message contains the message body content of the target message. For example, a Kafka delay message manager is pre-configured, and the delay message corresponding to the target message is generated through the Kafka delay message manager.

[0039] In some embodiments, as Figure 2 shown, step S101 can include sub-step S1011 and sub-step S1012.

[0040] S1011, obtaining delay information corresponding to the target message.

[0041] Exemplarily, the delay information comprises at least one of a delay level of the target message, and a number of times of retransmission of the target message.

[0042] Different delay levels correspond to different delay time lengths. For example, assume that three delay levels are preconfigured: delay level 1, delay level 2 and delay level 3, delay level 1 corresponds to a delay of 10 minutes, delay level 2 corresponds to a delay of 60 minutes, and delay level 3 corresponds to a delay of 24 hours, and the like.

[0043] Exemplarily, the delay information further comprises a key corresponding to the target message, a Kafka message queue corresponding to the target message, and a delay message creation timestamp, and the like.

[0044] S1012, encapsulate the message body content of the target message and the delay information according to a preset delay message format, to generate the delay message.

[0045] Exemplarily, the delay message format is preconfigured as: {"key":"key of the source message", "Topic":"Topic of the source message", "value":"{ \"_delay_message_parameter_\": { \"createdDate\": delay message creation timestamp, \"delayTimeLevel\": delay level, \"retriedCount\": number of times of retransmission}, message body content of the source message}"}.

[0046] The source message is the target message; "key" and "value" are a pair of key-value pairs, "key" is a keyword, and "value" is a value; "Topic" is a Kafka message queue. For example, if the target message is previously buffered in Kafka message queue Topic A, then "Topic of the source message" is Topic A.

[0047] The delay level of the target message, the number of times of retransmission of the target message, the key corresponding to the target message, the Kafka message queue corresponding to the target message, and the delay message creation timestamp are obtained by the Kafka delay message manager, and the message body content of the target message is encapsulated according to the above-mentioned configured delay message format, to generate the delay message corresponding to the target message.

[0048] It should be noted that the delay message format can be other message formats besides the above-mentioned message formats, and the delay message format is not specifically limited herein.

[0049] S102, send the delay message to a preconfigured target Kafka delay message queue.

[0050] Exemplarily, at least one Kafka delay message queue is pre-configured, and the Kafka delay message queue is used to buffer the generated delay message.

[0051] After the delay message corresponding to the target message is generated, the target Kafka delay message queue corresponding to the delay message is determined by the Kafka delay message manager, and the delay message is sent to the target Kafka delay message queue.

[0052] Exemplarily, if the Kafka delay message queue has only one, the Kafka delay message queue is directly determined as the target Kafka delay message queue, and the delay message is sent to the target Kafka delay message queue.

[0053] Exemplarily, if the Kafka delay message queue includes multiple, such as Figure 3 As shown in the figure, step S102 can include sub-step S1021 and sub-step S1022.

[0054] S1021, from the pre-configured multiple Kafka delay message queues, select the Kafka delay message queue matching the delay information as the target Kafka delay message queue;

[0055] S1022, send the delay message to the target Kafka delay message queue.

[0056] For example, different Kafka delay message queues correspond to buffering delay messages of different message types, and based on the message type of the generated delay message, from the multiple Kafka delay message queues, select the Kafka delay message queue corresponding to the message type of the delay information as the target Kafka delay message queue.

[0057] After the target Kafka delay message queue is determined, the delay message is sent to the target Kafka delay message queue by the Kafka delay message manager, and the delay message is buffered in the target Kafka delay message queue.

[0058] In some embodiments, from the pre-configured multiple Kafka delay message queues, select the Kafka delay message queue matching the delay information as the target Kafka delay message queue, can include:

[0059] Obtain the message retransmission number range corresponding to each Kafka delay message queue and the retransmission number of the target message;

[0060] When the number of times of retransmission of the target message is within the range of the number of times of retransmission of messages corresponding to the first Kafka delay message queue, the first Kafka delay message queue is determined as the target Kafka delay message queue.

[0061] When configuring each Kafka delay message queue, a range of the number of times of retransmission of messages corresponding to each Kafka delay message queue is configured. Different Kafka delay message queues correspond to different ranges of the number of times of retransmission of messages.

[0062] For example, N Kafka delay message queues, Delay_Topic_1, Delay_Topic_2, Delay_Topic_3,..., Delay_Topic_N, are pre-configured, each of which corresponds to a respective range of the number of times of retransmission of messages. For example, the Kafka delay message queue Delay_Topic_1 corresponds to the range [0, 3] of the number of times of retransmission of messages, the Kafka delay message queue Delay_Topic_2 corresponds to the range (3, 6] of the number of times of retransmission of messages, the Kafka delay message queue Delay_Topic_3 corresponds to the range (6, 10] of the number of times of retransmission of messages, and so on.

[0063] After generating the delay message corresponding to the target message, the range of the number of times of retransmission of messages corresponding to each pre-configured Kafka delay message queue is obtained, and the number of times of retransmission of the target message is determined. When the number of times of retransmission of the target message is within the range [0, 3] of the number of times of retransmission of messages corresponding to the Kafka delay message queue Delay_Topic_1, that is, the number of times of retransmission of the target message is not more than 3, the generated delay message corresponding to the target message is sent to the Kafka delay message queue Delay_Topic_1.

[0064] When the number of times of retransmission of the target message is within the range (3, 6] of the number of times of retransmission of messages corresponding to the Kafka delay message queue Delay_Topic_2, that is, the number of times of retransmission of the target message is more than 3 and not more than 6, the generated delay message corresponding to the target message is sent to the Kafka delay message queue Delay_Topic_2.

[0065] When the number of times of retransmission of the target message is within the range (6, 10] of the number of times of retransmission of messages corresponding to the Kafka delay message queue Delay_Topic_3, that is, the number of times of retransmission of the target message is more than 6 and not more than 10, the generated delay message corresponding to the target message is sent to the Kafka delay message queue Delay_Topic_3.

[0066] In some embodiments, the selecting the target Kafka delay message queue from the plurality of pre-configured Kafka delay message queues can comprise:

[0067] obtaining a message delay level corresponding to each of the plurality of Kafka delay message queues, and a delay level of the target message;

[0068] when the delay level of the target message is consistent with the message delay level corresponding to the second Kafka delay message queue, determining the second Kafka delay message queue as the target Kafka delay message queue.

[0069] In some embodiments, the obtaining the message delay level corresponding to each of the plurality of Kafka delay message queues can comprise:

[0070] For example, the message delay level corresponding to the Kafka delay message queue Delay_Topic_1 is pre-configured as delay level 1, the message delay level corresponding to the Kafka delay message queue Delay_Topic_2 is pre-configured as delay level 2, and the message delay level corresponding to the Kafka delay message queue Delay_Topic_3 is pre-configured as delay level 3.

[0071] In some embodiments, the obtaining the message delay level corresponding to each of the plurality of Kafka delay message queues can comprise:

[0072] In some embodiments, the obtaining the message delay level corresponding to each of the plurality of Kafka delay message queues can comprise:

[0073] querying a plurality of configuration information files corresponding to the plurality of Kafka delay message queues, wherein the configuration information file comprises configuration information corresponding to each of the plurality of Kafka delay message queues;

[0074] extracting the message delay level corresponding to each of the plurality of Kafka delay message queues from the configuration information corresponding to each of the plurality of Kafka delay message queues.

[0075] For example, when configuring the plurality of Kafka delay message queues, configuration information corresponding to each of the plurality of Kafka delay message queues is configured. The configuration information corresponding to each of the plurality of Kafka delay message queues comprises at least one of a message retransmission number range corresponding to each of the plurality of Kafka delay message queues and a message delay level corresponding to each of the plurality of Kafka delay message queues. The configuration information corresponding to each of the plurality of Kafka delay message queues is summarized to generate a corresponding configuration information file and saved.

[0076] After generating the delay message corresponding to the target message, the saved configuration information file is queried to obtain the configuration information corresponding to each Kafka delay message queue, and the message retransmission number range corresponding to each Kafka delay message queue, the message delay level corresponding to each Kafka delay message queue, and the like are extracted from the configuration information corresponding to each Kafka delay message queue.

[0077] For example, the mapping relationship between the message retransmission number range and the delay level is set in advance, for example, the retransmission number range [0, 3] corresponds to delay level 1, the retransmission number range (3, 6] corresponds to delay level 2, and the retransmission number range (6, 10] corresponds to delay level 3.

[0078] After obtaining the retransmission number of the target message, the retransmission number range in which the retransmission number of the target message is located is determined based on the mapping relationship between the message retransmission number range and the delay level, and the delay level corresponding to the retransmission number range is determined as the delay level of the target message.

[0079] For example, if the retransmission number of the target message is within the retransmission number range [0, 3], the delay level of the target message is determined as delay level 1. If the retransmission number of the target message is within the retransmission number range (3, 6], the delay level of the target message is determined as delay level 2. If the retransmission number of the target message is within the retransmission number range (6, 10], the delay level of the target message is determined as delay level 3.

[0080] When the delay level of the target message is delay level 1, the generated delay message corresponding to the target message is sent to the Kafka delay message queue Delay_Topic_1.

[0081] When the delay level of the target message is delay level 2, the generated delay message corresponding to the target message is sent to the Kafka delay message queue Delay_Topic_2.

[0082] When the delay level of the target message is delay level 3, the generated delay message corresponding to the target message is sent to the Kafka delay message queue Delay_Topic_3.

[0083] S103, when the delay message in the target Kafka delay message queue expires, the message body content in the delay message is extracted, and a retransmission message corresponding to the target message is generated according to the message body content.

[0084] Exemplarily, the Kafka delay message manager manages each delay message in each Kafka delay message queue, and judges whether each delay message is expired.

[0085] In some embodiments, before extracting the message body content in the delay message when the delay message in the target Kafka delay message queue is expired, the method can comprise:

[0086] adding the delay time corresponding to the delay level to the delay message creation timestamp to determine the delay expiration time corresponding to the delay message;

[0087] if the current time has not reached the delay expiration time, it is determined that the delay message has not expired;

[0088] if the current time reaches the delay expiration time, it is determined that the delay message is expired.

[0089] Exemplarily, the Kafka delay message manager extracts the delay message creation timestamp, delay level and other information in the delay message, and then adds the delay time corresponding to the delay level to the delay message creation timestamp to determine the delay expiration time corresponding to the delay message based on the delay time corresponding to the delay level.

[0090] For example, if the delay level corresponding to the delay message is delay level 1, and the delay time corresponding to delay level 1 is 10 minutes, then the delay expiration time corresponding to the delay message is determined as the time corresponding to the delay message creation timestamp plus 10 minutes.

[0091] For another example, if the delay level corresponding to the delay message is delay level 2, and the delay time corresponding to delay level 2 is 60 minutes, then the delay expiration time corresponding to the delay message is determined as the time corresponding to the delay message creation timestamp plus 60 minutes.

[0092] For another example, if the delay level corresponding to the delay message is delay level 3, and the delay time corresponding to delay level 3 is 24 hours, then the delay expiration time corresponding to the delay message is determined as the time corresponding to the delay message creation timestamp plus 24 hours.

[0093] if the current time has not reached the delay expiration time corresponding to the delay message, it is determined that the delay message has not expired.

[0094] For example, if it is determined that the delay message has not expired, the system determines whether the delay message has expired at a preset periodic time interval. For example, the preset periodic time is 1 second, and the system determines whether the delay message has expired every 1 second.

[0095] It can be understood that the preset periodic time can be flexibly set according to actual conditions, and is not specifically limited herein.

[0096] If the current time reaches the delay expiration time corresponding to the delay message, it is determined that the delay message has expired. At this time, the Kafka delay message manager extracts the message body content in the delay message, and generates a retransmission message corresponding to the target message according to the message body content.

[0097] In some embodiments, the obtained message body content is encapsulated in a preset retransmission message format to generate a retransmission message corresponding to the target message.

[0098] For example, the retransmission message format is preconfigured as follows:

[0099] {"_delay_message_parameter_":{"createdDate":delay message creation timestamp,"delayTimeLevel":delay level,"retriedCount":number of retransmissions},message body content of the source message}

[0100] The Kafka delay message manager obtains the delay level of the target message, the number of retransmissions of the target message, the delay message creation timestamp, and the message body content of the target message, and encapsulates them according to the above-mentioned configured retransmission message format to generate a retransmission message corresponding to the target message.

[0101] It should be noted that the retransmission message format can be other message formats besides the above-mentioned message formats, and the retransmission message format is not specifically limited herein.

[0102] S104, send the retransmission message to the Kafka message queue, so that the consumer obtains the retransmission message from the Kafka message queue for consumption.

[0103] After generating the retransmission message corresponding to the target message, the Kafka delay message manager sends the retransmission message to the Kafka message queue. For example, the retransmission message is sent to the Kafka message queue corresponding to the target message. For example, if the target message is originally cached in the Kafka message queue Topic A, the retransmission message is sent to the Kafka message queue Topic A.

[0104] It should be noted that the retransmission message can also be sent to other Kafka message queues other than the target message corresponding Kafka message queue, for example, if the target message is originally cached in Kafka message queue Topic A, the retransmission message is sent to Kafka message queue Topic B.

[0105] The consumer pulls the retransmission message from Kafka message queue Topic A or Topic B, and then performs corresponding business processing. If the retransmission message is consumed successfully, the processing of the target message ends, and other corresponding business processing can be continued. If the retransmission message is not consumed successfully, the message processing process for the target message in the above embodiment is continued, and thus is not described again.

[0106] By generating the retransmission message corresponding to the target message, the retransmission message is consumed, that is, the previous target message is re-consumed, thereby improving the success rate of target message consumption.

[0107] For example, as shown in FIG. 1, the message processing flow based on Kafka is as follows: Figure 4

[0108] ①, the message producer (producer) sends message a to Kafka message queue Topic A or Topic B;

[0109] ②, the message consumer (consumer) pulls message a from Topic A or Topic B, and then performs business processing. If message a is not consumed successfully, step ③ is performed; if message a is consumed successfully, step ⑥ is performed;

[0110] ③, the Kafka delay message manager packages message a into a delay message and sends it to one of the corresponding Kafka delay message queues Delay_Topic_1, …, …, Delay_Topic_N. For example, if the number of retransmissions of message a does not exceed 3 times, the delay message is sent to Delay_Topic_1; if the number of retransmissions of message a exceeds 3 times but does not exceed 6 times, the delay message is sent to Delay_Topic_2;

[0111] ④, the Kafka delay message manager consumes the messages of the Kafka delay message queue and judges whether the delay message is expired;

[0112] ⑤, if not, the interval 1 second interval time loop judges whether the delay message is expired; if yes, the delay message is pulled from the Kafka delay message queue, and the message body content is extracted and sent to Kafka message queue Topic A or Topic B;

[0113] ⑥, the consumption is successful, and the flow ends.​

[0114] In the above embodiment, when the target message consumption fails, a delay message corresponding to the target message is generated, the delay message contains the message body content of the target message, and the delay message is sent to a preconfigured target Kafka delay message queue. When the delay of the delay message in the target Kafka delay message queue expires, the message body content in the delay message is extracted, a resend message corresponding to the target message is generated, and the resend message is sent to the Kafka message queue for consumption by the consumer from the Kafka message queue.

[0115] Referring to Figure 5 , Figure 5 is a schematic block diagram of a Kafka-based message processing device provided by an embodiment of the present application, which can be configured in a computer device and used to execute the Kafka-based message processing method described above.

[0116] As Figure 5 shown, the Kafka-based message processing device 1000 includes a message generation module 1001, a message sending module 1002, a message management module 1003, and a message consumption module 1004.

[0117] The message generation module 1001 is configured to generate a delay message corresponding to a target message when the target message consumption fails, the delay message containing the message body content of the target message.

[0118] The message sending module 1002 is configured to send the delay message to a preconfigured target Kafka delay message queue.

[0119] The message management module 1003 is configured to extract the message body content in the delay message when the delay of the delay message in the target Kafka delay message queue expires, and generate a resend message corresponding to the target message according to the message body content.

[0120] The message consumption module 1004 is configured to send the resend message to a Kafka message queue for consumption by a consumer from the Kafka message queue.

[0121] In one embodiment, the message generation module 1001 is further configured to:

[0122] obtain delay information corresponding to the target message;

[0123] encapsulate the message body content of the target message and the delay information in a preset delay message format to generate the delay message.

[0124] In one embodiment, the message sending module 1002 is further configured to:

[0125] select a Kafka delay message queue matching the delay information from a plurality of preconfigured Kafka delay message queues as the target Kafka delay message queue;

[0126] send the delay message to the target Kafka delay message queue.

[0127] In one embodiment, the delay information includes the number of retransmissions of the target message, and the message sending module 1002 is further configured to:

[0128] obtain the number of retransmissions of messages corresponding to each Kafka delay message queue and the number of retransmissions of the target message;

[0129] when the number of retransmissions of the target message is within the number of retransmissions of messages corresponding to the first Kafka delay message queue, determine the first Kafka delay message queue as the target Kafka delay message queue.

[0130] In one embodiment, the delay information includes the delay level of the target message, and the message sending module 1002 is further configured to:

[0131] obtain the delay level of messages corresponding to each Kafka delay message queue and the delay level of the target message;

[0132] when the delay level of the target message is consistent with the delay level of messages corresponding to the second Kafka delay message queue, determine the second Kafka delay message queue as the target Kafka delay message queue.

[0133] In one embodiment, the message sending module 1002 is further configured to:

[0134] query configuration information files corresponding to a plurality of Kafka delay message queues; wherein the configuration information files include configuration information corresponding to each Kafka delay message queue;

[0135] extract the delay level of messages corresponding to each Kafka delay message queue from the configuration information corresponding to each Kafka delay message queue.

[0136] In one embodiment, the delay message comprises a delay level and a delay message creation timestamp, wherein different delay levels correspond to different delay durations; the message management module 1003 is further configured to:

[0137] adding the delay message creation timestamp and the delay duration corresponding to the delay level to determine a delay expiration time corresponding to the delay message;

[0138] if the current time has not reached the delay expiration time, determining that the delay message has not expired;

[0139] if the current time reaches the delay expiration time, determining that the delay message has expired.

[0140] Correspondingly, each module in the Kafka-based message processing apparatus 1000 described above corresponds to each step in the Kafka-based message processing method embodiment described above, and the functions and implementation processes thereof will not be repeated here.

[0141] The method and device of the present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in a distributed computing environment, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0142] For example, the method and device described above can be implemented as a computer program in the form of a computer program product, which can run on a computer device as shown in Figure 6 .

[0143] Please refer to Figure 6 , Figure 6 is a structural schematic block diagram of a computer device provided by an embodiment of the present application.

[0144] Please refer to Figure 6 , the computer device includes a processor and a memory connected by a system bus, wherein the memory can include a non-volatile storage medium and an internal memory.

[0145] The processor is configured to provide computing and control capabilities to support the operation of the entire computer device.

[0146] The internal memory provides an environment for the operation of a computer program in a non-volatile storage medium, which, when executed by the processor, can cause the processor to perform any kind of Kafka-based message processing method.

[0147] It should be understood that the processor can be a central processing unit (CPU), and the processor can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0148] In one embodiment, the processor is configured to execute a computer program stored in the memory to implement the following steps:

[0149] When the target message consumption fails, a delay message corresponding to the target message is generated, and the delay message contains the message body content of the target message;

[0150] The delay message is sent to a pre-configured target Kafka delay message queue;

[0151] When the delay message in the target Kafka delay message queue expires, the message body content in the delay message is extracted, and a retransmission message corresponding to the target message is generated according to the message body content;

[0152] The retransmission message is sent to the Kafka message queue, so that the consumer obtains the retransmission message from the Kafka message queue for consumption.

[0153] In one embodiment, when the processor implements the generation of the delay message corresponding to the target message, the processor is configured to implement:

[0154] Obtain delay information corresponding to the target message;

[0155] Encapsulate the message body content of the target message and the delay information according to a preset delay message format to generate the delay message.

[0156] In an embodiment, the processor, when implementing the sending of the delay message to the target Kafka delay message queue preconfigured, is configured to:

[0157] selecting a Kafka delay message queue matching the delay information from a plurality of preconfigured Kafka delay message queues as the target Kafka delay message queue;

[0158] sending the delay message to the target Kafka delay message queue.

[0159] In an embodiment, the delay information includes the number of retransmissions of the target message, and the processor, when implementing the selecting of a Kafka delay message queue matching the delay information from a plurality of preconfigured Kafka delay message queues as the target Kafka delay message queue, is configured to:

[0160] obtaining a range of the number of retransmissions of messages corresponding to each Kafka delay message queue and the number of retransmissions of the target message;

[0161] when the number of retransmissions of the target message is within the range of the number of retransmissions of messages corresponding to the first Kafka delay message queue, determining the first Kafka delay message queue as the target Kafka delay message queue.

[0162] In an embodiment, the delay information includes a delay level of the target message, and the processor, when implementing the selecting of a Kafka delay message queue matching the delay information from a plurality of preconfigured Kafka delay message queues as the target Kafka delay message queue, is configured to:

[0163] obtaining a delay level of messages corresponding to each Kafka delay message queue and the delay level of the target message;

[0164] when the delay level of the target message is consistent with the delay level of messages corresponding to the second Kafka delay message queue, determining the second Kafka delay message queue as the target Kafka delay message queue.

[0165] In an embodiment, the processor, when implementing the obtaining of the delay level of messages corresponding to each Kafka delay message queue, is configured to:

[0166] querying a configuration information file corresponding to a plurality of Kafka delay message queues; wherein the configuration information file includes configuration information corresponding to each Kafka delay message queue;

[0167] extract a message delay level corresponding to each Kafka delay message queue from the configuration information corresponding to each Kafka delay message queue.

[0168] In one embodiment, the delay message contains a delay level and a delay message creation timestamp, wherein different delay levels correspond to different delay durations.

[0169] Before the processor implements the step of extracting the message body content in the delay message when the delay of the delay message in the target Kafka delay message queue expires, the processor is configured to implement:

[0170] adding the delay message creation timestamp to the delay duration corresponding to the delay level to determine a delay expiration time corresponding to the delay message;

[0171] If the current time has not reached the delay expiration time, it is determined that the delay message has not expired.

[0172] If the current time reaches the delay expiration time, it is determined that the delay message has expired.

[0173] Embodiments of the present application also provide a computer readable storage medium.

[0174] The computer readable storage medium of the present application stores a computer program, and the computer program is executed by a processor to implement the steps of the Kafka-based message processing method as described above.

[0175] The computer readable storage medium can be an internal storage unit of the Kafka-based message processing apparatus or the computer device, such as a hard disk or a memory of the Kafka-based message processing apparatus or the computer device. The computer readable storage medium can also be an external storage device of the Kafka-based message processing apparatus or the computer device, such as a plug-in hard disk, a smart media card (SMC), a secure digital card (SD card), a flash card, etc.

[0176] Further, the computer readable storage medium can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required by a function, etc.; and the data storage area can store data created according to the use of the blockchain node, etc.

[0177] The blockchain referred to in the present application is a new application mode of distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm and other computer technologies. The blockchain is essentially a decentralized database, which is a series of data blocks associated using cryptographic methods, each data block contains a batch of network transaction information, which is used to verify the validity (anti-fake) of the information and generate the next block. The blockchain can include a blockchain underlying platform, a platform product service layer, and an application service layer, etc.

[0178] It should be noted that in this paper, the term "includes", "contains" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or system including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or includes elements inherent to such process, method, article or system. Without more limitations, the element defined by the statement "includes a" does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0179] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be covered within the protection scope of the present application.

Claims

1. A message processing method based on Kafka, characterized in that, The Kafka-based message processing method includes: When the consumption of the target message fails, a delayed message corresponding to the target message is generated. The delayed message contains the message body content of the target message. The delay information includes at least one of the delay level of the target message and the number of times the target message has been retransmitted. Send the delayed message to the pre-configured target Kafka delayed message queue; When the delayed message in the target Kafka delayed message queue expires, the message body content of the delayed message is extracted, and a retransmission message corresponding to the target message is generated based on the message body content; The retransmitted message is sent to the Kafka message queue so that the consumer can retrieve the retransmitted message from the Kafka message queue for consumption; Sending the delayed message to the pre-configured target Kafka delayed message queue includes: From the pre-configured multiple Kafka delayed message queues, select the Kafka delayed message queue that matches the delay information corresponding to the target message, and determine it as the target Kafka delayed message queue; The delayed message is sent to the target Kafka delayed message queue.

2. The message processing method based on Kafka as described in claim 1, characterized in that, The generation of the delayed message corresponding to the target message includes: Obtain the delay information corresponding to the target message; The message body content of the target message and the delay information are encapsulated according to a preset delay message format to generate the delay message.

3. The message processing method based on Kafka as described in claim 1, characterized in that, The delay information includes the number of times the target message has been retransmitted. The step of selecting a Kafka delayed message queue that matches the delay information from a pre-configured pool of multiple Kafka delayed message queues and determining it as the target Kafka delayed message queue includes: Obtain the range of retransmission counts for each Kafka delayed message queue and the retransmission count for the target message; When the number of times the target message has been resent is within the number of times the message corresponding to the first Kafka delayed message queue has been resent, the first Kafka delayed message queue is determined as the target Kafka delayed message queue.

4. The message processing method based on Kafka as described in claim 1, characterized in that, The latency information includes the latency level of the target message. The step of selecting a Kafka delayed message queue that matches the latency information from a pre-configured pool of multiple Kafka delayed message queues and determining it as the target Kafka delayed message queue includes: Obtain the message latency level corresponding to each Kafka delayed message queue and the latency level of the target message; When the latency level of the target message is consistent with the latency level of the message corresponding to the second Kafka delayed message queue, the second Kafka delayed message queue is determined as the target Kafka delayed message queue.

5. The message processing method based on Kafka as described in claim 4, characterized in that, The process of obtaining the message latency level corresponding to each Kafka delayed message queue includes: Query the configuration information files corresponding to multiple Kafka delayed message queues; wherein, the configuration information files contain the configuration information corresponding to each Kafka delayed message queue; Extract the message delay level corresponding to each Kafka delayed message queue from the configuration information corresponding to each Kafka delayed message queue.

6. The message processing method based on Kafka as described in any one of claims 1 to 5, characterized in that, The delayed message includes a delay level and a delay message creation timestamp, where different delay levels correspond to different delay durations; Before retrieving the message body from the delayed message when it expires in the target Kafka delayed message queue, the process includes: Add the delay duration corresponding to the delay level to the timestamp of the delayed message creation to determine the delay expiration time of the delayed message; If the current time has not reached the said delay expiration time, then it is determined that the delayed message has not yet expired; If the current time reaches the specified delay expiration time, then the delayed message is determined to have expired.

7. A message processing device based on Kafka, characterized in that, The Kafka-based message processing device includes: The message generation module is used to generate a delayed message corresponding to the target message when the consumption of the target message fails. The delayed message contains the message body content of the target message. The delay information includes at least one of the delay level of the target message and the number of times the target message has been retransmitted. The message sending module is used to send the delayed message to a pre-configured target Kafka delayed message queue; The message management module is used to extract the message body content from the delayed message when the delayed message in the target Kafka delayed message queue expires, and generate a retransmission message corresponding to the target message based on the message body content. The message consumption module is used to send the retransmitted message to the Kafka message queue, so that the consumer can obtain the retransmitted message from the Kafka message queue for consumption; The message sending module is further configured to select, from a pre-configured set of multiple Kafka delayed message queues, a Kafka delayed message queue that matches the delay information corresponding to the target message, and determine it as the target Kafka delayed message queue; and send the delayed message to the target Kafka delayed message queue.

8. A computer device, characterized in that, The computer device includes a memory and a processor; The memory is used to store computer programs; The processor is configured to execute the computer program and, in executing the computer program, implement the Kafka-based message processing method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the Kafka-based message processing method as described in any one of claims 1 to 6.

Images