A method and device for implementing message delay processing based on kafka

Abstract

The application discloses a method and device for realizing message delay processing based on kafka, wherein a kafka message middleware is divided into different shards in a delay consumption subject of the message middleware according to preset different delay consumption time lengths, and a corresponding consumer is created for each shard; a program message is received, the program message is sent to a corresponding shard of the delay consumption subject according to a selected time length in the program message, and the consumer corresponding to the shard sequentially consumes; and different shards are divided and corresponding consumers are created in the kafka message middleware according to preset delay consumption time lengths, so that the messages in each partition have the same delay time length, the problem that a consumer may be forced to wait for a message with the longest delay before consuming the following messages and unnecessary delay is avoided, and the message consumption efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of message delay processing technology, and in particular to a method and apparatus for implementing message delay processing based on Kafka. Background Technology

[0002] Introducing message middleware into system development can reduce the coupling between modules and improve the availability, reliability, and scalability of the system.

[0003] Due to its advantages such as high throughput, scalability, persistence and reliability, and support for multiple consumers, Kafka has been chosen as a message middleware in many projects.

[0004] In most cases, the system receives a user request, processes it, and then sends a message via Kafka. Other system modules receive and process the message. However, in some scenarios, a delayed consumption scheme is needed, meaning messages are consumed only after a certain delay. Examples include: clearing the cache after data is stored in the database, or implementing scheduled message push functionality.

[0005] In Kafka, delayed consumption is achieved by setting message timestamps and using Kafka's timestamp indexing feature, thus avoiding frequent polling and duplicate message sending. Furthermore, Kafka maintains an ordered, immutable message queue within each partition, enabling it to consume messages in order and supporting delayed message processing.

[0006] Within a partition, messages can only be consumed in the order they were written. If messages with different delays are mixed in the same partition, consumers may be forced to wait for the longest-delayed message before consuming subsequent messages, even if the delay time for the later messages has elapsed. For example, suppose there is a partition containing messages with a 10-minute delay and messages with a 1-minute delay. If a consumer consumes a message with a 10-minute delay, it must wait 10 minutes before consuming the next message, even if the next message only has a 1-minute delay. This causes unnecessary latency. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to provide a method and apparatus for message delay processing based on Kafka, so as to solve the problem of unnecessary delay in Kafka's delayed message processing.

[0008] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0009] A method for implementing message delay processing based on Kafka, including the following steps:

[0010] S1. The Kafka message middleware divides the message middleware into shards according to different preset delayed consumption durations, and creates a corresponding consumer for each shard.

[0011] S2. Receive program messages, and according to the duration selected in the program messages, send the program messages to the corresponding shards of the delayed consumption topic, so that the consumers corresponding to the shards can consume them sequentially.

[0012] To solve the above-mentioned technical problems, another technical solution adopted by the present invention is as follows:

[0013] An apparatus for message delay processing based on Kafka includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it performs the following steps:

[0014] S1. The Kafka message middleware divides the message middleware into shards according to different preset delayed consumption durations, and creates a corresponding consumer for each shard.

[0015] S2. Receive program messages, and according to the duration selected in the program messages, send the program messages to the corresponding shards of the delayed consumption topic, so that the consumers corresponding to the shards can consume them sequentially.

[0016] The beneficial effects of the present invention are as follows: The method and apparatus of the present invention for message delay processing based on Kafka divides different shards and creates corresponding consumers in the Kafka message middleware according to the predetermined delay consumption duration. This can ensure that messages in each partition have the same delay duration, thereby avoiding the problem that consumers may be forced to wait for the message with the longest delay before consuming subsequent messages, causing unnecessary delays and improving message consumption efficiency. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating a method for message delay processing based on Kafka, according to an embodiment of the present invention.

[0018] Figure 2 This is a structural diagram of a device for message delay processing based on Kafka, according to an embodiment of the present invention.

[0019] Figure 3 This is an example architecture diagram of a method for message delay processing based on Kafka according to an embodiment of the present invention;

[0020] Label Explanation:

[0021] 1. A device for message delay processing based on Kafka; 2. Processor; 3. Memory. Detailed Implementation

[0022] To explain in detail the technical content, objectives, and effects of the present invention, the following description is provided in conjunction with the embodiments and accompanying drawings.

[0023] Please refer to Figure 1 A method for implementing message delay processing based on Kafka, including the following steps:

[0024] S1. The Kafka message middleware divides the message middleware into shards according to different preset delayed consumption durations, and creates a corresponding consumer for each shard.

[0025] S2. Receive program messages, and according to the duration selected in the program messages, send the program messages to the corresponding shards of the delayed consumption topic, so that the consumers corresponding to the shards can consume them sequentially.

[0026] As can be seen from the above description, the beneficial effects of the present invention are as follows: The method and apparatus of the present invention for message delay processing based on Kafka divides different shards and creates corresponding consumers in the Kafka message middleware according to the predetermined delay consumption duration. This can ensure that messages in each partition have the same delay duration, thereby avoiding the problem that consumers may be forced to wait for the message with the longest delay before consuming subsequent messages, causing unnecessary delays and improving message consumption efficiency.

[0027] Furthermore, the program message includes a consumption time, which is determined based on the message sending time and the delay duration;

[0028] The sequential consumption includes the following steps:

[0029] S21. The consumer pulls the program messages in batches from the shards and starts multiple consumer threads to process the messages concurrently.

[0030] S22. The consumption thread determines whether the current time has reached the consumption time in the program message. If so, it consumes the message; otherwise, it sleeps according to the consumption time and consumes the message after the sleep ends.

[0031] As described above, consumers can retrieve program messages from shards in batches and process them using multi-threading, which effectively improves message processing efficiency.

[0032] Furthermore, it also includes the following steps:

[0033] S23. After the consumer finishes consuming the messages in the current batch, it submits the offset and updates the position of the consumer in the segment.

[0034] As described above, in Kafka, a consumer's position is maintained by an offset, representing the current position the consumer is consuming. After a consumer pulls a certain number of messages and completes processing, it needs to commit the offset.

[0035] Furthermore, the specific details of the submitted offset are as follows:

[0036] The offset is actively submitted by calling the API;

[0037] The API includes the commitSync() or commitAsync() methods.

[0038] As described above, offset commits can be made in two ways: manual commit and automatic commit. Automatic commit is the simplest method, which can be enabled by setting enable.auto.commit=true in the consumer configuration and the commit frequency can be controlled by auto.commit.interval.ms. However, this application uses the manual commit method, in which the consumer calls the API to commit the offset after processing the message, which has higher flexibility and accuracy.

[0039] Furthermore, step S1 specifically includes:

[0040] The Kafka message middleware is based on a preset custom partitioner. According to different preset delayed consumption durations, it divides the delayed consumption topics in the message middleware into shards and creates a corresponding consumer for each shard.

[0041] The custom partitioner inherits from the org.apache.kafka.clients.producer.Partitioner interface;

[0042] The partition method in the org.apache.kafka.clients.producer.Partitioner interface is pre-rewritten to calculate the shard to which a message should belong based on the message's delayed consumption duration.

[0043] As described above, Kafka can divide shards based on the latency of consumption and calculate the shard to which program messages belong.

[0044] Please refer to Figure 2An apparatus for message delay processing based on Kafka includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it performs the following steps:

[0045] S1. The Kafka message middleware divides the message middleware into shards according to different preset delayed consumption durations, and creates a corresponding consumer for each shard.

[0046] S2. Receive program messages, and according to the duration selected in the program messages, send the program messages to the corresponding shards of the delayed consumption topic, so that the consumers corresponding to the shards can consume them sequentially.

[0047] As can be seen from the above description, the beneficial effects of the present invention are as follows: The method and apparatus of the present invention for message delay processing based on Kafka divides different shards and creates corresponding consumers in the Kafka message middleware according to the predetermined delay consumption duration. This can ensure that messages in each partition have the same delay duration, thereby avoiding the problem that consumers may be forced to wait for the message with the longest delay before consuming subsequent messages, causing unnecessary delays and improving message consumption efficiency.

[0048] Furthermore, the program message includes a consumption time, which is determined based on the message sending time and the delay duration;

[0049] The sequential consumption includes the following steps:

[0050] S21. The consumer pulls the program messages in batches from the shards and starts multiple consumer threads to process the messages concurrently.

[0051] S22. The consumption thread determines whether the current time has reached the consumption time in the program message. If so, it consumes the message; otherwise, it sleeps according to the consumption time and consumes the message after the sleep ends.

[0052] As described above, consumers can retrieve program messages from shards in batches and process them using multi-threading, which effectively improves message processing efficiency.

[0053] Furthermore, it also includes the following steps:

[0054] S23. After the consumer finishes consuming the messages in the current batch, it submits the offset and updates the position of the consumer in the segment.

[0055] As described above, in Kafka, a consumer's position is maintained by an offset, representing the current position the consumer is consuming. After a consumer pulls a certain number of messages and completes processing, it needs to commit the offset.

[0056] Furthermore, the specific details of the submitted offset are as follows:

[0057] The offset is actively submitted by calling the API;

[0058] The API includes the commitSync() or commitAsync() methods.

[0059] As described above, offset commits can be made in two ways: manual commit and automatic commit. Automatic commit is the simplest method, which can be enabled by setting enable.auto.commit=true in the consumer configuration and the commit frequency can be controlled by auto.commit.interval.ms. However, this application uses the manual commit method, in which the consumer calls the API to commit the offset after processing the message, which has higher flexibility and accuracy.

[0060] Furthermore, step S1 specifically includes:

[0061] The Kafka message middleware is based on a preset custom partitioner. According to different preset delayed consumption durations, it divides the delayed consumption topics in the message middleware into shards and creates a corresponding consumer for each shard.

[0062] The custom partitioner inherits from the org.apache.kafka.clients.producer.Partitioner interface;

[0063] The partition method in the org.apache.kafka.clients.producer.Partitioner interface is pre-rewritten to calculate the shard to which a message should belong based on the message's delayed consumption duration.

[0064] As described above, Kafka can divide shards based on the latency of consumption and calculate the shard to which program messages belong.

[0065] The present invention provides a method and apparatus for message delay processing based on Kafka, which is applicable to system scenarios using Kafka message middleware where message delay processing is required.

[0066] Please refer to Figure 1 and Figure 3 Embodiment 1 of the present invention is as follows:

[0067] A method for implementing message delay processing based on Kafka, including the following steps:

[0068] S1. The Kafka message middleware divides the message middleware into shards according to different preset delayed consumption durations, and creates a corresponding consumer for each shard.

[0069] Step S1 is as follows:

[0070] The Kafka message middleware is based on a preset custom partitioner. According to different preset delayed consumption durations, it divides the delayed consumption topics in the message middleware into shards and creates a corresponding consumer for each shard.

[0071] The custom partitioner inherits from the org.apache.kafka.clients.producer.Partitioner interface;

[0072] The partition method in the org.apache.kafka.clients.producer.Partitioner interface is pre-rewritten to calculate the shard to which a message should belong based on the message's delayed consumption duration.

[0073] S2. Receive program messages, and according to the duration selected in the program messages, send the program messages to the corresponding shards of the delayed consumption topic, so that the consumers corresponding to the shards can consume them sequentially.

[0074] The program message includes a consumption time, which is determined based on the message sending time and the delay duration.

[0075] In this embodiment, when the program sends a message, it sets the duration of delayed consumption, such as 1 second, 5 seconds, 10 seconds, 30 seconds, 1 minute, or 5 minutes. The delay duration is selected based on the system's business scenario. In this embodiment, different delay duration options are preset to meet different business needs. The consumption time is set in the message body, i.e., the message's expected processing timestamp = current time + specified delayed consumption duration.

[0076] The sequential consumption includes the following steps:

[0077] S21. The consumer pulls the program messages in batches from the shards and starts multiple consumer threads to process the messages concurrently.

[0078] S22. The consumption thread determines whether the current time has reached the consumption time in the program message. If so, it consumes the message; otherwise, it sleeps according to the consumption time and consumes the message after the sleep ends.

[0079] S23. After the consumer finishes consuming the messages in the current batch, it submits the offset and updates the position of the consumer in the segment.

[0080] The specific offset to be submitted is as follows:

[0081] The offset is actively submitted by calling the API;

[0082] The API includes the commitSync() or commitAsync() methods.

[0083] In this embodiment, messages are arranged in order within each shard, with messages whose estimated processing time is earlier being consumed first. The consumer pulls a batch of messages and starts multi-threaded concurrent message processing. If the current time reaches the predetermined consumption time, the message is consumed. Otherwise, it sleeps for the predetermined consumption time, and then consumes messages after the sleep period ends. After message consumption is complete, the offset is committed.

[0084] In Kafka, a consumer's position is maintained by an "offset," representing its current consumption position. After a consumer pulls a certain number of messages and processes them, it needs to commit the offset, choosing between automatic or manual commit. Automatic commit: This is the simplest method, enabled by setting `enable.auto.commit=true` in the consumer configuration, with the commit frequency controlled by `auto.commit.interval.ms`. Manual commit: The consumer needs to explicitly call the API to commit the offset after processing messages. If processing records takes some time, manual commit provides better control because it ensures that the offset is only committed after the record has been fully processed. This avoids reprocessing records in case of failure.

[0085] In this embodiment, the offset is submitted manually. In other equivalent embodiments, the offset can also be submitted automatically.

[0086] The following example illustrates the use of a method for implementing message delay processing based on Kafka:

[0087] A studio's resource list uses Elasticsearch for data retrieval, caching the first 10 pages of results for 10 minutes. When data changes, the cache needs to be cleared. However, Elasticsearch's data update time has a delay, determined by the `refresh_interval` index refresh cycle. If the cache is cleared immediately after a data update, the data in Elasticsearch hasn't been updated yet. The cached search results will then be outdated, and the latest data won't be available until the cache expires. This manifests as administrators modifying resources finding the resource list unupdated and having to wait 10 minutes to see the changes, impacting operational efficiency.

[0088] Using this invention, based on the Elasticsearch index refresh cycle configuration, a delayed consumption message with a corresponding delay duration is sent when data is updated. Upon receiving the message, the consumer processes it at the predetermined time and deletes it from the cache. This way, when the administrator of a master teacher's studio modifies resources, the data in the resource list can be updated almost in real-time, allowing them to see the latest data promptly.

[0089] Please refer to Figure 2 Embodiment two of the present invention is as follows:

[0090] An apparatus 1 for message delay processing based on Kafka includes a processor 2, a memory 3, and a computer program stored in the memory 3 and executable on the processor 2. When the processor 2 executes the computer program, it implements the steps in the method for message delay processing based on Kafka in Embodiment 1 above.

[0091] In summary, the present invention provides a method and apparatus for message delay processing based on Kafka. In the Kafka message middleware, different shards are divided and corresponding consumers are created according to the predetermined delay consumption duration. This ensures that messages in each partition have the same delay duration, thereby avoiding the problem that consumers may be forced to wait for the message with the longest delay before consuming subsequent messages, which would cause unnecessary delays and improve message consumption efficiency.

[0092] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent modifications made based on the content of the present invention specification and drawings, or direct or indirect applications in related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for implementing message delay processing based on kafka, characterized in that, The method comprises the steps of: S1, the kafka message middleware divides the delay consumption theme of the message middleware into shards according to preset different delay consumption time lengths, and creates a corresponding consumer for each shard; S2, receiving a program message, sending the program message to the corresponding shard of the delay consumption theme according to the selected time length in the program message, and sequentially consuming the program message by the consumer corresponding to the shard; The program message comprises a consumption time, and the consumption time is determined according to the time of sending the message and the delay time length; The sequential consumption comprises the steps of: S21, the consumer batch-pulls the program message from the shard, and starts multiple consumption threads to concurrently process the message; S22, the consumption thread judges whether the current time reaches the consumption time in the program message, and if yes, consumes the message, otherwise sleeps according to the consumption time, and consumes the message after the sleep ends; Further comprising the steps of: S23, the consumer submits an offset after completing the current batch message consumption, and updates the current consumption position of the consumer in the shard; Step S1 is specifically: The kafka message middleware divides the delay consumption theme of the message middleware into shards according to preset different delay consumption time lengths based on a preset custom partitioner, and creates a corresponding consumer for each shard; The custom partitioner inherits from an org.apache.kafka.clients.producer.Partitioner interface; The partition method in the org.apache.kafka.clients.producer.Partitioner interface is pre-overwritten to calculate the shard to which the message belongs according to the delay consumption time length of the message.

2. The method for message delay processing based on kafka implementation according to claim 1, characterized in that, The offset is specifically submitted by: Active submission of offset by calling an API; The API comprises a commitSync() or commitAsync() method.

3. An apparatus for implementing message delay processing based on kafka, comprising a processor, a memory, and a computer program stored in the memory and capable of running on the processor, characterized in that, The processor implements the following steps when executing the computer program: S1, the kafka message middleware divides the delay consumption theme of the message middleware into shards according to preset different delay consumption time lengths, and creates a corresponding consumer for each shard; S2, receiving a program message, sending the program message to the corresponding shard of the delay consumption theme according to the selected time length in the program message, and sequentially consuming the program message by the consumer corresponding to the shard; The program message comprises a consumption time, and the consumption time is determined according to the time of sending the message and the delay time length; The sequential consumption comprises the steps of: S21, the consumer batch-pulls the program message from the shard, and starts multiple consumption threads to concurrently process the message; S22, the consumption thread judges whether the current time reaches the consumption time in the program message, and if yes, consumes the message, otherwise sleeps according to the consumption time, and consumes the message after the sleep ends; Further comprising the steps of: S23, after the current batch of messages is consumed, the consumer submits the offset and updates the current consumption position of the consumer in the shard; The step S1 is specifically: The kafka message middleware is divided into shards in the delay consumption topic of the message middleware according to preset different delay consumption time lengths based on a preset custom partitioner, and a corresponding consumer is created for each shard. The custom partitioner inherits from an org.apache.kafka.clients.producer.Partitioner interface. The partition method in the org.apache.kafka.clients.producer.Partitioner interface is pre-overridden to calculate the shard to which the message belongs according to the delay consumption time length of the message.

4. The device for implementing message delay processing based on kafka according to claim 3, characterized in that, The offset is submitted specifically as follows: The offset is actively submitted by calling an API. The API includes a commitSync() or commitAsync() method.

Images