Method and terminal for message consumption

Abstract

The application discloses a message consumption method and terminal, for a consumption group with a long execution time, the consumption group is split into a first sub-consumption group and a second sub-consumption group, wherein the first sub-consumption group is used for message consumption and stores the identification of the consumed message into a filter, and the second sub-consumption group is used for data warehousing of the consumed message in the filter according to a delay time. Since the original data warehousing operation is delayed when the message is consumed, the message is only subjected to the early message consumption, and the real-time requirement of the message is not high, compared with the prior art, in the message consumption process, the next message is processed after the consumption and warehousing each time, and the overall message consumption efficiency can be effectively improved.

Description

Technical Field

[0001] This invention relates to the technical field of message consumption, and in particular to a method and terminal for message consumption. Background Technology

[0002] In modern internet systems, message queues are typically used to decouple business logic between different systems or modules, reducing coupling between them. Kafka, as the most common message queue, is not only used for decoupling between systems or modules, but also for data writing in big data scenarios due to its high throughput.

[0003] In Kafka's architecture, there are multiple partitions and multiple consumer groups. However, its internal architecture and mechanisms dictate that a consumer within a group can consume messages from multiple partitions, and a partition can only be consumed by consumers within the same group. While this architecture can meet the needs of consuming large amounts of data, for messages with complex business logic, the message consumption and data storage time is relatively long, so the efficiency of message consumption still needs further improvement. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a message consumption method and terminal that can effectively improve the overall message consumption efficiency for messages with low real-time requirements.

[0005] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0006] A method for consuming messages, comprising the following steps:

[0007] The execution time of the consumer group in the monitoring cluster is increased. If the execution time exceeds the time threshold, the consumer group is split into a first sub-consumer group and a second sub-consumer group.

[0008] Use the first sub-consumer group to consume messages and store the identifiers of consumed messages in the filter;

[0009] The second sub-consumer group is used to store the consumed messages in the filter according to the delay time.

[0010] To solve the above-mentioned technical problems, another technical solution adopted by the present invention is as follows:

[0011] A message consumption terminal includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the message consumption method described above.

[0012] The beneficial effects of this invention are as follows: For consumer groups with long execution times, the consumer group is split into a first sub-consumer group and a second sub-consumer group. The first sub-consumer group is used for message consumption and stores the identifiers of consumed messages in a filter. The second sub-consumer group is used to store the consumed messages in the filter into the database according to the delay time. Since only the initial message consumption is required during message consumption, and the original data storage operation is delayed, this invention effectively improves the overall efficiency of message consumption, especially for messages with low real-time requirements, compared to existing technologies where each message needs to be consumed and stored before processing the next message. Attached Figure Description

[0013] Figure 1 This is a flowchart of a message consumption method according to an embodiment of the present invention;

[0014] Figure 2 This is a schematic diagram of a message consumption terminal according to an embodiment of the present invention;

[0015] Label Explanation:

[0016] 1. A message consumption terminal; 2. Memory; 3. Processor. Detailed Implementation

[0017] To explain in detail the technical content, objectives, and effects of the present invention, the following description is provided in conjunction with the embodiments and accompanying drawings.

[0018] Please refer to Figure 1 This invention provides a message consumption method, including the following steps:

[0019] The execution time of the consumer group in the monitoring cluster is increased. If the execution time exceeds the time threshold, the consumer group is split into a first sub-consumer group and a second sub-consumer group.

[0020] Use the first sub-consumer group to consume messages and store the identifiers of consumed messages in the filter;

[0021] The second sub-consumer group is used to store the consumed messages in the filter according to the delay time.

[0022] As described above, the beneficial effects of this invention are as follows: For consumer groups with long execution times, the consumer group is split into a first sub-consumer group and a second sub-consumer group. The first sub-consumer group is used for message consumption and stores the identifiers of consumed messages in a filter. The second sub-consumer group is used to store the consumed messages in the filter into the database according to the delay time. Since only the initial message consumption is required during message consumption, and the original data storage operation is delayed, this effectively improves the overall efficiency of message consumption, especially for messages with low real-time requirements, compared to the existing technology where each message needs to be consumed and stored before processing the next message.

[0023] Furthermore, message consumption is performed using the first sub-consumer group, and the identifiers of consumed messages are stored in a filter, including:

[0024] The first sub-consumer group performs business validation logic on messages. If the executed message meets the preset business rules, the identifier of the executed message is stored in the filter.

[0025] As described above, using filters to store identifiers of messages that have been completed can indicate whether a message exists in the filter, making it easier for the second sub-consumer group to store these messages in the database.

[0026] Further, the second sub-consumer group is used to store the consumed messages in the filter into the database according to the delay time, including:

[0027] When the second sub-consumer group receives a message, it determines whether the message identifier exists in the filter. If so, it stores the corresponding information of all messages that have reached the delay time into the database.

[0028] As described above, when a second sub-consumer group receives a message identifier in the filter, all messages that have reached the delay time are entered into the database. Therefore, by adopting the delayed entry method, the entry operation can be performed in batches, which is more efficient than the original single entry.

[0029] Further, message consumption is performed using the first sub-consumer group, and the identifiers of consumed messages are stored in a filter, followed by:

[0030] Update the identifier of the latest completed message of the first sub-consumer group in the local memory of the consumer group.

[0031] As described above, once the message identifier is written into the Boolean filter, the latest message identifier data consumed by the consumer group needs to be updated synchronously in the local memory so that the current local memory only contains the latest message ID data consumed by the first child consumer group, so as to facilitate subsequent judgment on whether to continue delaying.

[0032] Further, determining whether the message identifier exists in the filter includes the following steps:

[0033] Check if the message identifier exists in local memory. If it does, continue with a second delay according to the delay time.

[0034] As described above, when the message identifier of the second sub-consumer group is equal to the message identifier of the first sub-consumer group stored in local memory, it means that the second sub-consumer group has caught up with the speed of the first sub-consumer group and needs to continue to wait, that is, to perform a second delay, in order to ensure the reliability of the message consumption process.

[0035] Please refer to Figure 2 Another embodiment of the present invention provides a message consumption terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the various steps of the message consumption method described above.

[0036] The message consumption method and terminal described above are applicable to consuming log-type or tracing messages. They further improve message consumption efficiency through delayed consumption. The specific implementation methods are described below:

[0037] Example 1

[0038] Please refer to Figure 1 A method for consuming messages, comprising the following steps:

[0039] S1. Monitor the execution time of the consumer group in the cluster. If the execution time exceeds the time threshold, split the consumer group into a first sub-consumer group and a second sub-consumer group.

[0040] Specifically, in a typical Kafka cluster, consumer groups consume messages. The main business logic for consuming a message is: perform business validation for a period of time, followed by data storage for the same period of time. Therefore, under the current method, it takes a relatively long time for a single message to be analyzed and stored.

[0041] In this embodiment, the messages to be consumed are log messages or tracing messages, which do not have high real-time requirements and are allowed to be processed after a certain delay.

[0042] Therefore, in this embodiment, when a certain message type has complex verification logic and data writing time, the original consumer group is split into two sub-consumer groups for the consumer group of the partition corresponding to that message type by splitting the consumer group.

[0043] This allows for the differentiation of the main logic in message consumption, with execution time statistics for different logic segments, and the distinction of complex business processes based on the length of execution time.

[0044] S2. Use the first sub-consumer group to consume messages and store the identifiers of consumed messages in the filter.

[0045] Step S2 specifically includes the following steps:

[0046] The first sub-consumer group performs business validation logic on messages. If the executed message meets the preset business rules, the identifier of the executed message is stored in the filter.

[0047] Specifically, the first sub-consumer group mainly performs complex business verification logic. When the complex business verification logic is completed and the data needs to be written in accordance with the business rules, the ID of the message is recorded in the Boolean filter.

[0048] This filter is a commonly used method that supports fast writing and reading, and can support large amounts of data storage with relatively little memory. Its main function is to indicate whether a specific message ID exists in the filter.

[0049] Step S2 is followed by the following steps:

[0050] Update the identifier of the latest completed message of the first sub-consumer group in the local memory of the consumer group.

[0051] Specifically, after the message ID is written into the Boolean filter, the latest message ID data consumed by the consumer group needs to be updated synchronously in local memory, so that the current local memory only contains the latest message ID data consumed by the first child consumer group. In theory, at this time, the message consumption rate of the first child consumer group will be twice as fast as the original method.

[0052] S3. Use the second sub-consumer group to store the consumed messages in the filter according to the delay time.

[0053] Step S3 specifically includes the following steps:

[0054] When the second sub-consumer group receives a message, it determines whether the message identifier exists in the filter. If so, it stores the corresponding information of all messages that have reached the delay time into the database.

[0055] Specifically, the second sub-consumer group uses a half-hour delay for consumption, with the Kafka cluster supporting configuration of the delay time. When the second sub-consumer group consumes the same message, it first queries the boolean filter for the message ID. If the message ID does not exist, the message does not need to be processed; if the message ID exists, the message is inserted into the database.

[0056] Therefore, by delaying message entry into the database, messages of the same type can be logically grouped and distinguished. Since data entry is the most time-consuming operation, the delayed entry method can be used to perform entry operations in batches, which is more efficient than the original single entry method.

[0057] When the second sub-consumer group receives a message, it needs to first determine whether the message identifier exists in local memory. If it does, it will continue to delay the message a second time according to the delay time.

[0058] In this embodiment, it is necessary to retrieve the message IDs already consumed by the first sub-consumer group from local memory. When the message ID of the second sub-consumer group equals that of the first sub-consumer group, it means that the second sub-consumer group has caught up with the speed of the first sub-consumer group and needs to continue waiting. Because the first sub-consumer group has not finished processing its responsible business and there is no new data to be entered into the database, the second sub-consumer group needs to wait.

[0059] For example, if there are currently 100 messages, and the first sub-consumer group has consumed 50 messages, writing them to memory with id=50, and waits for half an hour, the second sub-consumer group starts consuming messages. At this point, the second sub-consumer group's consumption rate is faster than the first, so it will gradually catch up. Suppose that when id=75, the second sub-consumer group catches up with the first. At this point, the second sub-consumer group will have to wait for the first sub-consumer group to continue consuming messages before it can resume its own consumption.

[0060] Example 2

[0061] Please refer to Figure 2 A message consumption terminal 1 includes a memory 2, a processor 3, and a computer program stored on the memory 2 and executable on the processor 3. When the processor 3 executes the computer program, it implements the various steps of a message consumption method according to Embodiment 1.

[0062] In summary, the message consumption method and terminal provided by this invention, for consumption groups with long execution times, splits the consumption group into a first sub-consumption group and a second sub-consumption group. The first sub-consumption group is used for message consumption and stores the identifiers of consumed messages in a filter. The second sub-consumption group is used to store the consumed messages in the filter into the database according to the delay time. Since only the initial message consumption is required during message consumption, and the original data storage operation is delayed, this method effectively improves the overall efficiency of message consumption, especially for messages with low real-time requirements, compared to existing technologies where each message needs to be consumed and stored before processing the next message.

[0063] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent modifications made based on the content of the present invention specification and drawings, or direct or indirect applications in related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method of message consumption, characterized by, The consumed message is a log type or a link tracking message, and the steps include: Monitoring execution time of a consumer group in a cluster, and splitting the consumer group into a first sub-consumer group and a second sub-consumer group if the execution time exceeds a time threshold; Consuming messages using the first sub-consumer group and storing the identities of the consumed messages into a filter; Using the second sub-consumer group to perform data warehousing on the consumed messages in the filter according to a delay time.

2. The method of claim 1, wherein, The step of consuming messages using the first sub-consumer group and storing the identities of the consumed messages into a filter includes: Using the first sub-consumer group to perform business verification logic on the messages, and storing the identities of the messages that have been executed into the filter if the messages meet preset business rules.

3. The method of claim 1, wherein, The step of using the second sub-consumer group to perform data warehousing on the consumed messages in the filter according to a delay time includes: When the second sub-consumer group receives a message, determining whether the identity of the message exists in the filter, and performing data warehousing on information corresponding to all messages that meet the delay time if the identity exists.

4. The method of claim 3, wherein, The step of consuming messages using the first sub-consumer group and storing the identities of the consumed messages into a filter further includes: Updating the identities of the latest messages that have been executed in the first sub-consumer group in local memory of the consumer group.

5. The method of claim 4, wherein, The step of determining whether the identity of the message exists in the filter includes: Determining whether the identity of the message exists in the local memory, and continuing to perform secondary delay according to the delay time if the identity exists. 6.A terminal for message consumption, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein, The consumed message is a log type or a link tracking message, and the processor implements the following steps when executing the computer program: Monitoring execution time of a consumer group in a cluster, and splitting the consumer group into a first sub-consumer group and a second sub-consumer group if the execution time exceeds a time threshold; Consuming messages using the first sub-consumer group and storing the identities of the consumed messages into a filter; Using the second sub-consumer group to perform data warehousing on the consumed messages in the filter according to a delay time.

7. The message consuming terminal according to claim 6, wherein The step of consuming messages using the first sub-consumer group and storing the identities of the consumed messages into a filter includes: Using the first sub-consumer group to perform business verification logic on the messages, and storing the identities of the messages that have been executed into the filter if the messages meet preset business rules.

8. The message consuming terminal according to claim 6, wherein The step of using the second sub-consumer group to perform data warehousing on the consumed messages in the filter according to a delay time includes: When the second sub-consumer group receives a message, determining whether the identity of the message exists in the filter, and performing data warehousing on information corresponding to all messages that meet the delay time if the identity exists.

9. The message consuming terminal according to claim 8, wherein The step of consuming messages using the first sub-consumer group and storing the identities of the consumed messages into a filter further includes: Updating the identities of the latest messages that have been executed in the first sub-consumer group in local memory of the consumer group.

10. The message consuming terminal according to claim 9, wherein The step of determining whether the identity of the message exists in the filter includes: Determining whether the identity of the message exists in the local memory, and continuing to perform secondary delay according to the delay time if the identity exists.

Images