A message consumption optimization method and terminal

Abstract

The present invention discloses a message consumption optimization method and terminal. A corresponding number of consumption main threads are established according to the number of storage areas of a Kafka system, and each consumption main thread establishes a preset number of sub-threads. The consumption main thread obtains a preset number of messages from each partition of the Kafka system, and groups the messages according to the first business primary key of the message and a preset business logic condition. The consumption main thread distributes the messages to its own sub-threads for processing according to the grouping result. A consumption main thread is set for each storage on Kafka, which skips the design mode of Kafka, and a main thread consumes the messages in a partition and groups them, and a plurality of sub-threads process the messages according to the grouping result, thereby improving the consumption throughput of Kafka, and being able to control and adjust the message consumption rate by adjusting the number of threads, which is more flexible and changeable.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a message consumption optimization method and terminal. Background Art

[0002] The purpose of most MQ projects is to decouple different modules. Therefore, they can accumulate large amounts of data in the form of memory or files, and process accumulated messages asynchronously. Among them, Kafka is a common high-throughput distributed publish-subscribe messaging system that can handle all action stream data in consumer-scale websites. The number of consumer threads in the Kafka architecture is related to the partition (storage area) of the topic (Topic is the transmission medium between the message publisher (Pub) and the subscriber (Sub)). (One type of message has a topic, and when each topic is created, the number of partitions of the topic needs to be fixed.) At a certain moment, only one consumer thread can connect to a partition. At this point, there may be a problem: when the amount of messages that the project needs to process reaches a certain bottleneck, the message consumption rate will be limited.

[0003] In the face of the above problems, the old system generally increases the number of consumers by adding new partitions to the topic. However, this method requires deleting the topic and re-establishing the number of partitions related to the topic every time a new number of consumers is needed, or setting a larger number of partitions when a new topic is established for the first time. This method is not flexible, and the number of partitions cannot be expanded indefinitely. In addition, it cannot be flexibly expanded for scenarios that require sequential consumption of messages. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a message consumption optimization method and terminal, which can flexibly control the message consumption rate.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0006] A method for optimizing message consumption, comprising the steps of:

[0007] S1. Establish a corresponding number of consumer main threads according to the number of storage areas of the Kafka system, and each consumer main thread establishes a preset number of sub-threads;

[0008] S2. The consumer main thread obtains a preset number of messages from each partition of the Kafka system, and groups the messages according to the first business primary key of the message and the preset business logic condition;

[0009] S3. The consuming main thread distributes the message to its own child thread for processing according to the grouping result.

[0010] In order to solve the above technical problems, another technical solution adopted by the present invention is:

[0011] A message consumption optimization terminal includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0012] S1. Establish a corresponding number of consumer main threads according to the number of storage areas of the Kafka system, and each consumer main thread establishes a preset number of sub-threads;

[0013] S2. The consumer main thread obtains a preset number of messages from each partition of the Kafka system, and groups the messages according to the first business primary key of the message and the preset business logic condition;

[0014] S3. The consuming main thread distributes the message to its own child thread for processing according to the grouping result.

[0015] The beneficial effects of the present invention are as follows: the technical solution of the present invention sets a main consumption thread for each storage on Kafka, which bypasses the design mode of Kafka. A main thread consumes messages in a partition. At the same time, messages are grouped according to the first business primary key and preset business logic conditions, and multiple sub-threads process the messages according to the grouping results, which can greatly improve the message consumption rate and the consumption throughput of Kafka. Moreover, for the control of the consumption rate, there is no need to modify the partition, and the message consumption rate can be controlled by adjusting the number of threads, which is more flexible and changeable. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A flowchart of a method for optimizing message consumption according to an embodiment of the present invention;

[0017] Figure 2 A structural diagram of an optimized terminal for message consumption according to an embodiment of the present invention;

[0018] Description of labels:

[0019] 1. An optimized terminal for message consumption; 2. Processor; 3. Memory. DETAILED DESCRIPTION

[0020] In order to explain the technical content, achieved objectives and effects of the present invention in detail, the following is an explanation in combination with the implementation modes and the accompanying drawings.

[0021] Please refer to Figure 1 , a message consumption optimization method, comprising the steps of:

[0022] S1. Establish a corresponding number of consumer main threads according to the number of storage areas of the Kafka system, and each consumer main thread establishes a preset number of sub-threads;

[0023] S2. The consumer main thread obtains a preset number of messages from each partition of the Kafka system, and groups the messages according to the first business primary key of the message and the preset business logic condition;

[0024] S3. The consuming main thread distributes the message to its own child thread for processing according to the grouping result.

[0025] From the above description, it can be seen that the beneficial effects of the present invention are: the technical solution of the present invention sets a main consumption thread for each storage on Kafka, which bypasses the design method of Kafka. A main thread consumes messages in a partition. At the same time, the messages are grouped by the first business primary key and the preset business logic conditions, and multiple sub-threads process the messages according to the grouping results, which can greatly improve the message consumption rate and the consumption throughput of Kafka. Moreover, for the control of the consumption rate, there is no need to modify the partition, and the message consumption rate can be controlled by adjusting the number of threads, which is more flexible and changeable.

[0026] Furthermore, the grouping of the messages according to the first business primary key of the messages and the preset business logic condition in step S2 includes:

[0027] Identify the first business primary key in all the messages, group two or more messages with the same first business primary key into a duplicate set, and record the remaining messages into a non-duplicate set;

[0028] The step S3 is specifically as follows:

[0029] The consuming main thread randomly distributes all messages recorded in the non-repetitive set to its own child threads for concurrent processing, and randomly distributes the messages recorded in the repeated set to its own child threads in groups, and the child threads sequentially process the messages according to the time information in the messages.

[0030] From the above description, it can be seen that when there are multiple messages with a certain business primary key and the multiple messages have a time sequence, they are grouped together and processed by the same sub-thread, so that they can be processed sequentially according to the time information, avoiding the problem of disorganized processing order caused by assigning them to multiple threads.

[0031] Further, the identifying of the first business primary key in all the messages, grouping two or more messages with the same first business primary key into a group and recording them in a duplicate set, and recording the remaining messages in a non-duplicate set is specifically:

[0032] Identify the first business primary key in all the messages, calculate the hash value of the first business primary key, determine whether the messages have the same first business primary key based on the hash value, group two or more messages with the same first business primary key together and record them in a duplicate set, and record the remaining messages in a non-duplicate set.

[0033] It can be seen from the above description that by calculating the hash value of the first business message primary key and comparing the hash value to determine whether the first business primary key is the same, the comparison is more accurate.

[0034] Furthermore, the subthread sequentially processes the messages according to the time information in the messages as follows:

[0035] The child thread determines whether the number of the messages belonging to the same group is greater than a preset threshold. If so, the messages are grouped again according to the second business primary key to obtain secondary grouping information, and a grandchild thread corresponding to the number is created according to the secondary grouping information to process the messages.

[0036] From the above description, it can be seen that when the amount of message data in a group is large (greater than the preset threshold), we continue to group the messages again and create grandchild threads corresponding to the number of groups for processing. By expanding the number of threads for message processing, the efficiency of message processing is improved.

[0037] Furthermore, the step S3 further includes the following steps:

[0038] S31, determine whether all the messages have been processed, if so, return to step S2;

[0039] S4. Receive a user request, and adjust the number of the sub-threads of each consuming main thread according to the user request.

[0040] From the above description, we can see that after the current batch of messages is processed, another batch of messages will be pulled from Kafka for processing. At the same time, users can control the message processing efficiency by adjusting the number of child threads.

[0041] Please refer to Figure 2 , an optimized terminal for message consumption, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0042] S1. Establish a corresponding number of consumer main threads according to the number of storage areas of the Kafka system, and each consumer main thread establishes a preset number of sub-threads;

[0043] S2. The consumer main thread obtains a preset number of messages from each partition of the Kafka system, and groups the messages according to the first business primary key of the message and the preset business logic condition;

[0044] S3. The consuming main thread distributes the message to its own child thread for processing according to the grouping result.

[0045] From the above description, it can be seen that the beneficial effects of the present invention are: the technical solution of the present invention sets a main consumption thread for each storage on Kafka, which bypasses the design method of Kafka. A main thread consumes messages in a partition. At the same time, the messages are grouped by the first business primary key and the preset business logic conditions, and multiple sub-threads process the messages according to the grouping results, which can greatly improve the message consumption rate and the consumption throughput of Kafka. Moreover, for the control of the consumption rate, there is no need to modify the partition, and the message consumption rate can be controlled by adjusting the number of threads, which is more flexible and changeable.

[0046] Furthermore, the grouping of the messages according to the first business primary key of the messages and the preset business logic condition in step S2 includes:

[0047] Identify the first business primary key in all the messages, group two or more messages with the same first business primary key into a duplicate set, and record the remaining messages into a non-duplicate set;

[0048] The step S3 is specifically as follows:

[0049] The consuming main thread randomly distributes all messages recorded in the non-repetitive set to its own child threads for concurrent processing, and randomly distributes the messages recorded in the repeated set to its own child threads in groups, and the child threads sequentially process the messages according to the time information in the messages.

[0050] From the above description, it can be seen that when there are multiple messages with a certain business primary key and the multiple messages have a time sequence, they are grouped together and processed by the same sub-thread, so that they can be processed sequentially according to the time information, avoiding the problem of disorganized processing order caused by assigning them to multiple threads.

[0051] Further, the identifying of the first business primary key in all the messages, grouping two or more messages with the same first business primary key into a group and recording them in a duplicate set, and recording the remaining messages in a non-duplicate set is specifically:

[0052] Identify the first business primary key in all the messages, calculate the hash value of the first business primary key, determine whether the messages have the same first business primary key based on the hash value, group two or more messages with the same first business primary key together and record them in a duplicate set, and record the remaining messages in a non-duplicate set.

[0053] It can be seen from the above description that by calculating the hash value of the first business message primary key and comparing the hash value to determine whether the first business primary key is the same, the comparison is more accurate.

[0054] Furthermore, the subthread sequentially processes the messages according to the time information in the messages as follows:

[0055] The child thread determines whether the number of the messages belonging to the same group is greater than a preset threshold. If so, the messages are grouped again according to the second business primary key to obtain secondary grouping information, and a grandchild thread corresponding to the number is created according to the secondary grouping information to process the messages.

[0056] From the above description, it can be seen that when the amount of message data in a group is large (greater than the preset threshold), we continue to group the messages again and create grandchild threads corresponding to the number of groups for processing. By expanding the number of threads for message processing, the efficiency of message processing is improved.

[0057] Furthermore, the step S3 further includes the following steps:

[0058] S31, determine whether all the messages have been processed, if so, return to step S2;

[0059] S4. Receive a user request, and adjust the number of the sub-threads of each consuming main thread according to the user request.

[0060] From the above description, we can see that after the current batch of messages is processed, another batch of messages will be pulled from Kafka for processing. At the same time, users can control the message processing efficiency by adjusting the number of child threads.

[0061] The present invention discloses a message consumption optimization method and terminal, which are applied to an MQ (message queue) project based on a Kafka architecture for processing messages.

[0062] Please refer to Figure 1 , Embodiment 1 of the present invention is:

[0063] A method for optimizing message consumption, comprising the steps of:

[0064] S1. Establish a corresponding number of consumer main threads according to the number of storage areas of the Kafka system, and each consumer main thread establishes a preset number of sub-threads;

[0065] In this embodiment, the relevant Kafka cluster environment is pre-built, the relevant tool jar package is downloaded, and the relevant Kafka consumer client code is written. The consumer client program creates the relevant consumption thread.

[0066] S2. The consumer main thread obtains a preset number of messages from each partition of the Kafka system, and groups the messages according to the first business primary key of the message and the preset business logic condition;

[0067] The step S2 of grouping the messages according to the first business primary key of the messages and the preset business logic condition includes:

[0068] Identify the first business primary key in all the messages, group two or more messages with the same first business primary key into a duplicate set, and record the remaining messages into a non-duplicate set;

[0069] The step of identifying the first business primary key in all the messages, grouping two or more messages with the same first business primary key into a duplicate set, and recording the remaining messages into a non-duplicate set is specifically as follows:

[0070] Identify the first business primary key in all the messages, calculate the hash value of the first business primary key, determine whether the messages have the same first business primary key based on the hash value, group two or more messages with the same first business primary key together and record them in a duplicate set, and record the remaining messages in a non-duplicate set.

[0071] In this embodiment, the program pulls a batch of messages from Kafka through method calls, and the size and number of messages pulled can be preset. In this embodiment, there are 5 partitions in Kafka, and there are 5 consumer main threads to pull messages from these 5 partitions.

[0072] For the batch of messages obtained, we group them according to certain business logic conditions. Specifically, we perform hash processing based on the key key in the message, that is, the first business primary key, and compare the key key of the message based on the hash value, and group them according to the comparison result. For example, there are 100 messages, each of which has at least one business primary key. The business primary key may be repeated, indicating multiple messages of a certain record. For example, if there is a userId field in the message, messages with different userIds represent different account messages. If the userId is the same, it means multiple messages of the same account. At this time, this type of message needs to be consumed sequentially. We deduplicate based on userId. When there is only one record for each userId, it is recorded in a non-duplicate set. When there are at least 2 messages for a userId, they are grouped together and recorded in a duplicate set.

[0073] S3, the consuming main thread distributes the message to its own sub-thread for processing according to the grouping result;

[0074] The step S3 is specifically as follows:

[0075] The consumer main thread randomly distributes all messages recorded in the non-repeating set to its own child threads for concurrent processing, and randomly distributes the messages recorded in the repeating set to its own child threads in groups, and the child threads sequentially process the messages according to the time information in the messages;

[0076] The subthread sequentially processes the message according to the time information in the message as follows:

[0077] The child thread determines whether the number of the messages belonging to the same group is greater than a preset threshold. If so, the messages are grouped again according to the second business primary key to obtain secondary grouping information, and a grandchild thread corresponding to the number is created according to the secondary grouping information to process the messages.

[0078] In this embodiment, the data in the non-repeating set can be directly and evenly distributed to multiple subtask threads for concurrent consumption, because there will be no order problem, while the data in the repeating set is distributed according to each group. At the same time, when the message volume of a certain business primary key continues to increase, it is necessary to support the subtask processing thread to continue to perform hash comparison according to the second business primary key and then distribute it again, and distribute the message to the grandchild task processing thread for subsequent processing. For example, when the amount of synchronized data of a certain userId account reaches the preset threshold, we can group it according to another key key of the message, that is, the second business primary key, for example, according to the target location of the message, group it again, and distribute it to the grandchild thread according to the group, and the grandchild thread will process the message sequentially according to the group. In addition, for some specific first business primary keys or second business primary keys, special threads can be pre-configured to be responsible for processing or the number of sub-threads can be increased for processing.

[0079] The step S3 further includes the following steps:

[0080] S31. Determine whether all the messages have been processed. If so, return to step S2.

[0081] S4. Receive a user request, and adjust the number of the sub-threads of each consuming main thread according to the user request.

[0082] In this embodiment, when a large number of messages need to be consumed, the number of child threads can be artificially increased to increase the consumption throughput. In this way, the consumption rate of messages can be flexibly controlled to increase or decrease the throughput of message processing.

[0083] Please refer to Figure 2 , Embodiment 2 of the present invention is:

[0084] A message consumption optimization terminal includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps in the above embodiment 1 are implemented.

[0085] The main principle of the present invention is that a consumer thread processes the message of a partition, obtains a batch of messages from it and distributes them to different sub-threads for processing according to the business primary key, that is, it jumps out of the design method of Kafka and processes messages in a multi-threaded manner, which greatly improves the consumption rate of sequential messages and improves the consumption throughput of Kafka.

[0086] In summary, the present invention provides a message consumption optimization method and terminal, which sets a consumption main thread for each storage on Kafka, bypassing the design mode of Kafka. A main thread consumes messages in a partition. At the same time, the messages are grouped according to the first business primary key and the preset business logic conditions, and multiple sub-threads process the messages according to the grouping results, which can greatly improve the message consumption rate and the consumption throughput of Kafka. For the control of the consumption rate, there is no need to modify the partition, and the message consumption rate can be controlled by adjusting the number of threads, which is more flexible. At the same time, the messages are grouped according to the business primary key, and the messages in the same group are processed by the same thread model to ensure that the order of message processing is not disordered.

[0087] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent transformations made using the contents of the present invention's specification and drawings, or directly or indirectly applied in related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for optimizing message consumption, characterized in that: Includes steps: S1. Establish a corresponding number of consumer main threads according to the number of storage areas of the Kafka system, and each consumer main thread establishes a preset number of sub-threads; S2. The consumer main thread obtains a preset number of messages from each partition of the Kafka system, and groups the messages according to the first business primary key of the message and the preset business logic condition; The step S2 of grouping the messages according to the first business primary key of the messages and the preset business logic condition includes: Identify the first business primary key in all the messages, group two or more messages with the same first business primary key into a duplicate set, and record the remaining messages into a non-duplicate set; S3, the consuming main thread distributes the message to its own sub-thread for processing according to the grouping result; The step S3 is specifically as follows: The consuming main thread randomly distributes all the messages recorded in the non-repetitive set to its own child threads for concurrent processing, and at the same time randomly distributes the messages recorded in the repeated set to its own child threads in groups, and the child threads sequentially process the messages according to the time information in the grouped messages.

2. The method for optimizing message consumption according to claim 1, characterized in that: The step of identifying the first business primary key in all the messages, grouping two or more messages with the same first business primary key into a duplicate set, and recording the remaining messages into a non-duplicate set is specifically as follows: Identify the first business primary key in all the messages, calculate the hash value of the first business primary key, determine whether the messages have the same first business primary key based on the hash value, group two or more messages with the same first business primary key together and record them in a duplicate set, and record the remaining messages in a non-duplicate set.

3. The method for optimizing message consumption according to claim 1, characterized in that: The subthread sequentially processes the message according to the time information in the message as follows: The child thread determines whether the number of the messages belonging to the same group is greater than a preset threshold. If so, the messages are grouped again according to the second business primary key to obtain secondary grouping information, and a grandchild thread corresponding to the number is created according to the secondary grouping information to process the messages.

4. A message consumption optimization method according to claim 1, characterized in that: The step S3 further includes the following steps: S31, determine whether all the messages have been processed, if so, return to step S2; S4. Receive a user request, and adjust the number of the sub-threads of each consuming main thread according to the user request.

5. A message consumption optimization terminal, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the following steps are implemented: S1. Establish a corresponding number of consumer main threads according to the number of storage areas of the Kafka system, and each consumer main thread establishes a preset number of sub-threads; S2. The consumer main thread obtains a preset number of messages from each partition of the Kafka system, and groups the messages according to the first business primary key of the message and the preset business logic condition; The step S2 of grouping the messages according to the first business primary key of the messages and the preset business logic condition includes: Identify the first business primary key in all the messages, group two or more messages with the same first business primary key into a duplicate set, and record the remaining messages into a non-duplicate set; S3, the consuming main thread distributes the message to its own sub-thread for processing according to the grouping result; The step S3 is specifically as follows: The consuming main thread randomly distributes all the messages recorded in the non-repetitive set to its own child threads for concurrent processing, and at the same time randomly distributes the messages recorded in the repeated set to its own child threads in groups, and the child threads sequentially process the messages according to the time information in the grouped messages.

6. The message consumption optimization terminal according to claim 5, characterized in that: The step of identifying the first business primary key in all the messages, grouping two or more messages with the same first business primary key into a duplicate set, and recording the remaining messages into a non-duplicate set is specifically as follows: Identify the first business primary key in all the messages, calculate the hash value of the first business primary key, determine whether the messages have the same first business primary key based on the hash value, group two or more messages with the same first business primary key together and record them in a duplicate set, and record the remaining messages in a non-duplicate set.

7. The message consumption optimization terminal according to claim 5, characterized in that: The subthread sequentially processes the message according to the time information in the message as follows: The child thread determines whether the number of the messages belonging to the same group is greater than a preset threshold. If so, the messages are grouped again according to the second business primary key to obtain secondary grouping information, and a grandchild thread corresponding to the number is created according to the secondary grouping information to process the messages.

8. The message consumption optimization terminal according to claim 5, characterized in that: The step S3 further includes the following steps: S31, determine whether all the messages have been processed, if so, return to step S2; S4. Receive a user request, and adjust the number of the sub-threads of each consuming main thread according to the user request.

Images