Method and system for message aggregation consumption

Abstract

The application discloses a message aggregation consumption method and system, when a client receives a message, the type and data size of each message are recorded in advance, and the message queue is written according to the type and data size of the message; when the server consumes the message, the message list is obtained from the message queue, the messages are aggregated according to the type, and then the aggregated messages are sent to the thread pool; one aggregated message containing the same type of message only needs one thread in the thread pool to process. In this way, after corresponding aggregation according to the message size, the IO consumption of subsequent message consumption can be reduced, meanwhile, under the message aggregation mode, the thread usage in the thread pool can be effectively reduced, and the message consumption rate can be further improved.

Description

Technical Field

[0001] This invention relates to the field of message processing technology, and in particular to a method and system for message aggregation and consumption. Background Technology

[0002] Most systems currently have high concurrency and high throughput application scenarios. In order to decouple the various modules, the most common way is to use message middleware or message queues to decouple the system or modules.

[0003] The typical approach is as follows: after the client sends a message, it reaches the server, which then processes it directly. To support high concurrency, the server further stores received messages using a message queue. Then, it consumes messages from this queue and sends them to a thread pool, where threads handle the message tasks. This approach can improve message consumption speed to some extent. However, messages vary in size, and each message corresponds to a thread in the thread pool. When there are many small messages, this can significantly increase the number of tasks in the thread pool, leading to frequent thread switching and ultimately hindering the message consumption rate. Summary of the Invention

[0004] The technical problem to be solved by this invention is to provide a method and system for message aggregation and consumption that can improve message consumption rate.

[0005] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0006] A method for message aggregation and consumption includes the following steps:

[0007] When the client receives a message, it records the message type and data volume, uses the message type and data volume as keys, and stores the message as a value in the message queue.

[0008] When the server consumes messages, it retrieves a list of messages from the message queue, aggregates messages of the same type with the same key, sends the aggregated messages to the thread pool, and uses threads in the thread pool to split and consume the aggregated messages.

[0009] To solve the above-mentioned technical problems, another technical solution adopted by the present invention is as follows:

[0010] A message aggregation and consumption system includes:

[0011] The client is used to receive messages, record the type and amount of data of the messages, use the type and amount of data of the messages as keys, and store the messages as values ​​in a message queue.

[0012] On the server side, when consuming messages, a list of messages is retrieved from the message queue, messages of the same type with the same key are aggregated, and the aggregated messages are sent to a thread pool. Threads in the thread pool then split and consume the aggregated messages.

[0013] The beneficial effects of this invention are as follows: When the client receives a message, it pre-records the type and data size of each message and writes them into a message queue according to the message type and data size. When the server consumes messages, it retrieves a message list from the message queue, aggregates the messages by type, and then sends these aggregated messages to a thread pool. An aggregated message containing messages of the same type only needs to be processed by one thread in the thread pool. In this way, by aggregating messages according to their size, the IO consumption of subsequent message consumption can be reduced. At the same time, the message aggregation method can effectively reduce the number of threads used in the thread pool, further improving the message consumption rate. Attached Figure Description

[0014] Figure 1 This is a flowchart illustrating a message aggregation and consumption method according to an embodiment of the present invention;

[0015] Figure 2 This is a schematic diagram of a message aggregation and consumption system according to an embodiment of the present invention. Detailed Implementation

[0016] To explain in detail the technical content, objectives, and effects of the present invention, the following description is provided in conjunction with the embodiments and accompanying drawings.

[0017] Please refer to Figure 1 This invention provides a method for message aggregation and consumption, comprising the following steps:

[0018] When the client receives a message, it records the message type and data volume, uses the message type and data volume as keys, and stores the message as a value in the message queue.

[0019] When the server consumes messages, it retrieves a list of messages from the message queue, aggregates messages of the same type with the same key, sends the aggregated messages to the thread pool, and uses threads in the thread pool to split and consume the aggregated messages.

[0020] As described above, the beneficial effects of this invention are as follows: When the client receives a message, it pre-records the type and data size of each message and writes them into a message queue according to the message type and data size. When the server consumes messages, it retrieves a message list from the message queue, aggregates the messages by type, and then sends these aggregated messages to a thread pool. An aggregated message containing messages of the same type only needs to be processed by one thread in the thread pool. In this way, by aggregating messages according to their size, the IO consumption of subsequent message consumption can be reduced. At the same time, the message aggregation method can effectively reduce the number of threads used in the thread pool, further improving the message consumption rate.

[0021] Furthermore, storing the message type and data volume as keys and the message as values ​​in the message queue also includes:

[0022] In the message list of the message queue, the messages are sorted in ascending order according to the key name.

[0023] As described above, sorting messages in the message list facilitates subsequent aggregation of messages based on their type.

[0024] Furthermore, messages of the same type in the key names are aggregated, including:

[0025] Set the initial value of the first threshold, obtain the thread occupancy ratio of the thread pool, and adjust the first threshold upwards according to the occupancy ratio;

[0026] Messages of the same type in the key name are aggregated sequentially according to the message list, and the data size of the aggregated messages is less than or equal to the first threshold.

[0027] As described above, the upper limit of the data volume of aggregated messages can be set according to the specific situation of the thread pool, thereby performing message aggregation according to the specific situation of the thread pool and improving the utilization rate of threads in the thread pool.

[0028] Furthermore, the aggregated message is sent to the thread pool, including:

[0029] Set the initial value of the second threshold, obtain the thread occupancy ratio of the thread pool, and adjust the second threshold upwards based on the occupancy ratio;

[0030] Determine whether the number of aggregated messages that have not yet been sent to the thread pool has reached the second threshold. If so, send all aggregated messages that have not yet been sent to the thread pool to the thread pool.

[0031] As described above, the upper limit of the number of aggregated messages can be set according to the specific situation of the thread pool, thereby performing message aggregation according to the specific situation of the thread pool and improving the utilization rate of threads in the thread pool.

[0032] Furthermore, the aggregated message is split and consumed using threads in the thread pool, including:

[0033] The thread pool determines whether the received message is an aggregated message. If so, the aggregated message is split and the split messages are consumed one by one.

[0034] As described above, by using message aggregation and splitting aggregated messages, the number of threads used in the thread pool can be effectively reduced, thereby increasing the message consumption rate.

[0035] Please refer to Figure 2 Another embodiment of the present invention provides a message aggregation and consumption system, comprising:

[0036] The client is used to receive messages, record the type and amount of data of the messages, use the type and amount of data of the messages as keys, and store the messages as values ​​in a message queue.

[0037] On the server side, when consuming messages, a list of messages is retrieved from the message queue, messages of the same type with the same key are aggregated, and the aggregated messages are sent to a thread pool. Threads in the thread pool then split and consume the aggregated messages.

[0038] As described above, the beneficial effects of this invention are as follows: When the client receives a message, it pre-records the type and data size of each message and writes them into a message queue according to the message type and data size. When the server consumes messages, it retrieves a message list from the message queue, aggregates the messages by type, and then sends these aggregated messages to a thread pool. An aggregated message containing messages of the same type only needs to be processed by one thread in the thread pool. In this way, by aggregating messages according to their size, the IO consumption of subsequent message consumption can be reduced. At the same time, the message aggregation method can effectively reduce the number of threads used in the thread pool, further improving the message consumption rate.

[0039] Furthermore, storing the message type and data volume as keys and the message as values ​​in the message queue also includes:

[0040] In the message list of the message queue, the messages are sorted in ascending order according to the key name.

[0041] As described above, sorting messages in the message list facilitates subsequent aggregation of messages based on their type.

[0042] Furthermore, messages of the same type in the key names are aggregated, including:

[0043] Set the initial value of the first threshold, obtain the thread occupancy ratio of the thread pool, and adjust the first threshold upwards according to the occupancy ratio;

[0044] Messages of the same type in the key name are aggregated sequentially according to the message list, and the data size of the aggregated messages is less than or equal to the first threshold.

[0045] As described above, the upper limit of the data volume of aggregated messages can be set according to the specific situation of the thread pool, thereby performing message aggregation according to the specific situation of the thread pool and improving the utilization rate of threads in the thread pool.

[0046] Furthermore, the aggregated message is sent to the thread pool, including:

[0047] Set the initial value of the second threshold, obtain the thread occupancy ratio of the thread pool, and adjust the second threshold upwards based on the occupancy ratio;

[0048] Determine whether the number of aggregated messages that have not yet been sent to the thread pool has reached the second threshold. If so, send all aggregated messages that have not yet been sent to the thread pool to the thread pool.

[0049] As described above, the upper limit of the number of aggregated messages can be set according to the specific situation of the thread pool, thereby performing message aggregation according to the specific situation of the thread pool and improving the utilization rate of threads in the thread pool.

[0050] Furthermore, the aggregated message is split and consumed using threads in the thread pool, including:

[0051] The thread pool determines whether the received message is an aggregated message. If so, the aggregated message is split and the split messages are consumed one by one.

[0052] As described above, by using message aggregation and splitting aggregated messages, the number of threads used in the thread pool can be effectively reduced, thereby increasing the message consumption rate.

[0053] The message aggregation and consumption method and system described above are applicable to effectively reducing the number of threads used in a thread pool through message aggregation, thereby further improving the message consumption rate. The following is a detailed description of specific implementation methods:

[0054] Example 1

[0055] Please refer to Figure 1 A method for message aggregation and consumption, comprising the following steps:

[0056] S1. The client receives a message, records the message type and data volume, uses the message type and data volume as keys, and stores the message as a value in the message queue.

[0057] Specifically, when the client receives a message, it needs to first parse out the message type and obtain the actual size of the entire message. Then, it writes the message to the message queue using the type plus the actual size as the key. At this point, the message list in the message queue can be sorted in ascending order based on the key.

[0058] S2. When the server consumes messages, it retrieves a message list from the message queue, aggregates messages of the same type with the same key, sends the aggregated messages to the thread pool, and uses threads in the thread pool to split and consume the aggregated messages.

[0059] Specifically, when the server consumes messages, it retrieves a list of messages from the message queue and performs the following operations:

[0060] S21. Set the initial value of the first threshold, obtain the thread occupancy ratio of the thread pool, and increase the first threshold according to the occupancy ratio; aggregate messages of the same type in the key name according to the message list in order, and the data size of the aggregated messages is less than or equal to the first threshold.

[0061] In this embodiment, aggregation is only performed on messages of the same type, and the size of the aggregated message cannot exceed a first threshold.

[0062] S22. Set the initial value of the second threshold and obtain the thread occupancy ratio of the thread pool. Adjust the second threshold according to the occupancy ratio. Determine whether the number of aggregated messages that have not been sent to the thread pool has reached the second threshold. If so, send all aggregated messages that have not been sent to the thread pool to the thread pool.

[0063] In this embodiment, when the number of aggregated messages reaches a second threshold, such as when 10 aggregated messages are accumulated, they are sent all at once. If the number of aggregated messages does not reach 10, messages are retrieved from the message queue and aggregated until the number of aggregated messages reaches 10.

[0064] The aggregated messages are requested in batches to threads in the thread pool for subsequent task processing.

[0065] S23. Determine whether the received message is an aggregated message through the threads in the thread pool. If so, split the aggregated message and consume the split messages one by one.

[0066] Specifically, when a message is received in the thread pool, if it is a single message, it is processed directly on a single message basis; if it is an aggregated message, it needs to be split into individual messages first, and then processed as individual message tasks one by one.

[0067] Therefore, in this embodiment, the client sends the original message to the server and writes it directly into the actual message queue. Subsequently, an independent thread reads messages in batches from the actual message queue, aggregates the messages by type according to the specific situation of the subsequent thread pool, and consumes them again. The number and size of message groups can be dynamically and freely combined. Finally, the message groups are sent to the actual consumers.

[0068] Example 2

[0069] Please refer to Figure 2 A message aggregation and consumption system, comprising:

[0070] The client is used to receive messages, record the type and amount of data of the messages, use the type and amount of data of the messages as keys, and store the messages as values ​​in a message queue. In the message list of the message queue, the messages are sorted in ascending order according to the keys.

[0071] On the server side, when consuming messages, a list of messages is retrieved from the message queue, messages of the same type with the same key are aggregated, and the aggregated messages are sent to a thread pool. Threads in the thread pool then split and consume the aggregated messages.

[0072] Furthermore, messages of the same type in the key names are aggregated, including:

[0073] Set the initial value of the first threshold, obtain the thread occupancy ratio of the thread pool, and adjust the first threshold upwards according to the occupancy ratio;

[0074] Messages of the same type in the key name are aggregated sequentially according to the message list, and the data size of the aggregated messages is less than or equal to the first threshold.

[0075] Furthermore, the aggregated message is sent to the thread pool, including:

[0076] Set the initial value of the second threshold, obtain the thread occupancy ratio of the thread pool, and adjust the second threshold upwards based on the occupancy ratio;

[0077] Determine whether the number of aggregated messages that have not yet been sent to the thread pool has reached the second threshold. If so, send all aggregated messages that have not yet been sent to the thread pool to the thread pool.

[0078] Furthermore, the aggregated message is split and consumed using threads in the thread pool, including:

[0079] The thread pool determines whether the received message is an aggregated message. If so, the aggregated message is split and the split messages are consumed one by one.

[0080] In summary, the message aggregation and consumption method and system provided by this invention involves the following steps: When a client receives a message, it pre-records the type and data size of each message and writes them into a message queue based on their type and data size. When the server consumes messages, it retrieves a message list from the message queue, aggregates the messages by type, and then sends these aggregated messages to a thread pool. An aggregated message containing messages of the same type only requires one thread in the thread pool to process. This method, by aggregating messages according to their size, reduces the I / O consumption of subsequent message consumption. Furthermore, the message aggregation method effectively reduces the number of threads used in the thread pool, further improving the message consumption rate.

[0081] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent modifications made based on the content of the present invention specification and drawings, or direct or indirect applications in related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method of message aggregation consumption, the method comprising: The method comprises the steps of: The client receives a message, records the type and data volume of the message, stores the type and data volume of the message as a key name and the message as a value in a message queue; When the server performs message consumption, a message list is obtained from the message queue, messages of the same type in the key name are aggregated, the aggregated messages are sent to a thread pool, and the aggregated messages are split and consumed by threads in the thread pool; The type and data volume of the message are stored as a key name and the message is stored as a value in a message queue, and further comprising: In the message list of the message queue, the messages are sorted in ascending order of the key name; The messages of the same type in the key name are aggregated, comprising: Set the initial value of the first threshold value, and obtain the thread occupation ratio of the thread pool, and adjust the first threshold value according to the occupation ratio; According to the message list, the messages of the same type in the key name are aggregated in order, and the data volume of the aggregated messages is less than or equal to the first threshold value; The aggregated messages are sent to the thread pool, comprising: Set the initial value of the second threshold value, and obtain the thread occupation ratio of the thread pool, and adjust the second threshold value according to the occupation ratio; Determine whether the number of aggregated messages not sent to the thread pool currently reaches the second threshold value, if yes, all the aggregated messages not sent to the thread pool are sent to the thread pool.

2. The method of claim 1, wherein, The aggregated messages are split and consumed by threads in the thread pool, comprising: Through the threads in the thread pool, it is determined whether the received message is an aggregated message, if yes, the aggregated message is split, and the split messages are consumed one by one.

3. A system for message aggregation consumption, characterized in that, Comprise: The client receives a message, records the type and data volume of the message, stores the type and data volume of the message as a key name and the message as a value in a message queue; When the server performs message consumption, a message list is obtained from the message queue, messages of the same type in the key name are aggregated, the aggregated messages are sent to a thread pool, and the aggregated messages are split and consumed by threads in the thread pool; The type and data volume of the message are stored as a key name and the message is stored as a value in a message queue, and further comprising: In the message list of the message queue, the messages are sorted in ascending order of the key name; The messages of the same type in the key name are aggregated, comprising: Set the initial value of the first threshold value, and obtain the thread occupation ratio of the thread pool, and adjust the first threshold value according to the occupation ratio; According to the message list, the messages of the same type in the key name are aggregated in order, and the data volume of the aggregated messages is less than or equal to the first threshold value; The aggregated messages are sent to the thread pool, comprising: Set the initial value of the second threshold value, and obtain the thread occupation ratio of the thread pool, and adjust the second threshold value according to the occupation ratio; Determine whether the number of aggregated messages not sent to the thread pool currently reaches the second threshold value, if yes, all the aggregated messages not sent to the thread pool are sent to the thread pool.

4. The system for message aggregation consumption of claim 3, wherein, The aggregated messages are split and consumed by threads in the thread pool, comprising: The received message is judged by a thread in a thread pool whether it is an aggregated message, if yes, the aggregated message is split, and the split messages are consumed one by one.

Images