Message processing optimization method and system

Abstract

The invention relates to a message processing optimization method and system, and the method comprises the steps: carrying out the real-time monitoring and statistics of the number of topics subscribed by each consumer, marking the consumers with the number of topics exceeding a first threshold value as high-load consumers, and marking the consumers with the number of topics not exceeding the first threshold value as low-load consumers. High-load consumers with the same subscription characteristics are aggregated to generate N logic groups, and an exclusive consumption set is created for each logic group to receive and store topic messages of a corresponding subscription range. When a high-load consumer needs to obtain and consume a top message, the high-load consumer is allowed to directly obtain and consume from the consumption set of the corresponding logic group, and when a low-load consumer needs to obtain and consume a top message, a polling mechanism is adopted to obtain and consume from a top list subscribed by the low-load consumer. Therefore, resource consumption is reduced, the message processing throughput and response speed in a high-concurrency scene are improved, and the requirement for efficient message transmission of Internet applications is met.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to an optimization method and system for message processing. Background Technology

[0002] Currently, message middleware systems operate as follows: multiple message producers send messages to the middleware system, which then stores the received messages through internal components. Subsequently, multiple consumer programs subscribe to one or more topics and consume and process messages through polling or scanning. However, as the number of topics subscribed to by consumer programs increases, frequent polling leads to significant resource consumption, causing a decline in the overall operating efficiency of the middleware system. This is especially problematic in high-concurrency scenarios, potentially leading to performance bottlenecks and slow response times, failing to meet the efficient message delivery needs of internet applications. Summary of the Invention

[0003] The technical problem to be solved by this invention is: to provide an optimized message processing method and system that adopts a flexible message processing approach, reduces resource consumption, and improves message processing throughput and response speed in high-concurrency scenarios, thereby meeting the needs of efficient message transmission in Internet applications.

[0004] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: In a first aspect, the present invention provides an optimization method for message processing, comprising: The system monitors and counts the number of topics subscribed to by each consumer in real time, and determines whether the number of topics exceeds a first threshold. If so, consumers whose number of topics exceeds the first threshold are marked as high-load consumers; otherwise, consumers whose number of topics does not exceed the first threshold are marked as low-load consumers. Obtain the subscription information of all high-load consumers, and aggregate high-load consumers with the same subscription characteristics according to the subscription information to generate N logical groups, wherein each logical group contains a unique subscription range, which is a list of topics that high-load consumers in the logical group commonly subscribe to; Create a dedicated message set for each logical group. Each message set is used to receive and store topic messages for the corresponding subscription scope. When a high-load consumer needs to obtain and consume topic messages, it is allowed to obtain and consume them directly from the message set of the corresponding logical group. When a low-load consumer needs to obtain and consume topic messages, a polling mechanism is used to obtain and consume them from the topic list subscribed to by the low-load consumer.

[0005] The beneficial effects of this invention are as follows: It monitors and counts the number of topics subscribed to by each consumer in real time, marking consumers with more than a first threshold as high-load consumers. High-load consumers are aggregated into corresponding logical groups based on similar subscription characteristics. Through a dedicated message set for each logical group, high-load consumers can directly obtain and consume topic messages from the message set without polling, reducing resource consumption and improving message processing throughput and response speed. Furthermore, for consumers with fewer than the first threshold (i.e., low-load consumers), a traditional polling mechanism is used, ensuring compatibility with existing systems, broad applicability, and flexible adaptation to different consumer scenarios, meeting the needs of efficient message delivery in internet applications.

[0006] Optionally, determining whether the number of topics exceeds a first threshold includes: Obtain real-time resource information, including current CPU utilization and current memory utilization; Determine whether the current CPU utilization rate is lower than the first CPU threshold, and simultaneously determine whether the current memory utilization rate is lower than the first memory threshold. If the current CPU utilization rate is lower than the first CPU threshold, or the current memory utilization rate is lower than the first memory threshold, then dynamically adjust the first threshold according to the downsizing rule to obtain the dynamically adjusted first threshold. Determine whether the number of topics exceeds the dynamically adjusted first threshold.

[0007] As described above, the first threshold dynamically changes based on real-time resource information. When the current CPU utilization is lower than the first CPU threshold or the current memory utilization is lower than the first memory threshold, the first threshold is dynamically adjusted according to the downgrading rules. This means the criteria for identifying high-load consumers are lowered, allowing more consumers to be marked as high-load consumers and enter the message processing optimization mode, fully utilizing idle resources to improve message processing efficiency. Simultaneously, this avoids the shortcomings of fixed thresholds that cannot adapt to dynamic changes in system resources, preventing untimely message processing optimization when resources are scarce or insufficient optimization when resources are abundant, further improving resource utilization and operational stability.

[0008] Optionally, the subscription information includes the number of subscriptions and the subscription name, and the aggregation of high-load consumers with the same subscription characteristics based on the subscription information includes: Obtain the subscription quantity and subscription name of each high-load consumer from the subscription information, and aggregate high-load consumers with the same subscription quantity and subscription name.

[0009] As described above, high-load consumers with the same number of subscriptions and subscription names are aggregated to ensure that consumers within the same logical group have completely consistent topic subscription needs. This ensures that the topic messages received and stored in the message set of the subsequent logical group are all required by the consumers within the group, improving the accuracy of message distribution and consumption, and avoiding message redundancy or missing issues caused by mixing consumer groups with different subscription needs.

[0010] Optionally, each message set is used to receive and store topic messages corresponding to the subscription scope, including: The message status of a topic is monitored in real time. When the message status indicates that a new message has been received, the topic that received the new message is marked as a new topic. The subscription range of all logical groups is queried to determine whether the new topic exists in the subscription range of any logical group. If it does, the new topic will actively push the received new message to the message set of the corresponding logical group for storage. If not, the new topic will store the received new message in the corresponding native message queue.

[0011] As described above, for any topic receiving a new message, the subscription range of all logical groups is queried. If the new topic exists within the subscription range of any logical group, it will proactively push the received new message to the message set of the corresponding logical group for storage. This clarifies the implementation logic of message set reception and storage for logical groups, and uses proactive pushing to replace passive polling by consumers, significantly reducing message delivery latency and ensuring the timeliness of high-load consumers in acquiring and consuming topic messages. Simultaneously, for new topics not existing within the subscription range of any logical group, the received new messages are stored in the corresponding native message queue, ensuring the normal operation of the polling mechanism for low-load consumers and achieving seamless connection and compatibility between the two message processing paths.

[0012] Optionally, the new topic will actively push the received new messages to the message set of the corresponding logical group for storage, including: The message set stores new messages proactively pushed to a new topic in chronological order.

[0013] As described above, message sets have a time sequence, which allows high-load consumers to consume messages in order without additional filtering, reducing processing overhead on the consumer side and improving message processing efficiency.

[0014] Secondly, the present invention provides an optimized message processing system, comprising: The monitoring module is used to monitor and count the number of topics subscribed to by each consumer in real time, and determine whether the number of topics exceeds a first threshold. If so, consumers whose number of topics exceeds the first threshold are marked as high-load consumers; otherwise, consumers whose number of topics does not exceed the first threshold are marked as low-load consumers. The aggregation module is used to obtain the subscription information of all high-load consumers, and aggregate high-load consumers with the same subscription characteristics according to the subscription information to generate N logical groups, wherein each logical group contains a unique subscription range, which is a list of topics that high-load consumers in the logical group commonly subscribe to; The caching module is used to create a dedicated message set for each logical group. Each message set is used to receive and store topic messages for the corresponding subscription range. The consumption module is used to allow high-load consumers to directly obtain and consume topic messages from the message set of the corresponding logical group when they need to obtain and consume topic messages, and to use a polling mechanism to obtain and consume topic messages from the list of topics subscribed to by the low-load consumers when they need to obtain and consume topic messages.

[0015] The beneficial effects of this invention are as follows: It monitors and counts the number of topics subscribed to by each consumer in real time, marking consumers with more than a first threshold as high-load consumers. High-load consumers are aggregated into corresponding logical groups based on similar subscription characteristics. Through a dedicated message set for each logical group, high-load consumers can directly obtain and consume topic messages from the message set without polling, reducing resource consumption and improving message processing throughput and response speed. Furthermore, for consumers with fewer than the first threshold (i.e., low-load consumers), a traditional polling mechanism is used, ensuring compatibility with existing systems, broad applicability, and flexible adaptation to different consumer scenarios, meeting the needs of efficient message delivery in internet applications.

[0016] Optionally, the monitoring module specifically comprises: Obtain real-time resource information, including current CPU utilization and current memory utilization; Determine whether the current CPU utilization rate is lower than the first CPU threshold, and simultaneously determine whether the current memory utilization rate is lower than the first memory threshold. If the current CPU utilization rate is lower than the first CPU threshold, or the current memory utilization rate is lower than the first memory threshold, then dynamically adjust the first threshold according to the downsizing rule to obtain the dynamically adjusted first threshold. Determine whether the number of topics exceeds the dynamically adjusted first threshold.

[0017] As described above, the first threshold dynamically changes based on real-time resource information. When the current CPU utilization is lower than the first CPU threshold or the current memory utilization is lower than the first memory threshold, the first threshold is dynamically adjusted according to the downgrading rules. This means the criteria for identifying high-load consumers are lowered, allowing more consumers to be marked as high-load consumers and enter the message processing optimization mode, fully utilizing idle resources to improve message processing efficiency. Simultaneously, this avoids the shortcomings of fixed thresholds that cannot adapt to dynamic changes in system resources, preventing untimely message processing optimization when resources are scarce or insufficient optimization when resources are abundant, further improving resource utilization and operational stability.

[0018] Optionally, the subscription information includes the number of subscriptions and the subscription name, and the aggregation module specifically includes: Obtain the subscription quantity and subscription name of each high-load consumer from the subscription information, and aggregate high-load consumers with the same subscription quantity and subscription name.

[0019] As described above, high-load consumers with the same number of subscriptions and subscription names are aggregated to ensure that consumers within the same logical group have completely consistent topic subscription needs. This ensures that the topic messages received and stored in the message set of the subsequent logical group are all required by the consumers within the group, improving the accuracy of message distribution and consumption, and avoiding message redundancy or missing issues caused by mixing consumer groups with different subscription needs.

[0020] Optionally, the caching module specifically comprises: The message status of a topic is monitored in real time. When the message status indicates that a new message has been received, the topic that received the new message is marked as a new topic. The subscription range of all logical groups is queried to determine whether the new topic exists in the subscription range of any logical group. If it does, the new topic will actively push the received new message to the message set of the corresponding logical group for storage. If not, the new topic will store the received new message in the corresponding native message queue.

[0021] As described above, for any topic receiving a new message, the subscription range of all logical groups is queried. If the new topic exists within the subscription range of any logical group, it will proactively push the received new message to the message set of the corresponding logical group for storage. This clarifies the implementation logic of message set reception and storage for logical groups, and uses proactive pushing to replace passive polling by consumers, significantly reducing message delivery latency and ensuring the timeliness of high-load consumers in acquiring and consuming topic messages. Simultaneously, for new topics not existing within the subscription range of any logical group, the received new messages are stored in the corresponding native message queue, ensuring the normal operation of the polling mechanism for low-load consumers and achieving seamless connection and compatibility between the two message processing paths.

[0022] Optionally, the caching module specifically comprises: The message set stores new messages proactively pushed to a new topic in chronological order.

[0023] As described above, message sets have a time sequence, which allows high-load consumers to consume messages in order without additional filtering, reducing processing overhead on the consumer side and improving message processing efficiency. Attached Figure Description

[0024] Figure 1 This is a flowchart of an optimized message processing method provided in this embodiment; Figure 2 This is a schematic diagram of the overall process of an optimized message processing method provided in this embodiment; Figure 3 This is a schematic diagram of the structure of an optimized message processing system provided in this embodiment.

[0025] [Explanation of Labels in the Attached Image] 1. An optimized message processing system; 2. Monitoring module; 3. Aggregation module; 4. Caching module; 5. Consumption Module. Detailed Implementation

[0026] To better understand the above technical solutions, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present invention can be understood more clearly and thoroughly, and that the scope of the present invention can be fully conveyed to those skilled in the art.

[0027] Example 1 Please refer to Figures 1 to 2 This invention provides an optimization method for message processing, comprising the following steps: S1. Monitor and count the number of topics subscribed to by each consumer in real time, and determine whether the number of topics exceeds a first threshold. If so, mark the consumers whose number of topics exceeds the first threshold as high-load consumers; otherwise, mark the consumers whose number of topics does not exceed the first threshold as low-load consumers. In this embodiment, as Figure 2 As shown, the source code of the message middleware system is modified to introduce a statistics program to monitor and count the number of topics subscribed to by each consumer in real time. If the number of topics exceeds a first threshold, the consumer with the number of topics exceeding the first threshold is marked as a high-load consumer, and vice versa, it is marked as a low-load consumer.

[0028] At this point, determining whether the number of topics exceeds the first threshold in step S1 includes: S11. Obtain real-time resource information, including current CPU utilization and current memory utilization. S12. Determine whether the current CPU utilization rate is lower than the first CPU threshold, and at the same time determine whether the current memory utilization rate is lower than the first memory threshold. If the current CPU utilization rate is lower than the first CPU threshold, or the current memory utilization rate is lower than the first memory threshold, then dynamically adjust the first threshold according to the adjustment rule to obtain the dynamically adjusted first threshold. S13. Determine whether the number of topics exceeds the first threshold after dynamic adjustment.

[0029] In this embodiment, as Figure 2 As shown, the first threshold changes dynamically based on real-time resource information. Real-time resource information includes current CPU utilization and current memory utilization. If the current CPU utilization is lower than the first CPU threshold, or the current memory utilization is lower than the first memory threshold, the first threshold is dynamically adjusted according to the downward adjustment rule. That is, it is dynamically lowered to obtain the dynamically adjusted first threshold. Using the dynamically adjusted first threshold as the standard, it is determined whether the number of topics exceeds the dynamically adjusted first threshold, thus classifying high-load and low-load consumers. Conversely, if the number exceeds the first threshold, high-load and low-load consumers are classified according to the current value of the first threshold. Alternatively, the first threshold can be dynamically adjusted according to the upward adjustment rule, that is, dynamically increased, and the current first threshold is used as the standard for classifying high-load and low-load consumers.

[0030] S2. Obtain the subscription information of all high-load consumers, and aggregate high-load consumers with the same subscription characteristics according to the subscription information to generate N logical groups, wherein each logical group contains a unique subscription range, and the subscription range is a list of topics that high-load consumers in the logical group commonly subscribe to. At this point, the subscription information in step S2 includes the number of subscriptions and the subscription name, and the aggregation of high-load consumers with the same subscription characteristics based on the subscription information includes: S21. Obtain the subscription quantity and subscription name of each high-load consumer from the subscription information, and aggregate high-load consumers with the same subscription quantity and the same subscription name.

[0031] In this embodiment, as Figure 2 As shown, the subscription information of all high-load consumers is obtained. The subscription information includes, but is not limited to, subscription quantity, subscription name, subscription time and consumer ID. Based on the subscription information, high-load consumers with the same subscription characteristics are aggregated, that is, high-load consumers with the same subscription quantity and the same subscription name are aggregated to generate N logical groups. Each logical group contains a unique subscription scope, which is a list of topics that high-load consumers in the logical group have subscribed to.

[0032] S3. Create a dedicated message set for each logical group. Each message set is used to receive and store topic messages for the corresponding subscription scope. At this point, each message set mentioned in step S3 is used to receive and store topic messages corresponding to the subscription scope, including: S31. Monitor the message status of the topic in real time. When the message status is "new message received", mark the topic that received the new message as a new topic and query the subscription range of all logical groups to determine whether the new topic exists in the subscription range of any logical group. If so, the new topic will actively push the received new message to the message set of the corresponding logical group for storage. If not, the new topic will store the received new message in the corresponding native message queue.

[0033] In this embodiment, as Figure 2As shown, a dedicated message set is created for each logical group. Each message set is used to receive and store topic messages within its corresponding subscription range; that is, the message set acts as a temporary buffer. The message status of a topic is monitored in real time. When a new message is received, the topic receiving the new message is marked as a new topic, and the subscription ranges of all logical groups are queried. If the new topic exists within the subscription range of any logical group, it will proactively push the received new message to the corresponding logical group's message set for storage. In other words, for any topic receiving a new message, it is determined whether it exists within the subscription range of any logical group, thus proactively pushing information messages to the message set. Conversely, if the new topic does not exist within the subscription range of any logical group, it will store the received new message in the corresponding native message queue, ensuring the normal operation of the polling mechanism for low-load consumers.

[0034] At this point, step S31, which states that the new topic will actively push the received new messages to the message set of the corresponding logical group for storage, includes: S311. The message set stores new messages actively pushed to a new topic in chronological order.

[0035] In this embodiment, as Figure 2 As shown, the message set stores newly received messages actively pushed to a new topic in chronological order, meaning the message set has a temporal sequence. Furthermore, in this embodiment, the message set also possesses integrity and efficient read / write capabilities. Integrity refers to ensuring the integrity of the message storage process and preventing message loss and tampering. Efficient read / write capabilities refer to supporting fast message writing and reading in high-concurrency scenarios.

[0036] S4. When the high-load consumer needs to obtain and consume topic messages, it is allowed to obtain and consume them directly from the message set of the corresponding logical group. When the low-load consumer needs to obtain and consume topic messages, a polling mechanism is used to obtain and consume them from the topic list subscribed to by the low-load consumer.

[0037] In this embodiment, as Figure 2 As shown, a mode-based consumption model is adopted. For high-load consumers, topic messages are allowed to be directly obtained and consumed from the message set of the corresponding logical group. For low-load consumers, a polling mechanism is used to obtain and consume topic messages from the topic list subscribed to by the low-load consumers.

[0038] Example 2 Please refer to Figure 3The present invention provides an optimized message processing system 1, comprising: a monitoring module 2, an aggregation module 3, a caching module 4, and a consumption module 5.

[0039] Among them, the monitoring module 2 is used to monitor and count the number of topics subscribed to by each consumer in real time, and determine whether the number of topics exceeds a first threshold. If so, the consumer whose number of topics exceeds the first threshold is marked as a high-load consumer; otherwise, the consumer whose number of topics does not exceed the first threshold is marked as a low-load consumer. Aggregation module 3 is used to obtain the subscription information of all high-load consumers, and aggregate high-load consumers with the same subscription characteristics according to the subscription information to generate N logical groups, wherein each logical group contains a unique subscription range, and the subscription range is a list of topics that high-load consumers in the logical group commonly subscribe to; Cache module 4 is used to create a dedicated message set for each logical group. Each message set is used to receive and store topic messages for the corresponding subscription range. The consumption module 5 is used to allow the high-load consumer to directly obtain and consume topic messages from the message set of the corresponding logical group when the high-load consumer needs to obtain and consume topic messages, and to use a polling mechanism to obtain and consume topic messages from the list of topics subscribed to by the low-load consumer when the low-load consumer needs to obtain and consume topic messages.

[0040] Specifically, the monitoring module 2 is as follows: Obtain real-time resource information, including current CPU utilization and current memory utilization; Determine whether the current CPU utilization rate is lower than the first CPU threshold, and simultaneously determine whether the current memory utilization rate is lower than the first memory threshold. If the current CPU utilization rate is lower than the first CPU threshold, or the current memory utilization rate is lower than the first memory threshold, then dynamically adjust the first threshold according to the downsizing rule to obtain the dynamically adjusted first threshold. Determine whether the number of topics exceeds the dynamically adjusted first threshold.

[0041] Specifically, the subscription information includes the number of subscriptions and the subscription name, and the aggregation module 3 specifically comprises: Obtain the subscription quantity and subscription name of each high-load consumer from the subscription information, and aggregate high-load consumers with the same subscription quantity and subscription name.

[0042] Specifically, the cache module 4 is as follows: The message status of a topic is monitored in real time. When the message status indicates that a new message has been received, the topic that received the new message is marked as a new topic. The subscription range of all logical groups is queried to determine whether the new topic exists in the subscription range of any logical group. If it does, the new topic will actively push the received new message to the message set of the corresponding logical group for storage. If not, the new topic will store the received new message in the corresponding native message queue.

[0043] Specifically, the cache module is as follows: The message set stores new messages proactively pushed to a new topic in chronological order.

[0044] Since the systems / devices described in the above embodiments of the present invention are systems / devices used to implement the methods of the above embodiments of the present invention, those skilled in the art can understand the specific structure and modifications of the systems / devices based on the methods described in the above embodiments of the present invention, and therefore will not be repeated here. All systems / devices used in the methods of the above embodiments of the present invention fall within the scope of protection of the present invention.

[0045] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0046] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions.

[0047] It should be noted that any reference numerals placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In claims that enumerate several means, several of these means may be embodied by the same hardware. The use of the terms first, second, third, etc., is merely for convenience of expression and does not indicate any order. These terms can be understood as part of the component names.

[0048] Furthermore, it should be noted that in the description of this specification, the terms "one embodiment," "some embodiments," "embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0049] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the claims should be interpreted to include both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0050] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, then this invention should also include these modifications and variations.

Claims

1. An optimization method for message processing, characterized in that, include: The system monitors and counts the number of topics subscribed to by each consumer in real time, and determines whether the number of topics exceeds a first threshold. If so, consumers whose number of topics exceeds the first threshold are marked as high-load consumers; otherwise, consumers whose number of topics does not exceed the first threshold are marked as low-load consumers. Obtain the subscription information of all high-load consumers, and aggregate high-load consumers with the same subscription characteristics according to the subscription information to generate N logical groups, wherein each logical group contains a unique subscription range, which is a list of topics that high-load consumers in the logical group commonly subscribe to; Create a dedicated message set for each logical group. Each message set is used to receive and store topic messages for the corresponding subscription scope. When a high-load consumer needs to obtain and consume topic messages, it is allowed to obtain and consume them directly from the message set of the corresponding logical group. When a low-load consumer needs to obtain and consume topic messages, a polling mechanism is used to obtain and consume them from the topic list subscribed to by the low-load consumer.

2. The message processing optimization method as described in claim 1, characterized in that, The step of determining whether the number of topics exceeds the first threshold includes: Obtain real-time resource information, including current CPU utilization and current memory utilization; Determine whether the current CPU utilization rate is lower than the first CPU threshold, and simultaneously determine whether the current memory utilization rate is lower than the first memory threshold. If the current CPU utilization rate is lower than the first CPU threshold, or the current memory utilization rate is lower than the first memory threshold, then dynamically adjust the first threshold according to the downsizing rule to obtain the dynamically adjusted first threshold. Determine whether the number of topics exceeds the dynamically adjusted first threshold.

3. The message processing optimization method as described in claim 1, characterized in that, The subscription information includes the number of subscriptions and the subscription name. Aggregating high-load consumers with the same subscription characteristics based on the subscription information includes: Obtain the subscription quantity and subscription name of each high-load consumer from the subscription information, and aggregate high-load consumers with the same subscription quantity and subscription name.

4. The message processing optimization method as described in claim 1, characterized in that, Each message set is used to receive and store topic messages corresponding to the subscription scope, including: The message status of a topic is monitored in real time. When the message status indicates that a new message has been received, the topic that received the new message is marked as a new topic. The subscription range of all logical groups is queried to determine whether the new topic exists in the subscription range of any logical group. If it does, the new topic will actively push the received new message to the message set of the corresponding logical group for storage. If not, the new topic will store the received new message in the corresponding native message queue.

5. The message processing optimization method as described in claim 4, characterized in that, The new topic will proactively push newly received messages to the message set of the corresponding logical group for storage, including: The message set stores new messages proactively pushed to a new topic in chronological order.

6. An optimized message processing system, characterized in that, include: The monitoring module is used to monitor and count the number of topics subscribed to by each consumer in real time, and determine whether the number of topics exceeds a first threshold. If so, consumers whose number of topics exceeds the first threshold are marked as high-load consumers; otherwise, consumers whose number of topics does not exceed the first threshold are marked as low-load consumers. The aggregation module is used to obtain the subscription information of all high-load consumers, and aggregate high-load consumers with the same subscription characteristics according to the subscription information to generate N logical groups, wherein each logical group contains a unique subscription range, which is a list of topics that high-load consumers in the logical group commonly subscribe to; The caching module is used to create a dedicated message set for each logical group. Each message set is used to receive and store topic messages for the corresponding subscription range. The consumption module is used to allow high-load consumers to directly obtain and consume topic messages from the message set of the corresponding logical group when they need to obtain and consume topic messages, and to use a polling mechanism to obtain and consume topic messages from the list of topics subscribed to by the low-load consumers when they need to obtain and consume topic messages.

7. The message processing optimization system as described in claim 1, characterized in that, The monitoring module is specifically: Obtain real-time resource information, including current CPU utilization and current memory utilization; Determine whether the current CPU utilization rate is lower than the first CPU threshold, and simultaneously determine whether the current memory utilization rate is lower than the first memory threshold. If the current CPU utilization rate is lower than the first CPU threshold, or the current memory utilization rate is lower than the first memory threshold, then dynamically adjust the first threshold according to the downsizing rule to obtain the dynamically adjusted first threshold. Determine whether the number of topics exceeds the dynamically adjusted first threshold.

8. The message processing optimization system as described in claim 1, characterized in that, The subscription information includes the number of subscriptions and the subscription name, and the aggregation module specifically comprises: Obtain the subscription quantity and subscription name of each high-load consumer from the subscription information, and aggregate high-load consumers with the same subscription quantity and subscription name.

9. The message processing optimization system as described in claim 1, characterized in that, The cache module is specifically: The message status of a topic is monitored in real time. When the message status indicates that a new message has been received, the topic that received the new message is marked as a new topic. The subscription range of all logical groups is queried to determine whether the new topic exists in the subscription range of any logical group. If it does, the new topic will actively push the received new message to the message set of the corresponding logical group for storage. If not, the new topic will store the received new message in the corresponding native message queue.

10. The message processing optimization system as described in claim 9, characterized in that, The cache module is specifically: The message set stores new messages proactively pushed to a new topic in chronological order.

Images