High-concurrency message processing method and system

Abstract

The invention relates to the technical field of distributed system architecture, and particularly provides a high-concurrency message processing method and system, and the method comprises the following steps: S1, carrying out the registration of a dynamic consumer group; s2, adopting a multi-stage retry strategy to carry out exception handling chain design; and S3, memory and performance optimization. Compared with the prior art, an independent message queue assembly does not need to be deployed, Redis infrastructure is reused, and the operation and maintenance overhead is reduced by 50% or above; a single node supports 100,000 + TPS, which is improved by 3-5 times compared with a traditional message queue; if the message loss rate is lower than 0.001%, the retry mechanism covers network jitter and service fault scenes; dynamic adjustment of the number of consumers and Redis Cluster fragmentation is supported, and the service elastic requirement is met.

Description

Technical Field

[0001] The present invention relates to the technical field of distributed system architecture, and specifically provides a high-concurrency message processing method and system. Background Art

[0002] In the field of industrial data collection and real-time data processing, traditional message queues and RedisStream native solution architectures often face the following technical pain points when dealing with large-scale real-time data collection and high-concurrency business scenarios:

[0003] Traditional message queues (such as Kafka and RabbitMQ): Independent cluster deployment is required, resulting in high operation and maintenance costs. Disk I / O becomes a performance bottleneck, making it difficult to support the data throughput requirements of highly concurrent businesses.

[0004] There is a lack of a dynamic consumer group management mechanism, which makes it impossible to automatically balance the load; there are no defined message retry strategies and standardized processes for dead letter queues, and exception handling relies on manual intervention; the Redis single-threaded model and non-blocking I / O features are not fully utilized, resulting in low resource utilization in high-concurrency scenarios.

[0005] In high-concurrency scenarios, the risk of message accumulation is high, requiring real-time dynamic expansion; there is a lack of automatic recovery mechanism after message processing fails, resulting in data loss or business interruption. Summary of the Invention

[0006] The present invention aims to address the deficiencies of the above-mentioned prior art and provides a highly practical high-concurrency message processing method.

[0007] A further technical task of the present invention is to provide a high-concurrency message processing system that is rationally designed, safe and applicable.

[0008] The technical solution adopted by the present invention to solve its technical problem is:

[0009] A high-concurrency message processing method has the following steps:

[0010] S1, dynamic consumer group registration;

[0011] S2. Use a multi-level retry strategy to design an exception handling chain;

[0012] S3, memory, and performance optimization.

[0013] Furthermore, in step S1, the custom annotation scans the consumer implementation class and automatically registers it to the listening container to achieve automatic parallel processing of multiple consumer groups. A node heartbeat detection mechanism is added, and a scheduled task is used to scan the status of consumer nodes. Unregistered nodes are automatically added to the consumer group, and failed nodes are removed from the container.

[0014] Dynamically adjust the message distribution strategy based on the number of online nodes and CPU load to avoid overloading of a single node.

[0015] Furthermore, in step S2, retries are performed when consumption is abnormal, with the interval increasing exponentially. After exceeding the retry threshold, the message is transferred to an independent dead letter stream, preserving the original message context.

[0016] Furthermore, in step S3, the producer uses XADD to write in batches, and the consumer pulls messages in batches through XREADGROUP. The stream length is limited by the XTRIM command and MAXLEN to balance memory and data integrity. The connection pool is used to reuse Redis connections to reduce the overhead of frequently establishing or closing connections.

[0017] A high-concurrency message processing system that first performs dynamic consumer registration, then adopts a multi-level retry strategy, designs an exception handling chain, and finally optimizes memory and performance.

[0018] Furthermore, in dynamic consumer registration, custom annotations are used to scan consumer implementation classes and automatically register them in the listening container, enabling automatic parallel processing of multiple consumer groups. A node heartbeat detection mechanism is added, and scheduled tasks are used to scan the status of consumer nodes. Unregistered nodes are automatically added to the consumer group, and failed nodes are removed from the container.

[0019] Dynamically adjust the message distribution strategy based on the number of online nodes and CPU load to avoid overloading of a single node.

[0020] Furthermore, when a multi-level retry strategy is adopted, retries are performed when consumption is abnormal, and the interval time increases exponentially; after exceeding the retry threshold, the message is transferred to an independent dead letter stream, retaining the original message context.

[0021] To further optimize memory and performance, producers use XADD for batch writing, consumers use XREADGROUP for batch pulling of messages, and use the XTRIM command and MAXLEN to limit the stream length, balancing memory and data integrity. A connection pool is used to reuse Redis connections, reducing the overhead of frequently establishing or closing connections.

[0022] Compared with the prior art, the high-concurrency message processing method and system of the present invention have the following outstanding beneficial effects:

[0023] This invention eliminates the need to deploy independent message queue components and reuses Redis infrastructure, reducing operation and maintenance expenses by more than 50%. A single node supports 100,000+ TPS, which is 3-5 times higher than traditional message queues. The message loss rate is less than 0.001%, and the retry mechanism covers network jitter and service failure scenarios. It supports dynamic adjustment of the number of consumers and Redis Cluster sharding to meet business flexibility needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0025] Attachment Figure 1 It is a flowchart of a high-concurrency message processing method. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the solutions of the present invention, the present invention will be further described in detail below in conjunction with specific embodiments. Obviously, the embodiments described are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0027] A best embodiment is given below:

[0028] like Figure 1 As shown, a high-concurrency message processing method in this embodiment has the following steps:

[0029] S1, dynamic consumer group registration;

[0030] Scan consumer implementation classes through custom annotations, automatically register them to the listening container, and realize automatic parallel processing of multiple consumer groups; add a node heartbeat detection mechanism, use scheduled tasks to scan the status of consumer nodes, automatically add unregistered nodes to the consumer group, and remove failed nodes from the container to ensure high availability of the service.

[0031] Intelligent load distribution: Dynamically adjusts message distribution strategies based on the number of online nodes and CPU load to avoid overloading a single node.

[0032] S2. Use a multi-level retry strategy to design an exception handling chain;

[0033] When consumption is abnormal, retries are performed with an exponentially increasing interval. After exceeding the retry threshold, the message is transferred to an independent dead letter stream, preserving the original message context.

[0034] S3, memory, and performance optimization;

[0035] Batch operations reduce the number of network round trips. Producers use XADD for batch writing, and consumers use XREADGROUP for batch pulling of messages. The XTRIM command and MAXLEN are used to limit the stream length, balancing memory and data integrity. A connection pool is used to reuse Redis connections, reducing the overhead of frequently establishing and closing connections.

[0036] Based on the above method, a high-concurrency message processing system in this embodiment first performs dynamic consumer registration, then adopts a multi-level retry strategy, designs an exception handling chain, and finally optimizes memory and performance.

[0037] Furthermore, in dynamic consumer registration, custom annotations are used to scan consumer implementation classes and automatically register them in the listening container, enabling automatic parallel processing of multiple consumer groups. A node heartbeat detection mechanism is added, and scheduled tasks are used to scan the status of consumer nodes. Unregistered nodes are automatically added to the consumer group, and failed nodes are removed from the container.

[0038] Dynamically adjust the message distribution strategy based on the number of online nodes and CPU load to avoid overloading of a single node.

[0039] When a multi-level retry strategy is adopted, retries are performed when consumption is abnormal, and the interval time increases exponentially; after exceeding the retry threshold, the message is transferred to an independent dead letter stream, retaining the original message context.

[0040] When optimizing memory and performance, producers use XADD for batch writing, consumers use XREADGROUP for batch pulling of messages, and use the XTRIM command and MAXLEN to limit the stream length, balancing memory and data integrity. Connection pools are used to reuse Redis connections, reducing the overhead of frequently establishing or closing connections.

[0041] The above-mentioned specific implementation methods are only specific cases of the present invention. The patent protection scope of the present invention includes but is not limited to the above-mentioned specific implementation methods. Any technical solutions that conform to the above-mentioned specific implementation methods of the present invention and any appropriate changes or substitutions made thereto by ordinary technicians in the relevant technical field shall fall within the patent protection scope of the present invention.

[0042] This method, based on a high-concurrency message processing approach developed with Redis Stream, dynamically expands consumer groups, optimizes message confirmation and retry mechanisms, and improves load balancing and data distribution mechanisms. While significantly reducing resource consumption costs, it enables efficient processing of large-scale real-time data streams, ensuring the real-time, reliability, and accuracy of message processing.

[0043] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A high-concurrency message processing method, characterized in that: The steps are as follows: S1, dynamic consumer group registration; S2. Use a multi-level retry strategy to design an exception handling chain; S3, memory, and performance optimization.

2. A high-concurrency message processing method according to claim 1, characterized in that: In step S1, the custom annotation scans the consumer implementation class and automatically registers it to the listening container to achieve automatic parallel processing of multiple consumer groups. A node heartbeat detection mechanism is added, and a scheduled task is used to scan the status of consumer nodes. Unregistered nodes are automatically added to the consumer group, and failed nodes are removed from the container. Dynamically adjust the message distribution strategy based on the number of online nodes and CPU load to avoid overloading of a single node.

3. A high-concurrency message processing method according to claim 2, characterized in that: In step S2, when consumption is abnormal, retries are performed with an exponentially increasing interval. After exceeding the retry threshold, the message is transferred to an independent dead letter stream, preserving the original message context.

4. A high-concurrency message processing method according to claim 3, characterized in that: In step S3, the producer uses XADD to write in batches, and the consumer uses XREADGROUP to pull messages in batches. The XTRIM command and MAXLEN are used to limit the stream length, balance memory and data integrity, and use the connection pool to reuse Redis connections to reduce the overhead of frequently establishing or closing connections.

5. A high-concurrency message processing system, characterized in that: First, perform dynamic consumer registration, then adopt a multi-level retry strategy, design an exception handling chain, and finally optimize memory and performance.

6. A high-concurrency message processing system according to claim 5, characterized in that: In dynamic consumer registration, custom annotations are used to scan consumer implementation classes and automatically register them in the listening container, enabling automatic parallel processing of multiple consumer groups. A node heartbeat detection mechanism is added, and scheduled tasks are used to scan the status of consumer nodes. Unregistered nodes are automatically added to the consumer group, and failed nodes are removed from the container. Dynamically adjust the message distribution strategy based on the number of online nodes and CPU load to avoid overloading of a single node.

7. A high-concurrency message processing system according to claim 6, characterized in that: When a multi-level retry strategy is adopted, retries are performed when consumption is abnormal, and the interval time increases exponentially. After exceeding the retry threshold, the message is transferred to an independent dead letter stream, preserving the original message context.

8. A high-concurrency message processing system according to claim 7, characterized in that: When optimizing memory and performance, producers use XADD for batch writing, consumers use XREADGROUP for batch pulling of messages, and use the XTRIM command and MAXLEN to limit the stream length, balancing memory and data integrity. Connection pools are used to reuse Redis connections, reducing the overhead of frequently establishing or closing connections.

Images