Queue message backlog processing method, system and equipment based on coroutine technology
By dynamically adjusting the number of consumers and partitions using coroutine technology, combined with an asynchronous acknowledgment mechanism, the message queue backlog problem was solved, achieving efficient message processing, improving system scalability and throughput, and reducing latency.
Patent Information
- Application Number
- CN202511610104.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-05
- Publication Date
- 2026-02-10
AI Technical Summary
Existing technologies cannot effectively solve the message queue backlog problem in high-concurrency scenarios, resulting in high resource consumption, poor scalability, and delayed response, which cannot meet the real-time and throughput requirements of the system.
The number of consumers is dynamically adjusted and partitioned using coroutine technology. Coroutines are used to process messages concurrently, and an asynchronous acknowledgment mechanism is combined to achieve dynamic expansion of the coroutine pool and local caching, thereby optimizing the message processing flow.
It significantly reduces memory resource consumption, improves system scalability and responsiveness, increases throughput and reliability, and reduces message latency.
Smart Images
Figure CN121509335A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of message processing technology, specifically relating to a method, system, and device for processing queued message backlog based on coroutine technology. Background Technology
[0002] With the rapid development of internet technology, message queues, as a core module for asynchronous communication and peak shaving, are widely used in scenarios such as e-commerce order processing, log collection, and data synchronization between distributed systems. In practical applications, when business is at its peak, the message generation rate often far exceeds the consumer's processing speed, easily leading to a large backlog of messages in the queue. This message backlog not only significantly increases the storage pressure on the message queue but can also cause data expiration and business process delays and interruptions, seriously affecting system stability and service quality. To address the aforementioned message backlog problem, several solutions have been proposed in the existing technology, as follows: Adding consumer instances: This approach improves the overall message processing capacity of the system by horizontally scaling the number of consumer instances. However, this approach has significant drawbacks. On the one hand, it consumes a large amount of server resources (such as CPU, memory, and network bandwidth), increasing the system's operational burden. On the other hand, limited by the number of message queue partitions, when the number of consumer instances exceeds the number of partitions, newly added consumer instances cannot be allocated to message partitions, resulting in a non-linear performance improvement and difficulty in meeting the processing needs of high-concurrency scenarios. Optimizing Consumer Processing Logic: This solution improves the message processing speed of a single consumer instance by simplifying business processes and optimizing code execution efficiency. However, in practical applications, the optimization space for consumer processing logic is limited. When faced with sudden surges in concurrent messages, relying solely on optimizing the processing logic of a single consumer cannot fundamentally solve the message backlog problem, and the system's ability to withstand peak traffic is weak. Using a priority queue scheme: This scheme sets priorities for messages of different business types, allowing high-priority messages to be processed first, thereby reducing message backlog for critical business operations. However, this scheme can only alleviate the problem for some critical business operations, and the overall backlog problem of the message queue is not solved; at the same time, if the priority is not set properly, it may also lead to long-term delays for low-priority messages. In summary, existing solutions for message backlog generally suffer from core defects such as high resource consumption, poor scalability, and delayed response. They cannot meet the requirements of systems in high-concurrency scenarios for real-time performance, throughput, and resource utilization. There is an urgent need for a more efficient and lightweight message processing technology solution. Summary of the Invention
[0003] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method, system and device for processing queued message backlog based on coroutine technology.
[0004] To solve the above-mentioned technical problems, the present invention provides the following technical solution: The first objective of this invention is to provide a method for processing queued message backlog based on coroutine technology, comprising the following steps: (1) Monitor the backlog of the message queue in real time, and dynamically increase the number of coroutine consumers according to the backlog of the message queue; divide the message queue into multiple partitions and dynamically adjust the partition allocation according to the load of each coroutine consumer. (2) The coroutine consumer prefetches messages in batches from the message queue server. The number of messages prefetched is dynamically adjusted according to the current backlog and the consumer's processing speed. A local cache queue is built on the consumer side to store the prefetched messages. (3) The coroutine consumer uses coroutines to process multiple messages concurrently. After the message processing is completed, it sends an acknowledgment to the message queue in conjunction with the asynchronous acknowledgment mechanism.
[0005] Furthermore, in step (1), dynamically increasing the number of coroutine consumers based on the message queue backlog includes: Set a message queue backlog threshold. When the message backlog exceeds the preset threshold, automatically create coroutine consumer instances to expand the coroutine pool. When the backlog is below the preset threshold, automatically destroy idle coroutine consumer instances.
[0006] Furthermore, in step (2), the cache queue is stored in memory and a cache expiration mechanism is set to re-deliver expired messages.
[0007] Furthermore, in step (3), the coroutine consumer uses coroutines to concurrently process multiple messages, and rationally allocates the number of coroutines and task scheduling with the goal of maximizing the use of CPU resources.
[0008] Furthermore, in step (3), if message processing fails, the message is retried or marked as a dead message according to a preset strategy within the current coroutine.
[0009] Another object of the present invention is to provide a queue message backlog processing system based on coroutine technology, comprising: The coroutine consumer management module includes a coroutine pool dynamic expansion unit and a consumer load balancing unit. The coroutine pool dynamic expansion unit is used to monitor the message queue backlog in real time and dynamically increase the number of coroutine consumers according to the message queue backlog. The consumer load balancing unit is used to divide the message queue into multiple partitions and dynamically adjust the partition allocation according to the load of each coroutine consumer. The message prefetching and caching module includes a batch message prefetching unit and a local cache management unit. The batch message prefetching unit is used by the coroutine consumer to prefetch messages from the message queue server in batches. The number of messages prefetched is dynamically adjusted according to the current backlog and the consumer's processing speed. The local cache management unit is used to build a local cache queue on the consumer side to store the prefetched messages. The coroutine processing and acknowledgment module includes a concurrent message processing unit and a reliable acknowledgment mechanism unit. The concurrent message processing unit is used by the coroutine consumer to process multiple messages concurrently using coroutines. The reliable acknowledgment mechanism unit is used to send an acknowledgment to the message queue after message processing is completed, in conjunction with an asynchronous acknowledgment mechanism.
[0010] Another object of the present invention is to provide an electronic device, including a processor and a memory storing a computer program, wherein the processor executes the computer program to implement the queue message backlog processing method based on coroutine technology provided by the first object of the present invention.
[0011] In combination with the above technical solutions, the beneficial effects of the present invention compared with the prior art are as follows: Low resource consumption: Coroutines consume less than 1KB of memory, far less than the 1MB or more of memory consumed by threads. Under the same concurrency scenario, memory resources are saved by 99%. At the same time, coroutine switching is completed in user space, without the need for kernel space context switching, avoiding thread lock contention and context switching overhead. Single-core CPU utilization can reach more than 95%.
[0012] High scalability and fast response: Coroutines can be created and destroyed in user space in milliseconds. When message backlog is detected, coroutine expansion can be completed within 50ms without adding physical machines or restarting the service, achieving "soft expansion". Compared with traditional thread pool solutions, it can quickly respond to changes in message queue load and avoid the backlog from continuing to rise.
[0013] High throughput and low latency: By using batch prefetching and local caching, the number of interactions with the message queue server is reduced, thus reducing network overhead. This can reduce the number of network interactions by more than 90%. Combined with coroutine-level concurrent processing, throughput can be linearly increased without increasing the number of physical machines or threads. The asynchronous acknowledgment mechanism avoids blocking message processing by acknowledgment operations, reducing message processing latency by 80%.
[0014] High reliability: The in-coroutine failure retry mechanism avoids global rollback and ensures that the failure of processing a single message does not affect other messages; the dead letter marking strategy can isolate unprocessable messages in a timely manner and ensure stable system operation; the local cache expiration mechanism and asynchronous batch processing of receipts further ensure that messages are not lost or processed repeatedly.
[0015] In summary, this invention leverages the high concurrency advantages of coroutine technology to solve the message backlog problem in message queues. Through optimization techniques such as dynamically expanding the coroutine consumer pool, batch prefetching messages, and concurrent processing, it significantly improves the processing capacity of consumers and the throughput of the message queue, reduces message latency, and enhances the system's real-time performance and resource utilization. Attached Figure Description
[0016] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of a queue message backlog processing method based on coroutine technology provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the queue message backlog processing method based on coroutine technology provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of the queue message backlog processing system based on coroutine technology provided in an embodiment of the present invention. Detailed Implementation
[0017] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.
[0018] Example 1: like Figure 1 The image shows an embodiment of the queue message backlog processing method based on coroutine technology provided by the present invention, which specifically includes the following steps: S1: Monitor message queue backlog in real time and dynamically increase the number of coroutine consumers based on the message queue backlog; divide the message queue into multiple partitions and dynamically adjust partition allocation based on the load of each coroutine consumer. S2: The coroutine consumer prefetches messages in batches from the message queue server. The number of messages prefetched is dynamically adjusted according to the current backlog and the consumer's processing speed. A local cache queue is built on the consumer side to store the prefetched messages. S3: Coroutine consumers use coroutines to process multiple messages concurrently. After the message processing is completed, an acknowledgment is sent to the message queue in conjunction with an asynchronous acknowledgment mechanism.
[0019] Specifically, such as Figure 2As shown, this embodiment of the invention constructs a lightweight, high-concurrency message consumption architecture based on coroutines, and combines user-space scheduling, micro-thread-level concurrency, dynamic elastic scaling and other technologies to achieve millisecond-level response to changes in message backlog, support for tens of thousands of concurrent consumption tasks on a single node, and improve resource utilization by more than 50%. Moreover, it can linearly improve system throughput without increasing the number of physical machines or threads, fundamentally solving the message backlog problem.
[0020] Preferably, in step S1 of this embodiment of the invention, the step of dynamically increasing the number of coroutine consumers according to the backlog of the message queue includes: setting a backlog threshold for the message queue; when the message backlog exceeds the preset threshold, automatically creating coroutine consumer instances to expand the coroutine pool; and when the backlog is lower than the preset threshold, automatically destroying idle coroutine consumer instances.
[0021] This invention's embodiments dynamically expand the coroutine pool based on backlog volume, detecting backlog in milliseconds and completing coroutine expansion in seconds, without needing to restart services or add machines; on-demand elasticity avoids resource waste.
[0022] Preferably, in step S2 of this embodiment of the invention, the cache queue is stored in memory and a cache expiration mechanism is set to re-deliver expired messages.
[0023] The embodiments of the present invention achieve zero-latency message distribution through local memory caching, increasing throughput by 3 to 5 times.
[0024] Preferably, in step S3 of this embodiment of the invention, the coroutine consumer uses coroutines to concurrently process multiple messages, and rationally allocates the number of coroutines and task scheduling with the goal of maximizing the use of CPU resources, thereby improving message processing speed.
[0025] Preferably, in step S3 of this embodiment of the invention, if message processing fails, the message is retried or marked as a dead message according to a preset strategy within the current coroutine.
[0026] This invention avoids thread lock contention and eliminates context switching bottlenecks by using coroutine-level task sharding and event loop scheduling; it achieves zero CPU idle time and single-core utilization of over 95%; acknowledgment operations do not block message processing, reducing latency by 80%; and failed tasks are quickly retried within the coroutine, avoiding global rollback.
[0027] The advantages of the present invention will be further explained below with reference to specific experimental data.
[0028] 1. Experimental objective: Compared to the traditional "fixed thread pool consumption model", the "elastic coroutine consumption model" of this invention has better peak throughput, backlog clearing time, scalability and P99 latency performance under the same hardware and external traffic impact.
[0029] 2. System Configuration: (1) Hardware: 3 Alibaba Cloud servers (4C / 16T, 8g memory, 256g disk); (2) Software: OS: CentOS 7.6; Kafka: 3.5.1, 3-node cluster; Monitoring: Prometheus 2.45 + Grafana 10.0, metric collection cycle 1 second.
[0030] 3. Theme and Partition Configuration: Subject: test, 6 partitions; Message body: 1 KB JSON (with 20 fields); On the production side: kafka-producer-perf-test runs 20 processes × 10 threads for 30 minutes, continuously feeding in 200 messages per second to create a backlog scenario.
[0031] 4. Consumer-side parameters: This invention (coroutine) Language: golang + goroutine; For a single-process instance, the initial number of coroutines is 6 (1:1 with partitions), and the elasticity threshold is: the backlog of lag in a single partition is ≥ 5000 records. The maximum number of polls is 800, the throughput is 1 MB, the latency is 50ms, and the asynchronous commit interval is 1 second.
[0032] 5. Existing solution (thread pool): Fixed thread count = 6 (1:1 with partition), JDK8 runtime environment; Queue: LinkedBlockingQueue (unbounded queue, default); The parameters are the same as above (maximum number of polls 800, throughput 1 MB, latency 50ms, etc.) to ensure fair comparison. There is no automatic scaling; the script is manually restarted to adjust the number of consumer threads to N and then restarts on a rolling basis.
[0033] 6. Experimental steps: S1. Preheating: 5 minutes each for production and consumption, allowing the background cache to reach a steady state; S2. Baseline Recording: Normal consumption with no backlog, record average throughput and P99 latency; S3. Manufacturing Burst: Increase production flow from 0 to 2 million units / second within 30 seconds, and continue for 30 minutes; S4. Observation Indicators: Peak throughput = the maximum number of offsets successfully submitted within a 30-second sliding window; Backlog clearing time = the time from when the backlog amount first lag > 0 to when the backlog amount lag = 0; P99 Delay = End-to-end delay from message production to completion of commit; S5. Manual intervention comparison: When the backlog of lags reached 950,000 and P99 > 5 seconds, the operation and maintenance personnel manually restarted the service and increased the number of threads to 432, recording the restart time and recovery status.
[0034] 7. Experimental Data: a. Performance comparison: Peak throughput: This invention: 185,000 packets / second; Existing solutions: 52,000 packets / second; an improvement of 256%.
[0035] Backlog clearing time: This invention: 18 seconds; Existing solution: 192 seconds; Reduction: 90.6%.
[0036] b. Comparison of backlog Coroutine solution: If the backlog exceeds the threshold at the 5th second, start 2000 new coroutines within 50ms and clear the backlog within 18 seconds; Thread pool solution: Unable to automatically expand, backlog continues to rise to 950,000 threads, P99 latency surges to over 5 seconds, requiring manual service restart for expansion.
[0037] Example 2: like Figure 3 As shown, this embodiment of the invention provides a queue message backlog processing system based on coroutine technology, including: The coroutine consumer management module includes a coroutine pool dynamic expansion unit and a consumer load balancing unit. The coroutine pool dynamic expansion unit is used to monitor the message queue backlog in real time and dynamically increase the number of coroutine consumers according to the message queue backlog. The consumer load balancing unit is used to divide the message queue into multiple partitions and dynamically adjust the partition allocation according to the load of each coroutine consumer. The message prefetching and caching module includes a batch message prefetching unit and a local cache management unit. The batch message prefetching unit is used by the coroutine consumer to prefetch messages from the message queue server in batches. The number of messages prefetched is dynamically adjusted according to the current backlog and the consumer's processing speed. The local cache management unit is used to build a local cache queue on the consumer side to store the prefetched messages. The coroutine processing and acknowledgment module includes a concurrent message processing unit and a reliable acknowledgment mechanism unit. The concurrent message processing unit is used by the coroutine consumer to process multiple messages concurrently using coroutines. The reliable acknowledgment mechanism unit is used to send an acknowledgment to the message queue after message processing is completed, in conjunction with an asynchronous acknowledgment mechanism.
[0038] Example 3: The present invention provides an electronic device, including a processor and a memory storing a computer program. When the processor executes the computer program, it implements the queue message backlog processing method based on coroutine technology provided in embodiment 1 of the present invention.
[0039] It should be understood that although the steps in the flowcharts of the various embodiments of the present invention are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated in the present invention, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least some steps in the various embodiments may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.
[0040] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0041] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A queue message backlog processing method based on coroutine technology, characterized in that, The method includes the following steps: (1) Monitor the backlog of the message queue in real time, and dynamically increase the number of coroutine consumers according to the backlog of the message queue; divide the message queue into multiple partitions and dynamically adjust the partition allocation according to the load of each coroutine consumer. (2) The coroutine consumer prefetches messages in batches from the message queue server. The number of messages prefetched is dynamically adjusted according to the current backlog and the consumer's processing speed. A local cache queue is built on the consumer side to store the prefetched messages. (3) The coroutine consumer uses coroutines to process multiple messages concurrently. After the message processing is completed, it sends an acknowledgment to the message queue in conjunction with the asynchronous acknowledgment mechanism.
2. The queue message backlog processing method based on coroutine technology according to claim 1, characterized in that, In step (1), dynamically increasing the number of coroutine consumers based on the message queue backlog includes: Set a message queue backlog threshold. When the message backlog exceeds the preset threshold, automatically create coroutine consumer instances to expand the coroutine pool. When the backlog is below the preset threshold, automatically destroy idle coroutine consumer instances.
3. The queue message backlog processing method based on coroutine technology according to claim 1, characterized in that, In step (2), the cache queue uses memory storage and is configured with a cache expiration mechanism to redeliver expired messages, specifically including: Set a time-to-live threshold T1 to measure the maximum allowed time a message can remain in the local cache queue. Set a retry count threshold N1 to measure the maximum number of times the same message is redelivered due to cache expiration; If any message remains in the local cache queue for more than T1, it is determined to be an expired cache message, triggering the redelivery process. If the same message expires in the cache N1 times, it will not be redelivered and will be marked as a dead message.
4. The queue message backlog processing method based on coroutine technology according to claim 1, characterized in that, In step (3), the coroutine consumer uses coroutines to process multiple messages concurrently, and rationally allocates the number of coroutines and task scheduling with the goal of maximizing the use of CPU resources.
5. The queue message backlog processing method based on coroutine technology according to claim 1, characterized in that, In step (3), if message processing fails, retry or mark as a dead message according to the preset strategy within the current coroutine.
6. A queue message backlog processing system based on coroutine technology, characterized in that, The system includes: The coroutine consumer management module includes a coroutine pool dynamic expansion unit and a consumer load balancing unit. The coroutine pool dynamic expansion unit is used to monitor the message queue backlog in real time and dynamically increase the number of coroutine consumers according to the message queue backlog. The consumer load balancing unit is used to divide the message queue into multiple partitions and dynamically adjust the partition allocation according to the load of each coroutine consumer. The message prefetching and caching module includes a batch message prefetching unit and a local cache management unit. The batch message prefetching unit is used by the coroutine consumer to prefetch messages from the message queue server in batches. The number of messages prefetched is dynamically adjusted according to the current backlog and the consumer's processing speed. The local cache management unit is used to build a local cache queue on the consumer side to store the prefetched messages. The coroutine processing and acknowledgment module includes a concurrent message processing unit and a reliable acknowledgment mechanism unit. The concurrent message processing unit is used by the coroutine consumer to process multiple messages concurrently using coroutines. The reliable acknowledgment mechanism unit is used to send an acknowledgment to the message queue after message processing is completed, in conjunction with an asynchronous acknowledgment mechanism.
7. An electronic device comprising a processor and a memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the queue message backlog processing method based on coroutine technology as described in any one of claims 1 to 5.