A message queue processing method, system and medium
By designing a message pool and multi-level index, and employing a priority scheduling mechanism combined with an elastic scaling strategy, the operational complexity and performance bottlenecks of existing message queue technologies have been resolved. This has resulted in a lightweight, easy-to-deploy, and highly efficient resource scheduling message queue system, improving the system's robustness and operational efficiency.
Patent Information
- Application Number
- CN202610814775.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-06-08
- Publication Date
- 2026-08-25
AI Technical Summary
Existing mainstream message queue technologies suffer from problems such as high operational complexity, poor real-time performance, significant performance degradation when processing message backlog, bottlenecks in throughput and concurrency performance, and steep learning curves, making it difficult to achieve lightweight, easy-to-deploy and manage, and efficient resource scheduling capabilities.
It adopts an innovative model of message pool + multi-level index, combined with priority scheduling mechanism and elastic scaling strategy, and manages messages through globally unique identifiers to achieve lightweight storage and efficient resource scheduling, dynamically adjust the scale of processing threads and processes, and eliminate the dependence on external heavy components.
A message queue system that significantly reduces storage overhead and write latency, ensures efficient processing of large messages, supports high-concurrency scenarios, simplifies deployment and maintenance, improves system robustness and operational efficiency, and achieves adaptive capabilities.
Smart Images
Figure CN122633440A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of embedded software development, specifically to a message queue processing method, system, and medium. Background Technology
[0002] Message queues, as a core infrastructure of modern distributed systems, introduce an intermediate layer between message producers and consumers, achieving system decoupling, asynchronous processing, traffic shaping, and ensuring message order and reliability. These characteristics have led to the widespread application of message queues in many fields such as microservice architecture, big data processing, and the Internet of Things.
[0003] However, existing mainstream message queuing technologies, such as Apache Kafka, RabbitMQ, and RocketMQ, still exhibit some problems in practical applications:
[0004] High operational complexity and relatively poor real-time performance: These systems typically require the deployment and maintenance of external coordination services such as ZooKeeper, involving numerous configuration options and demanding high technical skills from operations and maintenance personnel. Furthermore, to ensure high throughput, their internal mechanisms (such as Kafka's batch processing) may sacrifice end-to-end message real-time performance.
[0005] Performance degrades significantly when dealing with a large backlog of messages: When the processing capacity of the consumer is insufficient, resulting in a large backlog of messages in the queue, the I / O performance and indexing efficiency of these systems will drop sharply, and may even trigger a service avalanche, with a long and complicated recovery process.
[0006] Throughput and concurrency performance bottlenecks exist: Although some systems claim to have high throughput, their concurrency processing capabilities and resource scheduling efficiency still have room for improvement when faced with massive amounts of small messages or mixed-size messages, especially in scenarios that require strict priority scheduling.
[0007] The learning curve is relatively steep: its complex conceptual model (such as Kafka topics, message partitions, consumer clusters, etc.) and configuration parameters make it difficult for developers to get started.
[0008] Therefore, how to design a lightweight, easy-to-deploy and manage message queue technology with efficient resource scheduling capabilities and high concurrency performance has become a technical problem that urgently needs to be solved in this field. Summary of the Invention
[0009] To address the problems of existing technologies and achieve lightweight, easy-to-deploy, high-concurrency, and efficient resource scheduling capabilities, this application provides a message queue processing method, system, and medium.
[0010] Firstly, a message queue processing method is provided, including the following steps:
[0011] Message persistence: Receive messages sent by the producer, store the message entities in the message pool, and assign a globally unique identifier to each message entity;
[0012] Index building: Based on the message priority, add the message's globally unique identifier to the index pointer of the corresponding priority queue;
[0013] Priority scheduling and consumption: According to the preset priority strategy, the globally unique identifier of the message to be processed is obtained from the index pointer, the complete message entity is read from the message pool, and the message entity is distributed to the consumer according to the set consumption rate; the priority strategy means that the high priority index queue is checked first. If it is not empty, the message pointer is taken out from it; if it is empty, the second priority index queue is checked, and so on.
[0014] Dynamic scaling: Real-time monitoring of message queue operation status indicators, and dynamic adjustment of the scale of processing threads and processes in the scheduling and consumption modules according to a preset elastic scaling strategy; the elastic scaling strategy refers to triggering an expansion operation when the message backlog of high-priority queues exceeds the backlog threshold for a continuous preset period of time, and the consumption delay exceeds the delay threshold, thereby doubling the number of processing threads and processes in the scheduling and consumption modules; when the message backlog of all queues remains below the shrinkage threshold, a shrinkage operation is triggered, thereby reducing the number of processing threads and processes in the scheduling and consumption modules.
[0015] Furthermore, when storing message entities in the message pool, they are stored in the log file in an append-only manner, which is sequential input / output.
[0016] Furthermore, when adding the message ID to the index pointer of the corresponding priority queue according to the message priority, the specific steps are as follows:
[0017] Each message priority index queue maintains an independent queue index; the index queues are implemented using a doubly linked list; the tails of adjacent priority index queues are connected to their heads; the message ID is added to the index pointer of the corresponding priority queue; when a message is processed by consumers of different priorities, pointers to the message are added to the index queues of different priorities respectively, but the message entity is stored only once in the message pool.
[0018] Furthermore, the elastic scaling strategy refers to triggering a scaling operation when the message backlog of a high-priority queue exceeds the backlog threshold of 1,000 messages within one minute and the consumption latency exceeds 500ms, thereby doubling the number of processing threads and processes in the scheduling and consumption module; and triggering a scaling operation when the message backlog of all queues remains below 100 messages, thereby reducing the number of processing threads and processes in the scheduling and consumption module.
[0019] Furthermore, the globally unique identifier is a message ID, which is a universally unique identifier (UUID) or an incrementing offset.
[0020] Secondly, a message queue processing system is provided to implement the message queue processing method described in the first aspect, including a message persistence module, an index management module, a scheduling and consumption module, and a monitoring and scaling module, as detailed below:
[0021] The message persistence module is used to receive messages sent by the producer, store message entities in the message pool, and assign globally unique identifiers to message entities.
[0022] The index management module, connected to the message persistence module, is used to add the globally unique identifier of a message to the index pointer of the corresponding priority queue according to the message's priority.
[0023] The scheduling and consumption module is connected to the index management module. It obtains the globally unique identifier of the message to be processed from the index pointer, reads the complete message entity from the message pool, and distributes the message entity to the consumer according to the set consumption rate.
[0024] The monitoring and scaling module connects to the scheduling and consumption module, monitors the message queue's operational status metrics in real time, and dynamically adjusts the scale of processing threads and processes in the scheduling and consumption module according to a preset elastic scaling strategy.
[0025] Thirdly, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the message queue processing methods described in the first aspect.
[0026] The beneficial effects of this invention are as follows:
[0027] The message queue processing method of this invention, through the construction of an innovative "message pool + multi-level index" model, requires only one instance of the same message that needs to be referenced by multiple consumers or multiple priority queues. The write operation occurs in the append-only message pool, and the creation of the index is a lightweight memory operation, thereby significantly reducing storage overhead and write latency, especially in high-concurrency write scenarios. Simultaneously, a priority scheduling mechanism ensures fine-grained control of computing resources when efficiently processing large messages; and an elastic scaling strategy is introduced, enabling the system to adaptively handle traffic fluctuations based on monitoring metrics, resulting in a message queue system with a simple architecture, easy deployment, and strong robustness.
[0028] By combining priority queues with a fixed-rate consumption mechanism, large message volumes of hundreds of megabytes can be effectively processed. The fixed-rate consumption mechanism allocates a fixed processing time slice or bandwidth to large messages, preventing them from occupying computing resources for extended periods, thus protecting the overall system responsiveness and preventing smaller messages from "starving" due to the processing of large messages. The combination of priority queues and fixed-rate consumption mechanism supports large message volumes and enables fine-grained resource scheduling.
[0029] By monitoring key metrics such as message backlog and consumption latency, the system can automatically detect load changes and dynamically adjust backend processing capacity, achieving automated elastic scaling. It automatically expands during peak traffic periods to maintain normal message queue operation and automatically shrinks during off-peak traffic periods to conserve resources. This gives the system adaptive capabilities, improving its robustness and operational efficiency.
[0030] This invention eliminates reliance on heavy external components such as ZooKeeper, resulting in a simpler architecture and significantly reduced deployment and maintenance complexity. Combined with built-in monitoring and automation capabilities, the entire system is easier to manage and use. The lightweight architecture facilitates deployment and management. Attached Figure Description
[0031] Figure 1 This is a system architecture diagram of an embodiment of the present invention.
[0032] Figure 2 This is an overall flowchart of the message queue processing method according to an embodiment of the present invention.
[0033] Figure 3 This is a schematic diagram of the "message pool + multi-level index" data structure in an embodiment of the present invention.
[0034] Figure 4 This is a processing logic diagram of the message queue processing method according to an embodiment of the present invention. Detailed Implementation
[0035] The present invention will now be described in further detail.
[0036] This invention provides a message queue system, such as Figure 1 As shown, it includes a message persistence module, an index management module, a scheduling and consumption module, and a monitoring and scaling module.
[0037] The message persistence module receives messages sent by producers, stores the message entities in a message pool, and assigns globally unique identifiers to each message entity. Messages sent by producers include business data, priority tags (such as HIGH, MEDIUM, LOW), and other metadata. The message persistence module stores the complete message entity content in one or more log files in an append-only manner, forming a unified "message pool." All messages, regardless of their priority or the number of times they are consumed, have only one copy of their entity content stored in this module. To uniquely identify each message, the message persistence module generates a globally unique message ID (e.g., an incrementing offset or a universally unique identifier, UUID). This append-only writing method is sequential input / output, resulting in high write performance.
[0038] The index management module, connected to the message persistence module, is used to create and maintain independent index pointers for message queues of different priorities based on message priority. The index pointers are lightweight, containing only references to the corresponding message entities in the message pool. Specifically, as follows: Figure 3 As shown, the message pool stores message entity data, such as Msg1, Msg2, Msg3, Msg4, etc. The index management module maintains multiple independent queue indexes based on message priority, such as priority 1 index queue, priority 2 index queue, priority 3 index queue, etc. The index queues are implemented using doubly linked lists, and the tails of adjacent priority index queues are connected to their respective heads; for example, the tail element of the priority 1 index queue is connected to the head element of the priority 2 index queue. These indexes do not store complete messages, but rather lightweight pointers. For example, when a priority 1 message Msg1 is written to the message pool, the index management module adds a pointer to message Msg1 (e.g., Pt11) to the priority 1 index queue. If message Msg2 needs to be processed by both priority 1 and priority 2 consumers, the index management module adds pointers to message Msg2 to both the priority 1 and priority 2 index queues, while the message entity Msg2 is stored only once in the message pool. This design greatly saves storage space and makes indexing operations (enqueue / dequeue) efficient memory operations.
[0039] The scheduling and consumption module, connected to the index management module, retrieves messages from index pointers according to a priority strategy and distributes them to consumers for processing at a preset consumption rate. The priority strategy involves first checking the high-priority index queue; if it's not empty, a message pointer is retrieved from it; if empty, the next-highest priority index queue is checked, and so on. The scheduling and consumption module includes a priority scheduling unit and a fixed-rate consumption unit. The priority scheduling unit employs a strict priority scheduling algorithm, always checking the high-priority index queue first; if it's not empty, a message pointer is retrieved from it; if empty, the next-highest priority index queue is checked, and so on. This ensures that high-priority tasks are always executed first. The fixed-rate consumption unit, after the priority scheduling unit obtains a message pointer, reads the complete message entity from the message pool. At this point, the fixed-rate consumption unit performs rate limiting processing based on the message size or a preset strategy. For example, for large messages exceeding 10MB, processing can be limited to within 1 second, consuming only fixed CPU and I / O resources during processing to prevent blocking the entire consumption thread. This achieves fine-grained protection of computing resources.
[0040] The monitoring and scaling module, connected to the scheduling and consumption module, monitors the operational status metrics of the message queues in real time and automatically adjusts the resource size of the backend work pool (the backend work pool refers to the processing threads and processes in the scheduling and consumption module) based on these metrics, achieving automated elastic scaling. The monitoring and scaling module continuously collects system operational metrics, such as the message backlog for each priority queue, the average latency from message enqueueing to consumption, production rate, and consumption rate. The monitoring and scaling module incorporates an elastic scaling strategy. This strategy triggers a scaling-up operation when the message backlog in a high-priority queue exceeds a backlog threshold for a consecutive preset duration, and the consumption latency exceeds a latency threshold. This doubles the number of processing threads and processes in the scheduling and consumption module. Conversely, it triggers a scaling-down operation when the message backlog in all queues remains below a shrinking threshold, reducing the number of processing threads and processes in the scheduling and consumption module to conserve resources. For example, the preset duration is 1 minute, the backlog threshold is 1000 messages, the latency threshold is 500ms, and the shrinking threshold is 100 messages. This automation mechanism enables the system to handle traffic surges with ease and release resources during idle periods.
[0041] The present invention also provides a message queue processing method, such as... Figure 2 As shown, it includes the following steps:
[0042] Step 1: Message Persistence: Receive messages sent by the producer, store the message entity in a unified message pool, and assign a globally unique identifier to each message entity. Details are as follows:
[0043] The message persistence module receives messages sent by producers. These messages can contain business data, priority tags (such as HIGH, MEDIUM, LOW), and other metadata. The complete message content is stored in one or more log files via append-only writing, forming a unified "message pool." To uniquely identify each message, a globally unique message ID (an incrementing offset or a universally unique identifier, UUID) is generated for each message. The append-only writing method is sequential input / output, resulting in high write performance.
[0044] Step 2, Index Construction: Based on the message's priority, add the message's unique identifier to the index pointer of the corresponding priority queue. Specifically:
[0045] The message pool stores the message entity data. Multiple independent queue indices are maintained based on message priority. These index queues are implemented using doubly linked lists, where the tail of an adjacent priority index queue is linked to its head; for example, the tail element of the priority 1 index queue is linked to the head element of the priority 2 index queue. These indices do not store the complete message but are lightweight pointers. When a message is processed by consumers of different priorities, a pointer to the message is added to the corresponding priority index queue, but the message entity is stored only once in the message pool. This saves storage space and makes index operations (enqueue / dequeue) efficient memory operations.
[0046] For example, the message pool stores message entity data such as Msg1, Msg2, Msg3, and Msg4. Based on message priority, multiple independent queue indices are maintained, namely, priority 1 index queue, priority 2 index queue, priority 3 index queue, etc. The index queues are implemented using doubly linked lists, and the tails of adjacent priority index queues are connected to their respective heads. For example, the tail element of the priority 1 index queue is connected to the head element of the priority 2 index queue; the tail element of the priority 2 index queue is connected to the head element of the priority 3 index queue, and so on. These indices do not store complete messages, but rather lightweight pointers. For example, when a priority 1 message Msg1 is written to the message pool, the multi-level index management module adds a pointer to message Msg1 (e.g., Pt11) to the priority 1 index queue. If message Msg2 needs to be processed by both priority 1 and priority 2 consumers, then pointers to message Msg2 are added to both the priority 1 and priority 2 index queues, while the message entity Msg2 is stored only once in the message pool.
[0047] Step 3: Priority Scheduling and Consumption: Based on the preset priority strategy, the unique identifier of the message to be processed is obtained from the index pointer. Then, the complete message entity is read from the message pool and distributed to consumers according to the set consumption rate. Details are as follows:
[0048] The scheduler employs a strict priority scheduling algorithm, always checking the high-priority index queue first. If the high-priority index queue is not empty, the message pointer is retrieved from it; if the high-priority index queue is empty, the next-highest priority index queue is checked, and so on. This ensures that high-priority tasks are always executed first.
[0049] After obtaining the message pointer, the complete message entity is read from the message pool. At this point, based on the message size or a preset strategy, it is rate-limited. For example, for large messages exceeding 10MB, processing can be limited to within 1 second, consuming only fixed CPU and I / O resources during processing to prevent blocking the entire consumer thread. This achieves fine-grained protection of computing resources.
[0050] Step 4: Dynamic Scaling: Monitor the message queue's operational status metrics in real time and dynamically adjust the size of the backend worker pool used for message processing according to the preset elastic scaling strategy. Details are as follows:
[0051] Continuously collect system performance metrics, such as message backlog for each priority queue, average latency from message enqueueing to consumption, production rate, and consumption rate. Determine if the system load is too high: If the message backlog in a high-priority queue exceeds the backlog threshold for a consecutive preset duration, and the consumption latency exceeds the latency threshold, it is considered overloaded, triggering a scaling operation (scaling up the backend worker pool) to double the number of processing threads and processes in the scheduling and consumption modules. If the message backlog in all queues remains below the shrinking threshold, it is considered that resources are too idle, triggering a scaling operation (shrinking the backend worker pool) to reduce the number of processing threads and processes in the scheduling and consumption modules to conserve resources. If the message backlog in a high-priority queue does not exceed the backlog threshold for a consecutive preset duration, or the consumption latency does not exceed the latency threshold, and the message backlog in all queues does not remain below the shrinking threshold, maintain the current resource configuration.
[0052] For example, the preset duration is 1 minute, the backlog threshold is 1000 entries, the delay threshold is 500ms, and the shrinkage threshold is 100 entries. This automated mechanism enables the system to handle traffic surges with ease and release resources during idle periods.
[0053] To enable those skilled in the art to better understand the present invention, the above method will be described in detail below with reference to specific examples.
[0054] Step one is implemented as follows: The producer sends a message containing business data and a priority of 1. After receiving the message, the message persistence module appends its content to the end of the message pool's log file and assigns it a globally unique ID, such as Offset=123.
[0055] The specific implementation of step two is as follows: the priority of the message is parsed as 1, so the pointer ID=123 is added to the tail of the priority 1 index queue.
[0056] Step three is implemented as follows: The priority scheduling unit of the scheduling and consumption module checks that the high-priority index is not empty and retrieves ID=123 from it. Then, based on ID=123, it locates and reads the complete message entity from the message pool. The fixed-rate consumption unit evaluates the message body size as 5MB, which is not considered a large message, and pushes it to the bound consumer at the normal rate.
[0057] Step four is implemented as follows: Throughout the process, the monitoring and scaling module runs continuously. Assume that a sudden surge of high-priority messages causes the message backlog in the priority 1 index queue to rapidly increase to 2000 messages. The monitoring and scaling module detects that this metric exceeds the preset expansion threshold of 1000 messages and immediately executes an expansion command, increasing the number of threads in the backend worker pool from 4 to 8, thereby improving the overall consumption rate and alleviating the backlog.
[0058] The processing logic of the message queue processing method of the present invention is as follows: Figure 4 As shown: The message persistence module receives messages, stores message entities in the message pool, and assigns globally unique identifiers; the index management module adds the globally unique identifiers to the corresponding priority indexes according to message priority; the scheduling and consumption module obtains the indexes according to the priority strategy and distributes messages at a fixed rate. The monitoring and scaling module determines whether the system load is too high: if the load is too high, it triggers the expansion of the backend worker pool; if the load is within the normal range, it determines whether there is too much idle resources. If there is too much idle resources, it triggers the reduction of the backend worker pool; if the idle resources are within the normal range, it keeps the current resource configuration unchanged.
[0059] The message queue processing method of this invention, through the construction of an innovative "message pool + multi-level index" model, requires only one instance of the same message that needs to be referenced by multiple consumers or multiple priority queues. The write operation occurs in the append-only message pool, and the creation of the index is a lightweight memory operation, thereby significantly reducing storage overhead and write latency, especially in high-concurrency write scenarios. Simultaneously, a priority scheduling mechanism ensures fine-grained control of computing resources when efficiently processing large messages; and an elastic scaling strategy is introduced, enabling the system to adaptively handle traffic fluctuations based on monitoring metrics, resulting in a message queue system with a simple architecture, easy deployment, and strong robustness.
[0060] By combining priority queues with a fixed-rate consumption mechanism, large message volumes of hundreds of megabytes can be effectively processed. The fixed-rate consumption mechanism allocates a fixed processing time slice or bandwidth to large messages, preventing them from occupying computing resources for extended periods, thus protecting the overall system responsiveness and preventing smaller messages from "starving" due to the processing of large messages. The combination of priority queues and fixed-rate consumption mechanism supports large message volumes and enables fine-grained resource scheduling.
[0061] By monitoring key metrics such as message backlog and consumption latency, the system can automatically detect load changes and dynamically adjust backend processing capacity, achieving automated elastic scaling. It automatically expands during peak traffic periods to maintain normal message queue operation and automatically shrinks during off-peak traffic periods to conserve resources. This gives the system adaptive capabilities, improving its robustness and operational efficiency.
[0062] This invention eliminates reliance on heavy external components such as ZooKeeper, resulting in a simpler architecture and significantly reduced deployment and maintenance complexity. Combined with built-in monitoring and automation capabilities, the entire system is easier to manage and use. The lightweight architecture facilitates deployment and management.
[0063] In some embodiments, certain aspects of the above-described techniques may be implemented by one or more processors of a processing system executing software. The software includes one or more sets of executable instructions stored or otherwise tangibly implemented on a non-transitory computer-readable storage medium. The software may include instructions and certain data that, when executed by one or more processors, manipulate one or more processors to perform one or more aspects of the above-described techniques. The non-transitory computer-readable storage medium may include, for example, magnetic or optical disk storage devices, solid-state storage devices such as flash memory, cache, random access memory (RAM), or other non-volatile memory devices. The executable instructions stored on the non-transitory computer-readable storage medium may be source code, assembly language code, object code, or other instruction formats interpreted or otherwise executed by one or more processors.
[0064] Computer-readable storage media can include any storage medium or combination of storage media that can be accessed by a computer system during use to provide instructions and / or data to the computer system. Such storage media can include, but are not limited to, optical media (e.g., optical discs (CDs), digital versatile optical discs (DVDs), Blu-ray discs), magnetic media (e.g., floppy disks, magnetic tapes, or magnetic hard disks), volatile memory (e.g., random access memory (RAM) or cache), non-volatile memory (e.g., read-only memory (ROM) or flash memory), or microelectromechanical systems (MEMS) based storage media. Computer-readable storage media can be embedded in a computing system (e.g., system RAM or ROM), fixedly attached to a computing system (e.g., a magnetic hard disk drive), removably attached to a computing system (e.g., an optical disc or universal serial bus-based (USB) flash memory), or coupled to a computer system via a wired or wireless network (e.g., network-accessible storage (NAS)).
[0065] While the present invention has been disclosed above with reference to preferred embodiments, these embodiments are not intended to limit the invention. Any equivalent changes or modifications made without departing from the spirit and scope of the invention are also within the scope of protection of the invention. Therefore, the scope of protection of the present invention should be determined by the claims of this application.
Claims
1. A message queue processing method, characterized in that, The steps include the following: Message persistence: Receive messages sent by the producer, store the message entities in the message pool, and assign a globally unique identifier to each message entity; Index building: Based on the message priority, add the message's globally unique identifier to the index pointer of the corresponding priority queue; Priority scheduling and consumption: According to the preset priority strategy, the globally unique identifier of the message to be processed is obtained from the index pointer, the complete message entity is read from the message pool, and the message entity is distributed to the consumer according to the set consumption rate; the priority strategy refers to first checking the high-priority index queue, and if it is not empty, then taking the message pointer from it. If it is empty, check the next priority index queue, and so on. Dynamic scaling: Real-time monitoring of message queue operation status indicators, and dynamic adjustment of the scale of processing threads and processes in the scheduling and consumption modules according to a preset elastic scaling strategy; the elastic scaling strategy refers to triggering an expansion operation when the message backlog of high-priority queues exceeds the backlog threshold for a continuous preset period of time, and the consumption delay exceeds the delay threshold, thereby doubling the number of processing threads and processes in the scheduling and consumption modules; when the message backlog of all queues remains below the shrinkage threshold, a shrinkage operation is triggered, thereby reducing the number of processing threads and processes in the scheduling and consumption modules.
2. The message queue processing method according to claim 1, characterized in that, When storing message entities in the message pool, they are stored in the log file in an append-only manner, which is sequential input / output.
3. The message queue processing method according to claim 1, characterized in that, When adding a message ID to the index pointer of the corresponding priority queue based on message priority, the specific steps are as follows: Each message priority index queue maintains an independent queue index; the index queues are implemented using a doubly linked list; the tails of adjacent priority index queues are connected to their heads; the message ID is added to the index pointer of the corresponding priority queue; when a message is processed by consumers of different priorities, pointers to the message are added to the index queues of different priorities respectively, but the message entity is stored only once in the message pool.
4. The message queue processing method according to claim 1, characterized in that, The elastic scaling strategy refers to the following: when the message backlog of a high-priority queue exceeds the backlog threshold of 1,000 messages within one minute and the consumption latency exceeds 500ms, a scaling operation is triggered, doubling the number of processing threads and processes in the scheduling and consumption module; when the message backlog of all queues remains below 100 messages, a scaling operation is triggered, reducing the number of processing threads and processes in the scheduling and consumption module.
5. The message queue processing method according to claim 1, characterized in that, The globally unique identifier is the message ID, which is either a universally unique identifier (UUID) or an incrementing offset.
6. A message queue processing system for implementing the message queue processing method as described in claim 1, characterized in that, It includes a message persistence module, an index management module, a scheduling and consumption module, and a monitoring and scaling module, as detailed below: The message persistence module is used to receive messages sent by the producer, store message entities in the message pool, and assign globally unique identifiers to message entities. The index management module, connected to the message persistence module, is used to add the globally unique identifier of a message to the index pointer of the corresponding priority queue according to the message's priority. The scheduling and consumption module is connected to the index management module. It obtains the globally unique identifier of the message to be processed from the index pointer, reads the complete message entity from the message pool, and distributes the message entity to the consumer according to the set consumption rate. The monitoring and scaling module connects to the scheduling and consumption module, monitors the message queue's operational status metrics in real time, and dynamically adjusts the scale of processing threads and processes in the scheduling and consumption module according to a preset elastic scaling strategy.
7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the message queue processing method as described in any one of claims 1-5.