Hybrid queue system and method for optimizing message processing flow

Abstract

This invention discloses a hybrid queue system and method for optimizing message processing, aiming to solve the problems of unbalanced resource allocation, rigid load adaptation, easy message loss, and slow fault recovery in existing hybrid queues. The system includes a partitioned memory queue, a bucketed file queue, a multi-dimensional intelligent scheduling module, an L1-L2 hybrid index, a dual-backup fault recovery module, and a timestamp-confirmed dequeue module. The method achieves this through a closed loop of message writing, intelligent migration, priority reading, timestamp confirmation, and fault recovery. This invention improves memory utilization, reduces bucketed file IOPS, ensures zero message loss, improves fault recovery efficiency, and is suitable for high-concurrency, million-level small queues, and highly reliable scenarios such as financial transactions.

Description

Technical Field

[0001] This invention belongs to the field of message processing technology, specifically relating to a hybrid queue system and method for optimizing message processing flow. Background Technology

[0002] In message processing systems, message queues serve as a core component, enabling asynchronous message transmission, buffering, and decoupling. Their performance and reliability directly impact the overall system efficiency. Existing message queue technologies are mainly divided into two categories: memory queues and file queues. Memory queues offer the advantage of fast read and write speeds, but are limited by memory capacity, cannot store large numbers of messages, and are susceptible to message loss. File queues have unlimited capacity and support message persistence, but have lower read and write performance, and are prone to queue blocking issues, especially in high-concurrency scenarios.

[0003] To balance capacity and performance, existing technologies have proposed hybrid queue schemes using both memory and file systems. For example, some schemes employ a double-buffered model using both memory and disk files. When the number of messages in memory exceeds a specified limit, some messages are saved to a file, while the header and tail messages are retained in memory to improve performance. Although this scheme achieves capacity expansion, its fixed-threshold migration strategy cannot adapt to dynamically changing system loads and message characteristics, easily leading to wasted memory resources or a surge in I / O pressure.

[0004] To address queue blocking issues in high-concurrency scenarios, related technologies employ multiple operation type queues with dynamically adjusted priorities, combined with message dependency detection to achieve ordered scheduling. However, this approach does not optimize the memory-file storage collaboration mechanism, and issues such as low index query efficiency and unordered message migration persist in scenarios with massive message volumes.

[0005] In scenarios with millions of small queues, existing solutions employ a multi-queue merging and shared block storage approach, combined with multi-level indexes to improve read and write performance. However, this approach suffers from complex index structure design, lacks collaborative optimization with in-memory queues, and relies on complete index reconstruction for fault recovery, resulting in low recovery efficiency and susceptibility to data consistency issues. Furthermore, existing hybrid queue solutions generally suffer from rigid message migration strategies, high index lookup overhead, and inadequate fault recovery mechanisms, making it difficult to meet the high-performance and high-reliability requirements of message processing in complex scenarios. Summary of the Invention

[0006] To address the problems of rigid migration strategies, low index query efficiency, and weak fault recovery capabilities in existing hybrid queue technologies, this invention provides a hybrid queue system and method that optimizes the message processing flow, achieving dynamic adaptation of message storage, efficient and accurate index query, and fast and reliable fault recovery, thereby improving the overall performance and stability of the message processing flow.

[0007] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: A hybrid queue system for optimizing message processing includes: Memory queue module: used to cache messages to be processed. The memory queue module is divided into a hot message area and a transition message area. The hot message area stores messages whose access frequency is higher than a threshold within a preset time window, and the transition message area stores newly received messages or messages to be migrated whose access frequency is lower than the threshold. The total memory usage and message status of the two areas are monitored in real time to provide a trigger basis for batch migration. This partitioning design can dynamically adjust the area size according to message access characteristics to improve memory resource utilization.

[0008] File queue module: Divide all file queues into multiple buckets, each bucket contains several queue groups, and each queue group manages a fixed number of queues as the smallest read and write unit; Each queue group is allocated an independent memory block to temporarily cache messages. When the amount of data in the memory block reaches a preset threshold, the messages are sorted by queue and compressed and written to disk to form file blocks. The file block header embeds a file block index to record the queue ownership, check code and storage information. Intelligent scheduling module: Based on message characteristics and system load, the migration strategy is dynamically adjusted, including: when the memory usage exceeds the first threshold, low-frequency messages in the transition message area are migrated to the bucketed file queue first; when the access frequency of a message in the file queue exceeds the set value continuously, it is migrated back to the hot message area; high-priority messages are kept in memory first, and when migration is required, they are stored in the high-priority dedicated bucket of the file queue. Hybrid Index Module: Constructs an index system that combines L1-L2 two-level memory indexes with file block indexes. The L1 index stores the starting information of multiple L2 indexes in chronological order, and the L2 index records the message offset, quantity, and compression information of each queue within each file block. Fault recovery module: It adopts a dual backup mechanism of memory snapshots and file logs. The memory queue periodically generates incremental snapshots, and the file queue records message operation logs. In case of failure, data recovery is performed based on snapshots and logs. Timestamp confirmation dequeue module: Generates and binds a unique millisecond-level timestamp for each message; after the consumer reads the message, the message is temporarily stored in the original queue, and the message is only removed from the queue when the consumer returns the corresponding timestamp confirmation signal, ensuring that the message is officially dequeued after processing.

[0009] Further optimizations include setting the access frequency threshold for the hot message area of ​​the memory queue module to 5-20 times per hour, and setting the preset time window to 0.5-2 hours; setting a message storage timeout for the transition message area, and automatically marking messages that have not been accessed within the timeout period as priority migration objects.

[0010] Further optimization includes the following migration strategy for the intelligent scheduling module: When the memory usage exceeds the first preset threshold, the messages with the lowest access frequency in the transition message area are migrated to the file queue module first. During the migration process, batch sorting is used to ensure the storage continuity of messages in a single queue. When the message access frequency of a certain queue in the file queue module exceeds a set value for a consecutive preset number of times, some messages of that queue will be migrated back to the hot message area of ​​the memory queue. The migration order is dynamically adjusted based on message priority. High-priority messages are retained in the memory queue module first, and if they need to be migrated, they are stored in the high-priority dedicated file block group of the file queue module.

[0011] Further optimization includes the following index building process for the hybrid index module: The L1 index is a first-level sparse index. Every 16 consecutive L2 indices correspond to one L1 index entry. The L1 index entry records the file ID to which the corresponding L2 index belongs and the starting sequence number of each queue message. The L2 index is a second-level dense index that corresponds one-to-one with each file block in the file queue module. It records the storage offset, compressed size, original size, and the offset and number of messages of each queue within the file block relative to the start position of the file block. The file block index is stored in the header of the corresponding file block and includes the file block's creation time, queue ownership information, and checksum, which is used for quick location and integrity verification of the file block.

[0012] This indexing system balances the query speed of in-memory indexes with the storage stability of file indexes. By using binary search of L1 indexes and sequential traversal of L2 indexes, the target message can be located quickly, significantly reducing the index query overhead.

[0013] Further optimization is made to the recovery process of the fault recovery module, which includes: When the system starts, it first loads the most recent memory snapshot into the memory queue module and restores the message state of the hot message area and the transition message area; Based on the file log, the message operations after the snapshot are replayed, and the message status of the memory queue module and the file queue module is updated. During the log replay process, operations that have been persisted are skipped. The recovered messages are indexed and reconstructed. The integrity of the file queue messages is verified by the file block index, and the consistency of the message association between the memory queue and the file queue is verified by the L1-L2 index.

[0014] A dual backup mechanism of memory snapshots and file logs is employed to ensure no data loss and efficient recovery. The memory queue periodically generates incremental snapshots, recording only message changes since the last snapshot, reducing snapshot storage overhead; the file queue records message operation logs, including message writing, migration, reading, deletion, and other operation information, ensuring operation traceability.

[0015] Further optimization involves the file queue module employing a bucket merging management mechanism, including: All queues are divided into multiple buckets using a hash algorithm. Each bucket contains multiple queue groups. Each queue group is the smallest read / write unit and manages a fixed number of queues. Each queue group is allocated an independent memory block for temporary message storage. When the amount of data in the memory block reaches a preset threshold, the messages in the memory block are sorted by queue and then compressed and written to disk to form a new file block. File block writing to disk adopts an asynchronous batch flushing method. The memory blocks to be written to disk are received through RingBuffer, and AIO technology is used to reduce the number of IO copies and improve the writing efficiency.

[0016] Further optimization involves using a hash algorithm to map the queue to hash buckets, specifically: 1) Queue identifier preprocessing: Convert the queue identifier to a UTF-8 encoded string, remove invalid characters (spaces, tabs) at the beginning and end, and then use the MD5 algorithm to generate a 128-bit fixed-length digest value; 2) Hash value calculation: The MurmurHash3-64 algorithm is used to perform hash operation on the 128-bit digest value to generate a 64-bit unsigned integer hash value. The seed value of the algorithm is configured as 0x12345678 (which can be dynamically adjusted through the system interface). 3) Bucket mapping logic: Take the modulo of the 64-bit hash value with the total number of hash buckets (512-2048 configurable) to obtain the bucket index, and map the queue to the corresponding hash bucket; if the total number of hash buckets is adjusted, update the mapping relationship by matching the range of the modulo result; 4) Conflict handling: When multiple queues are mapped to the same bucket index, a chained hash structure is used to store the queues in the queue linked list in the bucket according to the mapping order. Each linked list node is associated with the unique identifier of the queue and the ID of the queue group to which it belongs. 5) Dynamic scaling and adaptation: When the total number of hash buckets increases, new virtual nodes are added (the number is 10 times the number of new buckets), a hash value is generated for each virtual node and associated with a physical bucket; only the queue data whose virtual node mapping has changed is migrated, and the original bucket message reading and writing is kept uninterrupted during the migration process through "double write buffer".

[0017] A processing method for a hybrid queue system based on the above-mentioned optimized message processing flow includes the following steps: Step S1, Message Writing: Receive messages to be processed, store them in the corresponding area of ​​the memory queue module according to the access frequency and priority of the messages, and update the relevant index information of the hybrid index module at the same time. Step S2, Intelligent Migration: Monitor system load and message status in real time, and realize message migration between memory queue and file queue based on preset migration strategy, maintaining the FIFO order of messages and the continuity of queue storage during the migration process; Step S3, Message Reading: Receive a message reading request, quickly locate the message storage location through the hybrid index module, prioritize reading the message from the memory queue module, and if the message is stored in the file queue module, locate it through the index and read it, and determine whether to backtrack based on the access situation; Step S4, Timestamp Confirmation: After processing the message, the consumer returns a timestamp confirmation signal. After confirming the validity of the dequeue module's verification signal (timestamp existence, message not dequeue status), the consumer controls the removal of the corresponding message from the storage queue. Step S5, Fault Recovery: When a system failure occurs, the fault recovery module loads the most recent incremental snapshot to restore the partition memory queue state, replays the non-persistent operations after the snapshot through the operation log, updates the partition memory queue and bucket file queue state, rebuilds the hybrid index, verifies the integrity of file messages through the file block index, and verifies the consistency of memory and file message association through the L1-L2 index to ensure the continuation of the processing flow.

[0018] In a further optimization, the message writing step also includes classifying and identifying the message. The identification information includes message ID, generation time, initial access frequency, priority level, and queue identifier. The identification information is stored together with the message data and serves as the basic data for index construction.

[0019] Further optimization, step S2 intelligent migration specifically includes: Step S2.1, Multi-dimensional parameter acquisition unit: Real-time acquisition of message feature parameters and system load parameters. The message feature parameters include message access frequency counted by a 5-minute sliding window, message priority quantified into 3 levels, and message size divided into 3 levels. The system load parameters include real-time calculated memory usage, system-level CPU average utilization, and IOPS of the disk where the bucketed file queue is located. Step S2.2, Hierarchical Strategy Decision Unit: Employs a three-layer decision logic to output scheduling instructions, specifically including: Step S2.2.1, System load threshold judgment: Based on the initial memory utilization threshold T0, when CPU utilization is >80%, T0 is increased by 10%; when IOPS is >10000, T0 is decreased by 10%; when memory utilization is ≥ adjusted T0, "memory → file" migration is triggered; when the file queue message access frequency is ≥ hotspot threshold for 3-5 consecutive times, "file → memory" migration is triggered. Step S2.2.2, message value sorting: The message value is quantified by the scoring formula Score=(access frequency×0.4)+(priority×0.3)+(1 / size range×0.3). During migration, messages with a Score≤2 in the transition message area are selected, and messages with a Score≥5 in the file queue are selected during relocation. Step S2.2.3, Action Execution Decision: The migration batch is calculated as "(current memory occupancy rate - T0) × total memory capacity / average size of a single message", and the return batch is ≤ 80% of the remaining capacity of the hot message area. Both migration and return are sorted in ascending order of message timestamp. Step S2.3, Dynamic Execution and Status Synchronization Unit: Performs migration / reverse operations according to scheduling instructions. During migration, low-value messages are written in batches to the corresponding hash buckets in the bucketed file queue, and messages in the memory transition area are deleted synchronously. During reverse, high-value messages are written in batches to the memory hot spot area, and the corresponding messages in the file are marked as invalid synchronously. After the operation is completed, the status information of the hybrid index module and the fault recovery module is updated. Step S2.4, Exception Handling and Load Balancing Unit: Record the interruption sequence number and restart execution when migration / return is interrupted; use the average of the first 5 samples as a substitute when parameter acquisition fails; monitor hash bucket IOPS in real time; if the IOPS of a bucket is >5000 for 3 consecutive minutes, migrate queues with a score ≤2 in the bucket to an idle bucket; when the utilization rate of the hot message area is >90%, increase its memory share and migrate high-value messages to the transition area in advance.

[0020] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. Intelligent migration strategy: Adopting a dynamic migration strategy based on multi-dimensional parameters, the migration threshold and order are dynamically adjusted in combination with message characteristics and system load, avoiding resource waste or IO pressure surge caused by fixed threshold migration, and improving the system's adaptability to dynamic load.

[0021] 2. Improved Index Query Efficiency: A hybrid index system combining L1-L2 secondary memory indexes and file block indexes is constructed. Through the collaborative work of sparse and dense indexes, the index query overhead is significantly reduced, enabling rapid location of memory and file messages, with particularly significant advantages in scenarios with massive amounts of messages.

[0022] 3. Reliable Fault Recovery: The system employs a dual backup mechanism of memory snapshots and file logs. Incremental snapshots reduce storage overhead, while log replay ensures operational traceability, enabling rapid message recovery and data consistency verification, thereby improving the system's reliability and fault tolerance.

[0023] 4. Storage structure optimization: The file queue adopts a bucket merging management mechanism, which avoids the IOPS bottleneck caused by independent storage of a single small queue, and ensures the storage continuity of messages in a single queue. Combined with asynchronous batch disk flushing technology, it further improves file read and write performance. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the hybrid queue system architecture for optimizing message processing as described in this invention; Figure 2 This is a schematic diagram of the hybrid queue message processing flow for optimizing message processing as described in this invention. Detailed Implementation

[0025] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0026] In this embodiment, the system is deployed on a 4C8G ECS virtual machine, with Linux CentOS 7.9 as the operating system and EXT4 as the file system. The memory queue module is initially allocated 2G of memory (1G for hot message area and 1G for transition message area). The storage directory of the file queue module is set as an independent disk partition, supporting concurrent management of up to 1 million queues.

[0027] Module parameter configuration, such as Figure 1 As shown, it specifically includes: Memory queue module: The access frequency threshold for the hot message area is set to 10 times per hour, with a preset time window of 1 hour; the message storage timeout for the transition message area is set to 2 hours, and messages that are not accessed within the timeout period will trigger migration detection.

[0028] File queue module: It adopts a hash bucketing mechanism to divide all queues into 1024 buckets, each bucket contains 64 queue groups, and each queue group manages 16 queues; the queue group memory block threshold is configured to 16K, data compression adopts LZ4 algorithm, and the compression ratio is about 4:1; asynchronous disk flushing adopts RingBuffer caching, the buffer size is set to 1024 memory blocks, and the number of AIO thread pools is set to 8.

[0029] Intelligent scheduling module: The initial value of the first preset threshold for memory utilization is set to 70%, the threshold for CPU utilization is set to 80%, and the threshold for IOPS is set to 10,000; when the CPU utilization is higher than 80%, the first preset threshold for memory utilization is increased to 75%; when the IOPS is higher than 10,000, the first preset threshold for memory utilization is decreased to 65%; message priority is divided into 3 levels, and the migration delay coefficient for high-priority messages is set to 2, that is, migration is triggered only when the memory utilization reaches twice the adjusted threshold.

[0030] Hybrid index module: For every 16 L2 indexes, there is one index entry. The L1 index resides in memory, and the L2 index is managed using the LRU cache mechanism. The file block index size is set to 128 bytes, which includes file ID (4 bytes), creation time (8 bytes), queue ownership information (32 bytes), number of messages (4 bytes), checksum (16 bytes), and reserved field (64 bytes).

[0031] Fault recovery module: The memory snapshot generation cycle is set to 10 minutes, and incremental snapshot method is used; the file log adopts a rolling storage mechanism, the size of a single log file is set to 100M, and the log files of the most recent 7 days are retained; when replaying the log, the operations that have been persisted are skipped by message ID and operation timestamp.

[0032] like Figure 2 As shown, the queuing method flow of the hybrid queue system based on the above optimized message processing flow is as follows: Step S1, Message Writing: The client sends a message write request, which includes the queue identifier "queue_001", the message content "test_message", and the priority "high". After receiving the request, the system generates a message ID "msg_123456", a generation time "2025-12-06 10:00:00", and an initial access frequency of "1". Since the message priority is "high", it is stored in the hot message area of ​​the memory queue module. At the same time, the L2 index of the hybrid index module is updated, recording the offset of the message in the hot message area "0x1000", and the corresponding L1 index entry is updated.

[0033] Step S2, Intelligent Migration: After the system has been running for 1 hour, the memory usage reaches 70% (first preset threshold), and the intelligent scheduling module initiates migration detection. It detects that the message access frequency of queue "queue_002" in the transition message area is 2 times per hour (below the hotspot threshold) and the priority is "low". 100 messages in this queue are then used as batch migration targets. These 100 messages are sorted by message ID and compressed and written to the file block corresponding to the "bucket_001-group_002" queue group in the file queue module. After migration is complete, the L1-L2 index and file block index of the hybrid index module are updated, recording the message storage location in the file block.

[0034] Step S3, Message Reading: The client sends a message read request, which includes the queue identifier "queue_002" and the message ID "msg_123457". After parsing the request, the system uses a binary search of the L1 index to locate the corresponding L2 index range (file ID "file_005"). Then, it uses the L2 index to traverse sequentially to determine the message's offset "0x2000" in the file block. The system locates the file "file_005" using the file block index, reads the message content, and returns it to the client. After reading, the access frequency of the message is updated to "3". Since the backtracking threshold (10 times per hour) has not been reached, no backtracking operation is triggered.

[0035] Step S4, Fault Recovery: In the event of a sudden power outage, the fault recovery module is activated upon restart. First, the most recent memory snapshot (generated on 2025-12-06 09:50:00) is loaded to restore the message status of the hot and transitional message areas. Then, the file logs are read, and message operations from 2025-12-06 09:50:00 to the time of the fault (10:30:00) are replayed, totaling 1200 operations, skipping 800 already persisted operations. After replay, the L1-L2 indexes and file block indexes of the hybrid index module are rebuilt. The integrity of the file queue messages is verified using checksums, and the consistency between memory and file queue messages is confirmed using message sequence numbers. After recovery, the system initiates a self-check to confirm that the message processing flow can be resumed normally.

[0036] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A hybrid queue system for optimizing message processing flow, characterized in that, include: Memory queue module: used to cache messages to be processed. The memory queue module is divided into a hot message area and a transition message area. The hot message area stores messages whose access frequency is higher than a threshold within a preset time window, and the transition message area stores newly received messages or messages to be migrated whose access frequency is lower than a threshold. The total memory usage and the message status of the two areas are monitored in real time to provide a trigger basis for batch migration. File queue module: Divide all file queues into multiple buckets, each bucket contains several queue groups, and each queue group manages a fixed number of queues as the smallest read and write unit; Each queue group is allocated an independent memory block to temporarily cache messages. When the amount of data in the memory block reaches a preset threshold, the messages are sorted by queue and compressed and written to disk to form file blocks. The file block header embeds a file block index to record the queue ownership, check code and storage information. Intelligent scheduling module: Based on message characteristics and system load, the migration strategy is dynamically adjusted, including: when the memory usage exceeds the first threshold, low-frequency messages in the transition message area are migrated to the bucketed file queue first; when the access frequency of a message in the file queue exceeds the set value continuously, it is migrated back to the hot message area; high-priority messages are kept in memory first, and when migration is required, they are stored in the high-priority dedicated bucket of the file queue. Hybrid Index Module: Constructs an index system that combines L1-L2 two-level memory indexes with file block indexes. The L1 index stores the starting information of multiple L2 indexes in chronological order, and the L2 index records the message offset, quantity, and compression information of each queue within each file block. Fault recovery module: It adopts a dual backup mechanism of memory snapshots and file logs. The memory queue periodically generates incremental snapshots, and the file queue records message operation logs. In case of failure, data recovery is performed based on snapshots and logs. Timestamp confirmation dequeue module: Generates and binds a unique millisecond-level timestamp for each message; after the consumer reads the message, the message is temporarily stored in the original queue, and the message is only removed from the queue when the consumer returns the corresponding timestamp confirmation signal, ensuring that the message is officially dequeued after processing.

2. The system according to claim 1, characterized in that, The migration strategy of the intelligent scheduling module includes: When the memory usage exceeds the first preset threshold, the messages with the lowest access frequency in the transition message area are migrated to the file queue module first. During the migration process, batch sorting is used to ensure the storage continuity of messages in a single queue. When the message access frequency of a certain queue in the file queue module exceeds a set value for a consecutive preset number of times, some messages of that queue will be migrated back to the hot message area of ​​the memory queue. The migration order is dynamically adjusted based on message priority. High-priority messages are retained in the memory queue module first, and if they need to be migrated, they are stored in the high-priority dedicated file block group of the file queue module.

3. The system according to claim 1, characterized in that, The index building process of the hybrid index module includes: The L1 index is a first-level sparse index. Every 16 consecutive L2 indices correspond to one L1 index entry. The L1 index entry records the file ID to which the corresponding L2 index belongs and the starting sequence number of each queue message. The L2 index is a second-level dense index that corresponds one-to-one with each file block in the file queue module. It records the storage offset, compressed size, original size, and the offset and number of messages of each queue within the file block relative to the start position of the file block. The file block index is stored in the header of the corresponding file block and includes the file block's creation time, queue ownership information, and checksum, which is used for quick location and integrity verification of the file block.

4. The system according to claim 1, characterized in that, The recovery process of the fault recovery module includes: When the system starts, it first loads the most recent memory snapshot into the memory queue module and restores the message state of the hot message area and the transition message area; Based on the file log, the message operations after the snapshot are replayed, and the message status of the memory queue module and the file queue module is updated. During the log replay process, operations that have been persisted are skipped. The recovered messages are indexed and reconstructed. The integrity of the file queue messages is verified by the file block index, and the consistency of the message association between the memory queue and the file queue is verified by the L1-L2 index.

5. The system according to claim 1, characterized in that, The file queue module adopts a bucket merging management mechanism, including: All queues are divided into multiple buckets using a hash algorithm. Each bucket contains multiple queue groups. Each queue group is the smallest read / write unit and manages a fixed number of queues. Each queue group is allocated an independent memory block for temporary message storage. When the amount of data in the memory block reaches a preset threshold, the messages in the memory block are sorted by queue and then compressed and written to disk to form a new file block. File block writing to disk adopts an asynchronous batch flushing method. The memory blocks to be written to disk are received through RingBuffer, and AIO technology is used to reduce the number of IO copies and improve the writing efficiency.

6. The system according to claim 5, characterized in that, The mapping from queues to hash buckets is implemented using a hash algorithm, specifically as follows: 1) Queue identifier preprocessing: Convert the queue identifier to a UTF-8 encoded string, remove invalid characters at the beginning and end, and then use the MD5 algorithm to generate a 128-bit fixed-length digest value; 2) Hash value calculation: The MurmurHash3-64 algorithm is used to perform a hash operation on the 128-bit digest value to generate a 64-bit unsigned integer hash value; 3) Bucket mapping logic: Take the modulo of the 64-bit hash value with the total number of hash buckets to obtain the bucket index, and map the queue to the corresponding hash bucket; if the total number of hash buckets is adjusted, update the mapping relationship by matching the range of the modulo result; 4) Conflict handling: When multiple queues are mapped to the same bucket index, a chained hash structure is used to store the queues in the queue linked list in the bucket according to the mapping order. Each linked list node is associated with the unique identifier of the queue and the ID of the queue group to which it belongs. 5) Dynamic scaling and adaptation: When the total number of hash buckets increases, new virtual nodes are added, a hash value is generated for each virtual node and associated with a physical bucket; only the queue data whose virtual node mapping has changed is migrated, and the original bucket message reading and writing is kept uninterrupted during the migration process through "double write buffer".

7. A processing method for a hybrid queue system based on the optimized message processing flow according to any one of claims 1-6, characterized in that, Includes the following steps: Step S1, Message Writing: Receive messages to be processed, store them in the corresponding area of ​​the memory queue module according to the access frequency and priority of the messages, and update the relevant index information of the hybrid index module at the same time. Step S2, Intelligent Migration: Monitor system load and message status in real time, and realize message migration between memory queue and file queue based on preset migration strategy, maintaining the FIFO order of messages and the continuity of queue storage during the migration process; Step S3, Message Reading: Receive a message reading request, quickly locate the message storage location through the hybrid index module, prioritize reading the message from the memory queue module, and if the message is stored in the file queue module, locate it through the index and read it, and determine whether to backtrack based on the access situation; Step S4, Timestamp Confirmation: After processing the message, the consumer returns a timestamp confirmation signal. After confirming the validity of the signal by the dequeue module, the corresponding message is removed from the storage queue. Step S5, Fault Recovery: When a system failure occurs, the fault recovery module loads the most recent incremental snapshot to restore the partition memory queue state, replays the non-persistent operations after the snapshot through the operation log, updates the partition memory queue and bucket file queue state, rebuilds the hybrid index, verifies the integrity of file messages through the file block index, and verifies the consistency of memory and file message association through the L1-L2 index to ensure the continuation of the processing flow.

8. The method according to claim 7, characterized in that, Step S1 further includes classifying and identifying messages. The identification information includes message ID, generation time, initial access frequency, priority level, and queue identifier. The identification information is stored together with the message data and serves as the basic data for index construction.

9. The method according to claim 7, characterized in that, The intelligent migration step S2 specifically includes: Step S2.1, Multi-dimensional parameter acquisition unit: Real-time acquisition of message feature parameters and system load parameters. The message feature parameters include message access frequency counted by a 5-minute sliding window, message priority quantified into 3 levels, and message size divided into 3 levels. The system load parameters include real-time calculated memory usage, system-level CPU average utilization, and IOPS of the disk where the bucketed file queue is located. Step S2.2, Hierarchical Strategy Decision Unit: Employs a three-layer decision logic to output scheduling instructions, specifically including: Step S2.2.1, System load threshold judgment: Based on the initial memory utilization threshold T0, when CPU utilization is >80%, T0 is increased by 10%; when IOPS is >10000, T0 is decreased by 10%; when memory utilization is ≥ adjusted T0, "memory → file" migration is triggered; when the file queue message access frequency is ≥ hotspot threshold for 3-5 consecutive times, "file → memory" migration is triggered. Step S2.2.2, message value sorting: The message value is quantified by the scoring formula Score=(access frequency×0.4)+(priority×0.3)+(1 / size range×0.3). During migration, messages with a Score≤2 in the transition message area are selected, and messages with a Score≥5 in the file queue are selected during relocation. Step S2.2.3, Action Execution Decision: The migration batch is calculated according to "(current memory occupancy rate - T0) × total memory capacity / average size of a single message", and the return batch is ≤ 80% of the remaining capacity of the hot message area. Both migration and return are sorted in ascending order of message timestamp; Step S2.3, Dynamic Execution and Status Synchronization Unit: Performs migration / reverse operations according to scheduling instructions. During migration, low-value messages are written in batches to the corresponding hash buckets in the bucketed file queue, and messages in the memory transition area are deleted synchronously. During reverse, high-value messages are written in batches to the memory hot spot area, and the corresponding messages in the file are marked as invalid synchronously. After the operation is completed, the status information of the hybrid index module and the fault recovery module is updated. Step S2.4, Exception Handling and Load Balancing Unit: When migration / return is interrupted, record the interruption sequence number and restart execution; when parameter acquisition fails, use the average value of the first 5 samples instead; monitor hash bucket IOPS in real time; when the IOPS of a certain bucket is >5000 for 3 consecutive minutes, migrate queues with Score≤2 in the bucket to the idle bucket; when the utilization rate of the hot message area is >90%, increase its memory share and migrate high-value messages in the transition area in advance.

Images