A distributed message processing system based on dynamic sliding window and intelligent scheduling

Abstract

This invention discloses a distributed message processing system based on dynamic sliding windows and intelligent scheduling, aiming to solve the problems of processing latency, resource waste, and insufficient scalability caused by fixed grouping and polling strategies in traditional message processing technologies. It is suitable for dynamic message scheduling and load balancing in high-concurrency, high-throughput scenarios. A four-layer collaborative architecture is adopted, with a distributed message queue layer, a sliding grouping module, an intelligent scheduling module, and a cluster of processing units working together to form a closed-loop control through a state feedback mechanism. The system designs a hybrid sliding window and dynamic triggering grouping mechanism, a two-way feedback intelligent scheduling system, and a multi-dimensional elastic control strategy, which can dynamically optimize resource allocation based on message traffic fluctuations and processing unit load status, balancing processing timeliness and transaction consistency. It can be widely applied to scenarios such as financial transaction clearing, e-commerce flash sale traffic smoothing, and IoT data processing, achieving adaptive throughput improvement and efficient resource utilization, and possesses strong practicality and promotional value.

Description

Technical Field

[0001] This invention relates to a distributed message processing system based on dynamic sliding windows and intelligent scheduling, belonging to the field of distributed message processing technology. Background Technology

[0002] In distributed systems, real-time data processing, and high-concurrency scenarios, message distribution and processing systems are core infrastructure, responsible for data transmission, task scheduling, and resource coordination. Their performance directly determines business efficiency and service quality. With the advancement of digital transformation, message data is exhibiting characteristics of massive volume, high frequency, and dynamic nature, with increasingly stringent requirements for processing latency. Furthermore, the capabilities of processing units vary significantly, placing high demands on the system's adaptability, flexibility, and resource utilization.

[0003] Traditional systems often employ fixed grouping and simple polling distribution strategies, along with direct invocation or batch processing methods. These strategies can only meet the needs of low-concurrency and stable message volume scenarios. When faced with massive and high-frequency dynamic message streams, they suffer from numerous technical defects, which in turn restrict system performance and business adaptability.

[0004] These shortcomings are particularly evident in the message distribution process. Fixed grouping strategies group messages according to a preset number or time interval. The core problem with this strategy is that the grouping rules cannot be dynamically adjusted: when the message volume surges, excessively large groups will overload the processing unit and increase latency; when the message volume is sparse, excessively small groups will cause the processing unit to start and stop frequently, wasting resources and reducing processing efficiency.

[0005] Simple polling strategies distribute messages evenly to processing units in a fixed order, lacking awareness and scheduling of the real-time capabilities of each unit, achieving only a superficial balance. When processing units have unbalanced capabilities, weaker units are prone to message backlog and overload, while stronger units may remain idle, resulting in resource waste and an imbalance in overall processing efficiency.

[0006] Even though some systems adopt priority-based distribution strategies to prioritize core business processing, the aforementioned problems remain unresolved. This strategy only adjusts the message distribution order and does not optimize the matching logic between grouping rationality and processing unit capabilities. In scenarios with massive and high-frequency messages, issues such as improper grouping and resource imbalance still exist, failing to balance business priorities and overall system performance.

[0007] In the message processing stage, direct calls and batch processing methods further amplify system bottlenecks in high-concurrency scenarios. Batch processing relies on specific triggering conditions to start the process, making it impossible to respond to business needs in real time, resulting in processing delays; direct calls, on the other hand, will cause concentrated access and resource contention among nodes, with delays accumulating and forming performance bottlenecks, which may even lead to node failures in severe cases.

[0008] In summary, the core shortcomings of traditional message distribution and processing systems are poor flexibility, low resource utilization, insufficient processing efficiency, and limited scalability. Distribution methods cannot adapt to fluctuations in message volume and differences in processing unit capabilities; unreasonable grouping and allocation logic leads to resource waste; the latency characteristics of batch processing and resource contention issues reduce overall processing efficiency; and the system architecture relies on fixed strategies, making smooth expansion difficult and hindering business upgrades. Therefore, there is an urgent need for a dynamically adaptable, intelligently matched solution that balances real-time performance and resource utilization to address the deficiencies of traditional systems. Summary of the Invention

[0009] The technical problem to be solved by this invention is to provide a distributed message processing system based on dynamic sliding window and intelligent scheduling, which effectively solves the problems of processing delay, resource waste and insufficient scalability caused by fixed grouping or polling strategies in traditional technologies.

[0010] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: A distributed message processing system based on dynamic sliding window and intelligent scheduling includes a distributed message queue layer, a sliding grouping module, an intelligent scheduling module, and a processing unit cluster, which are sequentially linked and form a closed-loop control through a state feedback mechanism. Specifically: The distributed message queue layer is used to provide buffering and temporary storage for streaming messages, and to perform preliminary partitioning and storage of streaming messages based on message characteristics. The sliding grouping module is used to monitor the message backlog in the distributed message queue layer in real time and feed it back to the intelligent scheduling module. It is used to receive the message grouping strategy adjustment instructions sent back by the intelligent scheduling module, adjust the message grouping strategy based on the instructions, and group the messages in each area. The intelligent scheduling module is used to allocate the message groups obtained after grouping to each processing unit for processing, receive status confirmation signals from each processing unit, monitor the health status of each processing unit, and send instructions to the sliding grouping module to adjust the message grouping strategy based on the health status of each processing unit; when no status confirmation signal is received from the processing unit, the abnormal handling mechanism is activated for immediate response. The processing unit cluster includes several processing units. Each processing unit is used to process message groups assigned by the intelligent scheduling module and sends a status confirmation signal to the intelligent scheduling module after receiving the message group and completing the message group processing.

[0011] As a preferred embodiment of the present invention, the distributed message queue layer includes a message partition manager for performing preliminary partitioning and storage of streaming messages based on message characteristics.

[0012] As a preferred embodiment of the present invention, the sliding grouping module includes a sliding window controller and a grouping engine, wherein the sliding window controller monitors the message backlog of the distributed message queue layer in real time and dynamically adjusts the message grouping strategy, including window duration or window size, according to the processing capacity of the processing unit cluster combined with the traffic prediction model. The grouping engine performs message segmentation within each zone based on the message grouping strategy. The message grouping strategy includes a combination of at least one of the following conditions: window duration expiration trigger, accumulated message volume within the window reaching a first preset threshold trigger, and key field transition event trigger. Key field transition event trigger refers to grouping messages of different processing flows when messages that trigger different processing flows appear.

[0013] As a preferred embodiment of the present invention, the sliding grouping module dynamically adjusts the starting position of the window using a sliding offset mechanism during the grouping process based on the adjusted message grouping strategy to ensure grouping continuity.

[0014] As a preferred embodiment of the present invention, the intelligent scheduling module includes a status monitoring engine, a decision engine, and a load balancer. The status monitoring engine is used to receive a status confirmation signal from the processing unit after the processing unit completes the processing of the message group, and at the same time monitor the health status of the processing unit in real time. The status confirmation signal includes the number of messages received by the processing unit and the number of messages currently being processed by the processing unit. The health status includes CPU utilization, memory usage, and processing latency indicators. The load balancer is used to receive status confirmation information from the processing unit when a message group is allocated to the processing unit, and to send feedback to the decision engine when no status confirmation signal is received from the processing unit. The decision engine is used to allocate message groups based on an improved weighted round-robin algorithm, and to generate sliding group modules based on the health status of each processing unit and feed them back to the sliding group modules. It is also used to resend timed-out unacknowledged message groups to the originally allocated processing units when the load balancer does not receive status acknowledgment signals from the processing units. When the proportion of status acknowledgment signals fed back by a processing unit is lower than a second preset threshold, the processing unit is determined to be a faulty processing unit and is isolated. The proportion of status acknowledgment signals fed back by a processing unit is the ratio of the number of status acknowledgment signals fed back by the processing unit to the number of message groups allocated to that processing unit.

[0015] As a preferred embodiment of the present invention, the intelligent scheduling module, based on the number of messages received by each processing unit and the number of messages currently being processed by the processing unit, triggers a horizontal expansion instruction according to the queue depth prediction model to elastically scale the processing capacity of each processing unit.

[0016] As a preferred embodiment of the present invention, the processing unit supports a multi-mode processing engine, is configured with database transaction, file I / O and network communication processing components, adopts an asynchronous non-blocking architecture design, and has built-in retry and circuit breaker fault tolerance mechanisms.

[0017] Compared with the prior art, the present invention, employing the above technical solution, has the following technical effects: 1. This invention adopts a hybrid grouping mechanism that combines sliding windows and dynamic triggering, abandoning the rigid rules of traditional fixed grouping. By collecting message traffic characteristics and processing unit capabilities in real time, it dynamically adjusts the window size and grouping triggering conditions. This avoids processing delays caused by excessively large groups when message volume surges, and also prevents frequent start-stop of processing units caused by excessively small groups when message volume is sparse. It is suitable for high-concurrency and high-throughput scenarios and stably supports scenarios such as real-time clearing of financial transactions and traffic shaving for e-commerce flash sales.

[0018] 2. This invention constructs a two-way feedback intelligent scheduling system, breaking down the linkage barriers between the sliding grouping module and the intelligent scheduling module. The intelligent scheduling module collects the load status of the processing unit in real time and synchronizes indicators such as processing latency and success rate to the grouping module through the feedback loop to guide strategy adjustment. At the same time, the grouping results provide a basis for resource allocation, forming a closed-loop collaboration. This fundamentally solves the problems of uneven load distribution and coexistence of unit overload and idleness in the traditional polling strategy, and achieves optimized resource allocation.

[0019] 3. This invention adopts a multi-dimensional elastic control strategy, which enables adaptive adjustment of processing capacity as message traffic fluctuates through horizontal expansion of the distributed message queue layer and horizontal expansion instructions of the processing unit cluster. The processing unit adopts an asynchronous non-blocking architecture with built-in retry and circuit breaker fault tolerance mechanisms. While ensuring transaction consistency, it improves the parallel processing efficiency and anti-interference capability of the system, overcomes the shortcomings of insufficient scalability of traditional technologies, and is suitable for scenarios such as massive data processing of the Internet of Things and instant messaging message distribution.

[0020] 4. This invention can be widely adapted to high-concurrency scenarios in multiple fields such as finance, e-commerce, IoT, and instant messaging, and can be implemented without major modifications; at the same time, through resource optimization and elastic scaling, it reduces ineffective resource consumption and lowers deployment and maintenance costs, making it highly practical and economical. Attached Figure Description

[0021] Figure 1 This is an architecture diagram of the distributed message processing system based on dynamic sliding window and intelligent scheduling of the present invention; Figure 2 This is a flowchart of the operation of the distributed message processing system based on dynamic sliding window and intelligent scheduling of the present invention. Figure 3 This is an example of an architecture diagram of the system of the present invention used in a securities firm's post-trade clearing asset bookkeeping project. Detailed Implementation

[0022] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0023] like Figure 1 As shown, the present invention provides a distributed message processing system based on dynamic sliding window and intelligent scheduling, including a distributed message queue layer, a sliding grouping module, an intelligent scheduling module, and a processing unit cluster that are sequentially linked and form a closed-loop control through a state feedback mechanism.

[0024] The distributed message queue layer includes a message partition manager, which provides buffering for streaming messages and performs initial partitioning and storage of streaming messages based on message characteristics. After streaming messages enter the distributed message queue layer, the message partition manager completes the initial partitioning and buffering.

[0025] The sliding grouping module includes a sliding window controller and a grouping engine. The sliding window controller monitors the message backlog in the distributed message queue layer in real time and feeds it back to the intelligent scheduling module. Simultaneously, based on the processing capacity of the processing unit cluster and a traffic prediction model, it dynamically adjusts the message grouping strategy, including window duration or window size. The grouping engine, based on the adjusted message grouping strategy, performs message splitting within each zone. Grouping is triggered by at least one combination of window duration expiration, the accumulated message volume within the window reaching a first preset threshold, and a key field transition event. The key field transition event trigger refers to grouping messages that will trigger different processing flows when such messages appear. Furthermore, during the grouping process, the sliding grouping module uses a sliding offset mechanism to dynamically adjust the starting position of the window to ensure grouping continuity.

[0026] The sliding grouping module includes a sliding window controller and a grouping engine. The sliding window controller monitors the message backlog in the distributed message queue layer in real time and feeds it back to the intelligent scheduling module. Simultaneously, based on the processing capacity of the processing unit cluster and a traffic prediction model, it dynamically adjusts the message grouping strategy, including window duration or window size. The grouping engine, based on the adjusted message grouping strategy, performs message splitting within each zone. Grouping is triggered by at least one combination of window duration expiration, the accumulated message volume within the window reaching a first preset threshold, and a key field transition event. The key field transition event trigger refers to grouping messages that will trigger different processing flows when such messages appear. Furthermore, during the grouping process, the sliding grouping module uses a sliding offset mechanism to dynamically adjust the starting position of the window to ensure grouping continuity.

[0027] The processing unit cluster includes several processing units. Each processing unit is used to process message groups assigned by the intelligent scheduling module and sends a status confirmation signal to the intelligent scheduling module after receiving the message group and completing the message group processing. Each processing unit supports multi-mode processing engines, is configured with database transaction, file I / O and network communication processing components, adopts an asynchronous non-blocking architecture design, and has built-in retry and circuit breaker fault tolerance mechanisms.

[0028] like Figure 2 The diagram shown is a flowchart of the operation of the system of the present invention, as detailed below: 1. Message Injection Phase: Streaming messages enter a distributed queue for buffering and temporary storage, and the message partition manager performs preliminary classification and storage based on message characteristics; 2. Dynamic Grouping Phase: The sliding window controller monitors the message backlog in real time and dynamically adjusts the window threshold parameters (default window size 50-200 messages, time window 5-10 seconds) through a traffic prediction model; the grouping engine performs message splitting based on trigger conditions, including at least one combination of window duration expiration trigger, accumulated message volume reaching a target trigger, and key field jump event trigger; during the grouping process, a sliding offset mechanism is used to dynamically adjust the window starting position to ensure grouping continuity and avoid message omission; 3. Intelligent scheduling phase: The load balancer obtains the health status of each processing unit in real time; the decision engine allocates tasks based on an improved weighted round-robin algorithm, reduces the weight value of high-load nodes, and automatically obtains more message groups for idle nodes; the exception handling mechanism responds immediately, starts the replica resend process for message groups that have timed out and not been acknowledged, and automatically isolates faulty nodes. 4. Processing Feedback Phase: After each processing unit completes its transaction, it sends a status confirmation signal to the scheduling module. The intelligent scheduling module comprehensively analyzes indicators such as processing latency and success rate, and notifies the grouping module to adjust the grouping strategy through the feedback loop. At the same time, based on the queue depth prediction model, it triggers horizontal expansion instructions in advance to achieve elastic scaling of processing capacity.

[0029] Each module forms a closed-loop control system through a status feedback mechanism. The distributed message queue layer, sliding grouping module, intelligent scheduling module, and processing unit cluster work together to achieve efficient and stable message processing.

[0030] Figure 3 This demonstrates the successful implementation of this invention's solution in a securities firm's post-trade clearing and asset bookkeeping project. The system's core function is to process massive amounts of clearing and settlement data while ensuring real-time performance and consistency. The core objective of message distribution is to decouple upstream and downstream systems, ensuring the independence and accuracy of data processing through an efficient message routing mechanism. Specifically: Upstream sends transaction clearing and settlement logs to the Pulsar message queue cluster; The system calculates the hash value based on the account field (unifiedAcct) in the message, formats the account field, and facilitates the message queue to distribute the message evenly to different consumer nodes in key_shared mode according to a consistent grouping algorithm (including the sliding grouping module and the intelligent scheduling module). After the cluster node (sliding grouping module) consumes the message stream, it groups the messages according to the currently effective sliding grouping strategy; After being grouped, the messages enter the intelligent scheduling module of the asset bookkeeping, which delivers the grouped messages to the corresponding processing thread (processing unit) according to the currently effective load strategy (round-robin, weighted, or random). The intelligent scheduling module periodically monitors the queue length of the task queue for each business processing thread; The intelligent scheduling module adjusts the group message distribution strategy based on the monitoring results and feeds back message backlog data to the sliding group module; The sliding grouping module judges the feedback monitoring message data to determine the current system operation status, and dynamically changes the sliding grouping strategy according to the pre-set operation level. That is, the system will first preset several grouping strategies corresponding to different processing levels. The sliding grouping module feeds back the message backlog to the intelligent scheduling module, which judges the level and then initiates the corresponding level of grouping.

[0031] The above embodiments are merely illustrative of the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solutions based on the technical concept proposed in this invention shall fall within the scope of protection of this invention.

Claims

1. A distributed message processing system based on dynamic sliding windows and intelligent scheduling, characterized in that, This includes a distributed message queue layer, a sliding grouping module, an intelligent scheduling module, and a processing unit cluster, all linked sequentially and forming a closed-loop control through a status feedback mechanism. Specifically: The distributed message queue layer is used to provide buffering and temporary storage for streaming messages, and to perform preliminary partitioning and storage of streaming messages based on message characteristics. The sliding grouping module is used to monitor the message backlog in the distributed message queue layer in real time and feed it back to the intelligent scheduling module. It is used to receive the message grouping strategy adjustment instructions sent back by the intelligent scheduling module, adjust the message grouping strategy based on the instructions, and group the messages in each area. The intelligent scheduling module is used to allocate the message groups obtained after grouping to each processing unit for processing, receive status confirmation signals from each processing unit, monitor the health status of each processing unit, and send instructions to the sliding grouping module to adjust the message grouping strategy based on the health status of each processing unit; when no status confirmation signal is received from the processing unit, the abnormal handling mechanism is activated for immediate response. The processing unit cluster includes several processing units. Each processing unit is used to process message groups assigned by the intelligent scheduling module and sends a status confirmation signal to the intelligent scheduling module after receiving the message group and completing the message group processing.

2. The distributed message processing system based on dynamic sliding window and intelligent scheduling according to claim 1, characterized in that, The distributed message queue layer includes a message partition manager, which is used to perform preliminary partitioning and storage of streaming messages based on message characteristics.

3. The distributed message processing system based on dynamic sliding window and intelligent scheduling according to claim 1, characterized in that, The sliding grouping module includes a sliding window controller and a grouping engine. The sliding window controller monitors the message backlog of the distributed message queue layer in real time and dynamically adjusts the message grouping strategy, including window duration or window size, based on the processing capacity of the processing unit cluster and the traffic prediction model. The grouping engine performs message segmentation within each zone based on the message grouping strategy. The message grouping strategy includes a combination of at least one of the following conditions: window duration expiration trigger, accumulated message volume within the window reaching a first preset threshold trigger, and key field transition event trigger. Key field transition event trigger refers to grouping messages of different processing flows when messages that trigger different processing flows appear.

4. The distributed message processing system based on dynamic sliding window and intelligent scheduling according to claim 3, characterized in that, The sliding grouping module dynamically adjusts the starting position of the window using a sliding offset mechanism during the grouping process based on the adjusted message grouping strategy, ensuring grouping continuity.

5. The distributed message processing system based on dynamic sliding window and intelligent scheduling according to claim 1, characterized in that, The intelligent scheduling module includes a status monitoring engine, a decision engine, and a load balancer. The status monitoring engine is used to receive a status confirmation signal from the processing unit after the processing unit completes the processing of the message group, and at the same time monitor the health status of the processing unit in real time. The status confirmation signal includes the number and size of messages received by the processing unit and the size of messages currently being processed by the processing unit. The health status includes CPU utilization, memory usage, and processing latency indicators. The load balancer is used to receive status confirmation information from the processing unit when a message group is allocated to the processing unit, and to send feedback to the decision engine when no status confirmation signal is received from the processing unit. The decision engine is used to allocate message groups based on an improved weighted round-robin algorithm, and to generate sliding group modules based on the health status of each processing unit and feed them back to the sliding group modules. It is also used to resend timed-out unacknowledged message groups to the originally allocated processing units when the load balancer does not receive status acknowledgment signals from the processing units. When the proportion of status acknowledgment signals fed back by a processing unit is lower than a second preset threshold, the processing unit is determined to be a faulty processing unit and is isolated. The proportion of status acknowledgment signals fed back by a processing unit is the ratio of the number of status acknowledgment signals fed back by the processing unit to the number of message groups allocated to that processing unit.

6. The distributed message processing system based on dynamic sliding window and intelligent scheduling according to claim 5, characterized in that, The intelligent scheduling module, based on the number of messages received by each processing unit and the number of messages currently being processed by the processing unit, triggers a horizontal expansion command according to the queue depth prediction model to elastically scale the processing capacity of each processing unit.

7. The distributed message processing system based on dynamic sliding window and intelligent scheduling according to claim 1, characterized in that, The processing unit supports a multi-mode processing engine, is configured with database transaction, file I / O and network communication processing components, adopts an asynchronous non-blocking architecture design, and has built-in retry and circuit breaker fault tolerance mechanisms.

Images