A layered asynchronous processing method, system and computer readable storage medium for real-time messages

Abstract

The application discloses a layered asynchronous processing method and system for real-time messages and a computer readable storage medium, and relates to the technical field of communication. A receiving processing layer receives real-time messages and writes the real-time messages into a receiving buffer structure, and after pre-processing, the real-time messages or processing tasks are transferred to a preprocessing layer; after the preprocessing layer performs basic processing and generates to-be-analyzed tasks, the tasks are transferred to an analysis processing layer; the analysis processing layer performs analysis according to message types, generates analysis results and transfers the analysis results to a mapping processing layer; the mapping processing layer generates data objects according to data mapping rules and transfers the data objects to a distribution processing layer; and the distribution processing layer is matched and distributed to target units according to a data distribution strategy. The processing layers are asynchronously decoupled through a task transmission structure, and are scheduled by a scheduling control layer according to running load conditions, so that a pipeline type message processing link is formed, and synchronous waiting and resource contention are reduced, and the throughput capacity, time delay stability and running reliability in a continuous input scene are improved.

Description

Technical Field

[0001] This invention relates to the field of communication technology, and in particular to a hierarchical asynchronous processing method, system, and computer-readable storage medium for real-time messages. Background Technology

[0002] Current real-time message processing typically employs a serial processing approach, with all steps executed sequentially within the same processing thread. However, significant differences exist in the computational complexity and resource consumption patterns across different stages, such as message access, parsing, mapping, and distribution. High-time-consuming processing stages are prone to input blocking, message backlog, or processing chain jitter. When messages arrive suddenly or a processing step experiences increased computational load leading to longer processing times, it intensifies thread contention, resource consumption, and queue congestion. In severe cases, this can result in decreased system throughput, increased transmission latency, and even message loss. Simultaneously, some system resources remain underutilized, limiting overall system processing efficiency and making it difficult to maintain stable operation under continuous high-frequency real-time message input. Summary of the Invention

[0003] To address the aforementioned shortcomings of the prior art, the present invention aims to provide a layered asynchronous processing method for real-time messages, thereby resolving the problems of high processing link coupling, low scheduling efficiency, severe resource contention, and insufficient system stability in scenarios with high-frequency input, high throughput, and low latency. The invention also provides a system and a computer-readable storage medium.

[0004] The technical solution of the present invention is as follows: a layered asynchronous processing method for real-time messages, comprising the following steps:

[0005] Step 1: Input the real-time message into the receiving and processing layer. The receiving and processing layer has a receiving buffer structure. The receiving and processing layer writes the real-time message into the receiving buffer structure. The receiving and processing layer performs preprocessing on the received real-time message.

[0006] Step 2: Transfer the real-time message or the processing task corresponding to the real-time message processed by the receiving processing layer to the preprocessing layer. The preprocessing layer performs basic processing on the real-time message or the processing task and generates a task to be parsed.

[0007] Step 3: Transfer the task to be parsed to the parsing processing layer. The parsing processing layer performs parsing according to the type of the real-time message, extracts information and generates parsing results.

[0008] Step 4: The parsing result is transferred to the mapping processing layer, which performs data transformation processing on the parsing result according to the data mapping rules and generates a data object;

[0009] Step 5: Transfer the data object to the distribution processing layer. The distribution processing layer performs rule matching on the data object according to the data distribution strategy and distributes the results to the target unit.

[0010] A task passing structure is provided between the receiving processing layer, the preprocessing layer, the parsing processing layer, the mapping processing layer, and the distribution processing layer. The task passing structure realizes asynchronous decoupling between adjacent processing layers. The scheduling control layer performs asynchronous scheduling of the receiving processing layer, the preprocessing layer, the parsing processing layer, the mapping processing layer, and the distribution processing layer through the task passing structure to form a pipelined message processing link.

[0011] Furthermore, in adjacent processing layers, the subsequent processing layer reads the processing task from the previous task transfer structure, and after the previous processing layer completes its processing, it writes the processing result or the next stage task into the subsequent task transfer structure.

[0012] Furthermore, the task delivery structure employs a task delivery strategy to reduce the risk of blocking between different processing layers due to shared processing resources.

[0013] Furthermore, the task transfer strategy includes at least one of sequential writing, sequential reading, circular reuse, or batch transfer.

[0014] Furthermore, the scheduling control layer obtains the operating load status of the receiving processing layer, the preprocessing layer, the parsing processing layer, the mapping processing layer, and the distribution processing layer. When the operating load status reaches the scheduling condition of the scheduling control layer, a scheduling action is triggered.

[0015] Furthermore, the operating load includes task volume, queue length, processing latency, or resource consumption, and the scheduling action includes at least one of thread resource adjustment, task allocation adjustment, scheduling order adjustment, batch processing, rate limiting control, delayed scheduling, priority adjustment, or backpressure control.

[0016] Another technical solution of the present invention is:

[0017] A layered asynchronous processing system for real-time messages includes the following modules:

[0018] The receiving and processing module is used to receive real-time messages and write the real-time messages into the receiving buffer structure to complete the pre-processing of the real-time messages.

[0019] The preprocessing module is used to perform basic processing on the real-time messages output by the receiving and processing module before the parsing stage, and to generate parsing tasks.

[0020] The parsing and processing module is used to perform parsing based on the type of real-time message, extract information, and generate parsing results.

[0021] The mapping processing module is used to perform data transformation processing on the parsing results according to the data mapping rules to generate data objects;

[0022] The distribution processing module is used to distribute the data objects according to the data distribution strategy;

[0023] The task transfer module is used to transfer tasks among the receiving processing module, the preprocessing module, the parsing processing module, the mapping processing module, and the distribution processing module, thereby achieving asynchronous decoupling between adjacent processing modules.

[0024] The scheduling and control module is used to schedule the receiving processing module, the preprocessing module, the parsing processing module, the mapping processing module and the distribution processing module through the task delivery module according to the operating load or preset scheduling rules, forming a pipeline-style message processing link.

[0025] Furthermore, in adjacent processing modules, the subsequent processing module reads the processing task from the preceding task transfer module, and after the preceding processing module completes its processing, it writes the processing result or the next stage task into the subsequent task transfer module.

[0026] Furthermore, the task delivery module adopts a task delivery strategy to reduce the risk of blocking between different processing modules due to shared processing resources.

[0027] Furthermore, the task transfer strategy includes at least one of sequential writing, sequential reading, circular reuse, or batch transfer.

[0028] Furthermore, the scheduling control module obtains the operating load status of the receiving processing module, the preprocessing module, the parsing processing module, the mapping processing module, and the distribution processing module. When the operating load status reaches the scheduling conditions of the scheduling control module, the corresponding scheduling action is triggered.

[0029] Furthermore, the operating load status includes task volume, queue length, processing latency, or resource usage, and the scheduling action includes at least one of thread resource adjustment, task allocation adjustment, scheduling order adjustment, batch processing, rate limiting control, delayed scheduling, priority adjustment, or backpressure control.

[0030] Another technical solution of the present invention is: a computer-readable storage medium storing a computer program thereon, wherein when the computer program is executed by a processor, the aforementioned layered asynchronous processing method for real-time messages is implemented.

[0031] Compared with the prior art, the beneficial effects of the present invention are:

[0032] (1) By dividing the real-time message processing process into multiple processing layers, and passing the processing results between each processing layer through a task passing structure, a task passing and asynchronous decoupling relationship is established. This reduces the direct coupling between different processing links in the traditional serial processing mode from the overall architecture perspective. It helps to avoid the high-time-consuming processing of the later stage from continuously dragging down the message receiving capability of the previous stage, thereby improving the system's continuous processing capability for continuously input real-time messages.

[0033] (2) By constructing a hierarchical asynchronous pipeline message processing link, different processing stages can be carried out in parallel on relatively independent processing resources, thereby reducing the waiting and blocking problems caused by a single processing thread executing all processing tasks sequentially. This is conducive to improving the resource utilization efficiency in a multi-core hardware environment, enhancing the overall throughput of the system, and shortening the overall latency of message processing.

[0034] (3) By setting up a task transfer structure between different processing stages, the system can isolate, cache and buffer the task flow when faced with sudden message arrival, uneven stage load or local processing pressure increase, thereby suppressing the spread of task accumulation along the entire processing link, reducing the risk of system congestion, processing jitter or message loss, and enhancing the system's stable operation under high load conditions.

[0035] (4) By functionally splitting the preprocessing layer, parsing processing layer and mapping processing layer, different types of tasks such as protocol identification, basic verification, message semantic extraction and business object mapping belong to different processing stages. This helps to reduce the coupling between protocol parsing and subsequent business processing, reduce the performance bottleneck caused by complex processing concentrated in a single link, and thus improve the overall processing efficiency and structural scalability of the system.

[0036] (5) By introducing a scheduling and control mechanism based on task load, queue status, processing capacity and resource usage, the system can dynamically adjust the number of threads, task allocation method and processing order according to the running status, and trigger scheduling actions such as batch handling, flow control, delay scheduling, priority adjustment or back pressure control when the preset conditions are met, thereby taking into account the requirements of high throughput, low latency and high stability, and improving the system's adaptability to complex industrial communication scenarios.

[0037] (6) The layered asynchronous processing method and system provided by the present invention are suitable for real-time message processing in industrial communication scenarios. They have good versatility and engineering application value and can be applied to various processing environments that require receiving, parsing, mapping and distributing continuous input messages. Attached Figure Description

[0038] Figure 1 This is a schematic diagram of the overall process of a layered asynchronous processing method for real-time messages according to the present invention.

[0039] Figure 2 This is a structural block diagram of a layered asynchronous processing system for real-time messages according to the present invention.

[0040] Figure 3 This is a schematic diagram of the scheduling control process of the scheduling control module. Detailed Implementation

[0041] The specific technical solution of the present invention will be further described in detail below with reference to specific examples.

[0042] like Figures 1 to 3 As shown in the embodiment of the present invention, a layered asynchronous processing method for real-time messages includes the following steps:

[0043] Step 1: Real-time messages from the acquisition device or communication link are input into the receiving and processing layer. The receiving and processing layer has a receiving buffer structure. The receiving and processing layer writes the real-time messages into the receiving buffer structure, which may include a receiving buffer area, a message buffer pool, a receiving ring buffer, or other data structures for temporarily storing input messages. The receiving buffer structure is mainly used to temporarily store the raw real-time messages input from the acquisition device or communication link, absorbing short-term message input fluctuations and preventing changes in processing time in subsequent stages from directly blocking the front-end receiving process. The receiving and processing layer performs pre-processing on the received real-time messages. Pre-processing includes at least one of initial caching, message organization, or task encapsulation. The receiving and processing layer mainly undertakes front-end reception, raw message temporary storage, and task encapsulation functions, and writes the encapsulated processing tasks into the task transfer structure between the receiving and processing layer and the pre-processing layer to ensure that the input messages can stably enter the subsequent processing flow. The receiving and processing layer is not responsible for complete protocol parsing or business processing.

[0044] Step 2: After completing at least one of the following processes—initial caching, message organization, or task encapsulation—the receiving processing layer transfers the message or its corresponding processing task to the preprocessing layer via a task transfer structure. The preprocessing layer performs the basic processing required before entering the parsing processing layer, generating the task to be parsed. Basic processing includes at least one of the following: protocol identification, message type marking, source information extraction, length verification, integrity verification, or validity verification. The preprocessing layer marks, discards, or transfers tasks that do not meet format requirements or verification conditions to an exception handling process. The preprocessing layer does not perform complete business processing, thereby shortening the front-end processing path and reducing the impact of complex processing on message reception capabilities. The processing task corresponding to the message is a task description information formed by the receiving processing layer based on the encapsulation of the input message. The task description information may include at least one of the following: message cache address or message reference, message length, source identifier, reception time, task status, priority identifier, processing stage identifier, context number, or other information used to indicate subsequent processing flows. By passing task description information, the repeated transfer of complete message data between different processing layers can be avoided, enabling each processing layer to continuously complete preprocessing, parsing, mapping, and distribution based on the same task context. The task passing structure connects adjacent processing layers, establishing task buffering, task isolation, and asynchronous handover relationships between them. The task passing structure includes at least one of a task queue, buffer, circular buffer, or cache structure. In adjacent processing layers, the subsequent processing layer reads the task to be processed from its preceding task passing structure. After completing its processing, the preceding processing layer writes the processing result or the next stage task into its subsequent task passing structure. The preceding processing layer can continue to receive or process the next task after completing the task writing, without waiting for the subsequent processing layer to finish processing. The subsequent processing layer reads tasks from the task passing structure and executes them according to the running rhythm of its own thread, thread group, or processing unit. Through this producer-consumer task passing method, adjacent processing layers are no longer directly coupled through synchronous function calls, thus achieving asynchronous decoupling between processing stages. Between the receiving processing layer and the preprocessing layer, the task passing structure is used to receive the initial task written by the receiving processing layer, so that the receiving processing layer can return to the front-end receiving process after completing the message buffering and task encapsulation, avoiding the subsequent preprocessing time from directly blocking message reception.

[0045] Step 3: After the preprocessing layer generates the tasks to be parsed, it transfers these tasks to the parsing processing layer via a task transfer structure. The parsing processing layer parses the tasks based on the message type, which is identified using protocol syntax rules. It performs at least one of the following processing steps on the tasks to be parsed: field extraction, type determination, hierarchical relationship identification, or semantic conversion, to extract the target data content, identification information, and associated attribute information, and generate the parsing result. Between the preprocessing layer and the parsing processing layer, the task transfer structure temporarily stores tasks that have completed protocol identification, length verification, integrity verification, or legality verification, allowing the parsing processing layer to independently read the tasks according to task type, priority, or arrival order. The parsing processing layer focuses on completing the semantic parsing of the messages and does not directly handle subsequent business object generation and business distribution processing.

[0046] Step 4: After generating the parsing result, the parsing layer transfers the result to the mapping layer via a task transfer structure. The mapping layer performs data transformation processing on the parsing result according to the data mapping rules, generating data objects. Data mapping rules include preset data models, point models, data identification rules, or business object relationships. Data transformation processing includes at least one of the following: structure transformation, association mapping, data merging, object attribute filling, or state updating. After this process, the mapping layer generates data objects, which can be intermediate results or target objects. Between the parsing and mapping layers, the task transfer structure is used to transfer parsing results or references to parsing results, allowing the parsing layer to continue processing other parsing tasks without waiting for subsequent data object generation. By functionally separating preprocessing, parsing, and mapping, the mapping layer assigns different types of tasks, such as protocol identification, basic verification, message semantic extraction and data organization transformation, and business object mapping, to different processing stages. This reduces the coupling between protocol parsing and subsequent business processing, minimizes performance bottlenecks caused by concentrating complex processing in a single stage, and improves the overall system processing efficiency and structural scalability.

[0047] Step 5: After the mapping processing layer generates intermediate results or target objects, it transfers them to the distribution processing layer via a task transfer structure. The distribution processing layer performs rule matching on the intermediate results or target objects according to the data distribution strategy and sends the distribution results to the target units, thus completing the entire message processing process. Rule matching refers to selecting targets and determining sending paths for intermediate results or target objects based on the data distribution strategy. The data distribution strategy may include at least one of the following: preset subscription relationships, business categories, target channels, priority rules, target unit status, or data type rules. The distribution processing layer can determine that a data object needs to be sent to one or more target units based on the data object's business type, data identifier, priority, source information, or target unit subscription relationship. Target units may include business units, storage units, display units, or external interfaces. Distribution methods may include at least one of single-target distribution, multi-target distribution, broadcast distribution, or parallel distribution. Between the mapping processing layer and the distribution processing layer, the task transfer structure is used to transfer intermediate results or target objects, enabling the distribution processing layer to independently execute subsequent distribution processing according to subscription relationships, business categories, target channels, or priority rules.

[0048] By dividing the real-time message processing process into a receiving processing layer, a preprocessing layer, a parsing processing layer, a mapping processing layer, and a distribution processing layer, multiple relatively independent processing stages establish a task transfer structure and asynchronous decoupling relationship between adjacent processing stages. This reduces the direct coupling between different processing links in the traditional serial processing mode from the overall architecture, which helps to avoid the high-time-consuming processing of the later stage from continuously dragging down the message receiving capability of the earlier stage, thereby improving the system's continuous processing capability for continuously input real-time messages.

[0049] The scheduling control layer asynchronously schedules the receiving processing layer, preprocessing layer, parsing processing layer, mapping processing layer, and distribution processing layer through a task passing structure, forming a pipelined message processing link. The task passing structure enables stage isolation, task handover, buffering, and pressure buffering between adjacent processing layers, thereby suppressing task backlog along the entire processing link, reducing the risk of system congestion, processing jitter, or message loss, enhancing the system's stable operation under high load conditions, reducing the impact of time fluctuations in a single processing stage on the overall link throughput, improving resource utilization efficiency in multi-core hardware environments, increasing overall system throughput, and shortening the overall message processing latency. The task passing structure employs a task passing strategy to reduce the blocking risk caused by shared processing resources between different processing layers; the task passing strategy can be at least one of sequential write, sequential read, circular reuse, or batch handover. The scheduling control layer is used to obtain the runtime load status of the receiving processing layer, preprocessing layer, parsing processing layer, mapping processing layer, distribution processing layer, and task delivery structure. When the scheduling conditions are not met, the system maintains the current number of threads, thread division of labor, task allocation method, and scheduling order to avoid additional overhead. When the scheduling conditions are met, the scheduling control layer triggers the corresponding scheduling action. The specific scheduling action process is as follows:

[0050] When the queue length in the task delivery structure corresponding to a certain processing layer continuously exceeds the preset queue length threshold, or the average processing latency of the processing layer exceeds the preset processing latency threshold, the scheduling control layer can increase the number of processing threads corresponding to the processing layer, or allocate spare processing threads to the processing layer, in order to improve the task consumption capacity of the processing layer.

[0051] When the scheduling control layer detects that the system CPU utilization rate continuously exceeds the preset CPU utilization threshold, the scheduling control layer can limit the number of new threads, reduce the scheduling frequency of low-priority tasks, trigger batch processing to reduce the number of thread wake-ups, or perform flow control on the front-end processing layer to avoid further intensification of thread competition.

[0052] When the scheduling control layer detects that the system memory usage rate continuously exceeds the preset memory usage threshold, or the cache space occupied by the task delivery structure reaches the preset cache high water level, the scheduling control layer can delay the scheduling of low-priority tasks, reduce the batch size of batch transfers, limit the write speed of the preceding tasks, or trigger backpressure control to reduce the task entry rate of the preceding processing layer, thereby preventing the continuous growth of the cache from further increasing memory pressure.

[0053] When the queue length of the processing layer drops below the recovery threshold, or when the system resource usage reaches the preset upper limit, the scheduling control layer can reduce the number of corresponding processing threads to reduce resource usage and avoid thread contention.

[0054] When there is an imbalance in load between different processing layers, the scheduling control layer can adjust the thread division or task allocation method. For example, when the backlog of tasks in the parsing processing layer is significantly higher than that in other processing layers, while the preprocessing or mapping processing layer is under low load, the scheduling control layer can temporarily allocate reusable or shared threads to the parsing processing layer, or group and allocate tasks according to message type, task priority, source channel, or task complexity, thereby reducing local processing bottlenecks.

[0055] When the number of tasks in the task delivery structure reaches the batch transfer threshold, or when the task waiting time reaches the batch trigger time threshold, the scheduling control layer can trigger a batch transfer strategy to transfer multiple tasks as a batch from the previous processing layer to the next processing layer, thereby reducing the overhead caused by frequent scheduling and frequent wake-ups. When the task backlog in the next processing layer exceeds the high watermark threshold, the processing latency continues to rise, or the resource usage approaches the preset upper limit, the scheduling control layer can trigger rate limiting control or backpressure control to reduce the rate at which the previous processing layer writes tasks to the task delivery structure, or postpone the transfer time of low-priority tasks, to prevent the task backlog from continuing to spread.

[0056] When tasks of different priorities exist simultaneously in the system, the scheduling control layer can adjust the scheduling order according to priority, allowing high-priority tasks to enter subsequent processing layers first. For low-priority tasks, when queue pressure is high or resource consumption is high, a delayed scheduling strategy can be adopted, continuing processing only after a preset delay condition is met or the system load decreases. By using at least one of the above-mentioned thread resource adjustment, task allocation adjustment, scheduling order adjustment, batch processing, rate limiting control, delayed scheduling, priority adjustment, or backpressure control, where thread resource adjustment includes dynamically adjusting the number of threads or adjusting thread division of labor, it is possible to maintain a relatively stable pipeline processing capability for each processing layer under conditions of continuous high-frequency input, local congestion, or fluctuations in resource consumption.

[0057] By introducing a scheduling and control mechanism based on task load, queue status, processing capacity, and resource usage, the system can dynamically adjust the number of threads, task allocation method, and processing order according to the running status. When preset conditions are met, it can trigger scheduling actions such as batch processing, rate limiting control, delay scheduling, priority adjustment, or backpressure control, thereby taking into account the requirements of high throughput, low latency, and high stability, and improving the system's adaptability to complex industrial communication scenarios.

[0058] The hierarchical asynchronous processing system for real-time messages according to an embodiment of the present invention includes the following modules:

[0059] The receiving and processing module receives real-time messages from the acquisition device or communication link. It includes a receiving buffer structure, which the module writes into. This buffer primarily stores the raw real-time messages input from the acquisition device or communication link, absorbing short-term fluctuations in message input and preventing subsequent processing delays from directly blocking the front-end receiving process. The receiving buffer structure includes a receiving buffer area, a message buffer pool, a receiving circular buffer, or other data structures for temporarily storing input messages. The receiving and processing module completes message reception, initial buffering, message organization, and task encapsulation, and writes the encapsulated processing task to the task delivery module. The receiving and processing module then transfers the encapsulated processing task to the preprocessing module via the task delivery module for processing.

[0060] The preprocessing module performs basic processing on the real-time messages or corresponding processing tasks output by the receiving processing module before the parsing stage, and generates tasks to be parsed. Basic processing may include at least one of protocol identification, message type marking, source information extraction, length verification, integrity verification, and validity verification. The preprocessing module marks, discards, or transfers tasks that do not meet format requirements or verification conditions to an exception handling process, and forwards the tasks to be parsed to the parsing processing module through the task transfer module.

[0061] The parsing module parses messages based on their type, which is identified using protocol syntax rules. It performs at least one of the following processes on the task to be parsed: field extraction, type determination, hierarchical relationship identification, or semantic conversion, to extract the target data content, identification information, and associated attribute information, and generate the parsing result. The parsing module primarily focuses on the message semantic extraction process, and its parsing results are used as input to the mapping module through the task delivery module.

[0062] The mapping processing module transforms the parsed results according to data mapping rules, generating data objects, which are then transferred to the distribution processing module via the task transfer module. Data mapping rules can be preset data models, point models, data identification rules, or business object relationships. Data transformation processing includes structure conversion, association mapping, data merging, object attribute population, or status updates. Data objects can be intermediate results or target objects. The mapping processing module and the parsing processing module are set up independently, allowing message parsing and subsequent data organization and transformation to be completed in different processing stages, thereby reducing the coupling between protocol parsing and business processing.

[0063] The distribution processing module is used to perform rule matching on data objects according to the data distribution strategy and send the distribution results to the target unit. The data distribution strategy is at least one of subscription relationship, target business category, target channel or priority rule, and the distribution method is unicast distribution, broadcast distribution, multicast distribution or multi-target parallel distribution to adapt to different business consumption scenarios. The target unit is a business unit, storage unit, display unit or external interface.

[0064] The task transfer module is used to transfer tasks between the receiving processing module, preprocessing module, parsing processing module, mapping processing module, and distribution processing module, and to achieve asynchronous decoupling between adjacent processing modules. In adjacent receiving processing modules, the subsequent processing module reads processing tasks from the preceding task transfer module. After completing its processing, the preceding module writes the processing result or the next stage task to the subsequent task transfer module. The task transfer module employs a task transfer strategy to reduce the blocking risk caused by shared processing resources between different processing modules. The task transfer strategy can be at least one of sequential write, sequential read, circular reuse, or batch transfer. The task transfer module includes at least one of a task queue, buffer, circular buffer, or cache structure. The task transfer module can complete task transfer and stage isolation between modules.

[0065] The scheduling control module is used to obtain the task load, queue length, priority, or preset scheduling rules of the receiving processing module, preprocessing module, parsing processing module, mapping processing module, and distribution processing module, and to asynchronously schedule each processing module to form a pipelined message processing link. The scheduling control module can coordinate resources and adjust scheduling parameters for each module to ensure that the system can maintain continuous processing capacity under different load conditions.

[0066] The scheduling control module can detect and analyze the system's operating load status. It can obtain information such as task load, queue length, resource usage, priority, or preset scheduling rules, and determine whether the scheduling conditions have been met based on the obtained results.

[0067] When the scheduling conditions are not met, the system maintains the current scheduling strategy, that is, it maintains the existing number of threads, thread division of labor, task allocation method and scheduling order, thereby avoiding unnecessary frequent adjustments that may affect system stability.

[0068] When the scheduling conditions of the scheduling control module are met, the scheduling control module can trigger dynamic adjustment of the number of threads, adjustment of thread division of labor, adjustment of task allocation method, adjustment of scheduling order, and trigger at least one scheduling action among batch processing, rate limiting control, delayed scheduling, priority adjustment, or backpressure control strategies. The specific scheduling action process is as follows:

[0069] When the queue length in the task delivery module corresponding to a certain processing module continuously exceeds the preset queue length threshold, or the average processing latency of the processing module exceeds the preset processing latency threshold, the scheduling control module can increase the number of processing threads corresponding to the processing module, or allocate spare processing threads to the processing module to improve the task consumption capacity of the processing module.

[0070] When the scheduling control module detects that the system CPU utilization rate continuously exceeds the preset CPU utilization threshold, the scheduling control module can limit the number of new threads, reduce the scheduling frequency of low-priority tasks, trigger batch processing to reduce the number of thread wake-ups, or perform flow control on the front-end processing module to avoid further aggravation of thread competition.

[0071] When the scheduling control module detects that the system memory usage rate continuously exceeds the preset memory usage threshold, or the cache space occupied by the task delivery module reaches the preset cache high water level, the scheduling control module can delay the scheduling of low-priority tasks, reduce the batch size of batch transfers, limit the write speed of the preceding tasks, or trigger backpressure control to reduce the task entry rate of the preceding processing module, thereby preventing the continuous growth of the cache from further increasing memory pressure.

[0072] When the queue length of the processing module drops below the recovery threshold, or when the system resource usage reaches the preset upper limit, the scheduling control module can reduce the number of corresponding processing threads to reduce resource usage and avoid thread contention.

[0073] When the load is uneven among different processing modules, the scheduling control module can adjust the thread division or task allocation method. For example, when the backlog of tasks in the parsing processing module is significantly higher than that in other processing modules, while the preprocessing or mapping processing modules are under low load, the scheduling control module can temporarily allocate reusable or shared threads to the parsing processing module, or group and allocate tasks according to message type, task priority, source channel, or task complexity, thereby reducing local processing bottlenecks.

[0074] When the number of tasks in the task transfer module reaches the batch transfer threshold, or when the task waiting time reaches the batch trigger time threshold, the scheduling control module can trigger a batch transfer strategy, transferring multiple tasks as a batch from the previous processing module to the next processing module to reduce the overhead of frequent scheduling and frequent wake-ups. When the task backlog in the next processing module exceeds the high watermark threshold, the processing latency continues to rise, or the resource usage approaches the preset limit, the scheduling control module can trigger rate limiting control or backpressure control to reduce the rate at which the previous processing module writes tasks to the task transfer module, or postpone the transfer time of low-priority tasks to prevent the task backlog from spreading further.

[0075] When tasks of different priorities exist simultaneously in the system, the scheduling control module can adjust the scheduling order according to priority, allowing high-priority tasks to enter subsequent processing modules first. For low-priority tasks, when queue pressure is high or resource consumption is high, a delayed scheduling strategy can be adopted, continuing processing only after a preset delay condition is met or the system load decreases. By dynamically adjusting the number of threads, adjusting thread division of labor, adjusting task allocation methods, adjusting scheduling order, batch processing, rate limiting control, delayed scheduling, priority adjustment, or backpressure control, the processing modules can maintain relatively stable pipeline processing capabilities under conditions of continuous high-frequency input, local congestion, or fluctuations in resource consumption.

[0076] By introducing a scheduling and control mechanism based on task load, queue status, processing capacity, and resource usage, the system can dynamically adjust the number of threads, task allocation method, and scheduling order according to the running status. When preset conditions are met, it can trigger scheduling actions such as batch processing, rate limiting control, delayed scheduling, priority adjustment, or backpressure control, thereby taking into account the requirements of high throughput, low latency, and high stability, and improving the system's adaptability to complex industrial communication scenarios.

[0077] When a significant increase in task backlog is detected in a certain processing stage, the pressure can be mitigated by increasing the number of processing threads in that stage, changing the task allocation method, or limiting the input of preceding tasks. When a temporary shortage of processing capacity is detected in a certain stage, the scheduling order or priority rules can be adjusted to allow high-priority tasks to be processed first. When the entire link is in a state of continuous high load operation, the inbound rate of preceding tasks can be controlled through backpressure control or delayed scheduling strategies to maintain the overall balance of the system.

[0078] The receiving processing module, preprocessing module, parsing processing module, mapping processing module, and distribution processing module can each be executed by an independent thread, thread group, or processing unit. Different modules are connected through a task transfer module, and the scheduling and control module performs unified or hierarchical scheduling on the above modules, thus forming a layered asynchronous system structure for real-time message processing.

[0079] In addition, embodiments of the present invention also provide a computer-readable storage medium storing a computer program. When the computer program is executed by one or more processors, it implements the aforementioned layered asynchronous processing method for real-time messages. The computer-readable storage medium can be a non-transitory storage medium, including a disk, optical disk, magnetic tape, memory chip, or other medium capable of storing a computer program.

Claims

1. A layered asynchronous processing method for real-time messages, characterized in that, Includes the following steps: Step 1: Input the real-time message into the receiving and processing layer. The receiving and processing layer has a receiving buffer structure. The receiving and processing layer writes the real-time message into the receiving buffer structure. The receiving and processing layer performs preprocessing on the received real-time message. Step 2: Transfer the real-time message or the processing task corresponding to the real-time message processed by the receiving processing layer to the preprocessing layer. The preprocessing layer performs basic processing on the real-time message or the processing task and generates a task to be parsed. Step 3: Transfer the task to be parsed to the parsing processing layer. The parsing processing layer performs parsing according to the type of the real-time message, extracts information and generates parsing results. Step 4: The parsing result is transferred to the mapping processing layer, which performs data transformation processing on the parsing result according to the data mapping rules and generates a data object; Step 5: Transfer the data object to the distribution processing layer. The distribution processing layer performs rule matching on the data object according to the data distribution strategy and sends the distribution result to the target unit. A task passing structure is provided between the receiving processing layer, the preprocessing layer, the parsing processing layer, the mapping processing layer, and the distribution processing layer. The task passing structure realizes asynchronous decoupling between adjacent processing layers. The scheduling control layer performs asynchronous scheduling of the receiving processing layer, the preprocessing layer, the parsing processing layer, the mapping processing layer, and the distribution processing layer through the task passing structure to form a pipelined message processing link.

2. The layered asynchronous processing method for real-time messages according to claim 1, characterized in that, In adjacent processing layers, the next processing layer reads the processing task from the previous task transfer structure, and after the previous processing layer completes its processing, it writes the processing result or the next stage task into the next task transfer structure.

3. The layered asynchronous processing method for real-time messages according to claim 1, characterized in that, The task delivery structure employs a task delivery strategy to reduce the risk of blocking between different processing layers due to shared processing resources.

4. The layered asynchronous processing method for real-time messages according to claim 3, characterized in that, The task delivery strategy includes at least one of sequential writing, sequential reading, cyclic reuse, or batch transfer.

5. The layered asynchronous processing method for real-time messages according to claim 1, characterized in that, The scheduling control layer obtains the operating load status of the receiving processing layer, the preprocessing layer, the parsing processing layer, the mapping processing layer, and the distribution processing layer. When the operating load status reaches the scheduling condition of the scheduling control layer, a scheduling action is triggered.

6. The layered asynchronous processing method for real-time messages according to claim 5, characterized in that, The operational load status includes task volume, queue length, processing latency, or resource usage. The scheduling actions include at least one of thread resource adjustment, task allocation adjustment, scheduling order adjustment, batch processing, rate limiting control, delayed scheduling, priority adjustment, or backpressure control.

7. A layered asynchronous processing system for real-time messages, characterized in that, Includes the following modules: The receiving and processing module is used to receive real-time messages and write the real-time messages into the receiving buffer structure to complete the pre-processing of the real-time messages. The preprocessing module is used to perform basic processing required before the parsing stage on the real-time message or the processing task corresponding to the real-time message output by the receiving processing module, and generate the task to be parsed. The parsing and processing module is used to perform parsing based on the type of real-time message, extract information, and generate parsing results. The mapping processing module is used to process the parsing results according to the data mapping rules to generate data objects; The distribution processing module is used to perform rule matching on the data object according to the data distribution strategy and send the distribution result to the target unit. The task transfer module is used to transfer tasks among the receiving processing module, the preprocessing module, the parsing processing module, the mapping processing module, and the distribution processing module, thereby achieving asynchronous decoupling between adjacent modules. The scheduling control module is used to schedule the receiving processing module, the preprocessing module, the parsing processing module, the mapping processing module and the distribution processing module through the task transmission module according to the operating load or preset scheduling rules, so as to form a pipeline-style message processing link.

8. The layered asynchronous processing system for real-time messages according to claim 7, characterized in that, In adjacent processing modules, the next processing module reads the processing task from the previous task transfer module. After the previous processing module completes its processing, it writes the processing result or the next stage task into the next task transfer module.

9. A layered asynchronous processing system for real-time messages according to claim 7, characterized in that, The task delivery module adopts a task delivery strategy to reduce the risk of blocking between different processing modules due to shared processing resources.

10. A layered asynchronous processing system for real-time messages according to claim 9, characterized in that, The task delivery strategy includes at least one of sequential writing, sequential reading, cyclic reuse, or batch transfer.

11. A layered asynchronous processing system for real-time messages according to claim 7, characterized in that, The scheduling control module obtains the operating load status of the receiving processing module, the preprocessing module, the parsing processing module, the mapping processing module, and the distribution processing module. When the operating load status reaches the scheduling conditions of the scheduling control module, a scheduling action is triggered.

12. A layered asynchronous processing system for real-time messages according to claim 11, characterized in that, The operational load status includes task volume, queue length, processing latency, or resource usage. The scheduling actions include at least one of thread resource adjustment, task allocation adjustment, scheduling order adjustment, batch processing, rate limiting control, delayed scheduling, priority adjustment, or backpressure control.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the layered asynchronous processing method for real-time messages as described in any one of claims 1 to 6.

Images