Industrial edge data adaptive routing system and method

Abstract

The invention provides a self-adaptive routing system and method for industrial edge data, and the system comprises a transponder registry module, a high-performance buffer module, a batch processing event processing module, an asynchronous forwarding coordination module and a multi-protocol transponder realization module, and the transponder registry module realizes unified registration and dynamic management of a transponder. A high-performance buffer module ensures high-efficiency flow of data through a lock-free annular buffer area, a batch processing event processing module performs intelligent aggregation on the data, an asynchronous forwarding coordination module performs intelligent routing decision based on registry information, and a multi-protocol transponder implementation module provides multi-protocol support. According to the architecture, the system can dynamically adapt to routing change and support real-time adjustment of the routing strategy of the data repeater under the condition of no shutdown, the problem of low forwarding efficiency caused by a fixed forwarding path of a traditional edge gateway can be solved, and the forwarding efficiency of the industrial edge gateway is effectively improved.

Description

Technical Field

[0001] This application relates to the field of industrial control technology, and in particular to an industrial edge data adaptive routing system and method. Background Technology

[0002] In the context of the deep integration of Industrial Internet of Things (IIoT) and edge computing, industrial edge gateways, as the core hub connecting field devices and cloud platforms, have crucial data routing and forwarding capabilities. Edge gateways deployed in industrial fields need to simultaneously process high-frequency vibration, temperature, and pressure data (acquired in milliseconds) from thousands of sensors, as well as low-frequency but high-priority control data such as device status and alarm commands from programmable logic controllers (PLCs). This data needs to be forwarded reliably to different backend systems in real time, based on its type and purpose. However, existing edge data forwarding solutions mostly adopt a "statically configured multi-target forwarding mode," where the data forwarding logic and specific forwarding target addresses are fixed through hard coding during gateway system initialization. This mode has a fatal flaw: when a new forwarding target needs to be added or existing routing rules need to be modified, the entire gateway system must be shut down, the source code modified, and the entire system recompiled and redeployed. This routing mode lacks flexibility and affects data forwarding efficiency. Summary of the Invention

[0003] This application provides an industrial edge data adaptive routing system and method to solve the problem that the fixed forwarding path of industrial edge gateways affects data forwarding efficiency in related technologies.

[0004] The first aspect of this application provides an industrial edge data adaptive routing system, comprising: The repeater registry module is used to register the discovered available data repeaters and generate the repeater registry when the adaptive routing system starts up. A high-performance buffer module is used to receive and buffer data points from the data acquisition thread via a lock-free circular buffer; The batch processing event processing module is used to group the data points according to the topic identifier of the data points, and generate a data batch corresponding to each topic identifier when a preset trigger condition is met. An asynchronous forwarding coordination module is used to receive the data batch and call the corresponding target data forwarder in the forwarder registry according to the topic identifier; The multi-protocol forwarder implementation module is used to control the target data forwarder to call the corresponding forwarding protocol in a dedicated forwarding thread pool to execute the forwarding task corresponding to the data batch.

[0005] A second aspect of this application provides an adaptive routing method for industrial edge data, which is applied to an adaptive routing system for industrial edge data. The adaptive routing method for industrial edge data includes: When the adaptive routing system starts up, it registers the discovered available data forwarders and generates a forwarder registry. Data points from the data acquisition thread are received and buffered through an unlockless circular buffer. The data points are grouped according to their topic identifiers, and when a preset trigger condition is met, a data batch corresponding to each topic identifier is generated. The target data forwarder in the forwarder registry is invoked based on the topic identifier; The target data forwarder is controlled to execute the forwarding task corresponding to the data batch by calling the corresponding forwarding protocol in a dedicated forwarding thread pool.

[0006] In summary, the industrial edge data adaptive routing system and method provided in this application utilizes a repeater registry module to achieve unified registration and dynamic management of repeaters, laying the foundation for flexible routing. A high-performance buffer module ensures efficient data flow through a lock-free circular buffer. A batch event processing module intelligently aggregates data. An asynchronous forwarding coordination module executes intelligent routing decisions based on registry information. A multi-protocol repeater implementation module provides multi-protocol support. This architecture enables the system to dynamically adapt to routing changes without downtime, supports real-time adjustment of data repeater routing strategies, and solves the problem of low forwarding efficiency caused by fixed forwarding paths in traditional edge gateways, effectively improving the forwarding efficiency of industrial edge gateways. Attached Figure Description

[0007] Figure 1 A schematic diagram of the modules of the industrial edge data adaptive routing system provided in the embodiments of this application; Figure 2 This is a flowchart illustrating the adaptive routing method for industrial edge data provided in an embodiment of this application. Detailed Implementation

[0008] To make the inventive objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0009] To address the issue of fixed forwarding paths and lack of flexibility in related technologies, embodiments of this application provide an adaptive routing system for industrial edge data, such as... Figure 1 This is a flowchart illustrating the industrial edge data adaptive routing system provided in this embodiment. The industrial edge data adaptive routing system includes: The repeater registry module is used to register the discovered available data repeaters and generate the repeater registry when the adaptive routing system starts up. A high-performance buffer module is used to receive and buffer data points from the data acquisition thread via a lock-free circular buffer; The batch event processing module is used to group data points according to their topic identifiers and generate data batches corresponding to each topic identifier when preset trigger conditions are met. The asynchronous forwarding coordination module is used to receive data batches and call the corresponding target data forwarder in the forwarder registry according to the topic identifier; The multi-protocol forwarder implementation module is used to control the target data forwarder to call the corresponding forwarding protocol in a dedicated forwarding thread pool to execute the forwarding task corresponding to the data batch.

[0010] In this embodiment, the process of registering discovered available data forwarders when the adaptive routing system starts relies on scanning the capabilities of the forwarders and collecting metadata. By performing health checks and attribute readings on externally connectable forwarding endpoints, the system records the name, supported transport protocols, acceptable topic range, and load capacity, constructing a forwarder registry. The forwarder registry is a queryable data structure used to quickly locate data forwarders capable of handling a given topic based on its topic identifier (a topic identifier is an identifier used to mark the category or source of data content; data belonging to the same logical category can be centrally managed through topic identifiers). Internally, it binds topic identifiers to target data forwarders using key-value mapping, enabling accurate routing of subsequent data batches. After initializing this registry, the adaptive routing system can enter the data flow processing phase.

[0011] The high-performance buffer module employs a lock-free circular buffer for event reception and temporary storage. This lock-free circular buffer is a highly efficient data storage structure implemented through circular indexing, supporting multi-threaded concurrent access without traditional lock mechanisms, thus improving data write and read efficiency. During operation, the lock-free circular buffer continuously receives data points from the data acquisition thread and internally reuses slots in a circular structure, achieving concurrent writes and reads through coordination between producer and consumer sequence numbers. Because the lock-free structure avoids the overhead of mutual exclusion locks between threads, it can maintain stable throughput under high-frequency data injection conditions. The circular buffer outputs a data point stream to the upper-layer module and passes the data points to the grouping logic for further processing. The grouping logic maps the data points to corresponding independent batch containers based on the topic identifier carried within each data point and continuously accumulates data points within these containers.

[0012] The batch event processing module group and aggregates data points based on independent batch containers identified by topic. Therefore, it maintains a separate buffer set and first timestamp for each topic, which allows different batch sizes and timeout thresholds to be configured for different topics. The module determines that a batch is ready through a triple triggering mechanism (reaching the batch threshold, exceeding the timeout threshold, and receiving the buffer end flag). Then, it packages all data points in the ready container into a batch object of the associated topic and delivers it to the asynchronous forwarding coordination module asynchronously. At the same time, it ensures that high-frequency topics do not block low-frequency topics, thereby achieving a trade-off between fairness and low latency.

[0013] The asynchronous forwarding coordination module receives data batches and queries the forwarder registry based on the batch's topic identifier to determine the target forwarder. It also performs health checks and selection strategy judgments on the query results. Subsequently, it encapsulates the data batch and the target forwarder into an asynchronous forwarding task and submits it to a dedicated forwarding thread pool. This module is responsible for maintaining the task lifecycle, including retry strategies, priority control, and timeout monitoring. It also triggers fallback or rerouting decisions when forwarding fails or the forwarder's state changes, thereby decoupling the batch processing layer from the protocol execution layer.

[0014] After receiving a forwarding task from the asynchronous forwarding coordination module, the multi-protocol forwarder implementation module selects the appropriate sending logic based on the forwarding type bound to the target data forwarder and completes protocol execution within a dedicated forwarding thread pool. To maintain structural uniformity, this module provides a consistent entry point for different transmission methods through a protocol adaptation layer. This ensures that when a data batch enters the protocol processing stage, it only needs to provide a topic identifier, payload content, and necessary metadata, while the protocol adaptation layer is responsible for converting the batch into a sendable format for the corresponding protocol. When the forwarding type is Kafka sending, the module encapsulates the batch content into a message oriented towards a Kafka topic and sends it through the asynchronous interface of the Kafka client. The built-in acknowledgment mechanism of this interface can be used to record the task completion status, enabling the coordination module to manage subsequent processes based on the actual results. When the forwarding type is MQTT sending, the module publishes the batch content through a maintained MQTT session channel and manages the sending completion status according to the MQTT acknowledgment method. Because MQTT has topic routing and message acknowledgment semantics, messages can be distributed to the corresponding endpoints according to their topic paths after entering the session channel. When the forwarding type is UDP transmission, the module packages the batch content into data packets and sends them directly through the established UDP channel. Since UDP itself does not guarantee reliability, the module only performs lightweight transmission monitoring when necessary, to report potential connection instability to the coordination module. The entire multi-protocol forwarder implementation module forms a complete link through protocol adaptation, thread pool execution, and status feedback. This ensures that different transmission methods maintain a consistent entry point at the logical level and complete the actual transmission according to protocol characteristics at the execution level, thereby achieving a unified structure and well-defined behavior for multi-protocol processing in high-concurrency batch forwarding scenarios.

[0015] Preferably, the forwarder registry module includes: a first-level routing unit, a second-level routing unit, and a third-level routing unit; the first-level routing unit is used to receive a topic identifier and search for a first data forwarder that has established a binding relationship with the topic identifier in the forwarder registry. If the query is successful, the first data forwarder is used as the target data forwarder; the second-level routing unit is used to count the activation status of all data forwarders in the forwarder registry when the query of the first-level routing unit fails. If there is only one second data forwarder that is active, the second data forwarder is used as the target data forwarder; the third-level routing unit is used to obtain a preset default forwarder identifier when the query of the second-level routing unit fails, and determine the corresponding third data forwarder from the forwarder registry according to the default forwarder identifier, and use the third data forwarder as the target data forwarder.

[0016] In this embodiment, implementing the three-level routing mechanism involves the collaborative work of multiple consecutive routing units, with each unit's design logic and functional objectives closely integrated. The core function of the first-level routing unit is to receive a topic identifier and search for binding relationships in the forwarder registry based on that identifier. Upon receiving a topic identifier, the first-level routing unit searches the registry for the first data forwarder bound to that topic. If the search is successful (i.e., a forwarder bound to the topic identifier exists), this forwarder is returned as the target data forwarder to the asynchronous forwarding coordination module, thus completing the direct routing of specific data. When the first-level routing unit fails to return a query result, the second-level routing unit intervenes to perform further processing. The second-level routing unit monitors the activation status of all data forwarders, indicating whether the forwarder is in a running state capable of processing data. The second-level routing unit performs activation status statistics on all data forwarders in the registry. If the statistics show that only one second data forwarder is active, that forwarder is directly selected as the target data forwarder and returned. This mechanism ensures that, under certain circumstances, even if a topic identifier is not bound to a corresponding forwarder, data forwarding can still be completed through the only available active forwarder. When the secondary routing unit still hasn't returned a query result, the tertiary routing unit provides a final fallback solution. The tertiary unit retrieves a pre-configured default forwarder identifier, which specifies a generic forwarder to ensure data processing. Based on this identifier, it determines the corresponding third data forwarder from the forwarder registry and returns it as the target data forwarder. This mechanism ensures that even if the first two routing units fail to match specific conditions, the data still has a reliable processing path. The entire three-tier routing mechanism forms a complete data forwarding strategy through direct matching at the primary level, active state selection at the secondary level, and a default fallback at the tertiary level, enabling data to find the most suitable processing object under different circumstances and avoiding situations where there is no processing path.

[0017] Preferably, the high-performance buffer module includes: a ring array storage unit, a lock-free publishing coordination unit, and a waiting policy control unit; the ring array storage unit is used to pre-allocate a ring array with a fixed capacity during the initialization phase of the adaptive routing system, and create reusable event objects in the slots of the ring array; the lock-free publishing coordination unit is used to receive concurrent publishing requests from the data acquisition thread, allocate ring array slots for the data points to be published through the producer sequence number mechanism, and fill the event objects in the ring array slots with the data points; the waiting policy control unit is used to coordinate the speed difference between the consumer thread and the producer thread through a multi-stage state check process of the waiting policy when the data consumer thread reads data from the ring array storage unit.

[0018] In this embodiment, the circular array storage unit pre-allocates a fixed-capacity circular array during the adaptive routing system initialization phase. A circular array is a logically connected linear data structure where the next position after the end of the array points to the starting position, forming a closed-loop storage structure. Reusable event objects are created in each slot of the array. These event objects carry data point information and provide encapsulation and management of the data, ensuring efficient reuse during subsequent data publishing and consumption without frequent memory allocation and reclamation operations. The core objective of the lock-free publishing coordination unit is to receive concurrent publishing requests from the data acquisition thread and ensure data consistency and ordering under high concurrency conditions. The lock-free mechanism means that traditional mutex locks are not used during concurrent access, thus avoiding thread blocking and improving throughput. This unit allocates circular array slots for each data point to be published using a producer sequence number mechanism. The producer sequence number is an incrementing identifier used to identify the data production order, ensuring that the writing order of data in the circular array is consistent with the publishing order. After allocating slots, the lock-free publishing coordination unit fills the corresponding event object with the data point, completing the data storage. For example, if three data acquisition threads generate data simultaneously, the lock-free mechanism and sequence number allocation ensure that these three data entries are written to different slots in the circular array without overwriting each other, while maintaining the integrity of the data order within the array. The waiting strategy control unit plays a crucial role when the data consumption thread reads the circular array. Its main function is to coordinate the speed difference between the consumption and production threads, preventing data contention or delays caused by the consumer being too fast or too slow. The waiting strategy is implemented through a multi-stage state check process. The first stage is spin waiting, where the consumption thread rapidly and repeatedly checks whether the slots are filled within a short period to provide extremely low latency response capabilities. When the production speed lags slightly, spin waiting can allow data to arrive in a very short time, achieving non-blocking reading. If no data is detected during the spin stage, the waiting strategy enters the second stage, where the CPU yields. This involves the consumption thread voluntarily relinquishing scheduling control, allowing the operating system to schedule other threads to run, thus avoiding long-term CPU occupation and resource waste. If the target slot is still not filled after the concession, the process enters the third stage, which is a short sleep. This reduces CPU consumption by hibernating for a very short time slice, while giving the production thread enough time to complete data writing. The waiting strategy control unit can also dynamically adjust the waiting mode, optimizing the waiting time based on the occupancy of the circular array and the processing capacity of the consumer thread, thereby achieving traffic matching between producers and consumers.The entire adaptive routing system provides a fixed-capacity high-speed buffer through a ring array. The lock-free publishing coordination unit ensures that data points are written safely and sequentially under high concurrency. The waiting policy control unit achieves smooth connection between production and consumption through multi-stage checks and intelligent waiting mechanisms, which can improve data transmission and processing efficiency and maximize the use of computing resources while ensuring data integrity and order.

[0019] Preferably, the batch processing event processing module includes: a group container management unit, a batch trigger judgment unit, and a batch routing output unit; the group container management unit is used to distribute data points to independent batch containers corresponding to their respective topic identifiers for temporary storage based on the topic identifiers carried by the data points; the batch trigger judgment unit is used to monitor the number of data points in each independent batch container, the dwell time of the first data point, and receive a buffer end signal from the high-performance buffer module; when the number of data points reaches a preset size threshold, or the dwell time exceeds a preset timeout threshold, or a batch end signal is received, a batch ready signal is generated; the batch routing output unit is used to extract all data points from the corresponding independent batch container in response to the batch ready signal, assemble them into a data batch bound to the topic identifier, and send the data batch to the asynchronous forwarding coordination module.

[0020] In this embodiment, the group container management unit undertakes the core functions of temporary storage and distribution in the adaptive routing system. Its main role is to distribute data points to independent batch containers corresponding to their respective topic identifiers based on the topic identifiers carried by the data points. The group container management unit quickly locates the corresponding independent batch container based on the topic identifier of each data point. If the container does not exist, it dynamically creates a new batch container to ensure that the data of each topic can be stored independently and isolated, preventing data from different topics from interfering with each other. For example, in an IoT data acquisition scenario, data points from temperature sensors and humidity sensors are distributed to different batch containers so that temperature and humidity data can be analyzed and forwarded independently later. The batch trigger judgment unit plays a role in controlling batch generation during the data temporary storage process. Its core function is to monitor the number of data points in each independent batch container, the residence time of the first data point, and receive the buffer end signal from the high-performance buffer module. The residence time represents the time interval in which data points are stored in the batch container. By monitoring the residence time, it is possible to avoid data staying in the container for too long, which would lead to increased latency. The batch triggering judgment unit generates a batch ready signal based on various triggering conditions. A batch ready signal is triggered when the number of data points in an independent batch container reaches a preset size threshold, the dwell time of the first data point exceeds a preset timeout threshold, or a buffer end signal is received. For example, in a financial market data processing scenario, if a batch container contains 100 transaction records or the first record remains in the container for more than 50 milliseconds, the batch triggering judgment unit will generate a ready signal to facilitate timely batch processing, thus balancing throughput and latency performance. Upon receiving the batch ready signal, the batch routing output unit extracts all data points from the corresponding independent batch container and assembles them into data batches according to topic identifiers. Data batches are logical units bound according to topic identifiers, facilitating subsequent asynchronous forwarding and centralized processing by the adaptive routing system. After assembly, the batch routing output unit sends the data batches to the asynchronous forwarding coordination module. The asynchronous forwarding coordination module can independently process data from different batches, achieving data forwarding and distribution under high concurrency. For example, when a batch of temperature sensor data points is ready, the batch routing output unit packages all temperature data in that batch into a single batch and sends it to the asynchronous forwarding coordination module. Humidity data batches, however, remain independent, preventing data from different topics from being mixed, thereby improving the controllability and efficiency of data processing. This entire mechanism achieves efficient, low-latency data batching management through the close cooperation of the group container management unit, the batch trigger judgment unit, and the batch routing output unit. The group container management unit ensures that data is temporarily stored independently by topic, the batch trigger judgment unit ensures that batch generation occurs at the appropriate time, and the batch routing output unit ensures that batch data can be delivered to the asynchronous forwarding module quickly and accurately.In this way, stability and performance can be maintained in high-concurrency, multi-topic data environments, while taking into account data integrity, processing order, and response latency.

[0021] Preferably, the asynchronous forwarding coordination module includes: a forwarder decision unit, a task encapsulation and scheduling unit, and an execution status monitoring unit; the forwarder decision unit is used to receive data batches from the batch processing event processing module, extract the topic identifier bound to the data batch, and initiate a query request to the forwarder registry module according to the topic identifier to obtain the target data forwarder; the task encapsulation and scheduling unit is used to associate and encapsulate the data batch with the target data forwarder, generate an asynchronous forwarding task, and submit the asynchronous forwarding task to a dedicated forwarding thread pool for queuing and execution; the execution status monitoring unit is used to start a timeout timer when the asynchronous forwarding task is executed in the dedicated forwarding thread pool, and if no task completion notification is received within the preset timeout threshold, a forwarding timeout signal is generated and the corresponding task cancellation process is triggered.

[0022] In this embodiment, the forwarder decision unit plays a crucial role in routing and target forwarder selection throughout the entire data batch processing flow. Its main task is to receive data batches from the batch processing event processing module and extract the bound topic identifiers. The topic identifier identifies the logical category or source of the data batch, accurately mapping data batches of the same type to the corresponding processing channel. After extracting the topic identifier, the forwarder decision unit initiates a query request to the forwarder registry module to obtain the target data forwarder corresponding to that topic identifier. The target data forwarder includes a first data forwarder, a second data forwarder, or a third data forwarder. The first data forwarder is directly bound to the topic identifier; the second data forwarder is selected based on the uniqueness of its activation state when the first data forwarder fails to match; and the third data forwarder is a fallback default forwarder, ensuring that any unmatched data batch can still obtain a processing path. For example, if the subject of a data batch is identified as "order processing," the forwarder decision unit will query the registry. If a first data forwarder exists, it will be selected directly; if no first forwarder exists and only one is active, the second data forwarder will be selected; otherwise, the default third data forwarder will be selected, ensuring that the data batch can be effectively routed. After determining the target data forwarder, the task encapsulation and scheduling unit will associate the data batch with the target data forwarder to generate asynchronous forwarding tasks. Asynchronous forwarding tasks are logical units within the adaptive routing system used for non-blocking data processing. They bind the content of the data batch with the processing logic of the forwarder, thereby enabling independent processing and transmission of the data batch. The generated asynchronous forwarding tasks are then submitted to a dedicated forwarding thread pool for queued execution. The dedicated forwarding thread pool is a set of independent execution threads used to concurrently process asynchronous tasks, thereby improving the throughput of the adaptive routing system and isolating the execution impact between different tasks. For example, when multiple data batches are ready to be sent to different target forwarders simultaneously, the task encapsulation and scheduling unit will generate independent asynchronous tasks for each data batch and place them in the thread pool for queuing. The threads in the thread pool will then execute these tasks in parallel, achieving high-concurrency batch forwarding of data. The execution status monitoring unit plays a crucial role in the asynchronous forwarding task execution process. It starts a timeout timer for each task, which monitors whether the task is completed within a preset time to ensure that the adaptive routing system can respond promptly to abnormal or delayed tasks. If no task completion notification is received within the preset timeout threshold, the execution status monitoring unit generates a forwarding timeout signal and triggers the corresponding task cancellation process. The task cancellation process interrupts the unfinished asynchronous task and releases related resources to prevent prolonged occupation of thread pool resources and system blocking. For example, if a data batch cannot be forwarded in time due to network anomalies, the cancellation process will be triggered after the timeout timer expires. At the same time, the exception information will be recorded, and the batch can be rescheduled or recorded in the error log for subsequent processing.The entire process, through the close cooperation of the forwarder decision-making unit, task encapsulation and scheduling unit, and execution status monitoring unit, realizes a complete link for data batches from topic identification to target forwarding, asynchronous scheduling, and status monitoring. The forwarder decision-making unit ensures that data batches can be accurately routed, the task encapsulation and scheduling unit ensures that data batches are executed safely in parallel under high concurrency environments, and the execution status monitoring unit maintains the stability and reliability of the system through timeout mechanisms and task cancellation procedures.

[0023] Preferably, the execution status monitoring unit further includes: a buffer overflow detection subunit, used to monitor the lockless circular buffer of the high-performance buffer module, and when the buffer capacity is detected to be exhausted, triggering data point discarding or publishing blocking process according to the preset overflow handling strategy; a forwarder fault detection subunit, used to receive forwarder health status change notifications from the forwarder registry module, and when it is determined that the target data forwarder is in a fault state, triggering the process of routing subsequent data batches to the backup data forwarder.

[0024] In this embodiment, the buffer overflow detection subunit monitors the buffer capacity usage in real time. When the buffer capacity is exhausted, it handles the overflow according to a preset overflow handling strategy. The overflow handling strategy includes two methods: data point discarding and publishing blocking. The former discards newly arriving data points to avoid overwriting existing data, while the latter blocks the data publishing thread until the buffer has freed up space, thus ensuring data integrity. For example, in high-frequency data acquisition scenarios, if the amount of data points generated in a short period is too large, causing the buffer to fill up, the overflow detection subunit can trigger a blocking mechanism to temporarily stop new data writing, and continue publishing only after existing data has been consumed, thereby balancing system load and preventing data loss. The repeater fault detection subunit plays a crucial role in ensuring the reliability of data forwarding. It determines the availability of the target data repeater by receiving repeater health status change notifications from the repeater registry module. The repeater health status includes indicators such as active, faulty, or unavailable. When a target data repeater is detected to be in a faulty state, the repeater fault detection subunit triggers a process to route subsequent data batches to a backup data repeater. The backup data forwarder is a pre-configured redundant forwarder used to ensure uninterrupted data transmission in the event of a failure in the primary forwarder. For example, if the primary data forwarder used to process order data fails, the fault detection subunit will immediately route subsequent batches of order data to the backup data forwarder, ensuring the continuity of data forwarding and system stability. Simultaneously, it records the fault event in the background for maintenance personnel to troubleshoot and repair. The forwarding timeout detection subunit, buffer overflow detection subunit, and forwarder fault detection subunit together constitute a highly reliable monitoring mechanism. By identifying anomalies such as asynchronous task timeouts, buffer overloads, and forwarder failures in real time, it achieves comprehensive protection for the data batch processing flow. Forwarding timeout detection ensures tasks complete within a specified time; buffer overflow detection maintains a stable data flow under high concurrency; and forwarder fault detection ensures data can be flexibly switched to available forwarders, thereby improving the system's robustness, throughput, and fault tolerance in complex asynchronous environments.

[0025] According to the industrial edge data adaptive routing system provided in this application, the repeater registry module realizes unified registration and dynamic management of repeaters, laying the foundation for flexible routing. The high-performance buffer module ensures efficient data flow through a lock-free circular buffer. The batch event processing module intelligently aggregates data. The asynchronous forwarding coordination module executes intelligent routing decisions based on registry information. The multi-protocol repeater implementation module provides multi-protocol support. This architecture enables the system to dynamically adapt to routing changes without downtime, supports real-time adjustment of data repeater routing strategies, and solves the problem of low forwarding efficiency caused by fixed forwarding paths in traditional edge gateways, effectively improving the forwarding efficiency of industrial edge gateways.

[0026] Figure 2 This application provides an industrial edge data adaptive routing method, which is applied to the industrial edge data adaptive routing system described in the foregoing embodiments. For example... Figure 2 As shown, this industrial edge data adaptive routing method mainly includes: Step 210: When the adaptive routing system starts up, register the discovered available data forwarders and generate a forwarder registry. Step 220: Receive and buffer data points from the data acquisition thread through a lock-free circular buffer; Step 230: Group the data points according to their theme identifiers, and generate data batches corresponding to each theme identifier when the preset trigger conditions are met; Step 240: Invoke the corresponding target data forwarder in the forwarder registry based on the topic identifier; Step 250: Control the target data forwarder to call the corresponding forwarding protocol in the dedicated forwarding thread pool to execute the forwarding task corresponding to the data batch.

[0027] In this embodiment, the process of registering available data forwarders discovered during the startup of the adaptive routing system relies on scanning the forwarder's capabilities and collecting metadata. By performing health probes and attribute readings on externally connectable forwarding endpoints, the system records the name, supported transport protocols, acceptable topic range, and load capacity, constructing a forwarder registry. After initializing this registry, the system can enter the data stream processing phase. The lock-free circular buffer continuously receives data points from the data acquisition thread during operation and internally reuses slots in a circular structure, achieving concurrent writing and reading through the coordination of producer and consumer sequence numbers. Because the lock-free structure avoids the overhead of mutual exclusion locks between threads, it can maintain stable throughput under high-frequency data injection conditions. The circular buffer outputs a data point stream to the upper-layer module and passes the data points to the packetization logic for further processing. The packetization logic maps the data points to corresponding independent batch containers based on the topic identifiers carried within the data points and continuously accumulates data points within these containers. When the number of data points in a single batch container reaches a size threshold, or the dwell time of the first data point exceeds a timeout threshold, or a batch end signal is received from the buffer module, the contents of the container are considered ready to form a data batch. After generation, the data batch carries a topic identifier and is transmitted to the forwarding coordinator (part of the asynchronous forwarding coordination module). The forwarding coordinator queries the forwarder registry based on the topic identifier bound to the batch to locate the target data forwarder and selects a forwarding endpoint that matches the topic and is currently in a healthy state. After determining the target data forwarder, the corresponding forwarding protocol execution task is scheduled in a dedicated forwarding thread pool. The execution thread within the thread pool retrieves the asynchronous forwarding task from the task queue, calls the protocol logic bound to the target data forwarder (e.g., incremental push based on long connections or batch forwarding based on message channels), and submits the data batch to the target endpoint. This process is completely isolated from the acquisition thread and grouping logic, thus maintaining stable processing performance in high-concurrency environments. The entire process forms a continuous link through registry-driven forwarder selection, lock-free buffer-driven data input, high-precision batch triggering mechanism, and protocol execution logic within the thread pool, making the data path from collection to forwarding clear, controllable, and with high throughput characteristics.

[0028] In one optional implementation of this embodiment, the step of calling the corresponding target data forwarder in the forwarder registry based on the topic identifier includes: searching for a first data forwarder in the forwarder registry that is bound to the topic identifier; if the search is successful, returning the first data forwarder as the query result; when the first-level routing unit does not return a query result, counting the activation status of all data forwarders in the forwarder registry; if only one second data forwarder is active, returning the second data forwarder as the query result; when the second-level routing unit does not return a query result, obtaining a preset default forwarder identifier; determining the corresponding third data forwarder from the forwarder registry based on the default forwarder identifier, and returning the third data forwarder as the query result.

[0029] The specific scheme of this embodiment is consistent with the routing modes of the first-level routing unit, second-level routing unit and third-level routing unit in the above adaptive routing system, and will not be described in detail here.

[0030] In one optional implementation of this embodiment, the step of generating data batches corresponding to each topic identifier includes: starting parallel container monitoring for the independent batch containers corresponding to the topic identifiers; continuously acquiring the current number of data points in the independent batch containers during container monitoring and comparing it with a preset size threshold; recording the injection timestamp of the first data point in the independent batch container, calculating the corresponding retention time based on the current time, and comparing it with a preset timeout threshold; determining that the data in the independent batch containers is ready when the current number of data points reaches the size threshold, the retention time exceeds the timeout threshold, or a batch end signal is received, and generating data batches corresponding to the topic identifiers.

[0031] In this embodiment, the core of implementing parallel container monitoring for independent batch containers corresponding to the topic identifier lies in real-time tracking of the status of each container to ensure that data batches can be assembled and sent in a timely manner when conditions are met. Container monitoring continuously acquires the current number of data points within the independent batch container and compares this number with a preset size threshold. The size threshold is used to determine whether the batch capacity has reached the processing standard, thereby deciding whether to trigger the batch assembly operation. Simultaneously, during the monitoring process, the injection timestamp of the first data point in the container is recorded. The timestamp identifies the specific time the data point was placed into the container. By comparing it with the current system time, the dwell time is calculated to assess the time the data point stays in the container. The dwell time is compared with a preset timeout threshold, which is used to control the maximum waiting time of data in the container, avoiding data processing delays caused by incomplete batches. During monitoring, when the number of data points reaches a certain threshold, the dwell time of the first data point exceeds a timeout threshold, or a batch end signal is received, the system determines that the data in that independent batch container is ready, thus triggering batch assembly and transmission operations. The batch assembly operation packages all data points in the container into data batches according to topic identifiers, ensuring the integrity and order of data on the same topic during transmission, achieving efficient data flow. By monitoring multiple independent batch containers in parallel, the system can simultaneously track the data status of different topics, enabling concurrent assembly and transmission of multi-topic batches. This ensures timely data processing, fully utilizes system resources to improve throughput, and avoids data delays caused by backlog in a single container in high-concurrency scenarios, providing reliable support for efficient management of overall batch processing events.

[0032] In one optional implementation of this embodiment, the step of controlling the target data forwarder to execute forwarding tasks in a dedicated forwarding thread pool includes: encapsulating data batches with the target data forwarder to generate asynchronous forwarding tasks, and submitting the asynchronous forwarding tasks to the task queue of the dedicated forwarding thread pool; starting a timeout timer corresponding to the asynchronous forwarding task and recording the task forwarding duration; monitoring the execution status of the asynchronous forwarding task, and when the task forwarding duration exceeds a preset submission duration threshold, generating a task termination instruction and terminating the execution of the asynchronous forwarding task.

[0033] In this embodiment, the core of encapsulating data batches with the target data forwarder lies in binding data with processing logic to form asynchronous forwarding tasks, thereby achieving non-blocking data transmission. Task encapsulation combines the content of the data batch with the processing interface of the target forwarder to generate independent asynchronous forwarding tasks, ensuring that each task can execute independently without interfering with other tasks. The generated asynchronous forwarding tasks are then submitted to the task queue of a dedicated forwarding thread pool. The thread pool is a collection of multiple execution threads used to concurrently process asynchronous tasks. Queue management ensures that tasks are scheduled and executed efficiently in the order of submission. After a task is submitted, the system starts a timeout timer corresponding to the asynchronous forwarding task to record the forwarding duration and continuously monitor the task execution status to determine whether the task is completed or an error has occurred. During monitoring, the system continuously calculates the forwarding duration of the task through the timeout timer. If the forwarding duration exceeds a preset submission duration threshold, the task execution will be judged as abnormal or an unacceptable delay. At this time, the system generates a task termination instruction to forcibly interrupt the execution of the asynchronous forwarding task, releasing the occupied threads and related resources to prevent the task from occupying the thread pool for a long time, causing delays or blocking of other tasks. For example, in a financial market data push scenario, if a data batch fails to be sent to the target forwarder within 500 milliseconds, the monitoring logic will trigger a task termination command, interrupting further processing of the current task and recording the task in the exception management module for subsequent retry or log analysis, thereby ensuring the overall stability and high throughput of the system. Through the combination of task encapsulation, asynchronous scheduling, and timeout monitoring, the system can safely and efficiently manage the forwarding of data batches under high concurrency conditions. Task encapsulation ensures that data and forwarding logic are bound together and can be executed independently, while the thread pool queue provides concurrent execution capabilities. Timeout timers and execution status monitoring ensure that tasks do not wait indefinitely or consume resources, thus achieving the reliability, controllability, and low latency characteristics of asynchronous forwarding, providing a solid guarantee for real-time data processing and high-performance message forwarding.

[0034] According to the adaptive routing method for industrial edge data provided in this application, when the adaptive routing system starts, it registers the discovered available data forwarders and generates a forwarder registry; it receives and buffers data points from the data acquisition thread through a lock-free circular buffer; it groups the data points according to their topic identifiers, and generates data batches corresponding to each topic identifier when preset trigger conditions are met; it calls the corresponding target data forwarder in the forwarder registry according to the topic identifier; and it controls the target data forwarder to execute the forwarding task corresponding to the data batch by calling the corresponding forwarding protocol in a dedicated forwarding thread pool. This application can dynamically adapt to routing changes, supports real-time adjustment of the routing strategy of data forwarders, and can solve the problem of low forwarding efficiency caused by fixed forwarding paths in traditional edge gateways, effectively improving the forwarding efficiency of industrial edge gateways.

[0035] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An industrial edge data adaptive routing system, characterized by, include: The repeater registry module is used to register the discovered available data repeaters and generate the repeater registry when the adaptive routing system starts up. A high-performance buffer module is used to receive and buffer data points from the data acquisition thread via a lock-free circular buffer; The batch processing event processing module is used to group the data points according to the topic identifier of the data points, and generate a data batch corresponding to each topic identifier when a preset trigger condition is met. An asynchronous forwarding coordination module is used to receive the data batch and call the corresponding target data forwarder in the forwarder registry according to the topic identifier; The multi-protocol forwarder implementation module is used to control the target data forwarder to call the corresponding forwarding protocol in a dedicated forwarding thread pool to execute the forwarding task corresponding to the data batch.

2. The industrial edge data adaptive routing system of claim 1, wherein, The repeater registry module includes: a primary routing unit, a secondary routing unit, and a tertiary routing unit; The first-level routing unit is used to receive the topic identifier and search for the first data forwarder that has established a binding relationship with the topic identifier in the forwarder registry. If the query is successful, the first data forwarder is used as the target data forwarder. The secondary routing unit is used to count the activation status of all data forwarders in the forwarder registry when the primary routing unit fails to query. If there is only one active second data forwarder, then the second data forwarder is used as the target data forwarder. The third-level routing unit is used to obtain a preset default forwarder identifier when the second-level routing unit fails to query, and to determine the corresponding third data forwarder from the forwarder registry based on the default forwarder identifier, and to use the third data forwarder as the target data forwarder.

3. The industrial edge data adaptive routing system of claim 1, wherein, The high-performance buffer module includes: a ring array storage unit, a lock-free release coordination unit, and a wait strategy control unit; The circular array storage unit is used to pre-allocate a circular array with a fixed capacity during the system initialization phase, and to create reusable event objects in the slots of the circular array. The lock-free publishing coordination unit is used to receive concurrent publishing requests from the data acquisition thread, allocate circular array slots for the data points to be published through the producer sequence number mechanism, and fill the event objects of the circular array slots with the data points. The waiting strategy control unit is used to coordinate the speed difference between the consumer thread and the producer thread through a multi-stage state check process of the waiting strategy when the data consumer thread reads data from the circular array storage unit.

4. The industrial edge data adaptive routing system of claim 1, wherein, The batch event processing module includes: a group container management unit, a batch trigger judgment unit, and a batch routing output unit; The group container management unit is used to distribute the data points to independent batch containers corresponding to their respective topic identifiers for temporary storage, based on the topic identifiers carried by the data points. The batch trigger judgment unit is configured to monitor the number of data points and the residence time of the first data point of each independent batch container, and receive a buffer end signal from the high-performance buffer module; when the number of data points reaches a preset size threshold, or the residence time exceeds a preset timeout threshold, or the batch end signal is received, a batch ready signal is generated; The batch routing output unit is configured to extract all data points from the corresponding independent batch container in response to the batch ready signal, assemble the data batch bound to the topic identifier, and send the data batch to the asynchronous forwarding coordination module.

5. The industrial edge data adaptive routing system of claim 2, wherein, The asynchronous forwarding coordination module comprises a forwarder decision unit, a task packaging and scheduling unit, and an execution state monitoring unit. The forwarder decision unit is configured to receive the data batch from the batch processing event processing module, extract the topic identifier bound to the data batch, and initiate a query request to the forwarder registry module according to the topic identifier to obtain a target data forwarder; The task packaging and scheduling unit is configured to associate and package the data batch with the target data forwarder, generate an asynchronous forwarding task, and submit the asynchronous forwarding task to the dedicated forwarding thread pool for queuing and execution; The execution state monitoring unit is configured to start a timeout timer when the asynchronous forwarding task is executed in the dedicated forwarding thread pool, and generate a forwarding timeout signal and trigger a corresponding task cancellation process if a task completion notification is not received within a preset timeout threshold.

6. The industrial edge data adaptive routing system of claim 5, wherein, The execution state monitoring unit further comprises: A buffer overflow detection subunit configured to monitor the lock-free ring buffer of the high-performance buffer module, and trigger a data point discard or a publishing blocking process according to a preset overflow processing strategy when detecting that the buffer capacity is exhausted A forwarder fault detection subunit configured to receive a forwarder health state change notification from the forwarder registry module, and route subsequent data batches to a backup data forwarder when determining that the target data forwarder is in a fault state.

7. An industrial edge data adaptive routing method, characterized by, The industrial edge data adaptive routing method is applied to the industrial edge data adaptive routing system of claim 1, and the industrial edge data adaptive routing method comprises: Registering available data forwarders found when the adaptive routing system starts, and generating a forwarder registry; Receiving and buffering data points from data collection threads through a lock-free ring buffer; Grouping the data points according to the topic identifier of the data points, and generating data batches corresponding to each topic identifier when a preset trigger condition is met; Calling the target data forwarder corresponding to the topic identifier in the forwarder registry; Controlling the target data forwarder to call a corresponding forwarding protocol in a dedicated forwarding thread pool to execute a forwarding task corresponding to the data batch.

8. The industrial edge data adaptive routing method of claim 7, wherein, The step of calling the target data forwarder corresponding to the topic identifier in the forwarder registry comprises: Finding a first data forwarder bound with the topic identifier in the forwarder registry, and returning the first data forwarder as a query result if the finding succeeds; Counting activation states of all data forwarders in the forwarder registry when the primary routing unit does not return a query result; Returning a second data forwarder in an activation state as a query result if there is only one second data forwarder in an activation state; Obtaining a preset default forwarder identifier when the secondary routing unit does not return a query result; Determining a third data forwarder corresponding to the default forwarder identifier from the forwarder registry, and returning the third data forwarder as a query result.

9. The industrial edge data adaptive routing method of claim 7, wherein, The step of generating a data batch corresponding to each topic identifier comprises: Starting a parallel container monitoring on an independent batch container corresponding to the topic identifier; In the container monitoring, continuously obtaining a current data point quantity of the independent batch container, and comparing the current data point quantity with a preset size threshold; Recording an injection time stamp of a first data point in the independent batch container, and calculating a corresponding retention duration based on a current time, and comparing the retention duration with a preset timeout threshold; When the current data point quantity reaches the size threshold, the retention duration exceeds the timeout threshold, or a batch end signal is received, determining that data of the independent batch container is ready, and generating a data batch corresponding to the topic identifier.

10. The industrial edge data adaptive routing method of claim 7, wherein, The step of controlling the target data forwarder to execute a forwarding task in a dedicated forwarding thread pool comprises: Task encapsulating the data batch and the target data forwarder, generating an asynchronous forwarding task, and submitting the asynchronous forwarding task to a task queue of the dedicated forwarding thread pool; Starting a timeout timer corresponding to the asynchronous forwarding task, and recording a task forwarding duration; Monitoring an execution state of the asynchronous forwarding task, and generating a task termination instruction and terminating execution of the asynchronous forwarding task when the task forwarding duration exceeds a preset submission duration threshold.

Images