Heterogeneous device data acquisition and distribution method based on message middleware
Patent Information
- Application Number
- CN202511670768.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-14
- Publication Date
- 2026-02-13
AI Technical Summary
In industrial automation systems, data acquisition and management of heterogeneous devices face challenges such as high access barriers, difficulties in expansion, complex data diversion and priority processing, and poor system stability. Existing methods are particularly inadequate for meeting the needs of mixed access and unified management of multi-brand devices.
By adopting a message middleware-based approach, heterogeneous device data is uniformly collected and serialized, combined with dynamic routing and priority mechanisms, and leveraging Kafka's parallel consumption capabilities, high-priority messages can be processed quickly and stored hierarchically, supporting dynamic expansion and exception logging.
It enables unified data acquisition and management across heterogeneous devices, reduces access complexity, improves system scalability and flexibility, ensures rapid response to critical messages and system stability, and adapts to diverse industrial application needs.
Smart Images
Figure CN121530875A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of data processing, in particular to a heterogeneous device data acquisition and distribution method based on a message middleware. BACKGROUND
[0002] With the improvement of industrial automation, PLCs, numerical control systems and robot devices from different manufacturers (such as ABB, FANUC, Siemens, etc.) are widely deployed in production lines. However, due to the non-uniformity of communication protocols and interface standards of various devices, the development of data acquisition is complex and the access threshold is high. Meanwhile, in the traditional method, acquisition and backend processing are tightly coupled, and device abnormalities can easily affect system stability. Moreover, once new devices are added, the original system needs to be significantly modified, making it difficult to expand. In addition, if each device develops an acquisition interface and maintains connection logic independently, not only does it increase the overall development and operation and maintenance costs, but it also makes it difficult to meet the actual needs of mixed access and unified management of multiple brands of devices.
[0003] On the other hand, the data generated by the devices has different flow destinations and application requirements: some data needs real-time processing and quick response (such as alarm information and running status), while some data needs long-term storage and historical tracing (such as production statistics and quality analysis). How to shunt and manage different types of data in the acquisition and transmission process is a difficulty in existing methods.
[0004] At the same time, the industrial production scene also requires priority processing: critical events such as alarms and faults must be quickly transmitted and processed in high-priority channels to avoid being blocked by large-scale routine operation data, ensuring production safety and data transmission stability. Finally, the existing method has obvious shortcomings in scalability: when new device types or business requirements are added, the logic of the acquisition end and the processing end often needs to be modified, and there is a lack of unified data routing and scalable message distribution mechanism, which cannot meet the diversified and dynamic application requirements in complex production lines. SUMMARY
[0005] In view of the problems in the prior art, the present application proposes a heterogeneous device data acquisition and distribution method based on a message middleware. The method first uniformly acquires and serializes and packages multi-source heterogeneous device data; then, through the dynamic routing module of the middleware, the target Topic is determined from the routing rule table according to the device type and message priority, and when the matching fails, the fallback channel is entered; in the message distribution process, the system uses a priority mechanism to push alarm data and ordinary running data to independent Topics and Consumer Groups, respectively, thereby ensuring the priority processing of high-priority messages; finally, in the consumption processing layer, the data is written into Redis and MySQL, respectively, to realize layered storage of real-time and persistence.
[0006] The technical solution of the present application is as follows: A heterogeneous device data acquisition and distribution method based on a message middleware, comprising the following steps: Step 1: Data acquisition and serialization processing: for multi-source heterogeneous devices using different communication protocols, the unified acquisition of device operation data and state data is realized through protocol adaptation; the collected data is serialized and packaged in JSON format, and the packaged JSON data includes a header field and a metrics field, the header field contains device id, device type, message priority and timestamp, and the metrics field integrates the specific data collected in key-value mode; Step 2: Dynamic routing forwarding: the device type and message priority of the header field in the serialized JSON data are combined in the format of deviceType_priority to form a routing matching identifier; the corresponding target Topic of the routing matching identifier is queried from the dynamic routing rule table stored in Redis, if the matched target Topic is queried, the serialized data is sent to the target Topic; if the matched target Topic is not queried, the serialized data is sent to the preset fallback topic, and the corresponding log information is recorded; Step 3: Priority-based message distribution and processing: the messages forwarded by the dynamic routing are received through the consumer subscription of the message middleware, wherein each Topic corresponds to an independent Consumer Group; more processing resources are configured for the Consumer Group corresponding to the high priority message to ensure that the high priority message runs independently of the ordinary message channel; after the consumer pulls the message, deserialization processing is performed, the device id is obtained, and the data is stored in the corresponding device id information table; Step 4: Hierarchical data storage: according to the data attributes, the corresponding storage database is selected, wherein the alarm information is stored in the Redis database and the MySQL database, the Redis database is used for timely pushing of alarm information, and the MySQL database is used for archiving and checking of alarm information; among the device operation data, the real-time response type data is stored in the Redis database, and the persistent type data is stored in the MySQL database.
[0007] Further, the communication protocol in step 1 includes OPC protocol, Http protocol, PDSDK protocol and Focas protocol, and the communication acquisition with the device is realized by using the corresponding programming language for different communication protocols.
[0008] Furthermore, the message priority mentioned in step 1 is divided into two levels: high and normal. The high priority corresponds to alarm information, and the normal priority corresponds to device operation data. Furthermore, the JSON-formatted header field described in step 1 is also used to provide unified semantic support for subsequent dynamic routing and priority distribution, eliminating interface differences between different devices.
[0009] Furthermore, the dynamic routing rule table described in step 2 supports dynamic updates. When a new device is connected, there is no need to modify the system code; only the mapping relationship between the corresponding route matching identifier and the target Topic needs to be added to the Redis dynamic routing rule table.
[0010] Furthermore, the message middleware mentioned in step 3 is Kafka. By utilizing Kafka's parallel consumption characteristics, high-priority messages can be retrieved and processed preferentially.
[0011] Furthermore, the processing resources mentioned in step 3 include CPU resources and memory resources, and resource configuration ensures that high-priority messages are not blocked by low-priority messages.
[0012] Furthermore, the real-time response data mentioned in step 4 includes current data, torque data, and coordinate data, while the persistent demand data includes production data and statistics on the number of qualified parts.
[0013] Furthermore, the log information mentioned in step 2 includes message identifiers, device information, and timestamps for which no target Topic was matched, which are used for subsequent fault diagnosis and tracing.
[0014] The beneficial effects of this invention are as follows: 1) A unified data acquisition and serialization packaging method across heterogeneous devices.
[0015] This invention proposes a unified data acquisition mechanism for heterogeneous devices from multiple vendors (such as PLCs, CNC systems, and robots). By designing adapters for different protocols (such as OPC, Focas, and PDSDK), standardized data access is achieved. After acquisition, a unified JSON structure is used for serialization and encapsulation. The encapsulation structure includes tag information such as device identifier, device type, message priority, and timestamp, thereby eliminating interface differences between different devices, reducing access complexity, and providing unified semantic support for subsequent dynamic routing and priority distribution.
[0016] 2) A dynamic routing method based on labels and routing tables, which supports exception logging.
[0017] This invention achieves automatic matching and forwarding of message target topics by adding tags (such as deviceType and priority) to the message header and combining them with a dynamic routing table maintained by Redis / database. Compared with the traditional hard-coding method, this method does not require code modification when adding new device types or adjusting business logic; rule expansion can be completed simply by updating the routing table, significantly improving system scalability and flexibility. Furthermore, this method proposes a fallback mechanism for abnormal matching: when a message fails to match the routing table, it automatically enters the fallback topic and logs the event, ensuring no message loss and providing a basis for subsequent diagnostics.
[0018] 3) Combine the Kafka distribution mechanism with message priority to ensure that high-priority messages are processed first.
[0019] This invention determines the Kafka Topic by combining deviceType and priority, and configures independent Consumer Groups and more resources for high-priority topics (such as device alarms and fault information) on the consumer side, enabling them to run independently of ordinary message channels, thereby avoiding the blocking of critical messages by running data. Simultaneously, leveraging Kafka's parallel consumption characteristics, high-priority messages can be pulled and processed first, ensuring production security and real-time requirements. This mechanism balances batch data throughput with rapid response to critical events, demonstrating strong practical value in industrial scenarios. Attached Figure Description
[0020] Figure 1 This is an overall flowchart of the present invention; Figure 2 This is a flowchart of the data acquisition and serialization process of the present invention; Figure 3 This is a flowchart of the dynamic routing process of the present invention. Figure 4 This is a flowchart of the message priority distribution and processing of the present invention; Figure 5 This is a data storage flowchart of the present invention; Figure 6 This is a diagram of the routing rule table stored in Redis in this embodiment. Figure 7 This is a screenshot of the Kafka topic information statistics interface in this embodiment; Figure 8 This is a screenshot of the message details interface for the Kafka topic cnc_normal in the example. Figure 9 This is a diagram of the Kafka Topic interface of the present invention; Figure 10 This is a detailed interface diagram of the Kafka Consumer Group of this invention. Detailed Implementation
[0021] The present invention will be further described below with reference to the accompanying drawings.
[0022] 1. Data Acquisition and Serialization Processing Module: Unified access to data from multiple heterogeneous devices; Various production devices are connected to the computer via network cables. Different programming languages are used to communicate with these devices using different communication protocols, such as OPC, HTTP, PSDK, and FOCAS, enabling real-time acquisition of operational and status data. After acquisition, the data needs to be serialized and encapsulated in JSON format, as shown in the following structure: ; The device is tagged in the data header, including device ID, device type, message priority (high and normal levels), and timestamp. The collected data is integrated in the metrics section using a key-value model.
[0023] To meet the needs of industrial production, high priority mainly collects alarm information, while normal priority collects equipment operation data.
[0024] 2. Dynamic routing module: The main task is to write serialized data to Kafka from the data acquisition end. Before the data is officially sent to the corresponding topic in Kafka, the deviceType and priority of the header in the serialized data need to be combined in the form of deviceType_priority, such as CNC_normal. Then, the routing rule table (as shown in Table 1) is read from Redis, and the corresponding topic is obtained through this combination of deviceType_priority.
[0025] Table 1: Examples of Dynamic Routing Rules in Redis ; To prevent matching failures, if there is no match in the rule, it is unified into a fallback topic to avoid message loss, and this information can be written to the log.
[0026] Using the dynamic routing module, existing device data can be matched one-to-one. When new devices are connected in the future, no code changes are needed. Simply create a Name and Topic in the Redis dynamic routing table, and the data can be sent to the corresponding Topic.
[0027] 3. Middleware message receiving module It provides a unified Kafka message queue channel to the outside world, and the message middleware consumers mainly subscribe to and receive messages forwarded by dynamic routing; that is: receive dynamically routed messages → message middleware consumers subscribe to messages.
[0028] 4. Priority-based message distribution and processing module Combine business-defined priority strategies (such as ALARM→High, Normal→Low) to ensure that critical data enters the high-priority channel first.
[0029] In Kafka, each topic corresponds to an independent Consumer Group, as shown in Table 2 below: Table 2: Example of the relationship between message priority and message middleware consumer group ; By configuring resources for Kafka message queues, higher-priority groups receive more processing resources, such as CPU and memory. This ensures that high-priority queues are not blocked by lower-priority queues. Consumers pull messages from Kafka, deserialize the JSON data, parse it to obtain the device ID, and then store the data in the corresponding device ID information table.
[0030] 5. Data Storage and Application Module It supports multiple types of database storage (such as MySQL / Redis) and implements tiered storage based on data attributes (persistence vs. real-time).
[0031] When storing equipment alarm information, it is necessary not only to store it in a Redis database for timely push notifications, but also to archive it in a MySQL database for later verification. Equipment operating data, such as current, torque, and coordinates, can be stored directly in a Redis database for faster response. Production data and qualified piece count statistics can be stored in a MySQL database for persistent storage, and multiple pieces can be stored in batches to alleviate storage pressure. Example
[0032] For different equipment manufacturers, each has its own communication protocol, requiring the development of corresponding protocol adaptation modules. For devices supporting OPCUA, such as the Siemens 840DSL control system and advanced Siemens PLCs like the S7-1200 and S7-1500, data subscription and retrieval can be implemented using the Java-based OPCUA library. For FANUC CNC systems or robots, the Focas API in C# is used to read operating parameters such as spindle speed, coordinates, and alarms. For some quality inspection equipment controllers, HTTP requests can be sent, and Java can be used to receive and parse these requests. For lower-level Siemens PLCs, such as the S7-300, the Snap7 library is used to periodically poll specified database blocks.
[0033] After data is collected, it needs to be processed specifically and then converted into a unified JSON structure. For example, alarm information from Fanuc CNC systems or robots only contains numerical codes. The collected numerical codes need to be mapped to the corresponding alarm descriptions. The numerical codes and their Chinese meanings are shown in the table below: ; When acquiring data such as spindle speed, current position coordinates, torque, and current, it needs to be converted to a numeric type with two decimal places. Then, the data needs to be serialized into JSON, consisting of a header and metrics. The header contains the device number, type, priority, and timestamp. For example, if the device acquiring the data is a Siemens PLC, numbered 1, model S7-1200, and the time is October 10, 2025, 14:00:00, and the device has no abnormal alarm information, then the data format in the header would be: ; Where deviceId is in the format of device manufacturer-device type-device number, deviceType is the device type, priority is the priority type for no alarm information, and timestamp is the timestamp format of the current time.
[0034] The data information includes the relevant data collected from the equipment, such as rotational speed and temperature. ; The data displayed in both the header and the data information is encapsulated in key-value format, such as... Figure 6 As shown; the priority field is determined by the acquisition end. When device alarm information is acquired, it is set to high; otherwise, it is set to normal.
[0035] After collecting the data and encapsulating it into JSON data, it needs to be categorized and written into Kafka Topics based on the device type (deviceType) and priority (priority) in the JSON header. Figure 7 As shown, the corresponding topic in the Kafka message queue corresponds to the value in the dynamic routing table stored in Redis.
[0036] like Figure 8 As shown, these are the data values in the topic of cnc_normal. Subsequently, the consumer program listens for topic messages in real time and performs deserialization and parsing on each piece of data.
[0037] like Figure 9 As shown, each topic corresponds to the same consumer group, and each consumer group contains the corresponding consumers who consume the corresponding topic, such as... Figure 10 As shown.
[0038] Subsequently, a hierarchical storage strategy can be implemented based on message priority and data type, with alarms and real-time status data written to Redis, and historical statistics data written to MySQL in batches.
Claims
1. A method for data acquisition and distribution from heterogeneous devices based on message middleware, characterized in that, Includes the following steps: Step 1: Data Acquisition and Serialization Processing: For multi-source heterogeneous devices using different communication protocols, unified acquisition of device operation data and status data is achieved through protocol adaptation. The acquired data is serialized and encapsulated in JSON format. The encapsulated JSON data includes a header field and a metrics field. The header field contains the device ID, device type, message priority, and timestamp. The metrics field integrates the specific data acquired in a key-value pattern. Step 2: Dynamic Routing Forwarding: Combine the device type and message priority in the header field of the serialized JSON data according to the deviceType_priority format to form a route matching identifier; query the target topic corresponding to the route matching identifier from the dynamic routing rule table stored in Redis; if a matching target topic is found, send the serialized data to the target topic; if no matching target topic is found, send the serialized data to the preset fallback topic and record the corresponding log information. Step 3: Priority-based message distribution and processing: Consumers of the message middleware subscribe to receive messages forwarded via dynamic routing, where each Topic corresponds to an independent Consumer Group; allocate more processing resources to the Consumer Group corresponding to high-priority messages to ensure that high-priority messages run independently of the ordinary message channel; after the consumer pulls the message, it performs deserialization processing, parses to obtain the device ID, and stores the data in the corresponding device ID information table; Step 4: Layered data storage: Select the corresponding storage database according to the data attributes. Alarm information is stored in both Redis and MySQL databases. The Redis database is used to push alarm information in a timely manner, while the MySQL database is used for archiving and verifying alarm information. In the device operation data, real-time response data is stored in the Redis database, while persistent data is stored in batches in the MySQL database.
2. The method for heterogeneous device data acquisition and distribution based on message middleware according to claim 1, characterized in that, The communication protocols mentioned in step 1 include OPC protocol, HTTP protocol, PDSDK protocol, and Focas protocol. Corresponding programming languages are used to implement communication and data acquisition with the device for different communication protocols.
3. The method for heterogeneous device data acquisition and distribution based on message middleware according to claim 1, characterized in that, The message priority mentioned in step 1 is divided into two levels: high and normal. The high priority corresponds to alarm information, and the normal priority corresponds to device operation data.
4. The method for heterogeneous device data acquisition and distribution based on message middleware according to claim 1, characterized in that, The JSON-formatted header field mentioned in step 1 is also used to provide unified semantic support for subsequent dynamic routing and priority distribution, eliminating interface differences between different devices.
5. The method for heterogeneous device data acquisition and distribution based on message middleware according to claim 1, characterized in that, The dynamic routing rule table described in step 2 supports dynamic updates. When a new device is connected, there is no need to modify the system code. You only need to add the corresponding route matching identifier and the mapping relationship between the target topic to the dynamic routing rule table in Redis.
6. The method for heterogeneous device data acquisition and distribution based on message middleware according to claim 1, characterized in that, The message middleware mentioned in step 3 is Kafka. By utilizing Kafka's parallel consumption feature, high-priority messages can be retrieved and processed first.
7. The method for heterogeneous device data acquisition and distribution based on message middleware according to claim 1, characterized in that, The processing resources mentioned in step 3 include CPU resources and memory resources. Through resource configuration, high-priority messages are not blocked by low-priority messages.
8. The method for heterogeneous device data acquisition and distribution based on message middleware according to claim 1, characterized in that, The real-time response data mentioned in step 4 includes current data, torque data, and coordinate data, while the persistent demand data includes production data and statistics on the number of qualified parts.
9. The method for heterogeneous device data acquisition and distribution based on message middleware according to claim 1, characterized in that, The log information mentioned in step 2 includes message identifiers, device information, and timestamps for which no target Topic was matched, which are used for subsequent fault diagnosis and tracing.