Metering production equipment instrument data acquisition and integration method and system

Abstract

The invention relates to the technical field of Internet of Things and intelligent metering, in particular to a metering production equipment instrument data acquisition and integration method and system. Accessing different types of instruments by using a multi-protocol acquisition terminal, and unifying the format of protocol data through a protocol analysis engine; preprocessing the original data through edge computing nodes; performing data routing through a distributed message queue, and distributing the data to a corresponding cloud integration engine according to a production line to which the equipment belongs; and the cloud integration engine performs data modeling based on the unified data model. Through deep combination of theoretical innovation and engineering practice, the industrial measurement data acquisition and integration system with high compatibility, high real-time performance and high reliability is constructed, a standardized data infrastructure solution is provided for the fields of intelligent manufacturing, energy internet and the like, and the industrial measurement data acquisition and integration system has remarkable technical leading and industrial application value.

Description

Technical Field

[0001] The present invention relates to the field of industrial Internet of Things and intelligent metering technology, and in particular to a method and system for collecting and integrating meter data of metering production equipment. Background Art

[0002] In the context of Industry 4.0 and smart manufacturing, the intelligence of metering production equipment continues to increase. The number of instruments in a single factory has generally exceeded 100,000, covering more than 20 types of instruments such as pressure, flow, and electricity, and using more than 15 communication protocols (such as Yokogawa Instruments' Modbus RTU, Siemens Instruments' Profibus DP, and Emerson Instruments' HART). Traditional data acquisition systems have the following core problems:

[0003] 1. Inefficient protocol parsing: Test data from an automobile manufacturer showed that a traditional Modbus parser developed in Java experienced a parsing delay of 45ms per transaction when concurrently accessing 500 instruments. The CPU utilization rate exceeded 80%, extending the data collection cycle to over 600ms.

[0004] 2. Lack of timing synchronization mechanism: At a petrochemical plant, the clocks of multiple instruments were out of sync (with a maximum error of 1.2 seconds), resulting in a 15% deviation in production energy consumption statistics, directly affecting cost accounting accuracy.

[0005] 3. Crude abnormal data processing: The existing system can only detect over-range anomalies and lacks effective processing for complex abnormal scenarios such as data mutations (e.g., pressure fluctuations exceeding 10% of the range within 2ms) and jumps (data remaining unchanged for five consecutive cycles). For example, a steel plant failed to identify an instrument crash, resulting in 12 hours of distorted production data.

[0006] 4. Cloud load imbalance: When an electronics factory used a centralized database (MySQL) to connect to 8,000 instruments, write latency exceeded 300ms during peak periods, and the number of database connections exceeded the upper limit (the default is 1,000 connections), resulting in 20% data loss. Summary of the Invention

[0007] In view of the above problems in the prior art, the present invention is proposed.

[0008] Therefore, the technical problem to be solved by the present invention is: the problems of poor compatibility, low real-time performance, unstable data quality and insufficient scalability in the prior art.

[0009] To solve the above technical problems, the present invention provides the following technical solution: a method for collecting and integrating instrument data of measurement production equipment, comprising:

[0010] Use multi-protocol acquisition terminals to access different types of instruments, and use the protocol parsing engine to unify the protocol data into a unified format;

[0011] Preprocess the raw data through edge computing nodes:

[0012] Data is routed through distributed message queues, and data is distributed to the corresponding cloud integration engine based on the production line to which the equipment belongs;

[0013] The cloud-based integration engine performs data modeling based on a unified data model.

[0014] As a preferred solution of the method for data collection and integration of metering production equipment described in the present invention, wherein: the multi-protocol acquisition terminal supports wired and wireless hybrid networking;

[0015] When the wired network is disconnected, it automatically switches to wireless transmission mode, and the transmission rate is dynamically adjusted according to the signal strength;

[0016] The protocol analysis engine supports a protocol conversion throughput of ≥2000 times per second and a resolution error rate of ≤0.01%.

[0017] As a preferred solution of the method for collecting and integrating instrument data of a measuring production equipment according to the present invention, the pre-processing of the raw data by the edge computing node includes:

[0018] Perform time series alignment on multi-instrument data based on dynamic time warping algorithm;

[0019] The 3σ rule is used to detect data outliers, and abnormal data that exceeds the equipment range is marked and a re-sampling mechanism is triggered.

[0020] As a preferred solution of the method for collecting and integrating instrument data of measurement production equipment described in the present invention, the time series alignment of multiple instrument data based on the dynamic time warping algorithm includes:

[0021] When using the dynamic time warping algorithm, the master device clock is used as the benchmark to perform cubic spline interpolation on the slave device data to generate a synchronized data sequence with a time interval of 1ms. The mean square error of the synchronized data timestamp is ≤20ms.

[0022] As a preferred embodiment of the method for collecting and integrating metering production equipment data described in the present invention, the three-level cache system includes data classification into hot, warm, and cold data. Hot data, representing the top 10% of access frequency, is stored in device terminal registers, such as the ARM chip L1 cache, with a response time of less than 10ms. Warm data, representing the 20%-30% of access frequency, is stored in terminal memory, such as DDR4, and in edge server Redis clusters, with a response time of 50-100ms. Cold data, representing historical data, is stored in the HDFS distributed file system, with a batch query interface provided by the HBase column-oriented database.

[0023] , the present invention provides the following technical solutions: a measurement production equipment instrument data acquisition and integration system, an intelligent acquisition module, an edge processing module, a cloud integration module, and an application interface module.

[0024] A computer device includes a memory and a processor, wherein the memory stores a computer program, and is characterized in that when the processor executes the computer program, it implements the steps of the above-mentioned method for collecting and integrating instrument data of measurement production equipment.

[0025] A computer-readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, the steps of the above-mentioned method for collecting and integrating instrument data of measurement production equipment are implemented.

[0026] The beneficial effects of this invention are: support for rapid access to mainstream industrial protocols, shortening the new protocol development cycle from 2 weeks to 2 hours; edge layer processing delay <10ms, end-to-end data collection cycle ≤100ms (traditional solution >500ms); timing alignment error ≤50ms, abnormal data detection accuracy ≥99.5%; a single cloud node supports concurrent access of 20,000+ instruments, and cluster mode can be expanded to 100,000+ devices.

[0027] Through the in-depth combination of theoretical innovation and engineering practice, this invention has constructed a highly compatible, real-time and reliable industrial metrology data acquisition and integration system, providing standardized data infrastructure solutions for smart manufacturing, energy Internet and other fields, with significant technological leadership and industrial application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0029] Figure 1 A schematic diagram of a method for collecting and integrating instrument data for metrological production equipment provided by one embodiment of the present invention.

[0030] Figure 2 This is a system architecture design diagram of a metrological production equipment instrument data acquisition and integration system provided by one embodiment of the present invention.

[0031] Figure 3 A schematic diagram of a multi-protocol parsing plug-in architecture for a method for collecting and integrating instrument data for metrological production equipment provided by one embodiment of the present invention.

[0032] Figure 4 This is a logic diagram of edge node data processing for a method for collecting and integrating instrument data for metrological production equipment provided by one embodiment of the present invention.

[0033] Figure 5 This is a diagram of the edge node hardware architecture of a method for collecting and integrating instrument data for metrological production equipment provided by one embodiment of the present invention.

[0034] Figure 6 A cloud-based data modeling flow chart for a method for collecting and integrating instrument data for metrological production equipment provided in one embodiment of the present invention. DETAILED DESCRIPTION

[0035] To make the above-mentioned objects, features, and advantages of the present invention more clearly understood, the following detailed description of the specific embodiments of the present invention is given in conjunction with the accompanying drawings. It is obvious that the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in this field without creative work should fall within the scope of protection of the present invention.

[0036] Example 1, as Figure 1 、 Figure 3 、 Figure 4 FIG. 1 is an embodiment of the present invention, which provides a method for collecting and integrating instrument data of a metrological production equipment, including:

[0037] S1: Use a multi-protocol acquisition terminal to access different types of instruments, and use the protocol parsing engine to unify the protocol data into a unified format.

[0038] Design a pluggable protocol parser plug-in architecture that supports dynamic loading of protocols such as Modbus RTU / TCP, Profibus DP, HART, OPCUA, and MQTT. For example, for the Modbus RTU protocol, the serial port configuration tool automatically identifies the slave address (1 to 247) and generates a device communication parameter table.

[0039] Define a unified data interface specification. All protocol parsed data is converted into a JSON format containing the device_id, timestamp, value, unit, and quality fields. The example is as follows:

[0040] {

[0041] "device_id":"PRESS-001",

[0042] "timestamp":"2025-04-20T15:30:00.000+08:00",

[0043] "value":2.5,

[0044] "unit":"MPa",

[0045] "quality":"GOOD"

[0046] }

[0047] It uses an ARM Cortex-A7 processor (main frequency 1.2GHz), integrated RS-485 / 232 serial port (supports 32 slave devices), 10 / 100M Ethernet port, and 5G communication module (supports NSA / SA dual-mode); the protocol parsing plug-in is stored in the terminal flash memory (8GB eMMC), and the activation protocol is selected through the web configuration interface (for example, check the "Modbus RTU" plug-in and enter parameters such as slave address, baud rate, and parity bit).

[0048] S2: Preprocess the raw data through edge computing nodes.

[0049] Timing alignment: To address the issue of asynchronous data acquisition clocks across multiple instruments, the DTW algorithm is used to calculate the optimal matching path for each device's time series. Interpolation is then used to unify the data to millisecond-level timestamps (e.g., based on the master controller clock, data with an error exceeding 50ms triggers re-acquisition).

[0050] Using FPGA to implement parallel optimization of the DTW algorithm, the Euclidean distance calculation of the time series is decomposed into 16 parallel processing units. Each unit is responsible for the matching calculation of different time windows. Compared with pure software implementation, the efficiency is improved by 10 times, and the time required to align a 1000-point time series is reduced from 20ms to 2ms.

[0051] Synchronization accuracy verification: Using the PLC's 1ms clock as a benchmark, 100 Modbus RTU instruments were tested for 24 hours. The mean square error of the synchronized data timestamp was 18ms, with a maximum error of 45ms, meeting the requirements of the IEEE 1588 precise clock synchronization protocol.

[0052] Abnormal detection: Establish a threshold library for equipment operating parameters (such as the normal range of a pressure gauge is 0-4MPa), and monitor the data in real time to see if it exceeds the threshold by ±20% or the fluctuation rate exceeds 1MPa / s. Abnormal data is marked as "BAD" and the abnormality type (overrange / sudden change / jump) is recorded.

[0053] A lightweight anomaly detection model is deployed based on TensorFlow Lite. The input is the parameter values of the last 10 cycles, and the output is the anomaly type (overrange / mutation / jump / normal). The model inference time is ≤ 1ms, and the CPU load on the edge node is ≤ 5%.

[0054] Exception data tag format extension:

[0055] {

[0056] "device_id":"PRESS-001",

[0057] "timestamp":"2025-04-20T15:30:00.000+08:00",

[0058] "value":5.2,

[0059] "unit":"MPa",

[0060] "quality":"BAD",

[0061] "anomaly_type":"OVER_RANGE",

[0062] "raw_data":"0x010300010002C40B" / / Original hexadecimal data

[0063] }

[0064] For Modbus RTU instruments, the terminal sends a query command (function code 0x03) every 50ms. After receiving the data, it passes the CRC check (retransmitted when the error rate is greater than 1%) and parses the parameter value corresponding to the register address (for example, register 40001 corresponds to the pressure value, and the conversion formula is: value = raw_data × 0.01).

[0065] S3: Routes data through distributed message queues and distributes data to the corresponding cloud integration engine based on the production line to which the equipment belongs.

[0066] Message queue routing: Kafka is used as the data bus, and topics are partitioned by "production line ID + device type" (such as PROD-LINE-01-PRESSURE). Consistent hashing is used to achieve load balancing, ensuring a single-partition throughput of ≥1000 messages per second.

[0067] Unified data model: defines a three-layer data structure (basic equipment information, real-time operating parameters, and quality indicators), and supports version management of the data model (e.g., V1.0 adds a "spatial location coordinate" field, and V2.0 adds a "predictive maintenance parameter").

[0068] The acquisition terminal supports wired (Ethernet) and wireless (5G / 4G) dual-link communication, automatically switches the transmission mode through the link quality monitoring module (signal strength, bit error rate), and automatically synchronizes cached data after the wired network is restored (cache capacity ≥ 24 hours of data);

[0069] Edge nodes use a three-level cache (register-level hot data, memory-level warm data, and local disk cold data). Hot data caches the 1,000 pieces of data that have been frequently accessed in the last 10 minutes. Warm data stores the day's data and archives it hourly. Cold data is uploaded to the cloud regularly.

[0070] A digital twin model is built for each instrument. The normal operating range (mean ± 2σ) is trained based on historical data. The Mahalanobis distance of the current data is calculated in real time. When the threshold (3σ) is exceeded, a three-level warning is triggered:

[0071] Level 1 warning: Mark abnormal data and trigger immediate re-collection (at 10ms interval);

[0072] Level 2 warning: If there are three consecutive abnormal re-sampling events, the device will be marked as "suspected fault" and the edge node will be notified to increase the sampling frequency to 200ms.

[0073] Level 3 warning: If the abnormality lasts for 5 minutes, a fault ticket will be generated and pushed to the operation and maintenance terminal.

[0074] S4: The cloud integration engine performs data modeling based on a unified data model.

[0075] Data migration strategy: Statistical access logs are collected every 15 minutes. Warm data that has not been accessed for 30 consecutive minutes is migrated to cold storage, and newly generated frequently accessed cold data is upgraded to warm cache, thereby achieving dynamic data migration between different cache tiers.

[0076] The cloud uses an InfluxDB cluster to store real-time operating parameters, based on a three-dimensional sharding strategy of "device type + geographic location + time":

[0077] The device type (such as energy meter / pressure / flow) is used as the primary sharding key to isolate different business data;

[0078] Geographic location (factory / workshop / production line) is used as the secondary sharding key to meet regional data access requirements;

[0079] Time (by day / hour) is used as the third-level sharding key to optimize time series data query performance, with a single shard query throughput of ≥5,000 queries per second.

[0080] like Figure 3 As shown, step S1 mentions the use of a multi-protocol acquisition terminal to access different types of instruments and convert data formats, as well as completing data processing at the edge computing node. By integrating an FPGA coprocessor at the edge node, hardware acceleration of computationally intensive tasks such as protocol parsing and timing alignment is achieved, solving the potential problems of low protocol parsing efficiency and high edge processing latency in claim 1, reducing the overall processing latency of the edge layer to less than 10ms, and significantly improving the protocol conversion throughput (the protocol parsing engine supports a protocol conversion throughput of ≥2000 times per second) and edge processing performance.

[0081] like Figure 4 As shown, the raw data is pre-processed through the edge computing node in step S2, as well as the anomaly detection step. The edge computing node pre-processes the raw data, stipulating requirements such as timing alignment error of ±50ms and re-collection of abnormal data, and further limits the details of timing alignment using the dynamic time warping algorithm (DTW). A full-link data quality control system is constructed, and sub-millisecond timing alignment is achieved through the DTW algorithm + cubic spline interpolation, so that the mean square error of the synchronized data timestamp is ≤20ms. Combined with the digital twin model and machine learning algorithm, the abnormal data missed detection rate is reduced from 35% to 0.2%, which greatly improves the accuracy and reliability of data collection, and meets and exceeds the technical requirements related to data quality in the claims.

[0082] The cloud-based integration engine in S3 performs data modeling based on a unified data model. This involves transmitting data to the cloud-based integration platform via distributed message queues. Using a three-level expansion strategy of "consistent hashing + dynamic partitioning + intelligent sharding," the cloud cluster automatically scales based on load, supporting smooth access for tens to hundreds of thousands of instruments. This addresses issues such as load imbalance and data loss that can arise in high-concurrency cloud scenarios, ensuring high availability and scalability for large-scale device access, and enabling a maximum single-cluster access scale of 250,000 devices to meet the growing demand for device data collection and integration in industrial production.

[0083] Example 2, as Figure 2 FIG. 1 is an embodiment of the present invention, which provides a system for collecting and integrating meter data of production equipment, including:

[0084] Intelligent acquisition module, including multi-protocol acquisition terminal and adapter, supports analysis of at least 8 industrial protocols such as Modbus, Profibus, and MQTT;

[0085] Edge processing module, including edge computing nodes and local cache, to achieve data cleaning, timing calibration and exception handling;

[0086] Cloud integration module, including distributed message queue (Kafka), data modeling engine and unified storage database, supports real-time integration and quality assessment of data from tens of thousands of devices;

[0087] The application interface module provides RESTful API, OPC UA interface and data subscription services, supporting seamless integration with upper-level systems such as ERP and MES.

[0088] This embodiment further provides a computing device applicable to a method for collecting and integrating instrument data of a measurement production device, including:

[0089] Memory and processor; the memory is used to store computer-executable instructions, and the processor is used to execute computer-executable instructions to implement a method for collecting and integrating instrument data of measurement production equipment as proposed in the above embodiment.

[0090] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, a method for collecting and integrating instrument data of measurement production equipment as proposed in the above embodiment is implemented.

[0091] The storage medium proposed in this embodiment and the method for collecting and integrating instrument data of measuring production equipment proposed in the above embodiment belong to the same inventive concept. The technical details not fully described in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0092] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0093] Logic and / or steps otherwise described herein, which may be considered, for example, as an ordered list of executable instructions for implementing the logical functions, may be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" may be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0094] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0095] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

[0096] Example 3, as Figure 5 、 Figure 6As shown, take a new energy vehicle battery factory as an example (deployed 30,000 instruments, including 12 protocol types):

[0097]

[0098] Typical scenario test:

[0099] 1. Protocol switching test: After disconnecting from the wired network, the acquisition terminal completed the wireless link switching within 120ms, cached 23 data items during the switching period (cache capacity 1000 items), and completed data synchronization within 50ms after the network was restored, with no packet loss.

[0100] 2. Peak stress test: Simulating 20,000 instruments uploading data simultaneously, the cloud Kafka cluster CPU utilization remained stable at 65%, memory usage was 48%, and no partitions were overloaded (peak throughput of a single partition was 1,500 records / second).

[0101] like Figure 5 As shown in the figure, the key component is the FPGA coprocessor, which implements hardware acceleration functions such as Modbus RTU CRC check and HART Manchester encoding and decoding, reducing the load of the main control chip by more than 30%.

[0102] like Figure 6 As shown, the version management mechanism: when a new "predictive maintenance parameter" field is added, a model change notification is automatically generated to support compatible parsing of new and old version data, ensuring seamless integration of historical data and real-time data.

[0103] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A method for collecting and integrating instrument data of measurement production equipment, characterized by: include: Use multi-protocol acquisition terminals to access different types of instruments, and use the protocol parsing engine to unify the protocol data into a unified format; Preprocess the raw data through edge computing nodes: Data is routed through distributed message queues, and data is distributed to the corresponding cloud integration engine based on the production line to which the equipment belongs; The cloud integration engine performs data modeling based on a unified data model and builds a three-level cache system.

2. A method for collecting and integrating meter data of production equipment according to claim 1, characterized in that: The multi-protocol acquisition terminal supports wired and wireless hybrid networking; When the wired network is disconnected, it automatically switches to wireless transmission mode, and the transmission rate is dynamically adjusted according to the signal strength; The protocol analysis engine supports a protocol conversion throughput of ≥2000 times per second and a resolution error rate of ≤0.01%.

3. A method for collecting and integrating meter data of production equipment according to claim 2, characterized in that: The pre-processing of the raw data by the edge computing node includes: Perform time series alignment on multi-instrument data based on dynamic time warping algorithm; The 3σ rule is used to detect data outliers, and abnormal data that exceeds the equipment range is marked and a re-sampling mechanism is triggered.

4. A method for collecting and integrating meter data of production equipment according to claim 3, characterized in that: The time series alignment of multi-instrument data based on the dynamic time warping algorithm includes: When using the dynamic time warping algorithm, the master device clock is used as the benchmark to perform cubic spline interpolation on the slave device data to generate a synchronized data sequence with a time interval of 1ms. The mean square error of the synchronized data timestamp is ≤20ms.

5. A method for collecting and integrating meter data of production equipment according to claim 4, characterized in that: The three-level cache system includes the storage location and data source division of hot data, warm data, and cold data; Hot data uses the LRU-K algorithm to retain the last two access records; Use the LFU algorithm for warm data to eliminate the data with the lowest access frequency; Cold data is stored in the HDFS distributed file system, and the cache level is regularly updated through data hot and cold migration strategies.

6. A device for collecting and integrating data of metering production equipment according to any one of claims 1 to 5, characterized in that: include: Intelligent acquisition module, edge processing module, cloud integration module, and application interface module.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of a method for collecting and integrating instrument data of measurement production equipment according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of a method for collecting and integrating instrument data of measurement production equipment according to any one of claims 1 to 5 are implemented.

Images