Metering instrument equipment data intelligent acquisition management method based on Internet of Things

Abstract

The invention discloses a metering instrument equipment data intelligent acquisition management method based on Internet of Things, which belongs to the technical field of Internet of Things data management, comprises data acquisition, data processing, data storage, data analysis and user display, and solves the problems of unified efficient acquisition and intelligent analysis management of heterogeneous metering instrument equipment data. According to the invention, the technical problem of realizing intelligent collaboration and visual application of multi-source data is solved, multiple metering equipment access modes such as wired, wireless, API and Web services are supported, different manufacturers and communication protocols are adapted, local preprocessing and key event reporting are realized, bandwidth and cloud pressure are reduced, and equipment state prediction and accurate alarm are realized. Data security and access compliance are ensured by combining data desensitization encryption, authority control and OAuth2.0 authentication, and user perception and response capabilities are enhanced by cooperating with multi-channel message pushing.

Description

Technical Field

[0001] The present invention belongs to the technical field of RFID radio frequency tags and relates to a method for intelligently collecting and managing data of measuring instruments and equipment based on the Internet of Things. Background Art

[0002] With the rapid development of the Internet of Things (IoT), more and more measuring instruments and equipment are being connected to the internet, enabling automated data collection and remote monitoring. However, most solutions currently available on the market focus primarily on hardware-level automation and typically only support specific data collection methods. This results in a lack of a unified data management platform across different types of measuring instruments, leading to operational complexity and management inconvenience for users.

[0003] Existing hardware automation solutions suffer from the following shortcomings: poor compatibility. Most hardware solutions only support specific types of metering instruments and equipment, making it difficult to meet diverse device access requirements. Low scalability means that integrating new equipment or collection methods often requires hardware reconfiguration, increasing costs and time. Data silos exist, preventing data from different devices from being centrally managed and analyzed on the same platform, hindering the full realization of data value. Lack of intelligence means a lack of in-depth data analysis and intelligent prediction, making real-time monitoring and early warning of device status impossible.

[0004] Based on the above problems, there is an urgent need for a software platform-based solution that is compatible with multiple data collection methods, can achieve unified management and intelligent analysis of data of different types of measuring instruments and equipment, and improve the efficiency and accuracy of equipment management. Summary of the Invention

[0005] The purpose of the present invention is to provide a method for intelligent collection and management of measuring instrument equipment data based on the Internet of Things, which solves the technical problems of unified and efficient collection and intelligent analysis and management of heterogeneous measuring instrument equipment data, and realizes intelligent collaboration and visualization application of multi-source data.

[0006] To achieve the above object, the present invention adopts the following technical solutions: A method for intelligent data collection and management of measuring instrument equipment based on the Internet of Things, comprising the following steps: Step 1: Deploy data acquisition module, data processing module, cloud server module, data analysis module and user application module in the server cluster; The data acquisition module is compatible with multiple access methods for data collection, including wired connection, wireless connection, remote API interface or Web service access; it can realize real-time collection of heterogeneous data generated by measuring instruments during operation through the Internet; Step 2: The data processing module performs unified preprocessing on heterogeneous data and establishes a unified data resource set; Step 3: The cloud server module distributes the data resource collection to obtain the stored data, and uses a multi-level permission control mechanism and a desensitized data encryption strategy to process the stored data; Step 4: The data analysis module builds a machine learning model to predict the operating status of the metering equipment based on the stored data. It also uses anomaly detection algorithms to detect anomalies in historical data and generate equipment operating status reports and alarm information. Step 5: The user-side application module provides an interactive platform based on the web or mobile terminal to display historical data, equipment operation status reports and alarm information to users, thereby conducting intelligent visual management of metering equipment.

[0007] Preferably, when executing step 1, the specific steps performed by the data acquisition module are as follows: Step 1-1: Register and authenticate the metering device that is connected for the first time, and obtain basic information about the metering device, including device ID, model, manufacturer, and firmware version; After the registered metering device has passed the legitimacy verification, a unique device key is assigned to the metering device and the key is returned to the metering device through a secure channel; A temporary token mechanism is used during the key return process, and the metering device completes the binding and returns the signature verification within the predetermined time; Step 1-2: For metering devices with wired interfaces, connect the metering devices to the network through a serial port server or converter, and communicate with the data acquisition module through the network; For metering equipment with wireless communication, the data acquisition module exchanges data with the metering equipment through the wireless sensor access protocol; For metering devices with API interfaces, the data acquisition module provides a RESTful API, and the metering device uploads data to the data acquisition module through HTTP requests; For measuring devices with Web services, the data acquisition module exchanges data with the measuring devices through SOAP or REST; Steps 1-3: Deploy an edge gateway cluster on the metering device side. Each gateway has a built-in lightweight inference engine to perform abnormal pattern recognition and data downsampling on the raw data collected by the metering device, and only upload data fragments that are determined to be key events or representative features to the data collection module.

[0008] Preferably, when executing step 2, the preprocessing of the data processing module includes data format conversion, data cleaning and compression encryption, specifically the following steps: Step 2-1: Use the preset data standard dictionary to convert the data format of heterogeneous data to obtain standard data with a unified format; Step 2-1: Clean the outliers, missing values, and duplicate data in the standard data, filter out invalid data, and obtain the cleaned standard data; Step 2-3: Use the GZIP algorithm to compress the cleaned standard data, and encrypt the compressed data using symmetric encryption; Step 2-4: Gather the preprocessed data to establish a unified data resource collection.

[0009] Preferably, when executing step 3, the cloud server module specifically performs the following steps: Step 3-1: Establish a distributed database to store data resource collection; Step 3-2: Set up multi-level permission control and establish a mathematical model of users, roles, and permissions to express that different users have different permissions according to their roles; use OAuth2.0 authentication to authenticate the legitimacy of user identities; Step 3-3: Encrypt and store sensitive data, adopt an off-site backup strategy, and regularly back up the distributed database.

[0010] Preferably, when executing step 4, the data analysis module executes the following steps: Step 4-1: Use streaming data processing methods to process the stored data, calculate key indicators, and use them to detect the operating status of the metering equipment and generate historical data. Step 4-2: Build a machine learning model and train it using historical data. Use the trained machine learning model to predict the operating status of the metering equipment and generate an equipment operating status report. Step 4-3: The anomaly detection algorithm is an adaptive threshold algorithm. Specifically, it performs rolling window statistical analysis on historical data, automatically calculates the normal indicator distribution of the device under different time periods, different loads, and different business scenarios, and dynamically generates alarm thresholds based on quantiles. When the data exceeds the adaptive threshold, an alarm is triggered and an alarm message is generated. Step 4-4: Send historical data, equipment operation status report and alarm information to the user-side application module.

[0011] Preferably, when executing step 5, the user-side application module specifically performs the following steps: Step 5-1: For the web side, the front-end uses Vue.js and React frameworks; the user-side application module provides a RESTful API to interact with the front-end for data; The web terminal provides interfaces for data query, equipment operation status report query, and alarm information query; Step 5-2: For the mobile terminal, the front end establishes an APP terminal and exchanges data with the user-side application module through the mobile network; Step 5-3: The user-side application module provides multiple alarm channels such as SMS, email, and App push for pushing alarm information.

[0012] The method for intelligent data collection and management of measuring instruments and equipment based on the Internet of Things described in the present invention solves the technical problems of unified and efficient collection and intelligent analysis and management of heterogeneous measuring instrument and equipment data, and realizes intelligent collaboration and visualization application of multi-source data. The present invention supports multiple metering equipment access methods such as wired, wireless, API and Web services, adapts to different manufacturers and communication protocols, introduces edge gateways and lightweight inference engines, realizes local preprocessing and key event reporting, reduces bandwidth and cloud pressure, and realizes device status prediction and accurate alarm through machine learning models and adaptive anomaly detection algorithms. Combined with data desensitization encryption, permission control and OAuth2.0 authentication, it ensures data security and access compliance. The unified management interface of the Web and mobile terminals, combined with multi-channel message push, enhances user perception and responsiveness. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a schematic diagram of the system architecture of the present invention; Figure 2 is a flow chart of the data acquisition module of the present invention; Figure 3 is a flow chart of the data processing module of the present invention; Figure 4 is an architectural diagram of the cloud server module of the present invention; Figure 5 It is a flow chart of the data analysis module of the present invention. DETAILED DESCRIPTION

[0014] Depend on Figure 1-Figure 5 The method for intelligent data collection and management of measuring instrument equipment based on the Internet of Things includes the following steps: Step 1: Deploy data acquisition module, data processing module, cloud server module, data analysis module and user application module in the server cluster; The data acquisition module is compatible with multiple access methods for data collection, including wired connection, wireless connection, remote API interface or Web service access; it can realize real-time collection of heterogeneous data generated by measuring instruments during operation through the Internet; When executing step 1, the specific steps performed by the data acquisition module are as follows: Step 1-1: Register and authenticate the metering device that is connected for the first time, and obtain basic information about the metering device, including device ID, model, manufacturer, and firmware version; After the registered metering device has passed the legitimacy verification, a unique device key is assigned to the metering device and the key is returned to the metering device through a secure channel; A temporary token mechanism is used during the key return process, and the metering device completes the binding and returns the signature verification within the predetermined time; In this embodiment, the legitimacy verification is a process of verifying the legitimacy of a registered device, and the core of the process is to verify whether the identity of the device is authentic.

[0015] The specific method used in actual use is a two-factor authentication mechanism based on certificate chain or device fingerprint, combined with a temporary token to control the binding timeliness.

[0016] The device certificate verification is an X.509 certificate preset on the device side, and the server cluster verifies the legitimacy of the certificate.

[0017] In this embodiment, a unique identifier of the device is generated based on the basic information of the metering device, such as a hash value generated based on the MAC address (or device ID), firmware version, device type, etc.

[0018] After verifying the legitimacy, the server cluster generates a short-lived temporary access token (such as JWT or UUID token) and sends it to the metering device through the TLS channel. The metering device must complete the key binding request before the token expires.

[0019] The server cluster will assign a unique key to each device and ensure that the key is sent through a secure channel and verify that the binding is successful. The specific method can adopt an ECDH key exchange + signature verification mechanism.

[0020] The server generates a random key pair (using ECC or RSA) and assigns the public key portion of the public and private keys to the device. Alternatively, when the device initiates a key request, it generates a key pair itself and uploads only the public key, which the server records.

[0021] The server encrypts the assigned key (or the symmetric key negotiated with the device) and sends it to the device through the TLS or DTLS channel. At the same time, the server requires the device to sign the specified random data using the assigned key within the specified time and return the signature value to complete the binding authentication.

[0022] The server uses the device's public key to verify the signature result. If it is correct, the key binding is completed. If the device does not return the signature in time or the signature is incorrect, the temporary token becomes invalid and needs to be re-applied.

[0023] Step 1-2: For metering devices with wired interfaces, connect the metering devices to the network through a serial port server or converter, and communicate with the data acquisition module through the network; For metering equipment with wireless communication, the data acquisition module exchanges data with the metering equipment through the wireless sensor access protocol; For metering devices with API interfaces, the data acquisition module provides a RESTful API, and the metering device uploads data to the data acquisition module through HTTP requests; For measuring devices with Web services, the data acquisition module exchanges data with the measuring devices through SOAP or REST; In this embodiment, the metering device interacts with the data acquisition module through a RESTful interface and uses standard HTTP (such as POST or PUT) to upload structured measurement data in JSON format to the server.

[0024] However, during use, some older models or devices with specific industrial protocols only support data exchange based on WebService.

[0025] Therefore, in this embodiment, the data acquisition module supports SOAP over HTTP or RESTful Web Service calling mode, and actively calls the service interface exposed by the device or responds to device push at regular intervals.

[0026] Steps 1-3: Deploy an edge gateway cluster on the metering device side. Each gateway has a built-in lightweight inference engine to perform abnormal pattern recognition and data downsampling on the raw data collected by the metering device, and only upload data fragments that are determined to be key events or representative features to the data collection module.

[0027] In this embodiment, the edge gateway cluster will be identified by the server cluster as another type of "metering device", and its access method is similar to that of the metering device.

[0028] The gateway's built-in lightweight inference engine is IsolationForest, a lightweight anomaly detection tree model (existing technology model).

[0029] When in use, the model input data is generally N sampling values ​​in a sliding window (such as voltage, current, frequency, etc.), and the output data is the "abnormality score" of each sample. Scores higher than the threshold are judged as abnormal.

[0030] In this embodiment, data downsampling uses a sliding window + event trigger mechanism to upload only data that meets the following conditions: There are obvious fluctuations, transitions, and mutations (such as out-of-threshold jumps > 5%); Clips marked as anomalous (IsolationForest model output); Periodic representative points (e.g., 1 point every 5 minutes).

[0031] The specific algorithm for data downsampling can use the Top-K change rate screening algorithm to retain the data points with the most severe fluctuations, and use the LTTB (Largest-Triangle-Three-Buckets) algorithm to downsample the line chart and retain key trend features.

[0032] Step 2: The data processing module performs unified preprocessing on heterogeneous data and establishes a unified data resource set; When executing step 2, the preprocessing of the data processing module includes data format conversion, data cleaning, and compression encryption. The specific steps are as follows: Step 2-1: Use the preset data standard dictionary to convert the data format of heterogeneous data to obtain standard data with a unified format; Step 2-1: Clean the outliers, missing values, and duplicate data in the standard data, filter out invalid data, and obtain the cleaned standard data; Step 2-3: Use the GZIP algorithm to compress the cleaned standard data, and encrypt the compressed data using symmetric encryption; Step 2-4: Gather the preprocessed data to establish a unified data resource collection.

[0033] Step 3: The cloud server module distributes the data resource collection to obtain the stored data, and uses a multi-level permission control mechanism and a desensitized data encryption strategy to process the stored data; When executing step 3, the cloud server module specifically performs the following steps: Step 3-1: Establish a distributed database to store data resource collection; This embodiment establishes an HBase distributed database and adopts Hash Partitioning or Range Partitioning to perform distributed storage on a data resource collection.

[0034] Step 3-2: Set up multi-level permission control and establish a mathematical model of users, roles, and permissions to express that different users have different permissions according to their roles; use OAuth2.0 authentication to authenticate the legitimacy of user identities; The mathematical model used in this embodiment is a role-based access control model, namely, an RBAC model, which defines a mapping relationship of user→role→authority to control the access scope of different users.

[0035] The inputs to the RBAC model include: User: system login account, such as "admin" or "technician1"; Role: An abstract set of permissions, such as "Administrator", "Operator", and "Read-only User"; Permission: The specific operation capability on resources, such as "query data", "download report", and "device registration".

[0036] If the permissions are met, the model returns the access-granted flag; otherwise, it returns the access-denied flag.

[0037] Step 3-3: Encrypt and store sensitive data, adopt an off-site backup strategy, and regularly back up the distributed database.

[0038] In this embodiment, sensitive data encryption specifically uses a symmetric encryption algorithm (such as AES-256) to encrypt and store sensitive fields, such as device identification, user information, location information, etc.

[0039] The encryption keys are centrally managed by the Key Management Service (KMS), and field-level encryption is completed before data is stored.

[0040] The off-site backup strategy specifically adopts a cold and hot backup + off-site storage mechanism, regularly copying the master database snapshot to a backup site (such as a cloud storage bucket in another region or a local data center) to ensure disaster recovery capabilities.

[0041] In this embodiment, the master node database generates a snapshot every 6 hours, and the snapshot is asynchronously copied to the cloud and backed up using rsync, pg_basebackup or other distributed replication tools.

[0042] Step 4: The data analysis module builds a machine learning model to predict the operating status of the metering equipment based on the stored data. It also uses anomaly detection algorithms to detect anomalies in historical data and generate equipment operating status reports and alarm information. When executing step 4, the data analysis module performs the following steps: Step 4-1: Use streaming data processing methods to process the stored data, calculate key indicators, and use them to detect the operating status of the metering equipment and generate historical data. In this embodiment, the streaming data processing method specifically uses a stream processing framework (such as Apache Flink) to perform real-time calculations on stored data and extract key operating indicators (such as current, voltage, temperature, error rate, etc.).

[0043] The data analysis module performs sliding window aggregation based on the data timestamp and device ID, and calculates indicators such as mean, standard deviation, maximum value, minimum value, trend slope, etc. for subsequent model training and status evaluation.

[0044] Step 4-2: Build a machine learning model and train it using historical data. Use the trained machine learning model to predict the operating status of the metering equipment and generate an equipment operating status report. The machine learning model used in this embodiment is a supervised learning model, specifically an XGBoost model or an LSTM model. The machine learning model will predict the operating status of the device.

[0045] The input of the machine learning model is the historical key indicator features, and the output is the equipment operation status label (such as "normal", "deviation", "abnormal").

[0046] For example, the calculation results of the XGBoost model are as follows: Input features include [voltage_avg, temperature_slope, responseTime_stddev, load_rate]; The output features include the operating status classification label [0: normal, 1: abnormal trend, 2: obvious abnormality].

[0047] Step 4-3: The anomaly detection algorithm is an adaptive threshold algorithm. Specifically, it performs rolling window statistical analysis on historical data, automatically calculates the normal indicator distribution of the device under different time periods, different loads, and different business scenarios, and dynamically generates alarm thresholds based on quantiles. When the data exceeds the adaptive threshold, an alarm is triggered and an alarm message is generated. In this embodiment, an adaptive threshold algorithm based on quantiles is used for anomaly detection to adapt to changes in the business scenarios and operating conditions of the device.

[0048] Use a rolling window to perform statistical analysis on historical data, automatically estimate the "normal fluctuation range" of the current device, and generate dynamic thresholds based on this.

[0049] Specifically, the distribution of statistical indicators is calculated with a rolling cycle of 24 hours, and the 5th percentile (P5) and 95th percentile (P95) are extracted as dynamic upper and lower thresholds. Real-time data outside this range is considered abnormal.

[0050] Step 4-4: Send historical data, equipment operation status report and alarm information to the user-side application module.

[0051] Step 5: The user-side application module provides an interactive platform based on the web or mobile terminal to display historical data, equipment operation status reports and alarm information to users, thereby conducting intelligent visual management of metering equipment.

[0052] When executing step 5, the user-side application module specifically performs the following steps: Step 5-1: For the web side, the front-end uses Vue.js and React frameworks; the user-side application module provides a RESTful API to interact with the front-end for data; The web terminal provides interfaces for data query, equipment operation status report query, and alarm information query; Step 5-2: For the mobile terminal, the front end establishes an APP terminal and exchanges data with the user-side application module through the mobile network; Step 5-3: The user-side application module provides multiple alarm channels such as SMS, email, and App push for pushing alarm information.

[0053] The method for intelligent data collection and management of measuring instruments and equipment based on the Internet of Things described in the present invention solves the technical problems of unified and efficient collection and intelligent analysis and management of heterogeneous measuring instrument and equipment data, and realizes intelligent collaboration and visualization application of multi-source data. The present invention supports multiple metering equipment access methods such as wired, wireless, API and Web services, adapts to different manufacturers and communication protocols, introduces edge gateways and lightweight inference engines, realizes local preprocessing and key event reporting, reduces bandwidth and cloud pressure, and realizes device status prediction and accurate alarm through machine learning models and adaptive anomaly detection algorithms. Combined with data desensitization encryption, permission control and OAuth2.0 authentication, it ensures data security and access compliance. The unified management interface of the Web and mobile terminals, combined with multi-channel message push, enhances user perception and responsiveness.

Claims

1. A method for intelligent data collection and management of measuring instruments and equipment based on the Internet of Things, characterized by: The steps include: Step 1: Deploy data acquisition module, data processing module, cloud server module, data analysis module and user application module in the server cluster; The data acquisition module is compatible with multiple access methods for data collection, including wired connection, wireless connection, remote API interface or Web service access; it can realize real-time collection of heterogeneous data generated by measuring instruments during operation through the Internet; Step 2: The data processing module performs unified preprocessing on heterogeneous data and establishes a unified data resource set; Step 3: The cloud server module distributes the data resource collection to obtain the stored data, and uses a multi-level permission control mechanism and a desensitized data encryption strategy to process the stored data; Step 4: The data analysis module builds a machine learning model to predict the operating status of the metering equipment based on the stored data. It also uses anomaly detection algorithms to detect anomalies in historical data and generate equipment operating status reports and alarm information. Step 5: The user-side application module provides an interactive platform based on the web or mobile terminal to display historical data, equipment operation status reports and alarm information to users, thereby conducting intelligent visual management of metering equipment.

2. The method for intelligent data collection and management of measuring instruments and equipment based on the Internet of Things according to claim 1, characterized in that: When executing step 1, the specific steps performed by the data acquisition module are as follows: Step 1-1: Register and authenticate the metering device that is connected for the first time, and obtain basic information about the metering device, including device ID, model, manufacturer, and firmware version; After the registered metering device has passed the legitimacy verification, a unique device key is assigned to the metering device and the key is returned to the metering device through a secure channel; A temporary token mechanism is used during the key return process, and the metering device completes the binding and returns the signature verification within the predetermined time; Step 1-2: For metering devices with wired interfaces, connect the metering devices to the network through a serial port server or converter, and communicate with the data acquisition module through the network; For metering equipment with wireless communication, the data acquisition module exchanges data with the metering equipment through the wireless sensor access protocol; For metering devices with API interfaces, the data acquisition module provides a RESTful API, and the metering device uploads data to the data acquisition module through HTTP requests; For measuring devices with Web services, the data acquisition module exchanges data with the measuring devices through SOAP or REST; Steps 1-3: Deploy an edge gateway cluster on the metering device side. Each gateway has a built-in lightweight inference engine to perform abnormal pattern recognition and data downsampling on the raw data collected by the metering device, and only upload data fragments that are determined to be key events or representative features to the data collection module.

3. The method for intelligent data collection and management of measuring instruments and equipment based on the Internet of Things according to claim 1, characterized in that: When executing step 2, the preprocessing of the data processing module includes data format conversion, data cleaning, and compression encryption. The specific steps are as follows: Step 2-1: Use the preset data standard dictionary to convert the data format of heterogeneous data to obtain standard data with a unified format; Step 2-1: Clean the outliers, missing values, and duplicate data in the standard data, filter out invalid data, and obtain the cleaned standard data; Step 2-3: Use the GZIP algorithm to compress the cleaned standard data, and encrypt the compressed data using symmetric encryption; Step 2-4: Gather the preprocessed data to establish a unified data resource collection.

4. The method for intelligent data collection and management of measuring instruments and equipment based on the Internet of Things according to claim 1, characterized in that: When executing step 3, the cloud server module specifically performs the following steps: Step 3-1: Establish a distributed database to store data resource collection; Step 3-2: Set up multi-level permission control and establish a mathematical model of users, roles, and permissions to express that different users have different permissions according to their roles; Use OAuth2.0 authentication method to authenticate the legitimacy of user identity; Step 3-3: Encrypt and store sensitive data, adopt an off-site backup strategy, and regularly back up the distributed database.

5. The method for intelligent data collection and management of measuring instruments and equipment based on the Internet of Things according to claim 1, characterized in that: When executing step 4, the data analysis module performs the following steps: Step 4-1: Use streaming data processing methods to process the stored data, calculate key indicators, and use them to detect the operating status of the metering equipment and generate historical data. Step 4-2: Build a machine learning model and train it using historical data. Use the trained machine learning model to predict the operating status of the metering equipment and generate an equipment operating status report. Step 4-3: The anomaly detection algorithm is an adaptive threshold algorithm. Specifically, it performs rolling window statistical analysis on historical data, automatically calculates the normal indicator distribution of the device under different time periods, different loads, and different business scenarios, and dynamically generates alarm thresholds based on quantiles. When the data exceeds the adaptive threshold, an alarm is triggered and an alarm message is generated. Step 4-4: Send historical data, equipment operation status report and alarm information to the user-side application module.

6. The method for intelligent data collection and management of measuring instruments and equipment based on the Internet of Things according to claim 1, characterized in that: When executing step 5, the user-side application module specifically performs the following steps: Step 5-1: For the web side, the front-end uses Vue.js and React frameworks; the user-side application module provides a RESTful API to interact with the front-end for data; The web terminal provides interfaces for data query, equipment operation status report query, and alarm information query; Step 5-2: For the mobile terminal, the front end establishes an APP terminal and exchanges data with the user-side application module through the mobile network; Step 5-3: The user-side application module provides multiple alarm channels such as SMS, email, and App push for pushing alarm information.

Images