Internet of Things equipment monitoring data stream processing method and system

Abstract

The invention discloses an Internet of Things equipment monitoring data stream processing method and system. The method comprises the following steps: step 1, collecting and preprocessing multi-source heterogeneous data; step 2, dynamic routing distribution; step 3, edge calculation processing; 4, performing real-time feedback regulation and control; 5, performing anomaly detection and fault tolerance; step 6, data persistence storage; step 7, performing multi-dimensional analysis; and step 8, security enhancement processing. According to the method, intelligent distribution of data streams is realized through the dynamic routing engine, optimal nodes are matched, and resource waste is reduced; the edge computing node cluster completes preliminary screening and filtering, reduces core pressure, adopts an active-active storage subsystem, combines hot and cold storage, reduces cost and guarantees access performance, a safety protection module constructs a full-link safety system, and cooperates with a dual-mode redundancy architecture to improve system availability and fault switching second-level response.

Description

Technical Field

[0001] The present invention relates to the field of Internet of Things data processing systems, and in particular to a method and system for streaming processing of Internet of Things device monitoring data. Background Art

[0002] With the rapid development of IoT technology, efficient processing of massive amounts of device monitoring data has become a key bottleneck restricting system performance. Traditional batch processing models suffer from significant latency and cannot meet real-time monitoring requirements, while directly storing raw data leads to exponentially increased storage costs. While existing streaming processing technologies enable near-real-time analysis, they often suffer from rigid resource allocation and a lack of exception handling mechanisms. This is especially true in complex scenarios like smart campuses, where diverse device types and significantly different data characteristics exist. Traditional solutions struggle to balance processing efficiency and resource utilization.

[0003] The existing technology has three major defects: first, the static processing strategy leads to a waste of computing resources, for example, a unified processing frequency is used for periodically fluctuating data; second, the fault tolerance mechanism is weak, and a single node failure can easily cause a data chain interruption; third, the feedback adjustment is delayed, and the execution path cannot be dynamically optimized according to the real-time processing load. These problems are particularly prominent in data peak scenarios, which directly lead to increased response delays in the monitoring system and decreased accuracy in identifying equipment faults. Therefore, a method and system for streaming processing of IoT device monitoring data is proposed. Summary of the Invention

[0004] In view of the shortcomings of the existing technology, the present invention provides a method and system for streaming processing of IoT device monitoring data to solve the problems raised in the above background technology.

[0005] To achieve the above object, the present invention provides the following technical solution: a method for processing streaming data of IoT device monitoring data, comprising the following steps: Step 1: Multi-source heterogeneous data collection and preprocessing: Establish a multi-protocol adaptation layer to support concurrent access of Modbus, MQTT, and CoAP IoT protocols, parse the format of original data packets through the data feature recognition module, and perform data cleaning operations; Step 2: Dynamic routing distribution: Build a multi-dimensional labeling system based on data feature vectors, and intelligently distribute data streams to the optimal processing nodes through a load balancer combined with real-time resource monitoring data, supporting hot-swappable node expansion. Step 3: Edge computing processing: A lightweight rule engine is deployed on the edge gateway to perform preliminary filtering and format conversion preprocessing operations, and to calculate key indicators in real time. Step 4: Real-time feedback control: Build a closed-loop control loop to dynamically adjust the sampling frequency and algorithm complexity based on processing delay and resource utilization indicators; Step 5: Anomaly Detection and Fault Tolerance: Adopting dual-mode redundant architecture, the main processing channel and backup channel run in parallel, and the heartbeat detection mechanism realizes fault switching within seconds; Step 6: Data persistent storage: Design a hierarchical storage strategy to implement hot storage for critical data on alarm events and device status, and cold storage for raw data streams; Step 7: Multi-dimensional analysis: Establish a fusion architecture of time series database and graph database to support device status evolution tracking and correlation analysis, and use a distributed computing framework to achieve real-time data aggregation; Step 8: Security Enhancement Processing: Deploy the national secret SM4 encryption module in the data transmission link, implement dynamic desensitization processing for sensitive fields, and establish an access control list based on the zero-trust architecture; Establish a device fingerprint feature library, use regular expression matching to filter illegal characters, adopt a sliding window mechanism to achieve data deduplication, and identify and eliminate abnormal data sources by comparing the device fingerprint feature library to ensure data quality; Through multi-source heterogeneous data collection and preprocessing, the comprehensiveness and accuracy of the data are ensured; the dynamic routing distribution mechanism improves data processing efficiency, and support for hot-swappable node expansion enhances the flexibility and scalability of the system; edge computing processing reduces data transmission latency and realizes real-time calculation of key indicators; the real-time feedback control mechanism dynamically adjusts system parameters and optimizes resource utilization; anomaly detection and fault tolerance mechanisms ensure the stable operation of the system; the data persistence storage strategy balances storage costs and data access speed; multi-dimensional analysis supports in-depth tracking and correlation analysis of device status; security enhancement processing protects the security of data transmission and access, providing a comprehensive, efficient and secure solution for IoT device monitoring.

[0006] Preferably, the data cleaning operation includes an illegal character filtering mechanism based on regular expression matching, a sliding window mechanism to achieve data deduplication, and identification of abnormal data sources through a device fingerprint feature library; In illegal character filtering, a set of regular expressions covering common illegal characters is constructed. The input data is matched character by character, and the successfully matched illegal characters are eliminated. When deduplicating data, the sliding window size is set, and the data within the window is stored in a hash table. The hash table is checked before new data enters the window, and any existing data is discarded. When identifying abnormal data sources, normal device data features are collected in advance to build a fingerprint feature library. New data is compared with the features in the library. If the difference exceeds the threshold, it is determined to be an abnormal data source. Illegal character filtering is based on regular expressions, which can accurately identify and remove all kinds of illegal characters, ensure data format specifications, and avoid subsequent processing errors caused by illegal characters. The sliding window mechanism realizes data deduplication. While ensuring data integrity, it effectively reduces the storage resources and processing time occupied by duplicate data, improves system operation efficiency, and identifies abnormal data sources through the device fingerprint feature library. It can discover and process abnormal data in advance, enhance system stability and reliability, provide a high-quality data foundation for subsequent data analysis, and ensure the accuracy and effectiveness of IoT device monitoring data.

[0007] Preferably, the dynamic routing distribution includes establishing a QoS guarantee model to set up a three-level priority queue for key equipment data flows, and adopting a weighted round-robin algorithm combined with node processing capabilities to achieve differentiated service distribution; In dynamic routing distribution, the QoS guarantee model sets high, medium, and low priority queues for key device data flows to ensure that important data is transmitted first. The weighted polling algorithm assigns weights based on the real-time processing capabilities of edge computing nodes (such as CPU utilization and memory occupancy), dynamically adjusts the data flow distribution ratio, and achieves load balancing and differentiated services, ensuring that high-priority data flows are prioritized within the node's processing capacity. The dynamic routing distribution mechanism achieves efficient and intelligent management of IoT device monitoring data streams by establishing a QoS guarantee model and adopting a weighted polling algorithm. This mechanism ensures that data streams of key devices can be processed first, ensuring the real-time and reliability of the monitoring system. At the same time, combined with differentiated service distribution based on node processing capabilities, it effectively avoids resource waste and performance bottlenecks, improves the stability and response speed of the overall system, and provides solid technical support for remote monitoring and management of IoT devices.

[0008] Preferably, the edge computing process includes deploying a device behavior modeling module based on a finite state machine to predict abnormal device operating status through a pattern matching algorithm; Define a finite state machine model for normal equipment operation, clarifying each state and state transition conditions. Next, collect historical equipment operation data and extract typical behavior patterns. Finally, use a pattern matching algorithm to compare the real-time collected equipment data with predefined patterns. If deviations from the normal pattern are found, the equipment operation status is determined to be abnormal. By deploying a finite state machine-based device behavior modeling module and combining it with a pattern matching algorithm, it is possible to accurately predict the device's operating status. This approach enables the system to detect potential problems in advance and effectively avoid the occurrence or expansion of faults, thereby improving the stability and safety of device operation. At the same time, this prediction mechanism helps to achieve more efficient operation and maintenance management, reduce the need for manual inspections and intervention, reduce operation and maintenance costs, and improve overall operation and maintenance efficiency.

[0009] Preferably, the real-time feedback control includes establishing an adaptive threshold adjustment mechanism, dynamically correcting alarm triggering conditions through a sliding average algorithm, and optimizing resource allocation using a PID control model; The system continuously collects historical data and calculates its sliding average, dynamically adjusting the alarm threshold based on this value to ensure alarm accuracy. At the same time, the PID control model adjusts the resource allocation of processing nodes by comparing the deviation between current resource utilization and target values, such as increasing or decreasing computing resources, to optimize overall performance. The real-time feedback control mechanism significantly improves the flexibility and efficiency of the IoT device monitoring system through adaptive threshold adjustment and PID control model. The adaptive threshold can dynamically adjust the alarm conditions according to actual data, reduce false alarms and missed alarms, and ensure the system's rapid response to abnormal situations. The PID control model ensures that the system can maintain stable operation under high load by precisely adjusting resource allocation, while avoiding resource waste. This intelligent control mechanism not only improves the reliability and stability of the system, but also reduces operation and maintenance costs, providing a strong guarantee for the long-term stable operation of IoT devices.

[0010] Preferably, the anomaly detection and fault tolerance include implementing a data checksum mechanism to verify the integrity of data tampering during transmission, and ensuring the consistency of backup channel data through a distributed consensus algorithm; In the anomaly detection and fault tolerance mechanism, the data checksum mechanism verifies data integrity by calculating the hash value of the transmitted data and comparing it with the preset value. A distributed consensus algorithm (such as Paxos or Raft) synchronizes data status between backup channels and ensures data consistency through majority node confirmation. This ensures that when the primary channel fails, the backup channel can seamlessly take over without data loss. The anomaly detection and fault-tolerance mechanism effectively prevents data tampering during transmission by implementing a data checksum mechanism, ensuring the authenticity and integrity of the data. At the same time, the distributed consensus algorithm is used to ensure data consistency in the backup channel, which not only enhances the redundancy of the system, but also improves the reliability and fault tolerance of the system. When the main processing channel fails, the backup channel can quickly take over to ensure business continuity and data loss, providing a solid guarantee for the stable processing of IoT device monitoring data.

[0011] Preferably, the data persistent storage includes using a columnar storage engine to optimize time series data query performance, establishing a lifecycle management strategy based on data popularity, and supporting dynamic migration of storage media; Columnar storage engines such as ClickHouse are used to accelerate time series data queries through column storage features. Secondly, data heat thresholds are set based on data access frequency and timeliness, and data is divided into hot data, warm data, and cold data, which are stored in high-speed SSDs, medium-speed SAS hard drives, and low-speed tape libraries, respectively. Finally, automated scripts are used to monitor data heat changes and enable dynamic migration between storage media to ensure the optimal balance between storage cost and performance. By optimizing the query performance of time series data through the columnar storage engine, it can quickly respond to query requests and improve system efficiency. The lifecycle management strategy based on data popularity can reasonably allocate storage resources to ensure that high-frequency access data is stored on high-performance media, while reducing the storage cost of low-frequency data. Supporting dynamic migration of storage media further enhances the flexibility and scalability of the system, and can dynamically adjust the storage architecture according to actual needs to ensure data security and availability, providing a solid guarantee for the long-term preservation and efficient use of IoT device monitoring data.

[0012] Preferably, the security enhancement process includes deploying a hardware security module to implement full lifecycle management of encryption keys and implementing access frequency control through a token bucket algorithm; The hardware security module uses a dedicated chip that integrates key generation, storage, update, and destruction functions. It protects key security through physical isolation. The token bucket algorithm generates tokens at a fixed rate, consuming one token per access. Access is restricted if no token is available. Dynamically adjusting the token generation rate enables flexible access frequency control, effectively preventing brute force cracking and DDoS attacks. By deploying hardware security modules, the full life cycle management of encryption keys is achieved, ensuring the security of keys during generation, storage, use and destruction, and preventing the risk of key leakage. At the same time, the token bucket algorithm is used to implement access frequency control, effectively resisting malicious access and attacks, and ensuring the stable operation of the system in high-concurrency scenarios. This comprehensive security strategy not only enhances the confidentiality, integrity and availability of data transmission, but also improves the overall security protection capabilities of the system, providing a solid security guarantee for IoT device monitoring.

[0013] The system for stream processing of IoT device monitoring data adopts the above-mentioned method for stream processing of IoT device monitoring data, including: Multi-protocol access gateway, used to implement concurrent access and protocol conversion of Modbus / MQTT / CoAP IoT protocols; Dynamic routing engine, built-in load balancer and QoS policy configuration center to achieve intelligent distribution of data flow; Edge computing node clusters, consisting of edge gateways that deploy lightweight rule engines, perform initial filtering and real-time computing; Central control module, including closed-loop control circuit and adaptive threshold adjustment unit, to achieve dynamic resource allocation; Active-active storage subsystem, consisting of an NVMe SSD hot storage array and a distributed cold storage cluster, supports data lifecycle management; Real-time analysis engine, integrating time series database and graph database fusion architecture, performs multi-dimensional correlation analysis; Security protection module, including encryption processing unit, access control unit and audit log unit; The multi-protocol access gateway establishes a long connection with the dynamic routing engine through the message bus. The dynamic routing engine uses a two-way communication interface to connect to the edge computing node cluster and the central control module respectively. The active-active storage subsystem is connected to the real-time analysis engine through the RDMA high-speed interface. The security protection module is deployed in series on all data transmission channels in bypass monitoring mode. The multi-protocol access gateway uses multi-threading or asynchronous IO technology to achieve concurrent monitoring and parsing of Modbus, MQTT, and CoAP protocols. It uses protocol conversion middleware to uniformly convert data packets of different protocols into internal data formats, ensuring seamless access to multi-protocol data. The multi-protocol access gateway supports multiple IoT protocols, greatly expanding the system's compatibility. The dynamic routing engine ensures efficient and flexible processing by intelligently distributing data streams. The edge computing node cluster performs local data screening and real-time calculations, reducing data transmission latency. The central control module dynamically allocates resources to optimize system performance. The active-active storage subsystem supports data lifecycle management, ensuring data persistence and availability. The real-time analysis engine's multi-dimensional correlation analysis capabilities provide in-depth insights for device monitoring. The security protection module comprehensively guarantees the security of data transmission and storage.

[0014] Preferably, the encryption processing unit supports hardware acceleration of the SM4 / SM9 national encryption algorithm; the access control unit integrates the RBAC authority model and the device authentication submodule; the audit log unit uses blockchain evidence storage technology to ensure that operations are traceable; The encryption processing unit integrates the SM4 / SM9 national encryption algorithm through a dedicated hardware chip, and uses a hardware acceleration engine to improve encryption and decryption efficiency to ensure data transmission security. The access control unit configures user roles and permissions through the RBAC model, and combines the device authentication sub-module to verify device identity to achieve fine-grained access control. The audit log unit uses blockchain technology to record operation records on the chain, leveraging the blockchain's tamper-proof characteristics to ensure the integrity and traceability of operation logs. The encryption processing unit supports hardware acceleration of the SM4 / SM9 national secret algorithm, significantly improving the speed and efficiency of encryption and decryption, ensuring the security of IoT device monitoring data during transmission, and effectively preventing data leakage and illegal access. The access control unit integrates the RBAC permission model and device authentication sub-module to achieve fine-grained access control for users and devices, ensuring that only authorized users and devices can access system resources, enhancing the security and controllability of the system. The audit log unit uses blockchain evidence storage technology to ensure the non-tamperability and traceability of operation logs, providing a reliable basis for system security audits and incident investigations, and further improving the security and credibility of the system.

[0015] In summary, compared with the prior art, the present invention provides a method and system for processing IoT device monitoring data streams, which has the following beneficial effects: This invention builds a multi-dimensional label system through a dynamic routing engine, and realizes intelligent distribution of data streams in combination with real-time resource monitoring data, so that the system can dynamically match the optimal processing node according to data characteristics, effectively solving the resource waste problem caused by traditional static allocation. The lightweight rule engine deployed in the edge computing node cluster completes the initial screening and format conversion at the data source end, improves the interception rate of invalid data, and significantly reduces the data throughput pressure of the core processing link. In conjunction with the feedback adjustment mechanism established by the closed-loop control loop, it can automatically optimize the sampling frequency and algorithm complexity according to the real-time processing load, ensuring rapid convergence to the optimal execution path in the scenario of sudden device status changes, thereby reducing response delays. A dual-active storage subsystem is used to achieve full data lifecycle management. Through the collaborative work of the NVMe SSD hot storage array and the distributed cold storage cluster, the original data storage cost is reduced while ensuring millisecond-level access performance of key alarm data. The fusion architecture of the time series database and the graph database supports second-level tracing of the device status evolution trajectory, which improves the efficiency of correlation analysis compared to traditional solutions. The national secret SM4 encryption and dynamic desensitization mechanism deployed in the security protection module build a full-link security protection system. Combined with the access control strategy of the zero-trust architecture, the data transmission link has the ability to resist lateral penetration attacks. The overall system availability is improved through the dual-mode redundant architecture, and the fault switching time is compressed to the response range of seconds. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a step diagram of the invented method for streaming processing of IoT device monitoring data.

[0017] Figure 2 It is a schematic diagram of the system for monitoring data streaming of IoT devices of the invention. DETAILED DESCRIPTION

[0018] The present invention provides a technical solution, a method for processing streaming data of IoT device monitoring data, which includes the following steps: Step 1: Multi-source heterogeneous data collection and preprocessing: Establish a multi-protocol adaptation layer to support concurrent access of Modbus, MQTT, and CoAP IoT protocols, parse the format of original data packets through the data feature recognition module, and perform data cleaning operations; Step 2: Dynamic routing distribution: Build a multi-dimensional labeling system based on data feature vectors, and intelligently distribute data streams to the optimal processing nodes through a load balancer combined with real-time resource monitoring data, supporting hot-swappable node expansion. Step 3: Edge computing processing: A lightweight rule engine is deployed on the edge gateway to perform preliminary filtering and format conversion preprocessing operations, and to calculate key indicators in real time. Step 4: Real-time feedback control: Build a closed-loop control loop to dynamically adjust the sampling frequency and algorithm complexity based on processing delay and resource utilization indicators; Step 5: Anomaly Detection and Fault Tolerance: Adopting dual-mode redundant architecture, the main processing channel and backup channel run in parallel, and the heartbeat detection mechanism realizes fault switching within seconds; Step 6: Data persistent storage: Design a hierarchical storage strategy to implement hot storage for critical data on alarm events and device status, and cold storage for raw data streams; Step 7: Multi-dimensional analysis: Establish a fusion architecture of time series database and graph database to support device status evolution tracking and correlation analysis, and use a distributed computing framework to achieve real-time data aggregation; Step 8: Security Enhancement Processing: Deploy the national secret SM4 encryption module in the data transmission link, implement dynamic desensitization processing for sensitive fields, and establish an access control list based on the zero-trust architecture; Establish a device fingerprint feature library, use regular expression matching to filter illegal characters, adopt a sliding window mechanism to achieve data deduplication, and identify and eliminate abnormal data sources by comparing the device fingerprint feature library to ensure data quality; Through multi-source heterogeneous data collection and preprocessing, the comprehensiveness and accuracy of the data are ensured; the dynamic routing distribution mechanism improves data processing efficiency, and support for hot-swappable node expansion enhances the flexibility and scalability of the system; edge computing processing reduces data transmission latency and realizes real-time calculation of key indicators; the real-time feedback control mechanism dynamically adjusts system parameters and optimizes resource utilization; anomaly detection and fault tolerance mechanisms ensure the stable operation of the system; the data persistence storage strategy balances storage costs and data access speed; multi-dimensional analysis supports in-depth tracking and correlation analysis of device status; security enhancement processing protects the security of data transmission and access, providing a comprehensive, efficient and secure solution for IoT device monitoring.

[0019] See also Figure 1 ,The data cleaning operation includes an illegal character filtering mechanism based on ,regular expression matching, a sliding window mechanism to achieve data deduplication, and ,identification of abnormal data sources through the device fingerprint feature library; In illegal character filtering, a set of regular expressions covering common illegal characters is constructed. The input data is matched character by character, and the successfully matched illegal characters are eliminated. When deduplicating data, the sliding window size is set, and the data within the window is stored in a hash table. The hash table is checked before new data enters the window, and any existing data is discarded. When identifying abnormal data sources, normal device data features are collected in advance to build a fingerprint feature library. New data is compared with the features in the library. If the difference exceeds the threshold, it is determined to be an abnormal data source. Illegal character filtering is based on regular expressions, which can accurately identify and remove all kinds of illegal characters, ensure data format specifications, and avoid subsequent processing errors caused by illegal characters. The sliding window mechanism realizes data deduplication. While ensuring data integrity, it effectively reduces the storage resources and processing time occupied by duplicate data, improves system operation efficiency, and identifies abnormal data sources through the device fingerprint feature library. It can discover and process abnormal data in advance, enhance system stability and reliability, provide a high-quality data foundation for subsequent data analysis, and ensure the accuracy and effectiveness of IoT device monitoring data.

[0020] See also Figure 1 ,Dynamic routing distribution includes establishing a QoS guarantee model to set three-level priority queues for ,critical equipment data flows, and using a weighted round-robin algorithm combined with ,node processing capabilities to achieve differentiated service distribution; In dynamic routing distribution, the QoS guarantee model sets high, medium, and low priority queues for key device data flows to ensure that important data is transmitted first. The weighted polling algorithm assigns weights based on the real-time processing capabilities of edge computing nodes (such as CPU utilization and memory occupancy), dynamically adjusts the data flow distribution ratio, and achieves load balancing and differentiated services, ensuring that high-priority data flows are prioritized within the node's processing capacity. The dynamic routing distribution mechanism achieves efficient and intelligent management of IoT device monitoring data streams by establishing a QoS guarantee model and adopting a weighted polling algorithm. This mechanism ensures that data streams of key devices can be processed first, ensuring the real-time and reliability of the monitoring system. At the same time, combined with differentiated service distribution based on node processing capabilities, it effectively avoids resource waste and performance bottlenecks, improves the stability and response speed of the overall system, and provides solid technical support for remote monitoring and management of IoT devices.

[0021] See also Figure 1 ,Edge computing processing includes deploying a device behavior modeling module based on a finite state machine, and predicting abnormal device operating status through a pattern matching algorithm; Define a finite state machine model for normal equipment operation, clarifying each state and state transition conditions. Next, collect historical equipment operation data and extract typical behavior patterns. Finally, use a pattern matching algorithm to compare the real-time collected equipment data with predefined patterns. If deviations from the normal pattern are found, the equipment operation status is determined to be abnormal. By deploying a finite state machine-based device behavior modeling module and combining it with a pattern matching algorithm, it is possible to accurately predict the device's operating status. This approach enables the system to detect potential problems in advance and effectively avoid the occurrence or expansion of faults, thereby improving the stability and safety of device operation. At the same time, this prediction mechanism helps to achieve more efficient operation and maintenance management, reduce the need for manual inspections and intervention, reduce operation and maintenance costs, and improve overall operation and maintenance efficiency.

[0022] See also Figure 1 ,Real-time feedback control includes establishing an adaptive threshold adjustment mechanism,,dynamically correcting alarm trigger conditions through a sliding average,algorithm, and optimizing resource allocation using a PID control model; The system continuously collects historical data and calculates its sliding average, dynamically adjusting the alarm threshold based on this value to ensure alarm accuracy. At the same time, the PID control model adjusts the resource allocation of processing nodes by comparing the deviation between current resource utilization and target values, such as increasing or decreasing computing resources, to optimize overall performance. The real-time feedback control mechanism significantly improves the flexibility and efficiency of the IoT device monitoring system through adaptive threshold adjustment and PID control model. The adaptive threshold can dynamically adjust the alarm conditions according to actual data, reduce false alarms and missed alarms, and ensure the system's rapid response to abnormal situations. The PID control model ensures that the system can maintain stable operation under high load by precisely adjusting resource allocation, while avoiding resource waste. This intelligent control mechanism not only improves the reliability and stability of the system, but also reduces operation and maintenance costs, providing a strong guarantee for the long-term stable operation of IoT devices.

[0023] See also Figure 1 ,Anomaly detection and fault tolerance include implementing data checksum mechanisms to verify the integrity of data tampering during transmission, and ensuring the consistency of backup channel data through distributed consensus algorithms; In the anomaly detection and fault tolerance mechanism, the data checksum mechanism verifies data integrity by calculating the hash value of the transmitted data and comparing it with the preset value. A distributed consensus algorithm (such as Paxos or Raft) synchronizes data status between backup channels and ensures data consistency through majority node confirmation. This ensures that when the primary channel fails, the backup channel can seamlessly take over without data loss. The anomaly detection and fault-tolerance mechanism effectively prevents data tampering during transmission by implementing a data checksum mechanism, ensuring the authenticity and integrity of the data. At the same time, the distributed consensus algorithm is used to ensure data consistency in the backup channel, which not only enhances the redundancy of the system, but also improves the reliability and fault tolerance of the system. When the main processing channel fails, the backup channel can quickly take over to ensure business continuity and data loss, providing a solid guarantee for the stable processing of IoT device monitoring data.

[0024] See also Figure 1 ,Data persistence storage includes using a columnar storage engine to optimize time series data query performance, establishing a lifecycle management strategy based on data popularity, and supporting dynamic migration of storage media; Columnar storage engines such as ClickHouse are used to accelerate time series data queries through column storage features. Secondly, data heat thresholds are set based on data access frequency and timeliness, and data is divided into hot data, warm data, and cold data, which are stored in high-speed SSDs, medium-speed SAS hard drives, and low-speed tape libraries, respectively. Finally, automated scripts are used to monitor data heat changes and enable dynamic migration between storage media to ensure the optimal balance between storage cost and performance. By optimizing the query performance of time series data through the columnar storage engine, it can quickly respond to query requests and improve system efficiency. The lifecycle management strategy based on data popularity can reasonably allocate storage resources to ensure that high-frequency access data is stored on high-performance media, while reducing the storage cost of low-frequency data. Supporting dynamic migration of storage media further enhances the flexibility and scalability of the system, and can dynamically adjust the storage architecture according to actual needs to ensure data security and availability, providing a solid guarantee for the long-term preservation and efficient use of IoT device monitoring data.

[0025] See also Figure 1 ,Security enhancement processing includes deploying hardware security modules to ,implement encryption key life cycle management and access frequency ,control through the token bucket algorithm; The hardware security module uses a dedicated chip that integrates key generation, storage, update, and destruction functions. It protects key security through physical isolation. The token bucket algorithm generates tokens at a fixed rate, consuming one token per access. Access is restricted if no token is available. Dynamically adjusting the token generation rate enables flexible access frequency control, effectively preventing brute force cracking and DDoS attacks. By deploying hardware security modules, the full life cycle management of encryption keys is achieved, ensuring the security of keys during generation, storage, use and destruction, and preventing the risk of key leakage. At the same time, the token bucket algorithm is used to implement access frequency control, effectively resisting malicious access and attacks, and ensuring the stable operation of the system in high-concurrency scenarios. This comprehensive security strategy not only enhances the confidentiality, integrity and availability of data transmission, but also improves the overall security protection capabilities of the system, providing a solid security guarantee for IoT device monitoring.

[0026] The system for IoT device monitoring data stream processing adopts the above IoT device monitoring data stream processing method, please refer to Figure 1 and Figure 2 ,include: Multi-protocol access gateway, used to implement concurrent access and protocol conversion of Modbus / MQTT / CoAP IoT protocols; Dynamic routing engine, built-in load balancer and QoS policy configuration center to achieve intelligent distribution of data flow; Edge computing node clusters, consisting of edge gateways that deploy lightweight rule engines, perform initial filtering and real-time computing; Central control module, including closed-loop control circuit and adaptive threshold adjustment unit, to achieve dynamic resource allocation; Active-active storage subsystem, consisting of an NVMe SSD hot storage array and a distributed cold storage cluster, supports data lifecycle management; Real-time analysis engine, integrating time series database and graph database fusion architecture, performs multi-dimensional correlation analysis; Security protection module, including encryption processing unit, access control unit and audit log unit; The multi-protocol access gateway establishes a persistent connection with the dynamic routing engine via a message bus. The dynamic routing engine uses a bidirectional communication interface to connect to the edge computing node cluster and the central control module. The active-active storage subsystem connects to the real-time analysis engine via an RDMA high-speed interface. The security protection module is deployed in series on all data transmission channels in bypass monitoring mode. The multi-protocol access gateway uses multi-threading or asynchronous IO technology to achieve concurrent monitoring and parsing of Modbus, MQTT, and CoAP protocols. It uses protocol conversion middleware to uniformly convert data packets of different protocols into internal data formats, ensuring seamless access to multi-protocol data. The multi-protocol access gateway supports multiple IoT protocols, greatly expanding the system's compatibility. The dynamic routing engine ensures efficient and flexible processing by intelligently distributing data streams. The edge computing node cluster performs local data screening and real-time calculations, reducing data transmission latency. The central control module dynamically allocates resources to optimize system performance. The active-active storage subsystem supports data lifecycle management, ensuring data persistence and availability. The real-time analysis engine's multi-dimensional correlation analysis capabilities provide in-depth insights for device monitoring. The security protection module comprehensively guarantees the security of data transmission and storage.

[0027] See also Figure 1 and Figure 2 The encryption processing unit supports hardware acceleration of the SM4 / SM9 national encryption algorithm; the access control unit integrates the RBAC permission model and device authentication sub-module; the audit log unit uses blockchain evidence storage technology to ensure operation traceability; The encryption processing unit integrates the SM4 / SM9 national encryption algorithm through a dedicated hardware chip, and uses a hardware acceleration engine to improve encryption and decryption efficiency to ensure data transmission security. The access control unit configures user roles and permissions through the RBAC model, and combines the device authentication sub-module to verify device identity to achieve fine-grained access control. The audit log unit uses blockchain technology to record operation records on the chain, leveraging the blockchain's tamper-proof characteristics to ensure the integrity and traceability of operation logs. The encryption processing unit supports hardware acceleration of the SM4 / SM9 national secret algorithm, significantly improving the speed and efficiency of encryption and decryption, ensuring the security of IoT device monitoring data during transmission, and effectively preventing data leakage and illegal access. The access control unit integrates the RBAC permission model and device authentication sub-module to achieve fine-grained access control for users and devices, ensuring that only authorized users and devices can access system resources, enhancing the security and controllability of the system. The audit log unit uses blockchain evidence storage technology to ensure the non-tamperability and traceability of operation logs, providing a reliable basis for system security audits and incident investigations, and further improving the security and credibility of the system.

[0028] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

[0029] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A method for processing streaming data of IoT device monitoring, characterized in that: The steps include: Step 1: Multi-source heterogeneous data collection and preprocessing: Establish a multi-protocol adaptation layer to support concurrent access of Modbus, MQTT, and CoAP IoT protocols, parse the format of original data packets through the data feature recognition module, and perform data cleaning operations; Step 2: Dynamic routing distribution: Build a multi-dimensional labeling system based on data feature vectors, and intelligently distribute data streams to the optimal processing nodes through a load balancer combined with real-time resource monitoring data, supporting hot-swappable node expansion. Step 3: Edge computing processing: A lightweight rule engine is deployed on the edge gateway to perform preliminary filtering and format conversion preprocessing operations, and to calculate key indicators in real time. Step 4: Real-time feedback control: Build a closed-loop control loop to dynamically adjust the sampling frequency and algorithm complexity based on processing delay and resource utilization indicators; Step 5: Anomaly Detection and Fault Tolerance: Adopting dual-mode redundant architecture, the main processing channel and backup channel run in parallel, and the heartbeat detection mechanism realizes fault switching within seconds; Step 6: Data persistent storage: Design a hierarchical storage strategy to implement hot storage for critical data on alarm events and device status, and cold storage for raw data streams; Step 7: Multi-dimensional analysis: Establish a fusion architecture of time series database and graph database to support device status evolution tracking and correlation analysis, and use a distributed computing framework to achieve real-time data aggregation; Step 8: Security Enhancement Processing: Deploy the national secret SM4 encryption module in the data transmission link, implement dynamic desensitization processing on sensitive fields, and establish an access control list based on the zero trust architecture.

2. The method for streaming monitoring data of an Internet of Things device according to claim 1, wherein: The data cleaning operation includes an illegal character filtering mechanism based on regular expression matching, a sliding window mechanism for data deduplication, and identification of abnormal data sources through a device fingerprint feature library.

3. The method for streaming monitoring data of an Internet of Things device according to claim 1, wherein: The dynamic routing distribution includes establishing a QoS guarantee model to set up three-level priority queues for key equipment data flows, and adopting a weighted polling algorithm combined with node processing capabilities to achieve differentiated service distribution.

4. The method for streaming monitoring data of an Internet of Things device according to claim 1, wherein: The edge computing process includes deploying a device behavior modeling module based on a finite state machine, and predicting abnormal device operating status through a pattern matching algorithm.

5. The method for streaming monitoring data of an Internet of Things device according to claim 1, wherein: The real-time feedback control includes establishing an adaptive threshold adjustment mechanism, dynamically correcting alarm triggering conditions through a sliding average algorithm, and optimizing resource allocation using a PID control model.

6. The method for streaming monitoring data of an Internet of Things device according to claim 1, characterized in that: The anomaly detection and fault tolerance include implementing a data checksum mechanism to verify the integrity of data tampering during transmission, and ensuring the consistency of backup channel data through a distributed consensus algorithm.

7. The method for streaming monitoring data of an Internet of Things device according to claim 1, wherein: The data persistent storage includes using a columnar storage engine to optimize time series data query performance, establishing a lifecycle management strategy based on data popularity, and supporting dynamic migration of storage media.

8. The method for streaming monitoring data of an Internet of Things device according to claim 1, wherein: The security enhancement process includes deploying a hardware security module to implement full life cycle management of encryption keys and implementing access frequency control through a token bucket algorithm.

9. A system for streaming monitoring data of an Internet of Things device, comprising: a method for streaming monitoring data of an Internet of Things device according to any one of claims 1 to 8, wherein: include: Multi-protocol access gateway, used to implement concurrent access and protocol conversion of Modbus / MQTT / CoAP IoT protocols; Dynamic routing engine, built-in load balancer and QoS policy configuration center to achieve intelligent distribution of data flow; Edge computing node clusters, consisting of edge gateways that deploy lightweight rule engines, perform initial filtering and real-time computing; Central control module, including closed-loop control circuit and adaptive threshold adjustment unit, to achieve dynamic resource allocation; Active-active storage subsystem, consisting of an NVMe SSD hot storage array and a distributed cold storage cluster, supports data lifecycle management; Real-time analysis engine, integrating time series database and graph database fusion architecture, performs multi-dimensional correlation analysis; Security protection module, including encryption processing unit, access control unit and audit log unit; The multi-protocol access gateway establishes a long connection with the dynamic routing engine through the message bus. The dynamic routing engine uses a two-way communication interface to connect to the edge computing node cluster and the central control module respectively. The active-active storage subsystem is connected to the real-time analysis engine through the RDMA high-speed interface. The security protection module is deployed in series in all data transmission channels in bypass monitoring mode.

10. The system for streaming monitoring data of IoT devices according to claim 9, characterized in that: The encryption processing unit supports hardware acceleration of the SM4 / SM9 national encryption algorithm; the access control unit integrates the RBAC authority model and the device authentication sub-module; the audit log unit uses blockchain evidence storage technology to ensure that operations are traceable.

Images