Method and system for collecting and managing device information based on internet of things

Abstract

This invention discloses a device information collection and management method and system based on the Internet of Things (IoT), comprising the following steps: dynamic protocol loading; parallel processing architecture; intelligent routing selection; real-time data collection; data parsing and formatting; data uploading and storage; protocol sandbox isolation; fault detection and recovery; and data backup and recovery. This invention offers significant advantages through its IoT-based device information collection and management method and system. It can quickly and automatically adapt to new protocols, reduce development and maintenance costs, and accommodate more devices; it efficiently processes and transmits data, ensuring fast business response times; it enhances system security and maintainability, reduces security risks and data loss, and improves operational efficiency through a visual interface, ensuring stable and reliable system operation.

Description

Technical Field

[0001] This invention relates to the field of Internet of Things (IoT) technology, and in particular to a method and system for collecting and managing device information based on IoT. Background Technology

[0002] With the rapid development of IoT technology, various IoT devices are emerging rapidly and are widely used in many fields such as industry, home, and transportation. However, the protocols used by different devices are complex, diverse, and frequently updated, which brings huge challenges to the collection and management of device information.

[0003] Traditional methods often require large-scale system modifications when dealing with new protocol access, resulting in high development and maintenance costs. Furthermore, they are difficult to adapt to dynamic changes in protocols quickly, leading to low system compatibility with different devices and severely impacting business continuity.

[0004] Meanwhile, with the explosive growth in the number of IoT devices, the pressure on data processing and transmission is increasing daily. Traditional architectures have significant shortcomings in terms of data processing speed, transmission stability, and reliability, making it difficult to meet the needs of rapid and accurate collection of massive amounts of device data.

[0005] Furthermore, the security and maintainability of IoT systems are also pressing issues that need to be addressed. Potential security risks between protocols, data loss due to system failures, and inconvenience in operation and maintenance all pose threats to the stable operation and data security of IoT systems. Therefore, this paper proposes a device information collection and management method and system based on the Internet of Things (IoT). Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a device information collection and management method and system based on the Internet of Things (IoT) to solve the problems mentioned in the background section.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a device information collection and management method based on the Internet of Things, comprising the following steps:

[0008] Dynamic protocol loading steps:

[0009] A hot-swappable protocol library module is built based on the OSGi framework. The new access protocol is automatically verified for compatibility through protocol feature fingerprint recognition. After verification, it is dynamically registered to the service registry center.

[0010] Parallel processing architecture:

[0011] It adopts a multi-process architecture combined with thread pool technology. Each protocol instance runs in an independent Docker container. The main process allocates concurrent requests through the thread pool to achieve dynamic load balancing, and the container is configured with resource isolation strategies.

[0012] Smart routing selection steps:

[0013] A dual-channel routing mechanism is constructed. The primary channel uses a weighted scoring model to select the optimal cloud platform channel, while the backup channel achieves asynchronous transmission through a message queue. Automatic switching occurs when the quality of the primary channel deteriorates.

[0014] Real-time data acquisition steps:

[0015] Deploy edge computing nodes to perform data preprocessing and use a sliding window mechanism to verify sensor data in real time;

[0016] Data parsing and formatting steps:

[0017] Based on device type, dynamically load and parse templates, perform pattern matching and verification on the collected data, and generate structured data packets;

[0018] Data upload and storage steps:

[0019] Differentiated transmission is achieved through priority queues. Data from critical devices is transmitted directly, while data from ordinary devices is transmitted after compression. Cloud platform storage uses a distributed time-series database.

[0020] Protocol sandbox isolation steps:

[0021] Each protocol instance is deployed in a separate Docker container, with resource limits and access control policies configured.

[0022] Fault detection and recovery steps:

[0023] Establish a hot-switching mechanism for primary and backup processing units. When the primary unit times out, an election protocol is triggered to complete the failover.

[0024] Data backup and recovery steps:

[0025] An improved incremental backup algorithm is adopted to record data change logs, and erasure coding technology of distributed storage system is combined to achieve data protection;

[0026] In the dynamic protocol loading step, the specific implementation of protocol feature fingerprint recognition is as follows: extract features from the data packets of the new access protocol, generate feature vectors, and compare them with the pre-stored protocol feature library. If the matching degree exceeds the preset threshold, it is determined to be a legal protocol and dynamically registered to the service registry center.

[0027] The dynamic protocol loading mechanism, through the OSGi framework and feature fingerprint recognition, achieves automatic protocol compatibility and hot-swapping, greatly improving the system's flexibility and scalability. The parallel processing architecture, combined with multi-process and thread pool technologies and the resource isolation provided by Docker containers, ensures efficient and stable concurrent processing capabilities. Intelligent routing selection, through a dual-channel mechanism, guarantees the reliability and efficiency of data transmission. The combination of real-time data acquisition and edge computing nodes enables rapid data preprocessing and verification. Dynamic parsing template loading and data formatting ensure data accuracy and consistency. Priority queues and differentiated transmission strategies optimize data upload and storage efficiency. Protocol sandbox isolation enhances system security. Fault detection and recovery mechanisms ensure high system availability. An improved incremental backup algorithm and distributed storage system achieve efficient data protection and recovery. Overall, this method and system significantly improve the efficiency, reliability, and security of IoT device information collection and management.

[0028] Preferably, the protocol feature fingerprint recognition generates a feature vector by extracting features from protocol data packets, compares it with a pre-stored protocol feature library, and determines the legitimate protocol.

[0029] Deep packet inspection technology is used to parse protocol data packets and extract statistical features such as packet length distribution, port number, and payload entropy. Multidimensional feature vectors are constructed by combining these features with protocol behavior sequences. The feature library is incrementally updated through a distributed storage architecture, and legitimate protocol samples are collected periodically for model training. A version control mechanism is also introduced. The comparison process uses Locality Sensitive Hashing (LSH) to accelerate feature vector matching and dynamically adjusts the comparison threshold using a machine learning model. When a comparison fails, a secondary verification process is automatically triggered, including manual review of the interface and backtracking analysis of protocol samples. Sensitive fields are encrypted using the national cryptographic SM4 algorithm, and access permissions to the feature library are restricted by ACL.

[0030] Protocol fingerprinting technology significantly improves the security and efficiency of IoT device access management through an automated feature comparison mechanism. This technology replaces traditional manual protocol analysis, shortening the access cycle for new devices and reducing operation and maintenance costs. The dynamic update mechanism of the feature library can promptly identify new protocol variants, enhancing the system's compatibility with unknown protocols and avoiding access failures due to protocol version differences. Combined with encryption algorithms and access control policies, it effectively prevents protocol forgery attacks and ensures data security during feature extraction and comparison. In addition, this technology works in conjunction with the Docker containerized architecture to support hot-swappable upgrades of the protocol parsing module, completing feature library updates without business interruption and ensuring continuous and stable system operation.

[0031] Preferably, the multi-process architecture adopts a master-slave mode, where the master process is responsible for protocol instance management and global load monitoring, the slave processes achieve high concurrency processing through a multiplexing mechanism, and the thread pool dynamically adjusts the number of worker threads according to the system load.

[0032] In the master-slave mode of the multi-process architecture, the master process manages the lifecycle of each protocol instance by building a protocol instance registry and collects system load information in real time. The slave processes use multiplexing (such as epoll or kqueue) technology to listen to multiple network connections to achieve high-concurrency data processing. The thread pool has a built-in load detection algorithm that dynamically adjusts the number of worker threads according to the current CPU and memory usage of the system to ensure efficient utilization of system resources.

[0033] The master-slave architecture with a multi-process structure significantly improves the stability and scalability of the system in IoT device information collection and management. The master process focuses on protocol instance management and global load monitoring, ensuring the rational allocation and efficient utilization of system resources. The slave processes achieve high-concurrency processing through multiplexing mechanisms, effectively improving data processing throughput and meeting the needs of large-scale device access. The thread pool dynamically adjusts the number of worker threads according to the system load, avoiding resource waste and ensuring system response speed under high load conditions. This design enables the system to flexibly cope with IoT device access of different scales, while ensuring the real-time performance and accuracy of data processing, providing solid technical support for IoT applications.

[0034] Preferably, the main channel scoring model of the dual-channel routing mechanism comprehensively considers network latency, cloud platform load, and data priority to dynamically select the optimal channel;

[0035] Build a real-time monitoring system to continuously collect network latency data, current cloud platform load indicators (such as CPU utilization and memory usage), and data packet priority tags (such as critical data and ordinary data). Through a weighted algorithm, assign reasonable weights to each indicator, calculate the scores comprehensively, and dynamically select the cloud platform channel with the highest score as the main channel to ensure efficient and stable data transmission.

[0036] The dual-channel routing mechanism's primary channel scoring model intelligently selects data transmission paths by comprehensively considering network latency, cloud platform load, and data priority. The advantage of this design is that it dynamically adjusts the data transmission path based on real-time network conditions and cloud platform resources, ensuring that critical data is prioritized for rapid transmission through the optimal channel while avoiding network congestion and cloud platform overload. The backup channel, using a message queue for asynchronous transmission, serves as an effective supplement to the primary channel and automatically switches when the primary channel's quality deteriorates, further enhancing the system's reliability and fault tolerance. This dual-channel design not only optimizes data transmission efficiency but also enhances system stability and adaptability, providing strong support for IoT device information collection and management.

[0037] Preferably, the edge computing node deploys a rule engine that supports hot updates of parsed templates. When data anomalies are detected, it triggers local alarms and records the abnormal events.

[0038] The edge computing node has a built-in rule engine module. This module listens for and parses template update notifications (such as those from the cloud or local configuration center), dynamically loads new templates and replaces old ones. At the same time, the rule engine has built-in data verification logic. Once it detects that the data exceeds the preset threshold or does not conform to the expected pattern, it triggers a local alarm mechanism (such as sending emails, SMS messages or system logs) and records in detail the timestamp of the abnormal event, data content, triggering rules and other information for subsequent analysis and processing.

[0039] It enhances the system's flexibility and scalability, enabling parsing rules to be quickly adjusted according to business needs without system downtime or restart. Secondly, through real-time data verification and local alarm mechanisms, the system can quickly respond to data anomalies, reduce potential risks, and improve the accuracy and reliability of data processing. Furthermore, detailed anomaly event logs provide valuable data support for subsequent troubleshooting and system optimization, helping to continuously improve system performance. In short, this design not only improves the system's intelligence level but also enhances its ability to cope with complex and ever-changing environments, providing a solid technical guarantee for the information collection and management of IoT devices.

[0040] Preferably, the distributed time-series database adopts a cluster architecture, with data sharded by device ID, each shard configured with multiple replicas, and read and write operations meeting consistency requirements;

[0041] In the distributed time-series database, a consistent hashing algorithm is used to shard data based on the device ID to ensure that the data is evenly distributed. Each shard is configured with multiple replicas in the cluster. Consensus algorithms such as Paxos or Raft are used to ensure data consistency between replicas. During read and write operations, distributed locks or optimistic locking mechanisms are used to ensure data consistency under concurrent operations. At the same time, a caching layer is introduced to improve read and write performance.

[0042] The cluster architecture of this distributed time-series database effectively distributes storage and query loads by sharding data by device ID, improving system scalability and processing capabilities. Each shard is configured with multiple replicas, which not only enhances data redundancy and fault tolerance, ensuring no data loss in the event of partial node failure, but also guarantees consistency of read and write operations through data synchronization mechanisms between replicas, improving data accuracy and reliability. This design enables the system to handle large-scale device access and high-concurrency read and write scenarios, providing stable and efficient data storage support for IoT device information collection and management, and helping to improve the overall system stability and user experience.

[0043] Preferably, the failover process maintains uninterrupted TCP connections through session migration technology;

[0044] When the main processing unit establishes a TCP connection with the client, the connection information (such as source IP, destination IP, port number, sequence number, etc.) is synchronized to the backup processing unit. When the main unit fails, the backup unit immediately takes over the connection and continues to use the synchronized connection information to communicate with the client, ensuring the continuity of the TCP session and thus avoiding connection interruption.

[0045] It significantly improves the reliability and availability of the system, ensuring that the backup unit can seamlessly take over when the main processing unit fails, avoiding service interruption and thus guaranteeing the continuity and stability of equipment information collection. Secondly, this technology reduces data loss and retransmission overhead caused by connection interruption, improving the efficiency and accuracy of data transmission. In addition, session migration technology enhances the user experience because users do not need to perceive the service interruption or re-establishment process, thereby increasing user trust and satisfaction with the system. In summary, this patent, through session migration technology, optimizes data transmission efficiency and user experience while ensuring stable system operation.

[0046] Preferably, the improved incremental backup algorithm uses block-level difference tracking, generating a new version only when the amount of data changes exceeds a threshold;

[0047] The improved incremental backup algorithm sets up a change tracking mechanism at the data block level, performs hash calculations on data blocks and stores their initial state. In subsequent backup processes, the algorithm performs hash calculations on each data block again and compares it with the initial state. If the difference exceeds a preset threshold, it is determined that the data block has changed, and a new version backup containing the changed data is generated. This method ensures that only the data that has actually changed is backed up, reducing storage space usage.

[0048] This improved incremental backup algorithm employs block-level difference tracking technology, significantly enhancing backup efficiency and resource utilization. By generating a new version only when the amount of data changes exceeds a threshold, it avoids unnecessary full backups, effectively reducing storage space usage and backup time. Simultaneously, the algorithm ensures data integrity and recoverability; even if some data blocks are corrupted, recovery can be achieved through historical versions. Furthermore, by combining erasure coding technology from a distributed storage system, it further enhances data reliability and fault tolerance, enabling the system to maintain data availability and consistency even in the face of node failures. Overall, this algorithm provides an efficient and reliable data backup solution for IoT device information collection and management systems.

[0049] Preferably, the distributed storage system adopts an erasure coding scheme, which allows storage node failures without data loss, and the data reconstruction speed meets performance requirements;

[0050] Distributed storage systems divide data into multiple data blocks and generate redundant check blocks using erasure coding algorithms. These check blocks are stored on different nodes along with the data blocks. When a storage node fails, the system can use the remaining data blocks and check blocks to recover the lost data using erasure coding decoding algorithms, ensuring that no data is lost. The data reconstruction process is efficient and meets performance requirements.

[0051] Distributed storage systems employing erasure coding offer significant advantages. First, they greatly improve data reliability and availability. Even if some storage nodes fail, data can be quickly recovered using erasure coding algorithms, preventing data loss. Second, this scheme optimizes storage resource utilization by generating redundant check blocks instead of complete data copies, reducing storage space usage. Furthermore, the data reconstruction speed meets performance requirements, ensuring stable system operation in high-failure-rate environments. Finally, this scheme enhances system scalability and flexibility, enabling the storage system to easily handle node additions / removals and data migrations, providing a solid data storage foundation for IoT device information collection and management.

[0052] The IoT-based device information collection and management system, employing the aforementioned IoT-based device information collection and management method, includes:

[0053] Protocol management module: Enables hot-plugging of protocols and feature fingerprint recognition;

[0054] Data processing module: includes a parallel processing engine and intelligent routing components;

[0055] Security protection module: integrates Docker sandbox and blockchain evidence storage sub-modules;

[0056] Operations and maintenance management module: provides a visual monitoring interface and automated fault handling functions;

[0057] Extract data packet features of the new access protocol, such as packet header structure and specific field values, generate feature vectors, compare them with the pre-stored protocol feature library, determine whether it is a legal protocol through similarity calculation, and dynamically register it to the service registry center after verification;

[0058] Through the collaborative work of four core modules, the efficiency and security of IoT device information collection and management are significantly improved. The protocol management module supports hot-swappable protocols and feature fingerprint recognition, enabling the system to flexibly adapt to constantly changing protocol environments and reducing maintenance costs. The data processing module adopts a parallel processing engine and intelligent routing components to achieve efficient data processing and transmission, ensuring data real-time performance and accuracy. The security protection module integrates a Docker sandbox and blockchain evidence storage sub-module, enhancing system security and data immutability, and effectively resisting external attacks. The operation and maintenance management module provides a visual monitoring interface and automated fault handling functions, simplifying operation and maintenance processes, improving fault response speed, and ensuring stable system operation. These modules together construct an efficient, secure, and easy-to-maintain IoT device information collection and management system.

[0059] In summary, compared with the prior art, the present invention provides a device information collection and management method and system based on the Internet of Things, which has the following beneficial effects:

[0060] This invention achieves hot-plugging of protocols and feature fingerprint recognition through a dynamic protocol loading and protocol management module. The method and system possess extremely strong protocol compatibility. Faced with the complex and frequently updated protocols of IoT devices, it can quickly and automatically adapt to new protocols and register them without large-scale system modifications, reducing development and maintenance costs, improving the system's adaptability to different devices, and ensuring business continuity.

[0061] Employing a parallel processing architecture, intelligent routing selection, and data processing modules, data processing and transmission are highly efficient and reliable. Multi-process combined with thread pool technology, independent container operation, and dynamic load balancing improves data processing speed; a dual-channel routing mechanism ensures the stability and reliability of data transmission; dynamically loading and parsing templates to generate structured data packets enhances the accuracy and standardization of data processing, enabling the system to quickly respond to the data acquisition needs of massive devices.

[0062] By leveraging protocol sandbox isolation, fault detection and recovery, data backup and recovery modules, as well as security protection and operation and maintenance management modules, the system's security and maintainability are significantly improved. Independent container isolation, resource limits, and access control policies reduce security risks; the primary / backup hot-switch mechanism ensures high system availability; improved backup algorithms and erasure coding technology prevent data loss; and the security protection module and visual operation and maintenance interface make system management safer and more convenient, reducing manual intervention and improving operation and maintenance efficiency. Attached Figure Description

[0063] Figure 1 This is a flowchart illustrating the steps of the IoT-based device information collection and management method of this invention.

[0064] Figure 2 This is a schematic diagram of the IoT-based device information collection and management system of this invention. Detailed Implementation

[0065] This invention provides a technical solution: a device information collection and management method based on the Internet of Things (IoT). Please refer to [link / reference]. Figure 1 It includes the following steps:

[0066] Dynamic protocol loading steps:

[0067] A hot-swappable protocol library module is built based on the OSGi framework. The new access protocol is automatically verified for compatibility through protocol feature fingerprint recognition. After verification, it is dynamically registered to the service registry center.

[0068] Parallel processing architecture:

[0069] It adopts a multi-process architecture combined with thread pool technology. Each protocol instance runs in an independent Docker container. The main process allocates concurrent requests through the thread pool to achieve dynamic load balancing, and the container is configured with resource isolation strategies.

[0070] Smart routing selection steps:

[0071] A dual-channel routing mechanism is constructed. The primary channel uses a weighted scoring model to select the optimal cloud platform channel, while the backup channel achieves asynchronous transmission through a message queue. Automatic switching occurs when the quality of the primary channel deteriorates.

[0072] Quantitative standards for main channel quality degradation:

[0073] Network quality quantitative indicator system

[0074] Core detection indicators

[0075] Latency threshold: If a one-way latency >200ms (industrial scenario) or >500ms (consumer scenario) is detected for 5 consecutive sampling cycles (e.g., 100ms / cycle), a quality warning is triggered.

[0076] Packet loss rate threshold: The packet loss rate is >8% (critical data) or >15% (normal data) when the sliding window (window size = 100 data packets) is used, and the duration is >10 seconds.

[0077] Bandwidth utilization: The available bandwidth of the channel is less than 30% of the peak bandwidth and this continues for more than 3 detection cycles (e.g., 30 seconds).

[0078] Jitter amplitude: The latency jitter exceeds the average of ±50ms (industrial scenario) or ±100ms (consumer scenario), causing data frame timing disorder.

[0079] Cloud platform load metrics

[0080] A decrease in processing capacity is considered to occur when the target cloud platform's CPU utilization is greater than 85% and memory usage is greater than 90% for a duration of greater than 30 seconds.

[0081] Real-time data acquisition steps:

[0082] Deploy edge computing nodes to perform data preprocessing and use a sliding window mechanism to verify sensor data in real time;

[0083] Data parsing and formatting steps:

[0084] Based on device type, dynamically load and parse templates, perform pattern matching and verification on the collected data, and generate structured data packets;

[0085] Data upload and storage steps:

[0086] Differentiated transmission is achieved through priority queues. Data from critical devices is transmitted directly, while data from ordinary devices is transmitted after compression. Cloud platform storage uses a distributed time-series database.

[0087] Protocol sandbox isolation steps:

[0088] Each protocol instance is deployed in a separate Docker container, with resource limits and access control policies configured.

[0089] Fault detection and recovery steps:

[0090] Establish a hot-switching mechanism for primary and backup processing units. When the primary unit times out, an election protocol is triggered to complete the failover.

[0091] Data backup and recovery steps:

[0092] An improved incremental backup algorithm is adopted to record data change logs, and erasure coding technology of distributed storage system is combined to achieve data protection;

[0093] In the dynamic protocol loading step, the specific implementation of protocol feature fingerprint recognition is as follows: extract features from the data packets of the new access protocol, generate feature vectors, and compare them with the pre-stored protocol feature library. If the matching degree exceeds the preset threshold, it is determined to be a legal protocol and dynamically registered to the service registry center.

[0094] The dynamic protocol loading mechanism, through the OSGi framework and feature fingerprint recognition, achieves automatic protocol compatibility and hot-swapping, greatly improving the system's flexibility and scalability. The parallel processing architecture, combined with multi-process and thread pool technologies and the resource isolation provided by Docker containers, ensures efficient and stable concurrent processing capabilities. Intelligent routing selection, through a dual-channel mechanism, guarantees the reliability and efficiency of data transmission. The combination of real-time data acquisition and edge computing nodes enables rapid data preprocessing and verification. Dynamic parsing template loading and data formatting ensure data accuracy and consistency. Priority queues and differentiated transmission strategies optimize data upload and storage efficiency. Protocol sandbox isolation enhances system security. Fault detection and recovery mechanisms ensure high system availability. An improved incremental backup algorithm and distributed storage system achieve efficient data protection and recovery. Overall, this method and system significantly improve the efficiency, reliability, and security of IoT device information collection and management.

[0095] Please see Figure 1 Protocol feature fingerprinting extracts features from protocol data packets to generate feature vectors, which are then compared with a pre-stored protocol feature library to determine the legitimacy of the protocol.

[0096] Deep packet inspection technology is used to parse protocol data packets and extract statistical features such as packet length distribution, port number, and payload entropy. Multidimensional feature vectors are constructed by combining these features with protocol behavior sequences. The feature library is incrementally updated through a distributed storage architecture, and legitimate protocol samples are collected periodically for model training. A version control mechanism is also introduced. The comparison process uses Locality Sensitive Hashing (LSH) to accelerate feature vector matching and dynamically adjusts the comparison threshold using a machine learning model. When a comparison fails, a secondary verification process is automatically triggered, including manual review of the interface and backtracking analysis of protocol samples. Sensitive fields are encrypted using the national cryptographic SM4 algorithm, and access permissions to the feature library are restricted by ACL.

[0097] Incremental update triggering conditions and process of feature library:

[0098] I. Triggering Conditions

[0099] Scheduled trigger: Automatically scans the protocol sample library at preset intervals (e.g., 24 hours);

[0100] New protocol trigger: Automatically initiates updates when an unknown protocol is accessed and matching fails;

[0101] Anomaly trigger: The percentage of consecutive N unknown protocol requests or abnormal packets exceeds a threshold;

[0102] Manual trigger: Manually force an update or upload a custom sample through the operations and maintenance interface.

[0103] II. Update Process

[0104] Sample collection and validation

[0105] Automatically capture protocol data packets, manually review illegal samples, and encrypt and store them in a temporary pool;

[0106] Feature extraction and training

[0107] The sample is parsed to generate feature vectors, and the model is incrementally trained through a distributed framework to generate a new version;

[0108] Gray release and synchronization

[0109] First, deploy 10% of the nodes for verification, then perform incremental synchronization across the entire cluster using the Paxos algorithm, supporting version rollback;

[0110] Validation

[0111] Monitor matching success rate and version consistency, and issue alerts and initiate closed-loop optimization when anomalies occur.

[0112] Protocol fingerprinting technology significantly improves the security and efficiency of IoT device access management through an automated feature comparison mechanism. This technology replaces traditional manual protocol analysis, shortening the access cycle for new devices and reducing operation and maintenance costs. The dynamic update mechanism of the feature library can promptly identify new protocol variants, enhancing the system's compatibility with unknown protocols and avoiding access failures due to protocol version differences. Combined with encryption algorithms and access control policies, it effectively prevents protocol forgery attacks and ensures data security during feature extraction and comparison. In addition, this technology works in conjunction with the Docker containerized architecture to support hot-swappable upgrades of the protocol parsing module, completing feature library updates without business interruption and ensuring continuous and stable system operation.

[0113] Please see Figure 1 The multi-process architecture adopts a master-slave mode. The master process is responsible for protocol instance management and global load monitoring, while the slave processes achieve high concurrency processing through multiplexing mechanisms. The thread pool dynamically adjusts the number of worker threads according to the system load.

[0114] In the master-slave mode of the multi-process architecture, the master process manages the lifecycle of each protocol instance by building a protocol instance registry and collects system load information in real time. The slave processes use multiplexing (such as epoll or kqueue) technology to listen to multiple network connections to achieve high-concurrency data processing. The thread pool has a built-in load detection algorithm that dynamically adjusts the number of worker threads according to the current CPU and memory usage of the system to ensure efficient utilization of system resources.

[0115] The master-slave architecture with a multi-process structure significantly improves the stability and scalability of the system in IoT device information collection and management. The master process focuses on protocol instance management and global load monitoring, ensuring the rational allocation and efficient utilization of system resources. The slave processes achieve high-concurrency processing through multiplexing mechanisms, effectively improving data processing throughput and meeting the needs of large-scale device access. The thread pool dynamically adjusts the number of worker threads according to the system load, avoiding resource waste and ensuring system response speed under high load conditions. This design enables the system to flexibly cope with IoT device access of different scales, while ensuring the real-time performance and accuracy of data processing, providing solid technical support for IoT applications.

[0116] Please see Figure 1 The dual-channel routing mechanism's main channel scoring model comprehensively considers network latency, cloud platform load, and data priority to dynamically select the optimal channel.

[0117] Build a real-time monitoring system to continuously collect network latency data, current cloud platform load indicators (such as CPU utilization and memory usage), and data packet priority tags (such as critical data and ordinary data). Through a weighted algorithm, assign reasonable weights to each indicator, calculate the scores comprehensively, and dynamically select the cloud platform channel with the highest score as the main channel to ensure efficient and stable data transmission.

[0118] The dual-channel routing mechanism's primary channel scoring model intelligently selects data transmission paths by comprehensively considering network latency, cloud platform load, and data priority. The advantage of this design is that it dynamically adjusts the data transmission path based on real-time network conditions and cloud platform resources, ensuring that critical data is prioritized for rapid transmission through the optimal channel while avoiding network congestion and cloud platform overload. The backup channel, using a message queue for asynchronous transmission, serves as an effective supplement to the primary channel and automatically switches when the primary channel's quality deteriorates, further enhancing the system's reliability and fault tolerance. This dual-channel design not only optimizes data transmission efficiency but also enhances system stability and adaptability, providing strong support for IoT device information collection and management.

[0119] Please see Figure 1 The edge computing node deploys a rule engine that supports hot updates of parsed templates. When data anomalies are detected, it triggers local alarms and records the abnormal events.

[0120] Sub-steps for lightweight design of the rules engine:

[0121] (1) Execution engine selection:

[0122] It adopts an embedded engine based on Lua scripts and executes rules through pre-compiled bytecode, reducing interpreter overhead;

[0123] The rule logic is restricted to a two-part condition-action structure, loop statements are prohibited, and only a limited number of condition jumps are supported.

[0124] (2) Rule-based hierarchical strategy:

[0125] Basic rule layer: solidifies commonly used rules (such as data range validation and format conversion), implemented in C language and compiled into the kernel;

[0126] Extended rule layer: Handles complex logic (such as multi-sensor linkage judgment) through hot-uploaded Lua scripts.

[0127] (3) Preprocessing optimization:

[0128] Static analysis is performed during the rule compilation phase to eliminate redundant conditions and generate a decision tree structure.

[0129] Input data is pre-labeled with feature tags (such as device type and data priority), and a subset of rules is directly matched.

[0130] (4) Resource isolation mechanism:

[0131] Each rule instance runs in an independent coroutine, with a CPU time slice limit configured (e.g., no more than 50ms per execution).

[0132] High-priority rules (such as security alerts) are pre-allocated to a dedicated thread pool to avoid competing for resources with ordinary rules;

[0133] The edge computing node has a built-in rule engine module. This module listens for and parses template update notifications (such as those from the cloud or local configuration center), dynamically loads new templates and replaces old ones. At the same time, the rule engine has built-in data verification logic. Once it detects that the data exceeds the preset threshold or does not conform to the expected pattern, it triggers a local alarm mechanism (such as sending emails, SMS messages or system logs) and records in detail the timestamp of the abnormal event, data content, triggering rules and other information for subsequent analysis and processing.

[0134] It enhances the system's flexibility and scalability, enabling parsing rules to be quickly adjusted according to business needs without system downtime or restart. Secondly, through real-time data verification and local alarm mechanisms, the system can quickly respond to data anomalies, reduce potential risks, and improve the accuracy and reliability of data processing. Furthermore, detailed anomaly event logs provide valuable data support for subsequent troubleshooting and system optimization, helping to continuously improve system performance. In short, this design not only improves the system's intelligence level but also enhances its ability to cope with complex and ever-changing environments, providing a solid technical guarantee for the information collection and management of IoT devices.

[0135] Please see Figure 1The distributed time-series database adopts a cluster architecture, with data sharded by device ID, and each shard configured with multiple replicas, ensuring consistency for read and write operations.

[0136] In the distributed time-series database, a consistent hashing algorithm is used to shard data based on the device ID to ensure that the data is evenly distributed. Each shard is configured with multiple replicas in the cluster. Consensus algorithms such as Paxos or Raft are used to ensure data consistency between replicas. During read and write operations, distributed locks or optimistic locking mechanisms are used to ensure data consistency under concurrent operations. At the same time, a caching layer is introduced to improve read and write performance.

[0137] The cluster architecture of this distributed time-series database effectively distributes storage and query loads by sharding data by device ID, improving system scalability and processing capabilities. Each shard is configured with multiple replicas, which not only enhances data redundancy and fault tolerance, ensuring no data loss in the event of partial node failure, but also guarantees consistency of read and write operations through data synchronization mechanisms between replicas, improving data accuracy and reliability. This design enables the system to handle large-scale device access and high-concurrency read and write scenarios, providing stable and efficient data storage support for IoT device information collection and management, and helping to improve the overall system stability and user experience.

[0138] Please see Figure 1 The failover process maintains uninterrupted TCP connections through session migration technology;

[0139] When the main processing unit establishes a TCP connection with the client, the connection information (such as source IP, destination IP, port number, sequence number, etc.) is synchronized to the backup processing unit. When the main unit fails, the backup unit immediately takes over the connection and continues to use the synchronized connection information to communicate with the client, ensuring the continuity of the TCP session and thus avoiding connection interruption.

[0140] It significantly improves the reliability and availability of the system, ensuring that the backup unit can seamlessly take over when the main processing unit fails, avoiding service interruption and thus guaranteeing the continuity and stability of equipment information collection. Secondly, this technology reduces data loss and retransmission overhead caused by connection interruption, improving the efficiency and accuracy of data transmission. In addition, session migration technology enhances the user experience because users do not need to perceive the service interruption or re-establishment process, thereby increasing user trust and satisfaction with the system. In summary, this patent, through session migration technology, optimizes data transmission efficiency and user experience while ensuring stable system operation.

[0141] Please see Figure 1 The improved incremental backup algorithm uses block-level difference tracking, generating a new version only when the amount of data changes exceeds a threshold;

[0142] The improved incremental backup algorithm sets up a change tracking mechanism at the data block level, performs hash calculations on data blocks and stores their initial state. In subsequent backup processes, the algorithm performs hash calculations on each data block again and compares it with the initial state. If the difference exceeds a preset threshold, it is determined that the data block has changed, and a new version backup containing the changed data is generated. This method ensures that only the data that has actually changed is backed up, reducing storage space usage.

[0143] This improved incremental backup algorithm employs block-level difference tracking technology, significantly enhancing backup efficiency and resource utilization. By generating a new version only when the amount of data changes exceeds a threshold, it avoids unnecessary full backups, effectively reducing storage space usage and backup time. Simultaneously, the algorithm ensures data integrity and recoverability; even if some data blocks are corrupted, recovery can be achieved through historical versions. Furthermore, by combining erasure coding technology from a distributed storage system, it further enhances data reliability and fault tolerance, enabling the system to maintain data availability and consistency even in the face of node failures. Overall, this algorithm provides an efficient and reliable data backup solution for IoT device information collection and management systems.

[0144] Please see Figure 1 The distributed storage system uses erasure coding, which allows storage node failures without data loss, and the data reconstruction speed meets performance requirements.

[0145] Distributed storage systems divide data into multiple data blocks and generate redundant check blocks using erasure coding algorithms. These check blocks are stored on different nodes along with the data blocks. When a storage node fails, the system can use the remaining data blocks and check blocks to recover the lost data using erasure coding decoding algorithms, ensuring that no data is lost. The data reconstruction process is efficient and meets performance requirements.

[0146] Distributed storage systems employing erasure coding offer significant advantages. First, they greatly improve data reliability and availability. Even if some storage nodes fail, data can be quickly recovered using erasure coding algorithms, preventing data loss. Second, this scheme optimizes storage resource utilization by generating redundant check blocks instead of complete data copies, reducing storage space usage. Furthermore, the data reconstruction speed meets performance requirements, ensuring stable system operation in high-failure-rate environments. Finally, this scheme enhances system scalability and flexibility, enabling the storage system to easily handle node additions / removals and data migrations, providing a solid data storage foundation for IoT device information collection and management.

[0147] Key performance indicators of this solution:

[0148] I. Dynamic Protocol Loading

[0149] Time consumption: Single protocol hot loading < 80ms (including feature recognition + registration), 100 protocol batch loading < 500ms.

[0150] Technology: OSGi modularization + LSH hash comparison + etcd registration.

[0151] II. Distributed Databases

[0152] Read / write throughput: 100,000 TPS for writes / 150,000 TPS for reads (99% latency < 50ms).

[0153] Technology: Consistent hashing sharding + Raft consensus + LSM tree storage.

[0154] III. Incremental Backup Efficiency

[0155] Space saving: 85% less space compared to full backup (for scenarios with a daily change rate of <5%).

[0156] Technology: 16KB block-level difference tracking + 12+4 erasure coding.

[0157] IV. Fault Switching Delay

[0158] Service recovery: TCP connection migration < 50ms, RTO < 200ms.

[0159] Technology: Session pre-synchronization + Paxos election optimization.

[0160] V. Data Processing Capabilities

[0161] Single-node concurrency: Edge nodes support 5000 devices / second (2KB / device).

[0162] Technology: Multi-process master-slave + epoll multiplexing.

[0163] The system for collecting and managing device information based on the Internet of Things (IoT) employs the aforementioned IoT-based device information collection and management method. Please refer to [link / reference needed]. Figure 1 and Figure 2 ,include:

[0164] Protocol management module: Enables hot-plugging of protocols and feature fingerprint recognition;

[0165] Data processing module: includes a parallel processing engine and intelligent routing components;

[0166] Security protection module: integrates Docker sandbox and blockchain evidence storage sub-modules;

[0167] Operations and maintenance management module: provides a visual monitoring interface and automated fault handling functions;

[0168] Extract data packet features of the new access protocol, such as packet header structure and specific field values, generate feature vectors, compare them with the pre-stored protocol feature library, determine whether it is a legal protocol through similarity calculation, and dynamically register it to the service registry center after verification;

[0169] Through the collaborative work of four core modules, the efficiency and security of IoT device information collection and management are significantly improved. The protocol management module supports hot-swappable protocols and feature fingerprint recognition, enabling the system to flexibly adapt to constantly changing protocol environments and reducing maintenance costs. The data processing module adopts a parallel processing engine and intelligent routing components to achieve efficient data processing and transmission, ensuring data real-time performance and accuracy. The security protection module integrates a Docker sandbox and blockchain evidence storage sub-module, enhancing system security and data immutability, and effectively resisting external attacks. The operation and maintenance management module provides a visual monitoring interface and automated fault handling functions, simplifying operation and maintenance processes, improving fault response speed, and ensuring stable system operation. These modules together construct an efficient, secure, and easy-to-maintain IoT device information collection and management system.

[0170] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0171] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A device information collection management method based on the Internet of Things, characterized by, Includes the following steps: Dynamic protocol loading steps: A hot-pluggable protocol library module is built based on the OSGi framework. The new access protocol is automatically verified for compatibility through protocol feature fingerprint recognition. After the verification is successful, it is dynamically registered to the service registry center. Parallel processing architecture: It adopts a multi-process architecture combined with thread pool technology. Each protocol instance runs in an independent Docker container. The main process allocates concurrent requests through the thread pool to achieve dynamic load balancing, and the container is configured with resource isolation strategies. Smart routing selection steps: A dual-channel routing mechanism is constructed. The primary channel uses a weighted scoring model to select the optimal cloud platform channel, while the backup channel achieves asynchronous transmission through a message queue. Automatic switching occurs when the quality of the primary channel deteriorates. Real-time data acquisition steps: Deploy edge computing nodes to perform data preprocessing and use a sliding window mechanism to verify sensor data in real time; Data parsing and formatting steps: Based on device type, dynamically load and parse templates, perform pattern matching and verification on the collected data, and generate structured data packets; Data upload and storage steps: Differentiated transmission is achieved through priority queues. Data from critical devices is transmitted directly, while data from ordinary devices is transmitted after compression. Cloud platform storage uses a distributed time-series database. Protocol sandbox isolation steps: Each protocol instance is deployed in a separate Docker container, with resource limits and access control policies configured. Fault detection and recovery steps: Establish a hot-switching mechanism for primary and backup processing units. When the primary unit times out, an election protocol is triggered to complete the failover. Data backup and recovery steps: An improved incremental backup algorithm is used to record data change logs, and erasure coding technology of a distributed storage system is combined to achieve data protection. 2.The Internet of Things based device information collection management method according to claim 1, characterized in that: The protocol feature fingerprint recognition generates a feature vector by extracting features from protocol data packets, and compares it with a pre-stored protocol feature library to determine the legitimate protocol.

3. The device information collection and management method based on the Internet of Things according to claim 1, characterized in that: The multi-process architecture adopts a master-slave mode. The master process is responsible for protocol instance management and global load monitoring, while the slave processes achieve high concurrency processing through a multiplexing mechanism. The thread pool dynamically adjusts the number of worker threads according to the system load.

4. The device information collection and management method based on the Internet of Things according to claim 1, characterized in that: The dual-channel routing mechanism's main channel scoring model comprehensively considers network latency, cloud platform load, and data priority to dynamically select the optimal channel.

5. The device information collection and management method based on the Internet of Things according to claim 1, characterized in that: The edge computing node deploys a rule engine that supports hot updates of parsed templates. When data anomalies are detected, it triggers local alarms and records the abnormal events.

6. The device information collection and management method based on the Internet of Things according to claim 1, characterized in that: The distributed time-series database adopts a cluster architecture, with data sharded by device ID. Each shard is configured with multiple replicas, and read and write operations meet consistency requirements.

7. The device information collection and management method based on the Internet of Things according to claim 1, characterized in that: The failover process maintains uninterrupted TCP connections through session migration technology.

8. The device information collection and management method based on the Internet of Things according to claim 1, characterized in that: The improved incremental backup algorithm uses block-level difference tracking and generates a new version only when the amount of data changes exceeds a threshold.

9. The device information collection and management method based on the Internet of Things according to claim 1, characterized in that: The distributed storage system employs erasure coding, which allows storage node failures to prevent data loss, and the data reconstruction speed meets performance requirements.

10. A system for collecting and managing device information based on the Internet of Things (IoT), employing the device information collection and management method based on the IoT as described in any one of claims 1-9, characterized in that, include: Protocol management module: Enables hot-plugging of protocols and feature fingerprint recognition; Data processing module: includes a parallel processing engine and intelligent routing components; Security protection module: integrates Docker sandbox and blockchain evidence storage sub-modules; Operation and maintenance management module: Provides a visual monitoring interface and automated fault handling functions.

Images