Internet of Things data integration method and system based on dynamic semantic atlas and edge collaboration

Abstract

The invention discloses an Internet of Things data integration method and system based on dynamic semantic atlas and edge collaboration, and the method comprises the steps: carrying out the equipment access and protocol conversion, capturing the original data of equipment through an edge node, analyzing the average communication interval of 100ms + / -20ms through an LSTM model, generating an equipment fingerprint hash value, analyzing a message structure through a TinyBERT model, and extracting a key field, a temperature value and a timestamp. The method comprises the following steps: establishing a dynamic semantic graph, converting the dynamic semantic graph into a standard JSON format, uploading the standard JSON format to a cloud end through a service grid, constructing a dynamic semantic graph, collecting multi-source metadata by the cloud end, generating an entity relationship by utilizing a Transform model, dynamically optimizing a mapping rule based on reinforcement learning, automatically associating a Pressures field to a database column PSI, and carrying out collaborative optimization by an edge cloud; according to the method, the knowledge graph and the Transform model are combined, the data entity relation is updated in real time, the semantic drift problem of a traditional static graph is solved, the cross-domain data association accuracy is improved by 35%, the unique device fingerprint is generated through the message frequency and load distribution, and dynamic tracking of the mobile device is supported.

Description

Technical Field

[0001] The present invention relates to the field of Internet of Things communication technology, and in particular to an Internet of Things data integration method and system based on dynamic semantic graph and edge collaboration. Background Art

[0002] Identify the terminal devices connected to the Internet of Things power platform, parse the communication protocol through the protocol parser, and obtain the protocol features; build a matching knowledge graph of multi-channel adaptive protocols, use the protocol features and the node protocol features in the graph to perform feature identification and matching, and determine the common and different features; take the top-level target of the graph as the benchmark, perform conversion cost minimization evaluation, determine the protocol conversion path, adapt and convert the protocol features, and configure adaptive matching channels. This solves the problem of diversified communication protocols and data specifications in the power Internet of Things, which makes data sharing and system operation and maintenance complicated, and causes the adaptability and efficiency of communication collaboration. It enables equipment to be seamlessly connected on the Internet of Things power platform, improves the compatibility and scalability of the platform, and ensures efficient, accurate and stable operation of transmission.

[0003] In the Chinese patent CN115048441A, with a publication date of September 13, 2022, traditional IoT device protocols and data formats differ significantly. Traditional ETL tools rely on manual rule writing, have low integration efficiency, insufficient dynamic scalability, and require manual configuration of new devices. Zero-contact registration and adaptive protocol conversion cannot be achieved. Secondly, the edge cloud collaboration defect, the existing centralized cloud processing is difficult to meet the low latency requirements, and lacks a dynamic policy delivery mechanism. There are hidden dangers in security and privacy, data privacy protection, and device identity authentication in a distributed environment. Therefore, the present invention proposes an IoT data integration method and system based on dynamic semantic graphs and edge collaboration to solve the problems existing in the prior art. Summary of the invention

[0004] In view of the above problems, the purpose of the present invention is to propose an Internet of Things data integration method and system based on dynamic semantic graph and edge collaboration. The Internet of Things data integration method and system based on dynamic semantic graph and edge collaboration combine the knowledge graph and the Transformer model, update the data entity relationship in real time, solve the semantic drift problem of traditional static graphs, and improve the accuracy of cross-domain data association by 35%. It generates a unique device fingerprint through message frequency and load distribution, and supports dynamic tracking of mobile devices.

[0005] To achieve the purpose of the present invention, the present invention is implemented by the following technical scheme: an Internet of Things data integration method based on dynamic semantic graph and edge collaboration, comprising the following steps: Step 1: Device access and protocol conversion. The edge node captures the device raw data and uses the LSTM model to analyze the communication interval (average 100ms±20ms) to generate the device fingerprint hash value.

[0006] Step 2: The TinyBERT model parses the message structure, extracts key fields, temperature values, and timestamps, converts them into standard JSON format, and uploads them to the cloud through the service grid.

[0007] Step 3: Dynamic semantic graph construction, collecting multi-source metadata in the cloud, and generating entity relationships using the Transformer model.

[0008] Step 4: Based on the reinforcement learning dynamic optimization mapping rules, the Pressure field is automatically associated with the database column PSI.

[0009] Step 5: Edge-cloud collaborative optimization, edge nodes cache high-frequency data, temperature readings within 30 seconds, perform local aggregation, and reduce cloud load.

[0010] Step 6: The cloud sends the federated learning model parameters, the edge nodes jointly train the anomaly detection model, and the results are encrypted and synchronized to the blockchain.

[0011] A further improvement is that in the step 1, the communication interval is analyzed and the LSTM model generates a device fingerprint. The device fingerprint is a unique identifier based on the behavioral characteristics of the device to identify and distinguish different devices.

[0012] Further improvement is that in step 2, the TinyBERT model parses the message sent by the IoT device, extracts key fields from the message using the TinyBERT model, converts the extracted key fields into a standard JSON format, and finally uploads the JSON data converted by the service grid to the cloud.

[0013] A further improvement is that in step three, the cloud platform collects metadata from data sources, including databases, APIs, file systems, and log files, and the relationship information output by the Transformer model is used to construct a dynamic semantic graph.

[0014] A further improvement is that in step 4, the Pressure field is mapped to the PSI column, and the reward function distributes the reward according to the quality of the mapping rule selected by the agent.

[0015] A further improvement is that in step five, the edge node can aggregate data and extract useful information, calculate the average temperature over a period of time, and then send these aggregated results to the cloud, thereby reducing the amount of data transmission and lowering network latency.

[0016] A further improvement is that in step six, federated learning allows model training without sharing original data, thereby protecting the privacy of user data.

[0017] Further improvements are: the system includes an IoT device layer, an edge computing layer, a service grid layer, a cloud computing layer, an application layer, and a security and expansion layer; The IoT device layer is responsible for collecting data from the environment through sensors and actuators; Through the edge computing layer, the raw data collected from sensors and devices is cleaned, filtered and converted through edge nodes to reduce the transmission of invalid data; The service mesh layer parses heterogeneous protocols based on the lightweight BERT model and dynamically generates conversion rules; The cloud computing layer uses the Transformer model to build semantic associations of multi-source data and supports cross-domain reasoning. The edge nodes of the application layer perform local preprocessing for cloud management. Security and extension layer, for quantum encrypted communication and defense against quantum computing attacks.

[0018] The beneficial effects of the present invention are as follows: the present invention combines the knowledge graph with the Transformer model, updates the data entity relationship in real time, solves the semantic drift problem of the traditional static graph, improves the accuracy of cross-domain data association by 35%, generates a unique device fingerprint through message frequency and load distribution, supports dynamic tracking of mobile devices, uses TinyBERT protocol parsing, embeds lightweight models in edge nodes, realizes real-time conversion of non-standard protocols and private industrial protocols, reduces the manual workload of protocol adaptation by 70%, and reduces the delay to less than 50ms. Based on the reinforcement learning algorithm, computing tasks are distributed according to network bandwidth and node load, including key operations, device registration and data access, which are recorded in Hyperledger Fabric to ensure that data cannot be tampered with. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0020] Figure 1 This is a diagram showing the technical principle architecture of the present invention. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms installation, connection and connection should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0023] In file 202210493846.8, data received in different forms are managed and accommodated through the data source, the received data is detected through the data detection module, the data acquisition module receives and manages the detected data, the data processing module further optimizes the collected data, and the data application module forms usage units with different functions for the processed and summarized data, which can perform report extraction and information query on the data. When detecting data, the data detection module can send an alarm to the user terminal through the alarm unit, and at the same time, perform security protection on the network in the data source through the firewall unit. The existing centralized cloud processing solution is difficult to meet the low latency requirements, and lacks a dynamic policy delivery mechanism, and there are security and privacy risks.

[0024] according to Figure 1 As shown, this embodiment provides an IoT data integration method based on dynamic semantic graph and edge collaboration, including the following steps: Step 1: Device access and protocol conversion. The edge node captures the original data of the device and analyzes the communication interval of 100ms±20ms on average through the LSTM model to generate a device fingerprint hash value. By analyzing the communication interval, the LSTM model generates a device fingerprint. The device fingerprint is a unique identifier based on the device behavior characteristics to identify and distinguish different devices. In the process of generating the fingerprint, the device fingerprint is converted into a hash value for fast comparison and retrieval of data. Step 2: The TinyBERT model parses the message structure, extracts key fields, temperature value and timestamp, converts them into standard JSON format, and uploads them to the cloud through the service grid. The TinyBERT model is used to parse the message sent by the IoT device, and the key fields are extracted from the message using the TinyBERT model. The extracted key fields are converted into standard JSON format, and finally the JSON data converted by the service grid is uploaded to the cloud. Step 3: Dynamic semantic graph construction: The cloud collects metadata from multiple sources and generates entity relationships using the Transformer model. The cloud platform collects metadata from data sources, including databases, APIs, file systems, and log files. The relationship information output by the Transformer model is used to build a dynamic semantic graph. The nodes in the graph represent entities, and the graph will be updated. New metadata will be collected and added to the graph. Step 4: Based on reinforcement learning, the mapping rules are dynamically optimized to automatically associate the Pressure field with the database column PSI. The state space includes all mapping rules to configure the Pressure field to map to the PSI column. The action space includes all actions taken by the agent to map the Pressure field to the PSI column. The reward function distributes rewards based on the quality of the mapping rules selected by the agent. If the Pressure field is correctly mapped to the PSI column, the agent will receive a positive reward. If the mapping is incorrect, it will receive a negative reward. Step 5: Edge-cloud collaborative optimization, edge nodes cache high-frequency data, temperature readings within 30 seconds, perform local aggregation, reduce cloud load, data is generated in real time, where edge nodes are responsible for caching high-frequency data and performing local aggregation operations to reduce cloud load, and edge nodes process and analyze real-time generated data close to the data source. Through local aggregation, edge nodes can summarize data and extract useful information, calculate the average temperature over a period of time, and then send these aggregation results to the cloud, thereby reducing the amount of data transmission, while reducing network latency, and improving the response speed and efficiency of the entire system; Step 6: The cloud sends the federated learning model parameters, and the edge nodes jointly train the anomaly detection model. The results are encrypted and synchronized to the blockchain. Federated learning allows model training without sharing the original data, protecting the privacy of user data.

[0025] The system includes IoT device layer, edge computing layer, service grid layer, cloud computing layer, application layer and security and extension layer. This system realizes the automatic integration and governance of IoT data through three modules: dynamic semantic graph construction, edge intelligent collaboration and protocol adaptive conversion. The sensors and actuators in the IoT device layer are responsible for collecting data from the environment. The edge nodes set up by the edge computing layer clean, filter and convert the original data collected from sensors and devices to reduce the transmission of invalid data. The service grid layer parses heterogeneous protocols based on the lightweight BERT model and dynamically generates conversion rules. The cloud computing layer uses the Transformer model to build semantic associations of multi-source data and supports cross-domain reasoning. The edge nodes of the application layer perform local preprocessing for cloud management. Quantum encryption communication is carried out through the security and extension layer to defend against quantum computing attacks.

[0026] The IoT device layer is generated through multi-source heterogeneous data, covering IT systems, IoT device sensors, and OT device PLCs, with protocols supporting MQTT, Modbus, OPC UA, HTTP, HTTPS, and CoAP.

[0027] The core modules of the edge computing layer include device fingerprint recognition, which analyzes device communication behavior through the LSTM model, generates a unique hash fingerprint, supports zero-contact registration, and includes dynamic protocol conversion: the lightweight TinyBERT model parses non-standard protocols and dynamically generates field mapping rules.

[0028] The data preprocessing of the edge computing layer is carried out locally through denoising, aggregation, sliding window averaging, and format standardization. The technical support implements elastic expansion and contraction of edge nodes based on Docker and Kubernetes, and supports hot upgrades to add new protocols without downtime.

[0029] The dynamic routing of the service mesh layer includes optimization of data transmission paths based on network bandwidth, node load and business priority based on reinforcement learning algorithms. Policy management includes permission control and caching strategies, fine-grained access strategies based on attributes, and definition of edge caching cycles for high-frequency data.

[0030] The cloud computing layer includes a dynamic semantic graph, the construction method of which includes using the Transformer model to analyze multi-source metadata, device types, protocol headers and data patterns, generating entity relationships for storage and query, and using the Neo4j graph database to store dynamic graphs and support cross-domain association queries in the SPARQL language. Through federated learning, distributed training of anomaly detection models only shares model parameters to ensure data privacy, and through blockchain evidence storage, key operations are written to Hyperledger Fabric to support audit tracing.

[0031] The application layer includes a virtual data lake, which provides a unified SQL interface through Dremio and Snowflake, supports real-time analysis across IT and IoT systems, digital twins, and synchronization of physical equipment status to virtual models to achieve fault prediction and predictive maintenance by combining historical data with real-time streams to generate equipment health scores, triggering the work order system to automatically dispatch orders.

[0032] The security and extension layer includes device lifecycle management, dynamic extension of the protocol library, and quantum encryption communication. It tracks device status throughout the entire link, including online, offline, and faulty. It supports batch firmware upgrades and adds new protocol parsing modules through a microservice architecture. In the future, it can integrate QKD technology to defend against quantum computing attacks.

[0033] The core processes include: device access layer, generating device fingerprints through LSTM model to achieve zero-contact registration, protocol adaptation layer, parsing heterogeneous protocols based on lightweight BERT model, dynamically generating conversion rules, dynamic semantic graph, using Transformer model to build multi-source data semantic association, supporting cross-domain reasoning, edge cloud collaboration engine: edge nodes perform local preprocessing, and the cloud manages global strategies.

[0034] When the IoT data integration method and system based on dynamic semantic graph and edge collaboration is used, in step one, by analyzing the communication interval, the LSTM model generates a device fingerprint, which is a unique identifier based on the device behavior characteristics to identify and distinguish different devices. In the process of generating the fingerprint, the device fingerprint is converted into a hash value for fast comparison and retrieval of data. In step two, the TinyBERT model is used to parse the message sent by the IoT device, and the key fields are extracted from the message using the TinyBERT model, and the extracted key fields are converted into a standard JSON format. Finally, the service grid uploads the converted JSON data to the cloud. In step three, the cloud platform collects metadata from the data source, which includes databases, APIs, file systems, and log files. The relational information output by the Transformer model is used to construct a dynamic semantic graph. The nodes in the graph represent entities, and the graph will be updated. New metadata will be collected and added to the graph. In step four, the state space includes all mapping rule configurations Press The Pressure field is mapped to the PSI column. The action space includes all actions taken by the agent. The Pressure field is mapped to the PSI column. The reward function distributes rewards based on the quality of the mapping rule selected by the agent. If the Pressure field is correctly mapped to the PSI column, the agent will receive a positive reward. If the mapping is wrong, it will receive a negative reward. In step five, data is generated in real time and through rapid processing and response, the edge node performs local aggregation operations to summarize and analyze the cached data. The edge node filters data according to preset rules and sends important and abnormal data to the cloud, thereby reducing unnecessary data transmission and reducing the use of network bandwidth. In step six, federated learning allows model training without sharing original data, protecting the privacy of user data. Edge nodes can use local data for model training, improving training efficiency and model generalization ability. Blockchain provides a decentralized storage solution to ensure the security and transparency of model updates. Through joint training, the model can learn more comprehensive anomaly detection patterns and improve detection accuracy.

[0035] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. An IoT data integration method based on dynamic semantic graph and edge collaboration, characterized by: The following steps are involved: Step 1: Device access and protocol conversion: The edge node captures the device raw data, analyzes the communication interval through the LSTM model, and generates the device fingerprint hash value; Step 2: After the device fingerprint is converted into a hash value, the message structure is parsed through the TinyBERT model, key fields are extracted, converted into a standard JSON format, and uploaded to the cloud through the service grid; Step 3: Dynamic semantic graph construction, collecting multi-source metadata in the cloud, and generating entity relationships using the Transformer model; Step 4: Based on the reinforcement learning dynamic optimization mapping rules, the Pressure field is automatically associated with the database column PSI; Step 5: Edge-cloud collaborative optimization, edge nodes cache high-frequency data, temperature readings within 30 seconds, perform local aggregation, and reduce cloud load; Step 6: The cloud sends the federated learning model parameters, the edge nodes jointly train the anomaly detection model, and the results are encrypted and synchronized to the blockchain.

2. The method for integrating IoT data based on dynamic semantic graph and edge collaboration according to claim 1, characterized in that: In the step 1, the communication interval is analyzed, and the LSTM model generates a device fingerprint. The device fingerprint is a unique identifier based on the device behavior characteristics to identify and distinguish different devices.

3. The method for integrating IoT data based on dynamic semantic graph and edge collaboration according to claim 1, characterized in that: In the step 2, the TinyBERT model parses the message sent by the IoT device, extracts key fields from the message using the TinyBERT model, converts the extracted key fields into a standard JSON format, and finally uploads the JSON data converted by the service grid to the cloud.

4. The method for integrating IoT data based on dynamic semantic graph and edge collaboration according to claim 1, characterized in that: In step three, the cloud platform collects metadata from data sources, including databases, APIs, file systems, and log files. The relational information output by the Transformer model is used to construct a dynamic semantic graph.

5. The method for integrating IoT data based on dynamic semantic graph and edge collaboration according to claim 1, characterized in that: In step 4, the Pressure field is mapped to the PSI column, and the reward function distributes rewards based on the quality of the mapping rule selected by the agent.

6. The method for integrating IoT data based on dynamic semantic graph and edge collaboration according to claim 1, characterized in that: In step five, the edge node can aggregate data and extract useful information, calculate the average temperature over a period of time, and then send these aggregated results to the cloud, thereby reducing the amount of data transmission and reducing network latency.

7. The method for integrating IoT data based on dynamic semantic graph and edge collaboration according to claim 1, characterized in that: In step six, federated learning allows model training without sharing original data, thus protecting the privacy of user data.

8. The system of the Internet of Things data integration method based on dynamic semantic graph and edge collaboration according to claim 1 is characterized in that: It includes IoT device layer, edge computing layer, service grid layer, cloud computing layer, application layer, and security and expansion layer; The IoT device layer is responsible for collecting data from the environment through sensors and actuators; Through the edge computing layer, the raw data collected from sensors and devices is cleaned, filtered and converted through edge nodes to reduce the transmission of invalid data; The service mesh layer parses heterogeneous protocols based on the lightweight BERT model and dynamically generates conversion rules; The cloud computing layer uses the Transformer model to build semantic associations of multi-source data and supports cross-domain reasoning. The edge nodes of the application layer perform local preprocessing for cloud management. Security and extension layer, for quantum encrypted communication and defense against quantum computing attacks.

Images