Multi-source heterogeneous data dynamic protocol secure exchange adaptation system and method

Abstract

The invention discloses a multi-source heterogeneous data dynamic protocol security exchange adaptation system, which comprises a protocol analysis engine, a data encryption module, a data format standardization engine and a routing strategy engine, and is characterized in that the protocol analysis engine is respectively connected with a data source, the data encryption module and the data format standardization engine; the data format standardization engine is connected with the routing strategy engine, and the routing strategy engine is connected with the nodes. The invention discloses a multi-source heterogeneous data dynamic protocol secure exchange adaptation method. Comprising the following steps of protocol analysis and data conversion, data encryption and sensitive data desensitization, integrity verification and anomaly detection, routing strategy and intelligent scheduling, data distribution and storage, and dynamic extension and protocol adaptation, and the dynamic conversion of different protocol data ensures that various types of data can be smoothly interacted. Meanwhile, an advanced data security enhancement technology is integrated, multiple protections such as encryption and authentication are carried out on transmitted and stored data, and the data security is effectively improved.

Description

Technical Field

[0001] The present invention belongs to the field of Internet exchange technology, and in particular relates to a multi-source heterogeneous data dynamic protocol secure exchange adaptation system and method. Background Art

[0002] In the field of data exchange technology, existing solutions have significant technical bottlenecks when facing multi-source heterogeneous data scenarios, which are specifically manifested in three levels of defects: rigid protocol adaptation mechanism defects; traditional gateway devices use static protocol mapping tables (such as CoAP-HTTP fixed conversion templates), which cannot adapt to the new OPC UA nodes dynamically discovered in the industrial Internet of Things; the WSDL-based Web service description method makes it impossible for the protocol converter to automatically identify new features such as the will message of the MQTT 5.0 protocol; the conversion between the medical DICOM protocol and the financial FIX protocol requires manual XSLT script writing, and the conversion timeliness takes an average of 72 hours from requirement proposal to deployment; the fragmented security protection system defects: the existing system uses a layered security architecture (transport layer TLS + application layer PGP), which results in an 8.3-second plaintext exposure window for medical imaging data after DICOM-TLS decryption; the financial transaction system uses a fixed AES-256 key with a key update cycle of up to 30 days, which has caused key leakage incidents in the exchange; the access control of the manufacturing MES system is based on a static RBAC model. Unable to dynamically adapt to temporary access needs in supply chain collaboration scenarios, and lacking cross-domain data governance capabilities: Smart city projects lack unified semantic annotation for video streams (RTSP), sensor data (Modbus), and government documents (XML), resulting in cross-system retrieval accuracy rates below 47%. The use of a unified encryption strategy for cross-border data flows violates the EU GDPR's data minimization principle. Blockchain evidence storage systems fail to implement fine-grained data masking, leading to a 32% risk of commercially sensitive information leakage in supply chain finance scenarios. When using a traditional ESB architecture to connect a SCADA system (DNP3 protocol) with a new energy monitoring system (IEC 61850 protocol), excessive protocol conversion latency (up to 1.2 seconds) resulted in delayed frequency regulation response, causing a 0.3Hz deviation in the regional power grid. In similar projects, the lack of dynamic data encryption policies exposed AMI meter data to man-in-the-middle attacks, impacting the privacy and security of 120,000 users. These technical shortcomings stem from the existing architecture's failure to establish a dynamic evolution mechanism for protocol metamodels, a lack of security policy orchestration capabilities for the data lifecycle, and a lack of a knowledge graph-based semantic interoperability framework. Summary of the Invention

[0003] The present invention aims to overcome these shortcomings by providing a system and method for dynamically converting data from different protocols, ensuring smooth interaction between various data types. Furthermore, the system and method incorporate advanced data security enhancement technologies to provide multiple protections, such as encryption and authentication, for transmitted and stored data, effectively enhancing data security.

[0004] The present invention provides a source heterogeneous data dynamic protocol security exchange adaptation system, which includes a protocol parsing engine, a data encryption module, a data format standardization engine, and a routing policy engine, wherein the protocol parsing engine is respectively connected to the data source, the data encryption module, and the data format standardization engine, the data format standardization engine is connected to the routing policy engine, and the routing policy engine is connected to several nodes.

[0005] The protocol parsing engine supports real-time parsing of HTTP / HTTPS, MQTT, CoAP, FTP, and private TCP / UDP protocols, and implements protocol extension through plug-in design. The protocol parsing engine consists of a dynamic conversion rule library and protocol adaptation middleware. The dynamic conversion rule library defines protocol conversion rules based on YAML / JSON templates and supports hot loading of rules. The protocol adaptation middleware adopts an asynchronous non-blocking communication model to improve protocol conversion efficiency in high-concurrency scenarios.

[0006] The data encryption module transmission encryption adopts the national secret SM4 algorithm to perform real-time encryption on the data stream after dynamic protocol conversion. The key is dynamically negotiated through the ECDH algorithm. The storage encryption adopts the AES-256 algorithm to add secondary encryption to the persistent data. The key is managed by the hardware security module HSM to ensure the security of static data. The data encryption module consists of a sensitive data desensitization module and an integrity verification module. The sensitive data desensitization module matches sensitive fields through regular expressions and replaces the original data with a mask or hash value; Integrity verification module: Use HMAC-SHA256 to generate a data digest. The receiving end verifies data integrity and records abnormal logs.

[0007] The data format standardization engine converts XML, JSON, and binary streams into Apache Avro format and implements structured mapping based on Schema. A metadata management module is provided within the data format standardization engine. The metadata management module stores multi-source data association relationships through the graph database Neo4j and supports SPARQL queries to implement cross-source data association analysis.

[0008] The routing strategy engine consists of load balancing routing, service priority routing, dynamic routing table management, and an intelligent scheduling engine. Load balancing routing is based on real-time monitoring data of the target node's CPU, memory, and network bandwidth, and dynamically selects the node with the lowest load; service priority routing sets priority queues based on data labels, and high-priority data is preferentially routed to low-latency channels; dynamic routing table management uses a distributed key-value database to store routing tables, supporting millisecond-level routing information synchronization; node status changes are monitored through a subscription-publish mechanism; the intelligent scheduling engine integrates a machine learning model, predicts traffic peaks based on historical routing data, and pre-allocates bandwidth resources; supports A / B testing of routing paths, and dynamically optimizes routing strategies based on real-time performance feedback.

[0009] A method for securely exchanging and adapting a dynamic protocol for heterogeneous data from different sources, comprising the following steps: 1) Protocol analysis and data conversion: a. The protocol parsing engine receives data from various sources and performs real-time parsing. The parsing process defines protocol conversion rules through a dynamic conversion rule library and supports hot reloading to meet the real-time parsing requirements of different protocols. The parsed data is passed through the protocol adapter middleware, which uses an asynchronous non-blocking communication model to improve protocol conversion efficiency in high-concurrency scenarios. b. The data format standardization engine converts the parsed data into the Apache Avro format, implements structured mapping through Schema, and uses the metadata management module to store multi-source data relationships. It also supports SPARQL queries to implement cross-source data association analysis. 2) Data encryption and sensitive data desensitization: a. The data encryption module encrypts the data stream after dynamic protocol conversion during transmission. It uses the national secret SM4 algorithm for real-time data encryption. At the same time, the key is dynamically negotiated through the ECDH algorithm. For storage encryption, the AES-256 algorithm is added for secondary encryption of persistent data. The key is managed by the hardware security module HSM to ensure the security of static data. b. The sensitive data desensitization module matches sensitive fields through regular expressions and replaces the original data with masks or hash values to protect sensitive information.

[0010] 3) Integrity verification and anomaly detection: The integrity check module uses HMAC-SHA256 to generate a data digest. The receiving end verifies the data integrity and records abnormal logs to ensure the integrity and security of the data during transmission.

[0011] 4) Routing strategy and intelligent scheduling: a. The routing policy engine dynamically selects the node with the lowest load based on real-time monitoring data such as the target node's CPU, memory, and network bandwidth, and dynamically allocates data using a weighted calculation formula. At the same time, service priority routing sets priority queues based on data tags, prioritizing high-priority data to low-latency channels. b. Dynamic routing table management uses a distributed key-value database to store routing tables and supports millisecond-level routing information synchronization. It monitors node status changes through a subscription-publish mechanism. In the event of a failure, it automatically removes the faulty node and triggers route recalculation. c. The intelligent scheduling engine integrates machine learning models to predict traffic peaks and pre-allocate bandwidth resources based on historical routing data. It supports A / B testing of routing paths and dynamically optimizes routing strategies based on real-time performance feedback. 5) Data distribution and storage: The standardized data is distributed to several nodes through the routing policy engine. The nodes receive data based on load conditions and business needs. At the same time, the system supports persistent storage of data and hierarchical storage and backup of data through a hierarchical caching mechanism to improve data security and reliability. 6) Dynamic expansion and protocol adaptation: The system achieves dynamic expansion of protocols through plug-in design and supports rapid access and adaptation of new protocols; the protocol parsing engine can dynamically load new protocol templates and achieve automation and dynamic expansion of protocol parsing through the PMDL protocol description language.

[0012] Compared with existing technologies, this invention offers significant advantages. The technical solution outlined above demonstrates the following: The protocol parsing engine supports real-time parsing of protocols such as HTTP / HTTPS, MQTT, CoAP, FTP, and private TCP / UDP, enabling protocol expansion through a plug-in design. The dynamic conversion rule library defines protocol conversion rules based on YAML / JSON templates and supports hot rule loading. The protocol adaptation middleware utilizes an asynchronous, non-blocking communication model to improve protocol conversion efficiency in high-concurrency scenarios. The data encryption module uses the national secret SM4 algorithm for real-time encryption of data streams following dynamic protocol conversion, with keys dynamically negotiated using the ECDH (Elliptic Curve Diffie-Hellman) algorithm. Storage encryption employs AES-256 secondary encryption for persistent data, with the key managed by a hardware security module (HSM) to ensure data security at rest. The sensitive data desensitization module matches sensitive fields (such as ID card numbers and bank card numbers) using regular expressions and replaces the original data with a mask or hash value. The integrity verification module generates a data digest using HMAC-SHA256. The receiving end verifies data integrity and logs exceptions. Data format standardization engine: Converts XML, JSON, and binary streams to Apache Avro format, implementing structured mapping based on Schema. Metadata management module: Stores multi-source data relationships using a graph database (Neo4j), supporting SPARQL queries for cross-source data correlation analysis. Routing strategy engine: Load balancing routing: Dynamically selects the node with the lowest load based on real-time monitoring data of the target node's CPU, memory, and network bandwidth (weight calculation algorithm: node weight = 1 / (CPU utilization × 0.6 + memory usage × 0.4)). Business priority routing: Priority queues are set based on data tags (emergency alert, general monitoring), with high-priority data preferentially routed to low-latency channels. Dynamic routing table management: Utilizes a distributed key-value database to store routing tables, supporting millisecond-level routing information synchronization. Node status changes are monitored through a publish-subscribe mechanism (automatically removing faulty nodes and triggering route recalculation in the event of a node failure). Intelligent scheduling engine: Integrates machine learning models to predict traffic peaks and pre-allocate bandwidth resources based on historical routing data. It supports A / B testing of routing paths and dynamically optimizes routing strategies based on real-time performance feedback (throughput, packet loss rate). The protocol adaptation layer supports hot swapping, allowing new protocols to be added without downtime. An adaptive mapping algorithm automatically parses message fields, reducing manual configuration. Transport layer encryption (TLS 1.3) and application layer encryption (AES-256) provide dual security. Blockchain-based audit logs ensure traceability of data operations. An AI-driven load prediction model dynamically allocates bandwidth and computing resources, increasing throughput by over 30% and reducing latency to less than 50ms. The modular design supports rapid integration of third-party services (such as cloud platforms and edge nodes), and an open API facilitates secondary development and customization.It achieves dynamic conversion of data with different protocols, ensuring smooth interaction of various data types. At the same time, it incorporates advanced data security enhancement technologies to provide multiple protections such as encryption and authentication for transmitted and stored data, effectively improving data security. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a structural schematic diagram of the present invention. DETAILED DESCRIPTION

[0014] Example 1: The present invention provides a source heterogeneous data dynamic protocol security exchange adaptation system, which includes a protocol parsing engine, a data encryption module, a data format standardization engine, and a routing policy engine. The protocol parsing engine is respectively connected to the data source, the data encryption module, and the data format standardization engine; the data format standardization engine is connected to the routing policy engine; and the routing policy engine is connected to several nodes.

[0015] The protocol parsing engine supports real-time parsing of HTTP / HTTPS, MQTT, CoAP, FTP, and private TCP / UDP protocols, and implements protocol extension through plug-in design. The protocol parsing engine consists of a dynamic conversion rule library and protocol adaptation middleware. The dynamic conversion rule library defines protocol conversion rules based on YAML / JSON templates and supports hot loading of rules. The protocol adaptation middleware adopts an asynchronous non-blocking communication model to improve protocol conversion efficiency in high-concurrency scenarios.

[0016] The data encryption module transmission encryption adopts the national secret SM4 algorithm to perform real-time encryption on the data stream after dynamic protocol conversion. The key is dynamically negotiated through the ECDH (Elliptic Curve Diffie-Hellman) algorithm. The storage encryption adopts the AES-256 algorithm to add secondary encryption to the persistent data. The key is managed by the hardware security module (HSM) to ensure the security of static data. The data encryption module consists of a sensitive data desensitization module and an integrity verification module. The sensitive data desensitization module matches sensitive fields (ID number, bank card number) through regular expressions and replaces the original data with a mask or hash value; Integrity verification module: Use HMAC-SHA256 to generate a data digest. The receiving end verifies data integrity and records abnormal logs.

[0017] The data format standardization engine converts XML, JSON, and binary streams into Apache Avro format and implements structured mapping based on Schema. A metadata management module is provided within the data format standardization engine. The metadata management module stores multi-source data association relationships through a graph database (Neo4j) and supports SPARQL queries to implement cross-source data association analysis.

[0018] The routing strategy engine consists of load balancing routing, service priority routing, dynamic routing table management, and an intelligent scheduling engine. Load balancing routing is based on real-time monitoring data of the target node's CPU, memory, and network bandwidth, and dynamically selects the node with the lowest load (weight calculation algorithm: node weight = 1 / (CPU utilization × 0.6 + memory occupancy × 0.4)); service priority routing sets priority queues based on data tags (emergency alarm, general monitoring), and high-priority data is preferentially routed to low-latency channels; dynamic routing table management uses a distributed key-value database to store routing tables and supports millisecond-level routing information synchronization; node status changes are monitored through a subscription-publish mechanism (automatically eliminating faulty nodes and triggering route recalculation when a node fails); the intelligent scheduling engine integrates a machine learning model to predict traffic peaks and pre-allocate bandwidth resources based on historical routing data; supports A / B testing of routing paths and dynamically optimizes routing strategies based on real-time performance feedback (throughput, packet loss rate).

[0019] A method for securely exchanging and adapting a dynamic protocol for heterogeneous data from different sources, comprising the following steps: 1) Protocol analysis and data conversion: a. The protocol parsing engine receives data from various sources (such as HTTP / HTTPS, MQTT, CoAP, FTP, private TCP / UDP, etc.) and performs real-time parsing. The parsing process defines protocol conversion rules through a dynamic conversion rule library (based on YAML / JSON templates) and supports hot reloading to meet the real-time parsing requirements of different protocols. The parsed data is passed through the protocol adapter middleware, using an asynchronous non-blocking communication model to improve protocol conversion efficiency in high-concurrency scenarios. b. The data format standardization engine converts parsed data (XML, JSON, binary stream) into the Apache Avro format, implements structured mapping through Schema, and uses the metadata management module to store multi-source data relationships. It also supports SPARQL queries for cross-source data association analysis. 2) Data encryption and sensitive data desensitization: The data encryption module encrypts the data stream after dynamic protocol conversion during transmission, using the national secret SM4 algorithm for real-time data encryption. At the same time, the key is dynamically negotiated through the ECDH (Elliptic Curve Diffie-Hellman) algorithm. For storage encryption, the persistent data is encrypted twice with the AES-256 algorithm. The key is managed by the hardware security module (HSM) to ensure the security of static data. The sensitive data desensitization module matches sensitive fields (ID number, bank card number) through regular expressions and replaces the original data with masks or hash values to protect sensitive information.

[0020] 3) Integrity verification and anomaly detection: The integrity check module uses HMAC-SHA256 to generate a data digest. The receiving end verifies the data integrity and records abnormal logs to ensure the integrity and security of the data during transmission.

[0021] 4) Routing strategy and intelligent scheduling: a. The routing policy engine dynamically selects the node with the lowest load based on real-time monitoring data such as the target node's CPU, memory, and network bandwidth. It uses a weighted calculation formula (node weight = 1 / (CPU utilization × 0.6 + memory usage × 0.4)) to dynamically allocate traffic. Furthermore, service priority routing prioritizes data tags (emergency alert, general monitoring) by setting priority queues, prioritizing high-priority data to low-latency channels. b. Dynamic routing table management uses a distributed key-value database to store routing tables and supports millisecond-level routing information synchronization. It monitors node status changes through a subscription-publish mechanism. In the event of a failure, it automatically removes the faulty node and triggers route recalculation. c. The intelligent scheduling engine integrates machine learning models to predict traffic peaks and pre-allocate bandwidth resources based on historical routing data. It supports A / B testing of routing paths and dynamically optimizes routing strategies based on real-time performance feedback (throughput, packet loss rate). 5) Data distribution and storage: The standardized data is distributed to several nodes through the routing policy engine. The nodes receive data based on load conditions and business needs. At the same time, the system supports persistent storage of data and hierarchical storage and backup of data through a hierarchical caching mechanism to improve data security and reliability. 6) Dynamic expansion and protocol adaptation: The system achieves dynamic expansion of protocols through plug-in design, supporting the rapid access and adaptation of new protocols; the protocol parsing engine can dynamically load new protocol templates and realize the automation and dynamic expansion of protocol parsing through PMDL (Protocol Model Description Language).

Claims

1. A multi-source heterogeneous data dynamic protocol secure exchange adaptation system, including a protocol parsing engine, a data encryption module, a data format standardization engine, and a routing strategy engine, characterized in that: The protocol parsing engine is connected to the data source, the data encryption module, and the data format standardization engine respectively; the data format standardization engine is connected to the routing strategy engine; and the routing strategy engine is connected to several nodes.

2. The multi-source heterogeneous data dynamic protocol secure exchange adaptation system according to claim 1, characterized in that: The protocol parsing engine consists of a dynamic conversion rule library and a protocol adaptation middleware. The dynamic conversion rule library defines protocol conversion rules based on YAML / JSON templates and supports hot loading of rules. The protocol adaptation middleware adopts an asynchronous non-blocking communication model to improve the protocol conversion efficiency in high-concurrency scenarios.

3. The multi-source heterogeneous data dynamic protocol secure exchange adaptation system according to claim 1, characterized in that: The data encryption module consists of a sensitive data desensitization module and an integrity verification module. The sensitive data desensitization module matches sensitive fields through regular expressions and replaces the original data with a mask or hash value; the integrity verification module uses HMAC-SHA256 to generate a data summary, and the receiving end verifies the data integrity and records an exception log.

4. The multi-source heterogeneous data dynamic protocol secure exchange adaptation system according to claim 1, characterized in that: The data format standardization engine is provided with a metadata management module, which stores multi-source data association relationships through the graph database Neo4j and supports SPARQL queries to implement cross-source data association analysis.

5. The multi-source heterogeneous data dynamic protocol secure exchange adaptation system according to claim 1, characterized in that; The routing strategy engine consists of load balancing routing, service priority routing, dynamic routing table management, and intelligent scheduling engine. Load balancing routing is based on real-time monitoring data of the target node's CPU, memory, and network bandwidth, and dynamically selects the node with the lowest load; Business priority routing sets priority queues based on data tags, with high-priority data being routed to low-latency channels first. Dynamic routing table management uses a distributed key-value database to store routing tables, supporting millisecond-level routing information synchronization. Node status changes are monitored through a subscription-publish mechanism. The intelligent scheduling engine integrates machine learning models to predict traffic peaks and pre-allocate bandwidth resources based on historical routing data. It supports A / B testing of routing paths and dynamically optimizes routing strategies based on real-time performance feedback.

6. A method for securely exchanging and adapting dynamic protocols for multi-source heterogeneous data, comprising the following steps: 1) Protocol analysis and data conversion: a. The protocol parsing engine receives data from various sources and performs real-time parsing. The parsing process defines protocol conversion rules through a dynamic conversion rule library and supports hot reloading to meet the real-time parsing requirements of different protocols. The parsed data is passed through the protocol adapter middleware, which uses an asynchronous non-blocking communication model to improve protocol conversion efficiency in high-concurrency scenarios. b. The data format standardization engine converts the parsed data into the Apache Avro format, implements structured mapping through Schema, and uses the metadata management module to store multi-source data relationships. It also supports SPARQL queries to implement cross-source data association analysis. 2) Data encryption and sensitive data desensitization: a. The data encryption module encrypts the data stream after dynamic protocol conversion during transmission. It uses the national secret SM4 algorithm for real-time data encryption. At the same time, the key is dynamically negotiated through the ECDH algorithm. For storage encryption, the AES-256 algorithm is added for secondary encryption of persistent data. The key is managed by the hardware security module HSM to ensure the security of static data. b. The sensitive data desensitization module matches sensitive fields through regular expressions and replaces the original data with masks or hash values to protect sensitive information; 3) Integrity verification and anomaly detection: The integrity check module uses HMAC-SHA256 to generate a data digest. The receiving end verifies the data integrity and records abnormal logs to ensure the integrity and security of the data during transmission. 4) Routing strategy and intelligent scheduling: a. The routing policy engine dynamically selects the node with the lowest load based on real-time monitoring data such as the target node's CPU, memory, and network bandwidth, and dynamically allocates data using a weighted calculation formula. At the same time, service priority routing sets priority queues based on data tags, prioritizing high-priority data to low-latency channels. b. Dynamic routing table management uses a distributed key-value database to store routing tables and supports millisecond-level routing information synchronization. It monitors node status changes through a subscription-publish mechanism. In the event of a failure, it automatically removes the faulty node and triggers route recalculation. c. The intelligent scheduling engine integrates machine learning models to predict traffic peaks and pre-allocate bandwidth resources based on historical routing data. It supports A / B testing of routing paths and dynamically optimizes routing strategies based on real-time performance feedback. 5) Data distribution and storage: The standardized data is distributed to several nodes through the routing policy engine. The nodes receive data based on load conditions and business needs. At the same time, the system supports persistent storage of data and hierarchical storage and backup of data through a hierarchical caching mechanism to improve data security and reliability. 6) Dynamic expansion and protocol adaptation: The system achieves dynamic expansion of protocols through plug-in design and supports rapid access and adaptation of new protocols; the protocol parsing engine can dynamically load new protocol templates and achieve automation and dynamic expansion of protocol parsing through the PMDL protocol description language.

Images