A cross-system data intercommunication method, platform, device and medium

Abstract

The present application relates to a cross-system data intercommunication method, platform, device and medium, belonging to the field of data transmission, and the method includes: accessing at least two heterogeneous data source systems through a protocol adapter; parsing the field name, data type and constraint conditions of each source system, generating a standardized metadata file, and extracting the original data stream; performing data cleaning and format conversion on the original data stream, outputting the cleaned structured data and processing log; performing dynamic encryption according to the sensitivity level tag, generating a secure data packet queue based on the access permission attributes of the target system and sending it to the target system, and recording the transmission log; verifying the data consistency between the target system and the source system and performing abnormal retransmission, updating the transmission log, and generating a bloodline map containing the data flow path; dynamically adjusting the data cleaning rules and transmission parameters according to the bloodline map and the updated transmission log. The present application can improve the stability and efficiency of data intercommunication.

Description

Technical Field

[0001] The present application relates to the field of data transmission technology, and in particular to a cross-system data intercommunication method, platform, device and medium. Background Art

[0002] With the rapid development of information technology and the surge in global data volumes, more and more businesses, governments, and organizations are relying on information systems to process and manage massive amounts of data. The application of technologies such as big data, cloud computing, and the Internet of Things has made the interaction between diverse business systems and data sources a critical area in modern information technology. Consequently, cross-system data interoperability has become particularly important, especially in terms of interoperability, data security, and consistency management across heterogeneous systems.

[0003] Currently, many enterprises and institutions' IT environments are host to a vast array of heterogeneous data sources, including relational and non-relational databases, file systems, external APIs, and IoT devices. These data sources exhibit significant differences in format, storage methods, data types, and constraints. However, common cross-system data interoperability methods often rely on customized solutions, making them incapable of addressing the growing demand for cross-platform data exchange. This is particularly true for large-scale data transfers, where they are unable to adapt to the network conditions and complex scenarios between disparate systems, reducing the stability and efficiency of data interoperability. Summary of the Invention

[0004] In order to improve the stability and efficiency of data intercommunication, this application provides a cross-system data intercommunication method, platform, device and medium.

[0005] In the first aspect, the present application provides a cross-system data intercommunication method, which adopts the following technical solutions:

[0006] A cross-system data intercommunication method, the method comprising:

[0007] Access at least two heterogeneous data source systems through protocol adapters;

[0008] Parse the field names, data types, and constraints of each source system, generate standardized metadata files, and extract the original data stream; wherein the standardized metadata files include field semantic labels, field constraint rules, and sensitivity level tags;

[0009] Based on the field semantic labels and field constraint rules, the original data stream is cleaned and format converted, and the cleaned structured data and processing logs are output;

[0010] Dynamically encrypting the cleaned structured data according to the sensitivity level tag, and generating a secure data packet queue based on the access permission attributes of the target system;

[0011] Sending the secure data packet queue to the target system and recording a transmission log;

[0012] Based on the transmission log and processing log, verify the data consistency between the target system and the source system and perform abnormal retransmission, update the transmission log, and generate a lineage map containing the data flow path;

[0013] According to the bloodline map and the updated transmission log, data cleaning rules and transmission parameters are dynamically adjusted.

[0014] By adopting the above technical solutions, we ensure efficient, secure, and consistent data transmission and processing from data source access to target systems. We utilize protocol adapters to enable access to heterogeneous data sources, ensure data security through dynamic encryption and security tokens, ensure data integrity through intelligent transmission and data consistency verification, and enhance the system's overall performance and adaptability through dynamic adjustment mechanisms. Ultimately, this achieves cross-system data interoperability, making it suitable for multi-system collaboration applications such as cross-departmental data sharing, big data analysis, and IoT data synchronization. It effectively addresses industry pain points such as data silos, inefficient transmission, and high security risks.

[0015] Optionally, the step of dynamically encrypting the cleaned structured data according to the sensitivity level tag and generating a secure data packet queue based on the access permission attributes of the target system includes:

[0016] Generate a sensitive field mapping table based on the sensitivity level tags in the standardized metadata file;

[0017] Dynamically assign an encryption key and an encryption algorithm according to the sensitivity level tag, bind the encryption key to the shard ID in the structured data, and output a shard key table;

[0018] Based on the sensitive field mapping table and the sharding key table, dynamically encrypt the sensitive fields in the cleaned structured data, and output the encrypted data shards and the associated encryption algorithm identifiers;

[0019] Generate a permission policy document based on the access permission attributes of the target system, bind the shard ID to the permission policy document, and generate a permission token file based on an asymmetric encryption algorithm;

[0020] The encrypted data fragment, fragment ID and permission token file are encapsulated into a secure data packet, and the fragment ID, encryption algorithm identifier and target system attributes are written into the fragment metadata index table, and a secure data packet sequence is output.

[0021] By adopting the above technical solutions, based on dynamic encryption and secure data packet generation mechanisms, we ensure the encryption, access control, and secure transmission of sensitive data during cross-system data exchange. By combining the sensitive field mapping table with the sharding key table, data can be encrypted using different encryption algorithms based on its sensitivity, ensuring data confidentiality. The generation mechanism of permission policy documents and digital signatures ensures that only authorized systems can access sensitive data, thereby improving system security. Finally, the sharding metadata index table and protocol encapsulation ensure the reliability and consistency of data packet transmission, effectively meeting the data exchange needs of large-scale heterogeneous systems.

[0022] Optionally, the step of sending the secure data packet queue to the target system and recording a transmission log includes:

[0023] Obtaining the service type, time sensitivity rule, and user configuration policy of the encrypted data slice, and generating a real-time tag;

[0024] Based on the preset transmission strategy table, a transmission strategy is generated according to the target system attributes and real-time tags in the shard metadata index table;

[0025] Obtain network status monitoring data in real time;

[0026] Determining a transmission channel based on the transmission strategy and network status monitoring data;

[0027] Distributing the secure data packet queue to the transmission channel;

[0028] Obtain the shard ID, transmission channel type, transmission timestamp, and network time data, and write the structured transmission log according to the shard ID partition.

[0029] By employing these technical solutions, dynamically selecting transmission strategies and channel types, monitoring network status in real time, and recording detailed transmission logs, we achieve efficient and reliable packet transmission and logging. The use of real-time tags ensures prioritized transmission of different data types, while network status monitoring and dynamic policy adjustments optimize transmission efficiency and ensure data integrity. Structured transmission logs provide a valuable basis for subsequent troubleshooting, performance analysis, and optimization.

[0030] Optionally, based on the transmission log and the processing log, verifying the data consistency between the target system and the source system and performing abnormal retransmission, updating the transmission log, and generating a lineage map including the data flow path includes the following steps:

[0031] Based on the shard ID in the transmission log and the field change record in the processing log, a shard-level data snapshot table is generated by association; wherein the shard-level data snapshot table includes the association result of the source system hash value, the target system hash value and the transmission path node;

[0032] Perform data anomaly detection based on the source system hash value and the target system hash value, and output an abnormal shard list and an abnormal type mark;

[0033] Extracting original data segments or difference fields from the source system according to the abnormal segment list and the abnormal type mark, retransmitting them to the target system through the transmission channel, and outputting retry records and updated transmission logs;

[0034] Based on the retry records and the updated transmission logs, a lineage graph including data nodes, processing nodes and abnormal nodes is constructed.

[0035] By employing these technical solutions, the system can detect data anomalies in real time and repair data loss or corruption through an automated retransmission mechanism. The construction of a lineage map provides clear visualization of data flow paths, helping to promptly identify and resolve anomalies during transmission. This significantly improves the reliability, transparency, and maintainability of data transmission, and provides reliable support for subsequent data consistency checks and rollback operations.

[0036] Optionally, the step of dynamically adjusting data cleaning rules and transmission parameters according to the bloodline map and the updated transmission log includes:

[0037] Based on the bloodline map and the updated transmission log, a data flow-performance association table is constructed;

[0038] According to the data flow-performance association table, network delay, cleaning time and resource utilization indicators are aggregated and calculated by transmission channel type and cleaning rule version, and a performance analysis report is output;

[0039] Based on the performance analysis report, a strategy parameter table is generated through a reinforcement learning model, and tuning parameters are output; the tuning parameters include transmission channel switching instructions, compression algorithm adjustment suggestions, and cleaning rule simplification strategies;

[0040] Dynamically update the data cleaning rule configuration file and the transmission strategy table according to the strategy parameter table, and output the updated data cleaning rules and transmission parameters;

[0041] Collect updated transmission logs and real-time performance data, verify the tuning effect and feed it back to the reinforcement learning model, and output the model training iteration record.

[0042] By employing this technical solution, combining lineage graphs with real-time transmission logs and utilizing a reinforcement learning model for dynamic tuning, we can not only adjust data cleaning rules and transmission parameters based on real-time performance data, but also continuously optimize system performance through continuous feedback and iterative training. Ultimately, this technical solution significantly improves data processing efficiency, reduces latency, optimizes resource utilization, and ensures data consistency, resulting in more efficient data transmission and cleaning.

[0043] Optionally, the standardized metadata file, processing log, and permission token file are all stored in a distributed database and are associated and indexed by a unique shard ID.

[0044] Secondly, this application provides a cross-system data interoperability platform that adopts the following technical solutions:

[0045] A cross-system data intercommunication platform, comprising:

[0046] A source system access module, used to access at least two heterogeneous data source systems through a protocol adapter;

[0047] A parsing and extraction module is used to parse the field names, data types, and constraints of each source system, generate standardized metadata files, and extract the original data stream; wherein the standardized metadata files include field semantic labels, field constraint rules, and sensitivity level tags;

[0048] A data processing module is used to perform data cleaning and format conversion on the original data stream based on the field semantic labels and field constraint rules, and output the cleaned structured data and processing log;

[0049] a security data packet generation module, configured to dynamically encrypt the cleansed structured data according to the sensitivity level tag, and generate a security data packet queue based on the access permission attributes of the target system;

[0050] A transmission module, configured to send the secure data packet queue to the target system and record a transmission log;

[0051] A lineage map generation module is used to verify the data consistency between the target system and the source system based on the transmission log and the processing log, perform abnormal retransmission, update the transmission log, and generate a lineage map containing the data flow path;

[0052] The dynamic tuning module is used to dynamically adjust data cleaning rules and transmission parameters according to the blood relationship map and the updated transmission log.

[0053] Optionally, the cross-system data intercommunication platform includes a microservice module deployed in a Kubernetes cluster.

[0054] In a third aspect, the present application provides a computer device that adopts the following technical solution:

[0055] A computer device comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method according to the first aspect.

[0056] In a fourth aspect, the present application provides a computer-readable storage medium, which adopts the following technical solution:

[0057] A computer-readable storage medium stores a computer program capable of being loaded by a processor and executing any one of the methods in the first aspect.

[0058] In summary, this application has at least one of the following beneficial technical effects: By combining the lineage map with the transmission log, the system can verify data consistency in real time and perform abnormal retransmissions. It can also dynamically adjust data cleaning rules and transmission parameters based on feedback to ensure the accuracy and security of data flow, thereby achieving efficient and secure data exchange between heterogeneous data sources. This technical solution not only improves data transmission efficiency and accuracy, but also strengthens cross-system data security. It is suitable for complex data processing environments and has important practical application significance. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 This is the first flow chart of the cross-system data intercommunication method of one embodiment of the present application.

[0060] Figure 2 This is the second flow chart of the cross-system data intercommunication method of one embodiment of the present application.

[0061] Figure 3 This is the third flow chart of the cross-system data intercommunication method of one of the embodiments of the present application.

[0062] Figure 4 This is the fourth flow chart of the cross-system data intercommunication method according to one embodiment of the present application.

[0063] Figure 5 This is the fifth flow chart of the cross-system data intercommunication method according to one embodiment of the present application. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solutions and advantages of this application more clear, the following Figure 1-5 It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application.

[0065] The embodiment of the present application discloses a method for cross-system data intercommunication.

[0066] Reference Figure 1 , a cross-system data intercommunication method, the method comprising:

[0067] Step S101: access at least two heterogeneous data source systems through a protocol adapter;

[0068] By pre-installing multiple protocol adapters, heterogeneous data source interface types are dynamically identified and standardized communication links are established. Heterogeneous data sources may have different formats, structures, and storage methods. Therefore, protocol adapters (such as JDBC, HTTP, and MQTT) are required to enable cross-platform or cross-system data access. Protocol adapters act as a bridge between different systems, ensuring consistent access to diverse data sources.

[0069] Specifically, during implementation, the protocol adapter first needs to identify the data source type and select the appropriate connection method. For example, if the data source is a relational database (such as MySQL), the JDBC protocol is used for connection; if the data source is real-time data from an IoT device, the MQTT protocol may be used.

[0070] Step S102: parse the field names, data types, and constraints of each source system, generate standardized metadata files, and extract the original data stream;

[0071] Among them, the standardized metadata file includes field semantic labels, field constraint rules and sensitivity level tags;

[0072] Specifically, the generated standardized metadata files unify the structure, data types, and constraints of various data sources, providing consistent specifications for subsequent data processing. At the same time, the original data stream is retained for subsequent processing to ensure data integrity.

[0073] For example, Schema parsing tools (such as Apache Calcite) can be used to extract raw metadata such as field names, types, and primary key constraints. These metadata can then be mapped into semantic labels that are understandable across systems using a unified semantic model (such as JSON-LD). For example, source system A is a database with fields containing "user ID" and "name," while source system B is a CSV file with fields containing "customer ID" and "age." The system automatically parses these fields and normalizes them into a standard format.

[0074] Step S103: Based on the field semantic labels and field constraint rules, the original data stream is cleaned and format converted, and the cleaned structured data and processing log are output;

[0075] Using the field semantic labels and field constraint rules (such as non-null and value range) defined in standardized metadata files, the rule engine is invoked to clean the raw data stream and select transformation logic based on the semantic labels. Data cleansing aims to remove erroneous, incomplete, duplicate, or non-constrained data, while format conversion converts the data into a format acceptable to the target system.

[0076] Specifically, data cleansing is performed according to field constraint rules. For example, if a field's type is "integer" but a non-integer value appears in the data, the system will identify and remove or repair the value. Format conversion can be implemented programmatically, such as converting CSV to JSON or converting XML data to JSON via API calls. The system performs cleansing and conversion operations based on pre-defined rules to ensure data accuracy and consistency.

[0077] For example, suppose the data field "age" is required to be an integer, but there are non-numeric values ​​in the source data (such as "age unknown"). In this case, the cleaning rule will identify and fix it to the default value or delete the record.

[0078] Step S104 , dynamically encrypting the cleaned structured data according to the sensitivity level tag, and generating a secure data packet queue based on the access permission attributes of the target system;

[0079] The secure data packet sequence carries a shard ID and a permission token. To ensure data security during transmission, sensitive data (such as ID numbers and mobile phone numbers) must be encrypted or desensitized. Furthermore, the generated secure data packet must generate a permission token based on the target system's access permissions.

[0080] Specifically, based on the sensitivity level tag in the metadata (e.g., PII, financial data), an encryption algorithm is dynamically selected (AES-GCM for general sensitive data, and SM4 for high-security scenarios). Encryption algorithms such as AES-GCM can be used to encrypt data, ensuring it is not leaked during transmission. By generating a JWT (JSON Web Token) token, the access control of the data packet can be ensured to match the permission attributes of the target system. Each secure data packet contains a shard ID and a corresponding permission token, which is used to verify data access rights in the target system.

[0081] As you can understand, encryption ensures that sensitive data is not leaked during transmission, while the permission binding mechanism ensures that only authorized target systems or users can access the data. Security tokens and shard IDs are used to further ensure the security and integrity of data transmission.

[0082] For example, when the data contains an ID number field, the system will encrypt it based on the AES-GCM algorithm and generate a JWT token containing access permissions, such as "shard ID=12345, permission=read-only".

[0083] Step S105, sending the secure data packet queue to the target system and recording the transmission log;

[0084] The transmission log includes information such as the shard ID, network time consumption, transmission timestamp, and channel type. Based on the data type, transmission timeliness, and network conditions, the system selects the appropriate transmission mode to send the secure data packet queue to the target system. These modes include real-time transmission and batch transmission. Real-time transmission is suitable for time-sensitive data, while batch transmission is suitable for large amounts of data or historical data.

[0085] Specifically, real-time data transmission can be achieved through Kafka streaming, ensuring low latency and high throughput. Bulk data can be asynchronously transmitted using HDFS or similar distributed file systems after being partitioned and compressed. Network conditions (such as latency and bandwidth) during transmission affect the selected transmission method and path. The system optimizes the transmission path based on real-time network conditions. For example, for real-time monitoring data, the system uses Kafka for transmission; for historical data, the system may use the DistCp tool to transfer data in bulk to HDFS.

[0086] It is understood that by selecting appropriate transmission channels and optimizing transmission paths, efficient and stable data transmission can be ensured. The recorded transmission logs provide a basis for subsequent error handling and performance optimization.

[0087] Step S106: Based on the transmission log and processing log, verify the data consistency between the target system and the source system and perform abnormal retransmission, update the transmission log, and generate a lineage map containing the data flow path;

[0088] During data transmission, transmission failures or data corruption may occur, necessitating data consistency verification. By comparing the data hash values ​​of the source and target systems using the shard IDs in the transmission log and the field change records in the processing log, data consistency is ensured. If anomalies are detected, an automatic retry mechanism or data rollback is triggered.

[0089] Specifically, the data packet's hash value is compared with a CRC32 checksum to ensure that the data has not been tampered with during transmission. If a transmission fragment is lost or the hash value is inconsistent, the system will locate the abnormal data based on the transmission log and retransmit the lost fragment. If the retry limit is exceeded, the target system will perform a rollback operation to delete the damaged data in the target system and mark the abnormal status.

[0090] It can be understood that through data consistency verification and error recovery mechanisms, the data of the target system is ensured to be consistent with the source system, avoiding data incompleteness or loss due to transmission errors.

[0091] Step S107: Dynamically adjust data cleaning rules and transmission parameters based on the bloodline map and the updated transmission log.

[0092] Among them, by monitoring the network time consumption data in the bloodline map and transmission log, dynamically adjusting the data cleaning rules and transmission parameters, and analyzing historical data, the system can optimize the parallelism and transmission strategy of ETL tasks and improve system efficiency.

[0093] Specifically, reinforcement learning models can be used to predict optimal strategies for data cleaning and transmission, automatically adjusting parameters based on historical data flows and network conditions. For example, if network latency is high during a certain period, the system might automatically adjust data cleaning rules to reduce data processing complexity or adjust transmission parameters to avoid network bottlenecks. By dynamically adjusting strategies, the system can self-optimize, improving processing efficiency and transmission quality.

[0094] The above implementation ensures efficient, secure, and consistent data transmission and processing from data source access to the target system. Protocol adapters are used to enable access to heterogeneous data sources, dynamic encryption and security tokens are used to ensure data security, intelligent transmission and data consistency verification are used to ensure data integrity, and dynamic adjustment mechanisms are used to enhance the system's overall performance and adaptability. Ultimately, cross-system data interoperability is achieved, making it suitable for multi-system collaboration applications such as cross-departmental data sharing, big data analysis, and IoT data synchronization. This effectively addresses industry pain points such as data silos, inefficient transmission, and high security risks.

[0095] Reference Figure 2 As an implementation of step S104, the steps of dynamically encrypting the cleansed structured data according to the sensitivity level tag and generating a secure data packet queue based on the access permission attributes of the target system include:

[0096] Step S201: Generate a sensitive field mapping table based on the sensitivity level tags in the standardized metadata file;

[0097] During cross-system data interoperability, the cleaned structured data must first be labeled with a sensitivity level. This process begins by generating a sensitive field mapping table based on the sensitivity level tags in the standardized metadata file. This sensitive field mapping table accurately identifies which data fields are sensitive, allowing appropriate encryption protection measures to be implemented, providing the foundation for subsequent dynamic encryption.

[0098] For example, consider a field called "ID Number" that is marked as PII (Personally Identifiable Information) in the metadata file. The generated sensitive field mapping table associates this field with its PII sensitivity level, allowing subsequent processing to identify encryption requirements for this field.

[0099] Step S202: Dynamically assign encryption keys and encryption algorithms based on the sensitivity level tag, bind the encryption keys to the shard IDs in the structured data, and output a shard key table;

[0100] The shard key table includes the shard ID, encryption key, and encryption algorithm type. The shard ID is generated after data cleansing and serves as the core identifier for all subsequent steps. The shard ID is used to derive encryption parameters.

[0101] Specifically, the system dynamically selects an encryption algorithm based on the level of sensitive data (e.g., PII, financial data). The Key Management System (KMS) provides keys and generates corresponding keys based on the selected encryption algorithm. The system binds the keys to the shard ID, ensuring that each data shard has a unique key.

[0102] In this embodiment, sensitive data tags are mapped to encryption algorithms. For example, PII data is encrypted using the AES-GCM algorithm, and financial data is encrypted using the SM4 algorithm. Furthermore, the shard ID is bound to the key. The shard ID acts as a unique identifier and is bound to the corresponding encryption key. Each data shard has a different encryption key to ensure data security.

[0103] Step S203: Based on the sensitive field mapping table and the sharding key table, the sensitive fields in the cleaned structured data are dynamically encrypted, and the encrypted data shards and the associated encryption algorithm identifiers are output;

[0104] The system dynamically encrypts sensitive fields in the data based on the encryption algorithm and key in the shard key table. Each data shard derives encryption parameters based on its shard ID. Different sensitive fields are encrypted using different encryption algorithms (such as AES-GCM and SM4), ensuring the security of sensitive data during transmission.

[0105] Specifically, during the encryption process, encryption parameters are derived from the shard ID. For example, the AES-GCM algorithm uses the hash value of the shard ID as the additional authentication data (AAD), and the SM4 algorithm concatenates the shard ID and the target system ID to generate the initialization vector (IV). Each encrypted shard contains the encrypted ciphertext and the encryption algorithm identifier.

[0106] For example, assume that an ID number is encrypted using AES-GCM, and the shard ID is "12345." The system generates additional authentication data (AAD) based on the hash value of the shard ID. The encrypted data shard is output as ciphertext and marked as encrypted using AES-GCM.

[0107] Step S204: Generate a permission policy document based on the access permission attributes of the target system, bind the shard ID to the permission policy document, and generate a permission token file using an asymmetric encryption algorithm;

[0108] Among them, the permission policy document contains information such as the role, permissions, and accessible fields of the target system. The permission token file is a digital signature token carrying the shard ID, ensuring that the target system can access data only when it has the correct permissions.

[0109] Specifically, before data is transferred, the system generates a permission policy document based on the target system's access rights attributes (such as role, IP address, and list of accessible fields). This document binds the shard ID to the target system's access rights and is digitally signed using an asymmetric encryption algorithm (such as RSA-PSS) to generate a permission token file.

[0110] Step S205: Encapsulate the encrypted data fragment, fragment ID and permission token file into a secure data packet, write the fragment ID, encryption algorithm identifier and target system attributes into the fragment metadata index table, and output a secure data packet sequence.

[0111] The system encapsulates the encrypted data fragments, fragment IDs, and permission token files into secure data packets according to the protocol format. Each data packet contains a header field, a payload field, and a trailer field. Specifically, the header field contains the fragment ID and encryption algorithm identifier, the payload field contains the encrypted data ciphertext, and the trailer field contains the permission token and authentication tag (AES-GCM authentication tag or SM4 IV value).

[0112] In addition, after generating the security data packet, the system will write the shard ID, encryption algorithm identifier and target system properties into the shard metadata index table, which is used for subsequent transmission channel selection and data consistency verification.

[0113] In the above implementation, based on dynamic encryption and secure data packet generation mechanisms, encryption, access control, and secure transmission of sensitive data are ensured during cross-system data intercommunication. By combining the sensitive field mapping table with the sharding key table, data can be encrypted using different encryption algorithms based on its sensitivity, ensuring data confidentiality. The generation mechanism of permission policy documents and digital signatures ensures that only authorized systems can access sensitive data, thereby improving system security. Finally, through the sharding metadata index table and protocol encapsulation, the reliability and consistency of data packet transmission are ensured, which can effectively meet the data exchange needs of large-scale heterogeneous systems.

[0114] Reference Figure 3 As an implementation of step S105, the steps of sending the secure data packet queue to the target system and recording the transmission log include:

[0115] Step S301: Obtain the service type, time sensitivity rule, and user configuration policy of the encrypted data slice and generate a real-time tag;

[0116] Before data packets are transmitted, the encrypted data slices are analyzed. This analysis generates a corresponding real-time tag based on the encrypted data slice's service type, time sensitivity rules, and user-configured policies. The real-time tag determines the packet's transmission priority and guides the system in selecting the appropriate transmission channel.

[0117] Specifically, the time sensitivity of data is determined by categorizing business types based on business needs, such as financial transactions and real-time monitoring. For example, financial transaction data may require a higher real-time tag, while log data may have lower real-time requirements. Time sensitivity rules are determined based on the timeliness of data processing and define how quickly packets must be transmitted to the target system. For example, real-time data may require a high-priority tag, while non-real-time data can use a normal tag. Furthermore, real-time tags can be further adjusted based on user configuration policies. For example, some systems may require special real-time processing during packet transmission.

[0118] For example, if a data packet is of the “financial transaction” type and has a “high priority” time sensitivity, the system will generate a real-time tag with a “high priority” tag to ensure that the data packet is transmitted first.

[0119] Step S302: Generate a transmission strategy based on the preset transmission strategy table and the target system attributes and real-time tags in the shard metadata index table.

[0120] The transmission strategy includes candidate channel types, compression algorithms, and retry times. After generating the real-time tag, the system will generate an appropriate transmission strategy based on the preset transmission strategy table, the target system's interface protocol, the real-time tag, and other target system attributes.

[0121] Specifically, candidate channel types include real-time streaming channels or batch transmission channels, and the compression algorithm is selected according to the data type or the requirements of the transmission channel, such as Zstandard or Brotli; when the transmission fails, the system will set the number of retries to ensure that the data packet can be transmitted reliably.

[0122] Step S303, obtaining network status monitoring data in real time;

[0123] Network status monitoring data includes network latency, bandwidth utilization, and packet loss rate between the source and target systems. Specifically, network latency refers to the delay between the source and target systems and is used to assess the real-time nature of data transmission. Bandwidth utilization refers to the current network bandwidth usage, ensuring that bandwidth bottlenecks prevent data transmission. Packet loss rate refers to the percentage of packets lost during data transmission. A high packet loss rate can affect data integrity and accuracy.

[0124] Step S304, determining a transmission channel according to the transmission strategy and network status monitoring data;

[0125] Among them, the transmission channel includes a real-time streaming channel or a batch transmission channel. The real-time streaming channel is suitable for high-priority data (such as financial data), and the batch transmission channel is suitable for non-real-time data or large-scale data.

[0126] In some embodiments, when the system detects that the real-time label is "high priority" and the network status is good, the real-time stream channel is selected, and the Kafka transaction producer transmission can be used; when the real-time requirements of data transmission are low, or the network status is poor, the batch transmission channel is selected, and the Zstandard compression algorithm can be used for sharding and then asynchronously transmitted through the DistCp tool.

[0127] Step S305, distributing the secure data packet queue to the transmission channel;

[0128] Once the transmission channel is determined, the system will distribute the encrypted secure data packet queue according to the selected channel type. If Kafka is selected as the transmission channel, the data packet will be sent through the Kafka producer and a transaction identifier will be generated. If HDFS is selected, the data packet will be compressed with Zstandard before being written to the HDFS directory.

[0129] Step S306: Obtain the shard ID, transmission channel type, transmission timestamp, and network time-consuming data, and write the structured transmission log into partitions according to the shard ID.

[0130] Among them, the system will collect key information of each data packet, and the fragment ID is used to identify the uniqueness of the data packet; the transmission channel type includes real-time streaming channel or batch channel, the transmission timestamp is used to record the transmission time of the data packet, and the network time consumption data includes end-to-end transmission delay.

[0131] For example, the fragment ID of the data packet is "12345", the transmission time is "2024-08-13 12:30:00 UTC", the channel type is Kafka, and the network time is 150ms. This information will be stored in the transmission log.

[0132] The above implementation dynamically selects transmission strategies and channel types, monitors network status in real time, and maintains detailed transmission logs, achieving efficient and reliable data packet transmission and logging. The use of real-time tags ensures prioritized transmission of different data types, while network status monitoring and dynamic policy adjustments optimize transmission efficiency and ensure data integrity. Structured transmission logs provide a valuable basis for subsequent troubleshooting, performance analysis, and optimization.

[0133] Reference Figure 4 As an implementation method of step S106, based on the transmission log and the processing log, verifying the data consistency between the target system and the source system and performing abnormal retransmission, updating the transmission log, and generating a lineage map including the data flow path include the following steps:

[0134] Step S401: Generate a shard-level data snapshot table based on the shard ID in the transmission log and the field change record in the processing log;

[0135] The shard-level data snapshot table includes the association results of the source system hash value, the target system hash value, and the transmission path nodes;

[0136] Specifically, a shard-level data snapshot table is a structured data table used to record key state information about a data shard at a specific point in time during cross-system transmission. By correlating transmission and processing logs, it forms an "instantaneous state mirror" of the data shard in the source and target systems. This serves as the core basis for data consistency verification and lineage tracing.

[0137] In this embodiment of the present application, the source system hash value represents the unique fingerprint of the data in the source system, calculated using a hash algorithm (such as SHA-256) to ensure that the data has not been tampered with. The target system hash value represents the hash value of the data in the target system and is used to compare it with the source system hash value to check for data consistency. The transmission path node is used to record the various systems and nodes that the data passes through during transmission, helping to track the flow path of the data.

[0138] Step S402: Perform data anomaly detection based on the source system hash value and the target system hash value, and output an abnormal shard list and an abnormal type mark;

[0139] The system compares the hash value of the source system with the hash value of the target system to detect any data anomalies. Anomalies include shard loss and data corruption. If the target system does not have the corresponding hash value record, it indicates data loss. If the hash values ​​of the source and target systems do not match, it indicates that the data may have been corrupted during transmission.

[0140] Step S403: Extract the original data fragments or difference fields from the source system based on the abnormal fragment list and the abnormal type tag, retransmit them to the target system through the transmission channel, and output the retry record and the updated transmission log;

[0141] When the system detects abnormal sharding, it takes appropriate remediation measures based on the type of anomaly. If a shard is lost, the system extracts the complete data shard from the source system's storage pool (such as an HBase table) and resends it to the target system via a previously determined transmission channel (such as Kafka). If data is corrupted, incremental data patches are generated based on field change records and resynchronized to the target system via the HTTP API.

[0142] Step S404: Based on the retry records and the updated transmission logs, a lineage graph including data nodes, processing nodes and abnormal nodes is constructed.

[0143] Among them, if the number of retries exceeds the threshold or a version conflict is detected, the target system data is located according to the lineage map and a rollback operation is performed, and a consistency verification report and rollback record are output.

[0144] Specifically, the system constructs a data lineage graph based on retry records and updated transmission logs. This lineage graph helps the system track data flow paths, processing procedures, and exceptions. In the lineage graph, data nodes represent the source and destination of data, and are associated with the source system table name, target system table name, and storage path. Processing nodes are used to record the encryption services, compression algorithms, transmission channels, and other processing processes and resource consumption during data transmission. Exception nodes are used to record retry events, rollback operations, and related exception types to help analyze problems in data transmission.

[0145] In addition, when the system finds that the number of retries for a data shard exceeds the threshold or detects a version conflict, it locates the data based on the lineage map and performs a rollback operation, extracts the target system storage path from the lineage map, calls the deletion interface to remove the abnormal data, and then marks the shard as "pending resynchronization" and writes it to the RabbitMQ dead letter queue for subsequent processing.

[0146] In this implementation, the system can detect data anomalies in real time and repair data loss or corruption through an automated retransmission mechanism. The construction of the lineage map provides a clear visualization of the data flow path, helping to promptly identify and resolve anomalies that occur during transmission. This significantly improves the reliability, transparency, and maintainability of data transmission, and provides reliable support for subsequent data consistency verification and rollback operations.

[0147] Reference Figure 5 As an implementation of step S107, the step of dynamically adjusting data cleaning rules and transmission parameters according to the bloodline map and the updated transmission log includes:

[0148] Step S501: constructing a data flow-performance association table based on the lineage map and the updated transmission log;

[0149] The data flow-performance association table includes the association results of shard ID, cleaning rule version, transmission channel type, and end-to-end delay;

[0150] Specifically, the system combines the processing node attributes in the lineage graph with the performance data in the updated transmission log to construct a data flow-performance correlation table. This correlation table helps the system identify the different nodes each data shard passes through during transmission, as well as the associated performance indicators. Performance indicators include end-to-end latency (transmission time), data volume, encryption algorithm, etc., reflecting the performance of the data during transmission.

[0151] Step S502 , based on the data flow-performance association table, aggregate and calculate network delay, cleaning time, and resource utilization indicators by transmission channel type and cleaning rule version, and output a performance analysis report;

[0152] The system aggregates and calculates the performance data in the data flow-performance association table to analyze the performance of different transmission channels and cleaning rule versions. The calculation indicators include network delay, cleaning time and resource utilization.

[0153] Specifically, statistics are collected for network latency calculated by transmission channel type, especially the P95 percentile latency, to measure the efficiency of network transmission. The field processing time of different cleaning rule versions is counted, and the median processing time and anomaly rate (number of failures / total number of failures) are calculated to evaluate the efficiency of the cleaning rules. In addition, resource consumption is evaluated by analyzing the complexity of the cleaning rules, the type of compression algorithm, and system resource usage (such as CPU and memory).

[0154] For example, assume that when the system transmits data through Kafka, the average network latency is 150ms and the P95 latency is 200ms. Furthermore, when using cleansing rule version A, the median field processing time is 50ms and the failure rate is 3%. The system aggregates this data to generate a performance analysis report.

[0155] Step S503: Based on the performance analysis report, a strategy parameter table is generated through a reinforcement learning model, and tuning parameters are output;

[0156] Among them, the tuning parameters include transmission channel switching instructions, compression algorithm adjustment suggestions, and cleaning rule simplification strategies;

[0157] Specifically, based on the performance analysis report, a reinforcement learning model is used to automatically generate a tuning strategy. The core of the reinforcement learning model is to define the state space, action space, and reward function.

[0158] In one embodiment of this application, the state space includes the current network latency, the target system's CPU usage, and the complexity score of the scrubbing rules. The action space defines possible optimization actions, such as switching transmission channels, simplifying scrubbing rules, or replacing compression algorithms. The reward function calculates the reward value based on the optimized end-to-end latency and data consistency. Through training a reinforcement learning model, the system automatically generates a policy table that includes tuning parameters such as transmission channel switching, compression algorithm adjustment, and scrubbing rule simplification, thereby enabling dynamic adjustment of data transmission and scrubbing processes.

[0159] Step S504: Dynamically update the data cleaning rule configuration file and the transmission policy table according to the policy parameter table, and output the updated data cleaning rules and transmission parameters;

[0160] The system dynamically updates data cleaning rules and transmission strategies based on the tuning parameters generated from the reinforcement learning model. This involves loading the updated data cleaning rule configuration file (e.g., a YAML file) through a configuration management tool (such as Nacos / Apollo) and applying the new cleaning rules and transmission parameters to the system. Furthermore, the transmission agent service process is restarted to ensure that the new compression algorithm and transmission channel configuration take effect. All updates are recorded in the version control system, ensuring that every optimization adjustment can be traced and audited.

[0161] For example, suppose that based on tuning parameters, the system decides to update the cleaning rule configuration file to use simpler field validation rules and also updates the transport channel configuration to switch to RabbitMQ. The update record will be automatically recorded in the version control system, including the rule version number, timestamp, and operator ID.

[0162] Step S505: Collect updated transmission logs and real-time performance data, verify the tuning effect and feed it back to the reinforcement learning model, and output the model training iteration record.

[0163] Specifically, the system collects updated transmission logs and real-time performance data, comparing performance changes before and after optimization to verify the effectiveness of the tuning. The system evaluates the effectiveness of the optimization strategy by calculating the latency reduction ratio and the change in anomaly rate. If the optimization strategy significantly improves performance (for example, latency reduction of more than 10% and an increase in anomaly rate of no more than 2%), the strategy is marked as effective and added to the reinforcement learning model's training dataset for further optimization. All training records, including model accuracy and feedback timestamps, are also saved for analysis.

[0164] In this implementation, combining the lineage graph with real-time transmission logs and utilizing a reinforcement learning model for dynamic optimization, we not only adjust data cleaning rules and transmission parameters based on real-time performance data, but also continuously optimize system performance through continuous feedback and iterative training. Ultimately, this technical solution significantly improves data processing efficiency, reduces latency, optimizes resource utilization, and ensures data consistency, resulting in more efficient data transmission and cleaning.

[0165] In one of the embodiments of the present application, Q-learning can be selected as the algorithm model for reinforcement learning. Q-learning is an offline reinforcement learning algorithm based on value iteration. It is suitable for solving tasks where the state and action space are relatively clear and the strategy needs to be gradually optimized. It evaluates the expected return of taking an action in a specific state by learning a value function called a Q function. The Q function represents the cumulative reward that can be obtained by performing a certain action in a certain state. By continuously updating the Q function, Q-learning can help the model find the optimal strategy, so that the intelligent agent can make the most appropriate decision in a given environment.

[0166] Specifically, in the practical application of the Q-learning algorithm, it is necessary to first define the state space and action space. The state space includes indicators such as the current network latency, the target system's CPU utilization, and the complexity of the cleaning rules. These indicators can comprehensively reflect the performance of the data transmission and cleaning process. The action space includes optimization strategies such as switching transmission channels, simplifying cleaning rules, and changing compression algorithms. The system optimizes performance by executing these actions. Q-learning evaluates the effect of each action through a reward function. The reward function may be calculated based on factors such as end-to-end latency, data consistency, and resource consumption, thereby continuously updating the Q value and guiding the system to adopt the optimal strategy.

[0167] As training progresses, Q-learning updates the Q function based on observed rewards after each execution, thereby finding the optimal decision-making strategy. Initially, the system may conduct more exploration, trying different optimization schemes. Later, the system will select actions that maximize rewards to optimize data transmission performance. In this way, the Q-learning algorithm can automatically adjust data cleaning rules and transmission parameters to achieve efficient data processing under different network conditions. Over time, the system's performance will continue to improve, ensuring long-term optimization results.

[0168] In the embodiment of the present application, standardized metadata files, processing logs, and permission token files are all stored in a distributed database and indexed by a unique shard ID, opening up a full-link tracking and governance closed loop for cross-system data flow. By connecting metadata definitions, processing records, and dynamic permission credentials in series through a unique shard ID, not only can the status of each step from data collection, cleaning, to transmission be verified (such as matching encryption rule changes with log operations), but the root cause of anomalies can be located in seconds (such as shard loss caused by permission token expiration). At the same time, relying on the high concurrent read and write capabilities of distributed storage, it can support fine-grained access control of hundreds of millions of shards in scenarios such as financial risk control and cross-border compliance.

[0169] The embodiment of the present application also discloses a cross-system data intercommunication platform.

[0170] A cross-system data interoperability platform, comprising:

[0171] A source system access module, used to access at least two heterogeneous data source systems through a protocol adapter;

[0172] The parsing and extraction module is used to parse the field names, data types, and constraints of each source system, generate standardized metadata files, and extract the original data stream. The standardized metadata files include field semantic labels, field constraint rules, and sensitivity level tags.

[0173] The data processing module is used to clean and convert the original data stream based on field semantic labels and field constraint rules, and output the cleaned structured data and processing logs;

[0174] A security data packet generation module is used to dynamically encrypt the cleansed structured data according to the sensitivity level tag and generate a security data packet queue based on the access permission attributes of the target system;

[0175] A transmission module, used to send the secure data packet queue to the target system and record the transmission log;

[0176] The lineage map generation module is used to verify the data consistency between the target system and the source system based on the transmission log and processing log, perform abnormal retransmission, update the transmission log, and generate a lineage map containing the data flow path;

[0177] The dynamic tuning module is used to dynamically adjust data cleaning rules and transmission parameters based on the bloodline map and updated transmission logs.

[0178] In one embodiment of the present application, the cross-system data intercommunication platform includes a microservice module deployed in a Kubernetes cluster. Through containerized encapsulation and dynamic orchestration capabilities, elastic scaling and high availability of resources are achieved, which can automatically respond to business traffic fluctuations and ensure 99.99% availability throughout the year; its microservice architecture supports seamless integration of heterogeneous protocols, is compatible with traditional systems and cloud native components, and combines service grids to uniformly manage communication links. In scenarios such as real-time financial risk control and manufacturing IoT data synchronization, it can automatically trace data anomalies and trigger self-healing through bloodline maps, and dynamically optimize transmission strategies to reduce resource consumption by more than 40%. It can become the core foundation for enterprises to build a secure, agile, and intelligent data center, driving the digital transformation of complex businesses such as cross-cloud disaster recovery and compliance auditing.

[0179] The cross-system data intercommunication platform of the embodiment of the present application can implement any of the above-mentioned cross-system data intercommunication methods, and the specific working process of each module in the cross-system data intercommunication platform can refer to the corresponding process in the above-mentioned method embodiment.

[0180] In the several embodiments provided in this application, it should be understood that the provided methods and systems can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for example, the division of a module is merely a logical functional division, and in actual implementation, other division methods may be used, such as combining or integrating multiple modules into another system, or ignoring or not implementing certain features.

[0181] The embodiment of the present application also discloses a computer device.

[0182] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the cross-system data intercommunication method described above is implemented.

[0183] The embodiment of the present application also discloses a computer-readable storage medium.

[0184] A computer-readable storage medium stores a computer program that can be loaded by a processor and execute any one of the above-mentioned cross-system data intercommunication methods.

[0185] Among them, computer-readable storage media can be any tangible medium that contains or stores a program that can be used by or in combination with an instruction execution system, apparatus or device; the program code contained on the computer-readable medium can be transmitted using any appropriate medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0186] It should be noted that, in the above embodiments, the description of each embodiment has different emphases. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0187] The above are all preferred embodiments of the present application and are not intended to limit the scope of protection of this application. Unless otherwise stated, any feature disclosed in this specification (including the abstract and drawings) may be replaced by other equivalent or similar features. In other words, unless otherwise stated, each feature is merely an example of a series of equivalent or similar features.

Claims

1. A cross-system data intercommunication method, characterized in that: The method comprises: Access at least two heterogeneous data source systems through protocol adapters; Parse the field names, data types, and constraints of each source system, generate standardized metadata files, and extract the original data stream; wherein the standardized metadata files include field semantic labels, field constraint rules, and sensitivity level tags; Based on the field semantic labels and field constraint rules, the original data stream is cleaned and format converted, and the cleaned structured data and processing logs are output; Dynamically encrypting the cleaned structured data according to the sensitivity level tag, and generating a secure data packet queue based on the access permission attributes of the target system; Sending the secure data packet queue to the target system and recording a transmission log; Based on the transmission log and processing log, verify the data consistency between the target system and the source system and perform abnormal retransmission, update the transmission log, and generate a lineage map containing the data flow path; The steps of dynamically adjusting data cleaning rules and transmission parameters according to the bloodline map and the updated transmission log; dynamically encrypting the cleansed structured data according to the sensitivity level tag, and generating a secure data packet queue based on the access permission attributes of the target system include: Generate a sensitive field mapping table based on the sensitivity level tags in the standardized metadata file; Dynamically assign an encryption key and encryption algorithm based on the sensitivity level tag, bind the encryption key to the shard ID in the structured data, and output a shard key table; wherein the shard ID is bound to the corresponding encryption key as a unique identifier; based on the sensitive field mapping table and the shard key table, dynamically encrypt the sensitive fields in the cleansed structured data, and output the encrypted data shard and the associated encryption algorithm identifier; wherein each data shard derives encryption parameters based on the shard ID, and different sensitive fields are encrypted using different encryption algorithms; Generate a permission policy document based on the access rights attributes of the target system, bind the shard ID to the permission policy document, and generate a permission token file using an asymmetric encryption algorithm; wherein the permission policy document contains the role, permissions, and accessible fields of the target system, and the permission token file is a digitally signed token carrying the shard ID; The encrypted data fragment, fragment ID and permission token file are encapsulated into a security data packet, and the fragment ID, encryption algorithm identifier and target system properties are written into the fragment metadata index table, and a security data packet sequence is output; wherein, each security data packet includes a header field, a payload field and a tail field, the header field includes the fragment ID and encryption algorithm identifier, the payload field includes the encrypted data ciphertext, and the tail field includes the permission token file and the authentication tag.

2. A cross-system data intercommunication method according to claim 1, characterized in that: The steps of sending the secure data packet queue to the target system and recording a transmission log include: Obtaining the service type, time sensitivity rule, and user configuration policy of the encrypted data slice, and generating a real-time tag; Based on the preset transmission strategy table, a transmission strategy is generated according to the target system attributes and real-time tags in the shard metadata index table; Obtain network status monitoring data in real time; Determining a transmission channel based on the transmission strategy and network status monitoring data; Distributing the secure data packet queue to the transmission channel; Obtain the shard ID, transmission channel type, transmission timestamp, and network time data, and write the structured transmission log according to the shard ID partition.

3. A cross-system data intercommunication method according to claim 2, characterized in that: The steps of verifying data consistency between the target system and the source system based on the transmission log and the processing log, performing abnormal retransmission, updating the transmission log, and generating a lineage map including a data flow path include: Based on the shard ID in the transmission log and the field change record in the processing log, a shard-level data snapshot table is generated by association; wherein the shard-level data snapshot table includes the association result of the source system hash value, the target system hash value and the transmission path node; Perform data anomaly detection based on the source system hash value and the target system hash value, and output an abnormal shard list and an abnormal type mark; Extracting original data segments or difference fields from the source system according to the abnormal segment list and the abnormal type mark, retransmitting them to the target system through the transmission channel, and outputting retry records and updated transmission logs; Based on the retry records and the updated transmission logs, a lineage graph including data nodes, processing nodes and abnormal nodes is constructed.

4. A cross-system data intercommunication method according to claim 3, characterized in that: The steps of dynamically adjusting data cleaning rules and transmission parameters according to the bloodline map and the updated transmission log include: Based on the bloodline map and the updated transmission log, a data flow-performance association table is constructed; According to the data flow-performance association table, network delay, cleaning time and resource utilization indicators are aggregated and calculated by transmission channel type and cleaning rule version, and a performance analysis report is output; Based on the performance analysis report, a strategy parameter table is generated through a reinforcement learning model, and tuning parameters are output; the tuning parameters include transmission channel switching instructions, compression algorithm adjustment suggestions, and cleaning rule simplification strategies; Dynamically update the data cleaning rule configuration file and the transmission strategy table according to the strategy parameter table, and output the updated data cleaning rules and transmission parameters; Collect updated transmission logs and real-time performance data, verify the tuning effect and feed it back to the reinforcement learning model, and output the model training iteration record.

5. A cross-system data intercommunication method according to any one of claims 1 to 4, characterized in that: The standardized metadata files, processing logs, and permission token files are all stored in a distributed database and are associated and indexed by a unique shard ID.

6. A cross-system data intercommunication platform, characterized by: The platform is used to execute the cross-system data intercommunication method according to any one of claims 1 to 5, comprising: A source system access module, used to access at least two heterogeneous data source systems through a protocol adapter; A parsing and extraction module is used to parse the field names, data types, and constraints of each source system, generate standardized metadata files, and extract the original data stream; wherein the standardized metadata files include field semantic labels, field constraint rules, and sensitivity level tags; A data processing module is used to perform data cleaning and format conversion on the original data stream based on the field semantic labels and field constraint rules, and output the cleaned structured data and processing log; a security data packet generation module, configured to dynamically encrypt the cleansed structured data according to the sensitivity level tag, and generate a security data packet queue based on the access permission attributes of the target system; A transmission module, configured to send the secure data packet queue to the target system and record a transmission log; A lineage map generation module is used to verify the data consistency between the target system and the source system based on the transmission log and the processing log, perform abnormal retransmission, update the transmission log, and generate a lineage map containing the data flow path; The dynamic tuning module is used to dynamically adjust data cleaning rules and transmission parameters according to the blood relationship map and the updated transmission log.

7. A cross-system data intercommunication platform according to claim 6, characterized in that: The cross-system data intercommunication platform includes a microservice module deployed in a Kubernetes cluster.

8. A computer device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 5 when executing the program.

9. A computer-readable storage medium, characterized in that: A computer program is stored which can be loaded by a processor and execute the method according to any one of claims 1 to 5.

Images