Cross-system data intercommunication method, platform, equipment and medium

Abstract

The invention relates to a cross-system data intercommunication method, platform, device and medium, and belongs to the field of data transmission, the method comprises the following steps: accessing at least two heterogeneous data source systems through a protocol adapter; analyzing a field name, a data type and a constraint condition of each source system, generating a standardized metadata file, and extracting an original data stream; performing data cleaning and format conversion on the original data stream, and outputting cleaned structured data and processing logs; performing dynamic encryption according to the sensitive level mark, generating a security data packet queue based on the access permission attribute of the target system, sending the security data packet queue to the target system, and recording a transmission log; verifying the data consistency of the target system and the source system, carrying out abnormal retransmission, updating a transmission log, and generating a blood relationship map containing a data flow path; and dynamically adjusting a data cleaning rule and a transmission parameter according to the blood relationship map and the updated transmission log. The stability and efficiency of data intercommunication can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of data transmission, and in particular to a cross-system data intercommunication method, platform, device and medium. Background Art

[0002] With the rapid development of information technology and the surge in global data volume, more and more enterprises, government departments and organizations rely on information systems to process and manage massive amounts of data. The application of technologies such as big data, cloud computing, and the Internet of Things has made the interaction between different business systems and data sources an important area in modern information technology. Therefore, the problem of cross-system data interoperability is particularly important, especially in terms of interoperability, data security, and consistency management between multiple heterogeneous systems.

[0003] At present, there are a large number of heterogeneous data sources in the IT environment of many enterprises and institutions, including relational databases, non-relational databases, file systems, external API interfaces, IoT devices, etc. There are significant differences in the formats, storage methods, data types and constraints between these data sources. However, common cross-system data interoperability methods usually rely on customized solutions, which are difficult to cope with the growing demand for cross-platform data exchange, especially in large-scale data transmission. It cannot adapt to the network conditions and complex scenarios between different systems, thereby reducing the stability and efficiency of data interoperability. Summary of the invention

[0004] In order to improve the stability and efficiency of data intercommunication, the present application provides a cross-system data intercommunication method, platform, device and medium.

[0005] In the first aspect, the present application provides a cross-system data intercommunication method, which adopts the following technical solution: A cross-system data intercommunication method, the method comprising: Access at least two heterogeneous data source systems through a protocol adapter; Parse the field name, data type and constraint conditions of each source system, generate a standardized metadata file, and extract the original data stream; wherein the standardized metadata file includes field semantic labels, field constraint rules and sensitivity level tags; Based on the field semantic labels and field constraint rules, the original data stream is cleaned and format converted, and the cleaned structured data and processing logs are output; Dynamically encrypting the cleaned structured data according to the sensitivity level tag, and generating a secure data packet queue based on the access permission attributes of the target system; Sending the secure data packet queue to the target system and recording a transmission log; Based on the transmission log and the processing log, verify the data consistency between the target system and the source system and perform abnormal retransmission, update the transmission log, and generate a lineage map containing the data flow path; According to the bloodline map and the updated transmission log, data cleaning rules and transmission parameters are dynamically adjusted.

[0006] By adopting the above technical solutions, the efficiency, security and consistency of data transmission and processing from data source access to target system are ensured. Protocol adapters are used to achieve access to heterogeneous data sources, dynamic encryption and security tokens are used to ensure data security, intelligent transmission and data consistency verification are used to ensure data integrity, and the overall performance and adaptability of the system are improved through dynamic adjustment mechanisms. Finally, cross-system data interoperability is achieved, which is suitable for various multi-system collaboration application scenarios such as cross-departmental data sharing, big data analysis and IoT data synchronization, and effectively solves industry pain points such as data silos, low transmission efficiency, and high security risks.

[0007] Optionally, the step of dynamically encrypting the cleaned structured data according to the sensitivity level tag and generating a secure data packet queue based on the access permission attribute of the target system includes: Generate a sensitive field mapping table based on the sensitivity level tags in the standardized metadata file; Dynamically assign encryption keys and encryption algorithms according to the sensitivity level labels, bind the encryption keys to the shard IDs in the structured data, and output a shard key table; Based on the sensitive field mapping table and the sharding key table, dynamically encrypt the sensitive fields in the cleaned structured data, and output the encrypted data shards and the associated encryption algorithm identifiers; Generate a permission policy document based on the access permission attributes of the target system, bind the shard ID to the permission policy document, and generate a permission token file according to an asymmetric encryption algorithm; The encrypted data shard, shard ID and permission token file are encapsulated into a secure data packet, and the shard ID, encryption algorithm identifier and target system attributes are written into the shard metadata index table, and a secure data packet sequence is output.

[0008] By adopting the above technical solutions, based on dynamic encryption and secure data packet generation mechanism, the encryption, access control and secure transmission of sensitive data are ensured during cross-system data communication. By combining the sensitive field mapping table with the sharding key table, data can be encrypted using different encryption algorithms according to its sensitivity, ensuring the confidentiality of the data. The permission policy document and digital signature generation mechanism ensure that only authorized systems can access sensitive data, thereby improving the security of the system. Finally, through the sharding metadata index table and protocol encapsulation, the reliability and consistency of data packet transmission are ensured, which can effectively cope with the data exchange needs of large-scale heterogeneous systems.

[0009] Optionally, the step of sending the secure data packet queue to the target system and recording a transmission log includes: Obtaining the service type, time sensitivity rule and user configuration policy of the encrypted data slice, and generating a real-time tag; Based on the preset transmission strategy table, a transmission strategy is generated according to the target system attributes and real-time tags in the shard metadata index table; Obtain network status monitoring data in real time; Determine a transmission channel according to the transmission strategy and network status monitoring data; distributing the secure data packet queue to the transmission channel; Get the shard ID, transmission channel type, transmission timestamp and network time consumption data, and write the structured transmission log according to the shard ID partition.

[0010] By adopting the above technical solutions, dynamic selection of transmission strategies and channel types, real-time monitoring of network status, and recording of detailed transmission logs, efficient and reliable data packet transmission and logging are achieved. The use of real-time tags ensures the priority transmission of different types of data, while network status monitoring and dynamic adjustment strategies optimize transmission efficiency and ensure data integrity. Through structured transmission logs, the system can provide important basis for subsequent troubleshooting, performance analysis, and optimization.

[0011] Optionally, based on the transmission log and the processing log, verifying the data consistency between the target system and the source system and performing abnormal retransmission, updating the transmission log, and generating a lineage map including a data flow path includes: Based on the shard ID in the transmission log and the field change record in the processing log, a shard-level data snapshot table is generated by association; wherein the shard-level data snapshot table includes the association result of the source system hash value, the target system hash value and the transmission path node; Perform data anomaly detection based on the source system hash value and the target system hash value, and output an abnormal shard list and an abnormal type mark; Extracting original data slices or difference fields from the source system according to the abnormal slice list and the abnormal type mark, retransmitting them to the target system through the transmission channel, and outputting retry records and updated transmission logs; Based on the retry records and the updated transmission logs, a bloodline graph including data nodes, processing nodes and abnormal nodes is constructed.

[0012] By adopting the above technical solutions, the system can detect data anomalies in real time and repair data loss or damage problems through an automated retransmission mechanism. The construction of the bloodline map provides a clear visualization of the data flow path, helping to promptly discover and resolve anomalies that occur during the transmission process, significantly improving the reliability, transparency and maintainability of data transmission, and providing reliable support for subsequent data consistency verification and rollback operations.

[0013] Optionally, the step of dynamically adjusting data cleaning rules and transmission parameters according to the bloodline map and the updated transmission log includes: Based on the bloodline map and the updated transmission log, construct a data flow-performance association table; According to the data flow-performance association table, network delay, cleaning time and resource utilization indicators are aggregated and calculated according to the transmission channel type and cleaning rule version, and a performance analysis report is output; Based on the performance analysis report, a strategy parameter table is generated through a reinforcement learning model to output tuning parameters; the tuning parameters include transmission channel switching instructions, compression algorithm adjustment suggestions, and cleaning rule simplification strategies; According to the policy parameter table, dynamically update the data cleaning rule configuration file and the transmission policy table, and output the updated data cleaning rules and transmission parameters; Collect updated transmission logs and real-time performance data, verify the tuning effect and feed it back to the reinforcement learning model, and output the model training iteration record.

[0014] By adopting the above technical solutions, combining bloodline graphs with real-time transmission logs, and using reinforcement learning models for dynamic tuning, we can not only adjust data cleaning rules and transmission parameters based on real-time performance data, but also continuously optimize system performance through continuous feedback and iterative training. Ultimately, this technical solution can significantly improve data processing efficiency, reduce latency, optimize resource utilization, and ensure data consistency, thereby achieving a more efficient data transmission and cleaning process.

[0015] Optionally, the standardized metadata file, processing log, and permission token file are all stored in a distributed database and are associated and indexed through a unique shard ID.

[0016] In the second aspect, the present application provides a cross-system data intercommunication platform, which adopts the following technical solutions: A cross-system data intercommunication platform, comprising: A source system access module, used to access at least two heterogeneous data source systems through a protocol adapter; A parsing and extraction module, which is used to parse the field name, data type and constraint conditions of each source system, generate a standardized metadata file, and extract the original data stream; wherein the standardized metadata file includes field semantic labels, field constraint rules and sensitivity level tags; A data processing module, used to perform data cleaning and format conversion on the original data stream based on the field semantic labels and field constraint rules, and output the cleaned structured data and processing logs; A security data packet generation module, used to dynamically encrypt the cleansed structured data according to the sensitivity level tag, and generate a security data packet queue based on the access permission attributes of the target system; A transmission module, used for sending the secure data packet queue to the target system and recording a transmission log; A lineage map generation module is used to verify the data consistency between the target system and the source system and perform abnormal retransmission based on the transmission log and the processing log, update the transmission log, and generate a lineage map containing the data flow path; The dynamic tuning module is used to dynamically adjust data cleaning rules and transmission parameters according to the blood relationship map and the updated transmission log.

[0017] Optionally, the cross-system data intercommunication platform includes a microservice module deployed in a Kubernetes cluster.

[0018] In a third aspect, the present application provides a computer device, which adopts the following technical solution: A computer device comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method as described in the first aspect.

[0019] In a fourth aspect, the present application provides a computer-readable storage medium, which adopts the following technical solution: A computer-readable storage medium stores a computer program that can be loaded by a processor and execute any one of the methods in the first aspect.

[0020] In summary, the present application includes at least one of the following beneficial technical effects: through the combination of bloodline map and transmission log, the system can verify data consistency in real time and perform abnormal retransmission, dynamically adjust data cleaning rules and transmission parameters according to feedback to ensure the accuracy and security of data flow, and realize efficient and secure data exchange between different heterogeneous data sources. This technical solution not only improves the efficiency and accuracy of data transmission, but also strengthens data security across systems. It is suitable for complex data processing environments and has important practical application significance. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is the first flow chart of the cross-system data intercommunication method of one of the embodiments of the present application.

[0022] Figure 2 This is the second flow chart of the cross-system data intercommunication method of one of the embodiments of the present application.

[0023] Figure 3 This is the third flow chart of the cross-system data intercommunication method of one of the embodiments of the present application.

[0024] Figure 4 This is the fourth flow chart of the cross-system data intercommunication method of one of the embodiments of the present application.

[0025] Figure 5 This is the fifth flow chart of the cross-system data intercommunication method of one of the embodiments of the present application. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solutions and advantages of this application more clear, the following Figure 1-5 It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0027] The embodiment of the present application discloses a cross-system data intercommunication method.

[0028] Reference Figure 1 , a cross-system data intercommunication method, the method comprising: Step S101, accessing at least two heterogeneous data source systems through a protocol adapter; Among them, by presetting multiple types of protocol adapters, the interface types of heterogeneous data sources are dynamically identified and standardized communication links are established. Heterogeneous data sources mean that these data sources have different formats, structures, storage methods, etc., so protocol adapters (such as JDBC, HTTP, MQTT, etc.) are needed to achieve cross-platform or cross-system data access. The protocol adapter acts as a bridge between different systems, ensuring that different data sources can be accessed in a consistent manner.

[0029] Specifically, during the implementation process, the protocol adapter first needs to identify the data source type and select the appropriate connection method. For example, if the data source is a relational database (such as MySQL), the JDBC protocol is used for connection; if it is real-time data from an IoT device, the MQTT protocol may be used.

[0030] Step S102, parsing the field name, data type and constraint conditions of each source system, generating a standardized metadata file, and extracting the original data stream; Among them, the standardized metadata file includes field semantic labels, field constraint rules and sensitivity level tags; Specifically, the generated standardized metadata file can unify the structure, data type and constraints of various data sources, providing consistent specifications for subsequent data processing. At the same time, the original data stream is retained for subsequent processing to ensure data integrity.

[0031] For example, Schema parsing tools (such as Apache Calcite) can be used to extract original metadata such as field names, types, and primary key constraints, and mapped to semantic tags that can be understood across systems through a unified semantic model (such as JSON-LD). For example, source system A is a database with fields containing "user ID" and "name", and source system B is a CSV file with fields containing "customer ID" and "age". The system automatically parses out the respective fields and normalizes them into a standard format.

[0032] Step S103, based on the field semantic labels and field constraint rules, the original data stream is cleaned and format converted, and the cleaned structured data and processing log are output; The field semantic labels and field constraint rules (such as non-null and value range) defined in the standardized metadata file are used to call the rule engine to clean the original data stream and select the conversion logic based on the semantic labels. The purpose of data cleaning is to delete erroneous, incomplete, duplicate or non-constrained data, while format conversion is to convert the data into a format acceptable to the target system.

[0033] Specifically, data cleaning is performed according to field constraint rules. For example, if the field type is "integer" but a non-integer value appears in the data, the system will identify and delete or repair the value. Format conversion can be achieved through programming, such as converting CSV format to JSON format, or converting XML data to JSON data through API calls. The system performs cleaning and conversion operations based on pre-set rules to ensure data accuracy and consistency.

[0034] For example, suppose the data field "age" is required to be an integer, but there are non-numeric values ​​in the source data (such as "age unknown"), in which case the cleaning rule will identify and repair it to the default value or delete the record.

[0035] Step S104, dynamically encrypting the cleaned structured data according to the sensitivity level tag, and generating a secure data packet queue based on the access permission attributes of the target system; The security data packet sequence carries the shard ID and permission token. To ensure the security of data during transmission, sensitive data (such as ID card number, mobile phone number, etc.) needs to be encrypted or desensitized. At the same time, the generated security data packet also needs to generate a permission token based on the access rights of the target system.

[0036] Specifically, according to the sensitivity level mark in the metadata (such as PII, financial data), the encryption algorithm is dynamically selected (AES-GCM is used for ordinary sensitive data, and the national encryption SM4 is used for high security requirements). Encryption algorithms such as AES-GCM can be used to encrypt data to ensure that the data will not be leaked during transmission. By generating a JWT token (JSON Web Token), it can be ensured that the access control of the data packet matches the permission attributes of the target system. Each security data packet contains a shard ID and a corresponding permission token, which is used to verify data access rights in the target system.

[0037] It is understood that encryption ensures that sensitive data will not be leaked during transmission, while the permission binding mechanism ensures that only authorized target systems or users can access the data. Security tokens and shard IDs are used to further ensure the security and integrity of data transmission.

[0038] For example, when the data contains an ID number field, the system will encrypt it based on the AES-GCM algorithm and generate a JWT token containing access permissions, such as "Shard ID=12345, Permission=Read-only".

[0039] Step S105, sending the secure data packet queue to the target system and recording the transmission log; The transmission log includes shard ID, network time consumption data, transmission timestamp, channel type, etc. The system can select the appropriate transmission mode to send the secure data packet queue to the target system according to the data type, transmission timeliness and network conditions, such as real-time transmission and batch transmission. Real-time transmission is suitable for time-sensitive data, and batch transmission is suitable for large amounts of data or historical data.

[0040] Specifically, real-time data transmission can be achieved through Kafka streaming to ensure low latency and high throughput; batch data can be transmitted asynchronously using HDFS or similar distributed file systems after being sharded and compressed. The network conditions during the transmission process (such as network latency, bandwidth, etc.) will affect the selected transmission method and path, and the system optimizes the transmission path based on the real-time network status. For example, for real-time monitoring data, the system transmits through Kafka; for historical data, the system may use the DistCp tool to transfer data in batches to HDFS.

[0041] It is understandable that by selecting a suitable transmission channel and optimizing the transmission path, the efficiency and stability of data transmission can be ensured. The recorded transmission log provides a basis for subsequent error handling and performance optimization.

[0042] Step S106, based on the transmission log and the processing log, verify the data consistency between the target system and the source system and perform abnormal retransmission, update the transmission log, and generate a lineage map containing the data flow path; During the data transmission process, transmission failure or data corruption may occur, so data consistency verification is required. The data hash values ​​of the source system and the target system are compared by using the shard ID in the transmission log and the field change records in the processing log to ensure data consistency. If an abnormality is found, an automatic retry mechanism or data rollback operation is triggered.

[0043] Specifically, the hash value of the data packet is compared through CRC32 check to ensure that the data has not been tampered with during transmission. If the transmission fragment is found to be lost or the hash value is inconsistent, the system will locate the abnormal data according to the transmission log and retransmit the lost fragment; if the retry limit is exceeded, the target system will delete the damaged data in the target system through a rollback operation and mark the abnormal state.

[0044] It can be understood that the data consistency check and error recovery mechanism ensures that the data of the target system is consistent with the source system, avoiding incomplete or lost data due to transmission errors.

[0045] Step S107, dynamically adjust data cleaning rules and transmission parameters according to the bloodline map and the updated transmission log.

[0046] Among them, by monitoring the network time consumption data in the bloodline map and transmission log, dynamically adjusting the data cleaning rules and transmission parameters, and analyzing the historical data, the system can optimize the parallelism and transmission strategy of the ETL task and improve the system efficiency.

[0047] Specifically, the reinforcement learning model can be used to predict the best strategy for data cleaning and transmission, and automatically adjust parameters based on historical data flow and network conditions; for example, if the network delay is high during a certain period of time, the system may automatically adjust the data cleaning rules to reduce the complexity of data processing, or adjust the transmission parameters to avoid network bottlenecks. By dynamically adjusting the strategy, the system can self-optimize, thereby improving processing efficiency and transmission quality.

[0048] In the above implementation, the efficiency, security and consistency of data transmission and processing from data source access to target system are ensured. Protocol adapters are used to achieve access to heterogeneous data sources, dynamic encryption and security tokens are used to ensure data security, intelligent transmission and data consistency verification are used to ensure data integrity, and the overall performance and adaptability of the system are improved through dynamic adjustment mechanisms. Finally, cross-system data interoperability is achieved, which is suitable for various cross-departmental data sharing, big data analysis, IoT data synchronization and other multi-system collaboration application scenarios, effectively solving industry pain points such as data silos, low transmission efficiency, and high security risks.

[0049] Reference Figure 2 As an implementation method of step S104, the steps of dynamically encrypting the cleaned structured data according to the sensitivity level tag and generating a secure data packet queue based on the access permission attributes of the target system include: Step S201, generating a sensitive field mapping table based on the sensitivity level mark in the standardized metadata file; In the process of cross-system data intercommunication, the cleaned structured data must first be marked with sensitivity levels. The first step in this process is to generate a sensitive field mapping table based on the sensitivity level markings in the standardized metadata file. The sensitive field mapping table can accurately identify which data fields are sensitive data, and then take corresponding encryption protection measures, providing a basis for the subsequent dynamic encryption process.

[0050] For example, suppose there is a field "ID number", and the sensitivity level of this field in the metadata file is marked as PII (personal identity information). In the generated sensitive field mapping table, this field will be associated with its sensitivity level PII, so that the encryption requirement of this field can be identified in the subsequent processing process.

[0051] Step S202, dynamically assigning encryption keys and encryption algorithms according to the sensitivity level tags, binding the encryption keys to the shard IDs in the structured data, and outputting the shard key table; The shard key table includes the shard ID, encryption key, and encryption algorithm type. The shard ID is generated after data cleaning and serves as the core identifier for all subsequent steps. The shard ID is used to derive encryption parameters. Specifically, the system will dynamically select encryption algorithms based on the different levels of sensitive data (such as PII, financial data), and the key management system (KMS) will provide keys and generate corresponding keys based on the selected encryption algorithm. The system will bind the key to the shard ID so that each data shard has a unique key.

[0052] In the embodiment of the present application, sensitive data tags are mapped to encryption algorithms. For example, PII data is encrypted using the AES-GCM algorithm, and financial data is encrypted using the national secret SM4 algorithm. In addition, the shard ID is bound to the key. The shard ID is bound to the corresponding encryption key as a unique identifier. Each data shard will have a different encryption key to ensure data security.

[0053] Step S203: Based on the sensitive field mapping table and the sharding key table, the sensitive fields in the cleaned structured data are dynamically encrypted, and the encrypted data shards and the associated encryption algorithm identifiers are output; Among them, according to the encryption algorithm and key in the shard key table, the system dynamically encrypts the sensitive fields in the data, and each data shard derives encryption parameters according to its shard ID. Different sensitive fields are encrypted using different encryption algorithms (such as AES-GCM, SM4) to ensure the security of sensitive data during transmission.

[0054] Specifically, during the encryption process, encryption parameters are derived based on the shard ID. For example, the AES-GCM algorithm uses the hash value of the shard ID as the additional authentication data (AAD), and the SM4 algorithm uses the shard ID and the target system ID to generate the initialization vector (IV). Each encrypted shard contains the encrypted ciphertext and the encryption algorithm identifier.

[0055] For example, assume that the ID card number is encrypted using AES-GCM, and the shard ID is "12345". The system generates an additional authentication data (AAD) based on the hash value of the shard ID. The encrypted data shard is output as ciphertext and is marked as encrypted using AES-GCM.

[0056] Step S204, generating a permission policy document based on the access permission attributes of the target system, binding the shard ID to the permission policy document, and generating a permission token file according to an asymmetric encryption algorithm; Among them, the permission policy document contains information such as the role, permissions, and accessible fields of the target system. The permission token file is a digital signature token that carries the shard ID, ensuring that the target system can access data only when it has the correct permissions.

[0057] Specifically, before data transmission, the system generates a permission policy document based on the access rights attributes of the target system (such as role, IP address, accessible field list, etc.). This document binds the shard ID with the access rights of the target system, and digitally signs the document using an asymmetric encryption algorithm (such as RSA-PSS) to generate a permission token file.

[0058] Step S205, encapsulate the encrypted data shard, shard ID and permission token file into a secure data packet, write the shard ID, encryption algorithm identifier and target system attributes into the shard metadata index table, and output the secure data packet sequence.

[0059] The system encapsulates the encrypted data fragments, fragment IDs and permission token files into secure data packets according to the protocol format. Each data packet contains a header field, a payload field and a tail field. Specifically, the header field contains the fragment ID and encryption algorithm identifier, the payload field contains the encrypted data ciphertext, and the tail field contains the permission token and authentication tag (AES-GCM authentication tag or SM4 IV value).

[0060] In addition, after generating the security data packet, the system will write the shard ID, encryption algorithm identifier and target system properties into the shard metadata index table, which is used for subsequent transmission channel selection and data consistency verification.

[0061] In the above implementation, based on dynamic encryption and secure data packet generation mechanism, encryption, access control and secure transmission of sensitive data are ensured during cross-system data intercommunication. By combining the sensitive field mapping table with the sharding key table, data can be encrypted using different encryption algorithms according to its sensitivity, ensuring the confidentiality of the data. The generation mechanism of permission policy documents and digital signatures ensures that only authorized systems can access sensitive data, thereby improving the security of the system. Finally, through the sharding metadata index table and protocol encapsulation, the reliability and consistency of data packet transmission are ensured, which can effectively cope with the data exchange requirements of large-scale heterogeneous systems.

[0062] Reference Figure 3 As an implementation of step S105, the step of sending the secure data packet queue to the target system and recording the transmission log includes: Step S301, obtaining the service type, time sensitivity rule and user configuration policy of the encrypted data slice, and generating a real-time tag; Before data packets are transmitted, the encrypted data fragments need to be analyzed first. The analysis is based on the service type, time sensitivity rules and user configuration policies of the encrypted data fragments to generate corresponding real-time tags. The real-time tags will determine the transmission priority of the data packets and guide the system to select the appropriate transmission channel.

[0063] Specifically, the time sensitivity of the data is determined by dividing the business types according to business needs, such as financial transactions, real-time monitoring, etc. For example, financial transaction data may require a higher real-time label, while log data has lower real-time requirements. The time sensitivity rule is determined based on the processing timeliness of the data and defines how quickly the data packet needs to be transmitted to the target system; for example, real-time data may require a high-priority label, while non-real-time data can use a normal label. In addition, the real-time label can be further adjusted according to the user's configuration policy. For example, some systems may require special processing of real-time during data packet transmission.

[0064] For example, if a data packet is of the “financial transaction” type and has a “high priority” time sensitivity, the system will generate a real-time tag with a “high priority” tag to ensure that the data packet is transmitted first.

[0065] Step S302, based on the preset transmission strategy table, generating a transmission strategy according to the target system attributes and real-time tags in the shard metadata index table; The transmission strategy includes candidate channel type, compression algorithm and retry times. After generating the real-time tag, the system will generate a suitable transmission strategy based on the preset transmission strategy table, the interface protocol of the target system, the real-time tag and other target system attributes.

[0066] Specifically, candidate channel types include real-time streaming channels or batch transmission channels, and the compression algorithm is selected according to the data type or the requirements of the transmission channel, such as Zstandard or Brotli; when the transmission fails, the system will set the number of retries to ensure that the data packet can be transmitted reliably.

[0067] Step S303, acquiring network status monitoring data in real time; Among them, network status monitoring data includes network delay, bandwidth occupancy rate and packet loss rate between the source system and the target system. Specifically, network delay refers to the delay from the source system to the target system, which is used to evaluate the real-time performance of data transmission; bandwidth occupancy rate refers to the current network bandwidth occupancy, which ensures that the transmission channel will not be affected by bandwidth bottlenecks and data transmission; packet loss rate refers to the rate of packet loss during data transmission. Too high a packet loss rate will affect the integrity and accuracy of the data.

[0068] Step S304, determining a transmission channel according to the transmission strategy and network status monitoring data; Among them, the transmission channel includes a real-time streaming channel or a batch transmission channel. The real-time streaming channel is suitable for high-priority data (such as financial data), and the batch transmission channel is suitable for non-real-time data or large-scale data.

[0069] In some embodiments, when the system detects that the real-time label is "high priority" and the network status is good, the real-time stream channel is selected, and the Kafka transaction producer transmission can be used; when the real-time requirements of data transmission are low, or the network status is not good, the batch transmission channel is selected, and the Zstandard compression algorithm can be used for sharding, and then asynchronous transmission is performed through the DistCp tool.

[0070] Step S305, distributing the secure data packet queue to the transmission channel; Once the transmission channel is determined, the system will distribute the encrypted secure data packet queue according to the selected channel type. If Kafka is selected as the transmission channel, the data packet will be sent through the Kafka producer and a transaction identifier will be generated; if HDFS is selected, the data packet will be compressed by Zstandard and then written to the HDFS directory.

[0071] Step S306, obtain the shard ID, transmission channel type, transmission timestamp and network time consumption data, and write the structured transmission log according to the shard ID partition.

[0072] Among them, the system will collect key information of each data packet. The fragment ID is used to identify the uniqueness of the data packet; the transmission channel type includes real-time streaming channel or batch channel, the transmission timestamp is used to record the transmission time of the data packet, and the network time-consuming data includes end-to-end transmission delay.

[0073] For example, the fragment ID of the data packet is "12345", the transmission time is "2024-08-13 12:30:00 UTC", the channel type is Kafka, and the network time is 150ms. This information will be stored in the transmission log.

[0074] In the above implementation, dynamic selection of transmission strategy and channel type, real-time monitoring of network status and recording of detailed transmission logs achieve efficient and reliable data packet transmission and logging. The use of real-time tags ensures the priority transmission of different types of data, while network status monitoring and dynamic adjustment strategies optimize transmission efficiency and ensure data integrity. Through structured transmission logs, the system can provide important basis for subsequent troubleshooting, performance analysis and optimization.

[0075] Reference Figure 4 As an implementation method of step S106, based on the transmission log and the processing log, the data consistency between the target system and the source system is verified and abnormal retransmission is performed, the transmission log is updated, and the steps of generating a lineage map including the data flow path include: Step S401, based on the shard ID in the transmission log and the field change record in the processing log, a shard-level data snapshot table is generated by association; Among them, the shard-level data snapshot table includes the association results of the source system hash value, the target system hash value and the transmission path node; Specifically, the shard-level data snapshot table is a structured data table that records the key status information of a data shard at a certain point in time during the cross-system transmission process. By associating the transmission log and the processing log, an "instantaneous state mirror" of the data shard in the source system and the target system is formed, which is the core basis for data consistency verification and lineage tracing.

[0076] In the embodiment of the present application, the source system hash value represents the unique fingerprint of the data in the source system, which is calculated by a hash algorithm (such as SHA-256) to ensure that the data has not been tampered with. The target system hash value represents the hash value of the data in the target system, which is used to compare with the source system hash value to detect whether the data is consistent. The transmission path node is used to record the various systems and nodes that the data passes through during the transmission process, helping to track the flow path of the data.

[0077] Step S402, performing data anomaly detection according to the source system hash value and the target system hash value, and outputting an abnormal shard list and an abnormal type mark; The system compares the hash value of the source system with the hash value of the target system to detect whether the data is abnormal. Abnormal types include shard loss and data corruption. If the target system does not have the corresponding hash value record, it indicates that the data is lost; if the hash values ​​of the source system and the target system are inconsistent, it indicates that the data may be damaged during transmission.

[0078] Step S403: extract the original data slices or difference fields from the source system according to the abnormal slice list and the abnormal type mark, retransmit them to the target system through the transmission channel, and output the retry record and the updated transmission log; When the system detects abnormal shards, it will take appropriate repair measures according to the type of abnormality. When a shard is lost, the system will extract the complete data shard from the storage pool of the source system (such as HBase table) and resend it to the target system through the previously determined transmission channel (such as Kafka); when data is damaged, it will generate incremental data patches based on the field change records and resynchronize them to the target system through the HTTP API.

[0079] Step S404, based on the retry records and the updated transmission logs, construct a bloodline graph including data nodes, processing nodes and abnormal nodes.

[0080] Among them, if the number of retries exceeds the threshold or a version conflict is detected, the target system data is located according to the lineage map and a rollback operation is performed, and a consistency verification report and rollback record are output.

[0081] Specifically, the system builds a data lineage map based on retry records and updated transmission logs. The lineage map can help the system track the data flow path, processing process and abnormal situations. In the lineage map, the data node represents the source and target of the data, and associates the source system table name, target system table name and storage path. The processing node is used to record the encryption service, compression algorithm, transmission channel and other processing processes and resource consumption during the data transmission process. The exception node is used to record retry events, rollback operations and related exception types to help analyze problems in data transmission.

[0082] In addition, when the system finds that the number of retries for a data shard exceeds the threshold or detects a version conflict, it locates the data based on the lineage map and performs a rollback operation, extracts the target system storage path from the lineage map, calls the deletion interface to remove the abnormal data, and then marks the shard as "pending resynchronization" and writes it to the RabbitMQ dead letter queue for subsequent processing.

[0083] In the above implementation, the system can detect data anomalies in real time and repair data loss or damage problems through an automated retransmission mechanism. The construction of the bloodline map provides a clear visualization of the data flow path, helping to promptly discover and resolve anomalies that occur during the transmission process, significantly improving the reliability, transparency and maintainability of data transmission, and providing reliable support for subsequent data consistency verification and rollback operations.

[0084] Reference Figure 5 As an implementation method of step S107, the step of dynamically adjusting data cleaning rules and transmission parameters according to the blood relationship map and the updated transmission log includes: Step S501, constructing a data flow-performance association table based on the bloodline map and the updated transmission log; Among them, the data flow-performance association table includes the association results of the shard ID, the cleaning rule version, the transmission channel type, and the end-to-end delay; Specifically, the system combines the processing node attributes in the lineage graph with the performance data in the updated transmission log to build a data flow-performance association table, which will help the system identify the different nodes that each data shard has experienced during the transmission process, as well as the related performance indicators. Performance indicators include end-to-end latency (transmission time), data volume, encryption algorithm, etc., which reflect the performance of data during transmission.

[0085] Step S502, according to the data flow-performance association table, aggregate and calculate network delay, cleaning time and resource utilization indicators according to the transmission channel type and cleaning rule version, and output a performance analysis report; The system aggregates and calculates the performance data in the data flow-performance association table to analyze the performance of different transmission channels and cleaning rule versions. The calculation indicators include network delay, cleaning time and resource utilization.

[0086] Specifically, statistics are collected on network delays calculated by transmission channel type, especially P95 percentile delays, to measure the efficiency of network transmission. Statistics are collected on field processing times of different cleaning rule versions, and the median time and anomaly rate (number of failures / total number of failures) are calculated to evaluate the efficiency of cleaning rules. In addition, resource consumption is evaluated by analyzing the complexity of cleaning rules, compression algorithm type, and system resource usage (such as CPU, memory, etc.).

[0087] For example, assume that when the system transmits data through Kafka, the average network delay is 150ms and the P95 delay is 200ms; at the same time, when using cleaning rule version A, the median time spent on field processing is 50ms and the failure rate is 3%. The system aggregates this data to form a performance analysis report.

[0088] Step S503, based on the performance analysis report, generate a strategy parameter table through a reinforcement learning model and output tuning parameters; Among them, the tuning parameters include transmission channel switching instructions, compression algorithm adjustment suggestions, and cleaning rule simplification strategies; Specifically, based on the performance analysis report, a reinforcement learning model is used to automatically generate a tuning strategy. The core of the reinforcement learning model is to define the state space, action space, and reward function.

[0089] In one embodiment of the present application, the state space includes the current network delay, the CPU usage of the target system, and the complexity score of the cleaning rules; the action space defines possible optimization actions, such as switching transmission channels, simplifying cleaning rules, or replacing compression algorithms; the reward function calculates the reward value based on the tuned end-to-end delay and data consistency. Through the training of the reinforcement learning model, the system automatically generates a policy table including tuning parameters such as transmission channel switching, compression algorithm adjustment, and cleaning rule simplification, thereby realizing dynamic adjustment of data transmission and cleaning processes.

[0090] Step S504, dynamically updating the data cleaning rule configuration file and the transmission strategy table according to the strategy parameter table, and outputting the updated data cleaning rules and transmission parameters; The system dynamically updates the data cleaning rules and transmission strategies based on the tuning parameters generated from the reinforcement learning model. The specific operations include: loading the updated data cleaning rule configuration file (such as a YAML file) through the configuration management tool (such as Nacos / Apollo), and applying the new cleaning rules and transmission parameters to the system. In addition, the transmission agent service process will be restarted to ensure that the new compression algorithm and transmission channel configuration take effect. All update operations will be recorded in the version control system to ensure that each optimization adjustment can be traced and audited.

[0091] For example, suppose that based on the tuning parameters, the system decides to update the cleaning rule configuration file to use a simpler field validation rule, and also updates the configuration of the transmission channel to switch it to RabbitMQ. The update record will be automatically recorded in the version control system, including the rule version number, timestamp, and operator ID.

[0092] Step S505, collect updated transmission logs and real-time performance data, verify the tuning effect and feed it back to the reinforcement learning model, and output the model training iteration record.

[0093] Specifically, the system will collect updated transmission logs and real-time performance data, compare the performance changes before and after optimization, and verify the tuning effect. The system evaluates the effectiveness of the optimization strategy by calculating the latency reduction ratio and the abnormality rate change. If the optimization strategy significantly improves performance (such as latency reduction of more than 10% and abnormality rate increase of no more than 2%), the strategy will be marked as effective and added to the training data set of the reinforcement learning model for further optimization. All training records, including model accuracy and feedback timestamps, are also saved for analysis.

[0094] In the above implementation, the bloodline map and real-time transmission log are combined, and the reinforcement learning model is used for dynamic tuning, which can not only adjust the data cleaning rules and transmission parameters according to the real-time performance data, but also continuously optimize the system performance through continuous feedback and iterative training. Ultimately, this technical solution can significantly improve the efficiency of data processing, reduce latency, optimize resource utilization, and ensure data consistency, thereby achieving a more efficient data transmission and cleaning process.

[0095] In one of the embodiments of the present application, Q-learning can be selected as the algorithm model for reinforcement learning. Q-learning is an offline reinforcement learning algorithm based on value iteration, which is suitable for solving tasks where the state and action space are relatively clear and the strategy needs to be gradually optimized. It evaluates the expected return of taking an action in a specific state by learning a value function called the Q function. The Q function represents the cumulative reward that can be obtained by performing a certain action in a certain state. By continuously updating the Q function, Q-learning can help the model find the optimal strategy so that the agent can make the most appropriate decision in a given environment.

[0096] Specifically, in the practical application of the Q-learning algorithm, it is necessary to first define the state space and action space. The state space includes the current network delay, the CPU usage of the target system, and the complexity of the cleaning rules. These indicators can fully reflect the performance of data transmission and cleaning. The action space includes optimization strategies such as switching transmission channels, simplifying cleaning rules, and changing compression algorithms. The system optimizes performance by executing these actions. Q-learning evaluates the effect of each action through a reward function. The reward function may be calculated based on factors such as end-to-end delay, data consistency, and resource consumption, thereby continuously updating the Q value and guiding the system to adopt the optimal strategy.

[0097] As the training process progresses, Q-learning will update the Q function based on the observed rewards after each execution to find the best decision-making strategy. In the early stages, the system may conduct more exploration and try different optimization schemes, while in the later stages, the system will choose actions that maximize rewards to optimize data transmission performance. In this way, the Q-learning algorithm can automatically adjust data cleaning rules and transmission parameters to achieve efficient data processing under different network conditions, and the system performance will continue to improve over time to ensure long-term optimization results.

[0098] In the embodiment of the present application, standardized metadata files, processing logs, and permission token files are all stored in a distributed database and associated and indexed through a unique shard ID, opening up a full-link tracking and governance closed loop for cross-system data flow. By connecting metadata definitions, processing records, and dynamic permission credentials in series through a unique shard ID, not only can the state of each step of data from collection, cleaning to transmission be verified (such as matching encryption rule changes with log operations), but the root cause of the anomaly can be located in seconds (such as the loss of shards caused by the expiration of permission tokens). At the same time, relying on the high concurrent read and write capabilities of distributed storage, it can support fine-grained access control of hundreds of millions of shards in scenarios such as financial risk control and cross-border compliance.

[0099] The embodiment of the present application also discloses a cross-system data intercommunication platform.

[0100] A cross-system data intercommunication platform, the platform includes: A source system access module, used to access at least two heterogeneous data source systems through a protocol adapter; The parsing and extraction module is used to parse the field name, data type and constraint conditions of each source system, generate a standardized metadata file, and extract the original data stream; the standardized metadata file includes field semantic labels, field constraint rules and sensitivity level tags; The data processing module is used to clean and convert the original data stream based on field semantic labels and field constraint rules, and output the cleaned structured data and processing logs; A security data packet generation module is used to dynamically encrypt the cleaned structured data according to the sensitivity level tag and generate a security data packet queue based on the access permission attributes of the target system; A transmission module, used for sending the secure data packet queue to the target system and recording the transmission log; The lineage map generation module is used to verify the data consistency between the target system and the source system based on the transmission log and the processing log, perform abnormal retransmission, update the transmission log, and generate a lineage map containing the data flow path; The dynamic tuning module is used to dynamically adjust data cleaning rules and transmission parameters based on the bloodline map and updated transmission logs.

[0101] In one of the embodiments of the present application, the cross-system data intercommunication platform includes a microservice module deployed in a Kubernetes cluster. Through containerized encapsulation and dynamic orchestration capabilities, elastic scaling and high availability of resources are achieved, which can automatically respond to business traffic fluctuations and ensure 99.99% availability throughout the year; its microservice architecture supports seamless integration of heterogeneous protocols, is compatible with traditional systems and cloud native components, and combines service grids to uniformly manage communication links. In scenarios such as financial real-time risk control and manufacturing IoT data synchronization, it can automatically trace data anomalies and trigger self-healing through bloodline maps, and dynamically optimize transmission strategies to reduce resource consumption by more than 40%. It can become the core foundation for enterprises to build a secure, agile, and intelligent data center, driving the digital transformation of complex businesses such as cross-cloud disaster recovery and compliance auditing.

[0102] The cross-system data intercommunication platform of the embodiment of the present application can implement any of the above-mentioned cross-system data intercommunication methods, and the specific working process of each module in the cross-system data intercommunication platform can refer to the corresponding process in the above-mentioned method embodiment.

[0103] In the several embodiments provided in this application, it should be understood that the provided methods and systems can be implemented in other ways. For example, the system embodiments described above are only illustrative; for example, the division of a certain module is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0104] The embodiment of the present application also discloses a computer device.

[0105] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the cross-system data intercommunication method as described above is implemented.

[0106] The embodiment of the present application also discloses a computer-readable storage medium.

[0107] A computer-readable storage medium stores a computer program that can be loaded by a processor and execute any one of the above-mentioned cross-system data intercommunication methods.

[0108] Among them, computer-readable storage media can be any tangible medium that contains or stores a program, which can be used by or in conjunction with an instruction execution system, apparatus or device; the program code contained on the computer-readable medium can be transmitted using any appropriate medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0109] It should be noted that in the above embodiments, the description of each embodiment has different emphases, and for the parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0110] The above are all preferred embodiments of the present application, and are not intended to limit the protection scope of the present application. Any feature disclosed in this specification (including the abstract and drawings), unless otherwise stated, can be replaced by other equivalent or alternative features with similar purposes. That is, unless otherwise stated, each feature is only an example of a series of equivalent or similar features.

Claims

1. A cross-system data intercommunication method, characterized in that: The method comprises: Access at least two heterogeneous data source systems through a protocol adapter; Parse the field name, data type and constraint conditions of each source system, generate a standardized metadata file, and extract the original data stream; wherein the standardized metadata file includes field semantic labels, field constraint rules and sensitivity level tags; Based on the field semantic labels and field constraint rules, the original data stream is cleaned and format converted, and the cleaned structured data and processing logs are output; Dynamically encrypting the cleaned structured data according to the sensitivity level tag, and generating a secure data packet queue based on the access permission attributes of the target system; Sending the secure data packet queue to the target system and recording a transmission log; Based on the transmission log and the processing log, verify the data consistency between the target system and the source system and perform abnormal retransmission, update the transmission log, and generate a lineage map containing the data flow path; According to the bloodline map and the updated transmission log, data cleaning rules and transmission parameters are dynamically adjusted.

2. A cross-system data intercommunication method according to claim 1, characterized in that: The steps of dynamically encrypting the cleaned structured data according to the sensitivity level tag and generating a secure data packet queue based on the access permission attribute of the target system include: Generate a sensitive field mapping table based on the sensitivity level tags in the standardized metadata file; Dynamically assign encryption keys and encryption algorithms according to the sensitivity level labels, bind the encryption keys to the shard IDs in the structured data, and output a shard key table; Based on the sensitive field mapping table and the sharding key table, dynamically encrypt the sensitive fields in the cleaned structured data, and output the encrypted data shards and the associated encryption algorithm identifiers; Generate a permission policy document based on the access permission attributes of the target system, bind the shard ID to the permission policy document, and generate a permission token file according to an asymmetric encryption algorithm; The encrypted data shard, shard ID and permission token file are encapsulated into a secure data packet, and the shard ID, encryption algorithm identifier and target system attributes are written into the shard metadata index table, and a secure data packet sequence is output.

3. A cross-system data intercommunication method according to claim 2, characterized in that: The steps of sending the secure data packet queue to the target system and recording the transmission log include: Obtaining the service type, time sensitivity rule and user configuration policy of the encrypted data slice, and generating a real-time tag; Based on the preset transmission strategy table, a transmission strategy is generated according to the target system attributes and real-time tags in the shard metadata index table; Obtain network status monitoring data in real time; Determine a transmission channel according to the transmission strategy and network status monitoring data; distributing the secure data packet queue to the transmission channel; Get the shard ID, transmission channel type, transmission timestamp and network time consumption data, and write the structured transmission log according to the shard ID partition.

4. A cross-system data intercommunication method according to claim 3, characterized in that: Based on the transmission log and the processing log, verifying the data consistency between the target system and the source system and performing abnormal retransmission, updating the transmission log, and generating a lineage map including a data flow path include the following steps: Based on the shard ID in the transmission log and the field change record in the processing log, a shard-level data snapshot table is generated by association; wherein the shard-level data snapshot table includes the association result of the source system hash value, the target system hash value and the transmission path node; Perform data anomaly detection based on the source system hash value and the target system hash value, and output an abnormal shard list and an abnormal type mark; Extracting original data slices or difference fields from the source system according to the abnormal slice list and the abnormal type mark, retransmitting them to the target system through the transmission channel, and outputting retry records and updated transmission logs; Based on the retry records and the updated transmission logs, a bloodline graph including data nodes, processing nodes and abnormal nodes is constructed.

5. A cross-system data intercommunication method according to claim 4, characterized in that: The steps of dynamically adjusting data cleaning rules and transmission parameters according to the bloodline map and the updated transmission log include: Based on the bloodline map and the updated transmission log, construct a data flow-performance association table; According to the data flow-performance association table, network delay, cleaning time and resource utilization indicators are aggregated and calculated according to the transmission channel type and cleaning rule version, and a performance analysis report is output; Based on the performance analysis report, a strategy parameter table is generated through a reinforcement learning model to output tuning parameters; the tuning parameters include transmission channel switching instructions, compression algorithm adjustment suggestions, and cleaning rule simplification strategies; According to the policy parameter table, dynamically update the data cleaning rule configuration file and the transmission policy table, and output the updated data cleaning rules and transmission parameters; Collect updated transmission logs and real-time performance data, verify the tuning effect and feed it back to the reinforcement learning model, and output the model training iteration record.

6. A cross-system data intercommunication method according to any one of claims 1 to 5, characterized in that: The standardized metadata files, processing logs, and permission token files are all stored in a distributed database and are associated and indexed through a unique shard ID.

7. A cross-system data intercommunication platform, characterized in that: The platform includes: A source system access module, used to access at least two heterogeneous data source systems through a protocol adapter; A parsing and extraction module, which is used to parse the field name, data type and constraint conditions of each source system, generate a standardized metadata file, and extract the original data stream; wherein the standardized metadata file includes field semantic labels, field constraint rules and sensitivity level tags; A data processing module, used to perform data cleaning and format conversion on the original data stream based on the field semantic labels and field constraint rules, and output the cleaned structured data and processing logs; A security data packet generation module, used to dynamically encrypt the cleansed structured data according to the sensitivity level tag, and generate a security data packet queue based on the access permission attributes of the target system; A transmission module, used for sending the secure data packet queue to the target system and recording a transmission log; A lineage map generation module is used to verify the data consistency between the target system and the source system and perform abnormal retransmission based on the transmission log and the processing log, update the transmission log, and generate a lineage map containing the data flow path; The dynamic tuning module is used to dynamically adjust data cleaning rules and transmission parameters according to the blood relationship map and the updated transmission log.

8. A cross-system data intercommunication platform according to claim 7, characterized in that: The cross-system data intercommunication platform includes a microservice module deployed in a Kubernetes cluster.

9. A computer device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method according to any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium, characterized in that: A computer program is stored which can be loaded by a processor and execute the method according to any one of claims 1 to 6.

Images