A cross-network data sharing and exchange system

Abstract

The present invention relates to a cross-network data sharing and exchange system, and belongs to the field of data middle platform and data security. The system of the present invention includes a data sharing and exchange subsystem, a data acquisition subsystem, a data governance subsystem, a data sharing subsystem and a data exchange support tool. The present invention provides a method for the development of a cross-network data exchange system, which enhances the convenience and security of data in storage and circulation. The data exchange system provides a feasible method for data transmission across levels, regions, departments, systems, networks and confidentiality levels. The data exchanged using this method can be controlled, traceable, and can be documented and proved. In addition to being used for data circulation and transmission, the present invention also supports rule-based data circulation, so that business personnel can realize data circulation by defining business rules.

Description

Technical Field

[0001] The present invention belongs to the field of data middle platform and data security, and specifically relates to a cross-network data sharing and exchange system. Background Art

[0002] With the in-depth development of informatization and digitalization, promoting horizontal and vertical data exchange among departments and realizing cross-level, cross-regional, cross-departmental, cross-system, and cross-network data sharing and business collaboration have become the key to digitalization and even digital intelligence. Ensuring the convenient transmission and transmission security of structured data, semi-structured data, and unstructured data of various departments has become an urgent need for data sharing and exchange systems. Most shared exchange systems only realize the transmission of messages and files, and the transmitted data is point-to-point transmission. There is a lack of support for cross-layer, cross-domain, and cross-classification routing of data, and there is also a lack of security control over the transmitted data, which brings hidden dangers to data leakage.

[0003] The purpose of the present invention is to provide a cross-network data sharing and exchange system. The use of the present invention can solve: 1. cross-level, cross-regional, cross-departmental, cross-system, cross-network, and cross-classification transmission of data; 2. routing and transmission of structured data (relational database); 3. routing and transmission of unstructured and semi-structured data; 4. secure transmission of data. Summary of the invention

[0004] 1. Technical issues to be resolved

[0005] The technical problem to be solved by the present invention is how to provide a cross-network data sharing and exchange system to solve the above technical problems.

[0006] (II) Technical solution

[0007] In order to solve the above technical problems, the present invention proposes a cross-network data sharing and exchange system, which includes a data sharing and exchange subsystem, a data acquisition subsystem, a data governance subsystem, a data sharing subsystem and a data exchange support tool;

[0008] The data sharing and exchange subsystem realizes the vertical and horizontal flow and exchange of data through internal exchange, external exchange, data synchronization and file synchronization;

[0009] The data collection subsystem extracts and loads external data into the internal source library. The extracted data includes structured data, semi-structured data, unstructured data, streaming data and IoT data, realizes data collection and synchronization, and reconciles the collected data to ensure the consistency of data transmission;

[0010] The data sharing subsystem is used to provide more effective data resources for the data sharing exchange subsystem through data sharing means. The data sharing means include: data directory, data authority, data subscription distribution, data object scheme and data integration.

[0011] The data exchange support tool provides support for the data sharing exchange subsystem, data acquisition subsystem, data governance subsystem and data sharing subsystem, including: data extraction, data loading, secure transmission, file transfer, cross-domain isolation exchange adaptation, computing model management, health check, data verification, service engine, process engine, scheduling engine, rule engine, offline computing engine, real-time computing engine, ad hoc query engine and elastic computing engine.

[0012] (III) Beneficial effects

[0013] The present invention proposes a cross-network data sharing and exchange system. The present invention provides a method for the development of a cross-network data exchange system, which enhances the convenience and security of data in storage and circulation. The data exchange system provides a feasible method for data transmission across levels, regions, departments, systems, networks, and confidentiality levels. The data exchanged using this method can be controlled, traceable, and can be documented and proved. In addition to being used for data circulation and transmission, the present invention also supports rule-based data circulation, so that business personnel can achieve data circulation by defining business rules. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 This is a schematic diagram of the architecture of the cross-network data sharing and exchange system of the present invention;

[0015] Figure 2 Data flow diagram for internal exchange;

[0016] Figure 3 The data processing system architecture diagram for external exchange;

[0017] Figure 4 Data processing flow chart for external exchange;

[0018] Figure 5 Flowchart for file synchronization. DETAILED DESCRIPTION

[0019] In order to make the purpose, content and advantages of the present invention more clear, the specific implementation methods of the present invention are further described in detail below in conjunction with the drawings and examples.

[0020] A schematic diagram of a cross-network data sharing and exchange system architecture of the present invention is shown in FIG1 .

[0021] The cross-network data sharing and exchange system includes five subsystems: data sharing and exchange subsystem (see label ③), data collection subsystem (see label ⑯), data governance subsystem (see label ⑰), data sharing subsystem (see label ⑱), and data exchange support tools (see label ⑲). The data sharing and exchange subsystem (see label ③) uses the services and capabilities provided by the data collection subsystem (see label ⑯), data governance subsystem (see label ⑰), data sharing subsystem (see label ⑱), and data exchange support tools (see label ⑲).

[0022] (1) Data sharing and exchange subsystem (see label ③)

[0023] The data sharing and exchange subsystem realizes the vertical and horizontal flow and exchange of data through internal exchange, external exchange, data synchronization, file synchronization and other functions.

[0024] 1) Internal exchange: data flow within the local area network;

[0025] 2) External exchange: data exchange across levels, regions, departments, systems, networks, and confidentiality levels;

[0026] 3) Data synchronization: Due to network latency issues, this generally refers to the synchronization and backup of data (structured, semi-structured, and unstructured) in the same city;

[0027] 4) File synchronization: synchronization and backup of documents, audio, video, etc.

[0028] (2) Data acquisition subsystem (see reference ⑯)

[0029] The data collection subsystem extracts and loads external data into the internal system (source library). The extracted data includes structured data, semi-structured data, unstructured data, streaming data, and IoT data, realizing data collection and synchronization. At the same time, it can optionally reconcile the collected data to ensure the consistency of data transmission.

[0030] (3) Data governance subsystem (see number ⑰)

[0031] The data governance subsystem is used to place the data in the source library into the integrated library after data governance. Data governance includes steps such as data specification, data model construction, field collection, data classification, field mapping, and data quality.

[0032] (4) Data Sharing Subsystem (see reference ⑱)

[0033] The data sharing subsystem is used to provide more effective data resources for the data sharing exchange subsystem through data sharing means. Data sharing means include: data directory, data authority, data subscription distribution, data object scheme, data integration, etc.

[0034] Data Directory: Build a common information service operation platform and information resource sharing environment for the data to be shared, provide support services such as service registration and organization, publishing and discovery, directory data synchronization and sharing, subscription and delivery, organically link various business information service systems, and realize controlled sharing of various information resources.

[0035] Data authority: The data to be transmitted has been classified and graded. For data to be transferred from a low-level network to a high-level network, an approval process is required, or approval is exempted for specific positions or personnel. For data to be transferred from a high-level network to a low-level network, the confidentiality level of the data itself needs to be strictly reviewed to ensure that the confidentiality level of the target network to be transmitted is not lower than the confidentiality level of the data itself.

[0036] Data subscription and distribution: By utilizing the node discovery, information cataloging organization, directory synchronization, information resource search and other functions of the data directory, information resource search, information delivery and distribution services are realized.

[0037] Data object access: supports the identification, parsing, search, discovery and use of data in the form of digital objects across the entire domain.

[0038] Data integration: By defining templates for jobs and defining data processing strategies, data can be transmitted according to one or more defined templates and rules to achieve rule-based data flow.

[0039] (5) Data exchange support tool (see reference ⑲)

[0040] The data exchange support tool provides support for the data sharing exchange subsystem, data collection subsystem, data governance subsystem and data sharing subsystem, including: data extraction, data loading, secure transmission, file handling, cross-domain isolation exchange adaptation, computing model management, health check, data verification, service engine, process engine, scheduling engine, rule engine, offline computing engine, real-time computing engine, ad hoc query engine and elastic computing engine;

[0041] Data extraction: realize full and incremental data extraction from structured data based on rules;

[0042] Data loading: Load the received data into a structured database;

[0043] Secure transmission: realize encryption, decryption, decryption and other processing of data during transmission;

[0044] File handling: realize the reading, sub-packaging, merging, and writing of documents, videos, audios, spreadsheets, texts, ppt, html and other files;

[0045] Cross-domain isolation exchange adaptation: In cross-network transmission, the cross-network isolation exchange solutions and equipment of different manufacturers are different, and the corresponding APIs are also different. In order to enhance the environmental adaptability of the system, the data transmission and processing logic of upper-level applications such as data extraction and file transfer remains unchanged. The cross-network isolation exchange solutions and equipment of different manufacturers are adapted to ensure the consistency and integrity of application functions.

[0046] Computing model management: In order to ensure that data transmitted across the network can be restored normally, in certain strictly demanding environments, the support of computing models is required. The computing model includes model publishing registration and on-demand retrieval, so that various resources on the network can be organized and shared in a service-oriented manner, realizing the decoupling between resource users and resource providers.

[0047] Health check: Provide health check and operation status monitoring of application components, and report the health check results to the data center or sub-center node;

[0048] Data verification: Verify the consistency and repeatability of the data to be transmitted and the data that has been transmitted;

[0049] Service engine: management of service registration, discovery, elastic scaling, and monitoring;

[0050] Process engine: manages and controls the data flow process of internal exchange, external exchange, data synchronization, and file synchronization, as well as the definition and operation of approval processes;

[0051] Scheduling engine: manages the scheduling of processes and tasks, definition of execution rules, and adaptive optimization of scheduling algorithms.

[0052] Rule Engine: Separate business processes and business rules and flexibly manage rules through parsing, calling, executing, context management, and input and output data management of rule packages.

[0053] Offline computing engine: It uses distributed computing frameworks such as NoSQL, MapReduce, Spark, and Flink to provide offline batch processing capabilities for massive data and task operation and maintenance functions. It supports task operation status monitoring and control and executes parallel operation mode.

[0054] Real-time computing engine: It adopts a distributed stream processing computing framework to provide real-time high-concurrency streaming processing capabilities for massive data, supports offline channels and online channels dual-mode parallel data import, high-concurrency queries, supports external access to message queues, relational databases, NoSQL, real-time logs and other data sources, and provides real-time task management and monitoring capabilities.

[0055] Ad hoc query engine: provides efficient real-time query for relational databases, data warehouses, distributed databases, etc.

[0056] Elastic computing engine: Unified allocation and management of computing resources, monitoring of resource usage, and allocation and recycling of resources on demand.

[0057] The data processing flow of the present invention is as follows:

[0058] (1) Internal exchange

[0059] The data flow of internal exchange is shown in Figure 2.

[0060] Internal data exchange uses the data extraction function to extract data from the source end. After data cleaning, data conversion, dictionary alignment, value matching, rule definition, data filtering, data review and other processing, the data is routed to the target end and loaded into the target location at the target end according to predefined rules.

[0061] (2) External exchange

[0062] The data processing architecture of external exchange is shown in Figure 3.

[0063] Note: UTM refers to cross-domain firewall, intrusion detection, etc.

[0064] External switching requires the deployment of front-end switching nodes, switching nodes (low-code development) in different networks, and optional deployment of data sharing functions. Figure 3 In the example, Network A and Network B are in different domains, while Network C and Network B are in different departments in the same domain. When Network A needs to communicate with Network B, the data needs to pass through the cross-domain isolation exchange area. The data flow in the cross-domain isolation exchange area can be unidirectional or bidirectional. In institutions and units with high network security requirements, there may be two sets of "cross-domain isolation exchange areas", that is, each flow has its own set. Figure 3 The situation shown in the figure is that the network gate device supports bidirectional.

[0065] When data needs to be exchanged, the demander knows the metadata information of the source end, or obtains the metadata information of the data to be exchanged from the data sharing, builds the "external exchange process" in the local exchange node, defines the calculation model call, rule definition, etc., and after security processing, the data is packaged and processed, and the cross-domain isolation exchange adapter component is used to transmit the data across the network. After the target domain passes the data security check, the data is packaged and restored, and then the data is verified, the process is processed, the rule is processed, and the model is processed, and the data is loaded to the target location of the target network. The data processing flow of external exchange is shown in Figure 4.

[0066] (3) Data synchronization

[0067] The data synchronization process is consistent with the external exchange data processing process as shown in Figure 4.

[0068] (4) File synchronization

[0069] When files need to be synchronized (files can be single files, directories, files in the file system, or files in a distributed file system), the demander knows the metadata information of the source end, or obtains the metadata information of the data to be exchanged from the data sharing, and builds an "external exchange process" in the local exchange node, using files to move (read) files, define calculation model calls, rule definitions, etc. After security processing, the data is packaged and processed, and the cross-domain isolation exchange adapter component is used to transmit data across the network. After the target domain passes the data security check, the data is packaged and restored, and then the data is verified, the process is processed, the rules are processed, and the model is processed, and the file is moved (landed) to the target location of the target network. The file synchronization process of external exchange is shown in Figure 5.

[0070] The present invention provides a method for the development of a cross-network data exchange system, which enhances the convenience and security of data in storage and circulation. The data exchange system provides a feasible method for data transmission across levels, regions, departments, systems, networks, and confidentiality levels. The data exchanged using this method can be controlled, traceable, and can be documented and proved. In addition to being used for data circulation and transmission, the present invention also supports rule-based data circulation, so that business personnel can realize data circulation by defining business rules.

[0071] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A cross-network data sharing and exchange system, characterized in that: The system includes data sharing and exchange subsystem, data acquisition subsystem, data governance subsystem, data sharing subsystem and data exchange support tools; The data sharing and exchange subsystem realizes the vertical and horizontal flow and exchange of data through internal exchange, external exchange, data synchronization and file synchronization; The data collection subsystem extracts and loads external data into the internal source library. The extracted data includes structured data, semi-structured data, unstructured data, streaming data and IoT data, realizes data collection and synchronization, and reconciles the collected data to ensure the consistency of data transmission; The data governance subsystem is used to put the data in the source library into the integrated library after data governance. Data governance includes data specification, data model construction, field collection, data classification, field mapping and data quality steps; The data sharing subsystem is used to provide more effective data resources for the data sharing exchange subsystem through data sharing means. The data sharing means include: data directory, data authority, data subscription distribution, data object scheme and data integration. The data exchange support tool provides support for the data sharing exchange subsystem, data collection subsystem, data governance subsystem and data sharing subsystem, including: data extraction, data loading, secure transmission, file handling, cross-domain isolation exchange adaptation, computing model management, health check, data verification, service engine, process engine, scheduling engine, rule engine, offline computing engine, real-time computing engine, ad hoc query engine and elastic computing engine; in, The data sharing and exchange subsystem uses the services and capabilities provided by the data acquisition subsystem, data governance subsystem, data sharing subsystem, and data exchange support tools.

2. The cross-network data sharing and exchange system according to claim 1, characterized in that: Internal exchange refers to the flow of data within the local area network; external exchange refers to the exchange of data across levels, regions, departments, systems, networks, and confidentiality levels; data synchronization refers to the synchronization and backup of data in the same city; file synchronization refers to the synchronization and backup of documents, audio, and video.

3. The cross-network data sharing and exchange system according to claim 1, characterized in that: Data catalog refers to: building a common information service operation platform and information resource sharing environment for data to be shared, providing service registration and organization, publishing and discovery, directory data synchronization and sharing, subscription and delivery support services, organically linking various business information service systems, and realizing controlled sharing of various information resources; Data permissions mean that the data to be transmitted has been classified and graded. For data to be transferred from a low-level network to a high-level network, an approval process is required, or approval is exempted for specific positions or personnel. For data to be transferred from a high-level network to a low-level network, the confidentiality level of the data itself needs to be strictly reviewed to ensure that the confidentiality level of the target network to be transmitted is not lower than the confidentiality level of the data itself. Data subscription and distribution refers to: realizing information resource search and information delivery and distribution services by utilizing the node discovery, information cataloging organization, directory synchronization, and information resource search functions of the data directory; Data object access refers to: supporting the identification, parsing, search, discovery and use of data in the form of digital objects across the entire domain; Data integration means: by defining templates for jobs and defining data processing strategies, data is transmitted according to one or more defined templates and rules to achieve rule-based data flow.

4. The cross-network data sharing and exchange system according to claim 1, characterized in that: Data extraction refers to: extracting data in full and incremental form from structured data based on rules; Data loading means: loading the received data into a structured database; Secure transmission means: realizing encryption, decryption and decryption of data during transmission; File handling means: reading, splitting, merging and writing documents, videos, audios, spreadsheets, texts, ppt and html files; Cross-domain isolation exchange adaptation means: in cross-network transmission, different manufacturers have different cross-network isolation exchange solutions and devices, and the corresponding APIs are also different. In order to enhance the environmental adaptability of the system, the data transmission processing logic of data extraction and file transfer remains unchanged, and the cross-network isolation exchange solutions and devices of different manufacturers are adapted to ensure the consistency and integrity of application functions; Computing model management means: in order to ensure that data transmitted across the network can be restored normally, in certain strict environments, the support of computing models is required. The computing model includes model publishing registration and on-demand retrieval, so that various resources on the network can be organized and shared in a service-oriented manner, realizing the decoupling between resource users and resource providers; Health check means: providing health check and operation status monitoring of application components, and reporting the health check results to the data center or sub-center node; Data verification refers to: checking the consistency and repeatability of the data to be transmitted and the data that has been transmitted.

5. The cross-network data sharing and exchange system according to claim 4, characterized in that: The service engine refers to: registration, discovery, elastic scaling, and monitoring management of services; The process engine refers to: controlling the data flow process of internal exchange, external exchange, data synchronization, and file synchronization, as well as the definition and operation of the approval process; The scheduling engine refers to: managing the scheduling of processes and tasks, defining execution rules, and adaptive optimization of scheduling algorithms; The rule engine refers to: separation of business processes and business rules, and flexible management of rules through parsing, calling, executing, context management, and input and output data management of rule packages; Offline computing engine refers to: using a distributed computing framework, providing offline batch processing capabilities for massive data and task operation and maintenance functions, supporting task operation status monitoring and control, and executing parallel operation mode; Real-time computing engine refers to: using a distributed stream processing computing framework, providing real-time high-concurrency streaming processing capabilities for massive data, supporting offline channels, online channels dual-mode parallel data import, high-concurrency query, supporting external access to message queues, relational databases, NoSQL, and real-time log data sources, and providing real-time task management and monitoring capabilities; Ad hoc query engine refers to: providing efficient real-time query for relational databases, data warehouses, and distributed databases; Elastic computing engine refers to: unified allocation and management of computing resources, monitoring of resource usage, and allocation and recovery of resources on demand.

6. The cross-network data sharing and exchange system according to claim 5, characterized in that: The distributed computing frameworks are: NoSQL, MapReduce, spark and flink.

7. The cross-network data sharing and exchange system according to any one of claims 1 to 6, characterized in that: Internal exchange refers to the use of data extraction functions to extract data from the source end. After data cleaning, data conversion, dictionary alignment, value matching, rule definition, data filtering, and data review, the data is routed to the target end and loaded into the target location at the target end according to predefined rules.

8. The cross-network data sharing and exchange system according to claim 7, characterized in that: External exchange means: when data needs to be exchanged, the demander knows the metadata information of the source end, or obtains the metadata information of the data to be exchanged from data sharing, builds an "external exchange process" in the local exchange node, defines the calculation model call and rule definition, and after security processing, subpackages the data and uses cross-domain isolation exchange adapter components to transmit data across the network. After the target domain passes the data security check, the data is packaged and restored, and then the data is verified, processed by the process, rules, and model, and then the data is loaded to the target location of the target network.

9. The cross-network data sharing and exchange system according to claim 8, characterized in that: When communicating between networks in different domains, data needs to pass through the cross-domain isolation exchange area. The data flow in the cross-domain isolation exchange area is unidirectional or bidirectional. In institutions and units with high network security requirements, there will be two sets of "cross-domain isolation exchange areas", that is, each flow has an exclusive set.

10. The cross-network data sharing and exchange system according to claim 9, characterized in that: File synchronization means: when file synchronization is needed, the demander knows the metadata information of the source end, or obtains the metadata information of the data to be exchanged from the data sharing, builds an "external exchange process" in the local exchange node, uses files to move files, defines the calculation model call, and defines rules. After security processing, the data is subpackaged and processed, and the cross-domain isolation exchange adapter component is used to transmit the data across the network. After the target domain passes the data security check, the data is packaged and restored, and then the data is verified, the process is processed, the rules are processed, and the model is processed, and the file is moved to the target location of the target network.

Images