Data exchange system based on data set

Abstract

The invention relates to the technical field of data integration and management, in particular to a data exchange system based on datasets, comprising a dataset storage unit used for storing and managing a plurality of datasets, each dataset comprising one or more data files; the data set access module is used for allowing a user to access, query and retrieve the data set; the data set exchange interface is used for receiving and sending a data set; and the data set converter is used for converting the data sets with different formats and structures into data sets with uniform formats and structures. In the data exchange process, data exchanged each time is always a complete whole from data extraction to quality inspection, data auditing, data submission, data revocation, data query and the like. As the data set has the advantages of being customizable, easy to expand and revocable, the data can be conveniently maintained according to a certain theme, and a tree-shaped, multi-layer and multi-angle data set organization structure can be formed.

Description

Technical Field

[0001] The present invention relates to the technical field of data integration and management, and in particular to a data exchange system based on data sets. Background Art

[0002] The autonomy, distribution and heterogeneity of data between information systems of various business departments make data exchange an inevitable requirement. With the continuous in-depth research on data exchange technology, XML-based data exchange has been realized, which has solved the problem of heterogeneous data integration to a certain extent. However, there are still the following shortcomings: (1) Previous research focused on the process of data exchange implementation, but ignored the fact that data is the real core of data exchange, and did not pay attention to the collection, transmission, storage, review, management, rejection, organization, display and tracking process of data in data exchange. (2) In the past, data exchange files were just files and could not describe the structural information of the data model. They were invisible to users, so the exchange files could not be managed.

[0003] Most of the patient master index systems currently available on the market are judged by fixed identity identification information. The single calculation method cannot meet the complex business scenarios, and there are defects such as data redundancy and low calculation efficiency in big data scenarios. Existing technology: The system adopts a data set metamodel to realize data exchange between databases. By introducing the ideas of data sets and data meta logical models into data exchange, the definition, management, application and other technologies of data sets are studied, and data sets are applied to all aspects of the data exchange process. An XML-based data mapping rule is designed to facilitate the management of data model mapping and realize data exchange between heterogeneous databases. Dynamic migration of data between heterogeneous databases is realized to ensure data consistency.

[0004] However, the inventors have explored that the prior art still has at least the following defects: With the rapid development of informatization and the rapid advancement of digital transformation, enterprises are gradually turning to paperless office. Various business systems are difficult to interoperate due to different technologies and protocols. If enterprises want to integrate data or processes, they need to let business systems directly share data through interfaces. This method can quickly solve the data problems faced by enterprises at the beginning, but as the number of business systems gradually increases, the interface data between business systems increases in magnitude, making it difficult to unify and maintain. Secondly, the interface standardization is low, and each interface is provided by different manufacturers, which poses a hidden danger to the stability and reliability of the system. Thirdly, the coupling between systems is too strong, which increases the difficulty of business expansion and maintenance. The systems mainly rely on direct access sharing or database view calls to achieve integration purposes. There is no log record or redundancy guarantee for the interaction between some systems. There is a lack of transparency on whether the information interaction is successful, which is not convenient for tracking and monitoring. Finally, the data structure, message format, and communication protocol between heterogeneous systems are inconsistent, affecting data utilization. Summary of the invention

[0005] In view of this, an object of the present invention is to provide a data exchange system based on data sets.

[0006] Based on the above objectives, the present invention provides a data exchange system based on data sets.

[0007] A data exchange system based on data sets, comprising the following components:

[0008] a. A data set storage unit, used to store and manage multiple data sets, each of which contains one or more data files;

[0009] b. Dataset access module, used to allow users to access, query and retrieve data sets;

[0010] c. Dataset exchange interface, used to receive and send data sets;

[0011] d. Dataset converter, used to convert datasets of different formats and structures into a unified format and structure;

[0012] e. Dataset validator, used to verify and ensure the integrity and accuracy of the dataset;

[0013] f. Dataset update mechanism, used to update and synchronize changes in data sets in real time;

[0014] g. Visual modeling module, based on the definition of data elements in the data set, realizes visual modeling and automatically generates database physical tables.

[0015] Furthermore, the data exchange system includes the following functions:

[0016] a. Dataset search function, which allows users to search for the required datasets based on keywords, attributes or other conditions;

[0017] b. Dataset import function allows users to import external datasets into the system for management and use;

[0018] c. Dataset export function, which allows users to export data sets in the system into different formats for use in other applications or systems;

[0019] d. Dataset sharing function, allowing users to share datasets with other users or systems through the dataset exchange interface;

[0020] e. Dataset permission control function, which allows system administrators or dataset owners to set the access rights and confidentiality level of datasets;

[0021] f. Dataset version control function allows users to manage and track versions of datasets.

[0022] Furthermore, the data exchange system provides a visual data exchange process, including the following functions:

[0023] a. Publish and subscribe, which implements the publishing and subscription functions of data sets, allowing subscribers to obtain data sets published by publishers;

[0024] b. Synchronization agent, used to synchronize data updates between multiple data sets to maintain data consistency;

[0025] c. Service aggregation, which aggregates data from multiple data sets into one service to facilitate user access and use;

[0026] d. Input and output parameter conversion: convert the input and output parameters of different data sets to meet the needs of different systems;

[0027] e. Standard mapping: implement standard mapping between different data sets to ensure correct mapping and conversion of data.

[0028] Furthermore, the data set exchange system implements multiple message middleware switching functions, including:

[0029] a. Based on the configuration switching of mainstream MQ message middleware, users can switch different message middleware through simple configuration;

[0030] b. Supports commonly used message middleware, including Kafka, RabbitMQ and ActiveMQ.

[0031] Furthermore, the dataset exchange system provides incremental data extraction functions, including:

[0032] a. Based on the CDC and ETL dual data adaptation mode, incremental data extraction and conversion are realized;

[0033] b. According to the user's configuration, the incremental data is automatically converted into a standard data format to facilitate subsequent processing and analysis.

[0034] Furthermore, the data set exchange system has the following functions:

[0035] a. Real-time monitoring of data sets, allowing users to monitor the changes and updates of data sets in real time so as to make corresponding processing and decisions in a timely manner;

[0036] b. Dataset quality management, providing data set quality assessment and monitoring functions to ensure the accuracy, consistency and completeness of the data set;

[0037] c. Dataset performance optimization: by optimizing the way of data set access and query, the system performance and response speed can be improved;

[0038] d. Dataset backup and recovery: supports regular backup and recovery of data sets to protect data from accidental loss.

[0039] Furthermore, the data set exchange system has the following functions:

[0040] a. Dataset collaboration and collaborative editing, allowing multiple users to collaborate and edit the same dataset at the same time, ensuring dataset consistency and collaborative efficiency;

[0041] b. Data set auditing and tracing: recording and tracking the operation records and change history of data sets to meet compliance and audit requirements;

[0042] c. Dataset reporting and visual analysis: Provides data set reporting and visual analysis functions to help users better understand and utilize the value of data sets.

[0043] Beneficial effects of the present invention:

[0044] 1. In the data exchange process of the present invention, each exchanged data, from the initial data extraction to quality inspection, data review, data submission, data revocation, data query, etc., is always a complete whole. We call a set of data exchanged once a data set. Since the data set has the advantages of being customizable, easy to expand, and revocable, it is convenient to maintain data according to a certain theme, and a tree-like, multi-level, and multi-angle data set organization structure can be formed. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only for the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0046] Figure 1 A schematic diagram of a partial operation logic structure of an embodiment of the present invention;

[0047] Figure 2 It is a schematic diagram of a partial operation logic structure of an embodiment of the present invention;. DETAILED DESCRIPTION

[0048] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments.

[0049] Example:

[0050] A data exchange system based on a data set, comprising:

[0051] 1. Design and build a data set storage unit:

[0052] -Choose an appropriate database system or data storage technology for storing and managing multiple data sets.

[0053] -Create a data set storage unit and establish corresponding data tables or collections to store data sets.

[0054] 2. Develop dataset access module:

[0055] -Design user interfaces that allow users to access, query, and retrieve datasets.

[0056] -Implement dataset search functionality, allowing users to search for required datasets based on keywords, attributes, or other conditions.

[0057] 3. Implement the dataset exchange interface:

[0058] -Design dataset exchange interface for receiving and sending datasets.

[0059] -Implement the dataset import function, allowing users to import external datasets into the system for management and use.

[0060] -Implement the dataset export function, allowing users to export the datasets in the system into different formats.

[0061] 4. Develop dataset converter:

[0062] -Design dataset converters to convert datasets of different formats and structures into a unified format and structure.

[0063] -Implement input and output parameter conversion functions to convert the input and output parameters of different data sets to meet the needs of different systems.

[0064] 5. Design dataset validator:

[0065] - Develop a dataset validator to verify and ensure the completeness and accuracy of the dataset.

[0066] 6. Implement dataset update mechanism:

[0067] -Design and implement dataset update mechanisms to update and synchronize dataset changes in real time.

[0068] 7. Develop visual modeling module:

[0069] -Design a visual modeling module that allows users to perform visual modeling based on the definition of data elements in the dataset.

[0070] -Realize the function of automatically generating database physical tables and generate corresponding table structures based on visual modeling results.

[0071] 8. Implement other features:

[0072] -Develop dataset permission control function, allowing system administrators or dataset owners to set the access rights and confidentiality level of datasets.

[0073] -Implement the dataset version control function, allowing users to manage and track dataset versions.

[0074] Through the above embodiments, we can build a feature-rich data exchange system based on data sets, providing a series of functions such as data set storage, access, exchange, conversion, verification, update, visual modeling, etc. to meet user needs.

[0075] The present invention is intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A data exchange system based on data sets, characterized in that: Includes the following components: a. A data set storage unit, used to store and manage multiple data sets, each of which contains one or more data files; b. Dataset access module, used to allow users to access, query and retrieve data sets; c. Dataset exchange interface, used to receive and send data sets; d. Dataset converter, used to convert datasets of different formats and structures into a unified format and structure; e. Dataset validator, used to verify and ensure the integrity and accuracy of the dataset; f. Dataset update mechanism, used to update and synchronize changes in data sets in real time; g. Visual modeling module, based on the definition of data elements in the data set, realizes visual modeling and automatically generates database physical tables.

2. A data exchange system based on data sets according to claim 1, characterized in that: The data exchange system includes the following functions: a. Dataset search function, which allows users to search for the required datasets based on keywords, attributes or other conditions; b. Dataset import function allows users to import external datasets into the system for management and use; c. Dataset export function, which allows users to export data sets in the system into different formats for use in other applications or systems; d. Dataset sharing function, allowing users to share datasets with other users or systems through the dataset exchange interface; e. Dataset permission control function, which allows system administrators or dataset owners to set the access rights and confidentiality level of datasets; f. Dataset version control function allows users to manage and track versions of datasets.

3. A data exchange system based on data sets according to claim 2, characterized in that: The data set exchange system provides a visual data exchange process. Features include: a. Publish and subscribe, which implements the publishing and subscription functions of data sets, allowing subscribers to obtain data sets published by publishers; b. Synchronization agent, used to synchronize data updates between multiple data sets to maintain data consistency; c. Service aggregation, which aggregates data from multiple data sets into one service to facilitate user access and use; d. Input and output parameter conversion: convert the input and output parameters of different data sets to meet the needs of different systems; e. Standard mapping: implement standard mapping between different data sets to ensure correct mapping and conversion of data.

4. A data exchange system based on data sets according to claim 3, characterized in that: The data set exchange system implements multiple message middleware switching functions, including: a. Based on the configuration switching of mainstream MQ message middleware, users can switch different message middleware through simple configuration; b. Supports commonly used message middleware, including Kafka, RabbitMQ and ActiveMQ.

5. A data exchange system based on data sets according to claim 4, characterized in that: The data set exchange system provides incremental data extraction functions, including: a. Based on the CDC and ETL dual data adaptation mode, incremental data extraction and conversion are realized; b. According to the user's configuration, the incremental data is automatically converted into a standard data format to facilitate subsequent processing and analysis.

6. A data exchange system based on data sets according to claim 5, characterized in that: The data set exchange system further has the following functions: a. Real-time monitoring of data sets, allowing users to monitor the changes and updates of data sets in real time so as to make corresponding processing and decisions in a timely manner; b. Dataset quality management, providing data set quality assessment and monitoring functions to ensure the accuracy, consistency and completeness of the data set; c. Dataset performance optimization: by optimizing the way of data set access and query, the system performance and response speed can be improved; d. Dataset backup and recovery: supports regular backup and recovery of data sets to protect data from accidental loss.

7. A data exchange system based on data sets according to claim 6, characterized in that: The data set exchange system further has the following functions: a. Dataset collaboration and collaborative editing, allowing multiple users to collaborate and edit the same dataset at the same time, ensuring dataset consistency and collaborative efficiency; b. Data set auditing and tracing: recording and tracking the operation records and change history of data sets to meet compliance and audit requirements; c. Dataset reporting and visual analysis: Provides data set reporting and visual analysis functions to help users better understand and utilize the value of data sets.

Images