Metadata-driven integration systems

Abstract

This invention discloses a metadata-driven integration system. The platform includes an overall architecture module, a logical structure module, and an execution flow module. The overall architecture module provides data to enterprise users, offering integration operations including data visualization configuration, process visualization configuration, general capabilities, and basic capabilities. The logical structure module includes platform access layer components, platform business layer components, platform adaptation layer components, platform execution layer components, platform protocol layer components, and platform management system components. This invention provides the meaning and rules of business standards, unifies the standard model for data integration, reduces the cost of data integration for different enterprises, and provides metadata for implementing this data acquisition engine process. It adapts to different process scenarios to enable different task processing flows, reducing the workload of data integration for different data integration scenarios.

Description

Technical Field

[0001] This invention relates to the fields of metadata, databases, data integration, and computer systems, and in particular to an integration system based on metadata-driven systems. Background Technology

[0002] With the development of the Internet, the amount of data collected and used by enterprises each year has increased exponentially. The data in the enterprise big data environment is diverse in form and lacks uniformity in standards. It has become difficult to collect, disseminate and share these different types of data, which requires enterprises to manage these data in a unified standard.

[0003] In the field of data integration, many enterprises and organizations have conducted a great deal of beneficial exploration. For example, from the early enterprise-level data integration capabilities provided by Informatica and IBM, to Spark and Flink that integrate streaming and batch processing, and more recently, the exploration of data middle platforms, all of these have provided some impetus to the DT world. However, they have not changed the reality that the workload of data integration accounts for 70% of the total workload of data management. In fact, some of these new explorations have even technicalized data integration work, making business-driven data management increasingly distant and distancing business personnel from the process of exploring the value of data.

[0004] Current data integration solutions typically employ three integration methods: ETL data sharing, database sharing, and API sharing. The main problems are as follows:

[0005] 1. Fragmented enterprise information makes integration and management difficult. Enterprise data is typically fragmented, with a large number of systems, relationships between them, and corresponding tables that are difficult for enterprises to clarify. Consequently, automated collection of enterprise data, service information, and business information is not possible, making centralized management of enterprise information assets and data interaction and sharing challenging.

[0006] 2. Slow response to business data issues. In enterprises, business personnel often discover problems in analysis reports and request the IT department to make modifications. However, due to the long data processing chain, modifications may involve multiple departments or even the entire company, making it difficult to accurately locate the relevant tables and fields of the problematic data.

[0007] 3. High data integration costs. Data becomes disordered when the data model changes; the integration of structured, semi-structured, and unstructured data sources involves a large workload. Summary of the Invention

[0008] To overcome the shortcomings and deficiencies of existing technologies, this invention addresses the problem of separating enterprise data integration platforms from enterprise business operations. It introduces metadata-driven enterprise data integration, automatically generating enterprise data models adapted to the business models from the enterprise's generated business metadata through a categorized and layered business parsing model. These data models are then broken down into various business standard models within the enterprise data integration platform, enabling intelligent adaptation of the platform to enterprise business operations, improving data integration quality, and reducing integration costs. This invention provides a metadata-driven integration platform.

[0009] The technical solution adopted in this invention is a metadata-driven integration platform, which includes: an overall architecture module, a logical structure module, and an execution flow module.

[0010] The overall architecture module is used to provide enterprise users with corresponding integrated operations for datasets, including data visualization configuration, process visualization configuration, general capabilities, and basic capabilities.

[0011] The logical structure module includes a platform access layer component, a platform business layer component, a platform adaptation layer component, a platform execution layer component, a platform protocol layer component, and a platform management system component.

[0012] Furthermore, the data visualization configuration is used to generate business metadata, which includes metadata of the collected data source and metadata of the target data source, as well as the transformation mapping relationship between them.

[0013] Furthermore, the process visualization configuration is used to generate process metadata, wherein the process metadata includes a configurable dataset applicable to all job task details.

[0014] Furthermore, the general capabilities used for abstracting integration capabilities and reusing them to adapt to different integration scenarios include extension points, adapters, protocol converters, and field mappers.

[0015] Furthermore, the aforementioned basic capabilities are used to support the triggering of data integration processes and the handling of exceptions, including permissions, logging, scheduling, rate limiting, and retry strategies.

[0016] Furthermore, the platform access layer component is the service call entry point. It needs to encapsulate the call logic as a service, perform route adaptation, and complete some secure calls and access control.

[0017] Furthermore, the platform business layer components integrate the implementation of the platform's core functions, mainly including data exchange calls, service calls, job flow references, and metadata references.

[0018] Furthermore, the platform adaptation layer component is a crucial part of the integration platform, primarily comprising an adapter for receiving and converting data, an adapter for sending data, and a converter for data processing. It adapts to different system data formats and content through business metadata and unifies the output data standards within the system; it implements data parsing, structure conversion, field mapping, and rule execution functions through process metadata.

[0019] Furthermore, the platform execution layer component serves as the initiator of data retrieval or push, encompassing real-time synchronous calls, parallel asynchronous calls, and timed task calls. The execution layer manages all call execution logic and also supports trigger logic, allowing external execution to be triggered by triggers.

[0020] Furthermore, the platform protocol layer component integrates different interface protocols that are compatible with and adaptable to different systems, supporting RESTful, DB, MQ protocols, as well as JSON and XML data formats.

[0021] Furthermore, the management system components are capable of flexibly configuring each component within the integration platform.

[0022] Furthermore, the data integration refers to synchronizing data from one party to another, where the two parties are an internal system and an external system, or between two internal systems, or even between two external systems.

[0023] Furthermore, the data exchange adapts to various data protocols of the data source and the target, synchronizing the source database to the target database, or synchronizing the source API interface to the target database, or synchronizing the source API interface to the target API interface. The data exchange process is based on data links and implemented through job task scheduling. Each data exchange is a task, which is scheduled and triggered for execution by a distributed task component at regular intervals. The execution process is based on the configuration of the data link.

[0024] Furthermore, the data link execution includes:

[0025] 1) Read adapter: This abstracts the read adapter interface and routes the content to the corresponding read adapter through protocol metadata.

[0026] 2) Data channels are used to send data after it is read. Data channels are divided into two types: asynchronous channels, which are typically used for data exchange scenarios with large volumes and high concurrency; and synchronous channels, which are used for scenarios with small volumes of data that require fast reading and storage.

[0027] 3) Data transformation, used for data transformation before writing, including data format conversion, data mapping, and data processing. This process is implemented by the data converter in conjunction with process metadata. The data converter is plug-in on the writing end and can be flexibly expanded according to the actual needs of the scenario. Supported data transformation methods include: field mapping transformation or complex data transformation processing based on rule engine.

[0028] 4) Data writing end: The data writing end and the data reading end are separate, which helps to expand horizontally for each. If the reading end is not enough, the reading node can be expanded, and if the writing end is not enough, the writing node can be expanded. Like the reading end, the writing end also has writing adapters for various protocols.

[0029] Beneficial effects:

[0030] 1) A metadata-driven data integration system establishes a panoramic view of data flow through enterprise business metadata, fundamentally solving the problem of fragmented enterprise data information, which is conducive to centralized management of enterprise data assets and better completion of data exchange and sharing;

[0031] 2) A metadata-driven data integration system generates a wide range of configurations by parsing, transforming, and mapping metadata. These values ​​meet the needs of all data exchange steps, thus completing automated data integration.

[0032] 3) Metadata-driven data integration: By providing the meaning and rules of business standards through business standard metadata, a unified standard model for data integration is established, reducing the cost of data integration for different enterprises;

[0033] 4) Metadata-driven data integration provides a metadata framework for implementing the data acquisition engine process, adapting to different task processing flows and reducing the workload of data integration for different data integration scenarios. Attached Figure Description

[0034] Figure 1 This is a diagram illustrating the overall architecture of the present invention;

[0035] Figure 2 This is a logical structure diagram of the present invention;

[0036] Figure 3 This is a flowchart of the data acquisition process of the present invention;

[0037] Figure 4 This is a flowchart of the push process of the present invention. Detailed Implementation

[0038] It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other. The following describes the application in further detail with reference to the accompanying drawings and specific embodiments.

[0039] The metadata-driven integration platform includes: an overall architecture module, a logical structure module, and an execution flow module.

[0040] like Figure 1 As shown, the overall architecture module is used to provide corresponding integration operations for enterprise users, including data visualization configuration, process visualization configuration, general capabilities, and basic capabilities.

[0041] The logical structure modules include platform access layer components, platform business layer components, platform adaptation layer components, platform execution layer components, platform protocol layer components, and platform management system components;

[0042] The dataset provides enterprise users with corresponding integrated operations, including data visualization configuration, process visualization configuration, and viewing of task execution data.

[0043] Data visualization configuration generates business metadata, which includes metadata from the collected data source and metadata from the target data source, as well as the transformation mapping relationship between them.

[0044] The process is visualized and configured, generating process metadata, which includes a configurable dataset that applies to all job task details.

[0045] General capabilities, abstracted and integrated general capabilities, can be reused and adapted to different integration scenarios, including extension points, adapters, protocol converters, field mappers, etc.

[0046] Basic capabilities support the triggering of data integration processes and exception handling, including permissions, logging, scheduling, rate limiting, retry strategies, etc.

[0047] like Figure 2 As shown, the data integration platform is roughly divided into five layers, from top to bottom: platform access layer components, platform business layer components, platform adaptation layer components, platform execution layer components, and platform protocol layer components. In addition, the integration platform should also have a management function to control the operation of the entire platform, as detailed below:

[0048] 1) The platform access layer component is the service call entry point. It needs to encapsulate the call logic as a service, adapt the routing, and complete some secure calls and access control.

[0049] 2) Platform business layer components are the implementation of the platform's core functions, mainly including data exchange, service invocation, job flow reference, and metadata reference.

[0050] 3) The platform adaptation layer is a crucial component of the integration platform, primarily consisting of adapters for receiving and converting data, adapters for sending data, and converters for data processing. It adapts to different system data formats and content through business metadata and ensures unified data standards within the system; it also enables data parsing, structure conversion, field mapping, and rule execution through process metadata.

[0051] 4) Platform Execution Layer Components: The execution layer is where data retrieval or push is initiated. It includes real-time synchronous calls, as well as parallel asynchronous calls or timed task calls. The execution layer manages all call execution logic and also supports trigger logic, allowing external execution to be triggered by triggers.

[0052] 5) Platform protocol layer components, which integrate platform compatibility and adaptability to different interface protocols of different systems. Generally speaking, they should support common protocols such as RESTful, DB, MQ, as well as JSON and XML data formats.

[0053] 6) Platform management system components, which enable flexible configuration of various aspects of the integrated platform.

[0054] The core purpose of data integration is to synchronize data from one party to another. These two parties can be internal and external systems, or between two internal systems, or even between two external systems.

[0055] The data exchange adapts to various data protocols of the data source and the target. The source and target protocols can also be combined, that is, the source database can be synchronized to the target database, the source API interface can be synchronized to the target database, or the source API interface can be synchronized to the target API interface.

[0056] The data exchange process is based on data links and implemented through job task scheduling. Each data exchange is a task, which is scheduled and triggered for execution periodically by a distributed task component. The execution process is based on the configuration of the data link.

[0057] The data link execution mainly consists of three parts:

[0058] 1) Read adapter: Abstract the read adapter interface, and route to the corresponding read adapter through protocol metadata to realize content reading.

[0059] 2) Data Channels: After data is read, it will be sent to the data channel. There are two types of data channels: one is the asynchronous channel, which is usually used for data exchange scenarios with large data volumes and high concurrency. The other is the synchronous channel, which is used for scenarios with small data volumes where reading and storage are fast.

[0060] 3) Data Transformation: Before writing, data transformation is required, including data format conversion, data mapping, and data processing. This process is implemented by the data converter in conjunction with process metadata. The data converter is a plug-in on the writing end, which can be flexibly expanded according to the actual needs of the scenario. Supported data transformation methods include: field mapping transformation or complex data transformation processing based on a rule engine.

[0061] 4) Data writing end: The data writing end and data reading end are separate, which facilitates horizontal scaling for each. If the reading end is insufficient, additional reading nodes can be added; if the writing end is insufficient, additional writing nodes can be added. Like the reading end, the writing end also has write adapters for various protocols.

[0062] like Figure 3 As shown, the data collection process includes: selecting data source type, configuring data source, testing data source, selecting data source, parameter configuration, parsing original data configuration, selecting target data source, mapping metadata configuration, process metadata configuration, generating job tasks, scheduling frequency, task testing, saving tasks, starting tasks, task detection, task alarms, task recovery, and manual execution.

[0063] like Figure 4 As shown, the push process includes: selecting event type, configuring event, testing event, selecting event, parameter configuration, parsing metadata configuration, selecting receiving source, mapping metadata configuration, process metadata configuration, generating job task, scheduling frequency, task testing, saving task, starting task, task detection, task alarm, task recovery, and manual execution.

[0064] Metadata-driven data integration systems establish a panoramic view of data flow through enterprise business metadata, fundamentally solving the problem of fragmented enterprise data information, which is conducive to centralized management of enterprise data assets and better completion of data exchange and sharing.

[0065] Metadata-driven data integration systems generate a wide range of configurations by parsing, transforming, and mapping metadata. These values ​​meet the needs of all data exchange steps, thus achieving automated data integration.

[0066] Metadata-driven data integration provides the meaning and rules of business standards through business standard metadata, unifies the standard model of data integration, and reduces the cost of data integration for different enterprises.

[0067] Metadata-driven data integration provides a set of metadata for implementing the data acquisition engine process through process metadata, adapting to different task processing flows to reduce the workload of data integration when facing different data integration scenarios.

[0068] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "set," "install," "connect," "link," and "fix" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal communication between two components. Those skilled in the art will understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0069] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various equivalent changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A metadata-driven integration system, characterized in that... The system includes: an overall architecture module, a logical structure module, and an execution flow module; The overall architecture module is used to provide enterprise users with corresponding integrated operations for datasets, including data visualization configuration, process visualization configuration, general capabilities, and basic capabilities. The logical structure module includes a platform access layer component, a platform business layer component, a platform adaptation layer component, a platform execution layer component, a platform protocol layer component, and a platform management system component. The data visualization configuration is used to generate business metadata, which includes metadata of the collected data source and metadata of the target data source, as well as the transformation mapping relationship between them. The process visualization configuration is used to generate process metadata, which includes a configurable dataset applicable to all job task details; The general capabilities mentioned above are used to abstract general capabilities for integration, and reuse and adapt to different integration scenarios, including extension points, adapters, protocol converters, and field mappers. The aforementioned basic capabilities are used to support the triggering of data integration processes and the handling of exceptions, including permissions, logging, scheduling, rate limiting, and retry strategies. The platform access layer component is the service call entry point. It needs to encapsulate the call logic as a service, perform route adaptation, and complete some secure calls and access control. The platform business layer components integrate the implementation of the platform's core functions, including data exchange, service invocation, job flow reference, and metadata reference. The platform adaptation layer component is an important part of the integration platform, including an adapter for receiving and converting data, an adapter for sending data, and a converter for data processing; it adapts to the data formats and content of different systems through business metadata and outputs data in a unified manner within the system; and it implements data parsing, structure conversion, field mapping, and rule execution functions through process metadata. The platform execution layer component is the initiator of data retrieval or push. It includes real-time synchronous calls, as well as parallel asynchronous calls or timed task calls. The execution layer manages all call execution logic and also supports trigger logic, allowing external execution to be triggered by triggers. The platform protocol layer component integrates different interface protocols that are compatible with and adaptable to different systems, and supports RESTful, DB, MQ protocols, as well as JSON and XML data formats; The management system components can be flexibly configured for each link in the integration platform; The data integration refers to synchronizing data from one party to another, where the two parties are an internal system and an external system, or between two internal systems, or even between two external systems. The data exchange adapts to various data protocols of the data source and the target, synchronizing the source database to the target database, or synchronizing the source API interface to the target database, or synchronizing the source API interface to the target API interface. The data exchange process is based on the data link and implemented by job task scheduling. Each data exchange is a task, which is scheduled and triggered for execution by the distributed task component at regular intervals. The execution process is based on the configuration of the data link. The data link execution includes: 1) Read adapter: This abstracts the read adapter interface and routes the content to the corresponding read adapter through protocol metadata. 2) Data channels are used to send data after it is read. There are two types of data channels: one is the asynchronous channel, which is used to handle data exchange scenarios with large data volume and high concurrency; the other is the synchronous channel, which is used for scenarios with small data volume that are fast to read and fast to store. 3) Data transformation, used for data transformation before writing, including data format conversion, data mapping, and data processing. This process is implemented by the data converter in conjunction with process metadata. The data converter is plug-in on the writing end and can be flexibly expanded according to the actual needs of the scenario. Supported data transformation methods include: field mapping transformation or complex data transformation processing based on rule engine. 4) Data writing end: The data writing end and the data reading end are separate, which helps to expand horizontally for each. If the reading end is not enough, the reading node can be expanded, and if the writing end is not enough, the writing node can be expanded. Like the reading end, the writing end also has writing adapters for various protocols.

Images