Data integration method and device, computer equipment and readable storage medium

By calling source mode and table mode generation services during data integration, combined with automated converters, the problems of heterogeneous data source data structure differences and dynamic mode evolution are solved, and the automated adaptation and evolution of data are realized, thus reducing data The complexity of integration tasks and maintenance workload improve development efficiency.

CN119088865APending Publication Date: 2024-12-06CHINA TELECOM CORP LTD TECHNOLOGY INNOVATION CENTER +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411115444.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-14
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

During the data integration process, the data structure differences and dynamic pattern evolution of heterogeneous data sources lead to large maintenance workloads in data acquisition tasks, and traditional methods require manual configuration of mapping and rules, resulting in high development complexity, error-proneness, and difficulty in achieving automated synchronization. .

Method used

By calling source mode generation services and table mode generation services, the data on the source and destinations are enhanced in mode mode, and the structure mapping and conversion are used to achieve automated adaptation and evolution of data modes, reducing configuration complexity, and improving development efficiency.

Benefits of technology

The automated integration of data mode is realized, the failure rate and maintenance workload of data integration tasks are reduced, the development efficiency is improved, and the general processing and automation problems of heterogeneous data integration are solved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119088865A_ABST
    Figure CN119088865A_ABST
Patent Text Reader

Abstract

The invention relates to a data integration method and device, computer equipment, a computer readable storage medium and a computer program product. By calling the source mode generation service and the table mode generation service, mode enhancement is performed on data of a source end and a destination end, so that a basis is provided for adaptation and evolution of a data mode in a subsequent automatic processing data integration task. Therefore, the change of the data mode can be automatically processed while the data change synchronization is realized, so that the data integration with the automatic mode can be realized. Mapping of tables and fields is not needed, data processing rules do not need to be configured, and configuration files do not need to be modified for evolution of a table structure, so that extremely simplified configuration of a data integration task is realized. Through the automatic converter, various heterogeneous data sources can be automatically converted on the basis of a data channel specification. On the whole, the development efficiency of the data integration task can be improved, the failure rate of data integration is reduced, and the maintenance workload is reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of database technology, and in particular to a data integration method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] As enterprises continue to develop and operate, information systems continue to increase, and enterprise data will be distributed in various heterogeneous data sources, making data sharing very difficult. Therefore, it is necessary to establish a data middle platform to achieve data sharing, and establishing a data middle platform requires data integration first. Since heterogeneous data sources themselves have various data structures during data integration, there will be significant differences from the destination data source, and the data model will also evolve dynamically during the data integration process. Therefore, how to automatically synchronize the dynamically evolving data model when collecting data from heterogeneous data sources and reduce the workload of maintaining data collection tasks is a big challenge. Summary of the invention

[0003] Based on this, it is necessary to provide a data integration method, apparatus, computer equipment, computer-readable storage medium and computer program product that can reduce the workload of data integration in response to the above technical problems.

[0004] In a first aspect, the present application provides a data integration method, comprising:

[0005] Create a data integration task, which includes the link information of the source end and the destination end;

[0006] In the process of executing data integration tasks, data is extracted from the source end according to the link information to obtain the target data;

[0007] Call the source schema generation service to enhance the target data schema and obtain the source schema data;

[0008] Through the automated converter, the source schema data is mapped and converted from the source end to the destination end to obtain the intermediate data;

[0009] Call the table schema generation service to enhance the schema of the intermediate data and obtain the table schema data;

[0010] Write the table mode data to the destination according to the link information.

[0011] In one embodiment, a source schema generation service is called to perform schema enhancement on target data to obtain source schema data, including:

[0012] Generate source data model according to the data source of target data;

[0013] According to the source data model, the target data is converted into source model data, and the source model data includes data content, data model and metadata information of the source.

[0014] In one embodiment, generating a source data model according to a data source of target data includes:

[0015] When the data source of the target data is a structured data source, a pattern is extracted from the data source to obtain a data pattern of the data source and use it as a source-end data pattern;

[0016] When the data source of the target data is an unstructured data source, a source-side data model is generated according to the data format in the data source.

[0017] In one embodiment, the source schema data is mapped and converted from the source end to the target end by an automated converter to obtain intermediate data, including:

[0018] Map and convert the table names and data field names in the source schema data according to the naming conventions of the data center.

[0019] According to the data modeling specifications of the data center, map the data types in the source model data with the data types in the data center, and convert them according to the mapping relationship;

[0020] The data fields in the source mode data are enhanced to obtain intermediate data.

[0021] In one embodiment, a table schema generation service is called to perform schema enhancement on intermediate data to obtain table schema data, including:

[0022] Generate the destination data model based on the existing table structure of the destination;

[0023] According to the source data model, the intermediate data is converted into table model data. The table model data includes data content, data model, metadata information of the destination, and table structure evolution information.

[0024] In one embodiment, the method further comprises:

[0025] Data integration tasks are controlled through the management interface of the task execution engine.

[0026] In a second aspect, the present application also provides a data integration device, comprising:

[0027] A creation module is used to create a data integration task, which includes the link information of the source end and the destination end;

[0028] The extraction module is used to extract data from the source end according to the link information to obtain the target data during the execution of the data integration task;

[0029] The first calling module is used to call the source pattern generation service to perform pattern enhancement on the target data to obtain the source pattern data;

[0030] A conversion module is used to map and convert the source mode data from the source end to the destination end through an automated converter to obtain intermediate data;

[0031] The second calling module is used to call the table schema generation service to perform schema enhancement on the intermediate data to obtain table schema data;

[0032] The writing module is used to write the table mode data to the destination according to the link information.

[0033] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0034] Create a data integration task, which includes the link information of the source end and the destination end;

[0035] In the process of executing data integration tasks, data is extracted from the source end according to the link information to obtain the target data;

[0036] Call the source schema generation service to enhance the target data schema and obtain the source schema data;

[0037] Through the automated converter, the source schema data is mapped and converted from the source end to the destination end to obtain the intermediate data;

[0038] Call the table schema generation service to enhance the schema of the intermediate data and obtain the table schema data;

[0039] Write the table mode data to the destination according to the link information.

[0040] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the following steps are implemented:

[0041] Create a data integration task, which includes the link information of the source end and the destination end;

[0042] In the process of executing data integration tasks, data is extracted from the source end according to the link information to obtain the target data;

[0043] Call the source schema generation service to enhance the target data schema and obtain the source schema data;

[0044] Through the automated converter, the source schema data is mapped and converted from the source end to the destination end to obtain the intermediate data;

[0045] Call the table schema generation service to enhance the schema of the intermediate data and obtain the table schema data;

[0046] Write the table mode data to the destination according to the link information.

[0047] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0048] Create a data integration task, which includes the link information of the source end and the destination end;

[0049] In the process of executing data integration tasks, data is extracted from the source end according to the link information to obtain the target data;

[0050] Call the source schema generation service to enhance the target data schema and obtain the source schema data;

[0051] Through the automated converter, the source schema data is mapped and converted from the source end to the destination end to obtain the intermediate data;

[0052] Call the table schema generation service to enhance the schema of the intermediate data and obtain the table schema data;

[0053] Write the table mode data to the destination according to the link information.

[0054] The above-mentioned data integration method, device, computer equipment, computer-readable storage medium and computer program product, by calling the source mode generation service and the table mode generation service, enhances the mode of the data at the source end and the destination end, thereby providing a basis for the adaptation and evolution of the data mode in the subsequent automated processing of the data integration task. Thus, while realizing the synchronization of data changes, the change of the data mode can be automatically processed, so that the data integration of mode automation can be realized. At the same time, because there is no need to map tables and fields, there is no need to configure data processing rules, and the evolution of the table structure does not need to modify the configuration file, thereby realizing the extremely simplified configuration of the data integration task. In addition, through the automatic converter, various heterogeneous data sources can be automatically converted based on the data middle platform specification. Therefore, the overall development efficiency of the data integration task can be improved, the failure rate of data integration can be reduced, and the maintenance workload can be reduced. Finally, various heterogeneous data sources need to be integrated into a unified data warehouse. There are significant differences in the data and modes between various sources, and there are also significant differences in the data and modes of the target data warehouse data source. The unified description of data through the source mode and the table mode can solve the problem of general processing difficulty and automation difficulty of heterogeneous data integration. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0056] FIG1 is an application environment diagram of a data integration method in one embodiment;

[0057] FIG2 is a schematic diagram of a flow chart of a data integration method in one embodiment;

[0058] FIG3 is a framework diagram of a data integration system in one embodiment;

[0059] FIG4 is a schematic diagram of a flow chart of a data integration method in another embodiment;

[0060] FIG5 is a schematic diagram of the life cycle of a data integration task in one embodiment;

[0061] FIG6 is a schematic diagram of an execution flow of an automated converter in one embodiment;

[0062] FIG7 is a block diagram of a data integration device in one embodiment;

[0063] FIG. 8 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0065] As enterprises continue to grow and operate, information systems continue to increase, and enterprise data will be distributed in various heterogeneous data sources, making data sharing very difficult. This is followed by the problem of data silos. In order to solve the problem of data silos, tap the value of data, and improve the speed, quality, and transparency of decision-making, the construction of a data middle platform is essential. To build a data middle platform, data integration must be carried out first, integrating all data in order to unify data services and sharing.

[0066] Data integration is the integration of various heterogeneous data sources. When performing data integration, the destination and source of the data are also heterogeneous data sources. These heterogeneous data sources themselves have various data structures and types, which are significantly different from the destination data sources. At the same time, the data model will also evolve dynamically during the data integration process. Therefore, it is a great challenge to automatically synchronize the dynamically evolving data model when collecting data from heterogeneous data sources and reduce the workload of maintaining data collection tasks.

[0067] If we follow the traditional approach, we will generally perform source-to-destination configuration mapping for each table and each field in the data source. For different data types, type changes, and changes in table structures, such as adding fields, changing field types, or changing field lengths, if we follow the traditional approach, we must perform specific coding work to achieve compatibility and adaptation. As a result, data integration tasks are subject to heavy development workload, high complexity, prone to errors, and difficult maintenance, which has become an efficiency bottleneck in the construction of the data center.

[0068] To this end, the data integration method provided in the embodiment of the present application can be applied to the application environment shown in Figure 1. Among them, the source server 102, the destination server 104 and the data middle station server 106 communicate through the network. The data middle station server 106 creates a data integration task. In the process of executing the data integration task, according to the link information in the data integration task, data is extracted from the source server 102 to obtain the target data. The data middle station server 106 calls the source mode generation service to enhance the mode of the target data to obtain the source mode data; through the automatic converter, the source mode data is mapped and converted from the source end to the destination end structure to obtain the intermediate data; the table mode generation service is called to enhance the mode of the intermediate data to obtain the table mode data. The data middle station server 106 writes the table mode data to the destination server 104 according to the link information. Among them, the source server 102, the destination server 104 and the data middle station server 106 can be independent physical servers, or they can be a server cluster or distributed system composed of multiple physical servers, or they can be a cloud server providing cloud computing services.

[0069] In an exemplary embodiment, as shown in FIG2 , a data integration method is provided, which is described by taking the method applied to the data middle station server 106 in FIG1 as an example, and includes the following steps 202 to 212. Among them:

[0070] Step 202: Create a data integration task, which includes link information of the source end and the destination end.

[0071] Among them, data integration is to logically or physically integrate data of different sources, formats, characteristics and properties, so as to provide comprehensive data sharing. When creating a data integration task, it can be achieved through a configuration file. Different from the related art that requires the configuration of extractors, loaders and converters, the embodiment of the present application does not require a converter, but realizes automatic conversion through an automatic converter.

[0072] The data integration task may include the link information of the source end and the destination end, and the link information is mainly used to indicate the address of the database. In addition, data filtering rules may also be included to filter part of the data during extraction, which is not specifically limited in the present embodiment of the application.

[0073] Data integration tasks can be configured by relevant personnel as needed, and data integration tasks can be added, deleted or modified. In actual implementation, data integration tasks can be configured through code, such as including task name, database type of the source end, database link of the source end, account name and password, database type of the destination end, database link of the destination end, account name and password, etc., which are not specifically limited in the embodiments of the present application. Overall, configuring data integration tasks does not require mapping of configuration tables and fields, nor does it require configuring data processing rules. Data structure evolution does not require modifying configuration files, thereby improving development efficiency and reducing maintenance workload.

[0074] Step 204, in the process of executing the data integration task, data is extracted from the source end according to the link information to obtain the target data.

[0075] Specifically, data integration tasks can be executed by an execution engine, which can provide a running environment for data integration tasks. Data extraction refers to the act of extracting data from various data sources, which may include internal systems such as resource planning systems and relationship management systems, or external data sources such as social media and publicly released data sets. The purpose of data extraction is to prepare raw materials for subsequent data processing and analysis.

[0076] Step 206, calling the source schema generation service, performing schema enhancement on the target data, and obtaining source schema data.

[0077] The source schema generation service can uniformly describe the schema of the data extracted by the extractor to facilitate subsequent parsing and conversion according to the data center specifications. The source schema data can be represented by code, which can include: the name, type, length, whether it is the primary key and whether it is optional for each field; the ID, name and creation time of the data; the task name, source database type, database name, table name, operation type and timestamp of the metadata.

[0078] Step 208, using an automated converter, performs structure mapping and conversion on the source schema data from the source end to the destination end to obtain intermediate data.

[0079] Among them, the automatic converter can automatically convert the data table name, field name, data type and table structure in the source end according to the data format requirements of the data center. After the automatic converter, the data model and data content will form a unified view, which provides the basis for the subsequent table model generation service and the automatic processing of the loader.

[0080] Step 210, calling the table schema generation service, performing schema enhancement on the intermediate data, and obtaining table schema data.

[0081] The table schema generation service can re-describe the intermediate data according to the table structure of the destination end, so as to facilitate the subsequent writing of data through the loader. The table schema data can be represented by code, which can include: the name, type, length, whether it is the primary key and whether it is optional for each field; the ID, name and creation time of the data; the task name, source database type, database name, table name, operation type and timestamp of the metadata.

[0082] Step 212, write the table mode data to the destination end according to the link information.

[0083] Specifically, DDL (Data Definition Language) and DML (Data Manipulation Language) statements can be used to change the structure of the data table on the target end and store the data content in the table model data into the database.

[0084] In the above data integration method, by calling the source mode generation service and the table mode generation service, the data at the source and destination ends are enhanced in mode, thereby providing a basis for the adaptation and evolution of the data mode in the subsequent automated processing of data integration tasks. Thus, while realizing data change synchronization, the change of the data mode can be automatically processed, so that data integration with mode automation can be realized. At the same time, since there is no need to map tables and fields, there is no need to configure data processing rules, and the evolution of the table structure does not require modification of the configuration file, thereby realizing a very simplified configuration of the data integration task. In addition, through the automatic converter, various heterogeneous data sources can be automatically converted based on the data middle platform specification. Therefore, the overall development efficiency of data integration tasks can be improved, the failure rate of data integration can be reduced, and the maintenance workload can be reduced. Finally, various heterogeneous data sources need to be integrated into a unified data warehouse. There are significant differences in data and modes between various sources, and there are also significant differences in data and modes between the target data warehouse data source. The unified description of data through source mode and table mode can solve the problem of general processing difficulty and automation difficulty of heterogeneous data integration.

[0085] In an exemplary embodiment, calling a source schema generation service to perform schema enhancement on target data to obtain source schema data includes:

[0086] Generate source data model according to the data source of target data;

[0087] According to the source data model, the target data is converted into source model data, and the source model data includes data content, data model and metadata information of the source.

[0088] Specifically, the source data model refers to the way to describe the source data. For details, please refer to the content contained in the source model data when it is represented by code in the previous example. Metadata information refers to information used to describe data attributes, such as the task name, source database type, database name, table name, operation type, and timestamp mentioned in the metadata.

[0089] In this embodiment, since the source data can be enhanced by calling the source pattern generation service, a basis is provided for the subsequent automatic adaptation and evolution of the data pattern to realize pattern-automated data integration.

[0090] In an exemplary embodiment, generating a source data model according to a data source of target data includes:

[0091] When the data source of the target data is a structured data source, a pattern is extracted from the data source to obtain a data pattern of the data source and use it as a source-end data pattern;

[0092] When the data source of the target data is an unstructured data source, a source-side data model is generated according to the data format in the data source.

[0093] Specifically, a structured data source refers to data as a source that has a fixed format and limited length and can be described and stored through a unified data structure and specification, such as a common database. An unstructured data source refers to data as a source that is irregular or incomplete, has no predefined data model, and is not convenient to represent using a two-dimensional logical table in a database, such as documents, text, reports, images, and videos. If the data source of the target data is a structured data source, the corresponding data model can be used directly. If the data source of the target data is an unstructured data source, a source-side data model can be generated. For example, taking video data as an example, metadata of the video data, such as file name, file type, or file size, can be generated as content in the source-side data model.

[0094] In this embodiment, since the corresponding source-end data mode can be adapted according to the type of data source, a basis can be provided for subsequent mode enhancement of the source-end data to realize mode-automated data integration.

[0095] In an exemplary embodiment, the source schema data is mapped and converted from the source end to the target end by an automated converter to obtain intermediate data, including:

[0096] Map and convert the table names and data field names in the source schema data according to the naming conventions of the data center.

[0097] According to the data modeling specifications of the data center, map the data types in the source model data with the data types in the data center, and convert them according to the mapping relationship;

[0098] The data fields in the source mode data are enhanced to obtain intermediate data.

[0099] Specifically, the table names and field names in the source schema data can be extracted and converted according to the naming specifications of the data center. For example, if the naming specifications of the data center require that the table and field names be named using underscores, and the source schema data uses camel case naming, the camel case table names and field names in the source schema data can be modified to underscore names. If the naming specifications require lowercase letters, and the source schema data uses uppercase letters, the uppercase letters in the table and field names in the source schema data can be modified to lowercase letters. If the naming specifications require adding a system prefix, the system prefix can be added to the table and field names in the source schema data.

[0100] Furthermore, because the data types supported by the data center when storing data may differ from those supported by the source end, a mapping can be established between the data types in the source end and the data types supported by the data center, and the data types in the source model data can be converted to the data types supported by the data center according to the mapping. Finally, due to the continuous upgrade and iteration of the business system, the data table structure in the source model data will also change frequently, such as adding fields, changing field types, and changing field lengths, so that the data fields can be enhanced according to the actual changes to adapt to the changes.

[0101] In this embodiment, the automatic converter can automatically convert various heterogeneous data sources based on the data center specification. Therefore, the overall development efficiency of data integration tasks can be improved, the failure rate of data integration can be reduced, and the maintenance workload can be reduced.

[0102] In an exemplary embodiment, calling a table schema generation service to perform schema enhancement on intermediate data to obtain table schema data includes:

[0103] Generate the destination data model based on the existing table structure of the destination;

[0104] According to the source data model, the intermediate data is converted into table model data. The table model data includes data content, data model, metadata information of the destination, and table structure evolution information.

[0105] Specifically, the destination data model refers to the way to describe the source data. For details, please refer to the content contained in the table model data represented by code in the previous example. The table model generation service is called mainly to automatically realize the conversion from the source to the destination. When the table model data is used for subsequent actual writing to the data table in the destination, the model can be automatically evolved and adapted to achieve the effect of automatically changing the processing mode. Among them, the table structure evolution information is the information used to describe how the structure of the data table changes.

[0106] In this embodiment, since the table schema generation service can be called to enhance the schema of the data at the destination end, a basis is provided for the subsequent automatic adaptation and evolution of the data schema to achieve automated data integration of the schema.

[0107] In an exemplary embodiment, the method further comprises:

[0108] Data integration tasks are controlled through the management interface of the task execution engine.

[0109] Specifically, the task execution engine can provide a management interface for the data center to control the data integration task, wherein the control method may include starting, canceling and stopping the data integration task.

[0110] In this embodiment, since the execution process can be controlled according to requirements during the execution of the data integration task, the execution process is more flexible.

[0111] For ease of understanding, the implementation process provided by the embodiment of the present application is now described. The method flow provided in the present application can be implemented based on a data integration system, and the structure of the data integration system can be referred to in Figure 3. In Figure 3, the Web end is the front end, which may include a task configuration interface and a task management interface for relevant personnel to configure tasks and manage tasks. The integration end is the specific functional layer, which may have configuration management functions and task management functions. The execution engine can provide a runtime environment for the execution of data integration tasks, and has the functions of an extractor, an automatic converter, and a loader. In addition, the functional layer also has a source model generation service and a table model generation service. The storage end is the device that actually stores data, which may specifically include a data middle platform storage facility.

[0112] Combined with the system in FIG3 , in actual implementation, the main business process of the system can refer to FIG4 , and the specific implementation can be shown in the following process steps:

[0113] (1) Developers manage task configurations and start tasks after creating data integration tasks.

[0114] (2) Execute the extractor and extract data from the source end as the data source.

[0115] (3) Call the source model generation service to automatically generate the source data model according to the data source and data content of the extracted data, and convert the extracted data into source model data according to the source data model.

[0116] (4) The automated converter receives the source model data, automatically processes the table structure mapping and evolution, and obtains the intermediate data.

[0117] (5) Call the table schema generation service to automatically generate the destination data schema based on the existing table structure on the destination side, and convert the intermediate data into table schema data.

[0118] (6) The loader executes DML and DDL statements to change the table structure and store data based on the table schema data.

[0119] (7) When developers need to manage data integration tasks, they can call the control interface to execute corresponding control instructions.

[0120] In addition, the life cycle of data integration tasks can refer to Figure 5. It is mainly divided into three stages, namely creating and configuring data integration tasks, executing data integration tasks, and managing data integration tasks. Among them, configuring data integration tasks mainly configures the link information of the source and destination ends, and can also configure data filtering rules. Executing data integration tasks mainly parses the configuration content to generate the execution class of the data integration task, and then executes the data integration task through the execution engine. Managing data integration tasks mainly controls the start, cancellation, or stop of data integration tasks through the management interface of the execution engine.

[0121] The execution process of the automatic converter can refer to Figure 6, and the specific process can refer to the following process:

[0122] Step 602, automatically map and convert the source table name and field name according to the naming convention of the data center.

[0123] Step 604, according to the data modeling specifications of the data center, automatically map and convert the data type of the source end and the data type of the data center storage end.

[0124] Step 606, perform field enhancement on the data at the source end.

[0125] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0126] Based on the same inventive concept, the embodiment of the present application also provides a data integration device for implementing the data integration method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more data integration device embodiments provided below can refer to the limitations on the data integration method above, and will not be repeated here.

[0127] In an exemplary embodiment, as shown in FIG. 7 , a data integration device is provided, including: a creation module 702, an extraction module 704, a first calling module 706, a conversion module 708, a second calling module 710 and a writing module 712, wherein:

[0128] A creation module 702 is used to create a data integration task, wherein the data integration task includes link information of the source end and the destination end;

[0129] The extraction module 704 is used to extract data from the source end according to the link information to obtain the target data during the execution of the data integration task;

[0130] The first calling module 706 is used to call the source pattern generation service to perform pattern enhancement on the target data to obtain source pattern data;

[0131] The conversion module 708 is used to perform source-end-to-destination structure mapping and conversion on the source schema data through an automated converter to obtain intermediate data;

[0132] The second calling module 710 is used to call the table schema generation service to perform schema enhancement on the intermediate data to obtain table schema data;

[0133] The writing module 712 is used to write the table mode data to the destination end according to the link information.

[0134] In an exemplary embodiment, the first calling module 706 is used to generate a source data model according to the data source of the target data; according to the source data model, the target data is converted into source model data, and the source model data includes data content, data model and source metadata information.

[0135] In an exemplary embodiment, the first calling module 706 is used to extract the pattern of the data source when the data source of the target data is a structured data source, obtain the data pattern of the data source, and use it as the source data pattern; when the data source of the target data is an unstructured data source, generate the source data pattern according to the data format in the data source.

[0136] In an exemplary embodiment, the conversion module 708 is used to map and convert the table names and data field names in the source model data according to the naming specifications of the data middle platform; map the data types in the source model data with the data types in the data middle platform according to the data modeling specifications of the data middle platform, and convert them according to the mapping relationship; enhance the data fields in the source model data to obtain intermediate data.

[0137] In an exemplary embodiment, the second calling module 710 is used to generate a destination data model based on the existing table structure of the destination; according to the source data model, the intermediate data is converted into table model data, and the table model data includes data content, data model and metadata information and table structure evolution information of the destination.

[0138] In an exemplary embodiment, the creation module 702 is further used to control the data integration task through the management interface of the task execution engine.

[0139] Each module in the above data integration device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0140] In an exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG8. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data to be integrated. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a data integration method is implemented.

[0141] Those skilled in the art will understand that the structure shown in FIG. 8 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0142] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0143] Create a data integration task, which includes the link information of the source end and the destination end;

[0144] In the process of executing data integration tasks, data is extracted from the source end according to the link information to obtain the target data;

[0145] Call the source schema generation service to enhance the target data schema and obtain the source schema data;

[0146] Through the automated converter, the source schema data is mapped and converted from the source end to the destination end to obtain the intermediate data;

[0147] Call the table schema generation service to enhance the schema of the intermediate data and obtain the table schema data;

[0148] Write the table mode data to the destination according to the link information.

[0149] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0150] Generate source data model according to the data source of target data;

[0151] According to the source data model, the target data is converted into source model data, and the source model data includes data content, data model and metadata information of the source.

[0152] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0153] When the data source of the target data is a structured data source, a pattern is extracted from the data source to obtain a data pattern of the data source and use it as a source-end data pattern;

[0154] When the data source of the target data is an unstructured data source, a source-side data model is generated according to the data format in the data source.

[0155] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0156] Map and convert the table names and data field names in the source schema data according to the naming conventions of the data center.

[0157] According to the data modeling specifications of the data center, map the data types in the source model data with the data types in the data center, and convert them according to the mapping relationship;

[0158] The data fields in the source mode data are enhanced to obtain intermediate data.

[0159] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0160] Generate the destination data model based on the existing table structure of the destination;

[0161] According to the source data model, the intermediate data is converted into table model data. The table model data includes data content, data model, metadata information of the destination, and table structure evolution information.

[0162] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0163] Data integration tasks are controlled through the management interface of the task execution engine.

[0164] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0165] Create a data integration task, which includes the link information of the source end and the destination end;

[0166] In the process of executing data integration tasks, data is extracted from the source end according to the link information to obtain the target data;

[0167] Call the source schema generation service to enhance the target data schema and obtain the source schema data;

[0168] Through the automated converter, the source schema data is mapped and converted from the source end to the destination end to obtain the intermediate data;

[0169] Call the table schema generation service to enhance the schema of the intermediate data and obtain the table schema data;

[0170] Write the table mode data to the destination according to the link information.

[0171] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0172] Generate source data model according to the data source of target data;

[0173] According to the source data model, the target data is converted into source model data, and the source model data includes data content, data model and metadata information of the source.

[0174] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0175] When the data source of the target data is a structured data source, a pattern is extracted from the data source to obtain a data pattern of the data source and use it as a source-end data pattern;

[0176] When the data source of the target data is an unstructured data source, a source-side data model is generated according to the data format in the data source.

[0177] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0178] Map and convert the table names and data field names in the source schema data according to the naming conventions of the data center.

[0179] According to the data modeling specifications of the data center, map the data types in the source model data with the data types in the data center, and convert them according to the mapping relationship;

[0180] The data fields in the source mode data are enhanced to obtain intermediate data.

[0181] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0182] Generate the destination data model based on the existing table structure of the destination;

[0183] According to the source data model, the intermediate data is converted into table model data. The table model data includes data content, data model, metadata information of the destination, and table structure evolution information.

[0184] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0185] Data integration tasks are controlled through the management interface of the task execution engine.

[0186] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0187] Create a data integration task, which includes the link information of the source end and the destination end;

[0188] In the process of executing data integration tasks, data is extracted from the source end according to the link information to obtain the target data;

[0189] Call the source schema generation service to enhance the target data schema and obtain the source schema data;

[0190] Through the automated converter, the source schema data is mapped and converted from the source end to the destination end to obtain the intermediate data;

[0191] Call the table schema generation service to enhance the schema of the intermediate data and obtain the table schema data;

[0192] Write the table mode data to the destination according to the link information.

[0193] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0194] Generate source data model according to the data source of target data;

[0195] According to the source data model, the target data is converted into source model data, and the source model data includes data content, data model and metadata information of the source.

[0196] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0197] When the data source of the target data is a structured data source, a pattern is extracted from the data source to obtain a data pattern of the data source and use it as a source-end data pattern;

[0198] When the data source of the target data is an unstructured data source, a source-side data model is generated according to the data format in the data source.

[0199] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0200] Map and convert the table names and data field names in the source schema data according to the naming conventions of the data center.

[0201] According to the data modeling specifications of the data center, map the data types in the source model data with the data types in the data center, and convert them according to the mapping relationship;

[0202] The data fields in the source mode data are enhanced to obtain intermediate data.

[0203] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0204] Generate the destination data model based on the existing table structure of the destination;

[0205] According to the source data model, the intermediate data is converted into table model data. The table model data includes data content, data model, metadata information of the destination, and table structure evolution information.

[0206] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0207] Data integration tasks are controlled through the management interface of the task execution engine.

[0208] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0209] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0210] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data integration method, It is characterized in that The method comprises: Creating a data integration task, wherein the data integration task includes link information of a source end and a destination end; In the process of executing the data integration task, data is extracted from the source end according to the link information to obtain target data; Calling a source pattern generation service to perform pattern enhancement on the target data to obtain source pattern data; By means of an automated converter, the source mode data is structure-mapped and converted from a source end to a destination end to obtain intermediate data; Calling a table schema generation service to perform schema enhancement on the intermediate data to obtain table schema data; The table mode data is written to the destination end according to the link information.

2. The method according to claim 1, It is characterized in that The calling of the source pattern generation service to perform pattern enhancement on the target data to obtain source pattern data includes: Generate a source data model according to the data source of the target data; According to the source data mode, the target data is converted into source mode data, and the source mode data includes data content, data mode and metadata information of the source.

3. The method according to claim 2, It is characterized in that The step of generating a source data model according to the data source of the target data includes: In the case where the data source of the target data is a structured data source, a pattern is extracted from the data source to obtain a data pattern of the data source and use the data pattern as a source data pattern; In the case where the data source of the target data is an unstructured data source, a source-end data model is generated according to the data format in the data source.

4. The method according to claim 1, It is characterized in that The automatic converter is used to map and convert the source mode data from the source end to the destination end to obtain intermediate data, including: Map and convert the table names and data field names in the source schema data according to the naming conventions of the data center; According to the data modeling specifications of the data center, the data types in the source mode data are mapped with the data types in the data center, and converted according to the mapping relationship; The data fields in the source mode data are enhanced to obtain intermediate data.

5. The method according to claim 1, It is characterized in that The calling of the table schema generation service to perform schema enhancement on the intermediate data to obtain table schema data includes: Generate a destination data model according to the existing table structure of the destination; According to the source-end data model, the intermediate data is converted into table-mode data, wherein the table-mode data includes data content, data model, metadata information of the destination end, and table structure evolution information.

6. The method according to claim 1, It is characterized in that The method further comprises: The data integration task is controlled through the management interface of the task execution engine.

7. A data integration device, It is characterized in that The device comprises: A creation module, used to create a data integration task, wherein the data integration task includes link information of a source end and a destination end respectively; An extraction module, used to extract data from the source end according to the link information to obtain target data during the execution of the data integration task; A first calling module is used to call a source pattern generation service to perform pattern enhancement on the target data to obtain source pattern data; A conversion module, used to perform structure mapping and conversion from a source end to a destination end on the source mode data through an automated converter to obtain intermediate data; A second calling module is used to call a table schema generation service to perform schema enhancement on the intermediate data to obtain table schema data; A writing module is used to write the table mode data to the destination end according to the link information.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program. It is characterized in that When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.