Data integration method, device, server, medium and program

Through the full and incremental data acquisition methods, the intermediate library and data integration statements are used to solve the problems of low data integration efficiency and inconsistent data form across databases, and efficient, accurate integration and model unification of data are achieved.

CN118626496BActive Publication Date: 2024-12-03HANGZHOU SHUOPAN INTELLIGENT TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411095117.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-12
Publication Date
2024-12-03
Estimated Expiration
2044-08-12

AI Technical Summary

Technical Problem

The existing data integration technology is inefficient, making it difficult to achieve complete and accurate integration of data across databases, and the data forms are inconsistent between different source databases.

Method used

Full and incremental data acquisition methods are adopted to migrate data through intermediate libraries, data integration statements are used to perform data type conversion and field mapping, ensuring data integrity and accuracy, and unifying the data models of different source databases.

Benefits of technology

It improves the efficiency and performance of the data integration process, achieves complete and accurate data integration, and unifies the data model of different source databases, reducing network load and storage requirements.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118626496B_ABST
    Figure CN118626496B_ABST
Patent Text Reader

Abstract

The embodiment of the present application provides a data integration method, device, server, medium and program, wherein the method includes: after configuring the connection information of the source database, through full data collection, the full data of the source database is pulled to the intermediate table of the intermediate library, and the data is pulled from the intermediate table to the target table of the target library; when there is data update in the source database, the incremental data of the source database is pulled to the intermediate table through incremental data collection, and the data is pulled from the intermediate table to the target table; for full data collection, the full data is collected to the intermediate table, and then according to the data integration statement, the data is pulled from the intermediate table to the target table; for incremental data collection, the incremental data is collected to the intermediate table according to the incremental update field, and then the data integration statement with the incremental update clause is used to pull the data from the intermediate table to the target table; the data is pulled from the intermediate table to the target table and data type conversion and field mapping are also performed. The embodiment of the present application can improve the efficiency of the data integration process.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data processing technology, and specifically to a data integration method, device, server, medium and program. Background Art

[0002] Data integration is the process of logically or physically integrating data from different data sources to provide comprehensive data sharing for enterprises. Data integration plays a core and fundamental role in digital transformation, where digital transformation refers to the process of enterprises using digital technology to transform traditional business processes, improve business models and enhance customer experience; in the process of digital transformation, data integration is the cornerstone of data-driven decision-making, automated operations and improved data processing efficiency. Therefore, how to provide improved technical solutions to improve the efficiency of the data integration process has become a technical problem that technicians in this field need to solve urgently. Summary of the invention

[0003] In view of this, embodiments of the present application provide a data integration method, device, server, medium and program to improve the efficiency of the data integration process.

[0004] To achieve the above objectives, the embodiments of the present application provide the following technical solutions.

[0005] In a first aspect, an embodiment of the present application provides a data integration method, including:

[0006] After configuring the connection information of the source database, an active query request is sent to the source database to query whether there are data records or data updates in the source database;

[0007] When there are data records in the source database, the full amount of data of the source database is pulled into the intermediate table of the intermediate database through full data collection, and then pulled from the intermediate table to the target table of the target database; the intermediate database is the database corresponding to the intermediate storage system of the data integration platform, and the target database is a database using the enterprise standard model;

[0008] When there is data update in the source database, the incremental data of the source database is pulled to the intermediate table of the intermediate database through incremental data collection, and then pulled from the intermediate table to the target table of the target database;

[0009] Among them, for full data collection, the full data of the source database is collected into the intermediate table of the intermediate library, and then according to the data integration statement, the full data is pulled from the intermediate table of the intermediate library to the target table of the target library; for incremental data collection, according to the incremental update field, the incremental data of the source database is collected into the intermediate table of the intermediate library, and then the data integration statement with the incremental update clause is used to pull the incremental data from the intermediate table of the intermediate library to the target table of the target library; the process of pulling data from the intermediate table to the target table also performs data type conversion and field mapping.

[0010] In a second aspect, an embodiment of the present application provides a data integration device, including:

[0011] A query request sending module is used to send an active query request to the source database after configuring the connection information of the source database to query whether there is data record or data update in the source database;

[0012] The full data integration module is used to pull the full data of the source database to the intermediate table of the intermediate database through full data collection when there are data records in the source database, and pull it from the intermediate table to the target table of the target database; the intermediate database is the database corresponding to the intermediate storage system of the data integration platform, and the target database is a database using the enterprise standard model;

[0013] The incremental data integration module is used to pull the incremental data of the source database to the intermediate table of the intermediate database through incremental data collection when there is data update in the source database, and then pull it from the intermediate table to the target table of the target database;

[0014] Among them, for full data collection, the full data integration module collects the full data of the source database into the intermediate table of the intermediate library, and then pulls the full data from the intermediate table of the intermediate library to the target table of the target library according to the data integration statement; for incremental data collection, the incremental data integration module collects the incremental data of the source database into the intermediate table of the intermediate library according to the incremental update field, and then uses the data integration statement with the incremental update clause to pull the incremental data from the intermediate table of the intermediate library to the target table of the target library; the process of pulling data from the intermediate table to the target table also performs data type conversion and field mapping.

[0015] In a third aspect, an embodiment of the present application provides a server, comprising: at least one memory and at least one processor, the memory storing computer executable instructions, the processor calling the computer executable instructions to execute the data integration method as described in the first aspect above.

[0016] In a fourth aspect, an embodiment of the present application provides a storage medium, wherein the storage medium stores computer-executable instructions, and when the computer-executable instructions are executed, the data integration method as described in the first aspect above is implemented.

[0017] In a fifth aspect, an embodiment of the present application provides a computer program product, comprising computer executable instructions, which, when executed, implement the data integration method as described in the first aspect above.

[0018] The embodiment of the present application can cover the data migration path from the source database to the target database through the process of full data collection and incremental data collection, and introduce the intermediate library as temporary storage to ensure the integrity and accuracy of the data during the migration process, so as to meet the data integration requirements corresponding to the data record or data update when actively querying the source database, and improve the efficiency of the data integration process. Specifically, after configuring the connection information of the source database, the embodiment of the present application can send an active query request to the source database, so that when querying the source database for data records, the full data of the source database can be pulled to the intermediate library, and then according to the data integration statement, the full data is pulled from the intermediate library to the target library, ensuring that the target library has a complete snapshot of the source database, which can be applied to the initialization or periodic complete synchronization of the source database and the target library; when the source database is updated, the incremental data is pulled to the intermediate library, and then the data integration statement with the incremental update clause is used to pull the incremental data from the intermediate library to the target library, reducing the network load and storage requirements, while ensuring the latest data status of the target library. Furthermore, in the process of transferring data from the intermediate library to the target library, data type conversion and field mapping are performed, and the problem of inconsistent data formats that may exist between different source databases can be solved through the enterprise standard model of the target library, thereby unifying the data models of different source databases. Therefore, the embodiment of the present application can effectively manage data integration across databases. Whether it is necessary to regularly synchronize the full amount of data in the source database or update the incremental data in the source database, the embodiment of the present application can achieve complete and accurate data integration, and unify the data models of different source databases; and, by using special SQL statements to process full data collection and incremental data collection, it is possible to achieve optimized use of resources for data integration. Therefore, the embodiment of the present application can improve the efficiency of the data integration process and optimize the performance of data integration. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0020] Figure 1 This is an architecture diagram of the data integration platform.

[0021] Figure 2The flowchart is based on the data integration of the source database.

[0022] Figure 3 This is a flowchart of real-time data collection based on source database.

[0023] Figure 4 The flowchart is for data integration based on the source file system.

[0024] Figure 5 The flowchart of data integration based on source network API interface.

[0025] Figure 6 The flowchart of data integration based on source MQ is shown in FIG.

[0026] Figure 7 A block diagram of a data integration device. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0028] As an optional implementation, Figure 1 The data integration platform architecture diagram is shown as an example. The data integration platform can be a server platform for implementing data integration, and can be implemented by a server system consisting of multiple servers. Figure 1 The data integration platform may include a data collection system 110 , which is at least used to collect data from heterogeneous data sources and integrate it into a data storage and metadata management system 120 .

[0029] Heterogeneous data sources refer to multiple data sources with different data sources. In other words, heterogeneous data sources include multiple data sources with diverse data sources. For example, the data in different data sources has different formats, structures, storage methods or access methods. For example, heterogeneous data sources may include:

[0030] Source database (DB, Data Base), that is, the embodiment of the present application can collect data from a database as a data source. For the convenience of explanation, the database as a data source is called a source database, and the source database is, for example, a MySQL database of an enterprise;

[0031] A source file system, that is, the embodiment of the present application can collect data from a file system as a data source. For the convenience of description, the file system as a data source is called a source file system;

[0032] Source network API (Application Programming Interface) interface, that is, the embodiment of the present application can access the network through the network API interface as the data source to collect data; for the convenience of description, the network API interface as the data source is called the source network API interface;

[0033] Source MQ (Message Queue), that is, the embodiment of the present application can collect data from MQ as a data source. For the convenience of explanation, the MQ as a data source is called source MQ.

[0034] The following describes the data integration from the perspectives of source database, source file system, source network API interface, and source MQ.

[0035] As an optional implementation, Figure 2 The following is an example of a data integration flowchart based on the source database. Figure 2 As shown, the process may include the following steps.

[0036] Step S210, for full data collection, collect the full data of the source database into the intermediate table of the intermediate library; for incremental data collection, collect the incremental data of the source database into the intermediate table of the intermediate library according to the incremental update field.

[0037] The data integration based on the source database in the embodiment of the present application involves: a first stage of collecting data from the source database to the intermediate table of the intermediate database, and a second stage of pulling data from the intermediate table of the intermediate database to the target table of the target database. Optionally, the intermediate database is a database corresponding to the intermediate storage system of the data integration platform, which is used to store the intermediate data of the data integration, for example, in combination with Figure 1 As shown, the intermediate library may be an ODS (Operational Data Store) library in the data storage and metadata management system 120; the intermediate table may be a table in the intermediate library, for example, the intermediate table may be an ODS table in the ODS library. Figure 1 As shown, the target database may be a database adopting an enterprise standard model, for example, the data model of the data in the target database is the enterprise standard model; the target table is a table in the target database, and the data in the target table adopts the enterprise standard model.

[0038] It should be noted that in the embodiment of the present application, the ODS library is designed to temporarily store data from multiple data sources and act as a buffer layer for data integration. The enterprise standard model refers to a standardized and normalized data model defined in the embodiment of the present application, which is used to unify the data model of heterogeneous data sources.

[0039] Step S210 is mainly used to implement the first stage of data integration based on the source database, that is, to collect full data and incremental data from the source database to an intermediate table (such as an ODS table) through full data collection and incremental data collection.

[0040] In an optional implementation, full data collection can be performed in the early stages of data integration or when the data in the ODS table needs to be completely refreshed. Full data collection involves copying all data in the database to the ODS table. For example, full data collection can be used for initialization of the ODS table or regular full data synchronization.

[0041] As an optional implementation, for full data collection, taking the data integration tool using the DataX component as an example, the embodiment of the present application can adopt the DataX component to collect data from the source database to the intermediate table by configuring the Reader plug-in (such as the MySQL Reader plug-in) that adapts to the source database and the Writer plug-in (such as the Postgres Writer plug-in) of the intermediate library, and by passing the JSON (JavaScript Object Notation) configuration.

[0042] It should be noted that the DataX component is an open source data integration tool, which is mainly used for large-scale data migration and synchronization between various databases, data warehouses and data lakes. Of course, the embodiments of the present application can also support the use of other types of data integration tools, not limited to the DataX component; the DataX component supports a customizable plug-in architecture and can connect to various data sources and intermediate libraries (such as ODS libraries) through different Reader and Writer plug-ins.

[0043] In the DataX component, the Reader plug-in is responsible for reading data from the data source. Each Reader plug-in supports a specific type of data source. For example, the MySQL Reader plug-in is a plug-in provided by the DataX component and is used to read data from the MySQL database. The Writer plug-in in the DataX component is responsible for writing data to the intermediate library. For example, if the intermediate library is a PostgreSQL database (such as the ODS library is a PostgreSQL database), the Postgres Writer plug-in is used to write data to the PostgreSQL database. It should also be noted that the JSON format is a lightweight data exchange format that is easy for users to read and write, and is also easy for machines to parse and generate.

[0044] It should be further explained that, in the embodiment of the present application, the JSON configuration may be configuration information (such as configuration items) in a JSON configuration file, which expresses the processing method of extracting data from a data source and writing the data into an intermediate library. In other words, the JSON configuration enables the DataX tool to understand and execute the data migration task from the data source to the intermediate library. For example, the JSON configuration (such as the configuration items in the JSON configuration file) may include but is not limited to:

[0045] Reader configuration information, configuration information for reading data from the source database, such as data source connection information and metadata query information; the data source connection information is used by the DataX component to connect to the specified data source, and its content may include data source type, server address, port number, user name and password, etc., so that the data source connection information is set to correspond to the source database in the Reader configuration information configured in JSON (that is, the connection information of the source database is configured), which can ensure that the DataX component can access and read the data in the source database; the metadata query information is used to guide the DataX component to read data from the data source, and its content may include the executed SQL query statement, or the tables and fields to be migrated specified in the data source, so that the metadata query information is set in the Reader configuration information configured in JSON, which can define the specific behavior of the DataX component in reading data from the connected source database;

[0046] Writer configuration information, which is used to write data into the intermediate library, such as intermediate library connection information, intermediate table, and insert field information. The intermediate library connection information is used to set the intermediate library (such as ODS library) to which the DataX component is connected. Its content includes library type, server address, port number, user name, and password. Setting the intermediate library connection information in the JSON-configured Writer configuration information can ensure that the DataX component writes the data read from the source database into the intermediate library. The intermediate table and insert field information are used to specify the storage location and method of data in the intermediate library, such as the name and field information of the intermediate table. Setting the name and field information of the intermediate table in the JSON-configured Writer configuration information can ensure that the data is stored in the intermediate table of the intermediate library (such as the ODS table of the OSD library) as expected.

[0047] Transformer configuration information, which is used to indicate the transformation operations that need to be performed when data is transferred from the source database to the intermediate table. The content may include data cleaning, formatting, or field value calculation.

[0048] Config (settings) configuration information, such as setting task quantity information, error limit information, etc.; among them, the task quantity information defines the parallel processing capability of the DataX component when performing data migration, which can be the number of tasks executed in parallel, and the task quantity information needs to balance the data processing speed and system load; the error limit information is used to set the maximum number of errors allowed during the data migration process, and its content can be the upper limit of the number of errors allowed. By setting the error number threshold (i.e., the upper limit of the number of errors) corresponding to the error limit information, once the number of errors during the data migration process exceeds the error number threshold, the data migration task stops.

[0049] The data migration task described above in the embodiment of the present application may be a task of migrating data from a data source (eg, a source database) to an intermediate library (eg, an ODS library).

[0050] In a further optional implementation, the embodiment of the present application can allow multiple tasks to be performed simultaneously through the Channel task splitting technology, thereby improving the execution efficiency and performance of the data migration task. Specifically, Channel can be a data transmission channel used by data integration tools such as DataX tools for data transmission, that is, Channel is responsible for managing the data flow from the source database to the intermediate library, so that by splitting the data migration task into multiple subtasks, and multiple subtasks are divided into multiple Channels for parallel execution, multiple tasks can be performed simultaneously, thereby improving the efficiency and performance of data migration; for example, each Channel can run in a separate thread or process, so that the data migration task is divided into multiple independent Channels for execution, then multiple threads or processes corresponding to multiple Channels can run simultaneously, each thread or process executes tasks divided on different Channels, and multiple tasks can be performed simultaneously, thereby improving the efficiency and performance of data migration.

[0051] In an optional implementation, incremental data collection can be to migrate data records that already exist in the intermediate table but have changed data content from the source database to the intermediate table; incremental data collection relies on a mechanism that can identify data record changes in the source database, such as timestamps, log files, or data monitoring tools; incremental data collection can reduce the amount of data transmitted from the source database to the intermediate database, thereby improving data transmission efficiency.

[0052] In an optional implementation, for incremental data collection, the embodiment of the present application can set the attribute of Write Mode to update in the Writer configuration of the DataX tool (such as the above-mentioned Writer configuration information) to indicate that the current task is to update the existing records in the intermediate table instead of inserting new records, that is, to update the existing records (records, i.e., data records) in the intermediate table; at the same time, in the Writer configuration of the DataX tool (such as the above-mentioned Writer configuration information), specify field information (such as business primary key information) to indicate the fields in the intermediate table that belong to incremental data. Among them, the business primary key can be used to uniquely identify the records in the table, and the business primary key field is set in the Writer configuration, which can be used to identify the uniqueness of each record in the intermediate table; further, when executing the data update task, the DataX component can use the business primary key field to determine the records that need to be updated in the intermediate table. That is to say, the embodiment of the present application can collect the incremental data of the source database into the intermediate table of the intermediate library according to the updated write mode set in the Writer configuration information, and the field information belonging to the incremental data (such as the business primary key field).

[0053] Step S220, for full data collection, according to the data integration statement, pull the full data from the intermediate table of the intermediate library to the target table of the target library; for incremental data collection, use the data integration statement with the incremental update clause to pull the incremental data from the intermediate table of the intermediate library to the target table of the target library; wherein, the process of pulling data from the intermediate table to the target table performs data type conversion and field mapping.

[0054] Step S220 is mainly used to implement the second stage of data integration based on the source database. As an optional implementation, the embodiment of the present application can choose to use SQL components or DataX components to migrate the data of the intermediate table to the target table according to the needs.

[0055] Optionally, the embodiment of the present application can generate and execute data integration statements (such as SQL statements) to migrate data from the intermediate table to the target table. In an optional implementation example, the data integration statement (such as SQL statement) can extract data from one or more intermediate tables, and the extracted data is filtered and joined through the specified association conditions and screening conditions, and then inserted into the target table of the enterprise standard model; the target table is a table using the enterprise standard model, which is used to store sorted and converted data in a unified data model and format to support the enterprise's analysis and business decisions.

[0056] For example, a data integration statement (such as a SQL statement) can perform the following process:

[0057] Data pulling process, data pulling can be achieved through the SELECT clause, which is used to select and pull data from the intermediate table; for example, the SELECT clause specifies the fields of the pulled data and the one or more intermediate tables from which the fields of the pulled data come;

[0058] Data integration process, data integration can be achieved through the JOIN clause, which is used to merge data pulled from multiple intermediate tables through association conditions; for example, the JOIN clause can connect two intermediate tables based on the business primary key or other associated fields (such as customer ID or order ID);

[0059] Data filtering process: Data filtering can be achieved through the WHERE clause (conditional clause), which is used to select records that meet the filtering conditions from the data merged in the data integration process to achieve data filtering; for example, the WHERE clause can filter data based on multiple filtering conditions such as date, status, and value range;

[0060] The data insertion process can be implemented through the INSERT INTO clause (insert clause), which is used to specify the target table and field, that is, the target table and field into which the data of the above data pulling, data integration and data filtering are inserted.

[0061] As an optional implementation, for full data collection, the embodiment of the present application can pull the full amount of data from the intermediate table to the target table of the enterprise standard model through a data integration statement (such as an SQL statement) having the above-mentioned data pulling process, data integration process, data filtering process and data insertion process. For example, for full data collection, the embodiment of the present application can use the processing process of the above-mentioned data pulling process, data integration process and data filtering process of the data integration statement to obtain the full amount of data to be inserted from the intermediate table; then, after the full amount of data to be inserted is converted into data type and mapped into fields, it is inserted into the target table through the above-mentioned data insertion process of the data integration statement.

[0062] In an optional implementation, data type conversion and field mapping can be performed automatically during the migration of data from the intermediate table to the target table. Optionally, the embodiment of the present application can use the metadata information of the data asset platform and the mapping information of the page configuration to automatically perform data type conversion and field mapping.

[0063] Metadata information is data that describes data, providing detailed information about the data source, content, format, and structure. During data migration and integration, metadata can indicate the source of data, data type, and how data is processed. The mapping information configured on the page refers to the rules set through the user interface, which define the rules for mapping the data fields of the intermediate table to the corresponding fields of the target table. This mapping information can ensure that the data can be correctly mapped in the predetermined way when it is migrated from the intermediate table to the target table.

[0064] In an optional implementation, the embodiment of the present application can automatically convert the type of data pulled from the intermediate table to the data type indicated by the metadata information (i.e., the data type corresponding to the data in the target table) according to the data type indicated by the metadata information; and map the fields of the data pulled from the intermediate table to the corresponding fields of the target table according to the rules corresponding to the mapping information configured in the page. For example, taking the example of pulling a field named Date_of_Birth and a string type from the intermediate table, if the corresponding field name in the target database is DOB and the type is date, the string format can be converted to the date format according to the data type indicated by the metadata information, and according to the mapping information (Date_of_Birth to DOB ​​mapping), the converted data is stored in the DOB field of the target table.

[0065] For incremental data acquisition, when migrating data from the intermediate table to the target table, it involves inserting the records in the intermediate table into the target table. When the target table has a record with the same business primary key as the intermediate table, it is necessary to update the data of the record instead of inserting a new record. At this time, the embodiment of the present application can use a data integration statement with an incremental update clause to migrate the updated data (i.e., incremental data) in the intermediate table to the existing records in the target table. For example, the incremental update clause can perform the following process:

[0066] Conflict detection process: when trying to insert a record from the intermediate table into the target table, if the business primary key of the record already exists in the target table, a conflict is triggered;

[0067] Execute the update process. When a conflict occurs, for the conflicting record, use the incremental data in the intermediate table to replace the old data of the conflicting record in the target table; for example, update the field data of the conflicting record in the intermediate table to the corresponding field in the target table.

[0068] For example, the conflict problem of the business primary key can be automatically detected through the ON CONFLICT clause (conflict handling clause); the update operation is indicated by the DO UPDATE SET clause (execute update set clause): update the data of the field in the intermediate table where the conflict is recorded to the corresponding field in the target table. Therefore, when the intermediate table and the target table have records with the same business primary key, if the data in the record in the intermediate table changes, the corresponding data change will also be reflected in the corresponding record in the target table. In other words, the conflict detection of the ON CONFLICT clause and the update operation of the DO UPDATE SET clause can be used to handle business primary key conflicts, and are suitable for updating incremental data to the target table, thereby ensuring the data consistency of the target table.

[0069] Furthermore, when using data integration statements to pull data from the intermediate library to the target library, for incremental data, the embodiment of the present application can pull incremental data from the intermediate table to the target table of the enterprise standard model through an incremental update clause with the above-mentioned conflict detection process and update execution process. For example, when using data integration statements to pull data from the intermediate library to the target library, the embodiment of the present application can use the above-mentioned conflict detection and update execution process of the incremental update clause to obtain the incremental data to be inserted, and the incremental data to be inserted is inserted into the corresponding conflict record of the target table after data type conversion and field mapping.

[0070] In an optional implementation, the above Figure 2 The full data collection and incremental data collection shown may be non-real-time data collection, for example, may be triggered based on active query mode. Figure 2 The method shown. In an optional implementation, active query can be to pull data from the source database to the intermediate table and then to the target table by Pulling the full data and incremental data. In an optional implementation, the embodiment of the present application can send an active query request to the source database at a fixed time (for example, every day or every hour) or manually triggered by the user after configuring the connection information of the source database to query whether there is data record or data update (for example, data of existing records is updated) in the source database; when there is data record in the source database, the data collection system can collect the full data (refer to Figure 2 As shown in the figure), pull the full amount of data from the source database to the intermediate table of the intermediate database, and then pull it from the intermediate table to the target table of the target database; when there is data update in the source database, the data collection system can collect data through incremental data (refer to Figure 2 As shown in the figure, the incremental data of the source database is pulled into the intermediate table of the intermediate database, and then pulled from the intermediate table to the target table of the target database.

[0071] That is, in an optional implementation, Figure 2The method shown can be executed by sending an active query request to the source database after configuring the connection information of the source database, so that the full data covering the data records of the source database is pulled from the source database to the intermediate table and then to the target table through full data collection; when there is updated data in the source database, the corresponding incremental data is pulled from the source database to the intermediate table and then to the target table through incremental data collection. The above process can be implemented based on timed triggering or manual triggering after configuring the connection information of the source database.

[0072] The embodiment of the present application can send an active query request to the source database after configuring the connection information of the source database to query whether there is data record or data update in the source database; when there is data record in the source database, the full data of the source database is pulled to the intermediate table of the intermediate library through full data collection, and then pulled from the intermediate table to the target table of the target library; the intermediate library is a database corresponding to the intermediate storage system of the data integration platform, and the target library is a database using the enterprise standard model; when there is data update in the source database, the incremental data of the source database is pulled to the intermediate table of the intermediate library through incremental data collection, and then pulled from the intermediate table to the target table of the target library;

[0073] Among them, for full data collection, the full data of the source database is collected into the intermediate table of the intermediate library, and then according to the data integration statement, the full data is pulled from the intermediate table of the intermediate library to the target table of the target library; for incremental data collection, according to the incremental update field, the incremental data of the source database is collected into the intermediate table of the intermediate library, and then the data integration statement with the incremental update clause is used to pull the incremental data from the intermediate table of the intermediate library to the target table of the target library; the process of pulling data from the intermediate table to the target table also performs data type conversion and field mapping.

[0074] It can be seen that the embodiment of the present application can cover the data migration path from the source database to the target database through the process of full data collection and incremental data collection, and introduce the intermediate library as temporary storage to ensure the integrity and accuracy of the data during the migration process, thereby meeting the data integration requirements corresponding to the data record or data update when actively querying the source database, and improving the efficiency of the data integration process. Specifically, after configuring the connection information of the source database, the embodiment of the present application can send an active query request to the source database, so that when querying the source database for data records, the full data of the source database can be pulled to the intermediate library, and then according to the data integration statement, the full data is pulled from the intermediate library to the target library, ensuring that the target library has a complete snapshot of the source database, which can be applied to the initialization or periodic complete synchronization of the source database and the target library; when the source database is updated, the incremental data is pulled to the intermediate library, and then the data integration statement with the incremental update clause is used to pull the incremental data from the intermediate library to the target library, reducing the network load and storage requirements, while ensuring the latest data status of the target library. Furthermore, in the process of transferring data from the intermediate library to the target library, data type conversion and field mapping are performed, and the problem of inconsistent data formats that may exist between different source databases can be solved through the enterprise standard model of the target library, thereby unifying the data models of different source databases. Therefore, the embodiment of the present application can effectively manage data integration across databases. Whether it is necessary to regularly synchronize the full amount of data in the source database or update the incremental data in the source database, the embodiment of the present application can achieve complete and accurate data integration, and unify the data models of different source databases; and, by using special SQL statements to process full data collection and incremental data collection, it is possible to achieve optimized use of resources for data integration. Therefore, the embodiment of the present application can improve the efficiency of the data integration process and optimize the performance of data integration.

[0075] In a further optional implementation, the embodiment of the present application may also support real-time data collection of the source database, that is, capturing data change events of the source database, and synchronizing the data changed by the data change events to the intermediate library or the target library. For example, the real-time change data of the source database can be synchronized to the intermediate library, or synchronized to the intermediate library and then to the target library, or directly synchronized to the target library. It should be noted that, unlike the active query triggering method of the above-mentioned full data collection and incremental data collection, the real-time data collection of the source database can be carried out in real time. For example, the data collection system does not query the source database regularly or manually triggered by the user, but continuously monitors the change log of the source database, so that any data changes (insert, update, delete) of the source database will be written to the change log immediately; further, when the data collection system monitors the data change events in the change log, it can capture the data change events of the source database in real time and immediately reflect them in the intermediate library or the target library.

[0076] In an optional implementation, Figure 3 A flowchart of real-time data collection based on a source database is shown, such as Figure 3 As shown, the process may include the following steps.

[0077] Step S310: monitor the change log of the source database in real time to capture data change events of the source database in real time.

[0078] In an optional implementation, the embodiment of the present application may use the Debezium platform to perform step S310; wherein the Debezium platform is an open source distributed platform for real-time capture and forwarding source database change events, and the Debezium platform can monitor the change log of the source database and capture the change events corresponding to the change log. Thus, the embodiment of the present application may deploy the Debezium platform in the source database to monitor the change log of the source database in real time, such as real-time monitoring of MySQL's Binlog (Binary Log), PostgreSQL's WAL (Write Ahead Logging), etc.; by real-time monitoring of the change log of the source database, the Debezium platform can capture data change events of the source database in real time, such as data insertion, data update, and data deletion.

[0079] Step S320: Format the data change event into a message and send it to a specified topic of Kafka in real time.

[0080] In an optional implementation, the embodiment of the present application may use a Kafka connector in conjunction with the Debezium platform to achieve real-time data collection based on a source database. After the Debezium platform captures the data change events of the source database in real time, it can publish the data change events to the Kafka connector in real time. For example, the data change events captured by Debezium can be formatted as messages and sent to one or more specified topics in the Kafka cluster in real time through Kafka's Source Connector, so that Kafka, as an intermediate message queue, can store the messages corresponding to these data change events of the source database, so that Kafka's downstream systems or downstream services can access and process these messages in real time.

[0081] It should be noted that Kafka's Source Connector is a part of the Kafka connector, which is used to capture data from external systems and push this data to the Kafka cluster; Source Connector plays the role of data input, transferring data streams from different sources to the Kafka server in a unified manner. Topic is the category of Kafka's data stream or the identifier of the data source. Each topic contains a series of data records, and Kafka stores and manages data based on topics.

[0082] Step S330, consume the message of the specified topic through Kafka's downstream system or downstream service, and apply the changed data corresponding to the message to the intermediate library or target library in real time according to the consumed message.

[0083] As an optional implementation, Kafka's downstream system or downstream service can use Kafka's Sink Connector to subscribe to and consume messages of specified topics from Kafka; the messages of specified topics consumed can contain the type and detailed data of the data change event corresponding to the source database, such as the data record and change type (insert, update, delete, etc.) of the data change event; thus, Kafka's downstream system or downstream service can apply these data change events to the intermediate library or target library in real time according to the content of the consumed messages to ensure data consistency and real-time update between the intermediate library or target library and the source database.

[0084] It should be noted that the Sink Connector is used to transfer data from the Kafka cluster to an external system (such as a downstream system or downstream service). It is mainly responsible for consuming messages from Kafka topics and pushing the contents of these messages to external systems.

[0085] In a further optional implementation, the embodiments of the present application can realize real-time data collection for the source database based on the deployment of Kubernetes; for example, the Debezium platform and Kafka can be deployed on the Kubernetes platform, using the containerization and automated management features of Kubernetes to enhance availability and scalability; wherein, the Kubernetes environment provides the ability to automatically deploy, expand, and manage containerized applications.

[0086] In a further optional implementation, the embodiments of the present application can monitor and record the operating status, performance and potential system failures of the data acquisition system in real time through various monitoring tools and log recording functions, thereby facilitating timely problem detection, troubleshooting and system optimization.

[0087] As an optional implementation, the embodiment of the present application may also support data integration of the source file system. The data of the source file system may be integrated into the intermediate library, or integrated into the intermediate library and then into the target library, or directly integrated into the target library. Figure 4 The following is an example of a data integration flowchart based on the source file system. Figure 4 As shown, the process may include the following steps.

[0088] Step S410, for the full import of source files in the source file system, the source files to be imported are selected through user configuration, and the mapping relationship between the data page area of ​​the source file and the logical entity field of the intermediate table or the target table is configured; the page content of the source file is parsed, an SQL statement is generated according to the mapping relationship, and the SQL statement is executed to clear the existing data in the intermediate table or the target table, and all the data in the source file is inserted into the intermediate table or the target table.

[0089] The data integration of the source file system in the embodiment of the present application is mainly divided into full import and additional import. Step S410 mainly implements full import, and supports importing the source files of the source file system into the intermediate table or the target table in full. Full import can be regarded as a batch import of source files of the source file system. Taking the source file as an Excel file as an example, the embodiment of the present application can provide an import scheme configuration interface, so that the user can perform user configuration in the import scheme configuration interface. The user configuration may include: selecting the Excel file to be imported (that is, selecting the source file to be imported), and configuring the mapping relationship between the columns in the Excel file and the logical entity fields in the intermediate table or the target table (such as the fields in the intermediate table or the target table). In the case where the source file is an Excel file as an example, the area within the data page of the source file can be, for example, a column in an Excel file.

[0090] Thus, the data acquisition system provided in the embodiment of the present application can automatically parse the selected Excel file and read the page content in the Excel file (for example, the Sheet page content in the Excel file); and then the data acquisition system can generate corresponding SQL statements according to the mapping relationship configured by the user. Optionally, the SQL statement mainly includes a full delete statement and a full insert statement, wherein the full delete statement is used to delete (i.e., clear) the existing data in the intermediate table or the target table, and the full insert statement is used to insert all the data of the selected Excel file into the intermediate table or the target table; that is, the embodiment of the present application can clear the existing data in the intermediate table or the target table by full delete, and insert all the data of the Excel file into the intermediate table or the target table by full insert, so that by executing the SQL statement, the embodiment of the present application can realize the full import of the data of the Excel file into the intermediate table or the target table.

[0091] Step S420, for the additional import of the source file, reuse the user configuration; check the existing data in the intermediate table or the target table, and append the additional data of the source file to the intermediate table or the target table according to the mapping relationship configured by the user, so as to update the data with the existing records in the intermediate table or the target table.

[0092] Append import is suitable for scenarios where the data of the source file needs to be continuously updated, especially when new data of the source file is constantly generated and the data of the intermediate table or target table needs to be kept up to date. Under append import, the embodiment of the present application can reuse the corresponding user configuration of the full import, including the file structure and mapping relationship of the source file, etc.; thus, taking Excel file as an example, the embodiment of the present application can parse the selected Excel file and read the additional data that needs to be appended in the Excel file (for example, the updated data that needs to be updated); then, the embodiment of the present application can check the existing data in the intermediate table or target table, and append the additional data to the intermediate table or target table according to the configured mapping relationship, for example, based on the field in the intermediate table or target table corresponding to the column of the additional data in the Excel file indicated by the mapping relationship, append the additional data to the corresponding field in the intermediate table or target table; wherein, for records that already exist in the intermediate table or target table (which can be identified based on the business primary key field), the embodiment of the present application will not reinsert the record, but update the data of the record to ensure that the data in the record is up to date.

[0093] As an optional implementation, the embodiment of the present application may also support data integration of data from the source network API interface. The data from the source network API interface may be integrated into the intermediate library, or integrated into the intermediate library and then into the target library, or directly integrated into the target library. Figure 5 The following is an example of a data integration flowchart based on the source network API interface. Figure 5 As shown, the process may include the following steps.

[0094] Step S510, configuring the source network API interface, defining the data mapping relationship of the source network API interface in the data preview phase based on the configured source network API interface; and creating a corresponding intermediate table according to the data mapping relationship of the source network API interface.

[0095] In an optional implementation, an embodiment of the present application may provide an API configuration interface to allow a user to configure a source network API interface. For example, a user may enter the API URL (Uniform Resource Locator) and request parameters through the API configuration interface, such as the API access key (Token) and encryption and decryption configuration.

[0096] The embodiment of the present application further provides a data preview function, allowing the user to verify whether the API call is successful based on the configured source network API interface, and preview the data structure returned by the API. In the above-mentioned data preview stage (for example, the stage of previewing the data structure returned by the API), the embodiment of the present application can support the definition of the data mapping relationship of the source network API interface, and the definition of the data mapping relationship of the source network API interface can mainly include defining the data path and configuring the primary key field; for example, the user can define the data path (that is, define the used part of the data structure returned by the source network API interface) and the primary key field to ensure that the data returned by the API can be correctly mapped to the corresponding field of the intermediate table.

[0097] Furthermore, based on the user-defined data mapping relationship, the embodiment of the present application can create a corresponding ODS table (an example of an intermediate table) in the asset directory, and the table name and fields of the ODS table correspond to the data structure returned by the source network API interface.

[0098] In an optional implementation, the embodiment of the present application can parse the customer table information in the data mapping relationship of the source network API interface, so as to obtain the metadata of the customer table, for example, link to the customer table of the source network API interface based on the parsed customer table information, so as to obtain the metadata of the customer table; then, combine the obtained metadata with the table creation rules of the intermediate table, and check whether the corresponding intermediate table already exists. This is to ensure that the data is not created repeatedly. If the corresponding intermediate table already exists in the asset directory, it can be used directly; if it does not exist, it is necessary to proceed to the next step of intermediate table creation; that is, if the corresponding intermediate table is not found in the asset directory, a new intermediate table is created in the asset directory according to the obtained metadata. When creating the intermediate table, not only the basic structure of the intermediate table (such as field name and type) must be included, but also complete metadata information, such as table description and unique constraint, to ensure the integrity of the intermediate table and the consistency of the data.

[0099] Step S520, calling the source network API interface and processing the network request of the source network API interface to obtain data corresponding to the data path from the source network API interface; generating a JSON script based on the data obtained from the source network API interface to load the obtained data into the created intermediate table; wherein, in the process of loading data into the intermediate table, the primary key field configured by the data mapping relationship is used to handle data duplication or data conflict.

[0100] In an optional implementation, the user can set a scheduled task by configuring a cron expression (cron expressions are used to automatically execute tasks at a predetermined time), thereby automatically calling the source network API interface on a scheduled basis. When calling the source network API interface on a scheduled basis, the embodiment of the present application can send an HTTP (HyperText Transfer Protocol) request to the URL of the source network API interface; further, the embodiment of the present application can use the Restful Reader plug-in extended by the DataX component to process the network requests of the source network API interface, thereby ensuring that the data received from the source network API interface is correctly parsed and processed, for example, the part of the data used in the data path defined by the data mapping relationship can be correctly received. It should be noted that using the Restful Reader plug-in extended by the DataX component to process the network requests of the source network API interface may involve processing the HTTP requests and responses of the source network API interface.

[0101] It should be noted that the Restful Reader plug-in is a reading plug-in in the DataX component, which is used to read data from a data source that supports the Restful API. This plug-in can process HTTP requests and responses from the Restful service, thereby obtaining data and converting it into a format that the DataX component can handle for the data synchronization process.

[0102] Based on the data obtained from the source network API interface, the embodiment of the present application can generate a JSON script for the execution task of the DataX component. The JSON script is executed through the scheduling platform and can be responsible for loading the obtained data into the created intermediate table.

[0103] In the process of loading data into the intermediate table, the embodiment of the present application can use a primary key conflict update strategy to handle possible data duplication or conflict problems based on the primary key field defined by the data mapping relationship, thereby avoiding data redundancy and errors.

[0104] In a further optional implementation, the embodiment of the present application can monitor the task execution status of the DataX component in real time and record relevant logs to facilitate troubleshooting and performance optimization to ensure the stability and efficiency of data integration.

[0105] As an optional implementation, the embodiment of the present application may also support data integration of the source MQ data. The source MQ data may be integrated into the intermediate library, or integrated into the intermediate library and then into the target library, or directly integrated into the target library. Figure 6 The following is an example of a data integration flowchart based on source MQ: Figure 6 As shown, the process may include the following steps.

[0106] Step S610, configuring an MQ instance task, and providing a data preview function of the MQ instance task, so as to view message data read from the monitored MQ.

[0107] In an optional implementation, an embodiment of the present application may provide a visual interface to support users in configuring MQ instance tasks; for example, through the visual interface, users may input configuration information of the MQ instance, such as MQ connection information and the name of the message queue to be monitored, etc., and MQ connection information such as host address, port, user name, password, etc.; through the configuration information of the MQ instance, an embodiment of the present application may generate an MQ Source Connector, which may be used to read data from the specified MQ to be monitored; thus, the MQ Source Connector, as a connection component, may be connected to the MQ instance, that is, the MQ Source Connector depends on the connected MQ instance to obtain data from the specified MQ to be monitored.

[0108] Furthermore, the embodiment of the present application provides a data preview function for the MQ instance task, allowing the user to view the message data read from the MQ to be monitored after configuring the MQ instance task.

[0109] Further, if the read message data is in the form of a list, the embodiment of the present application supports converting the list into multiple independent messages for subsequent processing. As an optional implementation example, in the front-end interface, the user can select the JSON data path to be processed from the received MQ message, and the selected data path is persistently stored in the target library by the back-end system; in the case of using Kafka in combination, the message sent by MQ not only contains data, but also specifies the Kafka topic (Topic) corresponding to the message, which can be managed by the MQ Source Connector, so that the message can be directly pushed to the Kafka message queue corresponding to the specified topic; further, the embodiment of the present application can monitor the specified topic of Kafka and consume the message corresponding to the specified topic of Kafka. For example, the embodiment of the present application can use the selected data path to identify and process the message of the specified topic; then, after receiving each message data, if the message data contains data in the form of a list (for example, an array), the embodiment of the present application can traverse the list, extract each data in the list separately and format it into a format acceptable to the JDBC (Java Database Connectivity, Java database) Sink Connector, and the formatted data can be sent to another specified Kafka topic so that it can be finally sent to the intermediate library.

[0110] Step S620, configuring the JDBC Sink Connector to specify the connection information of the intermediate library; and writing the message data read from the MQ into the intermediate library.

[0111] JDBC Sink Connector is a tool for streaming data from Kafka to a database system that supports the JDBC protocol. This connector is responsible for reading data from a specified topic of Kafka and writing it to the configured database. Therefore, when configuring an MQ instance task and reading data from a specified MQ through MQ's MQ Source Connector, the data read from MQ can be pushed to the Kafka message queue, and then the embodiment of the application can read data from the Kafka message queue corresponding to the specified topic of Kafka through the JDBC Sink Connector and write it to the intermediate library.

[0112] In an optional implementation, the embodiment of the present application can configure the JDBC Sink Connector to specify the connection information of the intermediate library, such as library type, host address, port, etc.; further, the embodiment of the present application can also support selecting an intermediate table from the intermediate library through a drop-down menu, and then configure the table mapping information of the JDBC Sink Connector according to the selected intermediate table. The table mapping information is used to map the message data read from MQ to the corresponding fields of the intermediate table of the intermediate library.

[0113] As an optional implementation example, the user can make multiple selections when creating a new task instance, including determining the path of the data, the path of the primary key field, and selecting an OSD table. For example, an ODS table is created in the asset directory in advance, and the user can search and select the OSD table. After the user completes the above configuration and submits it, the background system can automatically persist the configuration information, such as saving it to the configuration database. Then, the system can read the user's configuration information and use a JDBC Sink Connector template (including some fixed configuration items and some variable parts) to replace the variable part in the template with the read configuration information to generate the final configuration file (for example, JSON format). The generated configuration file can be transmitted to the server of the Kafka connector through the RESTful API. Once the configuration file is uploaded and accepted by the server of the Kafka connector, the corresponding JDBC Sink Connector instance is created and the data synchronization task can be started.

[0114] Furthermore, the JDBC Sink Connector can read message data from the Kafka message queue corresponding to the specified topic of Kafka (if it is in list form, format conversion is also involved), and correctly map and write the message data to the corresponding fields of the intermediate table according to the configured table mapping information.

[0115] In a further optional implementation, the embodiment of the present application may provide a task monitoring interface so that users can view the execution status, progress and log information of the task in real time, so that users can track the performance of the data integration task and possible execution problems. Furthermore, by regularly checking the task status through the background polling mechanism, anomalies or errors can be discovered and handled in a timely manner; when anomalies or errors are found, the embodiment of the present application may notify the user through a preset alarm mechanism (such as email, instant messaging software, etc.) to ensure that the problem can be quickly responded to and handled.

[0116] In a further optional implementation, combining Figure 1 As shown, the embodiment of the present application can realize at least the integration of data from the data source to the intermediate library and then to the target library by intelligent matching and automatic mapping through the intelligent matching system 130. The intelligent matching and automatic integration of data can involve relational projection (RP, Relation Projection), entity relationship (ER, Entity Relationship) recommendation, and data integration strategies based on different data sources (such as API, MQ, etc.).

[0117] In an optional implementation, the embodiment of the present application can automatically analyze the table data structure (such as the order table and the user table) in the ERP database (an example of a source database), and automatically match the fields in the table data structure with the corresponding entities in the enterprise standard model, for example, the order_id field of the order table matches the corresponding field of the order entity in the enterprise standard model. Thus, based on the mapping and matching relationship between the fields in the table data structure and the corresponding fields of the entity in the enterprise standard model, the embodiment of the present application can automatically generate SQL statements to transfer the data from the ODS library to the target library of the enterprise standard model after transferring the data from the source database to the ODS library. That is to say, in the data integration based on the source database, the embodiment of the present application uses the mapping information of the fields in the table data structure of the source database and the corresponding fields of the entity in the enterprise standard model. Based on the mapping information, the embodiment of the present application can use the full data collection and incremental data collection solutions based on the data integration of the source database described above to realize the integration of data from the source database to the target library.

[0118] In an optional implementation example, the user can first configure the connection information of the ERP database (a form of the source database) and import the key data tables of the ERP database, such as the data dictionary of the order table and the user table; thus, the embodiment of the present application can use an intelligent matching algorithm to analyze the fields in the key data tables of the ERP database, such as order_id, user_id, order_date and other fields, and automatically match these fields with the corresponding entities in the enterprise standard model, and recommend the mapping information of the fields in the table data structure of the source database and the corresponding fields of the entities in the enterprise standard model according to the matching results, for example, mapping the order table to the order entity in the enterprise model, mapping the user table to the user entity, etc.; further, after confirming the above mapping information, the embodiment of the present application can automatically generate a data integration statement (such as an SQL statement) for pulling data from the intermediate table of the intermediate library to the target table of the target library according to the above mapping information after the data in the source database is collected into the intermediate library (such as the OSD library), and realize the field mapping of the process of pulling data from the intermediate table to the target table; the corresponding processing process of the data integration statement can refer to the description of the corresponding part of the previous text, which will not be repeated here.

[0119] In a further optional implementation, for complex mapping scenarios involving multiple data tables, such as integrating the data of a delivery note and a receipt into one table, it is necessary to analyze and confirm the entity relationship (ER) between these tables, so that the embodiment of the present application can perform ER matching. For example, after pulling the data from the source database to the intermediate library (ODS library), the data pulled to the intermediate library can be ER matched to obtain recommended ER relationships, such as using the dictionary and metadata (including business primary key information) of the data pulled to the intermediate library to recommend possible ER relationships (for example, if the user_id of the order table is associated with the user_id of the user table, a one-to-many relationship between the two tables can be recommended); further, the recommended ER relationship and its similarity information can be pushed to the data asset platform and the graph engine, so that the user can confirm the ER relationship on the platform, and further, after the user confirms the ER relationship, the embodiment of the present application can optimize the data model based on the ER relationship, determine the main table, and generate a many-to-one data integration statement for processing complex data relationships.

[0120] In an optional implementation example, after the ER relationship and its similarity information are pushed to the graph engine, the embodiment of the present application can use the graph engine to build a complex graph containing multiple logical entities and their relationships, that is, graph data. It should be noted that the graph engine can be used to maintain and operate data relationships in graph databases such as Neo4j; wherein, Neo4j is a graph database management system that uses nodes and edges to represent and store data, such as data entities can be represented as nodes, and the relationship between data can be represented as connecting edges between nodes, so that the use of a graph database can intuitively represent and maintain the relationship between entities, such as the relationship between orders and customers; further, the use of a graph database can easily query the relationship between multiple tables, for example, such as quickly finding a relationship chain through a series of associated nodes.

[0121] Therefore, after building the graph, determining the shortest path between multiple logical entities can be an application of the graph, which is valuable in certain business analyses (such as supply chain analysis, network optimization, etc.). For example, the shortest interaction path between two entities can be quickly found to help optimize processes or costs. Furthermore, when generating SQL statements for data integration, the determined master table (the table that dominates the data structure) directly affects the efficiency of data processing and querying. The embodiment of the present application can use the graph to dynamically determine the master table based on the relationship weights and connectivity between entities.

[0122] It is necessary to further explain that the above intelligent matching algorithm can receive metadata information and semantic content of data from the source database as input to provide basic data for subsequent matching and analysis; the intelligent matching algorithm can use a dual evaluation mechanism when performing matching and analysis:

[0123] Text similarity scoring, such as using a text similarity algorithm to calculate the similarity of metadata and semantic content between data, so as to evaluate the similarity of text information such as field names and descriptions and obtain similarity values;

[0124] The big model performs matching, for example, using machine algorithms (such as machine learning or deep learning models) to analyze and match the similarity of metadata and semantic content between data to obtain a similarity value;

[0125] Therefore, the embodiment of the present application may set up a policy engine to perform weighted average on the similarity values ​​calculated by the above-mentioned dual evaluation mechanism to obtain a comprehensive evaluation score, which is used for the final mapping information recommendation.

[0126] Furthermore, when recommending mapping information, the embodiment of the present application can select the top N tables and fields with the highest similarity according to the recommendation ranking information configured by the user, and then recommend mapping information of the tables and fields for constructing SQL statements.

[0127] Furthermore, when making ER relationship recommendations, the embodiments of the present application can pull data from multiple customer tables into the ODS library; analyze key information such as business primary keys of the data from multiple customer tables through algorithms, perform data matching, and construct an ER graph to represent the relationships between multiple customer tables; and then, output the relationship scores between multiple customer tables for graph construction and further data processing.

[0128] Furthermore, the embodiment of the present application can integrate mapping information and entity relationships. For example, the embodiment of the present application can construct complex SQL statements based on the results of RP recommendation (mapping information) and ER recommendation (entity relationships) to perform final data integration. For example, SQL statements can be constructed based on the field mapping obtained by RP recommendation and the table relationship obtained by ER relationship recommendation.

[0129] In a further optional implementation, smart matching can be divided into full matching and real-time matching; full matching can be a step performed before data collection, mainly used to generate mapping information between tables in the source database and target tables, including table mapping relationships and field mapping information, etc. Real-time matching is for tables that are not covered by full matching, that is, for tables that are not covered by full matching, ER relationships between tables are generated, thereby supporting the generation of complex multi-table association statements.

[0130] Furthermore, the embodiment of the present application can integrate the mapping information obtained by full matching and the ER relationship obtained by real-time matching to generate the data integration statements (such as SQL statements) of the embodiment of the present application, and the data integration statements are divided into full data collection statements and incremental data collection statements; wherein, the full data collection statement is used to collect the full data from the source database to the intermediate library and then migrate it to the target library when querying the existence of data records in the source database; the incremental data collection statement can be a data integration statement with an incremental update clause, which is used to process data updates in the source database and migrate the updated incremental data to the target library through the intermediate library.

[0131] In an optional implementation, the mapping information in the SQL statement may involve matching the target table and field specified in the INSERT INTO clause to determine how data is mapped from the table of the source database to the table of the target database. The ER relationship in the SQL statement may involve matching the association condition in the JOIN clause.

[0132] The embodiment of the present application uses an intelligent matching algorithm to process the text data of the source database, and combines machine algorithms such as machine models to analyze the semantic content of the data, so as to better understand and process the semantic level of the data and improve the accuracy and flexibility of matching; compared with the method of vectorizing the data into a series of numerical representations (involving feature extraction and dimensionality reduction, etc.), and then using a vector similarity meter to determine similar entity vectors, the embodiment of the present application does not ignore the semantic information of the data and can improve accuracy; that is, the calculation of vector similarity depends on accurate feature extraction, which may ignore certain semantic information of the data, resulting in inaccuracy. Furthermore, the embodiment of the present application can improve efficiency and reduce duplication of work by constructing a data knowledge graph. Furthermore, the use of weighted average and recommendation ranking strategies for similarity assessment allows for more detailed and personalized evaluation criteria.

[0133] The above intelligent matching and automated data integration methods can greatly improve the efficiency and accuracy of data integration. By automatically identifying and utilizing the relationship between data, it not only reduces the workload of manual configuration, but also improves the quality of data integration, allowing enterprises to better utilize their data assets, support complex business analysis and decision-making processes, and can be used in enterprise data environments with large data volumes, complex structures, and frequent updates.

[0134] In a further optional implementation, combining Figure 1 As shown, the data integration platform may also include a data storage and metadata management system 120 , a data query and application system 140 , and an infrastructure and monitoring system 150 .

[0135] For example, the data storage and metadata management system 120 may be configured with a Catalog component, an ODS library, and an enterprise standard model. The Catalog component is used to manage metadata to support data governance and data asset management. The introduction of the ODS library and the enterprise standard model can be referred to the previous description, which will not be expanded here.

[0136] The data query and application system 140 is used for data query and application. For example, the integrated data can be applied to the data query and application system to support application scenarios such as online transaction processing and online analytical processing. For example, the data query and application system may include:

[0137] OLTP (On-Line Transaction Processing) system, used to process online transaction requests, suitable for business operations that require fast response;

[0138] OLAP (On-Line Analytical Processing) systems are used for complex analytical queries and are suitable for in-depth data analysis;

[0139] Ad-hoc (temporary) query system, used to support ad hoc query of data, suitable for irregular data exploration and analysis needs.

[0140] The infrastructure and monitoring system 150 is used to provide basic capability support for the entire data integration platform, including but not limited to providing:

[0141] Open source container orchestration platforms, such as Kubernetes, for automated application container deployment, scaling, and management;

[0142] Graph lineage, which is used to track and visualize the flow path of data from source (e.g. data source) to target (e.g. final integrated data), such as detailed records of data movement, transformation, and processing;

[0143] Data asset management involves the management of the entire life cycle of data, such as the collection, storage, maintenance, use and archiving of data. Data asset management involves metadata management (managing data-related meta-information), data catalog management (providing an easily searchable data resource library so that users can quickly find the data they need), data governance management (managing data security, privacy protection, legality, etc.), data quality control (monitoring and improving data accuracy, availability and consistency), etc.;

[0144] The scheduling platform is responsible for managing and optimizing the execution of various data jobs in the data integration environment, involving task scheduling (automatically scheduling task execution time), task resource management (rationally allocating computing resources for tasks to ensure efficient operation of tasks), task dependency management (handling dependencies between tasks to ensure that tasks are executed in the correct order), monitoring and alarming (real-time monitoring of task status, providing a mechanism for rapid response and notification of relevant personnel in the event of anomalies), etc.

[0145] The data integration platform provided in the embodiment of the present application can be used to process data integration tasks from different data sources, including databases, Excel files, APIs, and MQ, etc.; the data integration platform can use full-link scheduling and data asset management technology, as well as intelligent matching and evaluation strategies, to optimize the data integration process. Specifically, the embodiment of the present application supports multiple types of data sources, including databases, Excel files, data accessed through APIs, and data streams from message queues; and realizes the automatic scheduling of data flow from the data source to the target library, ensuring the correctness and timeliness of the data during the entire transmission process; at the same time, the embodiment of the present application can handle full data collection and incremental data collection, effectively manage data load and synchronization frequency, and ensure the accuracy of full data and incremental data synchronization, thereby improving the efficiency of the data integration process.

[0146] Furthermore, the embodiments of the present application can use intelligent matching algorithms to analyze and recommend mapping information, ER relationships, and even integrate mapping relationships and ER relationships to construct data integration statements to ensure the efficiency and accuracy of data integration, avoid the technical barriers brought about by manually writing data integration statements (such as SQL statements), and reduce the limitations of data integration.

[0147] As an optional implementation, the embodiment of the present application further provides a data integration device, which can be regarded as a functional module device corresponding to the data integration method provided in the embodiment of the present application. The following description can be referenced to the above description.

[0148] Optional, Figure 7 An optional block diagram of a data integration device is shown as an example. Figure 7 As shown, the device may include:

[0149] The query request sending module 710 is used to send an active query request to the source database after configuring the connection information of the source database to query whether there is a data record or data update in the source database;

[0150] The full data integration module 720 is used to pull the full data of the source database to the intermediate table of the intermediate database through full data collection when there are data records in the source database, and pull it from the intermediate table to the target table of the target database; the intermediate database is the database corresponding to the intermediate storage system of the data integration platform, and the target database is the database using the enterprise standard model;

[0151] The incremental data integration module 730 is used to pull the incremental data of the source database to the intermediate table of the intermediate database through incremental data collection when there is data update in the source database, and then pull it from the intermediate table to the target table of the target database;

[0152] Among them, for full data collection, the full data integration module collects the full data of the source database into the intermediate table of the intermediate library, and then pulls the full data from the intermediate table of the intermediate library to the target table of the target library according to the data integration statement; for incremental data collection, the incremental data integration module collects the incremental data of the source database into the intermediate table of the intermediate library according to the incremental update field, and then uses the data integration statement with the incremental update clause to pull the incremental data from the intermediate table of the intermediate library to the target table of the target library; the process of pulling data from the intermediate table to the target table also performs data type conversion and field mapping.

[0153] Optionally, for the full data collection, collecting the full data of the source database into the intermediate table of the intermediate database includes:

[0154] By configuring the Reader plug-in that adapts to the source database and the Writer plug-in of the intermediate library, and by passing the JSON configuration, data is collected from the source database to the intermediate table;

[0155] The JSON configuration includes at least: Reader configuration information for configuring reading data from a source database, Writer configuration information for configuring writing data into an intermediate database, and Transformer configuration information for indicating a conversion operation in the process of transferring data from a data source to an intermediate table;

[0156] Optionally, for incremental data collection, collecting incremental data of the source database into an intermediate table of the intermediate database according to the incremental update field includes:

[0157] In the Writer configuration information, the attribute of the write mode is set to update to indicate that data of existing records in the intermediate table is updated, and in the Writer configuration information, field information is specified to indicate fields in the intermediate table that belong to incremental data;

[0158] According to the updated writing mode set in the Writer configuration information and the field information belonging to the incremental data, the incremental data of the source database is collected into the intermediate table of the intermediate library.

[0159] Optionally, pulling the full amount of data from the intermediate table of the intermediate database to the target table of the target database according to the data integration statement includes:

[0160] Pull all data from the intermediate table of the intermediate database to the target table of the target database through data integration statements with data pulling process, data integration process, data filtering process, and data insertion process;

[0161] Among them, the data pulling process is implemented through the SELECT clause, which is used to select and pull data from the intermediate table; the data integration process is implemented through the JOIN clause, which is used to merge the data pulled from multiple intermediate tables through association conditions; the data filtering process is implemented through the WHERE clause, which is used to select records that meet the filtering conditions from the data merged in the data integration process to achieve data filtering; the data insertion process is implemented through the INSERT INTO clause, which is used to specify the target table and field.

[0162] Optionally, the method of using a data integration statement with an incremental update clause to pull incremental data from an intermediate table of the intermediate database to a target table of the target database includes:

[0163] When using data integration statements to pull data from the intermediate database to the target database, incremental data is pulled from the intermediate table of the intermediate database to the target table of the target database through an incremental update clause with a conflict detection process and an update execution process;

[0164] Among them, the conflict detection process is used to trigger a conflict when trying to insert records from the intermediate table into the target table if the business primary key of the record already exists in the target table; the update execution process is used to replace the old data of the conflicting record in the target table with the incremental data in the intermediate table for the conflicting record when a conflict occurs.

[0165] In an optional implementation, a data integration statement may include a full data collection statement, and a data integration statement with an incremental update clause may include an incremental data collection statement; wherein the full data collection statement and the incremental data collection statement are determined according to the mapping information obtained by the full match and the ER relationship obtained by the real-time match; the full match is used to generate mapping information between a table in a source database and a target table, and the real-time match is used to generate an ER relationship between tables for tables not covered by the full match;

[0166] Furthermore, the mapping information matches the target table and field specified in the INSERT INTO clause; and the ER relationship matches the association condition in the JOIN clause.

[0167] Optionally, the data integration statement is generated at least based on mapping information, where the mapping information is mapping information between fields in a table data structure of a source database and corresponding fields of entities in an enterprise standard model.

[0168] Optional, combined Figure 7As shown, the device may also include an intelligent matching module 740, which is used to configure the connection information of the source database and import the key data table of the source database; use the intelligent matching algorithm to analyze the fields in the key data table of the source database, and match the fields with the corresponding entities in the enterprise standard model; based on the matching results, recommend the mapping information of the fields in the table data structure of the source database and the corresponding fields of the entities in the enterprise standard model;

[0169] Alternatively, after pulling the data from the source database to the intermediate database, ER matching is performed on the data pulled to the intermediate database to obtain the recommended ER relationship; after the recommended ER relationship is confirmed by the user, the data model is optimized based on the ER relationship, the main table is determined, and a many-to-one data integration statement is generated;

[0170] Alternatively, integrate mapping information and ER relationships to build data integration statements.

[0171] Optional, combined Figure 7 As shown, the device may also include a real-time data acquisition module 750, which is used to monitor the change log of the source database in real time to capture data change events of the source database in real time; format the data change events into messages, and send them to the specified topic of Kafka through the source connector of Kafka in real time; consume the messages of the specified topic through the downstream system or downstream service of Kafka, and apply the change data corresponding to the message to the intermediate library or target library in real time according to the consumed message; wherein the downstream system or downstream service of Kafka uses the aggregation connector of Kafka to subscribe to and consume the messages of the specified topic from Kafka.

[0172] Optional, combined Figure 7 As shown, the apparatus may further include: a source file system data integration module 760, and / or, an API data integration module 770, and / or, an MQ data integration module 780;

[0173] Among them, the source file system data integration module 760 performs data integration based on the source file system; the API data integration module 770 performs data integration based on the source network API interface; and the MQ data integration module 780 performs data integration based on the source MQ.

[0174] Optionally, performing data integration based on the source file system includes:

[0175] For the full import of source files in the source file system, the source files to be imported are selected through user configuration, and the mapping relationship between the data page area of ​​the source file and the logical entity fields of the intermediate table or the target table is configured; the page content of the source file is parsed, an SQL statement is generated according to the mapping relationship, and the SQL statement is executed to clear the existing data in the intermediate table or the target table, and insert all the data in the source file into the intermediate table or the target table;

[0176] For the additional import of the source file, the user configuration is reused; the existing data in the intermediate table or the target table is checked, and the additional data of the source file is appended to the intermediate table or the target table according to the mapping relationship configured by the user, so as to update the data with the existing records in the intermediate table or the target table.

[0177] Optionally, the performing of data integration based on the source network API interface includes:

[0178] Configure the source network API interface, and in the data preview stage based on the configured source network API interface, define the data mapping relationship of the source network API interface; and create a corresponding intermediate table according to the data mapping relationship of the source network API interface; wherein, defining the data mapping relationship of the source network API interface includes defining a data path and configuring a primary key field, and defining the data path is to define the used part of the data structure returned by the source network API interface;

[0179] The source network API interface is called and the network request of the source network API interface is processed to obtain the data corresponding to the data path from the source network API interface; a JSON script is generated according to the data obtained from the source network API interface to load the obtained data into the created intermediate table; wherein, in the process of loading the data into the intermediate table, the primary key field configured by the data mapping relationship is used to handle data duplication or data conflict.

[0180] Optionally, the performing of data integration based on the source MQ includes:

[0181] Configure MQ instance tasks and provide data preview function for MQ instance tasks so as to view the message data read from the monitored MQ;

[0182] Configure the JDBC aggregation connector to specify the connection information of the intermediate library; and write the message data read from MQ to the intermediate library;

[0183] Among them, the MQ instance task is connected to the MQ source connector, and the MQ source connector is used to read message data from the specified MQ to be listened, and push the read message data to the Kafka message queue corresponding to the specified topic. If the message data contains data in list form, the data in list form is formatted into the format accepted by the JDBC aggregation connector, so that the JDBC aggregation connector reads data from the Kafka message queue corresponding to the specified topic of Kafka and writes it into the intermediate library.

[0184] In a further optional implementation, the embodiment of the present application also provides a server, such as a server device corresponding to a data integration platform, comprising at least one memory and at least one processor; wherein the memory stores computer executable instructions, and the processor calls the computer executable instructions to execute a data integration method as provided in the embodiment of the present application.

[0185] In a further optional implementation, the embodiment of the present application also provides a storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed, the data integration method provided in the embodiment of the present application is implemented.

[0186] In a further optional implementation, the embodiment of the present application also provides a computer program product, which includes computer executable instructions. When the computer executable instructions are executed, the data integration method provided in the embodiment of the present application is implemented.

[0187] The above describes multiple implementation schemes provided by the embodiments of the present application. The various optional methods introduced in each implementation scheme can be combined and cross-referenced with each other without conflict, thereby extending a variety of possible implementation schemes, which can all be considered as implementation schemes disclosed and open in the embodiments of the present application.

[0188] Although the embodiments of the present application are disclosed above, the present application is not limited thereto. Any person skilled in the art may make various changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be subject to the scope defined by the claims.

Claims

1. A data integration method, characterized in that: By means of intelligent matching and automatic mapping, data integration from the data source to the intermediate database and then to the target database is at least realized; intelligent matching is divided into full matching and real-time matching; full matching is performed before data collection to generate mapping information between the tables in the source database and the target table, including the mapping relationship of the tables and the mapping information of the fields; Real-time matching generates ER relationships between tables that are not covered by full matching, and supports the generation of multi-table association statements; The method comprises: Integrate the mapping information obtained by full matching and the ER relationship obtained by real-time matching to generate data integration statements, so as to transfer the data from the source database to the target database of the enterprise standard model from the intermediate database after transferring the data from the source database to the intermediate database; the data integration statements are divided into full data collection statements and incremental data collection statements, and the incremental data collection statements are data integration statements with incremental update clauses; After configuring the connection information of the source database, an active query request is sent to the source database to query whether there are data records or data updates in the source database; When there are data records in the source database, the full data of the source database is pulled to the intermediate table of the intermediate database through full data collection, and then pulled from the intermediate table to the target table of the target database; full data collection involves copying all data; the intermediate database is the database corresponding to the intermediate storage system of the data integration platform, and the target database is the database of the data integration platform that adopts the enterprise standard model; When there is data update in the source database, incremental data of the source database is pulled to the intermediate table of the intermediate database through incremental data collection, and then pulled from the intermediate table to the target table of the target database; incremental data collection migrates data records that already exist in the intermediate table but the data content has changed from the source database to the intermediate table; Among them, for full data collection, the full data of the source database is collected into the intermediate table of the intermediate library, and then according to the full data collection statement, the full data is pulled from the intermediate table of the intermediate library to the target table of the target library; for incremental data collection, according to the incremental update field, the incremental data of the source database is collected into the intermediate table of the intermediate library, and then the data integration statement with the incremental update clause is used to pull the incremental data from the intermediate table of the intermediate library to the target table of the target library; the process of pulling data from the intermediate table to the target table also uses the metadata information of the data asset platform and the mapping information of the page configuration to perform data type conversion and field mapping.

2. The method according to claim 1, characterized in that: For the full data collection, collecting the full data of the source database into the intermediate table of the intermediate database includes: By configuring the Reader plug-in that adapts to the source database and the Writer plug-in of the intermediate library, and by passing the JSON configuration, data is collected from the source database to the intermediate table; The JSON configuration includes at least: Reader configuration information for configuring reading data from a source database, Writer configuration information for configuring writing data into an intermediate database, and Transformer configuration information for indicating a conversion operation in the process of transferring data from a data source to an intermediate table; For the incremental data collection, the incremental data of the source database is collected into the intermediate table of the intermediate database according to the incremental update field, including: In the Writer configuration information, the attribute of the write mode is set to update to indicate that data of existing records in the intermediate table is updated, and in the Writer configuration information, field information is specified to indicate fields in the intermediate table that belong to incremental data; According to the updated writing mode set in the Writer configuration information and the field information belonging to the incremental data, the incremental data of the source database is collected into the intermediate table of the intermediate library.

3. The method according to claim 1, characterized in that: Pulling the full amount of data from the intermediate table of the intermediate database to the target table of the target database according to the full amount of data collection statement includes: Pull all data from the intermediate table of the intermediate database to the target table of the target database through data integration statements with data pulling process, data integration process, data filtering process, and data insertion process; The data pulling process is implemented through the SELECT clause, which is used to select and pull data from the intermediate table; the data integration process is implemented through the JOIN clause, which is used to merge the data pulled from multiple intermediate tables through association conditions; the data filtering process is implemented through the WHERE clause, which is used to select records that meet the filtering conditions from the data merged in the data integration process to achieve data filtering; the data insertion process is implemented through the INSERT INTO clause, which is used to specify the target table and field; The method of using a data integration statement with an incremental update clause to pull incremental data from an intermediate table of an intermediate database to a target table of a target database includes: When using data integration statements to pull data from the intermediate database to the target database, incremental data is pulled from the intermediate table of the intermediate database to the target table of the target database through an incremental update clause with a conflict detection process and an update execution process; Among them, the conflict detection process is used to trigger a conflict when trying to insert records from the intermediate table into the target table if the business primary key of the record already exists in the target table; the update execution process is used to replace the old data of the conflicting record in the target table with the incremental data in the intermediate table for the conflicting record when a conflict occurs.

4. The method according to claim 3, characterized in that The mapping information obtained by the full match matches the target table and field specified in the INSERT INTO clause; the ER relationship matches the association condition in the JOIN clause.

5. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: Configure the connection information of the source database and import the key data tables of the source database; use the intelligent matching algorithm to analyze the fields in the key data tables of the source database and match the fields with the corresponding entities in the enterprise standard model; based on the matching results, recommend the mapping information between the fields in the table data structure of the source database and the corresponding fields of the entities in the enterprise standard model; Alternatively, after pulling the data from the source database to the intermediate database, ER matching is performed on the data pulled to the intermediate database to obtain recommended ER relationships; after the recommended ER relationships are confirmed by the user, the data model is optimized based on the ER relationships, the main table is determined, and many-to-one data integration statements are generated.

6. The method according to any one of claims 1 to 3, characterized in that: Also includes: Monitor the change log of the source database in real time to capture data change events of the source database in real time; Format data change events into messages and send them to the specified topic of Kafka in real time through Kafka's source connector; consume messages of the specified topic through Kafka's downstream system or downstream service, and apply the change data corresponding to the message to the intermediate library or target library in real time based on the consumed message; Kafka's downstream system or downstream service uses Kafka's aggregation connector to subscribe to and consume messages of the specified topic from Kafka; And / or, perform data integration based on the source file system, and / or, perform data integration based on the source network API interface, and / or, perform data integration based on the source MQ; The performing of data integration based on the source file system includes: For the full import of source files in the source file system, the source files to be imported are selected through user configuration, and the mapping relationship between the data page area of ​​the source file and the logical entity fields of the intermediate table or the target table is configured; the page content of the source file is parsed, an SQL statement is generated according to the mapping relationship, and the SQL statement is executed to clear the existing data in the intermediate table or the target table, and insert all the data in the source file into the intermediate table or the target table; For the additional import of the source file, the user configuration is reused; the existing data in the intermediate table or the target table is checked, and the additional data of the source file is appended to the intermediate table or the target table according to the mapping relationship configured by the user, so as to update the data with the existing records in the intermediate table or the target table; The execution of data integration based on the source network API interface includes: Configure the source network API interface, and in the data preview stage based on the configured source network API interface, define the data mapping relationship of the source network API interface; and create a corresponding intermediate table according to the data mapping relationship of the source network API interface; wherein, defining the data mapping relationship of the source network API interface includes defining a data path and configuring a primary key field, and defining the data path is to define the used part of the data structure returned by the source network API interface; Call the source network API interface and process the network request of the source network API interface to obtain data corresponding to the data path from the source network API interface; generate a JSON script based on the data obtained from the source network API interface to load the obtained data into the created intermediate table; wherein, in the process of loading data into the intermediate table, the primary key field configured by the data mapping relationship is used to handle data duplication or data conflict; The performing of data integration based on source MQ includes: Configure MQ instance tasks and provide data preview function for MQ instance tasks so as to view the message data read from the monitored MQ; Configure the JDBC aggregation connector to specify the connection information of the intermediate library; and write the message data read from MQ to the intermediate library; Among them, the MQ instance task is connected to the MQ source connector, and the MQ source connector is used to read message data from the specified MQ to be listened, and push the read message data to the Kafka message queue corresponding to the specified topic. If the message data contains data in list form, the data in list form is formatted into the format accepted by the JDBC aggregation connector, so that the JDBC aggregation connector reads data from the Kafka message queue corresponding to the specified topic of Kafka and writes it into the intermediate library.

7. A data integration device, characterized in that: By means of intelligent matching and automatic mapping, data integration from the data source to the intermediate database and then to the target database is at least realized; intelligent matching is divided into full matching and real-time matching; full matching is performed before data collection to generate mapping information between the tables in the source database and the target table, including the mapping relationship of the tables and the mapping information of the fields; Real-time matching generates ER relationships between tables that are not covered by full matching, and supports the generation of multi-table association statements; The device is used to integrate the mapping information obtained by full matching and the ER relationship obtained by real-time matching, and generate data integration statements to transfer the data from the intermediate database to the target database of the enterprise standard model after transferring the data from the source database to the intermediate database; The data integration statements are divided into full data collection statements and incremental data collection statements. Incremental data collection statements are data integration statements with incremental update clauses. The device comprises: A query request sending module is used to send an active query request to the source database after configuring the connection information of the source database to query whether there is data record or data update in the source database; The full data integration module is used to pull the full data of the source database to the intermediate table of the intermediate database through full data collection when there are data records in the source database, and pull it from the intermediate table to the target table of the target database; full data collection involves copying all data; the intermediate database is the database corresponding to the intermediate storage system of the data integration platform, and the target database is the database of the data integration platform that adopts the enterprise standard model; The incremental data integration module is used to pull the incremental data of the source database to the intermediate table of the intermediate database through incremental data collection when there is data update in the source database, and then pull it from the intermediate table to the target table of the target database; incremental data collection migrates data records that already exist in the intermediate table but the data content has changed from the source database to the intermediate table; Among them, for full data collection, the full data integration module collects the full data of the source database into the intermediate table of the intermediate library, and then pulls the full data from the intermediate table of the intermediate library to the target table of the target library according to the full data collection statement; for incremental data collection, the incremental data integration module collects the incremental data of the source database into the intermediate table of the intermediate library according to the incremental update field, and then uses the data integration statement with the incremental update clause to pull the incremental data from the intermediate table of the intermediate library to the target table of the target library; the process of pulling data from the intermediate table to the target table also uses the metadata information of the data asset platform and the mapping information of the page configuration to perform data type conversion and field mapping.

8. A server, characterized in that: include: At least one memory and at least one processor, the memory storing computer executable instructions, the processor calling the computer executable instructions to execute the data integration method according to any one of claims 1-6.

9. A storage medium, characterized in that: The storage medium stores computer executable instructions, and when the computer executable instructions are executed, the data integration method according to any one of claims 1 to 6 is implemented.

10. A computer program product, characterized in that It comprises computer executable instructions, and when the computer executable instructions are executed, it implements the data integration method as described in any one of claims 1 to 6.