Heterogeneous data synchronization method, device and computer-readable storage medium

Configure data integration tasks through the system interface and generate Json templates to automatically perform heterogeneous data synchronization, solving the problem of target library creation errors during multi-table synchronization, and improving the efficiency and consistency of heterogeneous data synchronization.

CN118606410BActive Publication Date: 2024-11-05SHENZHEN SMARTCITY TECH DEV GRP CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411096417.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-12
Publication Date
2024-11-05
Estimated Expiration
2044-08-12

AI Technical Summary

Technical Problem

In the case of multi-table synchronization, existing heterogeneous data synchronization methods require the creation of a large number of target tables in the target library in advance, which is prone to unnecessary errors and lead to low synchronization efficiency.

Method used

Receive configuration information through the system interface, configure the data source, data target and task configuration items of the data integration task, and generate integrated synchronization jobs and Json templates. Based on preset commands, the synchronization service component is triggered to execute data synchronization tasks, and adopt integrated automation and batch processing Method to reduce manual operation and workload.

Benefits of technology

Improves the synchronization efficiency of heterogeneous data, reduces the risk of configuration errors, and ensures the consistency and maintainability of data synchronization.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118606410B_ABST
    Figure CN118606410B_ABST
Patent Text Reader

Abstract

The present invention discloses a synchronization method, device and computer-readable storage medium for heterogeneous data, and belongs to the field of data processing. The synchronization method for heterogeneous data includes the following steps: based on the configuration information received by the system interface, the data source, data destination and task configuration items of the data integration task are configured, and the data integration task is released; according to the task instance of the data integration task, the data source and the data destination, an integrated synchronization job and a Json template are generated; when the task scheduling process is triggered, a synchronization job Json is generated according to the Json template and the integrated synchronization job; based on a preset command, the synchronization service component is triggered to execute the data synchronization task according to the synchronization job Json. Through the above method, integrated automation and batch processing are adopted to reduce manual operations and workload, and improve the synchronization efficiency of heterogeneous data.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data synchronization, and in particular to a method and device for synchronizing heterogeneous data and a computer-readable storage medium. Background Art

[0002] When implementing database data migration, in order to ensure data consistency between different systems, heterogeneous database data synchronization and processing is an indispensable part of the development of big data. The synchronization tasks implemented by current data synchronization technology usually integrate the full amount of offline heterogeneous data or a single table. If you want to synchronize data migration for a large number of tables, you need to create a large number of data synchronization tasks separately. This means that if you are synchronizing multiple tables, you need to create a large number of destination tables in the destination library in advance, which is prone to unnecessary errors and leads to low synchronization efficiency.

[0003] The above contents are only used to assist in understanding the technical solution of the present invention and do not constitute an admission that the above contents are prior art. Summary of the invention

[0004] The main purpose of the present invention is to provide a method, device and computer-readable storage medium for synchronizing heterogeneous data, aiming to solve the technical problem that the current method for synchronizing heterogeneous data requires creating a large number of destination tables in the destination library in advance when synchronizing multiple tables, which is prone to unnecessary errors and leads to low synchronization efficiency.

[0005] To achieve the above object, the present invention provides a method for synchronizing heterogeneous data, the method comprising the following steps:

[0006] Based on the configuration information received by the system interface, configure the data source, data destination and task configuration items of the data integration task, and publish the data integration task;

[0007] Generate an integrated synchronization job and a Json template according to the task instance of the data integration task, the data source and the data destination;

[0008] When the task scheduling process is triggered, a synchronization job Json is generated according to the Json template and the integrated synchronization job;

[0009] The synchronization service component is triggered based on the preset command to perform the data synchronization task according to the synchronization job Json.

[0010] Optionally, the step of configuring the data source, data destination and task configuration items of the data integration task based on the configuration information received by the system interface, and publishing the data integration task includes:

[0011] Based on the configuration mode triggered by the system interface, display the corresponding configuration page;

[0012] Determine and configure the data source, data destination and task configuration items of the data integration task based on the library table information, SQL statements and / or custom content written in the configuration page;

[0013] The configured task instance of the data integration task is saved and published.

[0014] Optionally, before the step of saving and publishing the task instance of the configured data integration task, the method further includes:

[0015] If the data integration task is a data synchronization task, the table structure and field type between the data source end and the data destination end are visually mapped and configured;

[0016] If the data integration task is a whole database synchronization task, when there is a table name conflict at the data destination end, use the existing table or delete the table to verify the table name of the newly created table, collect the data structure and attribute type in the metadata content according to the data source end table, and perform type mapping on the heterogeneous data destination end.

[0017] Optionally, the step of generating an integrated synchronization job and a Json template according to the task instance of the data integration task, the data source and the data destination includes:

[0018] According to the task instance, the data source and the data destination, configure the data synchronization strategy, scheduling rules, dependencies between tasks and exception handling strategy;

[0019] Generate the integrated synchronization job based on the data synchronization strategy, the scheduling rules, the dependency relationship and the exception handling strategy;

[0020] Based on the integrated synchronization job, the Json template is created.

[0021] Optionally, when the task scheduling process is triggered, the step of generating a synchronization job Json according to the Json template and the integrated synchronization job includes:

[0022] Based on the Json template, replace and fill in the connection configuration information of the data source and data destination;

[0023] If the synchronization mode of the integrated synchronization job is incremental, fill in the filter condition statement based on the Json template according to the incremental field and incremental value;

[0024] Based on the integrated synchronization job, replace the placeholder of the Json template;

[0025] The synchronization job Json is generated based on the filled content of the Json template.

[0026] Optionally, the step of triggering the synchronization service component to execute the data synchronization task according to the synchronization job Json based on a preset command includes:

[0027] Generate corresponding read port instance and write port instance based on the configuration information of the synchronization job Json;

[0028] Based on the read port instance, read data from the data source, and perform data extraction and cleaning on the data;

[0029] Based on the write port instance, the read data is written to the data destination end to perform the data synchronization task.

[0030] Optionally, after the step of triggering the synchronization service component to execute the data synchronization task according to the synchronization job Json based on the preset command, the method further includes:

[0031] Based on the callback function and the preset communication protocol, sending a result request of the task instance to the data integration service;

[0032] The data integration service updates the task execution status and synchronization information based on the result request;

[0033] The task execution status and the synchronization information are displayed based on the system interface.

[0034] Optionally, after the step of sending the result request of the task instance to the data integration service based on the callback function and the preset communication protocol, the step further includes:

[0035] If the integration service is an incremental task of data synchronization, the data value corresponding to the incremental field in the source end table is obtained based on the callback function;

[0036] Recording the data value into an integrated task table;

[0037] When executing the next incremental synchronization task, the incremental synchronization task is executed based on the data value in the integrated task table.

[0038] In addition, to achieve the above-mentioned purpose, the present invention also provides a heterogeneous data synchronization device, which includes: a memory, a processor, and a heterogeneous data synchronization program stored in the memory and executable on the processor, and the heterogeneous data synchronization program is configured to implement the steps of the heterogeneous data synchronization method as described above.

[0039] In addition, to achieve the above objectives, the present invention also provides a computer-readable storage medium, on which a synchronization program for heterogeneous data is stored, and when the synchronization program for heterogeneous data is executed by a processor, the steps of the heterogeneous data synchronization method as described above are implemented.

[0040] The embodiment of the present invention provides a method for synchronizing heterogeneous data. Based on the configuration information received by the system interface, the data integration task is configured. The configuration content includes the data source, data destination and other task configuration items. After the configuration is completed, the data integration task is published. The data integration service generates an integrated synchronization job and a Json template according to the task instance, data source and data destination of the data integration task. When the task scheduling process is triggered, the working node of the scheduling system generates a synchronization job Json according to the Json template and the integrated synchronization job. Finally, based on the preset command, the synchronization service component is triggered to execute the data synchronization task according to the synchronization job Json. Through the above steps, the integrated automation and batch processing method is adopted to reduce manual operations and workload, and improve the synchronization efficiency of heterogeneous data. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A schematic diagram of a process flow of a first embodiment of synchronization of heterogeneous data according to the present invention;

[0042] Figure 2 A schematic diagram of the process of the second embodiment of synchronization of heterogeneous data of the present invention;

[0043] Figure 3 A schematic diagram of a process flow of a third embodiment of synchronization of heterogeneous data according to the present invention;

[0044] Figure 4 It is a flow chart of the data integration and processing method of heterogeneous data sources of the present invention;

[0045] Figure 5 Data integration flow chart for data synchronization module;

[0046] Figure 6 Synchronize data integration flow chart for the entire database;

[0047] Figure 7 It is a schematic diagram of the terminal structure of the hardware operating environment involved in the embodiment of the present invention.

[0048] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0049] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0050] The embodiment of the present invention provides a method for synchronizing heterogeneous data. Figure 1 , Figure 1 The figure is a flow chart of a first embodiment of a method for synchronizing heterogeneous data according to the present invention.

[0051] In this embodiment, the method for synchronizing heterogeneous data includes:

[0052] Step S10: Based on the configuration information received by the system interface, the data source, data destination and task configuration items of the data integration task are configured, and the data integration task is published.

[0053] In this embodiment, refer to Figure 4 Users can configure data integration tasks through the system interface. Data integration tasks are divided into data integration and whole database synchronization. Users can create data synchronization or whole database synchronization tasks according to their own needs on the system interface. According to the synchronization requirements of data synchronization and whole database synchronization, synchronization processing can be performed for single table and multi-table synchronization respectively.

[0054] Furthermore, the user performs basic information of the data integration task in the system interface, mainly including data source configuration and data destination configuration. Through the configuration information received through the system interface, the data source, data destination, and task configuration items of the data integration task can be configured. Among them, the data source includes the type of data source, such as database, API (Application Programming Interface), file, etc.; connection information, including address, port, user name, password, etc.; data extraction method. The data destination includes the type of data destination, such as database, data warehouse, file system, etc., connection information and data loading method. The task configuration items mainly include scheduling strategies, task monitoring and alarm items, data conversion, cleaning, mapping and other specific task configuration contents.

[0055] In this embodiment, the data integration process can be automated by receiving configuration information and configuring data integration tasks through a system interface. This can save labor costs and time and improve work efficiency. At the same time, receiving configuration information through a unified system interface can ensure that each data integration task follows the same configuration standard, which helps maintain consistency and reduce the risk of configuration errors. In addition, storing configuration information in the system makes it easy to manage and maintain data integration tasks. When the configuration needs to be modified, it is only necessary to update the configuration information in the system without reconfiguring the entire task.

[0056] Step S20: Generate an integrated synchronization job and a Json template according to the task instance of the data integration task, the data source and the data destination.

[0057] In this embodiment, refer to Figure 4 After the configured data integration task is published, when the data integration task is executed manually or scheduled, the data integration service will generate an integration synchronization job and a Json template based on the task instance, data source, and data destination of the integration task. The data integration task defines the logic and rules of data processing, and the task instance is equivalent to the process actually executed at a specific point in time according to these logic and rules. The task instance includes the type of data source (database, API, file, etc.), data format, data structure and field mapping relationship. The data source and data destination ensure the data extraction, conversion and loading process. Through the task instance, data source and data destination, the integration synchronization job and Json template are further created. The integration synchronization job defines the steps and processes of data integration, and the Json template is used to ensure that data can be correctly transmitted and parsed between systems.

[0058] Further, in a feasible embodiment, the integrated task instance can be saved to the database, and the corresponding synchronization job can be created according to the requirements of the scheduling system, and the relevant task scheduling strategy, scheduling rules, dependencies between tasks, and exception handling strategies can be configured. According to the requirements of the data service datax (a data synchronization tool), the corresponding Json template of the data synchronization task is generated, including data source connection information, synchronization rules, target data source information, etc. Subsequently, the generated integrated synchronization job of the scheduling system and the Json template of the data service datax are published. It should be noted that the configuration information of the integrated synchronization job and the data Json template after publishing cannot be modified. If you want to edit it, you need to publish it offline. Among them, in the process of generating the Json template, it is necessary to determine the connection information of the source data and the target data, including the database address, port, user name, password, etc. This information will be used to connect the source data and the target data. According to the actual situation of data synchronization or whole database synchronization, determine the fields or library table contents that need to be synchronized, data conversion rules, etc. These rules are used to guide DataX on how to read, convert, and write target data from source data. In addition, this requires writing a JSON-formatted synchronization task template based on DataX specifications and requirements, combined with the connection information and synchronization rules defined above. The template should include data source connection information, synchronization rules, target data source information, etc.

[0059] In this embodiment, by generating integrated synchronization jobs and Json templates, the execution process of data integration tasks can be automated. Once the configuration is completed, these jobs are executed regularly through the scheduling system without manual intervention, thereby saving time and labor costs. In addition, the Json template defines the configuration information of the data integration task, including data sources, data destinations, field mapping, etc. By using templates, it can be ensured that each data integration task follows the same configuration standards, which helps to maintain consistency and maintainability.

[0060] Step S30: When the task scheduling process is triggered, a synchronization job Json is generated according to the Json template and the integrated synchronization job.

[0061] In this embodiment, refer to Figure 4 When the task scheduling process is triggered, the working node that executes the integration task in the scheduling system will generate the corresponding workflow task instance, that is, the synchronization job Json, based on the previously generated Json template and integration synchronization job, to ensure the mapping relationship between the data integration service instance and the workflow task instance, and ensure the uniqueness of the execution of the integration task.

[0062] Furthermore, in the specific execution process, the work node that executes task scheduling in the scheduling system requests the data integration service to obtain the task instance custom parameters and the Json template of the synchronization job corresponding to the previously generated integrated synchronization job through the HTTP (Hypertext Transfer Protocol) protocol service. Each synchronization job Json has a corresponding workflow ID, and the integrated synchronization job also has a corresponding synchronization job ID. The corresponding integration information can be obtained through the workflow ID and the synchronization job ID. Parameter placeholders are reserved in the Json template. During each execution process, these placeholders can be dynamically replaced by the template engine to update the data source connection configuration information between the data source end and the data destination end, and generate the synchronization job Json. In addition, if the work node determines that the synchronization mode is incremental through the custom parameters, it will splice according to the incremental field and the incremental value, generate the corresponding filter condition SQL statement and replace the corresponding field of the template. If the mapping field conversion is configured in the custom parameters, the corresponding converter plug-in will be generated for replacement.

[0063] Optionally, in another feasible implementation, configuration management tools may be used to store data source connection configuration information in these tools, and the latest configuration information may be dynamically obtained from the configuration management tools when executing tasks, and then the corresponding synchronization job Json may be constructed.

[0064] In this embodiment, the json template required for data synchronization is generated before executing the task, and after execution, different placeholders are dynamically replaced in the json template to generate a job json that can be executed for synchronization, which greatly improves the efficiency of data integration and synchronization processing. In the subsequent process, the working node of the scheduling system can trigger the data synchronization service datax component to execute the data synchronization process according to the json configuration through preset commands.

[0065] Step S40: triggering the synchronization service component to execute the data synchronization task according to the synchronization job Json based on the preset command.

[0066] In this embodiment, refer to Figure 4 The preset command refers to the shell (a command language) command. The work of the scheduling system triggers the DataX component of the data synchronization service through the shell command, and executes the data synchronization process according to the synchronization job Json configuration. As a feasible implementation method, a task can be set on the working node of the scheduling system, and a Shell script can be written in advance. The shell script is executed through the task, and the Json configuration information is received as a parameter, and the DataX component is called to execute the data synchronization task.

[0067] Furthermore, the data synchronization service datax will convert the submitted synchronization job Json into the reader (reading) end and writer writing end plug-in corresponding to the config (configuration), and execute the relevant heterogeneous data synchronization integration processing method. In the data integration service, the scheduling system generates the corresponding DataX configuration file by parsing the JSON configuration information of the synchronization job, and then triggers the DataX tool to execute the data synchronization task through the shell command. The DataX tool dynamically generates task instances including the reader end and the writer end according to the content in the configuration file, and executes the corresponding data conversion and synchronization process.

[0068] Among them, the reader side is responsible for reading data from the source data, and can extract and clean data according to the data source type, connection information and reading rules specified in the configuration file. The writer side is responsible for writing the processed data to the target data storage, and can also perform data conversion and integration according to the target source type, connection information and writing rules specified in the configuration file. DataX provides a rich plug-in mechanism that can support reading and writing operations of multiple data sources and data formats, and provides flexible conversion rules and filter condition settings to meet data integration needs in different scenarios. At the same time, DataX also supports task parallelism and distributed deployment, which can improve the efficiency and stability of data integration.

[0069] Further, refer to Figure 4After the data integration synchronization task is completed, the data synchronization and the whole database synchronization will perform a HOOK (callback function) callback, that is, in the callback function, the execution status and synchronization information of the updated integration task instance are sent to the data integration service through the HTTP protocol method. For the data synchronization instance result, the task success or failure status, the number of data read and written synchronously, and other information will be updated. For the whole database synchronization instance, the task success or failure status, the number of synchronization success tables and failure tables, and other information will be updated, and this information can be displayed through the system interface.

[0070] In addition, for the incremental tasks configured in data synchronization, the HOOK callback will obtain the latest data value of the corresponding incremental field in the source table through the JDBC connection method, and record the value in the integrated task table. The next time the incremental synchronization task is executed, data synchronization can be performed based on the saved incremental value. It should be noted that the first execution of the incremental task of data synchronization defaults to full synchronization. The update results of these integrated tasks obtained through the callback interface can be viewed in the data synchronization and whole database synchronization instances of the task monitoring module.

[0071] In this embodiment, efficient, secure and reliable integration of heterogeneous data can be achieved through the combination of data integration services and DataX tools. Through the execution of the above four steps, only based on the configured data source information, when creating a data synchronization task, the source and destination databases and other related metadata attributes are automatically obtained, breaking the original functional boundaries of data synchronization and reducing the overall cost of data synchronization and processing.

[0072] Further, see Figure 2 In the second embodiment of the synchronization of heterogeneous data of the present invention, the step S10 further includes the following steps:

[0073] Step S50: Based on the configuration mode triggered by the system interface, display the corresponding configuration page;

[0074] Step S60: Determine and configure the data source, data destination and task configuration items of the data integration task based on the library table information, SQL statements and / or custom content written in the configuration page;

[0075] Step S70: Save and publish the configured task instance of the data integration task.

[0076] In this embodiment, the task types of data integration tasks include data synchronization and whole database synchronization. The above two types correspond to different configuration modes in the system interface. Users can select the configuration mode in the system interface by themselves. According to the configuration mode triggered by the user, the corresponding configuration page is displayed for the user to write basic configuration information, which can be one or more of the three types of library table information, SQL statements and / or custom content. Through the above written content, the data source, data destination and other task configuration items of the data integration task are determined and configured. Subsequently, the task instance of the data integration task after the configuration is completed is saved and published, waiting for the next step to be executed.

[0077] Furthermore, different data integration task types have different configuration methods. The synchronization mode of the data synchronization source configuration provides two modes: normal and SQL. The normal mode automatically lists all table name contents under the corresponding library according to the database under the selected data source, and users can perform data synchronization operations without writing complex SQL statements. This mode is suitable for basic data synchronization needs. Users can directly select the tables to be synchronized from the system interface, which is simple and convenient. The SQL mode provides a higher degree of flexibility. Users can customize SQL statements on the system interface to perform data synchronization operations. This mode is mainly for relational databases with SQL language. Users can write complex SQL statements according to specific needs to achieve more refined data synchronization operations. The SQL mode is suitable for situations where there are specific requirements for the data synchronization process or complex data processing is required.

[0078] Furthermore, the data synchronization source configuration item also provides full and incremental filtering methods, which users can also configure through the system interface. Full means synchronizing all data in the source table, while incremental provides three methods: date, serial type, and custom. Among them, for date and serial type, the field content that conforms to the data type in the source table will be automatically listed, and custom supports users to fill in custom where conditions to filter SQL statements for synchronization. The incremental method is implemented by splicing the configured incremental fields before the data synchronization task is executed to form a conditional filtering SQL statement. After the single-table synchronization task is successfully executed, the latest data of the corresponding incremental field in the source table is obtained through the JDBC (Java Database ConnectivityJava, a standard interface for language access to relational databases) connection method, and the value content is recorded in the synchronization task table. When the single-table incremental synchronization task is executed again, it will be updated and synchronized according to the saved incremental value. The configuration of the source end of the whole database synchronization only requires selecting the source database information.

[0079] When configuring the data destination, the main configuration is the destination data source information. The configuration items for the destination are similar for the whole database synchronization and data synchronization. The destinations of different data sources can be personalized, and some write configurations will be dynamically adjusted according to different data sources. Among them, the data synchronization module also provides a method for quickly creating tables, that is, to perform relevant mapping type processing according to the data structure and field type of the source table, convert special types in different data sources, and generate the destination DDL (Data Definition Language, language statement of the database structure) based on this information. Users can also freely modify the above default generated DDL statements. Through the table creation statements submitted by users, the destination table can be automatically generated to achieve the function of quickly creating tables.

[0080] Optionally, the configuration content of the task configuration item may include scheduling rules, dependencies, monitoring and guarantee settings, etc. Scheduling rules are used to specify the scheduling rules of tasks, including execution time, execution frequency, retry strategy, etc., to ensure that tasks are executed on time and on demand. Dependencies are used to specify the dependencies between tasks to ensure the execution order and relevance of tasks and avoid confusion and errors during data synchronization. Monitoring and alarm settings are used to configure monitoring indicators, alarm rules, etc., to monitor the execution status of tasks in real time and to detect and handle abnormal situations in a timely manner. After that, the generated integration tasks will be loaded and parsed by the scheduling system, and the tasks will be scheduled and executed according to the configuration information therein. In this way, the automatic management and scheduling of data synchronization tasks can be achieved, and the efficiency and reliability of data synchronization can be improved.

[0081] Furthermore, to facilitate the distinction between the synchronization process of data synchronization and whole database synchronization, refer to Figure 5 , data synchronization first configures the data source connection information of the source and destination ends, that is, the first item is the basic information configuration of data synchronization source, data destination, etc., the second item is the field mapping configuration, the source and destination data type mapping and field conversion configuration, the third item is the data synchronization execution task configuration. After completing the task configuration, it is published to obtain the synchronization integration task. By running the task, the data synchronization job JSON is obtained, and a data synchronization instance is created. Finally, the data synchronization task is executed and the synchronization result callback is updated. Reference Figure 6 , the whole database synchronization data integration process needs to first configure the source and destination data source connection information, that is, the first item is the whole database synchronization source, data destination and other basic information configuration, the second item is data table selection, select multiple table name information that need to be migrated to the destination database, the third item is the whole database synchronization execution task, the task is published to obtain the whole database synchronization integration task, by executing the task, obtain the whole database synchronization job JSON, create a whole database synchronization instance, execute the whole database synchronization task, and update the whole database synchronization result callback.

[0082] In this embodiment, various data sources are integrated and a unified data access and management interface is provided through a one-stop data synchronization method, so that data can be seamlessly synchronized between different data sources. At the same time, the present invention also provides the needs of integrated synchronization of two modules, data synchronization and whole database synchronization, and processes single-table and multi-table synchronization respectively, reducing the difficulty of data synchronization and processing.

[0083] Reference Figure 3 The third embodiment of the method for synchronizing heterogeneous data of the present invention further includes the following steps before step S70:

[0084] Step S80: If the data integration task is a data synchronization task, the table structure and field type between the data source end and the data destination end are visually mapped and configured.

[0085] Step S90: If the data integration task is a whole database synchronization task, when there is a conflict in the table name at the data destination end, use the existing table or delete the table to verify the table name of the newly created table, collect the data structure and attribute type in the metadata content according to the data source end table, and perform type mapping on the heterogeneous data destination end.

[0086] In this embodiment, different configuration methods are used based on different task types of data integration tasks. If the data integration task is a data synchronization task, it is necessary to visually and dynamically map the table structure and field type between the data source end and the data destination end. Users can configure the source end data to be mapped to the corresponding columns of the destination end according to their needs. Optionally, in a feasible implementation, in order to improve the efficiency of data synchronization, the destination end of the Hive (a data warehouse name) database adopts the method of writing to HDFS (a distributed file system called Hadoop) for data synchronization, that is, the mapped fields are rearranged according to the order of the fields in the Hive table, and the unmapped fields are filled according to the null value identifier in the metadata of the Hive table, thereby realizing the synchronization of some Hive column data through HDFS. This approach can prevent users from selecting only some columns or mapping the column information in the Hive table in disorder without processing, resulting in confusion in the synchronized data. In addition, the data synchronization field mapping configuration also provides data cleaning and preprocessing methods for field conversion. Fields can choose from a variety of field conversion methods, such as desensitization conversion, that is, for some sensitive data such as mobile phone number information, select the corresponding desensitization rules for synchronization processing. Data conversion can eliminate inaccurate, redundant or invalid data, standardize and format the data to ensure data quality and consistency.

[0087] Furthermore, if the data integration task is a whole-database synchronization task, the second step of whole-database synchronization is data table selection configuration. Users can select multiple table information in the source database according to their own needs, and splice the prefix and suffix configuration of the custom table name. When executing synchronization, it will be processed according to the previous step configuration. In the case of conflicting table names on the destination side of heterogeneous data, the table name processing and verification of the newly created table is performed using an existing table or a deleted table. That is, in the whole-database multi-table synchronization scenario, for data tables that do not exist in the destination database, the heterogeneous data destination side will be type mapped according to the data structure and attribute type information obtained from the metadata content of the source side table collection, so as to obtain the destination table data structure and the destination side table creation DDL statement, and generate a SQL statement in the table creation statement whether to delete the table according to the table name conflict configuration content. If the synchronization is configured to use an existing table, no new table will be generated during the execution process. Therefore, during the whole-database synchronization execution process, it will first determine whether a new table needs to be created, and then perform data synchronization.

[0088] In this embodiment, for the configuration process of the two integration tasks of data synchronization and whole database synchronization, a visual dynamic mapping configuration is adopted, so that users can intuitively configure the mapping of source data to the corresponding columns of the destination according to their own needs, without having to deeply understand the underlying data structure and field types. In addition, both data synchronization and whole database synchronization provide a fast table creation function. When performing data synchronization, you can first create a destination table in the destination database according to the provided table creation statement; when performing whole database synchronization, if the destination target table does not exist in the heterogeneous destination database, the destination table will be automatically created. By integrating automation and batch processing, manual operations and workload are reduced, and data synchronization efficiency is improved.

[0089] Reference Figure 7 , Figure 7 The present invention is a schematic diagram of a structure of a synchronization device for heterogeneous data in a hardware operating environment according to an embodiment of the present invention.

[0090] like Figure 7As shown, the heterogeneous data synchronization device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (WIreless-FIdelity, WI-FI) interface). The memory 1005 may be a high-speed random access memory (Random Access Memory, RAM) memory, or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0091] Those skilled in the art will understand that Figure 7 The structure shown in the figure does not constitute a limitation on the synchronization device for heterogeneous data, and may include more or less components than those shown in the figure, or combine certain components, or arrange the components differently.

[0092] like Figure 7 As shown, the memory 1005 as a storage medium may include an operating system, a data storage module, a network communication module, a user interface module, and a synchronization program for heterogeneous data.

[0093] exist Figure 7 In the heterogeneous data synchronization device shown, the network interface 1004 is mainly used for data communication with other devices; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the heterogeneous data synchronization device of the present invention can be set in the heterogeneous data synchronization device, and the heterogeneous data synchronization device calls the heterogeneous data synchronization program stored in the memory 1005 through the processor 1001, and performs the following steps:

[0094] Based on the configuration information received by the system interface, configure the data source, data destination and task configuration items of the data integration task, and publish the data integration task;

[0095] Generate an integrated synchronization job and a Json template according to the task instance of the data integration task, the data source and the data destination;

[0096] When the task scheduling process is triggered, a synchronization job Json is generated according to the Json template and the integrated synchronization job;

[0097] The synchronization service component is triggered based on the preset command to perform the data synchronization task according to the synchronization job Json.

[0098] Furthermore, the heterogeneous data synchronization device calls the heterogeneous data synchronization program stored in the memory 1005 through the processor 1001, and further performs the following steps:

[0099] Based on the configuration mode triggered by the system interface, display the corresponding configuration page;

[0100] Based on the library table information, SQL statements and / or custom content written in the configuration page, determine and configure the data source, data destination and task configuration items of the data integration task;

[0101] The configured task instance of the data integration task is saved and published.

[0102] Furthermore, the heterogeneous data synchronization device calls the heterogeneous data synchronization program stored in the memory 1005 through the processor 1001, and further performs the following steps:

[0103] If the data integration task is a data synchronization task, the table structure and field type between the data source end and the data destination end are visually mapped and configured;

[0104] If the data integration task is a whole database synchronization task, when there is a table name conflict at the data destination end, use the existing table or delete the table to verify the table name of the newly created table, collect the data structure and attribute type in the metadata content according to the data source end table, and perform type mapping on the heterogeneous data destination end.

[0105] Furthermore, the heterogeneous data synchronization device calls the heterogeneous data synchronization program stored in the memory 1005 through the processor 1001, and further performs the following steps:

[0106] According to the task instance, the data source and the data destination, configure the data synchronization strategy, scheduling rules, dependencies between tasks and exception handling strategy;

[0107] Generate the integrated synchronization job based on the data synchronization strategy, the scheduling rules, the dependency relationship and the exception handling strategy;

[0108] Based on the integrated synchronization job, the Json template is created.

[0109] Furthermore, the heterogeneous data synchronization device calls the heterogeneous data synchronization program stored in the memory 1005 through the processor 1001, and further performs the following steps:

[0110] Based on the Json template, replace and fill in the connection configuration information of the data source and data destination;

[0111] If the synchronization mode of the integrated synchronization job is incremental, fill in the filter condition statement based on the Json template according to the incremental field and incremental value;

[0112] Based on the integrated synchronization job, replace the placeholder of the Json template;

[0113] The synchronization job Json is generated based on the filled content of the Json template.

[0114] Furthermore, the heterogeneous data synchronization device calls the heterogeneous data synchronization program stored in the memory 1005 through the processor 1001, and further performs the following steps:

[0115] Generate corresponding read port instance and write port instance based on the configuration information of the synchronization job Json;

[0116] Based on the read port instance, read data from the data source, and perform data extraction and cleaning on the data;

[0117] Based on the write port instance, the read data is written to the data destination end to perform the data synchronization task.

[0118] Furthermore, the heterogeneous data synchronization device calls the heterogeneous data synchronization program stored in the memory 1005 through the processor 1001, and further performs the following steps:

[0119] Based on the callback function and the preset communication protocol, sending a result request of the task instance to the data integration service;

[0120] The data integration service updates the task execution status and synchronization information based on the result request;

[0121] The task execution status and the synchronization information are displayed based on the system interface.

[0122] Furthermore, the heterogeneous data synchronization device calls the heterogeneous data synchronization program stored in the memory 1005 through the processor 1001, and further performs the following steps:

[0123] If the integration service is an incremental task of data synchronization, the data value corresponding to the incremental field in the source end table is obtained based on the callback function;

[0124] Recording the data value into an integrated task table;

[0125] When executing the next incremental synchronization task, the incremental synchronization task is executed based on the data value in the integrated task table.

[0126] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0127] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0128] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0129] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for synchronizing heterogeneous data, characterized in that: The method for synchronizing heterogeneous data comprises the following steps: Based on the configuration information received by the system interface, configure the data source, data destination and task configuration items of the data integration task, and publish the data integration task; Generate an integrated synchronization job and a Json template according to the task instance of the data integration task, the data source and the data destination; The step of generating an integrated synchronization job and a Json template according to the task instance of the data integration task, the data source and the data destination includes: According to the task instance, the data source and the data destination, configure the data synchronization strategy, scheduling rules, dependencies between tasks and exception handling strategy, wherein the scheduling rules include execution time, execution frequency and retry strategy to ensure that tasks are executed on time and on demand, and the dependencies are used to ensure the execution order and relevance of tasks to avoid confusion and errors in the data synchronization process, wherein the task instance includes the type of data source, data format, and data structure and field mapping relationship; Generate the integrated synchronization job based on the data synchronization strategy, the scheduling rules, the dependency relationship and the exception handling strategy; Based on the integrated synchronization job, create the Json template; When the task scheduling process is triggered, a synchronization job Json is generated according to the Json template and the integrated synchronization job; Trigger the synchronization service component based on the preset command to execute the data synchronization task according to the synchronization job Json; Wherein, when the task scheduling process is triggered, the step of generating the synchronization job Json according to the Json template and the integrated synchronization job includes: Based on the Json template, replace and fill in the connection configuration information of the data source and data destination; If the synchronization mode of the integrated synchronization job is incremental, fill in the filter condition statement based on the Json template according to the incremental field and incremental value; Based on the integrated synchronization job, replace the placeholder of the Json template; The synchronization job Json is generated based on the filled content of the Json template.

2. The method for synchronizing heterogeneous data according to claim 1, characterized in that: The step of configuring the data source, data destination and task configuration items of the data integration task based on the configuration information received by the system interface, and publishing the data integration task includes: Based on the configuration mode triggered by the system interface, display the corresponding configuration page; Based on the library table information, SQL statements and / or custom content written in the configuration page, determine and configure the data source, data destination and task configuration items of the data integration task; The configured task instance of the data integration task is saved and published.

3. The method for synchronizing heterogeneous data according to claim 2, characterized in that: Before the step of saving and publishing the task instance of the configured data integration task, the method further includes: If the data integration task is a data synchronization task, the table structure and field type between the data source end and the data destination end are visually mapped and configured; If the data integration task is a whole database synchronization task, when there is a table name conflict at the data destination end, use the existing table or delete the table to verify the table name of the newly created table, collect the data structure and attribute type in the metadata content according to the data source end table, and perform type mapping on the heterogeneous data destination end.

4. The method for synchronizing heterogeneous data according to claim 1, characterized in that: The step of triggering the synchronization service component to execute the data synchronization task according to the synchronization job Json based on the preset command includes: Generate corresponding read port instance and write port instance based on the configuration information of the synchronization job Json; Based on the read port instance, read data from the data source, and perform data extraction and cleaning on the data; Based on the write port instance, the read data is written to the data destination end to perform the data synchronization task.

5. The method for synchronizing heterogeneous data according to claim 1, characterized in that: After the step of triggering the synchronization service component to execute the data synchronization task according to the synchronization job Json based on the preset command, the method further includes: Based on the callback function and the preset communication protocol, sending a result request of the task instance to the data integration service; The data integration service updates the task execution status and synchronization information based on the result request; The task execution status and the synchronization information are displayed based on the system interface.

6. The method for synchronizing heterogeneous data according to claim 5, characterized in that: After the step of sending the result request of the task instance to the data integration service based on the callback function and the preset communication protocol, the method further includes: If the integration service is an incremental task of data synchronization, the data value corresponding to the incremental field in the source end table is obtained based on the callback function; Recording the data value into an integrated task table; When executing the next incremental synchronization task, the incremental synchronization task is executed based on the data value in the integrated task table.

7. A synchronization device for heterogeneous data, characterized in that: The heterogeneous data synchronization device includes: a memory, a processor, and a heterogeneous data synchronization program stored in the memory and executable on the processor, wherein the heterogeneous data synchronization program is configured to implement the steps of the heterogeneous data synchronization method as described in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a synchronization program for heterogeneous data, and when the synchronization program for heterogeneous data is executed by a processor, the steps of the synchronization method for heterogeneous data according to any one of claims 1 to 6 are implemented.