Synchronization method and device of heterogeneous data source, equipment and storage medium

Abstract

The invention provides a heterogeneous data source synchronization method and apparatus, a device and a storage medium. The method comprises the steps of obtaining database metadata information of a source end and a target end; determining a synchronization strategy according to the database metadata information; preprocessing a database of a target end according to the synchronization strategy to obtain a synchronization configuration file; and executing a data synchronization task according to the synchronization configuration file so as to synchronize data of the source end and the target end. By acquiring the metadata information of the databases at the source end and the target end, the structure and the characteristic of the databases at the two ends can be accurately understood, so that the optimal synchronization strategy is formulated. According to the mode, the data synchronization efficiency and accuracy can be effectively improved, and data redundancy and synchronization conflicts are reduced.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a synchronization method, device, equipment and storage medium for heterogeneous data sources. Background Art

[0002] As data-driven decision-making becomes increasingly important, the demand for data synchronization scenarios is also increasing. Especially in the context of enterprise digital transformation, the popularization of cloud computing, and the widespread application of big data analysis, data synchronization has become a key link in ensuring data consistency, improving business efficiency, and supporting real-time decision-making.

[0003] In the prior art, DataX, as an open source data synchronization tool developed by Alibaba, is widely used in various data synchronization scenarios due to its high efficiency, stability, ease of use, and scalability. However, with the complexity and diversification of enterprise needs, the existing synchronization solutions still have some shortcomings when dealing with heterogeneous data source environments. For example, when the data source changes or the table structure is modified, it is often necessary to manually update each JSON configuration file, which increases the manpower maintenance cost. In addition, the database connection information saved in the JSON file is at risk of being stolen, and the configuration file may fail to synchronize in a multi-server environment, resulting in data synchronization task failure.

[0004] In summary, the problems existing in the prior art need to be solved urgently. Summary of the invention

[0005] The present invention provides a synchronization method, device, equipment and storage medium for heterogeneous data sources, which are used to solve the defects in the prior art and improve the reliability of data synchronization.

[0006] The present invention provides a method for synchronizing heterogeneous data sources, comprising:

[0007] Obtain database metadata information on the source and target ends;

[0008] Determine a synchronization strategy according to the database metadata information;

[0009] Preprocessing the database on the target end according to the synchronization strategy to obtain a synchronization configuration file;

[0010] According to the synchronization configuration file, a data synchronization task is executed to synchronize the data of the source end and the target end.

[0011] According to a synchronization method for heterogeneous data sources provided by the present invention, the step of preprocessing the database of the target end according to the synchronization strategy to obtain a synchronization configuration file specifically includes:

[0012] Preprocessing the database on the target end according to the synchronization strategy;

[0013] Integrate the preprocessing results into the configuration template file to obtain the synchronization configuration file;

[0014] The configuration template file is constructed according to the acquired database metadata information.

[0015] According to a synchronization method for heterogeneous data sources provided by the present invention, the step of executing a data synchronization task according to the synchronization configuration file to synchronize data between the source end and the target end specifically includes:

[0016] Integrate the database connection information into the synchronization configuration file to obtain a target synchronization file;

[0017] According to the target synchronization file, a data synchronization task is executed to synchronize the data of the source end and the target end;

[0018] The database connection information includes a source database address and a target database address.

[0019] According to a synchronization method for heterogeneous data sources provided by the present invention, the step of executing a data synchronization task according to the target synchronization file to synchronize data between the source end and the target end specifically includes:

[0020] According to the resource usage of the server, the data synchronization task is assigned to the target server for execution.

[0021] According to a method for synchronizing heterogeneous data sources provided by the present invention, after the step of executing a data synchronization task according to the target synchronization file to synchronize data between the source end and the target end, the method further includes:

[0022] Obtaining the execution log of the data synchronization task, wherein the execution log includes the task execution status, collection amount and collection time;

[0023] The execution log is stored in a task execution database, and the task execution database is used to save the execution log of the data processing task.

[0024] According to a synchronization method for heterogeneous data sources provided by the present invention, the database metadata information includes data table structure, field type, field length and index information.

[0025] According to a synchronization method for heterogeneous data sources provided by the present invention, the synchronization strategy includes: full collection, incremental collection and receipt table collection.

[0026] The present invention also provides a synchronization device for heterogeneous data sources, comprising:

[0027] An information acquisition unit, used to acquire database metadata information of the source end and the target end;

[0028] A synchronization strategy unit, used to determine a synchronization strategy according to the database metadata information;

[0029] A configuration file unit, used to pre-process the database of the target end according to the synchronization strategy to obtain a synchronization configuration file;

[0030] The task execution unit executes the data synchronization task according to the synchronization configuration file to synchronize the data of the source end and the target end.

[0031] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a method for synchronizing heterogeneous data sources as described above is implemented.

[0032] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method for synchronizing heterogeneous data sources as described in any one of the above is implemented.

[0033] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method for synchronizing heterogeneous data sources as described in any one of the above is implemented.

[0034] The synchronization method, apparatus, device and storage medium of heterogeneous data sources provided by the present invention obtain database metadata information of the source and target ends; determine the synchronization strategy based on the database metadata information; pre-process the database of the target end according to the synchronization strategy to obtain a synchronization configuration file; and execute the data synchronization task according to the synchronization configuration file to synchronize the data of the source and target ends. By obtaining the database metadata information of the source and target ends, the present invention can accurately understand the structure and characteristics of the databases at both ends, thereby formulating the optimal synchronization strategy. This method can effectively improve the efficiency and accuracy of data synchronization and reduce data redundancy and synchronization conflicts. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0036] Figure 1 It is a flowchart of the synchronization method of heterogeneous data sources provided by the present invention;

[0037] Figure 2 It is a structural schematic diagram of a synchronization device for heterogeneous data sources provided by the present invention;

[0038] Figure 3 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] In order to solve the problems in the prior art, the present invention proposes a synchronization method for heterogeneous data sources to improve the reliability of data synchronization. The synchronization method for heterogeneous data sources is described below. Figure 1 As shown, including but not limited to the following steps:

[0041] Step 110: Obtain database metadata information of the source and target ends.

[0042] In step 110, first, metadata information is obtained from the databases of the source and target ends. Database metadata information includes, but is not limited to, data table structure, field type, field length, and index information. By obtaining this information, the current status of the source and target databases can be fully understood, providing a basis for subsequent synchronization strategy formulation and data processing.

[0043] Step 120: Determine a synchronization strategy based on the database metadata information.

[0044] In step 120, the synchronization strategy applicable to the data synchronization is determined by analyzing the obtained database metadata information. The selection of synchronization strategy may include full collection, incremental collection, and receipt table collection. Then, the optimal synchronization strategy is automatically selected based on factors such as differences in data table structure, changes in data volume, and index status.

[0045] Step 130: pre-process the database on the target end according to the synchronization strategy to obtain a synchronization configuration file.

[0046] Step 130, according to the selected synchronization strategy, first preprocess the target database. Preprocessing operations include but are not limited to adjusting field types, creating or deleting indexes, and changing table structures. After the preprocessing is completed, the preprocessing results are integrated into a configuration template file to generate a synchronization configuration file. The configuration template file is constructed based on the acquired database metadata information, and after preprocessing, it can reflect the latest status of the target database.

[0047] Step 140: Execute a data synchronization task according to the synchronization configuration file to synchronize data between the source end and the target end.

[0048] After the synchronization configuration file is generated, the database connection information (including the source database address and the target database address) is integrated into the synchronization configuration file to form a target synchronization file. Subsequently, the data synchronization task is executed according to the target synchronization file. During the execution process, the system will reasonably allocate the data synchronization task to the target server for execution based on the resource usage of the server to ensure the efficient completion of the synchronization task.

[0049] As an optional embodiment, the step of preprocessing the database of the target end according to the synchronization strategy to obtain the synchronization configuration file specifically includes:

[0050] Preprocessing the database on the target end according to the synchronization strategy;

[0051] Integrate the preprocessing results into the configuration template file to obtain the synchronization configuration file;

[0052] The configuration template file is constructed according to the acquired database metadata information.

[0053] As an optional embodiment, the synchronization strategy includes: full collection, incremental collection and receipt table collection.

[0054] In this embodiment, first, the database metadata information of the source end and the target end is obtained, and the information includes but is not limited to the data table structure, field type, field length, index information, etc.

[0055] Based on the metadata information obtained and the actual business needs, the system determines the appropriate data synchronization strategy. The synchronization strategy can include full data collection, incremental data collection, and receipt table collection.

[0056] Full data collection: The system clears the relevant table data in the target database (for example, through truncate or delete operations) to ensure complete data synchronization.

[0057] Incremental collection: The system will selectively update some data in the target database based on the latest data changes between the target database and the source database, thereby improving the efficiency of data synchronization.

[0058] Receipt table collection: The system collects and updates certain specific data segments in the target database based on the information recorded in the receipt table.

[0059] According to the above synchronization strategy, the target database is preprocessed to ensure that there is no redundancy or conflict in the data during the synchronization process.

[0060] After the preprocessing operation is completed, the preprocessing results (such as table structure, field mapping relationship, data screening conditions, etc.) are integrated into a configuration template file.

[0061] The configuration template file is pre-built based on the acquired database metadata information. The template file contains the basic configuration information required for the synchronization task and provides extensible fields and parameters for different synchronization strategies.

[0062] The final synchronization configuration file is generated by filling the specific results of the preprocessing into the corresponding part of the configuration template file. The synchronization configuration file is the basis for the execution of subsequent data synchronization tasks and can effectively guide the accurate transmission and processing of data from the source end to the target end.

[0063] The generated synchronization configuration file is used to guide the execution of data synchronization tasks. When executing synchronization tasks, the data on the source and target ends will be synchronized according to the parameters such as database connection information, data mapping relationship, data filtering conditions, etc. defined in the synchronization configuration file.

[0064] Through this process, the data consistency and integrity between the source and target ends can be ensured, while meeting the data synchronization requirements in different business scenarios.

[0065] Through the above steps, the present invention realizes intelligent preprocessing of the target database based on the synchronization strategy, generates a synchronization configuration file suitable for heterogeneous data source synchronization tasks, thereby greatly improving the efficiency and flexibility of data synchronization.

[0066] As an optional embodiment, the step of executing the data synchronization task according to the synchronization configuration file to synchronize the data of the source end and the target end specifically includes:

[0067] Integrate the database connection information into the synchronization configuration file to obtain a target synchronization file;

[0068] According to the target synchronization file, a data synchronization task is executed to synchronize the data of the source end and the target end;

[0069] The database connection information includes a source database address and a target database address.

[0070] First, the pre-processed target database information is read from the synchronization configuration file, and the data synchronization task is prepared to be executed according to the synchronization strategy.

[0071] Before executing the task, you need to integrate the database connection information (including the source database address and the target database address) into the synchronization configuration file. The database connection information is the key to ensure that data can be transmitted correctly, and it usually includes the following:

[0072] Source database address: used to identify the specific location of the database server where the source data is located (such as the IP address or domain name), as well as the corresponding database port number, database name, and authentication information (such as user name and password).

[0073] Target database address: used to identify the specific location of the target database for data synchronization (such as IP address or domain name), as well as the corresponding database port number, database name, and authentication information (such as user name and password).

[0074] The above database connection information is integrated with the preprocessed synchronization configuration file to generate a target synchronization file. This file contains not only the rules and parameters for synchronizing data, but also the database connection information required to perform data synchronization.

[0075] After obtaining the target synchronization file, first verify the validity of the database connection information to ensure that you can successfully connect to the source and target databases.

[0076] Execute the data synchronization task according to the data synchronization rules defined in the target synchronization file. This task can be performed according to different synchronization strategies, including:

[0077] Full synchronization: The system completely copies all the data in the source database after cleaning the target database to ensure that the target data is completely consistent with the source data.

[0078] Incremental synchronization: The system synchronizes only the newly added or updated data based on the data changes between the source and target databases to reduce the time and system resource consumption required for synchronization.

[0079] Receipt table synchronization: The system synchronizes only the data that needs to be updated based on the records in the receipt table, ensuring the accuracy and efficiency of the synchronization operation.

[0080] During the synchronization process, the execution status of the task will be monitored to ensure the integrity and consistency of the data during transmission.

[0081] After the data synchronization task is completed, the system will record the execution log of the synchronization task, including the execution status of the task, the amount of data synchronized, the execution time and other information, to facilitate subsequent monitoring and analysis.

[0082] These execution logs can be stored in the task execution database to support subsequent task scheduling and optimization.

[0083] As an optional embodiment, the step of executing the data synchronization task according to the target synchronization file to synchronize the data of the source end and the target end specifically includes:

[0084] According to the resource usage of the server, the data synchronization task is assigned to the target server for execution.

[0085] First, the server resource usage is evaluated in real time to ensure that the synchronization task can be executed efficiently in a suitable environment. This evaluation includes the following:

[0086] CPU usage: Monitor the CPU usage of the server to ensure that there are sufficient CPU resources when executing data synchronization tasks to avoid task delays or failures due to excessive usage.

[0087] Memory usage: Check the server memory usage to ensure that there is enough memory space for data processing and transmission to prevent task interruptions due to insufficient memory.

[0088] Network bandwidth: Evaluate the network bandwidth of the server, especially when the source and target databases are located in different geographical locations, to ensure the stability and efficiency of the data transfer process.

[0089] Disk I / O: Check the server's disk I / O performance to ensure that disk read and write operations do not become a bottleneck during data synchronization.

[0090] After completing the resource usage assessment, the data synchronization task is assigned to the appropriate target server for execution based on the current resource usage.

[0091] If the server resources are insufficient to execute the entire data synchronization task, you can use the following strategies to allocate tasks:

[0092] Task division: divide the data synchronization task into multiple small tasks and assign them to different target servers for parallel execution to make full use of server resources.

[0093] Task queuing: If resources are temporarily insufficient, the data synchronization task will be added to the task queue and executed after resources are released to ensure the order and reliability of task execution.

[0094] Dynamic scheduling: According to the dynamic changes of server resources, the task execution strategy is adjusted in real time to avoid task failure due to resource overload.

[0095] When a data synchronization task is assigned to the target server, the system will start executing the data synchronization task according to the synchronization rules in the target synchronization file.

[0096] During the synchronization process, the resource usage of the target server will be continuously monitored, and the execution progress and strategy of the synchronization task will be dynamically adjusted according to resource changes to ensure that the synchronization task can be completed smoothly.

[0097] If an exception occurs during the execution of a synchronization task (such as sudden exhaustion of server resources or network interruption), fault tolerance processing or retries will be automatically performed to ensure the integrity and consistency of data synchronization.

[0098] After the data synchronization task is completed, the occupied server resources will be automatically released and restored to an idle state to facilitate the execution of subsequent tasks.

[0099] At the same time, detailed logs of task execution will be recorded, including task execution time, resource usage, data synchronization volume, etc., for subsequent analysis and optimization.

[0100] As an optional embodiment, after the step of executing a data synchronization task according to the target synchronization file to synchronize data between the source end and the target end, the method further includes:

[0101] Obtaining the execution log of the data synchronization task, wherein the execution log includes the task execution status, collection amount and collection time;

[0102] The execution log is stored in a task execution database, and the task execution database is used to save the execution log of the data processing task.

[0103] In this embodiment, after the data synchronization task is completed, the execution log of the task is automatically generated and obtained. The execution log is a detailed record of the entire data synchronization process, including the following key information:

[0104] Task execution status: records the final status of the data synchronization task, such as success, failure, or partial success. If the task fails, the reason for the failure is also recorded for subsequent analysis and troubleshooting.

[0105] Collection volume: records the amount of data transferred from the source database to the target database during data synchronization. This information can help evaluate the workload and efficiency of the task.

[0106] Collection time: Record the start and end time of the entire data synchronization task to calculate the total time of the task. By analyzing the collection time, you can optimize future data synchronization tasks.

[0107] The execution logs of the acquired data synchronization tasks are stored in a dedicated task execution database. The task execution database is a database used to save the execution status of all data processing tasks, ensuring the persistent storage of log data for future query and analysis.

[0108] The design of the database includes but is not limited to the following structures:

[0109] Task ID: Uniquely identifies each data synchronization task to ensure the accuracy and traceability of log records.

[0110] Timestamp: records the time when the log is generated to facilitate tracking and analysis of the time sequence of task execution.

[0111] Task details: including detailed information such as task execution status, collection volume and collection time, so as to fully understand the execution status of the task.

[0112] Exception records: If any abnormal situation or error occurs during the task execution, the exception information will be recorded in the log for subsequent investigation and improvement.

[0113] The execution logs stored in the task execution database can be managed and queried through the query interface provided by the system. Users can retrieve the execution logs of specific tasks based on conditions such as task ID, execution status, and collection time.

[0114] By analyzing the execution logs, users can discover potential problems in data synchronization tasks and further optimize synchronization strategies and system performance.

[0115] As an optional embodiment, the database metadata information includes data table structure, field type, field length and index information. In an optional embodiment of the present invention, the database metadata information includes but is not limited to the following types of information:

[0116] Data table structure:

[0117] The data table structure refers to the organization and relationship of the tables in the database, including the table name, the relationship between tables (such as foreign keys), the hierarchical structure of the tables, etc. This information is used to determine the tables that need to be synchronized and their dependencies during data synchronization, thereby ensuring data consistency and integrity.

[0118] Field Type:

[0119] Field type refers to the data type of each field (column), such as integer (INTEGER), floating point (FLOAT), character (VARCHAR), date (DATE), etc. During data synchronization, different types of fields may require different processing methods, especially when synchronizing between heterogeneous data sources. Differences in field types may cause data conversion problems, so understanding field types is crucial to ensure synchronization accuracy.

[0120] Field Length:

[0121] Field length refers to the maximum length of a character field or the precision of a numeric field. During data synchronization, field length information is used to ensure that the field in the target database can accommodate the data in the source database, avoiding data truncation or overflow due to insufficient field length.

[0122] Index information:

[0123] Index information includes the index settings of each field in the table, such as primary key index, unique index, common index, etc. Indexes are used to speed up query operations in the database. Understanding index information helps optimize the efficiency of data writing during data synchronization and ensures that the synchronized database has query performance similar to that of the source database.

[0124] The following describes the synchronization device for heterogeneous data sources provided by the present invention. Figure 2 As shown, the synchronization device for heterogeneous data sources described below and the synchronization method for heterogeneous data sources described above can refer to each other.

[0125] A synchronization device for heterogeneous data sources, comprising:

[0126] The information acquisition unit 210 is used to acquire the database metadata information of the source end and the target end;

[0127] A synchronization strategy unit 220, configured to determine a synchronization strategy according to the database metadata information;

[0128] A configuration file unit 230, configured to pre-process the database of the target end according to the synchronization strategy to obtain a synchronization configuration file;

[0129] The task execution unit 240 executes a data synchronization task according to the synchronization configuration file to synchronize the data of the source end and the target end.

[0130] Figure 3 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 3As shown, the electronic device may include: a processor 310, a communication interface 320, a memory 330 and a communication bus 340, wherein the processor 310, the communication interface 320 and the memory 330 communicate with each other through the communication bus 340. The processor 310 may call the logic instructions in the memory 330 to execute the synchronization method of the heterogeneous data sources, and the method includes:

[0131] Obtain database metadata information on the source and target ends;

[0132] Determine a synchronization strategy according to the database metadata information;

[0133] Preprocessing the database on the target end according to the synchronization strategy to obtain a synchronization configuration file;

[0134] According to the synchronization configuration file, a data synchronization task is executed to synchronize the data of the source end and the target end.

[0135] In addition, the logic instructions in the above-mentioned memory 330 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM, Read-On l yMemory), a random access memory (RAM, Random Access Memory), a disk or an optical disk.

[0136] On the other hand, the present invention further provides a computer program product, the computer program product includes a computer program, the computer program can be stored in a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the synchronization method of heterogeneous data sources provided by the above methods, the method includes:

[0137] Obtain database metadata information on the source and target ends;

[0138] Determine a synchronization strategy according to the database metadata information;

[0139] Preprocessing the database on the target end according to the synchronization strategy to obtain a synchronization configuration file;

[0140] According to the synchronization configuration file, a data synchronization task is executed to synchronize the data of the source end and the target end.

[0141] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for synchronizing heterogeneous data sources provided by the above methods is implemented, and the method includes:

[0142] Obtain database metadata information on the source and target ends;

[0143] Determine a synchronization strategy according to the database metadata information;

[0144] Preprocessing the database on the target end according to the synchronization strategy to obtain a synchronization configuration file;

[0145] According to the synchronization configuration file, a data synchronization task is executed to synchronize the data of the source end and the target end.

[0146] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0147] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0148] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for synchronizing heterogeneous data sources, characterized in that: include: Obtain database metadata information on the source and target ends; Determine a synchronization strategy according to the database metadata information; Preprocessing the database on the target end according to the synchronization strategy to obtain a synchronization configuration file; According to the synchronization configuration file, a data synchronization task is executed to synchronize the data of the source end and the target end.

2. The method for synchronizing heterogeneous data sources according to claim 1, characterized in that: The step of preprocessing the database of the target end according to the synchronization strategy to obtain the synchronization configuration file specifically includes: Preprocessing the database on the target end according to the synchronization strategy; Integrate the preprocessing results into the configuration template file to obtain the synchronization configuration file; The configuration template file is constructed according to the acquired database metadata information.

3. The method for synchronizing heterogeneous data sources according to claim 1, characterized in that: The step of executing the data synchronization task according to the synchronization configuration file to synchronize the data of the source end and the target end specifically includes: Integrate the database connection information into the synchronization configuration file to obtain a target synchronization file; According to the target synchronization file, a data synchronization task is executed to synchronize the data of the source end and the target end; The database connection information includes a source database address and a target database address.

4. The method for synchronizing heterogeneous data sources according to claim 3, characterized in that: The step of executing the data synchronization task according to the target synchronization file to synchronize the data of the source end and the target end specifically includes: According to the resource usage of the server, the data synchronization task is assigned to the target server for execution.

5. The method for synchronizing heterogeneous data sources according to claim 4, characterized in that: After the step of executing a data synchronization task according to the target synchronization file to synchronize data between the source end and the target end, the method further includes: Obtaining the execution log of the data synchronization task, wherein the execution log includes the task execution status, collection amount and collection time; The execution log is stored in a task execution database, and the task execution database is used to save the execution log of the data processing task.

6. The method for synchronizing heterogeneous data sources according to claim 1, characterized in that: The database metadata information includes data table structure, field type, field length and index information.

7. The method for synchronizing heterogeneous data sources according to claim 1, characterized in that: The synchronization strategies include: full collection, incremental collection and receipt table collection.

8. A synchronization device for heterogeneous data sources, characterized in that: include: An information acquisition unit, used to acquire database metadata information of the source end and the target end; A synchronization strategy unit, used to determine a synchronization strategy according to the database metadata information; A configuration file unit, used to pre-process the database of the target end according to the synchronization strategy to obtain a synchronization configuration file; The task execution unit executes the data synchronization task according to the synchronization configuration file to synchronize the data of the source end and the target end.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method for synchronizing heterogeneous data sources as described in any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for synchronizing heterogeneous data sources as claimed in any one of claims 1 to 7 is implemented.

Images