Data synchronization method and device of heterogeneous data source, equipment, medium and product

Abstract

The invention relates to a data synchronization method and device for heterogeneous data sources, equipment, a medium and a product. The data synchronization method of the heterogeneous data source comprises the steps that a source database and a target database under a target synchronization task are determined, to-be-synchronized data synchronized between the source database and the target database is obtained, and the source database and the target database are heterogeneous databases; exporting the to-be-synchronized data from the source database according to export configuration information of the source database; and after the to-be-synchronized data is exported, importing the to-be-synchronized data into the target database according to the import configuration information of the target database. By adopting the method, the synchronization effect of data synchronization of the heterogeneous data source is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, apparatus, computer equipment, computer-readable storage medium, and computer program product for synchronizing data from heterogeneous data sources. Background Art

[0002] With the development of informatization, the amount of business data of enterprises has increased dramatically, and they are faced with the need to use a variety of different database systems. Therefore, how to synchronize heterogeneous data from different database systems is a crucial part of the enterprise information system integration process.

[0003] At present, data synchronization is usually achieved by using data synchronization tools, such as Apache Sqoop and Apache Flume, that is, data export and import are completed through a common interface. However, due to the different data structures of data from different source databases, the compatibility steps in the data synchronization process are cumbersome, which makes it easy for data synchronization of heterogeneous data sources to take too long or the synchronization performance to degrade. Therefore, the current synchronization effect of data synchronization of heterogeneous data sources is poor. Summary of the invention

[0004] Based on this, it is necessary to provide a data synchronization method, apparatus, computer device, computer-readable storage medium and computer program product for heterogeneous data sources to improve the synchronization effect of data synchronization of heterogeneous data sources in response to the above technical problems.

[0005] In a first aspect, the present application provides a data synchronization method for heterogeneous data sources, comprising:

[0006] Determine a source database and a target database under a target synchronization task, and obtain data to be synchronized between the source database and the target database, wherein the source database and the target database are heterogeneous databases;

[0007] Exporting the data to be synchronized in the source database according to the export configuration information of the source database;

[0008] After the export of the data to be synchronized is completed, the data to be synchronized is imported into the target database according to the import configuration information of the target database.

[0009] In one embodiment, exporting the to-be-synchronized data in the source database according to the export configuration information of the source database includes:

[0010] Extracting export configuration rules and export configuration time from the export configuration information;

[0011] According to the export configuration rule, the data to be synchronized is split into a plurality of sub-data to be synchronized;

[0012] The plurality of sub-data to be synchronized are exported in the source database according to the export configuration time.

[0013] In one embodiment, the target database includes a plurality of target sub-databases; and importing the to-be-synchronized data into the target database according to the import configuration information of the target database includes:

[0014] Extracting the data conversion formats corresponding to the multiple target sub-databases respectively from the import configuration information;

[0015] Converting the data to be synchronized into target heterogeneous data in each of the data conversion formats;

[0016] Import the target heterogeneous data in different data conversion formats into the corresponding target sub-database.

[0017] In one embodiment, the step of importing target heterogeneous data in different data conversion formats into corresponding target sub-databases includes:

[0018] Acquire a first synchronization performance characteristic of the source database and a second synchronization performance characteristic of the plurality of target sub-databases;

[0019] Generating a synchronization performance indicator of the target synchronization task during execution according to the first synchronization performance characteristic and the second synchronization performance characteristic;

[0020] According to the synchronization performance index, the target heterogeneous data in different data conversion formats are imported into the corresponding target sub-database.

[0021] In one embodiment, importing the target heterogeneous data in different data conversion formats into the corresponding target sub-database according to the synchronization performance indicator includes:

[0022] If the synchronization performance indicator is the first synchronization performance indicator, then at a preset import time point, the target heterogeneous data in different data conversion formats are imported into the corresponding target sub-database;

[0023] If the synchronization performance indicator is the second synchronization performance indicator, the database-related objects of the multiple target sub-databases are extracted from the import configuration information, and the target heterogeneous data in different data conversion formats are imported into the target database in sequence according to the import priority corresponding to the database-related objects.

[0024] In one embodiment, the data synchronization method of heterogeneous data sources further includes:

[0025] During the execution of the target synchronization task, detecting the task execution performance of the target synchronization task;

[0026] Detecting the synchronization status of the data to be synchronized according to the task execution performance;

[0027] When the synchronization state is an abnormal synchronization state, a synchronization strategy is adjusted for the data to be synchronized.

[0028] In a second aspect, the present application also provides a data synchronization method and device for heterogeneous data sources, including:

[0029] A determination module, used to determine a source database and a target database under a target synchronization task, and obtain data to be synchronized between the source database and the target database, wherein the source database and the target database are heterogeneous databases;

[0030] An export module, configured to export the data to be synchronized in the source database according to the export configuration information of the source database;

[0031] The import module is used to import the data to be synchronized into the target database according to the import configuration information of the target database after the export of the data to be synchronized is completed.

[0032] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0033] Determine a source database and a target database under a target synchronization task, and obtain data to be synchronized between the source database and the target database, wherein the source database and the target database are heterogeneous databases; export the data to be synchronized in the source database according to export configuration information of the source database; after the export of the data to be synchronized is completed, import the data to be synchronized into the target database according to import configuration information of the target database.

[0034] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0035] Determine a source database and a target database under a target synchronization task, and obtain data to be synchronized between the source database and the target database, wherein the source database and the target database are heterogeneous databases; export the data to be synchronized in the source database according to export configuration information of the source database; after the export of the data to be synchronized is completed, import the data to be synchronized into the target database according to import configuration information of the target database.

[0036] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0037] Determine a source database and a target database under a target synchronization task, and obtain data to be synchronized between the source database and the target database, wherein the source database and the target database are heterogeneous databases; export the data to be synchronized in the source database according to export configuration information of the source database; after the export of the data to be synchronized is completed, import the data to be synchronized into the target database according to import configuration information of the target database.

[0038] The above-mentioned data synchronization method, device, computer equipment, computer-readable storage medium and computer program product for heterogeneous data sources first determine the source database and target database under the target synchronization task, and obtain the data to be synchronized between the source database and the target database, wherein the source database and the target database are heterogeneous databases, that is, the data to be synchronized between the source database and the target database is the data to be synchronized between the heterogeneous data sources, and then according to the export configuration information of the source database, the data to be synchronized is exported in the source database, and after the export of the data to be synchronized is completed, the data to be synchronized is imported into the target database according to the import configuration information of the target database, so as to achieve the purpose of executing the target synchronization task of synchronizing the data to be synchronized from the heterogeneous source database to the target database. Since the export of the data to be synchronized is based on the export configuration information of the source database, in the process of exporting the data to be synchronized, There is no need to consider the compatibility between the data to be synchronized and the target database, and the import of the data to be synchronized is implemented based on the import configuration information of the target database. Therefore, in the process of importing the data to be synchronized, there is no need to consider the compatibility between the data to be synchronized and the source database. That is, in the process of data synchronization of the data to be synchronized, there is no need to consider the compatibility between the data to be synchronized and the heterogeneous databases. Therefore, the native optimization capabilities of the source database and the target database are fully utilized, rather than considering the compatibility issues in the entire stage of data synchronization of heterogeneous data sources, thereby setting additional parsing, conversion or storage steps. Therefore, the technical defects of cumbersome compatibility steps in the data synchronization process due to the different data structures of data from different source databases, which makes it easy for the data synchronization of heterogeneous data sources to take too long or the synchronization performance to decrease, etc., are overcome. Therefore, the synchronization effect of data synchronization of heterogeneous data sources is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 A schematic diagram of a process for synchronizing data of heterogeneous data sources in one embodiment;

[0041] Figure 2 A schematic diagram of a flow chart of a method for synchronizing data of heterogeneous data sources in another embodiment;

[0042] Figure 3 A schematic diagram of the structure of a data synchronization system for heterogeneous data sources of a method for synchronizing data with heterogeneous data sources in an embodiment;

[0043] Figure 4 A structural block diagram of a data synchronization device for heterogeneous data sources in one embodiment;

[0044] Figure 5 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0046] First of all, it should be understood that in the field of data processing, due to the use requirements of different database systems, how to synchronize data efficiently, accurately and with low latency between heterogeneous data sources has become a technical problem that needs to be solved in the process of enterprise information system integration. At present, traditional general-purpose data synchronization tools and middleware usually rely on general APIs or JDBC interfaces for data transmission. Although these tools can achieve data synchronization, they still have limitations in processing performance optimization of heterogeneous databases. It is difficult to fully utilize the native optimization capabilities between various database systems, resulting in performance bottlenecks, high latency and resource waste in the process of cross-source data synchronization. Among them, the performance bottleneck is mainly reflected in the fact that the data synchronization process is not synchronized. Fully utilizing the native optimization functions of each database results in low data transmission speed. High latency is mainly reflected in the high latency in the cross-source data transmission process, which affects the real-time data demand. Resource waste is mainly reflected in the data synchronization process. Resources have not been effectively managed and resource utilization efficiency is low, especially when processing large-scale data, resulting in unnecessary resource consumption. The complexity of data conversion is mainly reflected in the large differences in heterogeneous data formats and types, and cumbersome conversion, which increases the complexity and error risk of the synchronization process. Based on the above, the existing data synchronization method is prone to situations where data synchronization of heterogeneous data sources takes too long or the synchronization performance decreases. Therefore, there is an urgent need for a data synchronization method for heterogeneous data sources that can improve the synchronization effect of data synchronization of heterogeneous data sources.

[0047] In one embodiment, Figure 1As shown, a method for synchronizing data from heterogeneous data sources is provided. This embodiment takes the method applied to a terminal as an example. The terminal includes but is not limited to a personal computer, a laptop computer, a smart phone, and a tablet computer. The terminal includes a determination module, an export module, and an import module. The determination module is used to determine a source database and a target database under a target synchronization task, and obtain data to be synchronized between the source database and the target database, wherein the source database and the target database are heterogeneous databases. The export module is used to export the data to be synchronized in the source database according to the export configuration information of the source database. The import module is used to import the data to be synchronized into the target database according to the import configuration information of the target database after the export of the data to be synchronized is completed. Through information interaction among the determination module, the export module, and the import module, in the process of executing the target synchronization task of heterogeneous data, first, the source database and the target database that need to be synchronized under the target synchronization task are determined, and the data to be synchronized between the source database and the target database are obtained. The data to be synchronized between the heterogeneous databases of the target database are then exported in the source database with the help of the export configuration information of different source databases, thereby completing the export of the data to be synchronized. After the export of the data to be synchronized is completed, the data to be synchronized is imported into the target database with the help of the import configuration information of the target database. In the synchronization process of the data to be synchronized, the native optimization functions of the source database and the target database are fully utilized, and the compatibility steps in the data synchronization process are simplified. Therefore, the technical defects that the data synchronization process is cumbersome due to the different data structures of the data from different source databases, which makes it easy to have the situation that the data synchronization of heterogeneous data sources takes too long or the synchronization performance is reduced, etc., are overcome. Therefore, the synchronization effect of the data synchronization of heterogeneous data sources is improved. It can be understood that the method can also be applied to the server, and can also be applied to the system including the terminal and the server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps 202 to 206. Among them:

[0048] Step 202: determine the source database and the target database under the target synchronization task, and obtain the data to be synchronized between the source database and the target database, wherein the source database and the target database are heterogeneous databases.

[0049] It should be noted that the target database and the source database refer to different heterogeneous databases, which can be database systems of different types, different manufacturers or different versions, such as MySQL or Oracle. The target synchronization task is determined based on actual business needs. The synchronization process of heterogeneous data can be understood as the process of transmitting the business data in the source database to the target database in real time or on a scheduled basis. It can be understood that, usually, there is one source database and one target database. Furthermore, there can be multiple target databases, and the data structures of different target databases can be different. In the process of heterogeneous databases participating in synchronization, business data can be exported or imported through native tools provided by different databases. It can be understood that the target synchronization task can be manually controlled and executed by the user, or automatically controlled and executed after a timing, or can also be generated by a scheduling trigger. A target synchronization task can be regarded as a job instance. The data to be synchronized refers to the data waiting to be synchronized from the source database to the target database. For example, in an implementable manner, the user can select the first identifier of the source database and the second identifier of the target database in the task configuration interface of the target synchronization task, and place the first identifier and the second identifier in the specified position to manually create the target synchronization task, so that the source database and the target database can be determined according to the first identifier and the second identifier.

[0050] It should be noted that the target database refers to the database that stores the data to be synchronized. In the prior art, data synchronization of heterogeneous data sources is usually achieved through general data import and export tools. Among them, the native tools carried by the database system itself usually have functions such as connection management, data selection, data mapping and conversion, data verification and processing, and import logs and reports. The tool types of native tools can specifically be tools provided by database manufacturers, tools developed by third parties, open source tools, and self-developed tools. The purpose of synchronizing data from different source databases to the target database is to improve the convenience of the data to be synchronized in subsequent data analysis and business processing, and to eliminate the inconsistency between the data to be synchronized from different source databases.

[0051] As an example, step 202 includes: extracting a first identifier and a second identifier from the target synchronization task input by the user, querying the source database according to the first identifier, querying the target database according to the second identifier, and querying the source database for the data to be synchronized that needs to be synchronized from the source database to the target database according to the data synchronization identifier carried in the target synchronization task.

[0052] Step 204: export the data to be synchronized in the source database according to the export configuration information of the source database.

[0053] It should be noted that the export configuration information is used to export the data to be synchronized, and may specifically include the export data identifier, source database connection information, and data export format, etc. For example, in one implementable method, the data range of the data to be exported is set in the export configuration information, and the data range may specifically refer to a data table or data field in the database, etc. It can be understood that if there are multiple source databases, the native tools set up for different source databases may be the same or different. In the process of exporting the data to be synchronized, different source databases are only responsible for exporting the data to be synchronized within their own source databases.

[0054] As an example, step 204 includes: locating corresponding data to be synchronized in the source database according to the export configuration information of the source database, and exporting the data to be synchronized.

[0055] Step 206: After the export of the data to be synchronized is completed, the data to be synchronized is imported into the target database according to the import configuration information of the target database.

[0056] It should be noted that the import configuration information is used to import the data to be synchronized, and may specifically include the data source type, server address, database name, user name and password, etc. During the process of importing the data to be synchronized into the target database, the data to be synchronized may be imported randomly based on the performance of the target database import tool.

[0057] As an example, step 206 includes: after the export of the data to be synchronized is completed, according to the import configuration information of the target database, locating the storage space of the data to be synchronized in the target database, and importing the data to be synchronized into the storage space of the target database.

[0058] In the above-mentioned data synchronization method for heterogeneous data sources, firstly, according to the first identifier and the second identifier in the target synchronization task, the source database and the target database are respectively determined, and according to the data synchronization identifier carried in the target synchronization task, the to-be-synchronized data that needs to be synchronized from the source database to the target database is located in the source database, and then according to the export configuration information of the source database, the to-be-synchronized data is exported in the source database, and finally after the export of the to-be-synchronized data is completed, according to the import configuration information of the target database, the to-be-synchronized data is imported into the target database, so that the purpose of executing the target synchronization task of synchronizing the to-be-synchronized data from the heterogeneous source database to the target database can be achieved. Since the export of the to-be-synchronized data is realized based on the export configuration information of the source database, there is no need to consider the compatibility of the to-be-synchronized data and the target database during the export of the to-be-synchronized data, and The import of the data to be synchronized is implemented based on the import configuration information of the target database. Therefore, during the import of the data to be synchronized, there is no need to consider the compatibility between the data to be synchronized and the source database. That is, during the data synchronization of the data to be synchronized, there is no need to consider the compatibility between the data to be synchronized and the heterogeneous databases. Therefore, the native optimization capabilities of the source database and the target database are fully utilized, rather than considering the compatibility issues in the entire stage of data synchronization of heterogeneous data sources, thereby setting additional parsing, conversion or storage steps. Therefore, the technical defects of cumbersome compatibility steps in the data synchronization process due to the different data structures of data from different source databases, which makes it easy for the data synchronization of heterogeneous data sources to take too long or the synchronization performance to decrease, etc., are overcome. Therefore, the synchronization effect of data synchronization of heterogeneous data sources is improved.

[0059] In one embodiment, Figure 2 As shown, according to the export configuration information of the source database, the data to be synchronized is exported from the source database, including:

[0060] Step 302: extract the export configuration rule and the export configuration time from the export configuration information.

[0061] It should be noted that in different data synchronization tasks, the amount of data synchronized from different source databases is different. Therefore, in the data synchronization system of heterogeneous data sources that executes the data synchronization method of heterogeneous data sources, a task management scheduling module can also be deployed. Its function is to schedule cross-source data synchronization tasks and support parallel execution. The process is as follows: optimize the execution efficiency of multi-threaded tasks through intelligent sharding and concurrency control, and maximize the use of system resources. That is, the export configuration information can carry export configuration rules and export configuration time. The export configuration rules are used to divide the data to be synchronized, and the export configuration time is used to identify the time when each sub-data to be synchronized is exported from the corresponding source database.

[0062] As an example, step 302 includes: proposing export configuration rules and export configuration time in the export configuration information.

[0063] Step 304: split the data to be synchronized into multiple sub-data to be synchronized according to the export configuration rule.

[0064] It should be noted that the sub-data to be synchronized refers to the local data to be synchronized extracted from the data to be synchronized. The export configuration rules can be determined based on the user's actual business needs and the transmission performance of the source database. It can be understood that the data volume of different sub-data to be synchronized can be the same or different.

[0065] As an example, step 304 includes: splitting the data to be synchronized corresponding to the source database according to the export configuration rule to obtain a plurality of sub-data to be synchronized.

[0066] Step 306: Export multiple sub-data to be synchronized in the source database according to the export configuration time.

[0067] As an example, step 306 includes: at the export time point identified by the export configuration time, exporting the sub-data to be synchronized from the source database in sequence.

[0068] In this embodiment, in the process of exporting the data to be synchronized, the data to be synchronized corresponding to the source database can be intelligently sliced ​​based on the export configuration rules carried in the export configuration information, so as to obtain multiple sub-data to be synchronized, and finally multiple sub-data to be synchronized are exported in the source database depending on the export configuration time. That is, a task management scheduling module can be set in the data synchronization system of heterogeneous data sources, and intelligent slicing and concurrency control optimization of multi-threaded tasks can be implemented through the task management scheduling module, thereby ensuring that in the process of data synchronization tasks facing large-scale heterogeneous data, effective management of data synchronization can be performed, thereby improving the resource utilization of the source database. Therefore, while laying the foundation for improving the synchronization effect of data synchronization of heterogeneous data sources, the resource utilization of multiple source databases is improved.

[0069] In one embodiment, the target database includes a plurality of target sub-databases; importing the data to be synchronized into the target database according to the import configuration information of the target database includes:

[0070] Extract the data conversion formats corresponding to the multiple target sub-databases in the import configuration information; convert the data to be synchronized into target heterogeneous data under each data conversion format; import the target heterogeneous data under different data conversion formats into the corresponding target sub-database.

[0071] It should be noted that in some application scenarios, the data to be synchronized has large differences in data format and type, which leads to cumbersome conversion process during the conversion of the data to be synchronized. For example, the data to be synchronized in the source database needs to be imported into different target sub-databases respectively. When the data formats and types of different target sub-databases are very different, a data conversion compatibility module can be deployed in the data synchronization system of heterogeneous data sources. The data conversion compatibility module can automatically convert the source data into a data format compatible with the target sub-database. The process is as follows: perform format conversion during the data synchronization process to ensure data accuracy and completeness, that is, complete the deep integration of the source database native tools to adapt to the characteristics of each target sub-database, wherein the target heterogeneous data refers to the synchronized heterogeneous data converted in the data conversion format.

[0072] As an example, the data conversion formats corresponding to multiple target sub-databases are extracted from the import configuration information; the data to be synchronized is converted in the data conversion format to obtain multiple target heterogeneous data, wherein one target heterogeneous data corresponds to one target sub-database; and the target heterogeneous data under different data conversion formats are imported into the corresponding target sub-database.

[0073] In this embodiment, firstly, the data conversion formats corresponding to the multiple target sub-databases are extracted from the import configuration information, and then the multiple data format conversions of the data to be synchronized are completed with the targeted data conversion formats to obtain multiple target heterogeneous data, wherein one target heterogeneous data corresponds to one target sub-database, and finally the target heterogeneous data under different data conversion formats are imported into the corresponding target sub-database, so that in the process of importing the data to be synchronized into the target database, the differences between the heterogeneous data formats and types of different target sub-databases are eliminated, and the conversion process is simplified, that is, the purpose of importing the data to be synchronized into target sub-databases of different data formats and types is achieved. Therefore, while laying the foundation for improving the synchronization effect of data synchronization of heterogeneous data sources, the synchronization accuracy of data synchronization of heterogeneous data sources is improved.

[0074] In one embodiment, importing target heterogeneous data in different data conversion formats into corresponding target sub-databases includes:

[0075] Obtain a first synchronization performance characteristic of a source database and a second synchronization performance characteristic of multiple target sub-databases; generate a synchronization performance indicator of a target synchronization task during execution based on the first synchronization performance characteristic and the second synchronization performance characteristic; import target heterogeneous data in different data conversion formats into corresponding target sub-databases based on the synchronization performance indicator.

[0076] It should be noted that, since the performance of different target sub-databases is dynamic, in order to ensure the data synchronization performance of heterogeneous data sources, a performance optimization module can be deployed in the data synchronization system of the heterogeneous data source during the process of synchronizing the data to be synchronized to the heterogeneous data source. The performance optimization module can dynamically adjust the synchronization strategy according to the performance characteristics of the source database and the target database. The process can be as follows: analyze and collect database performance characteristics, optimize the data transmission method (point-to-point transmission or batch transmission), so as to select the optimal transmission method, and self-heal when an abnormality occurs to ensure synchronization performance. The synchronization performance characteristics are used to characterize the synchronization performance during the execution of the target synchronization task. For example, in an implementable manner, the first synchronization performance characteristics and the second synchronization performance characteristics can be spliced ​​into the synchronization performance indicators of the target synchronization task during the execution process. It can be understood that in the process of the heterogeneous data system executing the target synchronization task, the data conversion compatibility module, the task management scheduling module and the performance optimization module all act in the export stage of multiple data to be synchronized and the import stage of the data to be synchronized. The first synchronization performance indicator and the second synchronization performance indicator can specifically be execution time, throughput, error rate and resource usage.

[0077] As an example, a first synchronization performance characteristic of a source database and a second synchronization performance characteristic of multiple target sub-databases are obtained; the first synchronization performance characteristic and the second synchronization performance characteristic are spliced ​​into a synchronization performance indicator of the target synchronization task during execution; and according to the synchronization performance indicator, multiple target heterogeneous data are imported into the corresponding target sub-database in sequence.

[0078] In this embodiment, in the process of importing target heterogeneous data in different data conversion formats into corresponding target sub-databases, firstly, the first synchronization performance characteristics of the source database and the second synchronization performance characteristics of multiple target sub-databases are respectively obtained, and then the target heterogeneous data in different data conversion formats are imported into the corresponding target sub-database based on the synchronization performance characteristics generated by the first synchronization performance characteristics and the second synchronization performance characteristics. That is, through the performance optimization module deployed by the data synchronization system of the heterogeneous data source, data synchronization is always performed in the optimal data transmission method during the data synchronization process. Therefore, while laying the foundation for improving the synchronization effect of data synchronization of heterogeneous data sources, the database operation performance of data synchronization of heterogeneous data sources is improved.

[0079] In one embodiment, according to the synchronization performance index, target heterogeneous data in different data conversion formats are imported into the corresponding target sub-database, including:

[0080] If the synchronization performance indicator is the first synchronization performance indicator, the target heterogeneous data in different data conversion formats are imported into the corresponding target sub-database at the preset import time point; if the synchronization performance indicator is the second synchronization performance indicator, the database-associated objects of multiple target sub-databases are extracted from the import configuration information, and according to the import priority corresponding to the database-associated objects, the target heterogeneous data in different data conversion formats are imported into the target database in sequence.

[0081] It should be noted that the first synchronization performance indicator and the second synchronization performance indicator refer to different types of synchronization performance indicators. It can be understood that when the types of synchronization performance indicators are different, the methods of importing multiple target heterogeneous data into the target sub-database are different, wherein the preset import time point is an import time point set in advance for batch importing multiple target heterogeneous data, and the import priority can determine the import order of the target heterogeneous data under the database system from the source database to the target sub-database. The database-related object is used to display the database object associated with the target heterogeneous data, which can specifically be the tenant to which the database belongs, that is, the data for different tenants. According to synchronization, there are differences in import priorities. For example, database system a corresponds to tenant A, and database system b corresponds to tenant B. The target synchronization tasks include the synchronization requirements of target heterogeneous data under database system a from the source database to sub-database x, and the synchronization requirements of target heterogeneous data under database system b from the source database to sub-database y. If tenant A is the core tenant and tenant B is the edge tenant, the synchronization process of target heterogeneous data from the source database to sub-database x can be completed first. It can be understood that the import priorities of different target heterogeneous data can be comprehensively derived through business needs analysis, data value assessment and technical feasibility analysis.

[0082] As an example, if the synchronization performance indicator is the first synchronization performance indicator, the target heterogeneous data in different data conversion formats are imported into the corresponding target sub-database at the preset import time point; if the synchronization performance indicator is the second synchronization performance indicator, for any target sub-database, the database-associated object of the target sub-database is extracted from the import configuration information, and the import priority of the data to be synchronized is predicted according to the attribute information of the database-associated object, until each target sub-database completes the above steps, and the import priorities of multiple target sub-databases are obtained, and the multiple import priorities are sorted to obtain the data import order of multiple data to be synchronized, and according to the data import order, the target heterogeneous data in different data conversion formats are imported into the corresponding target sub-database in sequence.

[0083] In this embodiment, in the process of importing multiple target heterogeneous data into corresponding target sub-databases, different import methods are adopted to import the target heterogeneous data based on different types of synchronization performance indicators, that is, in the process of point-to-point transmission / import of multiple target heterogeneous data, the import priorities of different target sub-databases are determined through the database association objects of the multiple target sub-databases, and the multiple target heterogeneous data are imported into the corresponding target sub-database in sequence according to the import priority. Therefore, while laying the foundation for improving the synchronization effect of data synchronization of heterogeneous data sources, the synchronization flexibility of data synchronization of heterogeneous data sources is improved.

[0084] In one embodiment, the data synchronization method of heterogeneous data sources further includes:

[0085] During the execution of the target synchronization task, the task execution performance of the target synchronization task is detected; based on the task execution performance, the synchronization state of the data to be synchronized is detected; when the synchronization state is an abnormal synchronization state, the synchronization strategy of the data to be synchronized is adjusted.

[0086] It should be noted that in the data synchronization system of heterogeneous data sources, a real-time monitoring module can also be deployed. The function of the real-time monitoring module is to dynamically track the progress of synchronization tasks and monitor performance. The specific process is as follows: real-time monitoring during the synchronization process, supporting dynamic adjustment of synchronization strategies, and ensuring the stable operation of data synchronization tasks. That is, during the execution of the target synchronization task, the real-time monitoring module will monitor the synchronization status of the data to be synchronized during the export and import process in real time. When the synchronization status is a normal synchronization status, the synchronization status will be continuously monitored. When the synchronization status is an abnormal synchronization status, the synchronization strategy of the data to be synchronized can be adjusted in real time.

[0087] As an example, during the execution of the target synchronization task, the task execution performance of the target synchronization task is detected; based on the task execution performance, the synchronization status of the data to be synchronized is queried; when the synchronization status is identified as an abnormal synchronization status, the current synchronization strategy of the data to be synchronized is dynamically adjusted.

[0088] In this embodiment, during the execution of the target synchronization task, the synchronization status of synchronized heterogeneous data can be monitored in real time, and the synchronization strategy can be adjusted in real time based on the real-time detection status feedback to ensure the stable operation of the task, thereby improving the synchronization stability of data synchronization of heterogeneous data sources.

[0089] In one practicable manner, referring to Figure 3 , Figure 3The figure is a structural diagram of the data synchronization system for heterogeneous data sources, where the data synchronization system for heterogeneous data sources includes a data source adaptation module, a data conversion compatibility module, a task management and scheduling module, a performance optimization module and a real-time monitoring module. Assume that the target synchronization task is an actual business operation task of a tenant, where the source database is GaussDB and the target database is OceanBase. First, the data synchronization system of the heterogeneous data source automatically calls the GDS service of GaussDB (GaussDB's native tool) to export the data to be synchronized. After the source database completes the export of the data to be synchronized, the OBLOADER of OceanBase (OceanBase's native tool) is used to import the data to be synchronized. The GDS service supports multi-threaded concurrent export, and OBLOADER supports high-concurrency data writing and automatic load balancing to ensure that the target database can receive data quickly and efficiently. It can be understood that the target synchronization task is accompanied by format conversion compatibility, task management and scheduling, and performance optimization modules during operation to ensure stable and efficient data synchronization to the greatest extent.

[0090] Since the export of the data to be synchronized is implemented based on the export configuration information of the source database, there is no need to consider the compatibility between the data to be synchronized and the target database during the export process of the data to be synchronized, and the import of the data to be synchronized is implemented based on the import configuration information of the target database, there is no need to consider the compatibility between the data to be synchronized and the source database during the import process of the data to be synchronized, that is, in the process of data synchronization of the data to be synchronized, there is no need to consider the compatibility issues between the data to be synchronized and the heterogeneous databases. Therefore, the native optimization capabilities of the source database and the target database are fully utilized, instead of considering the compatibility issues in the entire stage of data synchronization of heterogeneous data sources, thereby setting additional parsing, conversion or storage steps. Therefore, the technical defects of cumbersome compatibility steps in the data synchronization process due to the different data structures of data from different source databases, which makes it easy for the data synchronization of heterogeneous data sources to take too long or the synchronization performance to decrease, etc., are overcome. Therefore, the synchronization effect of data synchronization of heterogeneous data sources is improved.

[0091] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0092] Based on the same inventive concept, the embodiment of the present application also provides a data synchronization device for heterogeneous data sources for implementing the data synchronization method for heterogeneous data sources involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in the embodiments of the data synchronization device for one or more heterogeneous data sources provided below can refer to the limitations of the data synchronization method for heterogeneous data sources above, and will not be repeated here.

[0093] In an exemplary embodiment, Figure 4 As shown, a data synchronization device for heterogeneous data sources is provided, including: a determination module 401, an export module 402 and an import module 403, wherein:

[0094] The determination module 401 is used to determine a source database and a target database under a target synchronization task, and obtain data to be synchronized between the source database and the target database, wherein the source database and the target database are heterogeneous databases;

[0095] An export module 402, configured to export the to-be-synchronized data in the source database according to the export configuration information of the source database;

[0096] The import module 403 is used to import the data to be synchronized into the target database according to the import configuration information of the target database after the export of the data to be synchronized is completed.

[0097] In one embodiment, the export module 402 is further used to:

[0098] Extracting export configuration rules and export configuration time from the export configuration information; splitting the data to be synchronized into multiple sub-data to be synchronized according to the export configuration rules; and exporting the multiple sub-data to be synchronized in the source database according to the export configuration time.

[0099] In one embodiment, the target database includes a plurality of target sub-databases; the import module 403 is further used to:

[0100] Extract the data conversion formats corresponding to each of the multiple target sub-databases from the import configuration information; convert the data to be synchronized into target heterogeneous data under each of the data conversion formats; import the target heterogeneous data under different data conversion formats into the corresponding target sub-database.

[0101] In one embodiment, the import module 403 is further used to:

[0102] Obtain a first synchronization performance characteristic of the source database and a second synchronization performance characteristic of the multiple target sub-databases; generate a synchronization performance indicator of the target synchronization task during execution based on the first synchronization performance characteristic and the second synchronization performance characteristic; and import target heterogeneous data in different data conversion formats into corresponding target sub-databases based on the synchronization performance indicator.

[0103] In one embodiment, the import module 403 is further used to:

[0104] If the synchronization performance indicator is the first synchronization performance indicator, the target heterogeneous data in different data conversion formats are imported into the corresponding target sub-database at a preset import time point; if the synchronization performance indicator is the second synchronization performance indicator, the database-associated objects of the multiple target sub-databases are extracted from the import configuration information, and according to the import priority corresponding to the database-associated objects, the target heterogeneous data in different data conversion formats are imported into the target database in sequence.

[0105] In one of the embodiments, the data synchronization device for heterogeneous data sources is further used for:

[0106] During the execution of the target synchronization task, the task execution performance of the target synchronization task is detected; based on the task execution performance, the synchronization state of the data to be synchronized is detected; when the synchronization state is an abnormal synchronization state, the synchronization strategy of the data to be synchronized is adjusted.

[0107] Each module in the above-mentioned data synchronization device for heterogeneous data sources can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0108] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 5As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface, the display unit and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a data synchronization method for heterogeneous data sources is implemented. Those skilled in the art can understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0109] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0110] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0111] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0112] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0113] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0114] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data synchronization method for heterogeneous data sources, characterized in that: The method comprises: Determine a source database and a target database under a target synchronization task, and obtain data to be synchronized between the source database and the target database, wherein the source database and the target database are heterogeneous databases; Exporting the data to be synchronized in the source database according to the export configuration information of the source database; After the export of the data to be synchronized is completed, the data to be synchronized is imported into the target database according to the import configuration information of the target database.

2. The method according to claim 1, characterized in that The exporting the to-be-synchronized data in the source database according to the export configuration information of the source database includes: Extracting export configuration rules and export configuration time from the export configuration information; According to the export configuration rule, the data to be synchronized is split into a plurality of sub-data to be synchronized; The plurality of sub-data to be synchronized are exported in the source database according to the export configuration time.

3. The method according to claim 1, characterized in that The target database includes a plurality of target sub-databases; and importing the to-be-synchronized data into the target database according to the import configuration information of the target database includes: Extracting the data conversion formats corresponding to the multiple target sub-databases respectively from the import configuration information; Converting the data to be synchronized into target heterogeneous data in each of the data conversion formats; Import the target heterogeneous data in different data conversion formats into the corresponding target sub-database.

4. The method according to claim 3, characterized in that The step of importing target heterogeneous data in different data conversion formats into corresponding target sub-databases includes: Acquire a first synchronization performance characteristic of the source database and a second synchronization performance characteristic of the plurality of target sub-databases; Generating a synchronization performance indicator of the target synchronization task during execution according to the first synchronization performance characteristic and the second synchronization performance characteristic; According to the synchronization performance index, the target heterogeneous data in different data conversion formats are imported into the corresponding target sub-database.

5. The method according to claim 4, characterized in that The step of importing target heterogeneous data in different data conversion formats into corresponding target sub-databases according to the synchronization performance indicator includes: If the synchronization performance indicator is the first synchronization performance indicator, then at a preset import time point, the target heterogeneous data in different data conversion formats are imported into the corresponding target sub-database; If the synchronization performance indicator is the second synchronization performance indicator, the database-related objects of the multiple target sub-databases are extracted from the import configuration information, and the target heterogeneous data in different data conversion formats are imported into the target database in sequence according to the import priority corresponding to the database-related objects.

6. The method according to claim 1, characterized in that The method further comprises: During the execution of the target synchronization task, detecting the task execution performance of the target synchronization task; Detecting the synchronization status of the data to be synchronized according to the task execution performance; When the synchronization state is an abnormal synchronization state, a synchronization strategy is adjusted for the data to be synchronized.

7. A data synchronization device for heterogeneous data sources, characterized in that: The device comprises: A determination module, used to determine a source database and a target database under a target synchronization task, and obtain data to be synchronized between the source database and the target database, wherein the source database and the target database are heterogeneous databases; An export module, configured to export the data to be synchronized in the source database according to the export configuration information of the source database; The import module is used to import the data to be synchronized into the target database according to the import configuration information of the target database after the export of the data to be synchronized is completed.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images