Data synchronization method and device, computer equipment and storage medium

Abstract

The invention relates to a data synchronization method and device, computer equipment and a storage medium. The method comprises the following steps: generating a data synchronization strategy in response to a data synchronization configuration operation of a user; creating a plurality of data synchronization tasks according to the data synchronization strategy, wherein different data synchronization tasks are used for synchronizing data in different databases to corresponding data warehouses; executing each data synchronization task according to the synchronization rule corresponding to each data synchronization task; and when any data synchronization task is executed, the synchronization process of the data synchronization task is monitored in real time, and when an abnormity or an error is monitored, troubleshooting processing is carried out. According to the embodiment of the invention, the method supports the real-time synchronization of data from a plurality of different relational databases to a plurality of data warehouses, and can effectively improve the real-time performance of data synchronization in the aspect of data synchronization.

Description

Technical Field

[0001] The present application relates to the technical field of data synchronization, and in particular to a data synchronization method, apparatus, computer device and storage medium. Background Art

[0002] With the rapid growth of data volume and the increasing demand for real-time data analysis, data warehouses, as key components of big data processing, are particularly important for ensuring data consistency. In existing data synchronization technologies, there are many solutions, such as trigger-based data synchronization solutions and timestamp-based data synchronization solutions. However, these solutions often have problems such as synchronization delays and performance bottlenecks, and cannot meet application scenarios with high real-time requirements. In addition, current data synchronization solutions usually cannot meet the data synchronization requirements of multiple sources and multiple targets (i.e., multiple source databases and multiple target databases). Summary of the invention

[0003] In response to the above-mentioned deficiencies or shortcomings, the present application provides a data synchronization method, apparatus, computer device and storage medium. The embodiments of the present application support real-time synchronization of data from multiple different relational databases to multiple data warehouses, and in terms of data synchronization, can also effectively improve the real-time performance of data synchronization.

[0004] The present application provides a data synchronization method according to a first aspect. In some embodiments, the method includes:

[0005] generating a data synchronization policy in response to a user's data synchronization configuration operation;

[0006] Create multiple data synchronization tasks based on the data synchronization strategy. Different data synchronization tasks are used to synchronize data from different databases to the corresponding data warehouse.

[0007] Execute each data synchronization task according to the synchronization rules corresponding to each data synchronization task;

[0008] When executing any data synchronization task, the synchronization process of the data synchronization task is monitored in real time, and when anomalies or errors are monitored, troubleshooting is performed.

[0009] In some embodiments, generating a data synchronization policy in response to a user's data synchronization configuration operation includes:

[0010] Receive the configuration file uploaded by the user through the configuration client, parse the configuration file, and generate a data synchronization policy based on the data synchronization configuration information parsed from the configuration file;

[0011] Alternatively, data synchronization configuration information transmitted by a user through a preset data synchronization configuration interface is received, and a data synchronization policy is generated according to the received data synchronization configuration information.

[0012] In some embodiments, the synchronization strategy includes multiple pieces of data synchronization configuration information; each piece of data synchronization configuration information includes a database identifier, a data warehouse identifier, a data table identifier and a field identifier to be synchronized, and data synchronization frequency information;

[0013] Create multiple data synchronization tasks based on the data synchronization strategy, including:

[0014] A data synchronization task is created according to the database identifier, data warehouse identifier, data table identifier to be synchronized and field identifier included in each piece of data synchronization configuration information, and a corresponding synchronization rule is set for the corresponding data synchronization task according to the data synchronization frequency information included in each piece of data synchronization configuration information.

[0015] In some embodiments, when executing any data synchronization task, a database identifier and a data warehouse identifier associated with the data synchronization task are obtained, and a database corresponding to the database identifier is used as a source database and a data warehouse corresponding to the data warehouse identifier is used as a target data warehouse;

[0016] Capture the change log of the source database in real time through the preset log parser;

[0017] Parse the change log to obtain the change data;

[0018] Perform data conversion and data cleansing on the changed data according to business requirements to obtain the target data to be synchronized;

[0019] Synchronize target data to the target data warehouse in real time.

[0020] In some embodiments, data conversion and data cleaning are performed on the changed data according to business requirements to obtain target data to be synchronized, including:

[0021] The data processing rules associated with the data synchronization task are obtained, and the changed data is converted and cleaned according to the data processing rules to obtain the target data to be synchronized.

[0022] In some embodiments, after creating multiple data synchronization tasks according to the data synchronization policy, the method further includes:

[0023] Receive data processing rules configured by users for each data synchronization task based on business needs; data processing rules include data conversion rules and data cleaning rules;

[0024] Store each data processing rule and establish an association between each data processing rule and its corresponding data synchronization task.

[0025] In some embodiments, the method further comprises:

[0026] When executing any data synchronization task, the synchronization process of the data synchronization task is optimized according to business needs and the amount of data to be synchronized.

[0027] According to a second aspect, the present application provides a data synchronization device. In some embodiments, the device includes:

[0028] A policy generation module, used to generate a data synchronization policy in response to a user's data synchronization configuration operation;

[0029] The task creation module is used to create multiple data synchronization tasks according to the data synchronization strategy. Different data synchronization tasks are used to synchronize data from different databases to the corresponding data warehouse.

[0030] A task execution module is used to execute each data synchronization task according to the synchronization rules corresponding to each data synchronization task;

[0031] The task monitoring module is used to monitor the synchronization process of any data synchronization task in real time when executing the task, and to perform troubleshooting when anomalies or errors are detected.

[0032] According to a third aspect, the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the data synchronization method provided in any of the above embodiments when executing the computer program.

[0033] According to a fourth aspect, the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the data synchronization method provided in any of the above embodiments are implemented.

[0034] The above-mentioned embodiments of the present application support real-time synchronization of data from multiple different relational databases to multiple data warehouses. Specifically, the present embodiment can generate a data synchronization strategy based on the user's data synchronization configuration operation, and then create multiple data synchronization tasks based on the data synchronization strategy, wherein different data synchronization tasks are used to synchronize data from different databases to corresponding data warehouses; then, each data synchronization task is executed according to the synchronization rules corresponding to each data synchronization task. When each data synchronization task is executed, it will capture the change log of the source database in real time, and synchronize the changed data in the source database to the relevant target data warehouse in real time to meet the needs of real-time data analysis. In addition, when executing any data synchronization task, the present embodiment monitors the synchronization process of the data synchronization task in real time, and performs troubleshooting when anomalies or errors are monitored, thereby improving the stability and reliability of the data synchronization process. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A flowchart of a data synchronization method provided by the present application according to one or more embodiments;

[0036] Figure 2 A schematic diagram of an operation flow of performing a data synchronization task provided by the present application according to one or more embodiments;

[0037] Figure 3 A structural block diagram of a data synchronization device provided by the present application according to one or more embodiments;

[0038] Figure 4 This is a diagram of the internal structure of a computer device provided by the present application according to one or more embodiments. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below in conjunction with the accompanying drawings. It should be clear that the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0040] When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0041] In the description of the present application, it should be understood that the terms "first", "second", "third", etc. are only used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence, nor can they be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances. In addition, in the description of the present application, unless otherwise specified, "multiple" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0042] In response to the deficiencies or defects of the prior art, the present application provides a data synchronization method, which supports real-time synchronization of data from multiple different relational databases to multiple data warehouses. Specifically, the method can generate a data synchronization strategy based on the user's data synchronization configuration operation, and then create multiple data synchronization tasks based on the data synchronization strategy, wherein different data synchronization tasks are used to synchronize data from different databases to corresponding data warehouses; then, each data synchronization task is executed according to the synchronization rules corresponding to each data synchronization task. When each data synchronization task is executed, it will capture the change log of the source database in real time, and synchronize the changed data in the source database to the relevant target data warehouse in real time to meet the needs of real-time data analysis. In addition, when executing any data synchronization task, the synchronization process of the data synchronization task will be monitored in real time, and when anomalies or errors are monitored, troubleshooting will be performed, thereby improving the stability and reliability of the data synchronization process.

[0043] In some embodiments, the method may be used in a data synchronization device, which may be a server independent of each database and data warehouse, and may be implemented using an independent server or a server cluster consisting of multiple servers. Figure 1 , Figure 1 The figure shows the steps included in the data synchronization method provided in this embodiment. The following takes the method applied to a data synchronization device as an example to describe the steps of the method in detail.

[0044] S110: Generate a data synchronization policy in response to a user's data synchronization configuration operation.

[0045] Among them, the operation of the data synchronization device generating a data synchronization policy in response to the user's data synchronization configuration operation can be to receive a configuration file uploaded by the user through a configuration client, parse the configuration file, and generate a data synchronization policy based on the data synchronization configuration information parsed from the configuration file; or, to receive data synchronization configuration information transmitted by the user through a preset data synchronization configuration interface, and generate a data synchronization policy based on the received data synchronization configuration information.

[0046] This embodiment supports real-time synchronization of data from multiple different relational databases (such as MySQL, Oracle, etc.) to multiple data warehouses (such as ElasticSearch, adb, etc.) to meet the diverse business needs of enterprises.

[0047] In this embodiment, the data synchronization device provides users with a flexible configuration method, which allows users to flexibly define data synchronization policies through configuration files or APIs (Application Programming Interface). Specifically, users can configure data synchronization policies by setting data configuration information in a configuration file of a predetermined format and then uploading the configuration file to the data synchronization device. Alternatively, data configuration information can be uploaded to the data synchronization device through an API interface pre-opened by the data synchronization device. The data synchronization device will generate a corresponding data synchronization policy based on the data synchronization configuration information parsed from the configuration file or the data synchronization configuration information received through the API interface. The policy is all data synchronization configuration information associated with the user.

[0048] Users can configure corresponding data synchronization configuration information for multiple different relational databases according to actual needs. Each data synchronization configuration information includes parameters such as the database identifier of the relevant relational database that needs to synchronize data, the identifier of the relevant data warehouse, the data table identifier and field identifier to be synchronized, and data synchronization frequency information.

[0049] S120: Creating multiple data synchronization tasks according to the data synchronization strategy, where different data synchronization tasks are used to synchronize data from different databases to corresponding data warehouses.

[0050] The data synchronization strategy includes multiple pieces of data synchronization configuration information; each piece of data synchronization configuration information includes a database identifier, a data warehouse identifier, a data table identifier and a field identifier to be synchronized, and data synchronization frequency information.

[0051] Correspondingly, the operation of creating multiple data synchronization tasks according to the data synchronization strategy can be to create data synchronization tasks according to the database identifier, data warehouse identifier, data table identifier to be synchronized and field identifier included in each data synchronization configuration information. Each data synchronization task is configured with a unique corresponding task identifier. The data synchronization device also sets corresponding synchronization rules for the corresponding data synchronization tasks according to the data synchronization frequency information included in each data synchronization configuration information.

[0052] S130: Execute each data synchronization task according to the synchronization rule corresponding to each data synchronization task.

[0053] The data synchronization device will execute each data synchronization task according to the data synchronization frequency corresponding to each data synchronization task.

[0054] In some embodiments, the operation flow of the data synchronization device when executing any data synchronization task is as follows: Figure 2 As shown, that is:

[0055] S210: Obtain the database identifier and data warehouse identifier associated with the data synchronization task, and use the database corresponding to the database identifier as the source database and the data warehouse corresponding to the data warehouse identifier as the target data warehouse. The data synchronization device can obtain the network addresses of the source database and the target data warehouse according to the database identifier and the data warehouse identifier, and establish connections with the source database and the target data warehouse respectively based on the network addresses.

[0056] S220: Capture the change log of the source database in real time through a preset log parser.

[0057] This embodiment uses log capture technology to capture the change log of the source database in real time. The change log can be a binlog file. The change log records the relevant information of each data changed in the source database, such as the corresponding data change type (such as addition, modification, deletion), the corresponding data table and field identification, the data value before and after the change, etc.

[0058] S230: Parse the change log to obtain change data.

[0059] This embodiment can use a general log parsing algorithm to parse the change log, so as to obtain relevant information of each data that has changed in the source database recorded therein. After obtaining this information, the data synchronization device filters out the data corresponding to the data table identifier and field identifier to be synchronized related to the data synchronization task as the change data.

[0060] S240: Perform data conversion and data cleansing on the changed data according to business requirements to obtain target data to be synchronized.

[0061] For the parsed data, this embodiment will perform a series of processing such as data conversion and cleaning to ensure the quality and accuracy of the data synchronized to the target data warehouse.

[0062] Among them, the operation of converting and cleaning the changed data according to business needs to obtain the target data to be synchronized can be to obtain the data processing rules associated with the data synchronization task, and then converting and cleaning the changed data according to the data processing rules to obtain the target data to be synchronized.

[0063] The above-mentioned data processing rules can be pre-configured by the user for the data synchronization task according to actual business needs. Accordingly, after creating multiple data synchronization tasks according to the data synchronization strategy, the method also includes: receiving the data processing rules configured by the user for each data synchronization task according to business needs; the data processing rules include data conversion rules and data cleaning rules; storing each data processing rule, and establishing an association relationship between each data processing rule and its corresponding data synchronization task.

[0064] After creating a data synchronization task, the user can pre-configure corresponding data conversion rules and data cleaning rules for each data synchronization task according to actual business needs. After the data synchronization device receives the data conversion rules and data cleaning rules configured by the user for each data synchronization task, it will establish an association relationship between the task identifier of each data synchronization task and the data conversion rules and data cleaning rules corresponding to each data synchronization task, so that when the data synchronization task is subsequently executed, the associated data conversion rules and data cleaning rules can be obtained according to the task identifier of the data synchronization task.

[0065] S250: Synchronize the target data to the target data warehouse in real time.

[0066] The target data is the data obtained after a series of processing such as data conversion and data cleaning. The data synchronization device will synchronize the target data to the target data warehouse in real time to meet the real-time data update requirements and keep the data in the source database and the target data warehouse consistent.

[0067] S140: When executing any data synchronization task, the synchronization process of the data synchronization task is monitored in real time, and when an abnormality or error is monitored, troubleshooting is performed.

[0068] Furthermore, when executing any data synchronization task, the data synchronization task monitors the synchronization process of the data synchronization task in real time, and the monitoring targets include but are not limited to data synchronization progress, data flow, error logs, etc. When an exception or error occurs during the synchronization process, an alarm notification is sent to relevant personnel in a timely manner. For example, when an exception is found during the data synchronization process or the synchronization delay exceeds the threshold, the data synchronization device automatically triggers an alarm notification so that relevant personnel can conduct troubleshooting in a timely manner to ensure the stability and reliability of data synchronization.

[0069] In some embodiments, the method further includes: when executing any data synchronization task, optimizing the performance of the synchronization process of the data synchronization task according to business requirements and the amount of data to be synchronized.

[0070] In this embodiment, the data synchronization device may also optimize the performance of the data synchronization process according to business requirements and the amount of data to improve synchronization efficiency.

[0071] It should be noted that, with respect to the various steps included in the data synchronization method provided in any of the above embodiments, unless otherwise clearly stated in this document, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a part of these steps may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0072] Based on the same inventive concept, the present application also provides a data synchronization device. In some embodiments, Figure 4 As shown, the data synchronization device includes the following modules:

[0073] A policy generation module 110, for generating a data synchronization policy in response to a user's data synchronization configuration operation;

[0074] A task creation module 120 is used to create multiple data synchronization tasks according to the data synchronization strategy, and different data synchronization tasks are used to synchronize data from different databases to corresponding data warehouses;

[0075] The task execution module 130 is used to execute each data synchronization task according to the synchronization rule corresponding to each data synchronization task;

[0076] The task monitoring module 140 is used to monitor the synchronization process of any data synchronization task in real time when executing the task, and to perform troubleshooting when anomalies or errors are detected.

[0077] In some embodiments, the policy generation module 110 is specifically used to: receive a configuration file uploaded by a user through a configuration client, parse the configuration file, and generate a data synchronization policy based on the data synchronization configuration information parsed from the configuration file; or, receive data synchronization configuration information transmitted by a user through a preset data synchronization configuration interface, and generate a data synchronization policy based on the received data synchronization configuration information.

[0078] In some embodiments, the synchronization strategy includes multiple data synchronization configuration information; each piece of data synchronization configuration information includes a database identifier, a data warehouse identifier, a data table identifier and a field identifier to be synchronized, and data synchronization frequency information; accordingly, the task creation module 120 is specifically used to: create a data synchronization task according to the database identifier, data warehouse identifier, data table identifier and field identifier to be synchronized included in each piece of data synchronization configuration information, and set corresponding synchronization rules for the corresponding data synchronization task according to the data synchronization frequency information included in each piece of data synchronization configuration information.

[0079] In some embodiments, the task execution module 130, when executing any data synchronization task, is specifically used to: obtain the database identifier and data warehouse identifier associated with the data synchronization task, use the database corresponding to the database identifier as the source database and the data warehouse corresponding to the data warehouse identifier as the target data warehouse; capture the change log of the source database in real time through a preset log parser; parse the change log to obtain the change data; perform data conversion and data cleansing on the change data according to business needs to obtain the target data to be synchronized; and synchronize the target data to the target data warehouse in real time.

[0080] In some embodiments, the task execution module 130 performs data conversion and data cleansing on the changed data according to business needs, and when the target data to be synchronized is obtained, it is specifically used to obtain the data processing rules associated with the data synchronization task, and performs data conversion and data cleansing on the changed data according to the data processing rules to obtain the target data to be synchronized.

[0081] In some embodiments, the apparatus further comprises:

[0082] A rule receiving module (not shown in the figure) is used to receive data processing rules configured by users for each data synchronization task according to business requirements; data processing rules include data conversion rules and data cleaning rules;

[0083] The rule storage module (not shown in the figure) is used to store various data processing rules and establish an association relationship between each data processing rule and its corresponding data synchronization task.

[0084] In some embodiments, the device further includes a performance optimization module (not shown in the figure). The performance optimization module is used to optimize the performance of the synchronization process of any data synchronization task according to business requirements and the amount of data to be synchronized when executing the data synchronization task.

[0085] For the specific definition of the data synchronization device, please refer to the definition of the data synchronization method above, which will not be repeated here. Each module in the above-mentioned data synchronization device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0086] The present application also provides a computer device, in some embodiments, its internal structure diagram can be as follows Figure 4As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as data processing rules and data synchronization strategies. The specific stored data can also refer to the definitions in the above method embodiments. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data synchronization method is implemented.

[0087] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0088] This embodiment also provides a computer device. In some embodiments, the computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the data synchronization method provided in any of the above embodiments can be implemented.

[0089] The present application also provides a computer-readable storage medium. In some embodiments, a computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, the steps of the data synchronization method provided in any of the above embodiments are implemented.

[0090] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0091] Those skilled in the art can understand that all or part of the processes in the above method embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink), DRAM (SLDRAM), memory bus (Rambus), direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0092] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0093] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A data synchronization method, characterized in that: The method comprises: generating a data synchronization policy in response to a user's data synchronization configuration operation; Creating multiple data synchronization tasks according to the data synchronization strategy, where different data synchronization tasks are used to synchronize data from different databases to corresponding data warehouses; Execute each data synchronization task according to the synchronization rules corresponding to each of the data synchronization tasks; When executing any of the data synchronization tasks, the synchronization process of the data synchronization task is monitored in real time, and when an abnormality or error is monitored, troubleshooting is performed.

2. The method according to claim 1, characterized in that Generating a data synchronization policy in response to a user's data synchronization configuration operation, including: Receiving a configuration file uploaded by a user through a configuration client, parsing the configuration file, and generating a data synchronization policy according to data synchronization configuration information parsed from the configuration file; Alternatively, data synchronization configuration information transmitted by a user through a preset data synchronization configuration interface is received, and a data synchronization policy is generated according to the received data synchronization configuration information.

3. The method according to claim 1 or 2, characterized in that The data synchronization strategy includes multiple pieces of data synchronization configuration information; each piece of data synchronization configuration information includes a database identifier, a data warehouse identifier, a data table identifier and a field identifier to be synchronized, and data synchronization frequency information; Create multiple data synchronization tasks according to the data synchronization strategy, including: A data synchronization task is created according to the database identifier, data warehouse identifier, data table identifier to be synchronized and field identifier included in each piece of the data synchronization configuration information, and a corresponding synchronization rule is set for the corresponding data synchronization task according to the data synchronization frequency information included in each piece of the data synchronization configuration information.

4. The method according to claim 3, characterized in that When executing any of the data synchronization tasks, obtaining a database identifier and a data warehouse identifier associated with the data synchronization task, taking the database corresponding to the database identifier as a source database and taking the data warehouse corresponding to the data warehouse identifier as a target data warehouse; Capturing the change log of the source database in real time through a preset log parser; Parsing the change log to obtain change data; According to business requirements, the changed data is converted and cleaned to obtain target data to be synchronized; The target data is synchronized to the target data warehouse in real time.

5. The method according to claim 4, characterized in that According to business requirements, the changed data is converted and cleaned to obtain the target data to be synchronized, including: The data processing rules associated with the data synchronization task are obtained, and the changed data is converted and cleaned according to the data processing rules to obtain the target data to be synchronized.

6. The method according to claim 5, characterized in that After creating multiple data synchronization tasks according to the data synchronization strategy, the method further includes: Receive data processing rules configured by the user for each of the data synchronization tasks according to business requirements; the data processing rules include data conversion rules and data cleaning rules; Each of the data processing rules is stored, and an association relationship is established between each of the data processing rules and its corresponding data synchronization task.

7. The method according to claim 1, characterized in that The method further comprises: When executing any of the data synchronization tasks, the synchronization process of the data synchronization task is optimized according to business requirements and the amount of data to be synchronized.

8. A data synchronization device, characterized in that: The device comprises: A policy generation module, used to generate a data synchronization policy in response to a user's data synchronization configuration operation; A task creation module, used to create multiple data synchronization tasks according to the data synchronization strategy, and different data synchronization tasks are used to synchronize data from different databases to corresponding data warehouses; A task execution module, used to execute each data synchronization task according to the synchronization rules corresponding to each of the data synchronization tasks; The task monitoring module is used to monitor the synchronization process of any of the data synchronization tasks in real time when executing the data synchronization tasks, and to perform troubleshooting when anomalies or errors are detected.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images