A method and apparatus for data synchronization

Abstract

This invention relates to the field of Fintech and discloses a method and apparatus for data synchronization. The method includes: monitoring a first operation event targeting metadata of a source data warehouse; determining a second operation event from the first operation event according to preset rules, generating synchronization tasks corresponding to the second operation event; and executing the synchronization tasks to achieve synchronization between the source file system and the destination file system. This invention addresses the causes of changes in source data to achieve more accurate and efficient data synchronization. By accurately, timely, and effectively storing and aggregating changes in source data and then synchronizing them to the destination file system, the time-consuming operations caused by recursively traversing data paths can be avoided. By monitoring and listening to changes in source data, the data comparison overhead is greatly reduced, improving the performance and efficiency of incremental synchronization.

Description

Technical Field

[0001] This invention relates to the field of financial technology (Fintech), and more particularly to a method and apparatus for data synchronization. Background Technology

[0002] With the development of computer technology, more and more technologies (such as distributed architecture, cloud computing or big data) are being applied in the financial field. The traditional financial industry is gradually transforming into financial technology, and big data technology is no exception. However, due to the security and real-time requirements of the financial and payment industries, higher requirements are also being placed on big data technology.

[0003] During cluster data migration, the data and metadata of the source cluster are usually fully synchronized to the destination cluster in one go to ensure that the data of the destination cluster is basically consistent with that of the source cluster. Then, the newly added data of the source cluster is periodically synchronized to the destination cluster in an incremental manner to keep the data of the two clusters dynamically consistent.

[0004] The main function of full synchronization is to significantly reduce the data differences between the source and destination clusters, creating conditions for incremental synchronization to keep the two clusters basically completely consistent.

[0005] Existing cross-cluster data synchronization solutions indirectly determine differences in metadata between the source and destination clusters during incremental synchronization by comparing data paths, and then synchronize metadata based on these differences. Specifically, incremental synchronization typically involves task configuration and data comparison at the database level (or table and partition level). Data comparison uses path data, including file size, number of directories, and number of files. The comparison process generally involves the following scenarios:

[0006] (1) If the data in the source cluster and the destination cluster are inconsistent, and the data in the destination cluster is more than that in the source cluster, the excess data in the destination cluster will be deleted.

[0007] (2) If the data in the source cluster and the destination cluster are inconsistent, and the data in the destination cluster is less than that in the source cluster, the corresponding data in the source cluster will be copied.

[0008] (3) If the data at both ends are completely consistent, the comparison is successful and no further processing is required.

[0009] The above approach has several drawbacks. First, it requires traversing and comparing all data paths in the database table, which is inefficient. Second, comparing data paths, which serves as an indirect basis for judging the differences between the target cluster and the source cluster's metadata, has a high probability of misjudgment and increases the overhead of repeatedly synchronizing data and metadata in the target cluster. Summary of the Invention

[0010] This application provides a method and apparatus for data synchronization, which solves the problem of how to perform data synchronization accurately and efficiently.

[0011] In a first aspect, embodiments of this application provide a data synchronization method applicable to a distributed file system with a data warehouse; the method includes:

[0012] Listen for the first operation event on the metadata of the source data warehouse;

[0013] A second operation event is determined from the first operation event according to a preset rule. The second operation event refers to an operation event that is related to changes in the source data stored in the source file system.

[0014] Generate the corresponding synchronization tasks for the second operation event;

[0015] The synchronization tasks are executed to achieve synchronization between the source file system and the destination file system.

[0016] The above solution differs from existing technologies that use data path comparison as an indirect basis for determining differences in metadata between the destination and source file systems. Instead, it directly monitors the first operation event on the metadata of the source data warehouse, starting from the cause of changes in the source data to achieve more accurate and efficient data synchronization. It accurately, promptly, and effectively stores and aggregates changes in source data before synchronizing them to the destination file system. This avoids the time-consuming operations caused by recursively traversing data paths. By monitoring and listening to changes in source data, it significantly reduces data comparison overhead and improves the performance and efficiency of incremental synchronization.

[0017] Optionally, each synchronization task corresponds to one operation event;

[0018] Executing the aforementioned synchronization tasks to achieve synchronization between the source file system and the destination file system includes:

[0019] If the synchronization task is for the metadata of the source data warehouse, then the metadata of the source data warehouse is synchronized to the destination data warehouse by executing the synchronization task.

[0020] If the synchronization task is for source data, then after executing the synchronization task, the source data corresponding to the synchronization task is obtained from the source file system and sent to the destination file system.

[0021] The above scheme improves the efficiency of data synchronization by different methods depending on whether the synchronization task is generated for the source data warehouse metadata or the source data. For metadata, the synchronization task is executed directly to synchronize the metadata of the destination data warehouse. For source data, after executing the synchronization task, the source data corresponding to the synchronization task is obtained from the source file system and sent to the destination file system.

[0022] Optionally, generating the synchronization tasks corresponding to the second operation event includes:

[0023] Determine a third operation event targeting the same operation object from the second operation event;

[0024] For each operation object, the operation time and operation type in the third operation event of the operation object are processed to obtain the fourth operation event of the operation object;

[0025] Generate the corresponding synchronous tasks for the fourth operation event of each operation object.

[0026] The above solution categorizes and merges operation events based on the operation time and operation type of the operation object, thereby reducing the number of synchronous tasks and avoiding the risk and hidden danger of event loss during data flow.

[0027] Optionally, based on the operation time and operation type in the third operation event of the operation object, a fourth operation event of the operation object is obtained, including:

[0028] For deletion-type operation events, delete the third operation event whose operation time precedes the deletion-type operation event; or

[0029] For operation events of the "create" or "modify" type, aggregate and merge various third-party operation events through parameter aggregation; or

[0030] For insertion type operation events, if the insertion attribute is overwrite, then the third operation event whose deletion operation time is before the latest insertion event is deleted; if the insertion attribute is non-overwrite, then all third operation events are merged.

[0031] The above solution reduces the number of synchronization tasks by removing invalid operation events (such as deletion events), merging operation events according to parameters (such as creation or modification events), and merging operation events according to whether they are overwritten (such as insertion events), thereby reducing the frequency of metadata and data operations. This minimizes the number of synchronization tasks and avoids the risk and potential problems of event loss during data flow.

[0032] Optionally, determining the second operation event from the first operation event includes:

[0033] A second operation event containing a defined metadata operation primitive is determined from the first operation event, wherein the defined metadata operation primitive includes at least one of the following:

[0034] Create a new partition, modify a partition, delete a partition, create a new table, modify a table, delete a table, and insert data.

[0035] The above solution standardizes operation events through metadata operation primitives, making subsequent synchronization tasks more efficient.

[0036] Optionally, listen for the first operation event on the metadata of the source data warehouse, including:

[0037] The operation events of the metadata of the source data warehouse are batched from the monitored data to obtain the first operation event of each batch.

[0038] The above scheme divides operation events into batches, thereby improving the efficiency of data synchronization.

[0039] Optionally, after executing the synchronization tasks to synchronize the source file system and the destination file system, the method further includes:

[0040] The synchronization status of the source file system and the destination file system is determined by path comparison.

[0041] The above solution improves the accuracy of synchronization by verifying the data through path comparison after synchronization.

[0042] Secondly, embodiments of this application provide a data synchronization apparatus, the apparatus comprising:

[0043] The listening module is used to listen for the first operation event on the metadata of the source data warehouse;

[0044] The processing module is used to determine a second operation event from the first operation event according to a preset rule. The second operation event refers to an operation event that is related to changes in the source data stored in the source file system.

[0045] Generate the corresponding synchronization tasks for the second operation event;

[0046] The synchronization tasks are executed to achieve synchronization between the source file system and the destination file system.

[0047] Optionally, each synchronization task corresponds to one operation event;

[0048] The processing module specifically includes:

[0049] If the synchronization task is for the metadata of the source data warehouse, then the metadata of the source data warehouse is synchronized to the destination data warehouse by executing the synchronization task.

[0050] If the synchronization task is for source data, then after executing the synchronization task, the source data corresponding to the synchronization task is obtained from the source file system and sent to the destination file system.

[0051] Optionally, the processing module specifically includes:

[0052] Determine a third operation event targeting the same operation object from the second operation event;

[0053] For each operation object, the operation time and operation type in the third operation event of the operation object are processed to obtain the fourth operation event of the operation object;

[0054] Generate the corresponding synchronous tasks for the fourth operation event of each operation object.

[0055] Optionally, the processing module specifically includes:

[0056] For deletion-type operation events, delete the third operation event whose operation time precedes the deletion-type operation event; or

[0057] For operation events of the "create" or "modify" type, aggregate and merge various third-party operation events through parameter aggregation; or

[0058] For insertion type operation events, if the insertion attribute is overwrite, then the third operation event whose deletion operation time is before the latest insertion event is deleted; if the insertion attribute is non-overwrite, then all third operation events are merged.

[0059] Optionally, the processing module specifically includes:

[0060] A second operation event containing a defined metadata operation primitive is determined from the first operation event, wherein the defined metadata operation primitive includes at least one of the following:

[0061] Create a new partition, modify a partition, delete a partition, create a new table, modify a table, delete a table, and insert data.

[0062] Optionally, the monitoring module specifically includes:

[0063] The operation events of the metadata of the source data warehouse are batched from the monitored data to obtain the first operation event of each batch.

[0064] Optionally, after executing the synchronization tasks to synchronize the source file system and the destination file system, the processing module further includes:

[0065] The synchronization status of the source file system and the destination file system is determined by path comparison.

[0066] Accordingly, embodiments of the present invention also provide a computing device, including:

[0067] Memory, used to store program instructions;

[0068] The processor is used to call program instructions stored in the memory and execute the above-mentioned data synchronization method according to the obtained program.

[0069] Accordingly, embodiments of the present invention also provide a computer-readable non-volatile storage medium, including computer-readable instructions, which, when read and executed by a computer, cause the computer to perform the aforementioned data synchronization method. Attached Figure Description

[0070] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0071] Figure 1 A schematic diagram illustrating data synchronization provided in an embodiment of the present invention;

[0072] Figure 2 This invention provides a system framework for a data synchronization method according to an embodiment of the invention.

[0073] Figure 3 A flowchart illustrating a data synchronization method provided in an embodiment of the present invention;

[0074] Figure 4 A flowchart illustrating a data synchronization method provided in an embodiment of the present invention;

[0075] Figure 5 This is a schematic diagram of a data synchronization device provided in an embodiment of the present invention. Detailed Implementation

[0076] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0077] First, some of the terms used in this application will be explained to enable those skilled in the art to understand them.

[0078] Transport: An accurate, stable, and efficient data synchronization tool based on the MapReduce computing framework, supporting TB and PB-level data synchronization for offline data platforms.

[0079] HMS (Hive Metastore): HMS provides services for clients to access all metadata in Hive, such as tables and partitions.

[0080] Event Listener: HMS provides an event subscription-based notification mechanism for subscribing to and processing event notification messages. This mechanism is called Event Listener.

[0081] Full synchronization: Historical data is copied at regular intervals to ensure that the data in the destination cluster is consistent with that in the source cluster.

[0082] Incremental synchronization: Relates newly added data to a target cluster based on a specific update time or checkpoint, ensuring that the newly added data in the target cluster remains consistent with that in the source cluster.

[0083] Before introducing the embodiments of the present invention, the prior art and basic principles are described below to enable a better understanding of the present invention.

[0084] It should be noted that this invention is primarily based on the Hadoop Distributed File System (HDFS).

[0085] HDFS is designed as a distributed file system suitable for running on commodity hardware. It shares many similarities with existing distributed file systems. However, its differences from other distributed file systems are also significant. HDFS is a highly fault-tolerant system, suitable for deployment on inexpensive machines. HDFS provides high-throughput data access, making it ideal for applications dealing with large-scale datasets. HDFS enables streaming reads of file system data.

[0086] Since querying files on HDFS requires writing a lot of code manually, Hive was created to solve this problem. Hive is based on a unified query and analysis layer and uses SQL statements to query, count, and analyze data on HDFS.

[0087] Furthermore, such as Figure 1 As shown, the relationship between Hive and Hadoop (MapReduce, HDFS) can be clearly seen.

[0088] Specifically, Hive is the top layer, the client layer or job submission layer. MapReduce is the middle layer, the computation layer. HDFS is the bottom layer, the storage layer. Hive stores metadata in the database. Metadata in Hive includes table names, table partitions and their attributes, and the directory where the table data is located.

[0089] When a user sends an SQL message, Hive processes it into a MapReduce function, then submits the task to Hadoop, and finally records the metadata in the database.

[0090] Based on this, existing data synchronization schemes based on HDFS data comparison indirectly determine the differences in related metadata between the source file system and the destination file system Hive table by comparing the corresponding HDFS data paths during incremental synchronization, and then synchronize the metadata based on the differences.

[0091] Furthermore, during the data migration process, the source data (HDFS data) and metadata of the source file system are generally fully synchronized to the destination file system at once to ensure that the destination file system is basically consistent with the source file system. Then, the newly added data of the source file system is periodically synchronized to the destination file system in an incremental synchronization manner to keep the data at both ends dynamically consistent.

[0092] The main purpose of full synchronization is to significantly reduce the data differences between the source and destination file systems, creating conditions for incremental synchronization to keep both ends essentially identical. Incremental synchronization is generally configured and data compared at the database level (or table level and partition level). The data comparison uses the Hive table to correspond to the HDFS path data, including file size, number of directories, and number of files.

[0093] However, existing technologies have several drawbacks. First, the incremental synchronization task configuration library-level task needs to traverse and compare all HDFS data paths in the library table every time it starts, which is inefficient. Second, the comparison of HDFS data paths, which serves as an indirect basis for judging the differences between the target file system and the source file system Hive table metadata, has a high probability of misjudgment, increasing the overhead of repeatedly synchronizing data and metadata in the target file system.

[0094] Based on this, embodiments of the present invention provide a data synchronization method, which can be applied to, for example... Figure 2 The system architecture shown includes a source file system 100, a destination file system 200, and a synchronization server 300.

[0095] Among them, the synchronization server 300 is used to listen for the first operation event on the metadata of the source data warehouse.

[0096] The second operation event is determined from the first operation event, and the corresponding synchronization tasks are generated. The synchronization tasks are executed to achieve synchronization between the source file system 100 and the destination file system 200.

[0097] It should be noted that, Figure 2This is merely an example of a system architecture in an embodiment of this application, and this application does not impose any specific limitations on it. In fact, the synchronization server 300 can be part of the source file system 100, and can be implemented through various functions set by the source file system 100 itself.

[0098] Based on the system architecture shown above Figure 3 This is a flowchart illustrating a data synchronization method provided in an embodiment of the present invention, as shown below. Figure 3 As shown, the method includes:

[0099] Step 301: Listen for the first operation event on the metadata of the source data warehouse.

[0100] Step 302: Determine the second operation event from the first operation event according to preset rules.

[0101] It should be noted that the second operation event refers to an operation event that is associated with changes to the source data stored in the source file system.

[0102] Step 303: Generate the corresponding synchronization tasks for the second operation event.

[0103] Step 304: Execute each synchronization task to achieve synchronization between the source file system and the destination file system.

[0104] The above solution differs from existing technologies that use data path comparison as an indirect basis for determining differences in metadata between the destination and source file systems. Instead, it directly monitors the first operation event on the metadata of the source data warehouse, starting from the cause of changes in the source data to achieve more accurate and efficient data synchronization. It accurately, promptly, and effectively stores and aggregates changes in source data before synchronizing them to the destination file system. This avoids the time-consuming operations caused by recursively traversing data paths. By monitoring and listening to changes in source data, it significantly reduces data comparison overhead and improves the performance and efficiency of incremental synchronization.

[0105] Specifically, in step 201, the operation events of the monitored metadata of the source data warehouse are batched to obtain the first operation event of each batch.

[0106] It should be noted that the first operation event in each batch corresponds to a batch number.

[0107] In one possible implementation, the operation events are batched by dividing the time, for example, multiple operation events every 5 minutes are grouped into a batch.

[0108] In step 301 of this embodiment, for example, a second operation event is first determined by enabling a preset rule. The preset rule can be the HMS configuration hive.metastore.event.listeners parameter, that is, the HMS metadata operation event is listened to through the HMS configuration hive.metastore.event.listeners parameter.

[0109] It should be noted that the operation events include commonly used operations such as creating a table (create table), clearing a table (drop table), creating a database (create database), and clearing a database (drop database).

[0110] Furthermore, HMS operation event listening can monitor and listen for changes in source HMS metadata, significantly reducing data comparison overhead and improving the performance and efficiency of incremental synchronization. However, since HMS is the carrier of data warehouse metadata, big data platforms generate a large number of HMS operations daily, reaching millions to tens of millions; therefore, the following drawbacks arise:

[0111] On the one hand, replaying large-scale operation events generates a large number of synchronization tasks; on the other hand, a large number of HMS operation events often contain redundant or invalid operation events, and there is a risk of loss of a large number of HMS operation events during data flow. This risk is relatively hidden and not easy to detect and monitor, posing a significant hidden danger to daily production.

[0112] Based on this, the second operation event is determined from the first operation event through step 302, that is, the first operation event is preprocessed to overcome the above-mentioned defects.

[0113] Specifically, a second operation event containing the set metadata operation primitive is determined from the first operation event. The set metadata operation primitive includes at least one of the following:

[0114] Create a new partition, modify a partition, delete a partition, create a new table, modify a table, delete a table, and insert data.

[0115] For example, seven metadata operation primitives are defined in the Event Playback module. These metadata operation primitives correspond one-to-one with the actual HMS operation events. Table 1 shows the seven basic metadata operation primitives defined.

[0116] It can be seen that the seven metadata operation primitives are ADD_PARTITION, ALTER_PARTITION, DROP_PARTITION, CREATE_TABLE, INSERT, DROP_TABLE, and ALTER_TABLE.

[0117] It should be noted that when inserting data using INSERT, you can insert data into either a table or a partition.

[0118] Table 1

[0119]

[0120] It should be noted that the above seven metadata operation primitives are only examples, and this embodiment does not impose any specific limitations on them.

[0121] Furthermore, in step 303, the process flow is as follows: Figure 4 As shown, the details are as follows:

[0122] Step 401: Determine the third operation event for the same operation object from the second operation event.

[0123] Step 402: For each operation object, process it according to the operation time and operation type in the third operation event of the operation object to obtain the fourth operation event of the operation object.

[0124] Step 403: Generate the corresponding synchronous tasks for the fourth operation event of each operation object.

[0125] Specifically, the operation events are preprocessed by merging and sorting according to <table name, partition name, operation event list>, where the table name and partition name are used as key-value indexes, and the operation event list is sorted according to time information to maintain order.

[0126] It should be noted that in step 401, the same operation object can be the same table or the same partition, and this application does not make specific limitations on this.

[0127] Specifically, in step 402, for deletion type operation events, the third operation event whose operation time is earlier than the deletion type operation event is deleted.

[0128] For example, deletion-type operation events generally fall into the following two categories:

[0129] 1. Partitioned Table: For events of the DROP type, the search is performed using the table name index and partition name index. If the same event is found, such as two identical DROP_PARTTION A events, the events are merged, that is, one of the two DROP_PARTTION A events is deleted, and only one DROP_PARTTION A event is kept.

[0130] Furthermore, if no matching event is found, all events with the same table name and partition name that precede the DROP event are deleted, and then the DROP event is written. For example, if the operation event stream is ALTER_PARTITION A, CREATE_PARTITION B, DROP_PARTITION A, and if listening to DROP_PARTITION A reveals an ALTER_PARTITION A with the same partition name preceding the DROP_PARTITION A operation event, then ALTER_PARTITION A is deleted.

[0131] 2. Non-partitioned tables: For DROP type events, the search is performed according to the table name index. If the same event is found, the events are merged. If the same event is found, such as three identical DROP_TABLE C, the events are merged, which means that two of the three DROP_TABLE C are deleted, and only one DROP_TABLE C is kept.

[0132] If no matching event is found, all events with the same table name that precede the DROP event will be deleted, and then the DROP event will be written. For example, if the operation event stream is ALTER _TABLE C, CREATE _TABLE D, DROP _TABLE C, and if DROP _TABLE C can be found to have the same table name as ALTER _TABLE C before the DROP _TABLE C operation event, then ALTER _TABLE C will be deleted.

[0133] Specifically, for operation events of the new or modified type, the various third operation events are aggregated and merged through parameter aggregation.

[0134] For example, for CREATE and ALTER type events, events of the same type can be merged by aggregating them using parameters based on table name and partition name.

[0135] It should be noted that the table types include partitioned tables and non-partitioned tables.

[0136] Specifically, the HiveMetastoreEventFactory class is implemented in the parameter aggregation module to deduplicatize and merge the parameters of various CREATE and ALTER operations, ultimately forming several large operation events.

[0137] For example, the operation event stream is as follows: the first ALTER_TABLE A modifies parameter a to b, the second ALTER_TABLE A modifies parameter b to c, the third ALTER_TABLE A modifies parameter c to d, and the fourth ALTER_TABLE A modifies parameter c to d. First, the third ALTER_TABLE A and the fourth ALTER_TABLE A are duplicates, so one of them is deleted. Then, the parameters are merged, and finally, the operation event ALTER_TABLE A modifies parameter a to d is formed.

[0138] Furthermore, for insertion type operation events, if the insertion attribute is overwrite, then the deletion operation time is the third operation event that precedes the latest insertion event; if the insertion attribute is non-overwrite, then all third operation events are merged.

[0139] For example, INSERT type events are merged based on table name and partition name. The INSERT type is divided into partitioned tables and non-partitioned tables, and further divided into overwrite writes and non-overwrite writes.

[0140] Specifically, for overwrite writes, INSERT operation events are merged according to table name and partition name. For example, the operation event stream is as follows: the first INSERT_TABLE A inserts parameter a, the second INSERT_TABLE A inserts parameter b, and the third INSERT_TABLE A inserts parameter c. Finally, the parameters are merged to form the operation event INSERT_TABLEA inserts parameter c.

[0141] Furthermore, for non-overwrite writes, deduplication and recording of all table names or partition names inserted into the INSERT operation are performed, ultimately resulting in several large operation events.

[0142] For example, the operation event stream is as follows: the first INSERT_TABLE E inserts parameter e, the second INSERT_TABLE E inserts parameter f, and the third INSERT_TABLE F inserts parameter g. Finally, the parameters are merged to form operation events INSERT_TABLE E inserting parameters e and f, and operation event INSERT_TABLE F inserting parameter g.

[0143] The above solution reduces the number of synchronization tasks by removing invalid operation events (such as deletion events), merging operation events according to parameters (such as creation or modification events), and merging operation events according to whether they are overwritten (such as insertion events), thereby reducing the frequency of metadata and data operations. This minimizes the number of synchronization tasks and avoids the risk and potential problems of event loss during data flow.

[0144] Furthermore, after the above process is completed, an optimized sequence of metadata operation primitives is formed, and then a corresponding task to be synchronized is generated for each operation event in the metadata operation sequence primitives.

[0145] Furthermore, in step 204 of this embodiment, each synchronization task corresponds to an operation event;

[0146] Specifically, if the synchronization task is for the metadata of the source data warehouse, then the metadata of the source data warehouse is synchronized to the destination data warehouse by executing the synchronization task.

[0147] If the synchronization task is for source data, then after executing the synchronization task, the source data corresponding to the synchronization task is obtained from the source file system and sent to the destination file system.

[0148] For example, for a drop-type operation event, the operation event is DROP_TABLE C. Since it targets the metadata of the source data warehouse, DROP_TABLE C can be executed directly on the destination file system.

[0149] For example, for an INSERT type operation event, such as INSERT _TABLE B, table B is retrieved from the source file system and then sent to the destination file system for storage.

[0150] The above solution improves data synchronization efficiency by executing synchronization tasks directly on the target data warehouse's metadata, depending on whether the synchronization task targets the source data warehouse's metadata or the source data itself.

[0151] In this embodiment, the metadata operation primitive sequence serves as the input parameters for job startup configuration and synchronization tasks. In one possible implementation, a SyncConf structure is constructed using the input parameters of each synchronization task. Then, a synchronization batch number is generated, and the batch number and the SyncConf structure are submitted to the GenerateTasks() function to automatically generate synchronization tasks. Finally, the generated synchronization tasks are periodically executed by the scheduler TaskExecute CrontabJob of the Transport scheduling module.

[0152] In this embodiment of the application, after step 204, it is determined whether the source file system and the destination file system have been synchronized by path comparison.

[0153] To better ensure the accuracy of data synchronization, the above scheme verifies the data by comparing paths after the synchronization task is completed, thus reducing the risk of errors during the data synchronization process.

[0154] Based on the same inventive concept Figure 5 An exemplary embodiment of the present invention provides a data synchronization apparatus, which can be a flow chart of a data synchronization method.

[0155] The device includes:

[0156] Listening module 501 is used to listen for the first operation event on the metadata of the source data warehouse;

[0157] Processing module 502 is used to determine a second operation event from the first operation event according to a preset rule, wherein the second operation event refers to an operation event that is related to changes in the source data stored in the source file system;

[0158] Generate the corresponding synchronization tasks for the second operation event;

[0159] The synchronization tasks are executed to achieve synchronization between the source file system and the destination file system.

[0160] Optionally, each synchronization task corresponds to one operation event;

[0161] The processing module 502 is specifically used for:

[0162] If the synchronization task is for the metadata of the source data warehouse, then the metadata of the source data warehouse is synchronized to the destination data warehouse by executing the synchronization task.

[0163] If the synchronization task is for source data, then after executing the synchronization task, the source data corresponding to the synchronization task is obtained from the source file system and sent to the destination file system.

[0164] Optionally, the processing module 502 is specifically used for:

[0165] Determine a third operation event targeting the same operation object from the second operation event;

[0166] For each operation object, the operation time and operation type in the third operation event of the operation object are processed to obtain the fourth operation event of the operation object;

[0167] Generate the corresponding synchronous tasks for the fourth operation event of each operation object.

[0168] Optionally, the processing module 502 is specifically used for:

[0169] For deletion-type operation events, delete the third operation event whose operation time precedes the deletion-type operation event; or

[0170] For operation events of the "create" or "modify" type, aggregate and merge various third-party operation events through parameter aggregation; or

[0171] For insertion type operation events, if the insertion attribute is overwrite, then the third operation event whose deletion operation time is before the latest insertion event is deleted; if the insertion attribute is non-overwrite, then all third operation events are merged.

[0172] Optionally, the processing module 502 is specifically used for:

[0173] A second operation event containing a defined metadata operation primitive is determined from the first operation event, wherein the defined metadata operation primitive includes at least one of the following:

[0174] Create a new partition, modify a partition, delete a partition, create a new table, modify a table, delete a table, and insert data.

[0175] Optionally, the monitoring module 501 is specifically used for:

[0176] The operation events of the metadata of the source data warehouse are batched from the monitored data to obtain the first operation event of each batch.

[0177] Optionally, after executing the synchronization tasks to achieve synchronization between the source file system and the destination file system, the processing module 502 is further configured to:

[0178] The synchronization status of the source file system and the destination file system is determined by path comparison.

[0179] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0180] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1The function specified in one or more boxes.

[0181] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0182] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0183] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for data synchronization, characterized in that, Suitable for distributed file systems with data warehouses; The distributed file system stores data, and the data warehouse stores metadata; the method includes: Listen for the first operation event on the metadata of the source data warehouse, which includes the table name, the table partitions and their attributes, and the directory where the table data is located; A second operation event is determined from the first operation event according to a preset rule. The second operation event refers to an operation event that is related to changes in the source data stored in the source file system. Generate each synchronization task corresponding to the second operation event, and each synchronization task corresponds to one operation event; If the synchronization task is for the metadata of the source data warehouse, then the metadata of the source data warehouse is synchronized to the destination data warehouse by executing the synchronization task. If the synchronization task is for source data, after executing the synchronization task, the source data corresponding to the synchronization task is obtained from the source file system and sent to the destination file system; whether the source file system and the destination file system have completed synchronization is determined by path comparison, and the path comparison is performed using path data, including file size, number of directories and number of files; The second operation event is determined from the first operation event, including: A second operation event containing a defined metadata operation primitive is determined from the first operation event, wherein the defined metadata operation primitive includes at least one of the following: Create a new partition, modify a partition, delete a partition, create a new table, modify a table, delete a table, and insert data.

2. The method as described in claim 1, characterized in that, Generate the synchronization tasks corresponding to the second operation event, including: Determine a third operation event targeting the same operation object from the second operation event; For each operation object, the operation time and operation type in the third operation event of the operation object are processed to obtain the fourth operation event of the operation object; Generate the corresponding synchronous tasks for the fourth operation event of each operation object.

3. The method as described in claim 2, characterized in that, Based on the operation time and operation type in the third operation event of the operation object, the fourth operation event of the operation object is obtained, including: For deletion-type operation events, delete the third operation event whose operation time precedes the deletion-type operation event; or For operation events of the "create" or "modify" type, aggregate and merge various third-party operation events through parameter aggregation; or For insertion type operation events, if the insertion attribute is overwrite, then the third operation event whose deletion operation time is before the latest insertion event is deleted; if the insertion attribute is non-overwrite, then all third operation events are merged.

4. The method as described in claim 1, characterized in that, Listen for the first operation event on the metadata of the source data warehouse, including: The operation events of the metadata of the source data warehouse are batched from the monitored data to obtain the first operation event of each batch.

5. A data synchronization device, characterized in that, Suitable for distributed file systems with data warehouses; The distributed file system stores data, and the data warehouse stores metadata; including: The listening module is used to listen for the first operation event on the metadata of the source data warehouse, which includes the table name, the table partitions and their attributes, and the directory where the table data is located. The processing module is used to determine a second operation event from the first operation event, wherein the second operation event refers to an operation event associated with changes in source data stored in the source file system; Generate each synchronization task corresponding to the second operation event, and each synchronization task corresponds to one operation event; If the synchronization task is for the metadata of the source data warehouse, then the metadata of the source data warehouse is synchronized to the destination data warehouse by executing the synchronization task. If the synchronization task is for source data, then after executing the synchronization task, the source data corresponding to the synchronization task is obtained from the source file system and sent to the destination file system; The path comparison method determines whether the source file system and the destination file system have completed synchronization. The path comparison uses path data, including file size, number of directories and number of files. The second operation event is determined from the first operation event, including: A second operation event containing a defined metadata operation primitive is determined from the first operation event, wherein the defined metadata operation primitive includes at least one of the following: Create a new partition, modify a partition, delete a partition, create a new table, modify a table, delete a table, and insert data.

6. A computing device, characterized in that, include: Memory, used to store program instructions; A processor is configured to invoke program instructions stored in the memory and execute the method according to any one of claims 1 to 4.

7. A computer-readable non-volatile storage medium, characterized in that, Includes computer-readable instructions that, when read and executed by a computer, cause the computer to perform the method as described in any one of claims 1 to 4.

Images