Data table synchronization method and device, computer device, and storage medium

Abstract

The present disclosure relates to a data table synchronization method and device, computer equipment and a storage medium. The method comprises: obtaining component version information of a source cluster and a target cluster; in response to the existence of version differences between the component version information of the source cluster and the component version information of the target cluster, converting metadata of a data table in the source cluster according to a difference point corresponding to the version differences; interacting with the target cluster, and writing the converted metadata into a metadata database of the target cluster; determining a storage path of main data of the target cluster according to the version differences, and synchronizing the main data of the source cluster and the target cluster by using the storage path. The present method can solve the problem that current data processing tools cannot copy or are incompatible between different Hive table versions.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to a data table synchronization method, apparatus, computer equipment, and storage medium. Background Technology

[0002] With the development of big data, the amount of data generated by various industries is growing exponentially. Large internet companies and financial institutions, in particular, need to process enormous amounts of data daily, often measured in terabytes (TB). Faced with such massive volumes, conventional data processing software struggles to meet the demands. Therefore, data processing tools are typically used to handle these large datasets.

[0003] Currently, data processing tools such as Sqoop, BDR, and Datax, when synchronizing data for complex business scenarios, typically only copy the original table structure, and not completely replicate it. For example, in different versions of clusters, the data storage location recorded in the table's metadata changes before and after copying. Furthermore, data processing tools like Sqoop, BDR, and Datax may encounter situations where copying is impossible or incompatible between different versions of Hive tables. Summary of the Invention

[0004] Therefore, it is necessary to provide a method, device, computer equipment, or storage medium that can solve the problem of data table synchronization between different Hive data table versions where current data processing tools cannot copy or are incompatible.

[0005] Firstly, this disclosure provides a method for synchronizing data tables. The method includes:

[0006] Obtain component version information for the source and target clusters;

[0007] In response to the version difference between the component version information of the source cluster and the component version information of the target cluster, the metadata of the data table in the source cluster is copied and transformed according to the difference points corresponding to the version difference.

[0008] Interact with the target cluster and write the transformed metadata into the target cluster's metadata database;

[0009] The storage path of the master data in the target cluster is determined based on the version differences, and the master data of the data tables in the source cluster and the target cluster is synchronized using the storage path.

[0010] In one embodiment, the method further includes:

[0011] If an exception occurs during the synchronization of master data, causing the synchronization to fail, the corresponding synchronization task for that master data will be added to a pre-created synchronization queue.

[0012] When the pre-set retry conditions are met, the master data of the source cluster and the target cluster that failed to synchronize in the synchronization queue are synchronized again.

[0013] In one embodiment, the method further includes:

[0014] In response to receiving a high-speed synchronization command, the storage path of the master data of the target cluster is determined solely based on the version difference, and the master data of the source cluster and the target cluster is synchronized using the storage path. The high-speed synchronization command indicates that metadata comparison is not required.

[0015] During the synchronization of multiple data tables in the source cluster and the target cluster, in response to receiving a failure log, the data table corresponding to the failure log is identified as having failed to synchronize, wherein the failure log indicates that the synchronization process of synchronizing the data table has terminated or failed.

[0016] Resume synchronization from the data table that was interrupted or failed, using breakpoint resume.

[0017] In one embodiment, synchronizing the master data of the source cluster and the target cluster using the storage path includes:

[0018] The number of parallel synchronization processes is determined based on the cluster performance and resources of the source or target cluster.

[0019] The source cluster and the target cluster master data are synchronized in parallel using the number of processes and the storage path.

[0020] In one embodiment, before converting the metadata of the data tables in the source cluster based on the difference points corresponding to the version differences, the method further includes:

[0021] In response to receiving modified parameters, the metadata of the data tables in the source cluster is modified according to the modified parameters.

[0022] In one embodiment, the method further includes:

[0023] During the synchronization of master data, snapshots are created in both the source and target clusters.

[0024] In response to receiving the synchronization log, the snapshot in the source cluster is deleted, and the consistency check of the master data in the target cluster is performed, wherein the synchronization log indicates that the synchronization is completed;

[0025] In response to receiving a kill command, the system retrieves the process corresponding to the synchronization process, terminates the process according to the kill command, and stops synchronization.

[0026] Secondly, this disclosure also provides a data table synchronization device. The device includes:

[0027] The information acquisition module is used to acquire component version information of the source cluster and the target cluster;

[0028] The data conversion module is used to copy and convert the metadata of the data tables in the source cluster according to the difference points corresponding to the version differences when there are version differences between the component version information of the source cluster and the component version information of the target cluster.

[0029] The data interaction module is used to interact with the target cluster and write the converted metadata into the target cluster's metadata database.

[0030] The data synchronization module is used to determine the storage path of the master data of the target cluster based on the version difference, and use the storage path to synchronize the master data of the data tables in the source cluster and the target cluster.

[0031] Thirdly, this disclosure also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of any method embodiment.

[0032] Fourthly, this disclosure also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of any method embodiment.

[0033] Fifthly, this disclosure also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the steps of any of the above-described method embodiments.

[0034] In the above embodiments, component version information of the source and target clusters is obtained. If there are version differences in the component version information, the metadata is copied and transformed. Then, the transformed metadata is written to the metadata database of the target cluster. Adjustments are made according to the metadata differences, so that data synchronization can still be performed between different component version information, eliminating the differences between different versions. In addition, the storage path of the master data of the target cluster is determined according to the version differences. The master data of the data tables in the source and target clusters is synchronized using the storage path. When there are differences in component version information between the source and target clusters, master data synchronization can be completed by transforming the master data storage path between different versions, ensuring that the type of the data table to be synchronized (the data table in the source cluster) does not change. In the case of differences in component version information, by using two different processing methods for metadata and master data, the problem of data tables being unable to be copied or incompatible between different Hive versions in current data processing tools can be solved. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the specific embodiments of this disclosure or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0036] Figure 1 This is a schematic diagram illustrating the application environment of a data table synchronization method in one embodiment;

[0037] Figure 2 This is a flowchart illustrating a data table synchronization method in one embodiment;

[0038] Figure 3 This is a flowchart illustrating the retry step in one embodiment;

[0039] Figure 4 This is a flowchart illustrating the steps of high-speed synchronization and breakpoint resume in one embodiment.

[0040] Figure 5 This is a flowchart illustrating steps S208 and S402 in one embodiment;

[0041] Figure 6 This is a flowchart illustrating the snapshot synchronization, consistency verification, and detection steps in one embodiment;

[0042] Figure 7 This is a flowchart illustrating the data table synchronization method in another embodiment;

[0043] Figure 8This is a schematic block diagram of the data table synchronization device in one embodiment;

[0044] Figure 9 This is a schematic diagram of the internal structure of a computer device in one embodiment. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this disclosure.

[0046] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings herein are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0047] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0048] As described in the background section, current data processing software includes tools such as Sqoop, which is based on traditional databases for cluster data synchronization. Sqoop is an Apache tool used to transfer data between Hadoop and relational databases. Its application is limited; it can only import data from relational databases such as MySQL, Oracle, and Postgres into HDFS, and vice versa. There are also data synchronization software programs specifically designed for big data cluster applications: Cloudera's BDR and Alibaba's Datax. BDR is an enterprise-level data synchronization tool provided by Cloudera. Integrated into Cloudera Manager, it allows users to perform data synchronization and backup operations through a page-based configuration interface. This software can synchronize Hive tables and HDFS data files between different CDH (Cloudera Distribution Hadoop) clusters and also exhibits good CDH version compatibility. Datax is an open-source heterogeneous data source offline synchronization tool from Alibaba, supporting most types of data sources. Furthermore, this software is easy to use, stable, and effectively guarantees data quality, making it widely used in China.

[0049] Currently, Hive is the most common storage solution for big data. Hive is a data warehouse architecture built on Hadoop, and common data loading, extraction, and transformation can all be implemented within Hive. Because its HQL syntax is relatively easy for users familiar with MySQL to understand, the learning curve is low. Furthermore, it is suitable for analyzing large volumes of data, making it widely used. Therefore, storing and backing up data tables in Hive has become a daily challenge for enterprises with large amounts of data.

[0050] Database table replication and backup are frequently used in production environments. When it comes to copying and backing up database tables, BDR (Browser Replication) can change the database name but cannot rename the tables. BDR cannot guarantee the consistency of table attributes (internal / external tables) when synchronizing across Hive versions. For example, when synchronizing from CDH6 to CDP7 (Cloudera Data Platform), all tables will become external tables, and the storage location will also differ between different version clusters. Datax allows manual specification of the data synchronization location, but requires modification of the configuration file each time, which is inconvenient for users. Datax focuses on higher-level data synchronization, requiring manual table creation in the Hive database before synchronization, and cannot automatically create tables, making the process cumbersome. When replicating across different version clusters, the source cluster operates on the data during the replication process, making it impossible to guarantee that the target data will be consistent with the original cluster data after replication. Most existing data table synchronization solutions are general data synchronization solutions, neglecting table synchronization and failing to guarantee consistency of basic table information.

[0051] Therefore, to solve the above problems, this disclosure provides a data table synchronization method that can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with both source cluster 104 and target cluster 106 via a network. Terminal 102 obtains component version information from both source cluster 104 and target cluster 106. Terminal 102 determines whether there are differences between the component version information of source cluster 104 and target cluster 106. In response to the discrepancy, terminal 102 copies and transforms the metadata of the data tables in source cluster 104 based on the differences. Terminal 102 interacts with target cluster 106, writing the transformed metadata into the metadata database of target cluster 106. Terminal 102 determines the storage path of the master data in target cluster 106 based on the version differences, and uses this storage path to synchronize the master data of the source and target clusters. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, etc. Source cluster 104 and target cluster 106 can be implemented using independent servers or a server cluster composed of multiple servers.

[0052] In one embodiment, such as Figure 2 As shown, a data table synchronization method is provided, which can be applied to... Figure 1 Taking terminal 102 as an example, the explanation includes the following steps:

[0053] S202, obtain component version information for the source and target clusters.

[0054] The source cluster can be a database or cluster that needs to be backed up. The target cluster can be a database or cluster to which data is synchronously written. Component version information is usually the Hive version, such as Hive 2.x or Hive 3.x, where x can represent a specific minor version number.

[0055] Specifically, it can connect to the source cluster and the target cluster, obtain the cluster configuration of the source cluster and the target cluster, and then obtain the component version information of Hive in the source cluster and the component version information of Hive in the target cluster.

[0056] In some exemplary embodiments, the cluster configurations of the source and target clusters can be obtained through local cluster files, which may include the version numbers of the components of the source and target clusters.

[0057] Understandably, before implementing this solution, users can select the source cluster, target cluster, source table in the source cluster, and target table in the target cluster. Selecting these parameters determines the source and target of the Hive data tables to be synchronized. The source cluster and source table information can locate the tables that need to be synchronized; the target cluster and target table information can locate the target location to which the source tables are synchronized.

[0058] S204, in response to the version difference between the component version information of the source cluster and the component version information of the target cluster, the metadata of the data table in the source cluster is copied and transformed according to the difference points corresponding to the version difference.

[0059] Version differences can refer to different versions of the big data platform, such as CDH6 and CDP7, CDH6 and CDH5, CDP7 and CDH5, and CDP7 and CDH6. The differences can be between Hive component versions within each version, such as the different Hive versions in CDH6 and CDP7. Conversion can involve converting a Hive table object from CDH6 to its corresponding Hive table object in CDP7. In this case, the metadata of the source and target clusters can represent databases, tables, table fields, partitions, indexes, and namespaces (a naming convention to ensure unique identification) within Hive. The table structure in Hive is part of the metadata, and the data in Hive tables can be called master data. Metadata is typically stored in a relational database. All metadata in Hive is usually maintained in HMS (Hive metastore), and the metadata of all Hive tables and partitions can be accessed through HMS.

[0060] Specifically, due to version differences between the components in the source and target clusters, to ensure the metadata is applicable when replicated to the target cluster, the metadata of the data tables in both clusters must first be compared. Typically, when version differences exist, the metadata will also differ after comparison. Therefore, the metadata needs to be transformed to eliminate Hive version differences without modifying the table structure in Hive. This can be done by retrieving the metadata of the source tables in the source cluster using HMS (Hive metastore), copying and transforming it based on the corresponding differences between the source and target clusters, and adapting the copied and transformed metadata to the component version information of the target cluster. Adjusting the metadata primarily aims to eliminate Hive version differences without modifying the table structure in Hive, thus ensuring the table structure remains unchanged.

[0061] In some exemplary embodiments, for example, the source cluster has a total of three partitions, namely A, B, and C, while the target cluster has a total of two partitions, namely C and D. In this case, the three partitions A, B, and C in the source cluster need to be adjusted to the three partitions A, B, and C or the four partitions A, B, C, and D in the target cluster. It should be understood that the above is only for illustrative purposes and does not represent actual application scenarios.

[0062] S206, Interact with the target cluster and write the transformed metadata into the target cluster's metadata database.

[0063] Specifically, after the metadata is copied and transformed as described above, it can interact with the target cluster's HMS (Hivemetastore). The copied and transformed metadata can be directly written to the target cluster's metadata database through the target cluster's HMS (Hive metastore), thus completing the metadata synchronization. This metadata database is typically a relational database that stores metadata.

[0064] S208, determine the storage path of the master data of the target cluster based on the version difference, and use the storage path to synchronize the master data of the data tables in the source cluster and the target cluster.

[0065] Specifically, after synchronizing the metadata, the master data can be synchronized. Because the source and target clusters have version differences—for example, CDH and CDP belong to different Hive versions, and their storage paths for internal and external master data are also different. For instance, when synchronizing from CDH6 to CDP7, the internal tables will change to external tables after synchronization, and the storage location will change from the original ` / user / hive / warehouse` directory to ` / warehouse / tablespace / managed / hive`. This not only makes it impossible to maintain consistency in table types but also changes the data storage location. Therefore, the data storage location needs to be adjusted. Continuing with the example of synchronizing from CDH6 to CDP7, if CDH6 is the Hive version in the source cluster and CDP7 is the Hive version in the target cluster, the storage path corresponding to CDH6 can be converted to the storage path corresponding to the target cluster's CDP7 version. After modification, because the storage paths for the Hive versions in the source and target clusters are the same, this storage path can be directly used to synchronize the master data of the tables in the source and target clusters. For example, master data from the source cluster can be written directly to the target cluster according to the storage path.

[0066] In the aforementioned data table synchronization method, component version information of the source and target clusters is obtained. If there are version differences, the metadata is copied and transformed. The transformed metadata is then written to the target cluster's metadata database. Adjustments are made based on the metadata differences, ensuring data synchronization between different component versions and eliminating version discrepancies. Furthermore, the storage path for the target cluster's master data is determined based on the version differences. This storage path is used to synchronize the master data of the data tables in the source and target clusters. Even when component version information differs between the source and target clusters, master data synchronization is achieved by converting the master data storage path between different versions, ensuring that the type of the data table to be synchronized (the data table in the source cluster) remains unchanged. By employing two different processing methods for metadata and master data when component version information differs, the problem of data table copying or incompatibility between different Hive versions in current data processing tools can be solved.

[0067] In one embodiment, such as Figure 3 As shown, the method further includes:

[0068] S302, If an abnormal situation occurs during the synchronization of master data, causing the synchronization to fail, the synchronization task corresponding to the master data synchronization is added to a pre-created synchronization queue.

[0069] S304, determine whether the pre-set retry conditions are met.

[0070] S306, when the pre-set retry conditions are met, the master data of the source cluster and the target cluster that failed to synchronize in the synchronization queue are synchronized again.

[0071] If the pre-set retry conditions are not met, synchronization will continue.

[0072] Abnormal situations can cause the DistCp task to fail and exit during execution, such as network jitter, network interruptions, or brief service failures of HDFS and YARN components (responsible for executing DistCp tasks). DistCp (Distributed Copy) is a tool for copying data within and between large-scale clusters. It uses MapReduce to implement file distribution, error handling and recovery, and report generation. It takes a list of files and directories as input to map tasks, and each task copies the master data from the source table. Pre-set retry conditions can include retrying after a certain period of time following a failure, such as retrying synchronization tasks in the synchronization queue after 5 or 10 minutes. Pre-set retry conditions can also include retrying synchronization tasks in the synchronization queue when the number of synchronization tasks in the queue exceeds a pre-set threshold. It is understood that those skilled in the art can flexibly set retry conditions according to different situations, and some disclosed embodiments do not impose specific restrictions on retry conditions.

[0073] Specifically, during master data synchronization, if an anomaly occurs, it may cause some tables or the entire synchronization task to fail, and corresponding error messages will appear in the execution log. When the corresponding error message is found in the execution log, the synchronization failure and the corresponding synchronization task can be identified. This synchronization task can be added to the tail of a pre-created synchronization queue. When the pre-set retry conditions are met, the synchronization task at the head of the synchronization queue can be retrieved sequentially for processing, and the master data of the failed source and target clusters can be synchronized again. This continues until there are no failed synchronization tasks in the synchronization queue. When the pre-set retry conditions are not met, the master data synchronization of the source and target clusters can continue.

[0074] In some exemplary embodiments, for example, four tables need to be synchronized: A, B, C, and D. If an anomaly occurs during the synchronization of master data in table B, causing the synchronization of all master data in table B to fail (i.e., the synchronization task for table B fails), the synchronization task for table B can be added to the tail of a pre-created synchronization queue. The synchronization task for table C continues. If an anomaly also occurs during the synchronization task for table C, causing the synchronization task for table C to fail, the synchronization task for table C can be added to the tail of the aforementioned synchronization queue. The corresponding synchronization queue contains, in order: the synchronization task for table B and the synchronization task for table C. When the pre-set retry conditions are met, the synchronization task for table B in the synchronization queue can be executed, followed by the synchronization task for table C. When the pre-set retry conditions are not met, the synchronization of master data in table D can continue.

[0075] In this embodiment, by adding the synchronization tasks corresponding to synchronization failures to the synchronization queue, it is ensured that failed synchronization tasks are not lost, reducing the number of task creations. Furthermore, when pre-set retry conditions are met, the synchronization tasks corresponding to failed synchronizations in the synchronization queue can be retried, ensuring that each synchronization task completes. Considering that task failures may occur due to cluster conditions, and the cluster state may be difficult to recover immediately, failed synchronization tasks can be moved to the end of the queue, while other normal tasks are executed first.

[0076] In one embodiment, such as Figure 4 As shown, the method further includes:

[0077] S402, in response to receiving the high-speed synchronization command, determine the storage path of the master data of the target cluster based solely on the version difference, and use the storage path to synchronize the master data of the source cluster and the target cluster;

[0078] The high-speed synchronization command indicates that no metadata comparison is required. High-speed synchronization commands are typically issued by users when they need to quickly synchronize data tables.

[0079] Specifically, when users need to quickly synchronize data tables, they typically issue a rapid synchronization command. Upon receiving this command, the default scenario is to synchronize the data from the source cluster to the target cluster for the first time. If the corresponding table already exists on the target cluster as it does in the source cluster, it will be considered outdated and overwritten. Therefore, upon receiving a rapid synchronization command, metadata comparison is not considered. It is assumed that the target cluster does not contain the same databases and tables as the source cluster; if they do exist, they are directly overwritten, with only metadata copying and conversion performed. The storage path can be directly used to synchronize the master data between the source and target clusters.

[0080] S404, during the synchronization of multiple data tables in the source cluster and the target cluster, in response to receiving a failure log, the data table corresponding to the failure log that failed to synchronize is determined.

[0081] S406, use breakpoint resume to continue synchronization from the data table that failed to synchronize.

[0082] The failure log indicates that the synchronization process for synchronizing the data table was terminated or failed.

[0083] Specifically, when a synchronization task involves synchronizing a large number of data tables, multiple data tables in the source and target clusters need to be synchronized. During the synchronization process, if a failure log is received, the corresponding data table can be identified for synchronization termination or failure. The breakpoint resume function can be used to continue synchronization from the terminated or failed data table. Failed synchronization tables can include both master data synchronization failures and metadata synchronization failures. When metadata synchronization fails, the metadata can be copied and transformed again. When master data synchronization fails, the storage path can be redefined and used to synchronize the master data.

[0084] In some exemplary embodiments, for example, a synchronization task needs to synchronize data from 1000 tables. If synchronization fails when synchronizing the 600th table due to a system failure of the node initiating the synchronization task, the synchronization program can be restarted, and the interruption point resumption function of the synchronization program can be used to continue synchronization from the 600th table. The synchronization process includes metadata copying and transformation, as well as master data synchronization.

[0085] In this embodiment, metadata and master data are synchronized directly without comparing metadata. By eliminating the metadata comparison step, the efficiency of data table synchronization is improved. The synchronization task is protected against breakpoints; if synchronization fails, it will not restart data synchronization from the beginning, ensuring the execution efficiency of the synchronization task.

[0086] In one embodiment, such as Figure 5 As shown, the step of synchronizing the master data of the source cluster and the target cluster using the storage path includes:

[0087] S502, determine the number of parallel synchronization processes based on the cluster performance and resources of the source cluster or target cluster.

[0088] S504, using the number of processes and the storage path, the master data of the source cluster and the target cluster are synchronized in parallel.

[0089] Cluster performance can be understood as the computing resources, I / O resources, and disk resources used during synchronization. Computing resources generally refer to the CPU resources, memory resources, and network resources required for a computer program to run. In this case, the resources typically refer to the computing resources needed to synchronize master data.

[0090] Specifically, the cluster performance for synchronization can be selected from either the source cluster or the target cluster. Therefore, either the source cluster or the target cluster can be chosen to provide computing resources for data synchronization (including metadata replication and transformation and master data synchronization). Then, the computing resources required for data synchronization are determined, and the number of processes that can be synchronized concurrently is determined based on the cluster performance and computing resources. Data synchronization is performed in parallel on the source and target clusters according to the modified number of processes and storage paths. In some exemplary embodiments, a resource pool can be used to control the number of processes. The resource pool manages the threads for data synchronization, i.e., a thread resource pool. Managing the thread resource pool controls the maximum number of parallel table synchronization processes. After master data synchronization is complete, the thread resource pool can be destroyed. If the cluster performance is 100 and the computing resources required for master data synchronization are 50, then the number of parallel synchronization processes can be determined to be 2. It is understood that the above cluster performance and computing resources are for illustrative purposes only.

[0091] In this embodiment, by determining the number of parallel synchronization processes, multiple processes can process simultaneously, which speeds up the synchronization of data tables and makes full use of the idle resources of the cluster.

[0092] In one embodiment, before converting the metadata of the data tables in the source cluster based on the differences corresponding to the version differences, the method further includes:

[0093] In response to receiving modified parameters, the metadata of the data tables in the source cluster is modified according to the modified parameters.

[0094] The parameters that need to be modified typically include the database name and table name.

[0095] Specifically, during the metadata conversion process, users can input modification parameters. Based on these parameters, the metadata of the data tables in the source cluster is copied and modified. The metadata of the source cluster's data tables (source tables) remains unchanged; that is, the modification parameters typically directly affect the metadata conversion. During metadata conversion, the modified metadata of the data tables in the source cluster can be converted. The target cluster can then generate the corresponding database name and table name based on the modification parameters in the converted metadata.

[0096] In this embodiment, during the data table synchronization process, the databases and tables of the target cluster can be renamed to meet user needs.

[0097] In one embodiment, such as Figure 6 As shown, the method further includes:

[0098] S602, During the process of synchronizing master data, snapshots are created in the source cluster and the target cluster respectively.

[0099] A snapshot is typically an HDFS snapshot, which is a read-only image of the HDFS file system at a certain point in time; it can be a complete file system or an image of a directory.

[0100] Specifically, a snapshot is created in the source cluster to ensure that the data in the source cluster does not change during data synchronization. Simultaneously, a snapshot is also created in the target cluster to prevent data corruption in the target cluster due to errors during master data synchronization.

[0101] S604, in response to receiving the synchronization log, delete the snapshot in the source cluster, and perform a consistency check on the master data in the target cluster, wherein the synchronization log indicates that the synchronization is complete.

[0102] Specifically, in the data synchronization task, each table is executed by a separate thread. Once this thread completes and returns successfully, it confirms that the master data and metadata of the table have been synchronized. Simultaneously, corresponding logs are output upon task completion. These logs can be synchronization logs. Upon receiving the synchronization logs, the snapshot of the source cluster is deleted. A consistency check is performed on the master data in the synchronized tables to confirm the correctness of the data synchronization results. If the synchronized results are inconsistent, data synchronization fails, and a notification can be issued, or master data synchronization can be performed again.

[0103] In some exemplary embodiments, consistency verification can be performed using CRC checksums, where CRC is used to generate checksums for the master data of the source and target clusters, and the checksums are compared to confirm data consistency. Other consistency verification methods include parity check and cumulative checksum.

[0104] S606, in response to receiving a kill command, obtain the process corresponding to the synchronization, terminate the process according to the kill command, and stop the synchronization.

[0105] Among them, the kill command can usually be the command to terminate synchronization.

[0106] Specifically, this function can detect and terminate ongoing synchronization tasks. This can be done by obtaining the process of the synchronization task in the operating system and then terminating that process. After the current table is synchronized, subsequent tables will not be synchronized.

[0107] In this embodiment, the source cluster snapshot ensures that the master data of the synchronized table remains unchanged, while the target cluster snapshot ensures its recoverability and prevents the target cluster from generating dirty data due to synchronization task anomalies. Simultaneously, the generated snapshots (from the source cluster) are cleaned up to avoid excessive space consumption in the source cluster.

[0108] In one embodiment, in response to receiving an operation command, the operation command is parsed, and the function corresponding to the operation command is executed. The function may include: parallel synchronization using parallel data, retrying synchronization tasks using retries, whether to use snapshots, whether to skip CRC checks, DistCp synchronization mode, single-table / multi-table mode, bandwidth size, DistCp MAP count, high-speed mode, breakpoint resume mode, and main task scanning mode, etc. Flexible parameter control of the operation command can meet various user needs.

[0109] The single-table and multi-table modes are designed to suit different application scenarios. Multi-table mode supports wildcard matching across multiple tables, enabling synchronization within the same task, but does not support renaming tables. Single-table mode does not support wildcards, but allows table renaming. Controlling bandwidth can limit network I / O (input and output) for synchronization tasks, preventing excessive network bandwidth consumption from impacting the normal operation of other tasks in the cluster.

[0110] Unlike existing technologies, current synchronization software has weak support for high-load data synchronization scenarios, offers few personalized configuration parameters, and has a limited scope of application, failing to cover current database and table synchronization needs. This solution adds numerous user-selectable configurations to the operation commands and includes various instructions, such as high-speed synchronization commands and scan commands, greatly enriching the types of tasks users can submit and generating the results they require.

[0111] In one embodiment, after the data tables of the source cluster and the target cluster are synchronized, the results of this data table synchronization can be summarized and analyzed, such as how many data tables were synchronized, the number of data tables that were successfully synchronized, the number of data tables that failed to synchronize, and the names of the data tables that failed to synchronize.

[0112] In one embodiment, this disclosure also provides another data table synchronization method, such as... Figure 7 As shown, it includes:

[0113] S702, obtain component version information for the source and target clusters.

[0114] S704, in response to a version difference between the component version information of the source cluster and the component version information of the target cluster, and in response to receiving modification parameters, the metadata of the data table in the source cluster is modified according to the modification parameters.

[0115] S706, Based on the differences corresponding to the version differences, the metadata of the data tables in the source cluster is copied and transformed.

[0116] S708, interact with the target cluster and write the transformed metadata into the target cluster's metadata database.

[0117] S710, determine the number of parallel synchronization processes based on the cluster performance and resources of the source cluster or target cluster.

[0118] S712, the master data of the source cluster and the target cluster are synchronized in parallel using the number of processes and the storage path.

[0119] S714: If an abnormal situation occurs during the synchronization of master data, causing the synchronization to fail, the synchronization task corresponding to the master data synchronization is added to a pre-created synchronization queue.

[0120] S716, when the pre-set retry conditions are met, the data tables of the source cluster and the target cluster that failed to synchronize in the synchronization queue are synchronized again.

[0121] S718, in response to receiving the high-speed synchronization command, determines the storage path of the master data of the target cluster based solely on the version difference, and uses the storage path to synchronize the master data of the source cluster and the target cluster. The high-speed synchronization command indicates that metadata comparison is not required.

[0122] S720, during the synchronization of multiple data tables in the source cluster and the target cluster, in response to receiving a failure log, the data table corresponding to the failure log that failed to synchronize is identified, wherein the failure log indicates that the synchronization process of synchronizing the data table was terminated or failed.

[0123] S722 uses breakpoint resume to continue synchronization from the data table that was terminated or failed.

[0124] S724, During the synchronization of master data, snapshots are created in the source cluster and the target cluster respectively.

[0125] S726, in response to receiving the synchronization log, delete the snapshot in the source cluster, and perform a consistency check on the master data in the target cluster, wherein the synchronization log indicates that the synchronization is complete.

[0126] S728, in response to receiving a kill command, obtain the process corresponding to the synchronization, terminate the process according to the kill command, and stop the synchronization.

[0127] For specific implementation methods and limitations in this embodiment, please refer to the above embodiments, which will not be repeated here.

[0128] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0129] Based on the same inventive concept, this disclosure also provides a data table synchronization device for implementing the data table synchronization method described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more data table synchronization device embodiments provided below can be found in the limitations of the data table synchronization method described above, and will not be repeated here.

[0130] In one embodiment, such as Figure 8 As shown, a data table synchronization device 800 is provided, including: an information acquisition module 802, a data conversion module 804, a data interaction module 806, and a data synchronization module 808, wherein:

[0131] The information acquisition module 802 is used to acquire component version information of the source cluster and the target cluster;

[0132] The data conversion module 804 is used to copy and convert the metadata of the data table in the source cluster according to the difference points corresponding to the version difference when there is a version difference between the component version information of the source cluster and the component version information of the target cluster.

[0133] The data interaction module 806 is used to interact with the target cluster and write the converted metadata into the metadata database of the target cluster.

[0134] The data synchronization module 808 is used to determine the storage path of the master data of the target cluster based on the version difference, and to synchronize the master data of the source cluster and the target cluster using the storage path.

[0135] In one embodiment of the device, the device further includes an exception handling module, which, in response to an exception occurring during the synchronization of the master data that causes synchronization failure, adds the synchronization task corresponding to the master data synchronization to a pre-created synchronization queue.

[0136] The retry module is used to re-synchronize the master data of the source cluster and the target cluster that failed to synchronize in the synchronization queue when the pre-set retry conditions are met.

[0137] In one embodiment of the device, the device further includes: a high-speed synchronization module, configured to, in response to receiving a high-speed synchronization instruction, determine the storage path of the master data of the target cluster based solely on the version difference, and synchronize the master data of the source cluster and the target cluster using the storage path, wherein the high-speed synchronization instruction indicates that metadata comparison is not required.

[0138] In one embodiment of the device, the data synchronization module 808 includes: a process number determination module, used to determine the number of parallel synchronization processes based on the cluster performance and resources of the source cluster or the target cluster.

[0139] The data synchronization submodule is used to synchronize the master data of the source cluster and the target cluster in parallel using the number of processes and the storage path.

[0140] In one embodiment of the device, the synchronization failure determination module is used to determine, in response to receiving a failure log, the data table corresponding to the failure log that failed to synchronize during the synchronization of multiple data tables in the source cluster and the target cluster, wherein the failure log indicates that the synchronization was terminated or failed during the synchronization of the data table.

[0141] The breakpoint resume module is used to resume synchronization from a data table that has terminated or failed.

[0142] In one embodiment of the apparatus, the apparatus further includes: a modification module, configured to modify the metadata of the data table in the source cluster according to the modification parameters received.

[0143] In one embodiment of the apparatus, the apparatus further includes a snapshot module for creating snapshots in the source cluster and the target cluster respectively during the synchronization of master data.

[0144] A synchronization processing module is configured to, in response to receiving a synchronization log, delete the snapshot in the source cluster and perform a consistency check on the master data in the target cluster, wherein the synchronization log indicates that synchronization has been completed.

[0145] The process killing module is used to respond to a killing command, obtain the process corresponding to the synchronization, terminate the process according to the killing command, and stop the synchronization.

[0146] Each module in the aforementioned data table synchronization device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0147] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 9 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores metadata, master data, or data tables. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements a data table synchronization method.

[0148] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present disclosure and does not constitute a limitation on the computer device to which the present disclosure is applied. A specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0149] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in any of the above method embodiments.

[0150] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in any of the above method embodiments.

[0151] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in any of the above method embodiments.

[0152] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this disclosure can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this disclosure may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this disclosure may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0153] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0154] The embodiments described above are merely illustrative of several implementations of this disclosure, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent disclosure. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this disclosure, and these all fall within the protection scope of this disclosure. Therefore, the protection scope of this disclosure should be determined by the appended claims.

Claims

1. A data table synchronization method, characterized in that, The method includes: Obtain component version information for the source and target clusters; In response to the version difference between the component version information of the source cluster and the component version information of the target cluster, the metadata of the data table in the source cluster is copied and transformed according to the difference points corresponding to the version difference. Interact with the target cluster and write the transformed metadata into the target cluster's metadata database; The storage path of the master data of the target cluster is determined based on the version differences, and the master data of the data tables in the source cluster and the target cluster is synchronized using the storage path; The method further includes: In response to receiving a high-speed synchronization command, the storage path of the master data of the target cluster is modified only according to the version difference, and the master data of the source cluster and the target cluster are synchronized using the storage path. The high-speed synchronization command indicates that there is no need to perform metadata comparison. During the synchronization of multiple data tables in the source cluster and the target cluster, in response to receiving a failure log, the data table corresponding to the failure log is identified as having failed to synchronize, wherein the failure log indicates that the synchronization process of synchronizing the data table has terminated or failed. Resume synchronization from the data table that was interrupted or failed, using breakpoint resume.

2. The method according to claim 1, characterized in that, The method further includes: If an exception occurs during the synchronization of master data, causing the synchronization to fail, the corresponding synchronization task for that master data will be added to a pre-created synchronization queue. When the pre-set retry conditions are met, the master data of the source cluster and the target cluster that failed to synchronize in the synchronization queue are synchronized again.

3. The method according to claim 1, characterized in that, The step of synchronizing the master data of the source cluster and the target cluster using the storage path includes: The number of parallel synchronization processes is determined based on the cluster performance and resources of the source or target cluster. The source cluster and the target cluster master data are synchronized in parallel using the number of processes and the storage path.

4. The method according to claim 1, characterized in that, Before converting the metadata of the data tables in the source cluster based on the differences corresponding to the version differences, the method further includes: In response to receiving modified parameters, the metadata of the data tables in the source cluster is modified according to the modified parameters.

5. The method according to claim 3, characterized in that, The method further includes: During the synchronization of master data, snapshots are created in both the source and target clusters. In response to receiving the synchronization log, the snapshot in the source cluster is deleted, and the consistency check of the master data in the target cluster is performed, wherein the synchronization log indicates that the synchronization is completed; In response to receiving a kill command, the system retrieves the process corresponding to the synchronization process, terminates the process according to the kill command, and stops synchronization.

6. A data table synchronization device, characterized in that, The device includes: The information acquisition module is used to acquire component version information of the source cluster and the target cluster; The data conversion module is used to copy and convert the metadata of the data tables in the source cluster according to the difference points corresponding to the version differences when there are version differences between the component version information of the source cluster and the component version information of the target cluster. The data interaction module is used to interact with the target cluster and write the converted metadata into the target cluster's metadata database. The data synchronization module is used to determine the storage path of the master data of the target cluster based on the version difference, and use the storage path to synchronize the master data of the data tables in the source cluster and the target cluster. The high-speed synchronization module is used to respond to a high-speed synchronization command by modifying the storage path of the master data of the target cluster only according to the version difference, and using the storage path to synchronize the master data of the source cluster and the target cluster. The high-speed synchronization command indicates that there is no need to perform metadata comparison. The synchronization failure determination module is used to determine the data table that failed to synchronize in response to receiving a failure log during the process of synchronizing multiple data tables of the source cluster and the target cluster. The failure log indicates that the synchronization was terminated or failed during the synchronization of the data table. The breakpoint resume module is used to resume synchronization from a data table that has terminated or failed.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images