A method and tool for implementing Hudi data cross-cluster migration based on Spark

Abstract

The present invention discloses a method for implementing cross-cluster migration of Hudi data based on Spark, which relates to the field of data migration technology, including: using Hive SQL to create a database under a target cluster, the database having the same name and path as the source cluster; using Hadoop and Linux commands to upload Hudi table data files on the source cluster HDFS to the database on the corresponding HDFS path of the target cluster; parsing the configuration parameters of the table, and using HiveSyncTool to synchronize the field information of the table data files to the Hive metadata according to the parsing parameters, and implementing Hive table creation in the database of the target cluster; reading the Hive metadata database MySQL table on the source cluster, obtaining the annotations of each field of each table, and synchronizing the annotation information to the Hive table. The present invention also discloses a tool for implementing cross-cluster migration of Hudi data based on Spark, which, combined with the above method, can realize automatic table creation and metadata synchronization, and can also migrate all the Hudi historical data of one cluster to another cluster, with a simple and efficient process.

Description

Technical Field

[0001] The present invention relates to the technical field of data migration, and more specifically to a method and tool for implementing Hudi data cross-cluster migration based on Spark. Background Art

[0002] The data lake is a very old concept, first proposed at the Hadoop World conference and having become a hot topic again in recent years. AWS was the first cloud service provider to launch a data lake solution on the cloud. AWS defines a data lake as a centralized repository that allows for the storage of data of any structure and can be applied to big data processing, real-time analytics, machine learning, and other related application scenarios. Currently, the main data lake technologies include Hudi, Iceberg, and DaltaLake.

[0003] Hudi is a data lake framework originally designed and developed by Uber engineers to meet their data analytics needs. Named after Hadoop Upserts Deletes and Incrementals, Hudi is a data lake framework that supports inserts, updates, deletes, and incremental processing. Furthermore, Hudi supports transactional ACID incremental processing, storage management, and time management. Hudi can manage extremely large analytical datasets in the cloud, exceeding hundreds of petabytes. It works out-of-the-box with all cloud services, making it incredibly convenient. It has been running stably within Uber for nearly four years.

[0004] Based on this, in order to realize the cross-cluster migration of historical data in the Hudi table, a method and tool for realizing cross-cluster migration of Hudi data based on Spark was designed and developed. Summary of the Invention

[0005] In response to the needs and shortcomings of current technological development, the present invention provides a method and tool for implementing cross-cluster migration of Hudi data based on Spark, so as to migrate all the historical Hudi data of one cluster to another cluster and synchronize the Hive metadata and field annotations.

[0006] First, the present invention provides a method for implementing cross-cluster migration of Hudi data based on Spark. The technical solutions adopted to solve the above technical problems are as follows:

[0007] A method for implementing cross-cluster migration of Hudi data based on Spark includes the following steps:

[0008] S1. Use Hive SQL to create a database in the target cluster. The database name and path are the same as those in the source cluster.

[0009] S2. Use Hadoop and Linux commands to upload the Hudi table data files on the source cluster's HDFS to the database on the corresponding HDFS path on the target cluster.

[0010] S3. Parse the table configuration parameters. HiveSyncTool synchronizes the field information of the table data file to the Hive metadata based on the parsed parameters and creates the Hive table in the database of the target cluster.

[0011] S4. Read the MySQL table of the Hive metadata repository on the source cluster, obtain the comments for each field in each table, and synchronize the comment information to the Hive table in step S3.

[0012] Optionally, when executing step S1, the number, name, and path of databases created in the target cluster are the same as the number, name, and path of databases in the source cluster.

[0013] Optionally, the specific implementation process of step S2 is:

[0014] S2.1. Use Hadoop's get command to download the Hudi table data file on the source cluster's HDFS.

[0015] S2.2. Use the Linux scp command to send the downloaded Hudi table data file to the target cluster;

[0016] S2.3. Use Hadoop's put command to upload the Hudi table data file to the database on the corresponding path of the target cluster's HDFS.

[0017] Optionally, the specific operation process of step S3 is:

[0018] S3.1. Use the Hadoop file system API to list the path of each table under the database path, and then parse the table path to obtain the table name.

[0019] S3.2, traverse the table name and determine whether there is a Hudi table metadata directory .hoodie under the table path,

[0020] If yes, then further determine whether it is a Hudi table.

[0021] If it is a Hudi table, proceed to step S3.3;

[0022] S3.3. Get the file system fs based on spark.sessionState.newHadoopConf and the Hudi table path;

[0023] S3.4. Obtain the configuration file and table path from the file system fs, and then create a HoodieTableMetaClient for each table.

[0024] S3.5. Obtain tableConfig, the configuration information of each table, using HoodieTableMetaClient.

[0025] S3.6. Use tableConfig to obtain the corresponding primary key field and partition field information;

[0026] S3.7. Construct HiveSyncConfig based on the database name, table name, Hudi table path, partition field, and primary key field, and use the file system fs.getConf to create the Hive configuration file hiveConf.

[0027] S3.8. Construct the Hive synchronization tool class HiveSyncTool using HiveSyncConfig, hiveConf, and fs. Use the syncHoodieTable method of HiveSyncTool to synchronize metadata and create a Hive table in the database of the target cluster.

[0028] Further optionally, the specific operation process of step S4 is:

[0029] S4.1. Use SparkSQL's show tables method to obtain all Hudi table names in the target cluster.

[0030] S4.2. Use the JDBC method of SparkSQL to read the table name field annotation information in the Hive metadata repository MySQL in the source cluster.

[0031] S4.3. Use SparkSQL's JDBC method to obtain the table name and field information in the MySQL Hive metabase in the target cluster. Remove the portion of the source cluster's table name and field comments that matches the target cluster's table name and field. Then, match the target cluster's table name and field to construct the modification comment SQL: alter table$new_hiveDatabaseName.$hiveTableName change$colName$colName$colType comment'$comments';

[0032] S4.4. Use Spark SQL to execute the comment SQL and automatically synchronize the comments to the newly created Hive table in the target cluster database.

[0033] Secondly, the present invention provides a tool for implementing cross-cluster migration of Hudi data based on Spark. The technical solutions adopted to solve the above technical problems are as follows:

[0034] A tool for migrating Hudi data across clusters based on Spark. Its structure includes:

[0035] Create a module that uses Hive SQL to create a database in the target cluster. The database has the same name and path as the source cluster.

[0036] The data transmission module is used to transfer the Hudi table data files on the source cluster's HDFS to the database on the corresponding HDFS path of the target cluster using Hadoop and Linux commands;

[0037] The parsing module parses the table configuration parameters and uses HiveSyncTool to synchronize the field information of the table data file to the Hive metadata, thus creating the Hive table in the database of the target cluster.

[0038] The data synchronization module is used to read the Hive metadata repository MySQL table on the source cluster, obtain the comments for each field in each table, and synchronize the comment information to the Hive table of the parsing processing module.

[0039] Optionally, the number, name, and path of databases created by the creation module in the target cluster are the same as those in the source cluster.

[0040] Optionally, the data transfer module uses Hadoop and Linux commands to transfer the Hudi table data files on the source cluster's HDFS to the database on the corresponding HDFS path in the target cluster. The specific implementation process is as follows:

[0041] The data transmission module first uses Hadoop's get command to download the Hudi table data file on the source cluster HDFS, then uses Linux's scp command to send the downloaded Hudi table data file to the target cluster, and then uses Hadoop's put command to upload the Hudi table data file to the database on the corresponding path of the target cluster HDFS.

[0042] Optionally, the parsing module parses the configuration parameters of the table, synchronizes the field information of the table data file to the Hive metadata, and implements the Hive table creation in the database of the target cluster. The specific implementation process is as follows:

[0043] Use the Hadoop file system API to list the path of each table under the database path, and then parse the table path to get the table name;

[0044] Traverse the table name and determine whether there is a Hudi table metadata directory .hoodie under the table path;

[0045] If so, further determine whether it is a Hudi table;

[0046] If it is a Hudi table, first obtain the file system fs according to spark.sessionState.newHadoopConf and the Hudi table path, then obtain the configuration file and table path according to the file system fs, and then create each table's HoodieTableMetaClient, and then obtain each table's configuration information tableConfig according to HoodieTableMetaClient, use tableConfig to obtain the corresponding primary key field and partition field information, and construct HiveSyncConfig according to the database name, table name, Hudi table path, partition field, and primary key field, and use the file system fs.getConf to create the Hive configuration file hiveConf, and finally construct the Hive synchronization tool class HiveSyncTool through HiveSyncConfig, hiveConf, and fs, and use the syncHoodieTable method of HiveSyncTool to synchronize metadata and implement Hive table creation in the database of the target cluster.

[0047] Optionally, the specific implementation process of the data synchronization module synchronizing the annotation information to the Hive table is as follows:

[0048] Use SparkSQL's show tables method to get all Hudi table names in the target cluster;

[0049] Use SparkSQL's JDBC method to read the table name and field annotation information in the Hive metadata repository MySQL in the source cluster.

[0050] Use SparkSQL's JDBC method to obtain the table name and field information in the Hive metabase MySQL in the target cluster. Remove the part of the source cluster's table name and field comments that matches the target cluster's table name and field. Then match the target cluster's table name and field to construct the modification comment SQL: alter table$new_hiveDatabaseName.$hiveTableName change$colName$colName$colType comment'$comments';

[0051] Use Spark SQL to execute the comment SQL and automatically synchronize the comments to the newly created Hive table in the target cluster database.

[0052] The present invention provides a method and tool for implementing Hudi data cross-cluster migration based on Spark, which has the following beneficial effects compared with the existing technology:

[0053] (1) The data migration method and tool of the present invention can automatically create tables and synchronize metadata to migrate all the historical Hudi data of one cluster to another cluster;

[0054] (2) The data migration method and tool of the present invention can also automatically synchronize Hive field annotation information, eliminating the need for manual table creation, making it easier for operation and maintenance personnel to use;

[0055] (3) The present invention uses Hudi source code to automatically parse the primary key and partition fields of the Hudi table. Compared with the method of exporting table creation statements in Hive, the operation is simpler, more efficient, and less prone to errors. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Attachment Figure 1 This is a flowchart of the implementation of the first embodiment of the present invention;

[0057] Attachment Figure 2 This is a module connection block diagram of the second embodiment of the present invention.

[0058] The reference numbers in the accompanying drawings represent:

[0059] 1. Creation module, 2. Data transmission module, 3. Parsing and processing module,

[0060] 4. Data synchronization module, 5. Database. DETAILED DESCRIPTION

[0061] In order to make the technical solution, the technical problems solved and the technical effects of the present invention more clear, the technical solution of the present invention is clearly and completely described below in conjunction with specific embodiments.

[0062] Example 1:

[0063] Combined with attachment Figure 1 This embodiment proposes a method for implementing cross-cluster migration of Hudi data based on Spark, including the following steps:

[0064] S1. Use Hive SQL to create a database in the target cluster. The database name and path are the same as those in the source cluster.

[0065] When executing step S1, the number, name, and path of databases created in the target cluster are the same as those in the source cluster.

[0066] S2. Use Hadoop and Linux commands to upload the Hudi table data files on the source cluster's HDFS to the database on the corresponding HDFS path in the target cluster.

[0067] The specific implementation process of step S2 is:

[0068] S2.1. Use Hadoop's get command to download the Hudi table data file on the source cluster's HDFS.

[0069] S2.2. Use the Linux scp command to send the downloaded Hudi table data file to the target cluster;

[0070] S2.3. Use Hadoop's put command to upload the Hudi table data file to the database on the corresponding path of the target cluster's HDFS.

[0071] S3. Parse the table configuration parameters. HiveSyncTool synchronizes the field information of the table data file to the Hive metadata based on the parsed parameters and creates a Hive table in the database of the target cluster. The specific operation process is as follows:

[0072] S3.1. Use the Hadoop file system API to list the path of each table under the database path, and then parse the table path to obtain the table name.

[0073] S3.2, traverse the table name and determine whether there is a Hudi table metadata directory .hoodie under the table path,

[0074] If yes, then further determine whether it is a Hudi table.

[0075] If it is a Hudi table, proceed to step S3.3;

[0076] S3.3. Get the file system fs based on spark.sessionState.newHadoopConf and the Hudi table path;

[0077] S3.4. Obtain the configuration file and table path from the file system fs, and then create a HoodieTableMetaClient for each table.

[0078] S3.5. Obtain tableConfig, the configuration information of each table, using HoodieTableMetaClient.

[0079] S3.6. Use tableConfig to obtain the corresponding primary key field and partition field information;

[0080] S3.7. Construct HiveSyncConfig based on the database name, table name, Hudi table path, partition field, and primary key field, and use the file system fs.getConf to create the Hive configuration file hiveConf.

[0081] S3.8. Construct the Hive synchronization tool class HiveSyncTool using HiveSyncConfig, hiveConf, and fs. Use the syncHoodieTable method of HiveSyncTool to synchronize metadata and create a Hive table in the database of the target cluster.

[0082] S4. Read the MySQL table in the Hive metadata repository on the source cluster, obtain the comments for each field in each table, and synchronize the comment information to the Hive table in step S3. The specific operation process is as follows:

[0083] S4.1. Use SparkSQL's show tables method to obtain all Hudi table names in the target cluster.

[0084] S4.2. Use the JDBC method of SparkSQL to read the table name field annotation information in the Hive metadata repository MySQL in the source cluster.

[0085] S4.3. Use SparkSQL's JDBC method to obtain the table name and field information in the MySQL Hive metabase in the target cluster. Remove the portion of the source cluster's table name and field comments that matches the target cluster's table name and field. Then, match the target cluster's table name and field to construct the modification comment SQL: alter table$new_hiveDatabaseName.$hiveTableName change$colName$colName$colType comment'$comments';

[0086] S4.4. Use Spark SQL to execute the comment SQL and automatically synchronize the comments to the newly created Hive table in the target cluster database.

[0087] Example 2:

[0088] Combined with attachment Figure 2 This embodiment proposes a tool for implementing cross-cluster migration of Hudi data based on Spark, whose structure includes:

[0089] Create module 1, which uses Hive SQL to create database 5 in the target cluster. Database 5 has the same name and path as the source cluster.

[0090] Data transmission module 2 is used to transfer the Hudi table data files on the source cluster HDFS to the database 5 on the corresponding HDFS path of the target cluster using Hadoop and Linux commands;

[0091] Parsing module 3 is used to parse the table configuration parameters and synchronize the field information of the table data file to the Hive metadata with the help of HiveSyncTool, and implement Hive table creation in database 5 of the target cluster;

[0092] The data synchronization module 4 is used to read the Hive metadata database MySQL table on the source cluster, obtain the comments of each field in each table, and synchronize the comment information to the Hive table of the parsing processing module 3.

[0093] In this embodiment, the number, name, and path of the databases 5 created by the creation module 1 in the target cluster are the same as the number, name, and path of the databases in the source cluster.

[0094] In this embodiment, the data transmission module 2 uses Hadoop and Linux commands to transfer the Hudi table data file on the source cluster HDFS to the database 5 on the corresponding HDFS path of the target cluster. The specific implementation process is as follows:

[0095] Data transmission module 2 first uses Hadoop's get command to download the Hudi table data file on the source cluster HDFS, then uses Linux's scp command to send the downloaded Hudi table data file to the target cluster, and then uses Hadoop's put command to upload the Hudi table data file to database 5 on the corresponding path of the target cluster HDFS.

[0096] In this embodiment, the parsing module 3 parses the configuration parameters of the table, synchronizes the field information of the table data file to the Hive metadata, and implements the Hive table creation in the database 5 of the target cluster. The specific implementation process is as follows:

[0097] Use the Hadoop file system API to list the path of each table under the database 5 path, and then parse the table path to get the table name;

[0098] Traverse the table name and determine whether there is a Hudi table metadata directory .hoodie under the table path;

[0099] If so, further determine whether it is a Hudi table;

[0100] If it is a Hudi table, first obtain the file system fs according to spark.sessionState.newHadoopConf and the Hudi table path, then obtain the configuration file and table path according to the file system fs, and then create each table's HoodieTableMetaClient, and then obtain each table's configuration information tableConfig according to HoodieTableMetaClient, use tableConfig to obtain the corresponding primary key field and partition field information, and construct HiveSyncConfig according to the name of database 5, the name of the table, the Hudi table path, the partition field, and the primary key field, and use the file system fs.getConf to create the Hive configuration file hiveConf, and finally construct the Hive synchronization tool class HiveSyncTool through HiveSyncConfig, hiveConf, and fs, and use the syncHoodieTable method of HiveSyncTool to synchronize metadata and implement Hive table creation in database 5 of the target cluster.

[0101] In this embodiment, the specific implementation process of the data synchronization module 4 synchronizing the annotation information to the Hive table is as follows:

[0102] Use SparkSQL's show tables method to get all Hudi table names in the target cluster;

[0103] Use SparkSQL's JDBC method to read the table name and field annotation information in the Hive metadata repository MySQL in the source cluster.

[0104] Use SparkSQL's JDBC method to obtain the table name and field information in the Hive metabase MySQL in the target cluster. Remove the part of the source cluster's table name and field comments that matches the target cluster's table name and field. Then match the target cluster's table name and field to construct the modification comment SQL: alter table$new_hiveDatabaseName.$hiveTableName change$colName$colName$colType comment'$comments';

[0105] Use Spark SQL to execute the comment SQL and automatically synchronize the comments to the newly created Hive table in the target cluster database 5.

[0106] In summary, by adopting the method and tool of the present invention for implementing cross-cluster migration of Hudi data based on Spark, all the historical Hudi data of one cluster can be migrated to another cluster.

[0107] The above specific examples are used to illustrate the principles and implementation methods of the present invention in detail. These examples are only used to help understand the core technical content of the present invention. Based on the above specific embodiments of the present invention, any improvements and modifications made by those skilled in the art without departing from the principles of the present invention should fall within the scope of patent protection of the present invention.

Claims

1. A method for implementing Hudi data cross-cluster migration based on Spark, characterized by , including the following steps: S1. Use Hive SQL to create a database in the target cluster. The database name and path are the same as those in the source cluster. S2. Use Hadoop and Linux commands to upload the Hudi table data files on the source cluster's HDFS to the database on the corresponding HDFS path on the target cluster. S3. Parse the table configuration parameters. HiveSyncTool synchronizes the field information of the table data file to the Hive metadata based on the parsed parameters and creates a Hive table in the database of the target cluster. The specific operation process is as follows: S3.

1. Use the Hadoop file system API to list the path of each table under the database path, and then parse the table path to obtain the table name. S3.2, traverse the table name and determine whether there is a Hudi table metadata directory .hoodie under the table path, If yes, then further determine whether it is a Hudi table. If it is a Hudi table, proceed to step S3.3; S3.

3. Get the file system fs based on spark.sessionState.newHadoopConf and the Hudi table path; S3.

4. Obtain the configuration file and table path from the file system fs, and then create a HoodieTableMetaClient for each table. S3.

5. Obtain tableConfig, the configuration information of each table, using HoodieTableMetaClient. S3.

6. Use tableConfig to obtain the corresponding primary key field and partition field information; S3.

7. Construct HiveSyncConfig based on the database name, table name, Hudi table path, partition field, and primary key field, and use the file system fs.getConf to create the Hive configuration file hiveConf. S3.

8. Construct the HiveSyncTool class using HiveSyncConfig, hiveConf, and fs. Use the syncHoodieTable method of HiveSyncTool to synchronize metadata and create a Hive table in the database of the target cluster. S4. Read the MySQL table in the Hive metadata repository on the source cluster, obtain the comments for each field in each table, and synchronize the comment information to the Hive table in step S3. The specific operation process is as follows: S4.

1. Use SparkSQL's show tables method to obtain all Hudi table names in the target cluster. S4.

2. Use the JDBC method of SparkSQL to read the table name field annotation information in the Hive metadata repository MySQL in the source cluster. S4.

3. Use SparkSQL's JDBC method to obtain the table name and field information in the MySQL Hive metabase in the target cluster. Remove the portion of the source cluster's table name and field comments that matches the target cluster's table name and field. Then, match the target cluster's table name and field to construct the modification comment SQL: alter table$new_hiveDatabaseName.$hiveTableName change$colName$colName$colType comment'$comments'; S4.

4. Use Spark SQL to execute the comment SQL and automatically synchronize the comments to the newly created Hive table in the target cluster database.

2. According to a method for implementing Hudi data cross-cluster migration based on Spark according to claim 1, it is characterized in that When executing step S1, the number, name, and path of databases created in the target cluster are the same as those in the source cluster.

3. According to a method for implementing Hudi data cross-cluster migration based on Spark according to claim 1, it is characterized in that The specific implementation process of step S2 is: S2.

1. Use Hadoop's get command to download the Hudi table data file on the source cluster's HDFS. S2.

2. Use the Linux scp command to send the downloaded Hudi table data file to the target cluster; S2.

3. Use Hadoop's put command to upload the Hudi table data file to the database on the corresponding path of the target cluster's HDFS.

4. A tool for implementing cross-cluster migration of Hudi data based on Spark, characterized by , its structure includes: Create a module that uses Hive SQL to create a database in the target cluster. The database has the same name and path as the source cluster. The data transmission module is used to transfer the Hudi table data files on the source cluster's HDFS to the database on the corresponding HDFS path of the target cluster using Hadoop and Linux commands; The parsing module parses the table configuration parameters and uses HiveSyncTool to synchronize the field information of the table data file to the Hive metadata, thus creating the Hive table in the database of the target cluster. The data synchronization module is used to read the Hive metadata repository MySQL table on the source cluster, obtain the comments for each field in each table, and synchronize the comment information to the Hive table of the parsing processing module; The parsing module parses the configuration parameters of the table, synchronizes the field information of the table data file to the Hive metadata, and implements the Hive table creation in the database of the target cluster. The specific implementation process is as follows: Use the Hadoop file system API to list the path of each table under the database path, and then parse the table path to get the table name; Traverse the table name and determine whether there is a Hudi table metadata directory .hoodie under the table path; If so, further determine whether it is a Hudi table; If it is a Hudi table, first obtain the file system fs according to spark.sessionState.newHadoopConf and the Hudi table path, then obtain the configuration file and table path according to the file system fs, and then create each table's HoodieTableMetaClient, and then obtain each table's configuration information tableConfig according to HoodieTableMetaClient, use tableConfig to obtain the corresponding primary key field and partition field information, and construct HiveSyncConfig according to the database name, table name, Hudi table path, partition field, and primary key field, and use the file system fs.getConf to create the Hive configuration file hiveConf, and finally construct the Hive synchronization tool class HiveSyncTool through HiveSyncConfig, hiveConf, and fs, and use the syncHoodieTable method of HiveSyncTool to synchronize metadata and implement Hive table creation in the database of the target cluster; The specific implementation process of the data synchronization module synchronizing annotation information to the Hive table is as follows: Use SparkSQL's show tables method to get all Hudi table names in the target cluster; Use SparkSQL's JDBC method to read the table name and field annotation information in the Hive metadata repository MySQL in the source cluster. Use SparkSQL's JDBC method to obtain the table name and field information in the Hive metabase MySQL in the target cluster. Remove the part of the source cluster's table name and field comments that matches the target cluster's table name and field. Then match the target cluster's table name and field to construct the modification comment SQL: alter table$new_hiveDatabaseName.$hiveTableNamechange$colName$colName$colType comment'$comments'; Use Spark SQL to execute the comment SQL and automatically synchronize the comments to the newly created Hive table in the target cluster database.

5. A tool for implementing Hudi data cross-cluster migration based on Spark according to claim 4, characterized in that: The number, name, and path of the databases created by the creation module under the target cluster are the same as those under the source cluster.

6. A tool for implementing Hudi data cross-cluster migration based on Spark according to claim 4, characterized in that: The data transmission module uses Hadoop and Linux commands to transfer the Hudi table data files on the source cluster HDFS to the database on the corresponding HDFS path of the target cluster. The specific implementation process is as follows: The data transmission module first uses Hadoop's get command to download the Hudi table data file on the source cluster HDFS, then uses Linux's scp command to send the downloaded Hudi table data file to the target cluster, and then uses Hadoop's put command to upload the Hudi table data file to the database on the corresponding path of the target cluster HDFS.

Images