Method for realizing Clickhome data migration process automation

Abstract

The invention discloses a method for realizing automation of a Clickhouse data migration process. The method comprises the following steps: acquiring all source data tables under all databases from a source-end Clickhouse database, inputting the name of a target-end Clickhouse database and the name of the source data table into a database building script, executing the database building script, and generating the target-end Clickhouse database and a data table in a specified path; according to a ClickHouse-Copier migration configuration file in the ZooKeeper, data in a data table of a source end ClickHouse database is migrated to a data table of a target end ClickHouse database by executing a data migration script; according to the method, a Shell script language and a Java programming language are used, on the basis of ClickHouse-Copier, the defects of high use cost, tedious configuration and the like are split according to the process and then packaged into scripts, and automation of the data migration process is achieved; when the data migration process is executed, data migration can be completed only by inputting database names or data table names and identification parameters of different logics, meanwhile, data migration failure reasons can be accurately positioned through deep learning, and rapid troubleshooting of data migration problems is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of data migration, and in particular to a method for realizing Clickhouse data migration process automation. Background Art

[0002] The currently commonly used ClickHouse data migration method is to use the official tool ClickHouse-client to export the table creation SQL of the database and data table on the data source side, and then execute the table creation SQL in the destination environment to create the database and data table to be migrated. Finally, the data is migrated to the ClickHouse cluster on the destination side through the following three methods.

[0003] 1. The remote function connects the databases at both ends, executes the query SQL on the data source side and then INSERTs to the destination side

[0004] 2. Export the data from the data source as a CSV file, and then import the CSV file into the destination.

[0005] 3. Redirect the data queried from the data source through the Linux pipeline and insert it to the destination

[0006] Any of the above methods can be used to migrate data from the ClickHouse data table on the data source side to the ClickHouse data table on the destination side.

[0007] The three commonly used migration methods have the advantages of easy operation, simple configuration, and low learning cost. However, they have the following disadvantages:

[0008] 1. Single process operation

[0009] 2. Few configurable parameters at runtime

[0010] 3. Small amount of data transferred during the migration process

[0011] 4. Data cannot be retransmitted from a breakpoint

[0012] The above tools cannot meet the current requirements for stable, reliable, high-concurrency, and high-throughput migration of massive data.

[0013] ClickHouse-Copier, the official data migration tool of ClickHouse, has features such as multi-process, distributed tables, data sharding, and ZooKeeper to record migration status, which can meet the migration requirements of stable and reliable data, high concurrency, and high throughput. However, it has shortcomings such as general ease of use, cumbersome configuration, and too few official reference materials, resulting in low efficiency in troubleshooting during data migration. Summary of the invention

[0014] The purpose of the present invention is to provide a method for automating the Clickhouse data migration process to solve the above-mentioned problems existing in the prior art.

[0015] The specific application is as follows:

[0016] A method for automating the ClickHouse data migration process, the method comprising running a Shell script to migrate data from a source ClickHouse database to a target ClickHouse database;

[0017] The running of the Shell script to migrate the data of the source ClickHouse database to the target ClickHouse database includes the following steps:

[0018] S1: Obtain all source data tables under the ClickHouse database on the source side, input the name of the ClickHouse database on the target side and the name of the source data table into the database building script, execute the database building script, and generate the ClickHouse database and data table on the target side in the specified path;

[0019] S2: Pass the SQL file path of the data table under the target ClickHouse database and the ClickHouse-Copier migration configuration template file name into the ClickHouse-Copier migration configuration file script through the incoming parameters, and generate ClickHouse-Copier migration configuration files named after the target ClickHouse data table in turn. The ClickHouse-Copier migration configuration files are all stored in the migration configuration file directory;

[0020] S3: Upload the ClickHouse-Copier migration configuration file to ZooKeeper by executing the upload script;

[0021] S4: According to the ClickHouse-Copier migration configuration file in ZooKeeper, migrate the data in the data table of the source ClickHouse database to the data table of the destination ClickHouse database by executing the data migration script.

[0022] Furthermore, in step S2, the specific implementation process of the SQL file path of the data table under the target ClickHouse database is:

[0023] First, connect to the source ClickHouse database through clickhouse-client, execute the SQL to view the database creation statement, and export the result to the target database file with the suffix sql; then use the sed command to remove the line breaks and \\ symbols in the file; then connect to the source ClickHouse database through clickhouse-client, export all data table lists under the database to the target database file, and use the depth-first search algorithm to traverse the exported data table list file to see if all are exported. If it is determined that the data table list file has not been exported, iteratively export the SQL of the creation statement of each table to the target data table file through clickhouse-client. After each data table is exported, use the sed command to remove the line breaks and \\ symbols in the target data table file until the traversal of the data table list file is completed; if it is determined that the data table list file has been exported, generate the SQL file path of the data table.

[0024] Furthermore, in step S2, the specific implementation process of the ClickHouse-Copier migration configuration file script is:

[0025] The ClickHouse-Copier migration configuration file script includes a ClickHouse data table SQL file module, a ClickHouse-Copier migration configuration template file and a ClickHouse-Copier migration configuration file to generate a Jar package; first, a script tool that can read the target data table file and the ClickHouse-Copier migration configuration template file is developed using the Java programming language, and the script tool reads the target data table file and the ClickHouse-Copier migration configuration template file into memory; then the script tool extracts various parameters of the data table in the target data table file, and the various parameters are filled into the ClickHouse-Copier migration configuration template file. If the filling is completed, the script tool This tool generates a ClickHouse-Copier migration configuration file named after the ClickHouse data table; then the script tool traverses the SQL files of all data tables under the SQL file path of the target ClickHouse data table through the Shell, and filters out data tables without the ENGINE engine through the grep command; runs the Jar package in the JVM virtual machine, and the Jar package passes the SQL file path of the ClickHouse data table and the ClickHouse-Copier migration configuration template file name through the incoming parameters, and generates the ClickHouse-Copier migration configuration file named after the ClickHouse data table in turn. At the same time, the script tool generates the ClickHouse-Copier migration configuration file script.

[0026] Furthermore, the specific implementation process of uploading the ClickHouse-Copier migration configuration file to ZooKeeper by executing the upload script in step S3 is:

[0027] The upload script uploads the ClickHouse-Copier migration configuration file in four ways, and the four ways specifically include a first upload method, a second upload method, a third upload method, and a fourth upload method;

[0028] The first upload method indicates the creation or update of a single file:

[0029] The upload script passes in three parameters, which are the operation type parameter, the ClickHouse database name, and the ClickHouse data table name. The upload script executes the creation or update process of a single file:

[0030] If it is a creation operation, first determine whether there is a ClickHouse database directory in ZooKeeper. If not, create it first. If it exists, create the ClickHouse data table directory. Then read the ClickHouse-Copier migration configuration file corresponding to the data table through the ZooKeeper cat command, write it to a file named description, and save it in the ClickHouse data table directory.

[0031] If it is an update operation, the content in the description file in the ClickHouse data table directory is reset through the ZooKeeper set command;

[0032] The second upload method indicates that multiple files are created:

[0033] The upload script passes in three parameters, which are the operation type parameter, the ClickHouse database name, and the ClickHouse data table name. The upload script executes the creation process of multiple files:

[0034] First, determine whether there is a ClickHouse database directory in ZooKeeper. If not, create the ClickHouse database directory first. If it exists, create the ClickHouse data table directory. Then traverse all migration configuration files in the migration configuration file directory and create the ClickHouse data table directory. Then read the ClickHouse-Copier migration configuration file corresponding to the data table through the ZooKeeper cat command, write it to a file named description, and save it in the ClickHouse data table directory.

[0035] The third upload method indicates the creation or update of multiple files in a partition:

[0036] The upload script passes in three parameters, which are the operation type parameter, ClickHouse database name, and ClickHouse data table name. The upload script executes multiple creation or update processes for partition-based migration data:

[0037] If it is a creation operation, first determine whether there is a ClickHouse database directory in ZooKeeper. If not, create the ClickHouse database directory first. If it exists, create the ClickHouse data table directory. Then traverse all configuration files in the partition migration configuration file directory, create the ClickHouse data table directory, and then read the ClickHouse-Copier migration configuration file corresponding to the data table through the ZooKeeper cat command, write it to a file named description, and save it in the ClickHouse data table directory.

[0038] If it is an update operation, reset the content in the description file in the ClickHouse data table directory through the ZooKeeper set command;

[0039] The fourth upload method indicates the creation or update of a partitioned single file:

[0040] The upload script passes in three parameters, which are the operation type parameter, ClickHouse database name, and ClickHouse data table name. The upload script executes the creation or update process of a single partition-by-partition migration data:

[0041] If it is a creation operation, first determine whether there is a ClickHouse database directory in ZooKeeper. If not, create the ClickHouse database directory first. If it exists, create the ClickHouse data table directory. Then read the ClickHouse-Copier migration configuration file corresponding to the data table through the ZooKeeper cat command, write it to a file named description, and save it in the ClickHouse data table directory.

[0042] If it is an update operation, the content in the description file in the ClickHouse data table directory is reset through the ZooKeeper set command.

[0043] Further, in step S4, according to the ClickHouse-Copier migration configuration file in ZooKeeper, the specific implementation process of migrating the data in the data table of the source ClickHouse database to the data table of the destination ClickHouse database by executing the data migration script is:

[0044] The data migration script supports three modes of functions, including a first mode, a second mode and a third mode;

[0045] The first mode is used for full migration of a single table, and is implemented as follows:

[0046] The data migration script passes in two parameters, and the two parameters Clickhouse database name and Clickhouse data table name are spliced ​​into the data migration configuration file path. The data migration configuration file path is used to represent the path in ZooKeeper, and then the data migration process is started to migrate data according to the data migration configuration file path; if the data migration is completed, clickhouse-client will query the total number of data in the migrated data table in the source ClickHouse database and the destination ClickHouse database respectively. If the total number of data is consistent, the data migration is successful; if the total number of data is inconsistent, the data migration fails, and the cause of the data migration failure is checked and then migrated again;

[0047] The second mode is used for full migration of multiple tables. The specific implementation method is as follows:

[0048] The data migration script passes in a Clickhouse database name parameter;

[0049] Traverse the migration configuration files of all ClickHouse data tables in the copier-tasks directory under the data migration configuration file directory to obtain the names of the ClickHouse data tables;

[0050] The data migration script starts to accumulate the number of ClickHouse data tables traversed from 0, presets the total number of ClickHouse data tables, and prints the information of which table is being migrated and how many tables are left to be migrated to the data migration script operation log;

[0051] The data migration configuration file path is spliced ​​by the passed-in ClickHouse database name and the traversed ClickHouse data table name. The data migration configuration file path is used to represent the path in ZooKeeper, and then the data migration process is started to migrate the data; if the data migration is completed, clickhouse-client will query the total number of data in the migrated data table in the source ClickHouse database and the destination ClickHouse database respectively. If the total number of data is consistent, the data migration is successful; if the total number of data is inconsistent, the data migration fails, and the cause of the data migration failure is checked and then migrated again; if the number of ClickHouse data tables traversed by the data migration script is equal to the preset total number of ClickHouse data tables, the process ends;

[0052] The third mode is used for full migration of multiple tables by partition, and the specific implementation method is as follows:

[0053] The data migration script passes in a Clickhouse database name parameter;

[0054] Traverse the migration configuration files of all ClickHouse data tables in the copier-tasks directory under the data migration configuration file directory to obtain the names of the ClickHouse data tables;

[0055] The data migration script starts to accumulate the number of ClickHouse data tables traversed from 0, presets the total number of ClickHouse data tables, and prints the information of which table is being migrated and how many tables are left to be migrated to the data migration script operation log;

[0056] Traverse the migration configuration files of all ClickHouse data tables in the incre_partition_copier_tasks directory under the data partition migration configuration file directory, obtain the name of the ClickHouse data table, and print the information of which table is being migrated and how many tables are left to be migrated to the data migration script running log;

[0057] Read the partition difference list file, and query the partition data of this table in the destination ClickHouse database through clickhouse-client. If the total data of the data table is greater than 0 and the number of partitions is equal to 1, the partition data of this table in the destination ClickHouse database will be DROPed; if the total data of the data table is greater than 0 and the number of partitions is equal to 2, the second partition will be found through the first partition and the second partition will be DROPed;

[0058] After the data in the data table partition of the destination ClickHouse database is deleted, the data migration configuration file path is spliced ​​into the passed-in ClickHouse database name and the traversed ClickHouse data table name. The data migration configuration file path is used to represent the path in ZooKeeper, and then the data migration process is started to migrate the data; if the data migration is completed, clickhouse-client will query the total number of data in the migrated data table in the source ClickHouse database and the destination ClickHouse database respectively. If the total number of data is consistent, the data migration is successful; if the total number of data is inconsistent, the data migration fails, and the cause of the data migration failure is checked and then migrated again; if the number of ClickHouse data tables traversed by the data migration script is equal to the preset total number of ClickHouse data tables, the process ends.

[0059] Furthermore, the specific implementation process of troubleshooting the cause of data migration failure is as follows:

[0060] Collect historical migration failure data, including script input parameter error data, database account password error data, script log error data, and network error data;

[0061] Performing normalized feature extraction on the historical migration failure data by using natural language technology to generate migration failure feature data, wherein a portion of the migration failure feature data is used for a training set and a portion of the migration failure feature data is used for a validation set;

[0062] Constructing a migration failure error deep learning model, performing model training using training set data, and stopping training if the training of the migration failure error deep learning model meets a preset recognition rate, wherein the input of the migration failure error deep learning model is the migration failure feature data, and the output of the migration failure error deep learning model is the migration error type;

[0063] The trained migration failure error deep learning model is used to analyze the causes of data migration failure in real time and output the migration error type.

[0064] Furthermore, the network error data includes errors caused by bandwidth, delay and packet loss of the network connection.

[0065] Furthermore, the feature extraction of the script log error data also includes:

[0066] The convolutional neural network (CNN) is used to extract the spatial features of the script log error data, and the long short-term memory (LSTM) network is used to capture the temporal sequence relationship of the script log error data to generate migration failure feature data.

[0067] Compared with the prior art, the embodiments of the present invention achieve the following beneficial effects:

[0068] The embodiment of the present invention provides a method for obtaining all source data tables under the ClickHouse database on the source side, inputting the name of the ClickHouse database on the target side and the name of the source data table into the database building script, executing the database building script, and generating the ClickHouse database and data table on the target side in the specified path; passing the SQL file path of the data table under the ClickHouse database on the target side and the file name of the ClickHouse-Copier migration configuration template into the ClickHouse-Copier migration configuration file script through the incoming parameters, and sequentially generating ClickHouse-Copier migration configuration files named after the ClickHouse data table on the target side, and the ClickHouse-Copier migration configuration files are all stored in the migration configuration file directory; by executing the upload script The ClickHouse-Copier migration configuration file is uploaded to ZooKeeper; according to the ClickHouse-Copier migration configuration file in ZooKeeper, the data in the data table of the source ClickHouse database is migrated to the data table of the destination ClickHouse database by executing the data migration script; the present invention uses Shell scripting language and Java programming language, and based on ClickHouse-Copier, the high cost of use and cumbersome configuration are divided according to the process and encapsulated into scripts to realize the automation of the data migration process; when executing the data migration process, only the database name or data table name and identification parameters of different logics need to be input to complete the data migration, and at the same time, the cause of data migration failure can be accurately located through deep learning, so as to realize the rapid troubleshooting of data migration problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 It is a flow chart of a method for realizing Clickhouse data migration process automation provided by an embodiment of the present invention;

[0070] Figure 2 A data table creation statement export flow chart of a method for realizing Clickhouse data migration process automation provided by an embodiment of the present invention;

[0071] Figure 3 A data migration configuration file generation flow chart of a method for realizing Clickhouse data migration process automation provided by an embodiment of the present invention;

[0072] Figure 4 A data migration configuration file uploading flow chart of a method for realizing Clickhouse data migration process automation provided by an embodiment of the present invention;

[0073] Figure 5 The present invention provides a data migration execution flow chart of a method for realizing Clickhouse data migration process automation. DETAILED DESCRIPTION

[0074] The present invention will be described in detail below in conjunction with the accompanying drawings.

[0075] Example 1

[0076] An embodiment of the present invention provides a method for automating the ClickHouse data migration process, the method comprising running a Shell script to migrate data from a source ClickHouse database to a destination ClickHouse database;

[0077] The Shell script is run to migrate the data of the source ClickHouse database to the target ClickHouse database, such as Figure 1 As shown, the following steps are included:

[0078] S1: Obtain all source data tables under the ClickHouse database on the source side, input the name of the ClickHouse database on the target side and the name of the source data table into the database building script, execute the database building script, and generate the ClickHouse database and data table on the target side in the specified path;

[0079] S2: Pass the SQL file path of the data table under the target ClickHouse database and the ClickHouse-Copier migration configuration template file name into the ClickHouse-Copier migration configuration file script through the incoming parameters, and generate ClickHouse-Copier migration configuration files named after the target ClickHouse data table in turn. The ClickHouse-Copier migration configuration files are all stored in the migration configuration file directory;

[0080] S3: Upload the ClickHouse-Copier migration configuration file to ZooKeeper by executing the upload script;

[0081] S4: According to the ClickHouse-Copier migration configuration file in ZooKeeper, migrate the data in the data table of the source ClickHouse database to the data table of the destination ClickHouse database by executing the data migration script.

[0082] Furthermore, in step S2, the specific implementation process of the SQL file path of the data table under the target ClickHouse database is:

[0083] First, connect to the source ClickHouse database through clickhouse-client, execute the SQL to view the database creation statement, and export the result to the target database file with the suffix sql; then use the sed command to remove the line breaks and \\ symbols in the file; then connect to the source ClickHouse database through clickhouse-client, export all data table lists under the database to the target database file, and use the depth-first search algorithm to traverse the exported data table list file to see if all are exported. If it is determined that the data table list file has not been exported, iteratively export the SQL of the creation statement of each table to the target data table file through clickhouse-client. After each data table is exported, use the sed command to remove the line breaks and \\ symbols in the target data table file until the traversal of the data table list file is completed; if it is determined that the data table list file has been exported, generate the SQL file path of the data table.

[0084] Furthermore, in step S2, the specific implementation process of the ClickHouse-Copier migration configuration file script is:

[0085] The ClickHouse-Copier migration configuration file script includes a ClickHouse data table SQL file module, a ClickHouse-Copier migration configuration template file and a ClickHouse-Copier migration configuration file to generate a Jar package; first, a script tool that can read the target data table file and the ClickHouse-Copier migration configuration template file is developed using the Java programming language, and the script tool reads the target data table file and the ClickHouse-Copier migration configuration template file into memory; then the script tool extracts various parameters of the data table in the target data table file, and the various parameters are filled into the ClickHouse-Copier migration configuration template file. If the filling is completed, the script tool This tool generates a ClickHouse-Copier migration configuration file named after the ClickHouse data table; then the script tool traverses the SQL files of all data tables under the SQL file path of the target ClickHouse data table through the Shell, and filters out data tables without the ENGINE engine through the grep command; runs the Jar package in the JVM virtual machine, and the Jar package passes the SQL file path of the ClickHouse data table and the ClickHouse-Copier migration configuration template file name through the incoming parameters, and generates the ClickHouse-Copier migration configuration file named after the ClickHouse data table in turn. At the same time, the script tool generates the ClickHouse-Copier migration configuration file script.

[0086] Furthermore, the specific implementation process of uploading the ClickHouse-Copier migration configuration file to ZooKeeper by executing the upload script in step S3 is:

[0087] The upload script uploads the ClickHouse-Copier migration configuration file in four ways, and the four ways specifically include a first upload method, a second upload method, a third upload method, and a fourth upload method;

[0088] The first upload method indicates the creation or update of a single file:

[0089] The upload script passes in three parameters, which are the operation type parameter, the ClickHouse database name, and the ClickHouse data table name. The upload script executes the creation or update process of a single file:

[0090] If it is a creation operation, first determine whether there is a ClickHouse database directory in ZooKeeper. If not, create it first. If it exists, create the ClickHouse data table directory. Then read the ClickHouse-Copier migration configuration file corresponding to the data table through the ZooKeeper cat command, write it to a file named description, and save it in the ClickHouse data table directory.

[0091] If it is an update operation, the content in the description file in the ClickHouse data table directory is reset through the ZooKeeper set command;

[0092] The second upload method indicates that multiple files are created:

[0093] The upload script passes in three parameters, which are the operation type parameter, the ClickHouse database name, and the ClickHouse data table name. The upload script executes the creation process of multiple files:

[0094] First, determine whether there is a ClickHouse database directory in ZooKeeper. If not, create the ClickHouse database directory first. If it exists, create the ClickHouse data table directory. Then traverse all migration configuration files in the migration configuration file directory and create the ClickHouse data table directory. Then read the ClickHouse-Copier migration configuration file corresponding to the data table through the ZooKeeper cat command, write it to a file named description, and save it in the ClickHouse data table directory.

[0095] The third upload method indicates the creation or update of multiple files in a partition:

[0096] The upload script passes in three parameters, which are the operation type parameter, ClickHouse database name, and ClickHouse data table name. The upload script executes multiple creation or update processes for partition-based migration data:

[0097] If it is a creation operation, first determine whether there is a ClickHouse database directory in ZooKeeper. If not, create the ClickHouse database directory first. If it exists, create the ClickHouse data table directory. Then traverse all configuration files in the partition migration configuration file directory, create the ClickHouse data table directory, and then read the ClickHouse-Copier migration configuration file corresponding to the data table through the ZooKeeper cat command, write it to a file named description, and save it in the ClickHouse data table directory.

[0098] If it is an update operation, reset the content in the description file in the ClickHouse data table directory through the ZooKeeper set command;

[0099] The fourth upload method indicates the creation or update of a partitioned single file:

[0100] The upload script passes in three parameters, which are the operation type parameter, ClickHouse database name, and ClickHouse data table name. The upload script executes the creation or update process of a single partition-by-partition migration data:

[0101] If it is a creation operation, first determine whether there is a ClickHouse database directory in ZooKeeper. If not, create the ClickHouse database directory first. If it exists, create the ClickHouse data table directory. Then read the ClickHouse-Copier migration configuration file corresponding to the data table through the ZooKeeper cat command, write it to a file named description, and save it in the ClickHouse data table directory.

[0102] If it is an update operation, the content in the description file in the ClickHouse data table directory is reset through the ZooKeeper set command.

[0103] Further, in step S4, according to the ClickHouse-Copier migration configuration file in ZooKeeper, the specific implementation process of migrating the data in the data table of the source ClickHouse database to the data table of the destination ClickHouse database by executing the data migration script is:

[0104] The data migration script supports three modes of functions, including a first mode, a second mode and a third mode;

[0105] The first mode is used for full migration of a single table, and is implemented as follows:

[0106] The data migration script passes in two parameters, and the two parameters Clickhouse database name and Clickhouse data table name are spliced ​​into the data migration configuration file path. The data migration configuration file path is used to represent the path in ZooKeeper, and then the data migration process is started to migrate data according to the data migration configuration file path; if the data migration is completed, clickhouse-client will query the total number of data in the migrated data table in the source ClickHouse database and the destination ClickHouse database respectively. If the total number of data is consistent, the data migration is successful; if the total number of data is inconsistent, the data migration fails, and the cause of the data migration failure is checked and then migrated again;

[0107] The second mode is used for full migration of multiple tables. The specific implementation method is as follows:

[0108] The data migration script passes in a Clickhouse database name parameter;

[0109] Traverse the migration configuration files of all ClickHouse data tables in the copier-tasks directory under the data migration configuration file directory to obtain the names of the ClickHouse data tables;

[0110] The data migration script starts to accumulate the number of ClickHouse data tables traversed from 0, presets the total number of ClickHouse data tables, and prints the information of which table is being migrated and how many tables are left to be migrated to the data migration script operation log;

[0111] The data migration configuration file path is spliced ​​by the passed-in ClickHouse database name and the traversed ClickHouse data table name. The data migration configuration file path is used to represent the path in ZooKeeper, and then the data migration process is started to migrate the data; if the data migration is completed, clickhouse-client will query the total number of data in the migrated data table in the source ClickHouse database and the destination ClickHouse database respectively. If the total number of data is consistent, the data migration is successful; if the total number of data is inconsistent, the data migration fails, and the cause of the data migration failure is checked and then migrated again; if the number of ClickHouse data tables traversed by the data migration script is equal to the preset total number of ClickHouse data tables, the process ends;

[0112] The third mode is used for full migration of multiple tables by partition, and the specific implementation method is as follows:

[0113] The data migration script passes in a Clickhouse database name parameter;

[0114] Traverse the migration configuration files of all ClickHouse data tables in the copier-tasks directory under the data migration configuration file directory to obtain the names of the ClickHouse data tables;

[0115] The data migration script starts to accumulate the number of ClickHouse data tables traversed from 0, presets the total number of ClickHouse data tables, and prints the information of which table is being migrated and how many tables are left to be migrated to the data migration script operation log;

[0116] Traverse the migration configuration files of all ClickHouse data tables in the incre_partition_copier_tasks directory under the data partition migration configuration file directory, obtain the name of the ClickHouse data table, and print the information of which table is being migrated and how many tables are left to be migrated to the data migration script running log;

[0117] Read the partition difference list file, and query the partition data of this table in the destination ClickHouse database through clickhouse-client. If the total data of the data table is greater than 0 and the number of partitions is equal to 1, the partition data of this table in the destination ClickHouse database will be DROPed; if the total data of the data table is greater than 0 and the number of partitions is equal to 2, the second partition will be found through the first partition and the second partition will be DROPed;

[0118] After the data in the data table partition of the destination ClickHouse database is deleted, the data migration configuration file path is spliced ​​into the passed-in ClickHouse database name and the traversed ClickHouse data table name. The data migration configuration file path is used to represent the path in ZooKeeper, and then the data migration process is started to migrate the data; if the data migration is completed, clickhouse-client will query the total number of data in the migrated data table in the source ClickHouse database and the destination ClickHouse database respectively. If the total number of data is consistent, the data migration is successful; if the total number of data is inconsistent, the data migration fails, and the cause of the data migration failure is checked and then migrated again; if the number of ClickHouse data tables traversed by the data migration script is equal to the preset total number of ClickHouse data tables, the process ends.

[0119] Furthermore, the specific implementation process of troubleshooting the cause of data migration failure is as follows:

[0120] Collect historical migration failure data, including script input parameter error data, database account password error data, script log error data, and network error data;

[0121] Performing normalized feature extraction on the historical migration failure data by using natural language technology to generate migration failure feature data, wherein a portion of the migration failure feature data is used for a training set and a portion of the migration failure feature data is used for a validation set;

[0122] Constructing a migration failure error deep learning model, performing model training using training set data, and stopping training if the training of the migration failure error deep learning model meets a preset recognition rate, wherein the input of the migration failure error deep learning model is the migration failure feature data, and the output of the migration failure error deep learning model is the migration error type;

[0123] The trained migration failure error deep learning model is used to analyze the causes of data migration failure in real time and output the migration error type.

[0124] Furthermore, network error data includes errors caused by bandwidth, latency, and packet loss of the network connection.

[0125] Furthermore, the feature extraction of the script log error data also includes:

[0126] The convolutional neural network (CNN) is used to extract the spatial features of the script log error data, and the long short-term memory (LSTM) network is used to capture the temporal sequence relationship of the script log error data to generate migration failure feature data.

[0127] Embodiment 2, as Figure 2 , including the following steps:

[0128] 1. Execute the script through the sh command in the Linux server terminal, and pass the name of the ClickHouse database whose table creation statement is to be exported after the script;

[0129] 2. Determine whether the ck_databases and ck_tables directories exist. If they do exist, clear the files in the two directories. If they do not exist, create two new directories.

[0130] 3. Use clickhouse-client to export the database creation statement to a file named after the database with the suffix sql, and save the file to the ck_databases directory;

[0131] 4. Eliminate invalid characters in the database creation statement;

[0132] 5. Export the list of data tables in the database through clickhouse-client;

[0133] 6. Traverse the list of data tables, export the table creation statements of each data table into a file named after the data table with the suffix sql through clickhouse-client, and save the file to the ck_tables directory;

[0134] 7. Eliminate invalid characters in the database creation statement;

[0135] 8. End the run.

[0136] Embodiment 3, as Figure 3 , including the following steps:

[0137] 1. Run the sh command on the Linux server terminal to execute the script and enter the database name for which the data migration configuration file is to be generated after the script.

[0138] 2. Check whether there is a copier_tasks directory. If it does, clear the files in the directory. If it does not, create a new directory.

[0139] 3. Traverse all the data table creation files under the database directory in the ck_tables directory to determine whether there is ENGINE in the creation statement. If yes, enter the data migration configuration file generation process. If no, print a prompt message;

[0140] 4. Pass the data table creation statement file and the data migration configuration file template name to the data migration configuration file creation Jar package, start creating the data migration configuration file for the data table, and after successful creation, write the content of the configuration file into a file named after the data table with the suffix xml, and save it to the copier_tasks directory;

[0141] 5. End the run.

[0142] Embodiment 4, as Figure 4 , including the following steps:

[0143] 1. Execute the script through the sh command in the Linux server terminal, and pass the operation type parameter 1, create or update, and ClickHouse data table name after the script. At this time, the process of uploading or updating the data migration configuration file of a single data table is executed;

[0144] 2. After the ZooKeeper client connects to ZooKeeper, it determines whether there is a directory named after the database of the data table in ZooKeeper. If it does not exist, it will be created;

[0145] 3. If the second parameter is create, the ZooKeeper client creates a directory named after the data table in the database directory, and writes the contents of the data migration configuration file in the copier_tasks directory with the same name as the data table to a file named description in the data table directory;

[0146] 4. If the second parameter is update, the ZooKeeper client uses the set command to reset the description file content in the database / data table directory to the data migration configuration file content in the copier_tasks directory with the same name as the data table;

[0147] 5. End the run.

[0148] Embodiment 5, as Figure 4 , including the following steps:

[0149] 1. Execute the script through the sh command in the Linux server terminal, and pass in the operation type parameter 2, create, and ClickHouse database name after the script. At this time, the process of uploading all data migration configuration files in the copier_tasks directory is executed;

[0150] 2. After the ZooKeeper client connects to ZooKeeper, it determines whether there is a directory named after the database of the data table in ZooKeeper. If it does not exist, it will be created;

[0151] 3. Traverse all data migration configuration files in the copier_tasks directory. The ZooKeeper client creates directories named after the data table in the database directory in turn, and writes the contents of the data migration configuration files in the copier_tasks directory with the same name as the data table into a file named description in the data table directory.

[0152] 4. End the run.

[0153] Embodiment 6, as Figure 4 The method of using this embodiment includes the following steps:

[0154] 1. Execute the script through the sh command in the Linux server terminal, and pass the operation type parameter 3, create or update, and ClickHouse database name after the script. At this time, the process of uploading or updating all data migration configuration files in the incre_copier_tasks directory is executed;

[0155] 2. After the ZooKeeper client connects to ZooKeeper, it determines whether there is a directory named after the database of the data table in ZooKeeper. If it does not exist, it will be created;

[0156] 3. If the second parameter is create, all data migration configuration files in the incre_copier_tasks directory are traversed, and the ZooKeeper client creates directories named after the data table in the database directory in turn, and writes the contents of the data migration configuration files in the incre_copier_tasks directory with the same name as the data table to the file named description in the data table directory;

[0157] 4. If the second parameter is update, all data migration configuration files in the incre_copier_tasks directory are traversed, and the ZooKeeper client resets the description file content in the database / data table directory to the data migration configuration file content in the incre_copier_tasks directory with the same name as the data table;

[0158] 5. The embodiment ends operation.

[0159] Embodiment 7, as Figure 4 The method of using this embodiment includes the following steps:

[0160] 1. Execute the script through the sh command in the Linux server terminal, and pass in the operation type parameter 4, create or update, and ClickHouse data table name after the script. At this time, the process of uploading or updating the data migration configuration file with the same name as the data table in the incre_copier_tasks directory is executed;

[0161] 2. After the ZooKeeper client connects to ZooKeeper, it determines whether there is a directory named after the database of the data table in ZooKeeper. If it does not exist, it will be created;

[0162] 3. If the second parameter is create, the ZooKeeper client writes the contents of the data migration configuration file with the same name as the data table in the incre_copier_tasks directory to a file named description in the data table directory;

[0163] 4. If the second parameter is update, the ZooKeeper client resets the content of the description file in the data table directory to the content of the data migration configuration file in the incre_copier_tasks directory with the same name as the data table;

[0164] 5. The embodiment ends operation.

[0165] Embodiment 8, as Figure 5 The method of using this embodiment includes the following steps:

[0166] 1. Execute the script through the sh command in the Linux server terminal, and pass the operation type parameter 1 and the ClickHouse data table name after the script. At this time, the process of migrating a single data table is executed;

[0167] 2. clickhouse-copier obtains the configuration file information of data migration from ZooKeeper through the ZooKeeper configuration file zookeeper.xml and the passed-in ClickHouse data table name, starts the data migration process, completes the data migration, and ends the service process;

[0168] 3. Clickhouse-client connects to the data source cluster and the data destination cluster respectively, counts the total number of data in the migrated data table, and determines whether the total number of data tables on both ends is consistent. If the total number of data is consistent, it prints the information that the data migration is successful; if it is inconsistent, it prints the information that the data migration failed; if the total number of data on the data source is 0, it prompts the user to manually create a table on the data destination;

[0169] 4. The embodiment ends operation.

[0170] Embodiment 9, as Figure 5 The method of using this embodiment includes the following steps:

[0171] 1. Run the script in the Linux server terminal using the sh command and pass in the operation type parameter 2 after the script to execute the migration process of multiple data tables.

[0172] 2. Traverse all data table migration configuration files in the copier_tasks directory, obtain the data table names, pass them to clickhouse-copier one by one, and execute the following steps;

[0173] 3. clickhouse-copier obtains the configuration file information of data migration from ZooKeeper through the ZooKeeper configuration file zookeeper.xml and the passed-in ClickHouse data table name, starts the data migration process, completes the data migration, and ends the service process;

[0174] 4. Clickhouse-client connects to the data source cluster and the data destination cluster respectively, counts the total number of data in the migrated data table, and determines whether the total number of data tables on both ends is consistent. If the total number of data is consistent, it prints the information that the data migration is successful; if it is inconsistent, it prints the information that the data migration failed; if the total number of data on the data source is 0, it prompts the user to manually create a table on the data destination;

[0175] 5. After a loop is completed, continue to execute from step 3 to step 4 until all data table migration configuration files in the copier_tasks directory are traversed;

[0176] 6. This embodiment ends operation.

[0177] Embodiment 10, as Figure 5 The method of using this embodiment includes the following steps:

[0178] 1. Run the script in the Linux server terminal using the sh command and pass in the operation type parameter 4 after the script. This will execute the process of migrating multiple data tables by partition.

[0179] 2. Traverse all data table migration configuration files in the incre_copier_tasks directory to obtain the data table name;

[0180] 3. Enter the partition data difference directory, read the file named after the data table, and then count the number of partitions in the data table and the amount of data in each partition and store them in the variable;

[0181] 4. If the total number of partition data is greater than 0 and there is only one partition, delete the partition through clickhouse-client; if the total number of partition data is greater than 0 and there are two partitions, delete both partitions through clickhouse-client;

[0182] 5. If the total number of partition data = 0, no operation is required and go directly to the next step;

[0183] 7. clickhouse-copier obtains the configuration file information of data migration from ZooKeeper through the ZooKeeper configuration file zookeeper.xml and the passed-in ClickHouse data table name, starts the data migration process, completes the data migration, and ends the service process;

[0184] 8. Clickhouse-client connects to the data source cluster and the data destination cluster respectively, counts the total number of data in the migrated data table, and determines whether the total number of data tables on both ends is consistent. If the total number of data is consistent, it will print the information that the data migration is successful; if it is inconsistent, it will print the information that the data migration failed; if the total number of data on the data source is 0, it will prompt the user to manually create a table on the data destination;

[0185] 9. After a loop is completed, continue to execute from step 3 to step 8 until all data table migration configuration files in the incre_copier_tasks directory are traversed.

[0186] The embodiment ends running.

[0187] The algorithm and display provided herein are not inherently related to any particular computer, virtual system or other device. Various general purpose systems can also be used together with the teachings based on this. According to the above description, it is obvious that the structure required for constructing such systems. In addition, the present invention is not directed to any specific programming language either. It should be understood that various programming languages ​​can be utilized to realize the content of the present invention described herein, and the description of the above specific languages ​​is for disclosing the best mode of the present invention.

[0188] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0189] Similarly, it should be understood that in order to streamline the present disclosure and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting the intention that the claimed invention requires more features than those explicitly recited in each claim. More specifically, as reflected in the claims below, inventive aspects lie in less than all the features of the individual embodiments disclosed above. Therefore, the claims that follow the specific embodiment are hereby expressly incorporated into the specific embodiment, with each claim itself serving as a separate embodiment of the present invention.

[0190] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0191] In addition, those skilled in the art will appreciate that, although some embodiments herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.

[0192] The various component embodiments of the present invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) may be used in practice to implement some or all of the functions of some or all of the components in the apparatus according to an embodiment of the present invention. The present invention may also be implemented as a device or device program (e.g., a computer program and a computer program product) for executing part or all of the methods described herein. Such a program implementing the present invention may be stored on a computer-readable medium, or may have the form of one or more signals. Such a signal may be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

Claims

1. A method for automating the Clickhouse data migration process, characterized in that: The method includes running a Shell script to migrate data from a source ClickHouse database to a destination ClickHouse database; The running of the Shell script to migrate the data of the source ClickHouse database to the target ClickHouse database includes the following steps: S1: Obtain all source data tables under the ClickHouse database from the source end, input the name of the ClickHouse database on the destination end and the name of the source data table into the database building script, execute the database building script, and generate the ClickHouse database and data table on the destination end in the specified path; S2: Pass the SQL file path of the data table under the target ClickHouse database and the ClickHouse-Copier migration configuration template file name into the ClickHouse-Copier migration configuration file script through the incoming parameters, and generate ClickHouse-Copier migration configuration files named after the target ClickHouse data table in turn. The ClickHouse-Copier migration configuration files are all stored in the migration configuration file directory; S3: Upload the ClickHouse-Copier migration configuration file to ZooKeeper by executing the upload script; S4: According to the ClickHouse-Copier migration configuration file in ZooKeeper, migrate the data in the data table of the source ClickHouse database to the data table of the destination ClickHouse database by executing the data migration script.

2. According to claim 1, a method for realizing Clickhouse data migration process automation is characterized in that: In step S2, the specific implementation process of the SQL file path of the data table under the target ClickHouse database is: First, connect to the source ClickHouse database through clickhouse-client, execute the SQL to view the database creation statement, and export the result to the target database file with the suffix sql; then use the sed command to remove the line breaks and \\ symbols in the file; then connect to the source ClickHouse database through clickhouse-client, export all data table lists under the database to the target database file, and use the depth-first search algorithm to traverse the exported data table list file to see if all are exported. If it is determined that the data table list file has not been exported, iterate through clickhouse-client to export the SQL of the creation statement of each table to the target data table file. After each data table is exported, use the sed command to remove the line breaks and \\ symbols in the target data table file until the traversal of the data table list file is completed; If it is determined that the data table list file has been exported, the SQL file path of the data table is generated.

3. According to claim 2, a method for realizing Clickhouse data migration process automation is characterized in that: In step S2, the specific implementation process of the ClickHouse-Copier migration configuration file script is: The ClickHouse-Copier migration configuration file script includes a ClickHouse data table SQL file module, a ClickHouse-Copier migration configuration template file and a ClickHouse-Copier migration configuration file generation Jar package; first, a script tool that can read the target data table file and the ClickHouse-Copier migration configuration template file is developed using the Java programming language, and the script tool reads the target data table file and the ClickHouse-Copier migration configuration template file into memory; then the script tool extracts various parameters of the data table in the target data table file, and the various parameters are filled into the ClickHouse-Copier migration configuration template file. If the filling is completed, the script tool generates a ClickHouse-Copier migration configuration file named after the ClickHouse data table; Then the script tool uses Shell to traverse the SQL files of all data tables under the SQL file path of the target ClickHouse data table, and filters out data tables without the ENGINE engine through the grep command; the Jar package is run in the JVM virtual machine, and the Jar package passes the SQL file path of the ClickHouse data table and the ClickHouse-Copier migration configuration template file name through the input parameters, and generates the ClickHouse-Copier migration configuration file named after the ClickHouse data table in turn. At the same time, the script tool generates the ClickHouse-Copier migration configuration file script.

4. According to claim 3, a method for realizing Clickhouse data migration process automation is characterized in that: The specific implementation process of uploading the ClickHouse-Copier migration configuration file to ZooKeeper by executing the upload script in step S3 is: The upload script uploads the ClickHouse-Copier migration configuration file in four ways, and the four ways specifically include a first upload method, a second upload method, a third upload method, and a fourth upload method; The first upload method indicates the creation or update of a single file: The upload script passes in three parameters, which are the operation type parameter, the ClickHouse database name, and the ClickHouse data table name. The upload script executes the creation or update process of a single file: If it is a creation operation, first determine whether there is a ClickHouse database directory in ZooKeeper. If not, create it first. If it exists, create the ClickHouse data table directory. Then read the ClickHouse-Copier migration configuration file corresponding to the data table through the ZooKeeper cat command, write it to a file named description, and save it in the ClickHouse data table directory. If it is an update operation, the content in the description file in the ClickHouse data table directory is reset through the ZooKeeper set command; The second upload method indicates that multiple files are created: The upload script passes in three parameters, which are the operation type parameter, the ClickHouse database name, and the ClickHouse data table name. The upload script executes the creation process of multiple files: First, determine whether there is a ClickHouse database directory in ZooKeeper. If not, create the ClickHouse database directory first. If it exists, create the ClickHouse data table directory. Then traverse all migration configuration files in the migration configuration file directory and create the ClickHouse data table directory. Then read the ClickHouse-Copier migration configuration file corresponding to the data table through the ZooKeeper cat command, write it to a file named description, and save it in the ClickHouse data table directory. The third upload method indicates the creation or update of multiple files in a partition: The upload script passes in three parameters, which are the operation type parameter, ClickHouse database name, and ClickHouse data table name. The upload script executes multiple creation or update processes for partition-based migration data: If it is a creation operation, first determine whether there is a ClickHouse database directory in ZooKeeper. If not, create the ClickHouse database directory first. If it exists, create the ClickHouse data table directory. Then traverse all configuration files in the partition migration configuration file directory, create the ClickHouse data table directory, and then read the ClickHouse-Copier migration configuration file corresponding to the data table through the ZooKeeper cat command, write it to a file named description, and save it in the ClickHouse data table directory. If it is an update operation, reset the content in the description file in the ClickHouse data table directory through the ZooKeeper set command; The fourth upload method indicates the creation or update of a partitioned single file: The upload script passes in three parameters, which are the operation type parameter, ClickHouse database name, and ClickHouse data table name. The upload script executes the creation or update process of a single partition-by-partition migration data: If it is a creation operation, first determine whether there is a ClickHouse database directory in ZooKeeper. If not, create the ClickHouse database directory first. If it exists, create the ClickHouse data table directory. Then read the ClickHouse-Copier migration configuration file corresponding to the data table through the ZooKeeper cat command, write it to a file named description, and save it in the ClickHouse data table directory. If it is an update operation, the content in the description file in the ClickHouse data table directory is reset through the ZooKeeper set command.

5. According to claim 4, a method for realizing Clickhouse data migration process automation is characterized in that: In step S4, according to the ClickHouse-Copier migration configuration file in ZooKeeper, the specific implementation process of migrating the data in the data table of the source ClickHouse database to the data table of the destination ClickHouse database by executing the data migration script is: The data migration script supports three modes of functions, including a first mode, a second mode and a third mode; The first mode is used for full migration of a single table, and is implemented as follows: The data migration script passes in two parameters, and the two parameters Clickhouse database name and Clickhouse data table name are concatenated into the data migration configuration file path. The data migration configuration file path is used to represent the path in ZooKeeper, and then the data migration process is started to migrate data according to the data migration configuration file path; If the data migration is completed, clickhouse-client will query the total number of data in the migrated data table in the source ClickHouse database and the destination ClickHouse database respectively. If the total number of data is consistent, the data migration is successful. If the total number of data is inconsistent, it will prompt that the data migration has failed. Find out the cause of the data migration failure and migrate again; The second mode is used for full migration of multiple tables. The specific implementation method is as follows: The data migration script passes in a Clickhouse database name parameter; Traverse the migration configuration files of all ClickHouse data tables in the copier-tasks directory under the data migration configuration file directory to obtain the names of the ClickHouse data tables; The data migration script starts to accumulate the number of ClickHouse data tables traversed from 0, presets the total number of ClickHouse data tables, and prints the information of which table is being migrated and how many tables are left to be migrated to the data migration script operation log; The data migration configuration file path is spliced ​​by the incoming ClickHouse database name and the traversed ClickHouse data table name. The data migration configuration file path is used to represent the path in ZooKeeper, and then the data migration process is started to start migrating data; If the data migration is completed, clickhouse-client will query the total number of data in the migrated data table in the source ClickHouse database and the destination ClickHouse database respectively. If the total number of data is consistent, the data migration is successful. If the total number of data is inconsistent, it will prompt that the data migration has failed. Find out the cause of the data migration failure and migrate again; If the number of ClickHouse data tables traversed by the data migration script is equal to the total number of preset ClickHouse data tables, the process ends; The third mode is used for full migration of multiple tables by partition, and the specific implementation method is as follows: The data migration script passes in a Clickhouse database name parameter; Traverse the migration configuration files of all ClickHouse data tables in the copier-tasks directory under the data migration configuration file directory to obtain the names of the ClickHouse data tables; The data migration script starts to accumulate the number of ClickHouse data tables traversed from 0, presets the total number of ClickHouse data tables, and prints the information of which table is being migrated and how many tables are left to be migrated to the data migration script operation log; Traverse the migration configuration files of all ClickHouse data tables in the incre_partition_copier_tasks directory under the data partition migration configuration file directory, obtain the name of the ClickHouse data table, and print the information of which table is being migrated and how many tables are left to be migrated to the data migration script running log; Read the partition difference list file, and query the partition data of this table in the destination ClickHouse database through clickhouse-client. If the total data of the data table is greater than 0 and the number of partitions is equal to 1, the partition data of this table in the destination ClickHouse database will be DROPed; if the total data of the data table is greater than 0 and the number of partitions is equal to 2, the second partition will be found through the first partition and the second partition will be DROPed; After the data in the data table partition of the destination ClickHouse database is deleted, the data migration configuration file path is spliced ​​by the incoming Clickhouse database name and the traversed ClickHouse data table name. The data migration configuration file path is used to represent the path in ZooKeeper, and then the data migration process is started to start migrating data; If the data migration is completed, clickhouse-client will query the total number of data in the migrated data table in the source ClickHouse database and the destination ClickHouse database respectively. If the total number of data is consistent, the data migration is successful. If the total number of data is inconsistent, it will prompt that the data migration has failed. Find out the cause of the data migration failure and migrate again; If the number of ClickHouse data tables traversed by the data migration script is equal to the total number of preset ClickHouse data tables, the process ends.

6. According to claim 5, a method for realizing Clickhouse data migration process automation is characterized in that: The specific implementation process of troubleshooting the cause of data migration failure is as follows: Collect historical migration failure data, including script input parameter error data, database account password error data, script log error data, and network error data; Performing normalized feature extraction on the historical migration failure data by using natural language technology to generate migration failure feature data, wherein a portion of the migration failure feature data is used for a training set and a portion of the migration failure feature data is used for a validation set; Constructing a migration failure error deep learning model, performing model training using training set data, and stopping training if the training of the migration failure error deep learning model meets a preset recognition rate, wherein the input of the migration failure error deep learning model is the migration failure feature data, and the output of the migration failure error deep learning model is the migration error type; The trained migration failure error deep learning model is used to analyze the causes of data migration failure in real time and output the migration error type.

7. A method for realizing Clickhouse data migration process automation according to claim 6, characterized in that: The network error data includes errors caused by bandwidth, delay and packet loss of the network connection.

8. According to claim 7, a method for realizing Clickhouse data migration process automation is characterized in that: The feature extraction of the script log error data also includes: The convolutional neural network (CNN) is used to extract the spatial features of the script log error data, and the long short-term memory (LSTM) network is used to capture the temporal sequence relationship of the script log error data to generate migration failure feature data.

Images