Data Migration Method, Device, Equipment and Computer Readable Storage Medium

Abstract

The present invention discloses a data migration method, device, equipment and computer-readable storage medium. The method includes: circularly reading a preset configuration file, and when a migration message is read from the preset configuration file, obtaining an egress cluster server; invoking a first preset framework to remotely log in to the egress cluster server to execute, on the egress cluster server, a table creation statement for the database table in the migration message and output the table creation statement to a specified directory under a pre-specified file of the egress cluster server; invoking a second preset framework to synchronize the table creation statement on the egress cluster server and perform table formatting parsing on the synchronized table creation statement to obtain migration information and an executable table creation statement; executing the executable table creation statement to create a corresponding Hive table, and migrating corresponding data files to the Hive table based on the migration information. The present invention can solve the problems of low migration efficiency and inability to perform batch migration on demand in the existing data migration methods.

Description

Technical Field

[0001] The present invention relates to the technical field of financial technology (Fintech), and particularly to a data migration method, device, equipment and computer-readable storage medium. Background Art

[0002] With the development of computer technology, more and more technologies (big data, distributed, blockchain, artificial intelligence, etc.) are applied in the financial field. The traditional financial industry is gradually transforming into financial technology (Fintech). However, due to the security and real-time requirements of the financial industry, higher requirements are also put forward for technologies.

[0003] With the continuous development of information technology, the amount of information data continues to grow, and the data of various enterprises has also experienced explosive growth. Therefore, there is an urgent need for the ability to operate and process large-scale data. As an open-source project with distributed storage and computing capabilities, Hadoop uses a parallel computing framework for efficient distributed computing and has its own distributed file system HDFS, which can provide scalable and robust data storage. Therefore, it has quickly attracted the attention of various industries and is widely used in the fields of finance, commerce, education, etc.

[0004] As a massive data storage system, a distributed system needs to solve an important problem, which is to determine the data distribution strategy in the cluster. When the storage capacity and processing capacity of a certain database cluster reach the upper limit of the cluster capacity, it is necessary to relieve the storage pressure and load pressure of the original server through data migration. Currently, in the process of data migration, the hdfs data files and hive (a data warehouse tool based on Hadoop) metadata are usually migrated separately. Specifically, the dhfs data files are migrated across large data clusters through methods such as disk copying and distcp (distributed copying), and then the hive table structure and partition values (i.e., hive metadata) are exported and migrated in batches from the hive metastore (a service for storing hive metadata). In the above data migration process, it is necessary to manually input the clusters and library tables to be migrated in batches, resulting in low data migration efficiency and inability to perform on-demand batch migration. Summary of the Invention

[0005] The main purpose of the present invention is to provide a data migration method, device, equipment and computer-readable storage medium, aiming to solve the problems of low existing data migration efficiency and inability to perform on-demand batch migration.

[0006] To achieve the above object, the present invention provides a data migration method, and the data migration method includes:

[0007] Read the preset configuration file cyclically. When an in-migration message is read from the preset configuration file, obtain the out-migration cluster server in the in-migration message;

[0008] Call the first preset framework to remotely log in to the out-migration cluster server, execute the table creation statement of the library table in the in-migration message on the out-migration cluster server, and output the table creation statement to the specified directory under the pre-specified file of the out-migration cluster server;

[0009] Call the second preset framework to synchronize the table creation statement on the out-migration cluster server, and perform table formatting parsing on the synchronized table creation statement to obtain migration information and an executable table creation statement;

[0010] Execute the executable table creation statement to create the corresponding hive table, and migrate the corresponding data file to the hive table based on the migration information.

[0011] Optionally, the migration information includes partition information and the storage location of the data file. The step of migrating the corresponding data file to the hive table based on the migration information includes:

[0012] Judge whether the partition information is no partition or there is a partition;

[0013] If the partition information is no partition, call the preset interface to obtain the data file corresponding to the storage location of the data file, and migrate the obtained data file to the hive table.

[0014] Optionally, after the step of judging whether the partition information is no partition or there is a partition, it further includes:

[0015] If the partition information is there is a partition, call the first preset framework to remotely log in to the out-migration cluster server, execute the query of the partition value of the library table on the out-migration cluster server, and output the partition value to the specified directory under the pre-specified file of the out-migration cluster server;

[0016] Call the second preset framework to synchronize the partition value on the out-migration cluster server, and perform partition formatting parsing on the synchronized partition value to obtain an executable command for adding a partition;

[0017] Execute the command for adding a partition to add the corresponding partition to the hive table;

[0018] Call the preset interface to obtain the data file corresponding to the storage location of the data file, and migrate the obtained data file to the corresponding partition of the hive table.

[0019] Optionally, the data migration method further includes:

[0020] Obtaining the size of the data file before migration and the size of the data file after migration through the preset interface;

[0021] Detecting whether the size of the data file before migration is consistent with the size of the data file after migration;

[0022] If the size of the data file before migration is inconsistent with the size of the data file after migration, generating corresponding error prompt information.

[0023] Optionally, the first preset framework is the expect + ssh framework, the second preset framework is the expect + scp framework, and the preset interface is the distributed copy distcp interface.

[0024] In addition, to achieve the above object, the present invention further provides a data migration device, and the data migration device includes:

[0025] A first acquisition module, configured to cyclically read a preset configuration file, and when an in-migration message is read from the preset configuration file, acquire the out-migration cluster server in the in-migration message;

[0026] A remote login module, configured to remotely log in to the out-migration cluster server by invoking a first preset framework, so as to execute, on the out-migration cluster server, a table creation statement of a library table in the in-migration message, and output the table creation statement to a specified directory under a pre-specified file of the out-migration cluster server;

[0027] A statement synchronization module, configured to synchronize the table creation statement on the out-migration cluster server by invoking a second preset framework, and perform table formatting parsing on the synchronized table creation statement to obtain migration information and an executable table creation statement;

[0028] A data migration module, configured to execute the executable table creation statement to create a corresponding hive table, and migrate corresponding data files to the hive table based on the migration information.

[0029] Optionally, the migration information includes partition information and a data file storage location, and the migration module includes:

[0030] An information determination unit, configured to determine whether the partition information is no partition or there is a partition;

[0031] A first migration unit, configured to, if the partition information is no partition, invoke a preset interface to obtain a data file corresponding to the data file storage location, and migrate the obtained data file to the hive table.

[0032] Optionally, the migration module further includes:

[0033] a remote login unit, configured to, if the partition information indicates that a partition exists, call the first preset framework to remotely log in to the migration-out cluster server, query the partition value of the library table on the migration-out cluster server, and output the partition value to a specified directory under a pre-specified file on the migration-out cluster server;

[0034] A partition value synchronization unit, configured to call the second preset framework to synchronize the partition values on the migrated cluster server, and perform partition formatting and parsing on the synchronized partition values to obtain an executable command for adding a partition;

[0035] A partition adding unit, configured to execute the partition adding command to add a corresponding partition in the Hive table;

[0036] The second migration unit is configured to call the preset interface to obtain the data file corresponding to the data file storage location, and migrate the obtained data file to the partition corresponding to the hive table.

[0037] In addition, to achieve the above-mentioned purpose, the present invention also provides a data migration device, which includes: a memory, a processor, and a data migration program stored on the memory and runnable on the processor, and when the data migration program is executed by the processor, the steps of the data migration method described above are implemented.

[0038] In addition, to achieve the above-mentioned purpose, the present invention further provides a computer-readable storage medium, on which a data migration program is stored. When the data migration program is executed by a processor, the steps of the data migration method described above are implemented.

[0039] The present invention provides a data migration method, device, equipment and computer-readable storage medium. By cyclically reading a preset configuration file, when an in-migration message is read, the out-cluster server in the in-migration message is obtained; the first preset framework is called to remotely log in to the out-cluster server to execute the table creation statement of the database table in the in-migration message on the out-cluster server, and the table creation statement is output to a specified directory under a pre-specified file on the out-cluster server; the second preset framework is called to synchronize the table creation statement on the out-cluster server and perform table formatting parsing on the synchronized table creation statement to obtain migration information and an executable table creation statement; the executable table creation statement is executed to create a corresponding Hive table, and the corresponding data file is migrated to the created Hive table based on the migration information. By the above method, the present invention can realize the complete cross-cluster migration of Hive tables, and can directly realize on-demand batch migration intelligently by reading the in-migration message in the configuration file. Compared with the existing data migration process that requires manual input of the clusters and database tables to be migrated in batches, the present invention can reduce human intervention and thus improve the overall efficiency of migration. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a schematic diagram of the device structure of the hardware operating environment related to the solution of the embodiment of the present invention;

[0041] Figure 2 It is a schematic flowchart of the first embodiment of the data migration method of the present invention;

[0042] Figure 3 It is a schematic diagram of the function modules of the first embodiment of the data migration device of the present invention.

[0043] The implementation, functional features and advantages of the object of the present invention will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0044] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0045] Refer to Figure 1 , Figure 1 It is a schematic diagram of the device structure of the hardware operating environment related to the solution of the embodiment of the present invention.

[0046] The data migration device in the embodiment of the present invention can be a smart phone, or a terminal device such as a PC (Personal Computer), a tablet computer, a portable computer, etc.

[0047] Such as Figure 1As shown in the figure, the data migration device may include: a processor 1001, such as a CPU, a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to implement connection communication between these components. The user interface 1003 may include a display screen (Display) and an input unit such as a keyboard (Keyboard). Optionally, the user interface 1003 may further include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed RAM memory or a stable memory (non-volatile memory), such as a disk memory. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0048] Those skilled in the art can understand that Figure 1 the structure of the data migration device shown in does not constitute a limitation on the data migration device, and it may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0049] As Figure 1 shown, the memory 1005, as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a data migration program.

[0050] In Figure 1 the terminal shown, the network interface 1004 is mainly used to connect to the background server and communicate with the background server for data; the user interface 1003 is mainly used to connect to the client and communicate with the client for data; and the processor 1001 may be used to call the data migration program stored in the memory 1005 and execute the following steps of the data migration method.

[0051] Based on the above hardware structure, various embodiments of the data migration method of the present invention are proposed.

[0052] The present invention provides a data migration method.

[0053] Referring to Figure 2 , Figure 2 is a schematic flowchart of the first embodiment of the data migration method of the present invention.

[0054] In this embodiment, the data migration method includes:

[0055] Step S10, cyclically read a preset configuration file, and when an in-migration message is read from the preset configuration file, obtain the out-migration cluster server in the in-migration message;

[0056] With the development of computer technology and the advent of big data, the application of computer technology in financial institutions (such as banks, insurance companies, securities firms, and wealth management institutions) has become increasingly widespread. In current financial institutions, data migration is underway, and there are higher requirements for migration efficiency and accuracy. Due to the increasing amount of data, when current financial institutions perform data migration, they usually migrate the hdfs data files and hive (a data warehouse tool based on Hadoop) metadata separately. Their migration plans are isolated and incomplete, and the data migration process requires manual input of the clusters and library tables to be migrated in batches, resulting in relatively low data migration efficiency and the inability to perform on-demand batch migration. This situation seriously does not meet the requirements of financial institutions.

[0057] The data migration method in this embodiment is implemented by a data migration device. Taking the incoming cluster server as an example for illustration, a cluster server is a server that can interact with a hadoop (the infrastructure of a distributed system) cluster. The incoming cluster server refers to the server corresponding to the target hadoop cluster to which the data is migrated. For example, when migrating tables a, b, and c under library B from the B hadoop cluster to the corresponding tables a, b, and c in library E of the A hadoop cluster, at this time, the server corresponding to the A hadoop cluster is called the incoming cluster server, and the server corresponding to the B hadoop cluster is called the outgoing cluster server. In this embodiment, each cluster server will repeatedly read a preset configuration file. When an incoming message is read from the preset configuration file, that is, when an incoming message to migrate data from another cluster to this incoming cluster server is read, at this time, the outgoing cluster server in the incoming message is obtained. The preset configuration file is used to record the cluster information and library table information that need to be migrated, which can be set batch by humans or automatically set according to the data migration request when the staff initiates a data migration request through software. For example, in the above example, when it is necessary to migrate tables a, b, and c under library B from the B hadoop cluster to the corresponding tables a, b, and c in library E of the A hadoop cluster, correspondingly, the following can be configured in the preset configuration file:

[0058] B a E a

[0059] B b E b

[0060] B c E c

[0061] Correspondingly, after the A cluster server starts the main program, it can repeatedly read the preset configuration file. When an incoming message is read from the preset configuration file, the outgoing cluster server in the incoming message can be obtained as the B cluster server.

[0062] Step S20, call the first preset framework to remotely log in to the migration-out cluster server, execute on the migration-out cluster server to query the table creation statements of the library tables in the migration-in message, and output the table creation statements to the specified directory under the pre-specified file of the migration-out cluster server;

[0063] After obtaining the migration-out cluster server, call the first preset framework to remotely log in to the migration-out cluster server, execute on the migration-out cluster server to query the table creation statements of the library tables in the migration-in message, and output the table creation statements to the specified directory under the pre-specified file of the migration-out cluster server. Among them, the first preset framework is the expect+ssh framework. Expect is a software used to implement automatic interaction functions. ssh is the abbreviation of secure shell, which is an encrypted network protocol. Its common application programs include remote command-line login and remote command execution, but any network service can be protected by ssh. Under liunx (an operating system), expect can be used to achieve automatic ssh login and execute scripts.

[0064] Step S30, call the second preset framework to synchronize the table creation statements on the migration-out cluster server, and perform table formatting parsing on the synchronized table creation statements to obtain migration information and executable table creation statements;

[0065] Then, call the second preset framework to synchronize the table creation statements on the migration-out cluster server, and perform table formatting parsing on the synchronized table creation statements to obtain migration information and executable table creation statements. Among them, the second preset framework is the expect+scp framework. Expect is a software used to implement automatic interaction functions. scp is the abbreviation of secure copy, which is a transmission command for cross-server encryption. expect+scp can be used to transfer files across machines and automatically synchronize files. When performing table formatting parsing on the synchronized table creation statements, the sed (a Linux instruction) or awk (a text processing tool) text processing tool of liunx can be used for table formatting parsing. Specifically, to prevent errors caused by the table already existing in the cluster to be migrated in (cluster a in the above example), first use sed to replace the create table character with createtable if not exists, and then judge whether the table a needs to specify a storage path. If it has been specified, replace the path after location in the table creation statement with the specified path; if not, delete all the attributes after the location line in the table creation statement, and use the default path and other attributes of the table to obtain the executable table creation statement. In addition, after performing table formatting parsing, migration information can also be parsed, where the migration information includes partition information and data file storage locations.

[0066] Step S40, execute the executable table creation statement to create a corresponding Hive table, and migrate the corresponding data files to the Hive table based on the migration information.

[0067] Finally, execute the executable table creation statement to create a corresponding Hive table, and migrate the corresponding data files to the created Hive table based on the migration information.

[0068] Among them, the migration information includes partition information and the storage location of data files. The step of "migrating the corresponding data files to the Hive table based on the migration information" includes:

[0069] Step a1, determine whether the partition information indicates non - existent partitions or existent partitions;

[0070] Step a2, if the partition information indicates non - existent partitions, call a preset interface to obtain the data files corresponding to the storage location of the data files, and migrate the obtained data files to the Hive table.

[0071] If the partition information indicates non - existent partitions, at this time, call a preset interface to obtain the data files corresponding to the storage location of the data files (that is, the data files stored under the storage location of the data files), and then migrate the obtained data files to the Hive table. Among them, the preset interface is the distcp (distributed copy) interface. Of course, in a specific embodiment, the data files can also be migrated by means of disk copying.

[0072] In addition, after step a2, it further includes:

[0073] Step a3, if the partition information indicates existent partitions, call the first preset framework to remotely log in to the source cluster server to execute a query on the partition values of the library table on the source cluster server, and output the partition values to a specified directory under a pre - specified file on the source cluster server;

[0074] If the partition information indicates that a partition exists, then it is necessary to call the first preset framework to remotely log in to the migrated cluster server to execute the partition value of the query library table on the migrated cluster server, and output the partition value to the specified directory under the pre-specified file of the migrated cluster server. Among them, the first preset framework is the expect+ssh framework. Expect is a software used to implement automatic interaction functions. SSH is the abbreviation of secure shell, which is an encrypted network protocol. Its common applications include remote command line login and remote command execution. However, any network service can be secured through SSH. Under liunx (an operating system), expect can be used to implement SSH automatic login and script execution.

[0075] Step a4: calling the second preset framework to synchronize the partition values on the migrated cluster server, and performing partition formatting and parsing on the synchronized partition values to obtain an executable command for adding a partition;

[0076] Then, call the second preset framework to synchronize the partition values on the migrated cluster server, and perform partition formatting and parsing on the synchronized partition values to obtain an executable command to add partitions. Among them, the second preset framework is the expect+scp framework. When performing partition formatting and parsing on the synchronized partition values, you can use liunx's sed or awk text processing tools to perform table formatting and parsing. Specifically, first process the content in the partition file: 1) first delete the first line in the partition file, 2) replace / with ', 3) replace = with =', 4) add partition(, 5) at the end of each line; then, splice use E; alter table A add if not exists characters, and it will be formatted into an executable command to add partitions in hive. For example, in the above example, the partition value found on table a in cluster B is:

[0077] partition

[0078] ds=20150529 / city=sz

[0079] ds=20150529 / city=sh

[0080] After processing the contents of the partition file by calling the expect+scp framework, the partition can be converted into:

[0081] partition(ds='20150529',city='sz')

[0082] partition(ds='20190519',city='sh')

[0083] Finally, the executable add statement format in Cluster A can be obtained as follows:

[0084] use E; alter table A add if not exists partition(ds='20150529',city='sz')

[0085] partition(ds='20190519',city='sh')

[0086] Step a5: Execute the command to add the partition to add the corresponding partition to the Hive table;

[0087] Step a6: Call the preset interface to obtain the data file corresponding to the storage location of the data file, and migrate the obtained data file to the corresponding partition of the Hive table.

[0088] After obtaining the executable command to add the partition, execute the command to add the partition to add the corresponding partition to the created Hive table. Furthermore, call the preset interface to obtain the data file corresponding to the storage location of the data file, and migrate the obtained data file to the corresponding partition of the Hive table. Among them, the preset interface is the distcp interface.

[0089] An embodiment of the present invention provides a data migration method. By circularly reading a preset configuration file, when an in-migration message is read, the out-cluster server in the in-migration message is obtained; the first preset framework is called to remotely log in to the out-cluster server to execute the table creation statement of the database table in the in-migration message on the out-cluster server, and output the table creation statement to the specified directory under the pre-specified file of the out-cluster server; the second preset framework is called to synchronize the table creation statement on the out-cluster server, and perform table formatting parsing on the synchronized table creation statement to obtain migration information and an executable table creation statement; execute the executable table creation statement to create a corresponding Hive table, and migrate the corresponding data file to the created Hive table based on the migration information. Through the above method, the embodiment of the present invention can achieve complete cross-cluster migration of Hive tables, and can directly realize on-demand batch migration intelligently by reading the in-migration message in the configuration file. Compared with the existing data migration process that requires manual input of the clusters and database tables to be migrated one by one, the embodiment of the present invention can reduce human intervention, thereby improving the overall efficiency of migration.

[0090] Further, based on Figure 2 the first embodiment shown, a second embodiment of the data migration method of the present invention is proposed.

[0091] In this embodiment, the data migration method further includes:

[0092] Step A, obtaining the size of the data file before migration and the size of the data file after migration through the preset interface;

[0093] In this embodiment, after the data migration is completed, the sizes of the data files before and after migration can be verified to detect whether the data is completely migrated. Specifically, the size of the data file before migration and the size of the data file after migration can be obtained through this preset interface, where the preset interface is the dictcp interface.

[0094] Step B, detecting whether the size of the data file before migration is consistent with the size of the data file after migration;

[0095] Step C, if the size of the data file before migration is not consistent with the size of the data file after migration, generating a corresponding error prompt message.

[0096] Then, detect whether the size of the data file before migration is consistent with the size of the data file after migration. If the size of the data file before migration is not consistent with the size of the data file after migration, it means that there is a problem in the data migration process. At this time, a corresponding error prompt message is generated. Specifically, the error prompt message can be displayed in the log file.

[0097] In the prior art, another script needs to be written to verify the sizes of the data files before and after migration to detect whether the data migration is successful. In this embodiment, when migrating through the distcp interface, the sizes of the data files before and after migration can be directly obtained, and then the verification can be automatically performed. Therefore, intelligent and automatic data verification can be realized to detect whether the data migration is successful.

[0098] The present invention can be specifically applied to the data migration process in financial institutions. For example, when a banking institution migrates its computer room, it is necessary to migrate the data on the old cluster in the original computer room to the new cluster in the new computer room. Migration information can be set in a preset configuration file in advance, and then the main programs of the cluster servers corresponding to each new cluster in the new computer room are started. As a result, each cluster server reads the preset configuration file in a loop. When an in-migration message is read from the preset configuration file, the out-migration cluster server in the in-migration message is obtained; the first preset framework is called to remotely log in to the out-migration cluster server to execute the table creation statement of the database table in the in-migration message on the out-migration cluster server, and output the table creation statement to the specified directory under the pre-specified file of the out-migration cluster server; the second preset framework is called to synchronize the table creation statement on the out-migration cluster server and perform table formatting analysis on the synchronized table creation statement to obtain migration information and an executable table creation statement; the executable table creation statement is executed to create the corresponding Hive table, and the corresponding data file is migrated to the created Hive table based on the migration information. Through the above method, a complete cross-cluster migration of Hive tables can be achieved, and intelligent on-demand batch migration can be directly realized by reading the in-migration message in the configuration file, so as to migrate the data on the old cluster to the new cluster. Compared with the need to manually input the clusters and database tables to be migrated one by one in the existing data migration process, the embodiment of the present invention can reduce human intervention and thus improve the overall efficiency of migration.

[0099] The present invention also provides a data migration device.

[0100] Referring to Figure 3 , Figure 3 is a schematic diagram of the functional modules of the first embodiment of the data migration device of the present invention.

[0101] As Figure 3 shown, the data migration device includes:

[0102] The first acquisition module 10 is configured to read the preset configuration file in a loop, and when an in-migration message is read from the preset configuration file, obtain the out-migration cluster server in the in-migration message;

[0103] The remote login module 20 is configured to call the first preset framework to remotely log in to the out-migration cluster server to execute the table creation statement of the database table in the in-migration message on the out-migration cluster server, and output the table creation statement to the specified directory under the pre-specified file of the out-migration cluster server;

[0104] The statement synchronization module 30 is configured to call the second preset framework to synchronize the table creation statement on the out-migration cluster server and perform table formatting analysis on the synchronized table creation statement to obtain migration information and an executable table creation statement;

[0105] The data migration module 40 is configured to execute the executable table creation statement to create a corresponding hive table, and migrate the corresponding data file to the hive table based on the migration information.

[0106] Furthermore, the migration information includes partition information and data file storage location, and the migration module 40 includes:

[0107] An information judging unit, configured to judge whether the partition information indicates that the partition does not exist or exists;

[0108] The first migration unit is configured to, if the partition information indicates that the partition does not exist, call a preset interface to obtain a data file corresponding to the data file storage location, and migrate the obtained data file to the hive table.

[0109] Furthermore, the migration module 30 further includes:

[0110] a remote login unit, configured to, if the partition information indicates that a partition exists, call the first preset framework to remotely log in to the migration-out cluster server, query the partition value of the library table on the migration-out cluster server, and output the partition value to a specified directory under a pre-specified file on the migration-out cluster server;

[0111] A partition value synchronization unit, configured to call the second preset framework to synchronize the partition values on the migrated cluster server, and perform partition formatting and parsing on the synchronized partition values to obtain an executable command for adding a partition;

[0112] A partition adding unit, configured to execute the partition adding command to add a corresponding partition in the Hive table;

[0113] The second migration unit is configured to call the preset interface to obtain the data file corresponding to the data file storage location, and migrate the obtained data file to the partition corresponding to the hive table.

[0114] Furthermore, the data migration device further includes:

[0115] A second acquisition module is used to acquire the size of the data file before migration and the size of the data file after migration through the preset interface;

[0116] A size detection module, configured to detect whether the size of the data file before migration is consistent with the size of the data file after migration;

[0117] The error prompt module is configured to generate corresponding error prompt information if the size of the data file before migration is inconsistent with the size of the data file after migration.

[0118] Further, the first preset framework is the expect + ssh framework, the second preset framework is the expect + scp framework, and the preset interface is the distributed copy distcp interface.

[0119] Among them, the function realization of each module in the above data migration device corresponds to each step in the above data migration method embodiment, and its function and implementation process will not be elaborated here one by one.

[0120] The present invention also provides a computer-readable storage medium, on which a data migration program is stored. When the data migration program is executed by a processor, the steps of the data migration method described in any one of the above embodiments are realized.

[0121] The specific embodiments of the computer-readable storage medium of the present invention are basically the same as those of the above data migration method embodiments, and will not be elaborated here.

[0122] It should be noted that in this article, the terms "including", "comprising" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or system. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of another identical element in the process, method, article or system including that element.

[0123] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments.

[0124] Through the description of the above embodiments, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium as described above (such as ROM / RAM, magnetic disk, optical disc), and includes several instructions for causing a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in various embodiments of the present invention.

[0125] The above are only the preferred embodiments of the present invention, and do not limit the patent scope of the present invention accordingly. Any equivalent structure or equivalent process transformation made by using the specification and drawings of the present invention, or directly or indirectly applied to other related technical fields, shall be equally included in the patent protection scope of the present invention.

Claims

1. A data migration method, characterized in that, The data migration method includes: Read the preset configuration file in a loop. When an in-migration message is read from the preset configuration file, obtain the out-migration cluster server in the in-migration message; Call the first preset framework to remotely log in to the out-migration cluster server, execute the table creation statement of the library table in the in-migration message on the out-migration cluster server, and output the table creation statement to the specified directory under the pre-specified file of the out-migration cluster server; Call the second preset framework to synchronize the table creation statement on the out-migration cluster server, and perform table formatting analysis on the synchronized table creation statement to obtain migration information and an executable table creation statement; the migration information includes partition information and the data file storage location; Execute the executable table creation statement to create a corresponding Hive table, and determine whether the partition information indicates no partition or there is a partition; If the partition information indicates no partition, call the preset interface to obtain the data file corresponding to the data file storage location, and migrate the obtained data file to the Hive table; If the partition information indicates there is a partition, call the first preset framework to remotely log in to the out-migration cluster server, execute the query of the partition values of the library table on the out-migration cluster server, and output the partition values to the specified directory under the pre-specified file of the out-migration cluster server; Call the second preset framework to synchronize the partition values on the out-migration cluster server, and use the sed or awk text processing tool of Liunx to perform partition formatting analysis on the synchronized partition values to obtain an executable partition addition command; Execute the partition addition command to add a corresponding partition to the Hive table; Call the preset interface to obtain the data file corresponding to the data file storage location, and migrate the obtained data file to the corresponding partition of the Hive table.

2. The data migration method according to claim 1, wherein The data migration method further includes: Obtain the size of the data file before migration and the size of the data file after migration through the preset interface; Detect whether the size of the data file before migration is consistent with the size of the data file after migration; If the size of the data file before migration is inconsistent with the size of the data file after migration, generate a corresponding error prompt message.

3. The data migration method according to any one of claims 1 to 2, characterized in that The first preset framework is the expect+ssh framework, the second preset framework is the expect+scp framework, and the preset interface is the distributed copy distcp interface.

4. A data migration device, characterized in that, The data migration device includes: The first acquisition module is used to read the preset configuration file in a loop. When an in-migration message is read from the preset configuration file, obtain the out-migration cluster server in the in-migration message; The remote login module is used to call the first preset framework to remotely log in to the out-migration cluster server, execute the query of the table creation statement of the library table in the in-migration message on the out-migration cluster server, and output the table creation statement to the specified directory under the pre-specified file of the out-migration cluster server; A statement synchronization module, configured to call a second preset framework to synchronize the table creation statements on the migrated cluster server, and perform table formatting parsing on the synchronized table creation statements to obtain migration information and executable table creation statements; the migration information includes partition information and data file storage locations; A data migration module, configured to execute the executable table creation statements to create corresponding Hive tables, and determine whether the partition information indicates non-existent partitions or existing partitions; if the partition information indicates non-existent partitions, call a preset interface to obtain data files corresponding to the data file storage locations, and migrate the obtained data files to the Hive tables; if the partition information indicates existing partitions, call the first preset framework to remotely log in to the migrated cluster server to execute queries on the partition values of the library tables on the migrated cluster server, and output the partition values to a specified directory under a pre-specified file on the migrated cluster server; call the second preset framework to synchronize the partition values on the migrated cluster server, and perform partition formatting parsing on the synchronized partition values using sed or awk text processing tools of Liunx to obtain executable commands for adding partitions; execute the commands for adding partitions to add corresponding partitions to the Hive tables; call the preset interface to obtain data files corresponding to the data file storage locations, and migrate the obtained data files to the corresponding partitions of the Hive tables.

5. A data migration device, characterized in that, The data migration device includes: a memory, a processor, and a data migration program stored on the memory and executable on the processor, and when the data migration program is executed by the processor, the steps of the data migration method according to any one of claims 1 to 3 are implemented.

6. A computer-readable storage medium, characterized in that, A data migration program is stored on the computer-readable storage medium, and when the data migration program is executed by a processor, the steps of the data migration method according to any one of claims 1 to 3 are implemented.

Images