P-level data migration method and system

Abstract

The embodiment of the present application provides a P-level data migration method and system. The method includes: obtaining an HDFS directory file that needs to be migrated, and obtaining a scan file based on the HDFS directory file that needs to be migrated; creating a target cluster, wherein the target file is configured in the target cluster creation based on the scan file; migrating the HDFS directory file that needs to be migrated according to the configured target file; after completing the migration, verifying the migration data according to preset rules, judging whether the HDFS directory file of the source cluster and the HDFS directory file in the target cluster after the migration are consistent, and outputting a migration completion signal if they are consistent. The present invention verifies the consistency of data by comparing the number and size of files in the corresponding directories of the source cluster and the target cluster and the number of tables and the number of rows in the corresponding Hive library, thereby improving the migration efficiency and accuracy of P-level data.

Description

Technical Field

[0001] The present application relates to the technical field of data migration, and in particular to a P-level data migration method and a P-level data migration system. Background Art

[0002] Hadoop is a distributed system infrastructure. Users can develop distributed programs without understanding the underlying details of the distributed system, and make full use of the power of the cluster for high-speed computing and storage. Hadoop processes data in a reliable, efficient, and scalable way. Because of this feature, Hadoop is often used to store historical data of enterprises. For example, the historical data of the financial system. With the expansion of the scale of the data center system and the need for rapid business development, the historical data of each system is increasing. The financial system generally retains various historical data of about 10 years, and stores historical data by building a Hadoop cluster. The hardware facilities also basically use X86-based servers.

[0003] In recent years, the financial system has gradually implemented the localization of server equipment, piloted at the infrastructure (IAAS) layer, introduced domestic Huawei ARM servers, and migrated some big data applications on X86 servers to ARM servers. That is, using domestic equipment based on the Hadoop architecture, rebuilding the Hadoop cluster, and migrating the original cluster data to the new cluster, while verifying the data consistency of the new and old clusters.

[0004] Existing Hadoop clusters often store a very large amount of data, reaching the P level, that is, the total amount of data exceeds 1024T. It is very difficult to migrate such a large amount of data. Existing data migration methods have problems such as large bandwidth requirements, difficulty in consistency verification, and the risk of Java memory overflow. To address these problems, a new P-level data migration method needs to be created. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a P-level data migration method and system to at least solve the problems of large bandwidth requirements, difficulty in consistency verification and Java memory overflow risks in existing data migration methods.

[0006] In order to achieve the above-mentioned objectives, the first aspect of the present application provides a P-level data migration method, which is implemented based on the Hadoop architecture and is used to transfer P-level data from a source cluster to a target cluster. The method includes: obtaining an HDFS directory file that needs to be migrated, and obtaining a scan file based on the HDFS directory file that needs to be migrated; creating a target cluster, wherein a target file is configured in the target cluster creation based on the scan file; migrating the HDFS directory file that needs to be migrated according to the configured target file; after completing the migration, verifying the migration data according to preset rules to determine whether the HDFS directory file of the source cluster is consistent with the HDFS directory file in the target cluster after the migration, and outputting a migration completion signal if they are consistent.

[0007] In an embodiment of the present application, obtaining the HDFS directory file that needs to be migrated includes: collecting data migration requirements entered by users, and extracting the time range required for migration from the data migration requirements; scanning the source cluster file directory based on the time range, classifying the data within the time range as stock data, and classifying the data outside the time range as incremental data; and using the stock data as the HDFS directory file that needs to be migrated.

[0008] In an embodiment of the present application, obtaining a scan file according to the HDFS directory file that needs to be migrated includes: performing a recursive scan of the HDFS directory file that needs to be migrated to obtain a corresponding scan file; wherein the scan file includes: an HDFS file list and an HDFS directory list.

[0009] In an embodiment of the present application, the target file configuration in the target cluster creation based on the scanned file includes: creating a target directory in the target cluster; and in the target cluster, restoring the corresponding target cluster permissions based on the HDFS file directory permissions of the source cluster.

[0010] In an embodiment of the present application, the creating a directory in the target cluster includes: creating a corresponding target directory in the target cluster according to the HDFS directory list; wherein the created target directory corresponds to and is consistent with the HDFS directory list.

[0011] In an embodiment of the present application, in the target cluster, permissions of the corresponding target cluster are restored based on the HDFS file directory permissions of the source cluster, including: collecting users and corresponding permission information of the HDFS file list in the source cluster; based on the collected users and corresponding permission information, modifying the corresponding users and corresponding permission information of the HDFS file list in the target cluster.

[0012] In an embodiment of the present application, the method also includes: during the migration process of the HDFS directory file that needs to be migrated, in response to a trigger signal for the front-end directory file of the HDFS directory file to start transmitting, starting a logging function to record the log signal of the migration process in real time.

[0013] In an embodiment of the present application, the migration data verification is performed according to preset rules, including: performing HDFS directory file size and number verification, including: calculating the file size and file number of the HDFS directory files in the target cluster; determining whether the file size and file number of the HDFS directory files to be migrated match the file size and file number of the HDFS directory files in the target cluster.

[0014] In an embodiment of the present application, the migration data verification according to preset rules also includes: performing MD5 value verification of the HDFS directory file, including: calculating the MD5 value of the corresponding HDFS directory file according to the HDFS directory file list of the scanned file, as the MD5 value of the HDFS directory file of the source cluster; calculating the MD5 value of the corresponding HDFS directory file according to the directory file list in the target cluster after migration as the MD5 value of the HDFS directory file of the target cluster; and determining whether the MD5 value of the HDFS directory file of the source cluster is consistent with the MD5 value of the HDFS directory file of the target cluster.

[0015] In an embodiment of the present application, the migration data verification according to preset rules also includes: performing HIVE database verification, including: mapping the target cluster to a corresponding HIVE database table; performing a spot check on the corresponding HIVE database table according to preset spot check rules to determine whether the access performance of the HIVE database table and the data consistency with the source cluster meet expectations; wherein the preset spot check rules include at least: spot checking according to a preset date and spot checking according to the importance of the data.

[0016] The second aspect of the present application provides a P-level data migration system, including: a collection unit, used to obtain HDFS directory files that need to be migrated; a processing unit, used to: obtain a scan file according to the HDFS directory files that need to be migrated; create a target cluster, wherein the target file is configured in the target cluster creation based on the scan file; a migration unit, used to migrate the HDFS directory files that need to be migrated according to the configured target files; a verification unit, used to verify the migration data according to preset rules after completing the migration, and determine whether the HDFS directory files of the source cluster are consistent with the HDFS directory files in the target cluster after the migration; an output unit, used to output a migration completion signal when the verification unit determines that the HDFS directory files of the source cluster are consistent with the HDFS directory files in the target cluster after the migration.

[0017] In an embodiment of the present application, obtaining the HDFS directory file that needs to be migrated includes: during the migration process of the HDFS directory file that needs to be migrated, in response to a trigger signal for the front-end directory file of the HDFS directory file to start transmitting, starting a logging function; the system also includes: a logging unit for recording log signals during the migration process in real time.

[0018] A third aspect of the present application provides a machine-readable storage medium having instructions stored thereon, which, when executed by a processor, configure the processor to execute the above-mentioned P-level data migration method.

[0019] A fourth aspect of the present application provides a computer program product, including a computer program, which implements the above-mentioned P-level data migration method when executed by a processor.

[0020] Through the above technical solution, the target file is configured in the target cluster creation through the HDFS directory file that needs to be migrated, and then the file is migrated based on the configuration file. After the file migration, consistency verification is performed, and the data migration of multiple HDFS directories and different Hive data partitions is realized through HDFS and Hive data storage directories. The risk of memory overflow is prevented by configuring the JVM running memory parameters of the Hadoop client. The consistency of the data is verified by comparing the number and size of files in the corresponding directories of the source cluster and the target cluster and the number of tables and the number of rows in the corresponding Hive library, thereby improving the migration efficiency and accuracy of P-level data.

[0021] Other features and advantages of the embodiments of the present application will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the following specific implementations, they are used to explain the embodiments of the present application, but do not constitute a limitation on the embodiments of the present application. In the accompanying drawings:

[0023] Figure 1 A flowchart of the steps of a P-level data migration method according to an embodiment of the present application is schematically shown;

[0024] Figure 2 A flowchart schematically shows the steps of data consistency verification of a P-level data migration method according to an embodiment of the present application;

[0025] Figure 3 A system structure diagram of a P-level data migration system according to an embodiment of the present application is schematically shown;

[0026] Figure 4 The internal structure diagram of a computer device according to an embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the specific implementation methods described herein are only used to illustrate and explain the embodiments of the present application, and are not used to limit the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0028] It should be noted that if the embodiments of the present application involve directional indications (such as up, down, left, right, front, back...), the directional indications are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0029] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present application, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0030] Hadoop is a distributed system infrastructure developed by the Apache Foundation. Users can develop distributed programs without understanding the underlying details of the distribution. Take full advantage of the power of the cluster for high-speed computing and storage. Hadoop implements a distributed file system, one of which is HDFS (Hadoop Distributed File System). HDFS has the characteristics of high fault tolerance and is designed to be deployed on low-cost hardware; it also provides high throughput to access application data, which is suitable for applications with large data sets. Hadoop processes data in a reliable, efficient and scalable manner. Because of this feature of Hadoop, it is often used to store enterprise historical data. For example, the historical data of the financial system, with the expansion of the scale of the data center system and the need for rapid business development, the historical data of each system is increasing. At present, the financial system generally retains various historical data of about 10 years, and will store historical data by building a Hadoop cluster, and the hardware facilities basically use X86-based servers. In recent years, the financial system has gradually implemented the localization of server equipment, piloted at the infrastructure (IAAS) layer, introduced domestic Huawei ARM servers, and migrated some big data applications on X86 servers to ARM servers. Using domestic equipment based on the Hadoop architecture, the Hadoop cluster was rebuilt, and the original cluster data was migrated to the new cluster, while verifying the data consistency of the new and old clusters.

[0031] In existing Hadoop clusters, the amount of data stored is often very large, reaching the P level, that is, the total amount of data exceeds 1024T. It is very difficult to migrate such a large amount of data. The existing data migration methods mainly have the following problems.

[0032] The first type is that the source cluster has nearly 4TB of data that needs to be archived every day, and nearly 30,000 batches are running on the cluster. It is not possible to migrate offline by stopping the cluster, and online migration must be performed to ensure that there is no impact on the original cluster.

[0033] The second type is that because the data volume is very large, if the data migration needs to be completed in a very short time, the required bandwidth is very large. In addition, during the data migration, there will be a continuous stream of new data or updated data, that is, the data changes dynamically, which also makes it very difficult to verify the data consistency during the migration process.

[0034] The third type is that Hive tables have internal and external tables, some of which have three partitions, such as hds-datamode, hds-sdate, and hds-edate. However, some tables have only one data partition, which increases the difficulty of migration and verification. Furthermore, due to the huge amount of data during the migration process, there is a risk of Java memory overflow when executing the migration and verification scripts through Spark.

[0035] The solution of the present invention is based on the above-mentioned problem, and proposes a P-level data migration method and system. The specific implementation concept is to divide the data into full data and incremental data based on the Hadoop architecture, and to solve the problem of data fluctuation by corresponding different migration and verification methods. Online migration is achieved by building mutual trust between the source cluster and the target cluster, limiting the migration time and building a dedicated migration bandwidth. Data migration of multiple directories of HDFS and different partitions of Hive data is achieved by snapshotting the HDFS and Hive data storage directories. The risk of memory overflow is prevented by configuring the JVM running memory parameters of the Hadoop client. Data consistency is verified by comparing the number and size of files in the corresponding directories of the source cluster and the target cluster and the number of tables and the number of rows in the corresponding Hive library.

[0036] Figure 1 The flowchart of the P-level data migration method according to the embodiment of the present application is schematically shown. Figure 1 As shown, in one embodiment of the present application, a P-level data migration method is provided, comprising the following steps:

[0037] Step S10: Obtain the HDFS directory file that needs to be migrated, and obtain the scan file according to the HDFS directory file that needs to be migrated.

[0038] Specifically, it is known that because the data in the source cluster is dynamically changing, a huge amount of new data will flow in every day, that is, there is a problem that the target data migration has not been completed and is covered by new data. In order to solve this problem, it is necessary to classify the data, that is, fix the data that needs to be migrated, and discard the new data to avoid interference of the new data on the migrated data. Based on this, the data in the source cluster is divided into stock data and incremental data. Among them, the stock data is the fixed data that needs to be transferred, and the incremental data is the data that is allowed to change dynamically to avoid interference. Preferably, the distinction between stock data and incremental data is based on the time node, because the longer the data is stored, the lower the probability of its change. For example, if the preset time node is November 5, the data between the storage and November 5 will be regarded as stock data, and all the data after this time will be regarded as incremental data. By defining the stock data, its inherent nature is guaranteed to avoid changes that cause migration and verification difficulties.

[0039] The stored data is taken as the HDFS directory file that needs to be migrated, and then the HDFS directory file that needs to be migrated is recursively scanned to obtain a complete HDFS file list and HDFS directory list.

[0040] Step S20: creating a target cluster, wherein target file configuration is performed in the target cluster creation based on the scan file.

[0041] Specifically, before data migration, the target directory in the target cluster is first created. Based on the created target directory, the directionality of subsequent data migration is guaranteed. And unlike conventional methods, after the data is completed, the data is reorganized based on the original data, which increases the difficulty of executing subsequent steps. In this application, the target directory is created first, which realizes data migration of different partitions and improves migration efficiency. Therefore, when creating the target directory, it must be the same as the directory in the source cluster. Therefore, based on the HDFS directory list obtained above, the corresponding target cluster HDFS directory list is created. The two need to correspond one to one, so that the files under the source cluster directory can be migrated to the directory of the target cluster based on the corresponding relationship.

[0042] In addition to creating the target file, you also need to restore the permissions of the target cluster to build mutual trust between the source cluster and the target cluster, as well as subsequent normal performance. Get the user or user group with permissions for each HDFS file in the source cluster, and obtain their corresponding permission information. Then, in the constructed target directory, modify the information of the corresponding directory file, and change its permissions to the corresponding user or user group. The specific permission information is also the same as the permission information in the source cluster.

[0043] Step S30: Migrate the HDFS directory files that need to be migrated according to the configured target file.

[0044] Specifically, after completing the target file configuration, that is, the mutual trust relationship between the source cluster and the target cluster and the creation of the target directory have been completed, data migration is performed based on the fixed pointing correspondence. Specifically, the distcp tool is used to migrate the HDFS file from the source cluster to the corresponding directory of the target cluster. Its input parameter is the HDFS file list, and the last layer of HDFS file directory after migration is output. Based on the output last layer of HDFS file directory trigger signal, it is determined that the data migration is completed. In an embodiment of the present invention, based on the pre-established mutual trust relationship and target directory creation between the source cluster and the target cluster, the migration time can be customized and a dedicated migration bandwidth can be established to realize online migration. Because the data has good directionality and there is no interfering data, its transmission efficiency is very high, the dependence on bandwidth is reduced, and it will not cause too much bandwidth burden.

[0045] Step S40: After completing the migration, the migration data is verified according to preset rules to determine whether the HDFS directory file of the source cluster is consistent with the HDFS directory file in the target cluster after the migration, and a migration completion signal is output if they are consistent.

[0046] Specifically, this method provides three data verification rules, namely, HDFS file size and quantity verification, HDFS file MD5 value verification, and HIVE database verification. Among them, HDFS file size and quantity verification is a preliminary verification, which is used to verify whether there are omissions in the overall file transfer; HDFS file MD5 value verification is an advanced verification, which is used to verify the consistency of source cluster files and target cluster files; HIVE database verification is the ultimate verification, which is used to verify whether the functions of the files in the target cluster meet expectations. Figure 2 , including the following steps:

[0047] Step S401: Verify the size and quantity of HDFS files.

[0048] Specifically, in response to the data migration completion signal, the size and number of files in the HDFS file directory of the migrated data are calculated in the target cluster, and then the size and number of files in the HDFS file directory are compared between the source cluster and the target cluster to determine the consistency of the size and number of files in the HDFS file directory of the source cluster and the target cluster. If the two are consistent, it means that the HDFS directory list in the data to be migrated is the same as the HDFS directory list in the target cluster, and the migration of all target files has been completed. If the two are inconsistent, it means that there are omissions in the migration, then the corresponding migration log record is retrieved, the legacy data is screened based on the migration process information in the log record, and the missing data is re-migrated until the migration is completed.

[0049] Step S402: Verify the MD5 value of the HDFS file.

[0050] Specifically, the MD5 Message-Digest Algorithm is a widely used cryptographic hash function that can generate a 128-bit (16-byte) hash value to ensure that information transmission is complete and consistent. The principle of the MD5 algorithm can be briefly described as follows: the MD5 code processes the input information in 512-bit groups, and each group is divided into 16 32-bit sub-groups. After a series of processing, the output of the algorithm consists of four 32-bit groups. After the four 32-bit groups are cascaded, a 128-bit hash value will be generated. It is often used for password management, electronic signatures, and spam filtering.

[0051] The solution of the present invention conveniently utilizes the characteristics of the MD5 algorithm to verify the consistency of the source cluster and the target cluster. First, the MD5 value of the corresponding HDFS file is calculated based on the HDFS file list in the scan file, which is used as the MD5 value of the HDFS file of the source cluster. Then, the MD5 value of the corresponding HDFS file is calculated based on the file list of the target cluster after the file transfer, which is used as the MD5 value of the HDFS file of the target cluster. Finally, the MD5 value of the HDFS file of the source cluster is compared with the MD5 value of the HDFS file of the target cluster to determine whether the two are consistent. If the two are the same, it means that the consistency is the same.

[0052] Step S403: Perform HIVE database verification.

[0053] Specifically, HIVE is a data warehouse tool based on Hadoop, which is used for data extraction, transformation, and loading. It is a mechanism that can store, query, and analyze large-scale data stored in Hadoop. The HIVE data warehouse tool can map structured data files to a database table, and provides SQL query functions, which can convert SQL statements into MapReduce tasks for execution. The advantage of HIVE is that it has a low learning cost and can implement fast MapReduce statistics through similar SQL statements, making MapReduce simpler without having to develop dedicated MapReduce applications. Therefore, HIVE is very suitable for statistical analysis of data warehouses.

[0054] The solution of the present invention is to use its characteristics suitable for statistical analysis to verify data functionality. The target cluster verifies HIVE data information, the number of HIVE tables in each database, the number and data of HIVE tables for spot checks, and the access of HIVE libraries of different HIVE users. First, the target cluster is mapped to the corresponding HIVE database table, and then its access performance and consistency with the source cluster data are judged based on the preset spot check rules to see whether they meet expectations. For example, the tables that need to be sampled are sorted out by database, and the sampling date is determined. The sampling date of each library can be dispersed. Alternatively, the source cluster and the target cluster count the number of rows of the sampling table of each database, the partition table is counted by partition, and the non-partitioned table (the data has not changed after relocation) is counted in full. Compare the statistical results to see if the number of data rows in the corresponding tables is the same. For the sampled data tables, export the text to check whether each table can read the data normally and whether the data display is normal. For tables where no data is found, it is necessary to go to the source cluster to check the reason.

[0055] In one embodiment, the importance of each type of data is marked. For files with higher importance, the need to ensure their correct transfer is higher, so when spot checking, priority is given to spot checking important data. Important tables are sorted out from all data tables as spot check targets. When verifying the spot check targets, it is necessary to judge whether the table is within the normal range and whether it is consistent with the old table. Because even if the directory is the same and the file name is the same, it may still lead to the situation where the data transfer is indeed inaccessible, so the necessity of functional verification is very high, and only the migration data that meets the expected function is meaningful data.

[0056] Specifically, when it is determined that the files in the target cluster are consistent with the files in the source cluster in terms of directory and function, and it is determined that the data has been completely and correctly migrated, a corresponding migration completion signal is output to remind the user to review.

[0057] In one embodiment, Figure 3 As shown, a P-level data migration system is provided, including: a collection unit, used to obtain HDFS directory files that need to be migrated; a processing unit, used to: obtain a scan file according to the HDFS directory files that need to be migrated; create a target cluster, wherein the target file is configured in the target cluster creation based on the scan file; a migration unit, used to migrate the HDFS directory files that need to be migrated according to the configured target files; a verification unit, used to verify the migration data according to preset rules after completing the migration, and determine whether the HDFS directory files of the source cluster are consistent with the HDFS directory files in the target cluster after the migration; an output unit, used to output a migration completion signal when the verification unit determines that the HDFS directory files of the source cluster are consistent with the HDFS directory files in the target cluster after the migration.

[0058] In one embodiment, obtaining the HDFS directory files that need to be migrated includes: during the migration process of the HDFS directory files that need to be migrated, in response to a trigger signal for the front-end directory file of the HDFS directory file to start transmitting, starting a logging function; the system also includes: a logging unit for recording log signals during the migration process in real time.

[0059] The processor includes a kernel, which calls the corresponding program unit from the memory. One or more kernels can be set, and the P-level data migration method is implemented by adjusting kernel parameters.

[0060] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0061] An embodiment of the present application provides a storage medium on which a program is stored. When the program is executed by a processor, the above-mentioned P-level data migration method is implemented.

[0062] An embodiment of the present application provides a processor, which is used to run a program, wherein the above-mentioned P-level data migration method is executed when the program is running.

[0063] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor A01, a network interface A02, a memory (not shown in the figure) and a database (not shown in the figure) connected through a system bus. Among them, the processor A01 of the computer device is used to provide computing and control capabilities. The memory of the computer device includes an internal memory A03 and a non-volatile storage medium A04. The non-volatile storage medium A04 stores an operating system B01, a computer program B02 and a database (not shown in the figure). The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 in the non-volatile storage medium A04. The network interface A02 of the computer device is used to communicate with an external terminal through a network connection. When the computer program B02 is executed by the processor A01, a P-level data migration method is implemented.

[0064] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0065] In one embodiment, the P-level data migration system provided by the present application can be implemented in the form of a computer program. The computer program can be Figure 4 The computer device shown in the figure is run. The memory of the computer device can store program modules constituting the P-level data migration system. The computer program composed of each program module enables the processor to execute the steps of the P-level data migration method of each embodiment of the present application described in this specification.

[0066] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0067] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0068] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0069] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0070] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0071] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0072] Computer readable media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0073] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0074] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A P-level data migration method, implemented based on Hadoop architecture, for transferring P-level data from a source cluster to a target cluster, characterized in that: The method comprises: Collect the data migration requirements entered by users and extract the time range required for migration from the data migration requirements; Scan the source cluster file directory based on the time range, classify the data within the time range as stock data, and classify the data outside the time range as incremental data; The stock data is used as an HDFS directory file that needs to be migrated, and a scan file is obtained according to the HDFS directory file that needs to be migrated; Creating a target cluster, wherein target file configuration is performed in the target cluster creation based on the scan file; Migrate the HDFS directory files that need to be migrated according to the configured target file; After the migration is completed, the migration data is verified according to the preset rules to determine whether the HDFS directory file of the source cluster is consistent with the HDFS directory file in the target cluster after the migration, and a migration completion signal is output if they are consistent; wherein, The verification of migration data according to preset rules includes: Verify the size and number of files in the HDFS directory, including: Calculate the file size and number of files in the HDFS directory in the target cluster; Determine whether the file size and the number of files of the HDFS directory files to be migrated match the file size and the number of files of the HDFS directory files in the target cluster.

2. The method according to claim 1, characterized in that The step of obtaining a scan file according to the HDFS directory file that needs to be migrated includes: Perform a recursive scan of the HDFS directory files that need to be migrated to obtain corresponding scan files; wherein the scan files include: HDFS file listing and HDFS directory listing.

3. The method according to claim 2, characterized in that The configuring of the target file in the target cluster creation based on the scanned file includes: Create a target directory in the target cluster; and In the target cluster, permissions of the corresponding target cluster are restored based on the HDFS file directory permissions of the source cluster.

4. The method according to claim 3, characterized in that The creating a directory in the target cluster includes: A corresponding target directory is created in the target cluster according to the HDFS directory list; wherein the created target directory corresponds to and is consistent with the HDFS directory list.

5. The method according to claim 3, characterized in that: In the target cluster, restoring the permissions of the corresponding target cluster based on the HDFS file directory permissions of the source cluster includes: Collect the user and corresponding permission information of the HDFS file list in the source cluster; Based on the collected user and corresponding permission information, the corresponding user and corresponding permission information of the HDFS file list are modified in the target cluster.

6. The method according to claim 1, characterized in that The method further comprises: During the migration process of the HDFS directory files that need to be migrated, in response to a trigger signal for the front-end directory file of the HDFS directory files to start transmitting, a log recording function is started to record log signals during the migration process in real time.

7. The method according to claim 1, characterized in that The verifying the migration data according to the preset rules also includes: Verify the MD5 value of the HDFS directory file, including: According to the HDFS directory file list of the scanned file, the MD5 value of the corresponding HDFS directory file is calculated as the MD5 value of the HDFS directory file of the source cluster; According to the directory file list in the target cluster after migration, calculate the MD5 value of the corresponding HDFS directory file as the MD5 value of the HDFS directory file of the target cluster; Determine whether the MD5 value of the HDFS directory file of the source cluster is consistent with the MD5 value of the HDFS directory file of the target cluster.

8. The method according to claim 7, characterized in that The verifying the migration data according to the preset rules also includes: Perform HIVE database verification, including: Map the target cluster to the corresponding HIVE database table; Perform a spot check on the corresponding HIVE database table according to the preset spot check rules to determine whether the access performance of the HIVE database table and the data consistency with the source cluster meet expectations; wherein, The preset sampling rules at least include: Random checks are conducted based on preset dates and based on the importance of the data.

9. A P-level data migration system, characterized in that: The system comprises: Acquisition unit for: Collect the data migration requirements entered by users and extract the time range required for migration from the data migration requirements; Scan the source cluster file directory based on the time range, classify the data within the time range as stock data, and classify the data outside the time range as incremental data; The existing data is used as the HDFS directory file that needs to be migrated; A processing unit for: Obtain a scan file according to the HDFS directory file that needs to be migrated; Creating a target cluster, wherein target file configuration is performed in the target cluster creation based on the scan file; A migration unit, used to migrate the HDFS directory files that need to be migrated according to the configured target files; A verification unit is used to verify the migration data according to preset rules after completing the migration, and determine whether the HDFS directory file of the source cluster is consistent with the HDFS directory file in the target cluster after the migration; wherein, The verification of migration data according to preset rules includes: Verify the size and number of files in the HDFS directory, including: Calculate the file size and number of files in the HDFS directory in the target cluster; Determine whether the file size and file quantity of the HDFS directory file to be migrated match the file size and file quantity of the HDFS directory file in the target cluster; The output unit is used to output a migration completion signal when the verification unit determines that the HDFS directory file of the source cluster is consistent with the HDFS directory file in the target cluster after migration.

10. The system according to claim 9, characterized in that Obtain the HDFS directory files that need to be migrated, including: During the migration process of the HDFS directory file that needs to be migrated, in response to a trigger signal for starting the transmission of the frontmost directory file of the HDFS directory file, starting a log recording function; The system further comprises: The log unit is used to record the log signals in the migration process in real time.

11. A machine-readable storage medium having instructions stored thereon, characterized in that: When the instruction is executed by a processor, the processor is configured to execute the P-level data migration method according to any one of claims 1 to 8.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the P-level data migration method according to any one of claims 1 to 8.

Images