Data migration method, device and equipment based on Hadoop snapshot and storage medium

Abstract

The invention discloses a data migration method, device and equipment based on a Hadoop snapshot and a storage medium, and relates to the technical field of big data storage and processing, and the method comprises the steps: obtaining a path parameter of a data source cluster sent by a user side, and determining a data position of a to-be-processed data source based on the path parameter; creating a target snapshot of the to-be-processed data source by using a Hadoop command based on the data position, and generating a timestamp corresponding to creation of the target snapshot; and determining to-be-migrated data from the to-be-processed data source according to the target snapshot and the timestamp, and migrating the to-be-migrated data to a target cluster by using a preset thread. In this way, the migration efficiency can be greatly improved by creating the source cluster data snapshot, recognizing the incremental data according to the snapshot timestamp and processing the data transmission in parallel in combination with the multi-thread technology.

Description

Technical Field

[0001] The present invention relates to the technical field of big data storage and processing, and in particular to a data migration method, device, equipment and storage medium based on Hadoop snapshots. Background Art

[0002] In the era of big data, data volumes are exploding, prompting businesses and organizations to continuously upgrade, optimize, or expand their data storage and processing architectures. Data migration is a critical component of this process. Traditional data migration methods present numerous challenges. For example, full migrations require copying large amounts of data, which is time-consuming and consumes significant network bandwidth. Incremental migrations struggle to accurately identify and migrate newly added or changed data, leading to data inconsistencies. Furthermore, operations are complex, rely heavily on manual intervention, and are inefficient.

[0003] Hadoop, a widely used big data processing platform, offers a new approach to data migration through its snapshot feature. Hadoop snapshots create read-only copies of the file system at a specific moment in time, recording data status and accurately capturing data changes, providing strong support for full and incremental data migrations. However, a mature full and incremental data migration solution based on Hadoop snapshots is currently lacking.

[0004] Therefore, developing an efficient and reliable data migration method is a problem that needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of this, the present invention aims to provide a data migration method, apparatus, device, and storage medium based on Hadoop snapshots, which can reduce data migration costs and improve data processing efficiency. The specific solution is as follows:

[0006] In a first aspect, the present application discloses a data migration method based on Hadoop snapshots, comprising:

[0007] Obtaining the path parameters of the data source cluster sent by the user terminal, and determining the data location of the data source to be processed based on the path parameters;

[0008] Creating a target snapshot of the data source to be processed based on the data location and using Hadoop commands, and generating a timestamp corresponding to the creation of the target snapshot;

[0009] Data to be migrated is determined from the data source to be processed according to the target snapshot and the timestamp, and the data to be migrated is migrated to the target cluster using a preset thread.

[0010] Optionally, obtaining a path parameter of a data source cluster sent by the user terminal and determining a data location of the data source to be processed based on the path parameter includes:

[0011] Get the path parameters of the data source cluster sent by the user;

[0012] If the path parameter is an HDFS path, the HDFS path is verified using the Hadoop API to determine whether the HDFS path is legal, and if the HDFS path is legal, the HDFS path is parsed to determine the data location of the data source to be processed;

[0013] If the path parameters are Hive database names and table names, a connection is established with the Metastore through the Hive metadata service, and a query request for the data source to be processed is sent according to the Hive database name and table name to determine the data location of the data source to be processed based on the metadata information returned by the Metastore.

[0014] Optionally, creating a target snapshot of the data source to be processed based on the data location and using a Hadoop command, and generating a timestamp corresponding to the creation of the target snapshot, includes:

[0015] Determining whether the data source to be processed allows creation of a snapshot based on the data location;

[0016] If snapshot creation is not allowed, send the corresponding snapshot creation instruction to the NameNode of the Hadoop cluster through the Hadoop management command to enable the snapshot creation function;

[0017] A Hadoop snapshot creation command is used to create a target snapshot of the data source to be processed, and a timestamp corresponding to the creation of the target snapshot is generated.

[0018] Optionally, determining the data to be migrated from the data source to be processed according to the target snapshot and the timestamp, and migrating the data to be migrated to the target cluster using a preset thread includes:

[0019] determining all data in the to-be-processed data source as data to be migrated according to the target snapshot and the timestamp;

[0020] The data to be migrated is copied to the corresponding HDFS path of the target cluster using a preset thread and a preset network transmission tool to complete the migration of the data to be migrated.

[0021] Optionally, copying the data to be migrated to a corresponding HDFS path of the target cluster using a preset thread and a preset network transmission tool to complete the migration of the data to be migrated includes:

[0022] Processing the data to be migrated using a preset data processing method to obtain processed data, and determining a target number of preset threads based on the amount of the processed data; the preset data processing method includes data compression processing and data block processing;

[0023] The processed data is copied to the corresponding HDFS path of the target cluster using the target number of preset threads and the preset network transmission tool to complete the migration of the data to be migrated.

[0024] Optionally, determining the data to be migrated from the data source to be processed according to the target snapshot and the timestamp, and migrating the data to be migrated to the target cluster using a preset thread includes:

[0025] Obtaining a historical snapshot corresponding to a previous timestamp based on the timestamp, and determining data to be migrated from the data source to be processed by using the target snapshot and the historical snapshot;

[0026] If the data to be migrated is incremental data, the incremental data is added to the corresponding position of the target cluster using a preset thread;

[0027] If the data to be migrated is modified data, using the preset thread to replace old data corresponding to the modified data in the target cluster with the modified data;

[0028] If the data to be migrated is deleted data, the deleted data is deleted from the target cluster using the preset thread.

[0029] Optionally, after determining the data to be migrated from the data source to be processed according to the target snapshot and the timestamp, and migrating the data to be migrated to the target cluster using a preset thread, the method further includes:

[0030] During the data migration process, a preset data verification mechanism is used to verify the data information of the first data in the to-be-processed data source and the second data in the target cluster to determine whether the second data is completely identical to the first data;

[0031] If the second data is inconsistent with the first data, the preset data repair process is started to jump to the step of determining the data to be migrated from the data source to be processed according to the target snapshot and the timestamp, and migrating the data to be migrated to the target cluster using a preset thread.

[0032] In a second aspect, the present application discloses a data migration device based on Hadoop snapshots, comprising:

[0033] A data location determination module, configured to obtain the path parameters of the data source cluster sent by the user terminal, and determine the data location of the data source to be processed based on the path parameters;

[0034] A data snapshot module is used to create a target snapshot of the data source to be processed based on the data location and using Hadoop commands, and generate a timestamp corresponding to the creation of the target snapshot;

[0035] The data migration module is configured to determine the data to be migrated from the data source to be processed according to the target snapshot and the timestamp, and migrate the data to be migrated to the target cluster using a preset thread.

[0036] In a third aspect, the present application discloses an electronic device, comprising:

[0037] Memory, used to store computer programs;

[0038] A processor is configured to execute the computer program to implement the aforementioned data migration method based on Hadoop snapshot.

[0039] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program, wherein the computer program implements the aforementioned Hadoop snapshot-based data migration method when executed by a processor.

[0040] It can be seen that the path parameters of the data source cluster sent by the user end are obtained, and the data location of the data source to be processed is determined based on the path parameters; based on the data location and using Hadoop commands, a target snapshot of the data source to be processed is created, and a corresponding timestamp is generated for the target snapshot creation; based on the target snapshot and the timestamp, the data to be migrated is determined from the data source to be processed, and the data to be migrated is migrated to the target cluster using a preset thread. That is, by creating a snapshot of the source cluster data, identifying incremental data based on the snapshot timestamp, and combining multi-threading technology to process data transmission in parallel, the migration efficiency can be greatly improved. By creating a snapshot of the source cluster data, identifying incremental data based on the snapshot timestamp, and combining multi-threading technology to process data transmission in parallel, the migration efficiency can be greatly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0042] Figure 1This is a flow chart of a data migration method based on Hadoop snapshots disclosed in this application;

[0043] Figure 2 This is a schematic diagram of the structure of a data migration device based on Hadoop snapshot disclosed in this application;

[0044] Figure 3 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0046] Hadoop, a widely used big data processing platform, offers a new approach to data migration through its snapshot functionality. Hadoop snapshots create read-only copies of the file system at a specific moment in time, recording data status and accurately capturing data changes, providing powerful support for both full and incremental data migrations. Therefore, this application will specifically introduce a data migration method based on Hadoop snapshots, which can significantly improve migration efficiency.

[0047] See also Figure 1 As shown, the embodiment of the present application discloses a data migration method based on Hadoop snapshot, including:

[0048] Step S11: Acquire the path parameters of the data source cluster sent by the user terminal, and determine the data location of the data source to be processed based on the path parameters.

[0049] In this embodiment, obtaining the path parameters of the data source cluster sent by the user and determining the data location of the data source to be processed based on the path parameters includes: obtaining the path parameters of the data source cluster sent by the user; if the path parameters are HDFS paths, validating the HDFS paths using the Hadoop API to determine whether the HDFS paths are valid; and if the HDFS paths are valid, parsing the HDFS paths to determine the data location of the data source to be processed; if the path parameters are Hive database and table names, establishing a connection with the Metastore via the Hive metadata service, and sending a query request for the data source to be processed based on the Hive database and table names, to determine the data location of the data source to be processed based on the metadata information returned by the Metastore. Specifically, the relevant path parameters of the data source cluster sent by the user are obtained and analyzed to ultimately determine the specific data location of the data source to be processed. Specifically, the source cluster path parameters received from the user can be a specific Hadoop Distributed File System (HDFS) directory path, a Hive database name, or a Hive table name. If an HDFS (Hadoop Distributed File System) path is entered, the Hadoop API first verifies the path's validity. This verification process checks whether the path conforms to HDFS naming conventions, exists in the cluster, and whether the user has access permissions. If the path is valid, the path structure is parsed, identifying the first-level directory as the database name and the second-level directory as the table name. This parsing method clearly defines the data's source and ownership, facilitating subsequent migration operations. If a Hive (Hadoop-based data warehouse infrastructure) database or table name is entered, the corresponding HDFS path is retrieved using Hive metadata services (such as the Metastore). During the query process, a connection is established with the Metastore, a query request is sent, and the returned metadata is parsed to accurately locate the data's actual storage location in HDFS. This step ensures the source data location for the migration operation is clearly defined, laying the foundation for subsequent migration processes.

[0050] Step S12: creating a target snapshot of the data source to be processed based on the data location and using Hadoop commands, and generating a timestamp corresponding to the creation of the target snapshot.

[0051] In this embodiment, creating a target snapshot of the data source to be processed based on the data location and using a Hadoop command, and generating a timestamp corresponding to the creation of the target snapshot, includes: judging whether the data source to be processed allows snapshot creation based on the data location; if snapshot creation is not allowed, sending a corresponding snapshot creation instruction to the NameNode of the Hadoop cluster through a Hadoop management command to enable the snapshot creation function; creating a target snapshot of the data source to be processed using a Hadoop snapshot creation command, and generating a timestamp corresponding to the creation of the target snapshot.

[0052] Specifically, if the data to be migrated is full data, for the determined source data path, check whether it allows the creation of snapshots. If not, enable the snapshot function through Hadoop management commands. When enabling the snapshot function, it is necessary to send corresponding instructions to the NameNode of the Hadoop cluster to ensure that the snapshot can be created smoothly later. Use Hadoop commands to create a full snapshot of the source data and record the creation timestamp. The recording of timestamps is very important for subsequent incremental migration and data tracing, as it can accurately identify the status and change time of the data. If the data to be migrated is incremental data, create a new snapshot of the source data again and record the new timestamp. The creation of a new snapshot can capture data changes since the last full or incremental migration.

[0053] Step S13: determining the data to be migrated from the data source to be processed according to the target snapshot and the timestamp, and migrating the data to be migrated to the target cluster using a preset thread.

[0054] In this embodiment, determining the data to be migrated from the data source to be processed based on the target snapshot and the timestamp, and migrating the data to be migrated to the target cluster using a preset thread, includes: determining all data in the data source to be processed as the data to be migrated based on the target snapshot and the timestamp; copying the data to be migrated to the corresponding HDFS path of the target cluster using a preset thread and a preset network transmission tool to complete the migration of the data to be migrated.

[0055] Specifically, using preset threads and a preset network transmission tool to copy the data to be migrated to the corresponding HDFS path of the target cluster to complete the migration of the data to be migrated includes: processing the data to be migrated using a preset data processing method to obtain processed data, and determining a target number of preset threads based on the volume of the processed data; the preset data processing method includes data compression and data segmentation; and copying the processed data to the corresponding HDFS path of the target cluster using the target number of preset threads and the preset network transmission tool to complete the migration of the data to be migrated. Specifically, for the full amount of data, multi-threaded tasks are appropriately allocated based on the source data scale and target cluster performance. When allocating threads, factors such as the number and size of source data files, as well as the network bandwidth and disk I / O (Input / Output) capacity of the target cluster are comprehensively considered. Each thread is responsible for copying a portion of data files from the source data snapshot to the corresponding HDFS path of the target cluster via a network transmission tool (such as DistCp or SCP). During the copy process, a data verification mechanism (such as MD5 hashing) is used to ensure data integrity. An MD5 hash check calculates a hash value for each data file and recalculates it on the destination cluster. The two hash values ​​are compared to ensure consistency. Any inconsistency indicates data corruption during transfer. After the copy is complete, the integrity and correctness of the data files are verified on the destination cluster. If errors are detected, the files are promptly retransmitted. Full migration information (such as source path, destination path, migration time, and data volume) is recorded in log files and a database for easy query and management. These records can help administrators understand migration history, analyze potential issues, and optimize migration strategies.

[0056] In this embodiment, the data to be migrated is determined from the data source to be processed based on the target snapshot and the timestamp, and the data to be migrated is migrated to the target cluster using a preset thread, including: obtaining a historical snapshot corresponding to the previous timestamp based on the timestamp, and determining the data to be migrated from the data source to be processed using the target snapshot and the historical snapshot; if the data to be migrated is incremental data, adding the incremental data to the corresponding position of the target cluster using a preset thread; if the data to be migrated is modified data, replacing the old data corresponding to the modified data in the target cluster with the modified data using the preset thread; if the data to be migrated is deleted data, deleting the deleted data from the target cluster using the preset thread. Specifically, for incremental data, the timestamps of the new snapshot and the full snapshot (or the previous incremental snapshot) are compared, and by analyzing the snapshot metadata, data files (newly added, modified, or deleted files) that have changed within the time interval between the two snapshots are identified. When analyzing the metadata, information such as the file creation time, modification time, and file size are compared to accurately identify the incremental data. The number of threads is dynamically adjusted based on the resource availability of the target cluster, with each thread responsible for migrating a portion of the incremental data files to the target cluster. During the migration process, modified data files overwrite the old versions in the target cluster; new files are directly added to the corresponding locations in the target cluster; and deletions are executed in the target cluster. Similar to full migrations, a data validation mechanism is used to ensure the integrity and consistency of incremental data migrations, and data verification is performed after the migration is complete. Detailed information about the incremental migration (such as the incremental data file list, migration time, and data volume changes) is recorded in log files and databases to facilitate tracking of data changes and migration status.

[0057] In addition, this application proposes a visual interface that displays the data migration progress in real time, including information such as the amount of data migrated, the amount of data remaining, the migration speed, and the estimated completion time. This interface will utilize intuitive charts and numerical displays to provide users with a clear understanding of the migration progress. Upon completion, the migration results will be displayed, including detailed information such as the success rate, the total amount of data migrated, and the duration of both full and incremental migrations. Furthermore, a monitoring system will be established to monitor key metrics such as network status, disk I / O, CPU (Central Processing Unit), and memory usage during the migration process. If anomalies (such as network outages or insufficient disk space) occur, prompt alerts will be issued and relevant information will be recorded, enabling administrators to quickly identify and resolve issues, ensuring the stability and reliability of the migration process. The monitoring system will set appropriate thresholds and issue prompt alerts when indicators exceed these thresholds. Detailed information about the anomaly will also be recorded, providing a basis for subsequent troubleshooting and resolution.

[0058] It should be noted here that data conflicts are inevitable during the full and incremental migration processes, such as when a file with the same name already exists in the target cluster and the content is inconsistent. In response to this situation, the present invention sets a priority strategy to decide which version of the data to retain based on factors such as data update time and data importance. For example, if the newly migrated data is updated later, the old data in the target cluster will be overwritten; if critical business data is involved, the administrator can manually intervene in the decision. To ensure data consistency, before each migration operation, the data that may be affected in the target cluster is backed up. The backup data will be stored in a safe location, such as a dedicated backup directory or other storage media. If errors or data inconsistencies occur during the migration process, it can be quickly rolled back to the state before the migration to ensure data reliability. The rollback operation will restore the original state of the target cluster based on the backup data to ensure that the data is not affected by abnormal conditions during the migration process.

[0059] That is, after determining the data to be migrated from the data source to be processed according to the target snapshot and the timestamp, and migrating the data to be migrated to the target cluster using a preset thread, it also includes: during the data migration process, using a preset data verification mechanism to verify the data information of the first data in the data source to be processed and the second data in the target cluster to determine whether the second data is exactly the same as the first data; if the second data is inconsistent with the first data, starting a preset data repair process to jump to the step of determining the data to be migrated from the data source to be processed according to the target snapshot and the timestamp, and migrating the data to be migrated to the target cluster using a preset thread. As mentioned above, when an error occurs in data migration, a rollback operation can be performed to ensure that data migration is not affected.

[0060] Furthermore, the migration process is repeated multiple times, and with each completed migration, the amount of data required for subsequent incremental migrations gradually decreases. After each migration, the number of threads and resource allocation strategy are dynamically adjusted. As the amount of data decreases, the number of threads is appropriately reduced to avoid wasted resources. At the same time, data transmission algorithms are optimized to improve migration efficiency. Advanced technologies, such as data compression and block-based transmission, are employed to reduce both the amount and time of data transferred. This approach gradually shortens migration time, ultimately enabling incremental migrations to be kept within a relatively short timeframe (e.g., within one minute), meeting real-time data requirements. Furthermore, various parameters during the migration process are monitored and analyzed in real time, and optimization strategies are continuously adjusted based on actual conditions to achieve optimal migration results.

[0061] After the data migration is complete, this application synchronizes the metadata between the source and target clusters. For Hive table data migration, key metadata such as the table structure, partition information, and storage location in the Hive metadata store (e.g., a MySQL database) is synchronized to the target cluster. During metadata synchronization, specialized tools or scripts are used to ensure accurate metadata replication and updates. In this application, the migrated data is fully verified by comparing hash values, file counts, directory structure, and other information between the source and target data. If data inconsistencies or missing data are detected, a data repair process is initiated to re-migrate or supplement missing data to ensure that the target data is fully consistent with the source data. The data repair process provides targeted processing based on specific error conditions, such as retransmitting damaged files or supplementing missing files, to ensure data integrity and consistency. Furthermore, various errors may occur during the data migration process, such as network failures, file corruption, and insufficient permissions. To address these situations, the present invention has designed a comprehensive error handling and retry mechanism. When an error is detected, the system first records detailed error information, including the time, location, and type of error. Errors are then categorized and handled based on their type. For recoverable errors, such as temporary network outages, the system automatically retries the operation. The number of retries and the interval can be configured to ensure the migration can continue after the network is restored. For unrecoverable errors, such as irreparable file corruption, the system promptly notifies the administrator and provides a detailed error report for manual intervention and resolution.

[0062] Furthermore, to ensure security during data migration, the present invention utilizes secure authentication and encrypted transmission technologies. Before data migration, identity authentication is performed on both the source and target clusters to ensure that only authorized users and systems can perform data migration operations. Identity authentication can be performed using methods such as username and password authentication and certificate authentication to ensure the security of the migration process. During data transmission, data is encrypted using advanced encryption algorithms (such as AES encryption, Advanced Encryption Standard) to prevent data theft or tampering during transmission. Encryption keys are securely stored and managed to ensure that only authorized personnel can access and use them.

[0063] It can be seen that the path parameters of the data source cluster sent by the user end are obtained, and the data location of the data source to be processed is determined based on the path parameters; based on the data location and using Hadoop commands, a target snapshot of the data source to be processed is created, and a corresponding timestamp is generated for the target snapshot creation; based on the target snapshot and the timestamp, the data to be migrated is determined from the data source to be processed, and the data to be migrated is migrated to the target cluster using a preset thread. That is, by creating a snapshot of the source cluster data, identifying incremental data based on the snapshot timestamp, and combining multi-threading technology to process data transmission in parallel, the migration efficiency can be greatly improved. By creating a snapshot of the source cluster data, identifying incremental data based on the snapshot timestamp, and combining multi-threading technology to process data transmission in parallel, the migration efficiency can be greatly improved.

[0064] refer to Figure 2 The embodiment of the present application further discloses a data migration device based on Hadoop snapshots, including:

[0065] The data location determination module 11 is configured to obtain the path parameters of the data source cluster sent by the user terminal, and determine the data location of the data source to be processed based on the path parameters;

[0066] A data snapshot module 12 is configured to create a target snapshot of the data source to be processed based on the data location and using Hadoop commands, and generate a timestamp corresponding to the creation of the target snapshot;

[0067] The data migration module 13 is configured to determine the data to be migrated from the data source to be processed according to the target snapshot and the timestamp, and migrate the data to be migrated to the target cluster using a preset thread.

[0068] As can be seen, in this embodiment, by creating a snapshot of the source cluster data, identifying incremental data based on the snapshot timestamp, and combining multi-threading technology to parallelize data transmission, migration efficiency can be greatly improved. By creating a snapshot of the source cluster data, identifying incremental data based on the snapshot timestamp, and combining multi-threading technology to parallelize data transmission, migration efficiency can be greatly improved.

[0069] In some specific embodiments, the data location determination module 11 may specifically include:

[0070] A path parameter acquisition unit, used to acquire the path parameters of the data source cluster sent by the user end;

[0071] a first data location determining unit configured to, if the path parameter is an HDFS path, verify the HDFS path using a Hadoop API to determine whether the HDFS path is legal, and if the HDFS path is legal, parse the HDFS path to determine a data location of a data source to be processed;

[0072] The second data location determination unit is configured to establish a connection with the Metastore through the Hive metadata service if the path parameter is a Hive database name and a table name, and send a query request for the data source to be processed according to the Hive database name and the table name, so as to determine the data location of the data source to be processed based on the metadata information returned by the Metastore.

[0073] In some specific embodiments, the data snapshot module 12 may specifically include:

[0074] A location determination unit, configured to determine whether the data source to be processed allows creation of a snapshot based on the data location;

[0075] A snapshot creation function enabling unit, configured to send a corresponding snapshot creation instruction to the NameNode of the Hadoop cluster through a Hadoop management command to enable the snapshot creation function if snapshot creation is not permitted;

[0076] The timestamp generating unit is used to create a target snapshot of the data source to be processed by using a Hadoop snapshot creation command, and generate a timestamp corresponding to the creation of the target snapshot.

[0077] In some specific embodiments, the data migration module 13 may specifically include:

[0078] a first to-be-migrated data determining unit, configured to determine all data in the to-be-processed data source as to-be-migrated data according to the target snapshot and the timestamp;

[0079] The data migration submodule is used to copy the data to be migrated to the corresponding HDFS path of the target cluster using a preset thread and a preset network transmission tool to complete the migration of the data to be migrated.

[0080] In some specific embodiments, the data migration submodule may specifically include:

[0081] a data processing unit configured to process the data to be migrated using a preset data processing method to obtain processed data, and determine a target number of preset threads based on the amount of the processed data; the preset data processing method includes data compression processing and data block processing;

[0082] The data migration unit is used to copy the processed data to the corresponding HDFS path of the target cluster using the target number of preset threads and the preset network transmission tool to complete the migration of the data to be migrated.

[0083] In some specific embodiments, the data migration module 13 may specifically include:

[0084] a second to-be-migrated data determining unit, configured to obtain a historical snapshot corresponding to a previous timestamp based on the timestamp, and determine the to-be-migrated data from the to-be-processed data source by using the target snapshot and the historical snapshot;

[0085] an incremental data migration unit, configured to, if the data to be migrated is incremental data, add the incremental data to a corresponding position of the target cluster using a preset thread;

[0086] a modified data migration unit, configured to, if the data to be migrated is modified data, use the preset thread to replace old data corresponding to the modified data in the target cluster with the modified data;

[0087] The deleted data migration unit is configured to, if the data to be migrated is deleted data, delete the deleted data from the target cluster using the preset thread.

[0088] In some specific embodiments, the Hadoop snapshot-based data migration device may further include:

[0089] A data verification module is used to verify the data information of the first data in the to-be-processed data source and the second data in the target cluster using a preset data verification mechanism during the data migration process to determine whether the second data is completely identical to the first data;

[0090] A step jump module is used to start a preset data repair process if the second data is inconsistent with the first data, so as to jump to the step of determining the data to be migrated from the data source to be processed according to the target snapshot and the timestamp, and migrating the data to be migrated to the target cluster using a preset thread.

[0091] Furthermore, the embodiment of the present application also discloses an electronic device, Figure 3 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram should not be considered as any limitation to the scope of application of the present application.

[0092] Figure 3This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps of the Hadoop snapshot-based data migration method disclosed in any of the aforementioned embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0093] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device. The communication protocol it follows is any communication protocol that can be applied to the technical solution of this application and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world. Its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0094] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or CD, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0095] The operating system 221 is used to manage and control the hardware devices on the electronic device 20 and the computer program 222. The operating system 221 can be Windows Server, NetWare, Unix, Linux, etc. In addition to including a computer program capable of implementing the Hadoop snapshot-based data migration method performed by the electronic device 20 as disclosed in any of the aforementioned embodiments, the computer program 222 can further include computer programs capable of completing other specific tasks.

[0096] Furthermore, this application discloses a computer-readable storage medium for storing a computer program. When executed by a processor, the computer program implements the aforementioned Hadoop snapshot-based data migration method. The specific steps of this method can be found in the corresponding content disclosed in the aforementioned embodiments and will not be further described here.

[0097] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. Reference can be made to the descriptions of the identical or similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and the relevant parts can be referred to the descriptions of the methods.

[0098] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0099] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0100] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0101] The above is a detailed introduction to the technical solution provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the ideas of the present application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A data migration method based on Hadoop snapshot, characterized in that: include: Obtaining the path parameters of the data source cluster sent by the user terminal, and determining the data location of the data source to be processed based on the path parameters; Creating a target snapshot of the data source to be processed based on the data location and using Hadoop commands, and generating a timestamp corresponding to the creation of the target snapshot; Data to be migrated is determined from the data source to be processed according to the target snapshot and the timestamp, and the data to be migrated is migrated to the target cluster using a preset thread.

2. The data migration method based on Hadoop snapshot according to claim 1, characterized in that: The obtaining of the path parameter of the data source cluster sent by the user terminal and determining the data location of the data source to be processed based on the path parameter includes: Get the path parameters of the data source cluster sent by the user; If the path parameter is an HDFS path, the HDFS path is verified using the Hadoop API to determine whether the HDFS path is legal, and if the HDFS path is legal, the HDFS path is parsed to determine the data location of the data source to be processed; If the path parameters are Hive database names and table names, a connection is established with the Metastore through the Hive metadata service, and a query request for the data source to be processed is sent according to the Hive database name and table name to determine the data location of the data source to be processed based on the metadata information returned by the Metastore.

3. The data migration method based on Hadoop snapshot according to claim 1, characterized in that: The step of creating a target snapshot of the data source to be processed based on the data location and using a Hadoop command, and generating a timestamp corresponding to the creation of the target snapshot, includes: Determining whether the data source to be processed allows creation of a snapshot based on the data location; If snapshot creation is not allowed, send the corresponding snapshot creation instruction to the NameNode of the Hadoop cluster through the Hadoop management command to enable the snapshot creation function; A Hadoop snapshot creation command is used to create a target snapshot of the data source to be processed, and a timestamp corresponding to the creation of the target snapshot is generated.

4. The data migration method based on Hadoop snapshot according to claim 3, characterized in that: Determining the data to be migrated from the data source to be processed according to the target snapshot and the timestamp, and migrating the data to be migrated to the target cluster using a preset thread, includes: determining all data in the to-be-processed data source as data to be migrated according to the target snapshot and the timestamp; The data to be migrated is copied to the corresponding HDFS path of the target cluster using a preset thread and a preset network transmission tool to complete the migration of the data to be migrated.

5. The data migration method based on Hadoop snapshot according to claim 4, characterized in that: The method of copying the data to be migrated to the corresponding HDFS path of the target cluster by using a preset thread and a preset network transmission tool to complete the migration of the data to be migrated includes: Processing the data to be migrated using a preset data processing method to obtain processed data, and determining a target number of preset threads based on the amount of the processed data; the preset data processing method includes data compression processing and data block processing; The processed data is copied to the corresponding HDFS path of the target cluster using the target number of preset threads and the preset network transmission tool to complete the migration of the data to be migrated.

6. The data migration method based on Hadoop snapshot according to claim 1, characterized in that: Determining the data to be migrated from the data source to be processed according to the target snapshot and the timestamp, and migrating the data to be migrated to the target cluster using a preset thread, includes: Obtaining a historical snapshot corresponding to a previous timestamp based on the timestamp, and determining data to be migrated from the data source to be processed by using the target snapshot and the historical snapshot; If the data to be migrated is incremental data, the incremental data is added to the corresponding position of the target cluster using a preset thread; If the data to be migrated is modified data, using the preset thread to replace old data corresponding to the modified data in the target cluster with the modified data; If the data to be migrated is deleted data, the deleted data is deleted from the target cluster using the preset thread.

7. The data migration method based on Hadoop snapshot according to any one of claims 1 to 6, characterized in that: After determining the data to be migrated from the data source to be processed according to the target snapshot and the timestamp, and migrating the data to be migrated to the target cluster using a preset thread, the method further includes: During the data migration process, a preset data verification mechanism is used to verify the data information of the first data in the to-be-processed data source and the second data in the target cluster to determine whether the second data is completely identical to the first data; If the second data is inconsistent with the first data, the preset data repair process is started to jump to the step of determining the data to be migrated from the data source to be processed according to the target snapshot and the timestamp, and migrating the data to be migrated to the target cluster using a preset thread.

8. A data migration device based on Hadoop snapshot, characterized in that: include: A data location determination module, configured to obtain the path parameters of the data source cluster sent by the user terminal, and determine the data location of the data source to be processed based on the path parameters; A data snapshot module is used to create a target snapshot of the data source to be processed based on the data location and using Hadoop commands, and generate a timestamp corresponding to the creation of the target snapshot; The data migration module is configured to determine the data to be migrated from the data source to be processed according to the target snapshot and the timestamp, and migrate the data to be migrated to the target cluster using a preset thread.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the data migration method based on Hadoop snapshot according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that Used to store a computer program, which, when executed by a processor, implements the data migration method based on Hadoop snapshot according to any one of claims 1 to 7.

Images