Big data migration method and device, computer equipment, storage medium and computer program product

Abstract

The invention relates to the technical field of data information, in particular to a big data migration method and device, computer equipment, a storage medium and a computer program product. The method comprises the steps that in response to a migration request of to-be-migrated data, data features of the to-be-migrated data are acquired; determining a target migration strategy in candidate migration strategies based on the data features, wherein the candidate migration strategies comprise a migration strategy based on an HDFS directory and a migration strategy based on an HDFS directory snapshot; on the basis of a preset migration configuration algorithm and a preset index, migration parameters associated with the target migration strategy are generated, and the migration parameters comprise the number of parallel tasks and the maximum bandwidth; and generating a migration instruction based on the migration parameter, calling a corresponding target API interface to submit a data migration job based on the migration instruction, and executing the data migration job. By adopting the method, the efficiency and stability of data migration can be improved.

Description

Technical Field

[0001] This application relates to the field of data information technology, and in particular to a big data migration method, apparatus, computer equipment, storage medium, and computer program product. Background Technology

[0002] With the rapid development of information technology, big data technology has become a core driving force for enterprise digital transformation. In modern enterprises, various business systems, IoT devices, and user interaction platforms generate massive amounts of data daily. This data includes not only structured data, such as transaction records and user information, but also a large amount of unstructured or semi-structured data, such as log files, images, videos, and social media content. Big data technology, through distributed storage, parallel computing, and intelligent analysis, helps enterprises extract valuable information from this data, supporting optimized decision-making, improved operational efficiency, and the construction of new business models.

[0003] However, with the continuous expansion of data scale and the diversification of data types, enterprise data environments are becoming increasingly complex, posing numerous challenges to data storage and management. On the one hand, data is often scattered across different physical devices, virtual machines, or cloud platforms, resulting in heterogeneous storage and poor system compatibility. On the other hand, as a core enterprise asset, the security, availability, and consistency of data are crucial for business continuity. Against this backdrop, data migration and backup have become key aspects of enterprise data management. Data migration primarily refers to the process of transferring data from one storage location, format, or system to another environment. Its application scenarios are diverse, including system upgrades, data center consolidation, cloud resource switching, and disaster recovery system construction.

[0004] In related technologies, existing data migration techniques are mostly based on traditional data transmission and conversion methods. In addition, techniques based on file system snapshots or block-level replication are also widely used in data migration and backup scenarios. Regarding backup, traditional techniques mainly rely on a strategy combining scheduled full backups and incremental backups. Full backups periodically perform a complete copy of the dataset, while incremental backups only record data changes since the last backup.

[0005] However, current big data migration methods have the following technical problems:

[0006] Existing big data migration methods still have significant shortcomings in terms of adaptability to heterogeneous environments, real-time performance, automation, and resource utilization efficiency, and need to be optimized. Summary of the Invention

[0007] Therefore, it is necessary to provide a big data migration method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can improve the efficiency and stability of data migration in response to the above-mentioned technical problems.

[0008] Firstly, this application provides a big data migration method. The method includes:

[0009] In response to a migration request for the data to be migrated, the data characteristics of the data to be migrated are obtained;

[0010] Based on the data characteristics, a target migration strategy is determined from the candidate migration strategies, including migration strategies based on HDFS directories and migration strategies based on HDFS directory snapshots.

[0011] Based on a preset migration configuration algorithm and preset metrics, migration parameters associated with the target migration strategy are generated. The migration parameters include the number of parallel tasks and the maximum bandwidth.

[0012] Based on the migration parameters, a migration instruction is generated, and based on the migration instruction, the corresponding target API interface is called to submit the data migration job and execute the data migration job.

[0013] In one embodiment, the method includes:

[0014] In response to the target migration strategy being the HDFS directory-based migration strategy, a list of files under the source HDFS directory is generated during each migration.

[0015] Based on the file list, perform a full migration or an incremental migration.

[0016] In one embodiment, the method includes:

[0017] In response to the target migration strategy being the HDFS directory snapshot-based migration strategy, a full snapshot migration is performed during the first migration to migrate the entire source directory snapshot to the target cluster;

[0018] During incremental migration, only the difference files identified by the comparison are migrated by comparing the old version snapshot and the new version snapshot of the source directory.

[0019] In one embodiment, migration parameters associated with the target migration strategy are generated based on a preset migration configuration algorithm and preset metrics. These migration parameters include the number of parallel tasks and the maximum bandwidth.

[0020] Determine the total number of files and directories involved in the data migration operation;

[0021] The recommended number of parallel tasks is determined based on the preset maximum number of files per task and the default number of parallel tasks.

[0022] In one embodiment, migration parameters associated with the target migration strategy are generated based on a preset migration configuration algorithm and preset metrics. These migration parameters include the number of parallel tasks and the maximum bandwidth.

[0023] Determine the average data volume of the migration files involved in the data migration operation;

[0024] The recommended maximum bandwidth is calculated based on the preset maximum bandwidth per task and the average data volume.

[0025] In one embodiment, the step of generating migration instructions based on the migration parameters, submitting a data migration job by calling the corresponding target API interface based on the migration instructions, and executing the data migration job includes:

[0026] The migration command invokes a predefined RESTful API interface.

[0027] The data migration jobs are submitted and executed in batches through the RESTful API interface.

[0028] In one embodiment, after generating migration instructions based on the migration parameters, submitting a data migration job by calling the corresponding target API interface based on the migration instructions, and executing the data migration job, the method further includes:

[0029] In response to the completion of the data migration operation, the Hive metadata of the target cluster is repaired;

[0030] Perform consistency verification on the migrated data.

[0031] In one embodiment, the consistency verification of the migrated data includes:

[0032] After a single migration job is completed, data verification is performed on the source and target directories;

[0033] And / or after the overall migration operation is completed, perform a full comparison of the migration data between the source cluster and the target cluster.

[0034] In one embodiment, after generating migration instructions based on the migration parameters, submitting a data migration job by calling the corresponding target API interface based on the migration instructions, and executing the data migration job, the method further includes:

[0035] Obtain the business requirement information associated with the migration request;

[0036] Based on the aforementioned business requirements information, configure access permissions for the HDFS directory migrated to the target cluster.

[0037] In one embodiment, the step of generating migration instructions based on the migration parameters, submitting a data migration job by calling the corresponding target API interface based on the migration instructions, and executing the data migration job includes:

[0038] During the execution of the data migration job, the migration process is monitored to obtain progress data and status data;

[0039] A visual display interface is generated based on the progress data and status data.

[0040] In one embodiment, before generating migration instructions based on the migration parameters, submitting a data migration job by calling the corresponding target API interface based on the migration instructions, and executing the data migration job, the method further includes:

[0041] Configure a cross-domain mutual trust channel between the source cluster and the target cluster based on the preset permission scope;

[0042] Communication between the source cluster and the target cluster is achieved based on the cross-domain mutual trust channel.

[0043] Secondly, this application also provides a big data migration apparatus. The apparatus includes:

[0044] The data feature module is used to obtain the data features of the data to be migrated in response to a migration request.

[0045] A migration strategy module is used to determine a target migration strategy from candidate migration strategies based on the data characteristics. The candidate migration strategies include migration strategies based on HDFS directories and migration strategies based on HDFS directory snapshots.

[0046] The migration parameter module is used to generate migration parameters associated with the target migration strategy based on a preset migration configuration algorithm and preset metrics. The migration parameters include the number of parallel tasks and the maximum bandwidth.

[0047] The migration execution module is used to generate migration instructions based on the migration parameters, call the corresponding target API interface based on the migration instructions to submit the data migration job, and execute the data migration job.

[0048] In one embodiment, the device further includes:

[0049] The file list module is used to generate a file list under the source HDFS directory during each migration in response to the target migration strategy being the HDFS directory-based migration strategy.

[0050] The list migration module is used to perform full migration or incremental migration based on the file list.

[0051] In one embodiment, the device includes:

[0052] The full migration module is used to perform a full snapshot migration during the first migration in response to the target migration strategy being the HDFS directory snapshot-based migration strategy, so that the entire source directory snapshot is migrated to the target cluster.

[0053] The incremental migration module is used to migrate only the difference files determined by comparison when performing incremental migration by comparing the old version snapshot and the new version snapshot of the source directory.

[0054] In one embodiment, the migration parameter module is further configured to:

[0055] Determine the total number of files and directories involved in the data migration operation;

[0056] The recommended number of parallel tasks is determined based on the preset maximum number of files per task and the default number of parallel tasks.

[0057] In one embodiment, the migration parameter module is further configured to:

[0058] Determine the average data volume of the migration files involved in the data migration operation;

[0059] The recommended maximum bandwidth is calculated based on the preset maximum bandwidth per task and the average data volume.

[0060] In one embodiment, the migration execution module is further configured to:

[0061] The migration command invokes a predefined RESTful API interface.

[0062] The data migration jobs are submitted and executed in batches through the RESTful API interface.

[0063] In one embodiment, after the migration execution module, the system further includes:

[0064] The repair module is used to repair the Hive metadata of the target cluster in response to the completion of the data migration job;

[0065] The verification module is used to verify the consistency of the migrated data.

[0066] In one embodiment, the verification module includes:

[0067] The single-time verification module is used to verify the data in the source and target directories after a single migration job is completed.

[0068] The full verification module is used to perform a full comparison of the migration data between the source cluster and the target cluster after the overall migration job is completed.

[0069] In one embodiment, after the migration execution module, the system further includes:

[0070] The business requirements module is used to obtain business requirements information associated with the migration request;

[0071] The permission configuration module is used to configure access permissions for the HDFS directory migrated to the target cluster based on the business requirement information.

[0072] In one embodiment, the migration execution module includes:

[0073] The monitoring module is used to monitor the migration process during the execution of the data migration job and obtain progress data and status data.

[0074] The visualization module is used to generate a visual display interface based on the progress data and status data.

[0075] In one embodiment, prior to the migration execution module, the following is also included:

[0076] The mutual trust channel module is used to configure a cross-domain mutual trust channel between the source cluster and the target cluster based on a preset permission scope;

[0077] A communication module is used to enable communication between the source cluster and the target cluster based on the cross-domain mutual trust channel.

[0078] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of a big data migration method as described in any embodiment of the first aspect.

[0079] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of a big data migration method as described in any embodiment of the first aspect.

[0080] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the steps of a big data migration method as described in any embodiment of the first aspect.

[0081] The above-described big data migration method, apparatus, computer equipment, storage medium, and computer program product, derived from the technical features in the embodiments, can achieve the following beneficial effects to address the technical problems in the background art:

[0082] This application provides a big data migration method, comprising: in response to a migration request for data to be migrated, obtaining data characteristics of the data to be migrated; determining a target migration strategy from candidate migration strategies based on the data characteristics, the candidate migration strategies including a migration strategy based on an HDFS directory and a migration strategy based on an HDFS directory snapshot; generating migration parameters associated with the target migration strategy based on a preset migration configuration algorithm and preset indicators, the migration parameters including the number of parallel tasks and maximum bandwidth; generating a migration instruction based on the migration parameters; and submitting and executing a data migration job by calling the corresponding target API interface based on the migration instruction. In implementation, efficient and reliable data migration is achieved through data-driven strategy selection and parameter optimization. First, in response to the migration task, data characteristics such as the scale, directory structure, number and size of files of the data to be migrated are automatically analyzed to provide an objective basis for subsequent decision-making. Based on this, the system can intelligently select the most suitable target migration strategy from candidate migration strategies: if it needs to process regular full or incremental data, it adopts a migration strategy based on HDFS directories; if the source directory file volume is extremely large and the increment is small, to avoid the performance bottleneck of full file comparison, it automatically enables a migration strategy based on HDFS directory snapshots, only synchronizing the difference files between snapshots, thereby significantly improving efficiency. After the strategy is determined, the solution further dynamically generates key migration parameters, especially the number of parallel tasks and maximum bandwidth, through a preset migration configuration algorithm. This step helps the migration task maximize the use of the cluster's computing and network resources, while avoiding resource contention and improving the stability of the migration process. Finally, the system automatically generates migration instructions and calls the target API interface to submit the job, transforming the strategy and parameters into specific distributed execution tasks. Overall, this method effectively solves the problems of low efficiency, high resource consumption, and difficulty in guaranteeing consistency that are common in traditional big data migration through a coherent technical chain of feature awareness, strategy adaptation, parameter optimization, and automated execution, achieving a comprehensive improvement in the efficiency, accuracy, and controllability of the migration process. Attached Figure Description

[0083] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0084] Figure 1 This is a schematic diagram of the first process of a big data migration method in one embodiment;

[0085] Figure 2 This is a schematic diagram of the second process of a big data migration method in another embodiment;

[0086] Figure 3 This is a schematic diagram of the third process of a big data migration method in another embodiment;

[0087] Figure 4 This is a schematic diagram of the fourth process of a big data migration method in another embodiment;

[0088] Figure 5 This is a schematic diagram of the fifth process of a big data migration method in another embodiment;

[0089] Figure 6 This is a schematic diagram of the sixth process of a big data migration method in another embodiment;

[0090] Figure 7 This is a schematic diagram of the seventh process of a big data migration method in another embodiment;

[0091] Figure 8 This is a schematic diagram of the eighth process of a big data migration method in another embodiment;

[0092] Figure 9 This is a schematic diagram of the ninth process of a big data migration method in another embodiment;

[0093] Figure 10 This is a schematic diagram of the tenth process of a big data migration method in another embodiment;

[0094] Figure 11 This is a schematic diagram of the eleventh step of a big data migration method in another embodiment;

[0095] Figure 12 This is a flowchart illustrating a migration strategy based on an HDFS directory in one embodiment.

[0096] Figure 13 This is a flowchart illustrating a migration strategy based on HDFS directory snapshots in one embodiment.

[0097] Figure 14 This is a structural block diagram of a big data migration device in one embodiment;

[0098] Figure 15 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0099] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0100] In one embodiment, such as Figure 1 As shown, a big data migration method is provided. This embodiment illustrates the method applied to a terminal, but it is understood that the method can also be applied to a server, and to a system including both a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0101] Step 102: In response to the migration request of the data to be migrated, obtain the data characteristics of the data to be migrated.

[0102] Step 104: Determine the target migration strategy from the candidate migration strategies based on the data characteristics. The candidate migration strategies include migration strategies based on HDFS directories and migration strategies based on HDFS directory snapshots.

[0103] Step 106: Based on the preset migration configuration algorithm and preset metrics, generate migration parameters associated with the target migration strategy. The migration parameters include the number of parallel tasks and the maximum bandwidth.

[0104] Step 108: Generate migration instructions based on the migration parameters, call the corresponding target API interface based on the migration instructions to submit the data migration job and execute the data migration job.

[0105] In the above-mentioned big data migration method, by reasonably deducing the technical features in the embodiments, the beneficial effect of solving the technical problems raised in the background art is achieved:

[0106] This application provides a big data migration method, comprising: in response to a migration request for data to be migrated, obtaining data characteristics of the data to be migrated; determining a target migration strategy from candidate migration strategies based on the data characteristics, the candidate migration strategies including a migration strategy based on an HDFS directory and a migration strategy based on an HDFS directory snapshot; generating migration parameters associated with the target migration strategy based on a preset migration configuration algorithm and preset indicators, the migration parameters including the number of parallel tasks and maximum bandwidth; generating a migration instruction based on the migration parameters; and submitting and executing a data migration job by calling the corresponding target API interface based on the migration instruction. In implementation, efficient and reliable data migration is achieved through data-driven strategy selection and parameter optimization. First, in response to the migration task, data characteristics such as the scale, directory structure, number and size of files of the data to be migrated are automatically analyzed to provide an objective basis for subsequent decision-making. Based on this, the system can intelligently select the most suitable target migration strategy from candidate migration strategies: if it needs to process regular full or incremental data, it adopts a migration strategy based on HDFS directories; if the source directory file volume is extremely large and the increment is small, to avoid the performance bottleneck of full file comparison, it automatically enables a migration strategy based on HDFS directory snapshots, only synchronizing the difference files between snapshots, thereby significantly improving efficiency. After the strategy is determined, the solution further dynamically generates key migration parameters, especially the number of parallel tasks and maximum bandwidth, through a preset migration configuration algorithm. This step helps the migration task maximize the use of the cluster's computing and network resources, while avoiding resource contention and improving the stability of the migration process. Finally, the system automatically generates migration instructions and calls the target API interface to submit the job, transforming the strategy and parameters into specific distributed execution tasks. Overall, this method effectively solves the problems of low efficiency, high resource consumption, and difficulty in guaranteeing consistency that are common in traditional big data migration through a coherent technical chain of feature awareness, strategy adaptation, parameter optimization, and automated execution, achieving a comprehensive improvement in the efficiency, accuracy, and controllability of the migration process.

[0107] In one embodiment, it can be as follows Figure 2 As shown, the method includes:

[0108] Step 202: In response to the target migration strategy being the HDFS directory-based migration strategy, generate a list of files in the source HDFS directory during each migration.

[0109] Step 204: Based on the file list, perform a full migration or incremental migration.

[0110] In this embodiment, when the system determines that an HDFS-based directory migration strategy is to be adopted, the method dynamically generates a complete file list of the source directory during each migration, laying the foundation for precise control of the migration scope. Subsequently, based on this list, the system flexibly performs a full migration to ensure data integrity, or performs an efficient incremental migration by comparison, only synchronizing the changed files. This helps avoid unnecessary full data scanning and transmission, significantly improving migration efficiency while ensuring data consistency, and is especially suitable for routine change scenarios.

[0111] In one embodiment, it can be as follows Figure 3 As shown, the method includes:

[0112] Step 302: In response to the target migration strategy being the HDFS directory snapshot-based migration strategy, a full snapshot migration is performed during the first migration to migrate the entire source directory snapshot to the target cluster.

[0113] Step 304: During incremental migration, migrate only the difference files determined by comparison by comparing the old version snapshot and the new version snapshot of the source directory.

[0114] In this embodiment, when employing a migration strategy based on HDFS directory snapshots, the migration scope is precisely controlled through the snapshot mechanism. The initial migration utilizes a full copy of the snapshot to lay the data foundation; subsequent incremental migrations intelligently compare old and new snapshots, selecting and transferring only the files that have changed. This fundamentally avoids the huge overhead of performing a full directory scan on massive files for each incremental migration, greatly improving the execution efficiency of the incremental migration phase, and is particularly suitable for scenarios with a large total number of files but small incremental changes.

[0115] In one embodiment, it can be as follows Figure 4 As shown, step 106 includes:

[0116] Step 402: Determine the total number of files and directories involved in the data migration operation.

[0117] Step 404: Determine the recommended number of parallel tasks based on the preset maximum number of files per task and the default number of parallel tasks.

[0118] In this embodiment, the total number of files and directories in the migration task is quantitatively analyzed, and the optimal number of parallel tasks is intelligently calculated based on the preset single-task processing capacity limit (maximum number of files per task) and the system baseline parallelism (default number of parallel tasks). This ensures that system resources are maximized, which can improve migration throughput through high concurrency, while avoiding resource contention and scheduling overload due to too many tasks, thus achieving the best balance between efficiency and stability.

[0119] In one embodiment, it can be as follows Figure 5 As shown, step 106 includes:

[0120] Step 502: Determine the average data volume of the migration files involved in the data migration operation.

[0121] Step 504: Calculate the recommended maximum bandwidth based on the preset maximum bandwidth per task and the average data volume.

[0122] In this embodiment, by analyzing the average data size of the migrated files and combining it with a preset single-task bandwidth limit, the optimal maximum bandwidth limit is intelligently calculated. This effectively avoids overall network congestion caused by excessive network resource consumption by a single task, while ensuring that available bandwidth is fully utilized during large file transfers. This achieves refined management of network resources, maintaining the stability of the overall cluster network environment while ensuring migration speed.

[0123] In one embodiment, it can be as follows Figure 6 As shown, step 108 includes:

[0124] Step 602: Invoke the predefined Restful API interface based on the migration instruction.

[0125] Step 604: Submit and execute the data migration job in batches through the Restful API interface.

[0126] In this embodiment, the optimized migration strategy and parameters are transformed into specific cluster job instructions by calling a predefined RESTful API interface. This standardized interface-based calling method enables automated batch submission and execution of data migration jobs, significantly reducing manual intervention and the risk of operational errors. Simultaneously, this design provides a technical foundation for an integrated job scheduling and management system, greatly enhancing the controllability and execution efficiency of large-scale migration tasks.

[0127] In one embodiment, it can be as follows Figure 7 As shown, after step 108, the method further includes:

[0128] Step 702: In response to the completion of the data migration job, repair the Hive metadata of the target cluster.

[0129] Step 704: Perform consistency verification on the migrated data.

[0130] In this embodiment, after the data migration operation is completed, the subsequent data integrity assurance process is automatically triggered. By repairing the Hive metadata of the target cluster, it is ensured that all migrated data tables can be correctly identified and queried by the Hive Metastore, solving the problem of data "existing but not visible". The subsequent data consistency verification confirms the complete consistency between the source and target data at the file level or checksum level, forming the final security barrier for data accuracy and business continuity, realizing a complete closed loop from physical data migration to logical usability.

[0131] In one embodiment, it can be as follows Figure 8 As shown, step 704 includes:

[0132] Step 802: After a single migration job is completed, perform data verification on the source directory and the target directory.

[0133] Step 804: and / or after the overall migration operation is completed, perform a full difference comparison of the migration data between the source cluster and the target cluster.

[0134] In this embodiment, by constructing a multi-layered data consistency verification system, the directory-level verification performed immediately after each job can quickly locate and repair any local data anomalies that may occur during the migration, ensuring the accuracy of the incremental process. The full-scale difference comparison performed after the overall migration is completed provides a final review at a macro level, completely eliminating any potential data inconsistency risks. This point-to-surface verification mechanism provides dual protection for the integrity and reliability of the migrated data, which is key to achieving highly reliable data migration.

[0135] In one embodiment, it can be as follows Figure 9 As shown, after step 108, the method further includes:

[0136] Step 902: Obtain the business requirement information associated with the migration request.

[0137] Step 904: Configure access permissions for the HDFS directory migrated to the target cluster based on the business requirement information.

[0138] In this embodiment, after successful data migration, the permission configuration of the target cluster's HDFS directory is automatically completed based on business requirements. This process directly translates business-level access control policies into underlying cluster security settings, ensuring that the migrated data can be immediately used in a controlled and secure environment. This not only prevents data leakage and unauthorized access but also achieves seamless integration and automated management of data security policies from the source cluster to the target cluster, greatly improving the security and readiness efficiency of the overall migration process.

[0139] In one embodiment, it can be as follows Figure 10 As shown, step 108 includes:

[0140] Step 1002: Monitor the migration process during the execution of the data migration job and obtain progress data and status data.

[0141] Step 1004: Generate a visual display interface based on the progress data and status data.

[0142] In this embodiment, the progress and status data of the migration task are monitored and collected in real time during the data migration operation. By continuously tracking these key indicators, the system can dynamically generate an intuitive visual display interface, providing users with a transparent view of information such as migration rate, completion percentage, and current running status. This greatly enhances the operation and maintenance personnel's awareness and control over the migration process, enabling potential problems to be discovered and intervened in a timely manner, thereby ensuring the controllability and operational transparency of large-scale data migration tasks.

[0143] In one embodiment, it can be as follows Figure 11 As shown, before step 108, the procedure further includes:

[0144] Step 1102: Configure a cross-domain mutual trust channel between the source cluster and the target cluster based on the preset permission scope.

[0145] Step 1104: Implement communication between the source cluster and the target cluster based on the cross-domain mutual trust channel.

[0146] In this embodiment, before initiating the data migration operation, a cross-domain mutual trust channel based on a preset permission scope is established between the source cluster and the target cluster. The establishment of this secure channel allows the migration tool to obtain bidirectional access to protected services (such as HDFS and Hive) within both clusters after single identity authentication. This not only eliminates the need for complex multi-credential management but also ensures that the entire migration process is conducted under strict identity verification and authorization mechanisms at the underlying communication level, laying a solid security foundation for efficient data transmission.

[0147] In one embodiment, the flowchart of the migration strategy based on the HDFS directory can be illustrated as follows: Figure 12 As shown in the diagram, the migration strategy based on HDFS directory snapshots can be illustrated as follows: Figure 13 As shown.

[0148] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0149] Based on the same inventive concept, this application also provides a big data migration apparatus for implementing the big data migration method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations of one or more embodiments of the big data migration apparatus provided below can be found in the limitations of the big data migration method described above, and will not be repeated here.

[0150] In one embodiment, such as Figure 14 As shown, a big data migration device is provided, including: a data feature module, a migration strategy module, a migration parameter module, and a migration execution module, wherein:

[0151] The data feature module is used to obtain the data features of the data to be migrated in response to a migration request.

[0152] A migration strategy module is used to determine a target migration strategy from candidate migration strategies based on the data characteristics. The candidate migration strategies include migration strategies based on HDFS directories and migration strategies based on HDFS directory snapshots.

[0153] The migration parameter module is used to generate migration parameters associated with the target migration strategy based on a preset migration configuration algorithm and preset metrics. The migration parameters include the number of parallel tasks and the maximum bandwidth.

[0154] The migration execution module is used to generate migration instructions based on the migration parameters, call the corresponding target API interface based on the migration instructions to submit the data migration job, and execute the data migration job.

[0155] In one embodiment, the device further includes:

[0156] The file list module is used to generate a file list under the source HDFS directory during each migration in response to the target migration strategy being the HDFS directory-based migration strategy.

[0157] The list migration module is used to perform full migration or incremental migration based on the file list.

[0158] In one embodiment, the device includes:

[0159] The full migration module is used to perform a full snapshot migration during the first migration in response to the target migration strategy being the HDFS directory snapshot-based migration strategy, so that the entire source directory snapshot is migrated to the target cluster.

[0160] The incremental migration module is used to migrate only the difference files determined by comparison when performing incremental migration by comparing the old version snapshot and the new version snapshot of the source directory.

[0161] In one embodiment, the migration parameter module is further configured to:

[0162] Determine the total number of files and directories involved in the data migration operation;

[0163] The recommended number of parallel tasks is determined based on the preset maximum number of files per task and the default number of parallel tasks.

[0164] In one embodiment, the migration parameter module is further configured to:

[0165] Determine the average data volume of the migration files involved in the data migration operation;

[0166] The recommended maximum bandwidth is calculated based on the preset maximum bandwidth per task and the average data volume.

[0167] In one embodiment, the migration execution module is further configured to:

[0168] The migration command invokes a predefined RESTful API interface.

[0169] The data migration jobs are submitted and executed in batches through the RESTful API interface.

[0170] In one embodiment, after the migration execution module, the system further includes:

[0171] The repair module is used to repair the Hive metadata of the target cluster in response to the completion of the data migration job;

[0172] The verification module is used to verify the consistency of the migrated data.

[0173] In one embodiment, the verification module includes:

[0174] The single-time verification module is used to verify the data in the source and target directories after a single migration job is completed.

[0175] The full verification module is used to perform a full comparison of the migration data between the source cluster and the target cluster after the overall migration job is completed.

[0176] In one embodiment, after the migration execution module, the system further includes:

[0177] The business requirements module is used to obtain business requirements information associated with the migration request;

[0178] The permission configuration module is used to configure access permissions for the HDFS directory migrated to the target cluster based on the business requirement information.

[0179] In one embodiment, the migration execution module includes:

[0180] The monitoring module is used to monitor the migration process during the execution of the data migration job and obtain progress data and status data.

[0181] The visualization module is used to generate a visual display interface based on the progress data and status data.

[0182] In one embodiment, prior to the migration execution module, the following is also included:

[0183] The mutual trust channel module is used to configure a cross-domain mutual trust channel between the source cluster and the target cluster based on a preset permission scope;

[0184] A communication module is used to enable communication between the source cluster and the target cluster based on the cross-domain mutual trust channel.

[0185] The modules in the aforementioned big data migration device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0186] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 15As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a large data migration method.

[0187] Those skilled in the art will understand that Figure 15 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0188] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0189] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0190] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0191] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0192] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0193] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0194] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A big data migration method, characterized in that, The method includes: In response to a migration request for the data to be migrated, the data characteristics of the data to be migrated are obtained; Based on the data characteristics, a target migration strategy is determined from the candidate migration strategies, including migration strategies based on HDFS directories and migration strategies based on HDFS directory snapshots. Based on a preset migration configuration algorithm and preset metrics, migration parameters associated with the target migration strategy are generated. The migration parameters include the number of parallel tasks and the maximum bandwidth. Based on the migration parameters, a migration instruction is generated, and based on the migration instruction, the corresponding target API interface is called to submit the data migration job and execute the data migration job.

2. The method according to claim 1, characterized in that, The method includes: In response to the target migration strategy being the HDFS directory-based migration strategy, a list of files under the source HDFS directory is generated during each migration. Based on the file list, perform a full migration or an incremental migration.

3. The method according to claim 1, characterized in that, The method includes: In response to the target migration strategy being the HDFS directory snapshot-based migration strategy, a full snapshot migration is performed during the first migration to migrate the entire source directory snapshot to the target cluster; During incremental migration, only the difference files identified by the comparison are migrated by comparing the old version snapshot and the new version snapshot of the source directory.

4. The method according to claim 1, characterized in that, The migration parameters, based on a preset migration configuration algorithm and preset metrics, are generated and associated with the target migration strategy. These migration parameters include the number of parallel tasks and the maximum bandwidth. Determine the total number of files and directories involved in the data migration operation; The recommended number of parallel tasks is determined based on the preset maximum number of files per task and the default number of parallel tasks.

5. The method according to claim 1, characterized in that, The migration parameters, based on a preset migration configuration algorithm and preset metrics, are generated and associated with the target migration strategy. These migration parameters include the number of parallel tasks and the maximum bandwidth. Determine the average data volume of the migration files involved in the data migration operation; The recommended maximum bandwidth is calculated based on the preset maximum bandwidth per task and the average data volume.

6. The method according to claim 1, characterized in that, The step of generating migration instructions based on the migration parameters, submitting a data migration job by calling the corresponding target API interface based on the migration instructions, and executing the data migration job includes: The migration command invokes a predefined RESTful API interface. The data migration jobs are submitted and executed in batches through the RESTful API interface.

7. The method according to claim 1, characterized in that, After generating migration instructions based on the migration parameters, submitting a data migration job by calling the corresponding target API interface based on the migration instructions, and executing the data migration job, the process further includes: In response to the completion of the data migration operation, the Hive metadata of the target cluster is repaired; Perform consistency verification on the migrated data.

8. The method according to claim 7, characterized in that, The consistency verification of the migrated data includes: After a single migration job is completed, data verification is performed on the source and target directories; And / or after the overall migration operation is completed, perform a full comparison of the migration data between the source cluster and the target cluster.

9. The method according to claim 1, characterized in that, After generating migration instructions based on the migration parameters, submitting a data migration job by calling the corresponding target API interface based on the migration instructions, and executing the data migration job, the process further includes: Obtain the business requirement information associated with the migration request; Based on the aforementioned business requirements information, configure access permissions for the HDFS directory migrated to the target cluster.

10. The method according to claim 1, characterized in that, The step of generating migration instructions based on the migration parameters, submitting a data migration job by calling the corresponding target API interface based on the migration instructions, and executing the data migration job includes: During the execution of the data migration operation, the migration process is monitored to obtain progress data and status data; A visual display interface is generated based on the progress data and status data.

11. The method according to claim 1, characterized in that, Before generating migration instructions based on the migration parameters, submitting the data migration job by calling the corresponding target API interface based on the migration instructions, and executing the data migration job, the process further includes: Configure a cross-domain mutual trust channel between the source cluster and the target cluster based on the preset permission scope; Communication between the source cluster and the target cluster is achieved based on the cross-domain mutual trust channel.

12. A big data migration device, characterized in that, The device includes: The data feature module is used to obtain the data features of the data to be migrated in response to a migration request. A migration strategy module is used to determine a target migration strategy from candidate migration strategies based on the data characteristics. The candidate migration strategies include migration strategies based on HDFS directories and migration strategies based on HDFS directory snapshots. The migration parameter module is used to generate migration parameters associated with the target migration strategy based on a preset migration configuration algorithm and preset metrics. The migration parameters include the number of parallel tasks and the maximum bandwidth. The migration execution module is used to generate migration instructions based on the migration parameters, call the corresponding target API interface based on the migration instructions to submit the data migration job, and execute the data migration job.

13. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 11.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 11.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 11.

Images