Data migration method and device, computer program product and data migration system

Abstract

The invention provides a data migration method and device, a computer program product and a data migration system. The method comprises the following steps: acquiring source data in a source cluster; a snapshot is created for the source data at the current moment to obtain snapshot data, and the snapshot data at least comprises one or more of a data source, snapshot time, a data size and a data partition; the snapshot data are sent to a target cluster, data are loaded in the target cluster according to the snapshot data, restored source data are obtained, and in the process of migrating data according to the snapshot data, services of the source cluster are not interrupted; acquiring newly added data of the source cluster from the starting moment of data migration according to the snapshots to the ending moment of data migration according to the snapshots to obtain incremental data; the incremental data is migrated into the target cluster, and in the incremental data migration process, the service of the source cluster is temporarily interrupted. The method solves the problem that the service is interrupted for a long time during data migration in the prior art.

Description

Technical Field

[0001] This application relates to the field of big data technology, and more specifically, to a data migration method, a data migration device, a computer program product, and a data migration system. Background Technology

[0002] With the rapid development of information technology, big data technology is facing a comprehensive upgrade and transformation. Currently, cloud-native data lakes are gradually being applied in various industries, achieving efficient and flexible data storage and analysis with the help of cloud-native technology. Through a new data processing architecture that combines cloud computing and data lake technology, leading big data solutions are provided to relevant enterprises.

[0003] Currently, cloud-native data lakes generally use high-version big data components. The low-version big data components previously used by various industries are relatively old. Migrating big data from low-version components to cloud-native data lakes presents several technical challenges, such as ensuring data consistency, minimizing business interruption, and achieving efficient verification.

[0004] Traditional data migration methods require long-term interruption of source cluster services, which results in low efficiency. Summary of the Invention

[0005] The main objective of this application is to provide a data migration method, a data migration device, a computer program product, and a data migration system to at least solve the problem of long-term business interruption during data migration in the prior art.

[0006] To achieve the above objectives, according to one aspect of this application, a data migration method is provided, comprising: acquiring source data from a source cluster; creating a snapshot of the source data at the current moment to obtain snapshot data, wherein the snapshot data includes at least one or more of a data source, snapshot time, data size, and data partition; sending the snapshot data to a target cluster, loading data in the target cluster based on the snapshot data to obtain restored source data, wherein the services of the source cluster are not interrupted during the data migration process based on the snapshot data; acquiring newly added data in the source cluster from the start time of the data migration based on the snapshot data to the end time of the data migration based on the snapshot data to obtain incremental data; and migrating the incremental data to the target cluster, wherein the services of the source cluster are temporarily interrupted during the incremental data migration process.

[0007] Optionally, after acquiring the source data in the source cluster, the method further includes: sending a data scheduling instruction to the source cluster and the target cluster using a data migration scheduling service, wherein the data migration scheduling service is used to control the execution of data migration tasks, and the data scheduling instruction is used to instruct the source cluster and the target cluster to start data migration; sending a data access control instruction to the source cluster and the target cluster using a data access control service, wherein the data access control service is used to monitor the read / write status of the data tables in the source cluster, control the source cluster's services to remain uninterrupted when using snapshot data migration, and control the source cluster to pause new data writing when using incremental data migration, wherein the data migration scheduling service and the data access control service interact, and the data access instruction is used to instruct the source cluster and the target cluster to start data migration and instruct the source cluster to pause data writing.

[0008] Optionally, after sending data scheduling instructions to the source cluster and the target cluster using a data migration scheduling service, the method further includes: obtaining the number of running snapshot migration tasks, wherein the snapshot migration task is a data migration task generated based on a snapshot; if the number is greater than or equal to a preset number threshold, waiting for a first preset time, wherein the next snapshot migration task is not executed during the waiting period; and if the number is less than the preset number threshold, running the snapshot migration task.

[0009] Optionally, after sending a data access instruction to the source cluster and the target cluster using the data access control service, the method further includes: using the data access control service to determine whether there is a data table currently performing a write operation; if there is a data table currently performing a write operation, waiting for a second preset time period, wherein the incremental migration task is not executed during the waiting period, wherein the incremental migration task is a data migration task generated based on the incremental data; if there is no data table currently performing a write operation, pausing new write operations on the data tables of the source cluster, and executing the incremental migration task.

[0010] Optionally, after executing the incremental migration task, the method further includes: verifying the source data in the source cluster and the migrated data in the target cluster to obtain a verification result, wherein the verification includes verifying data consistency and verifying data integrity; if the verification result indicates that the verification has passed, updating the data status of the source data in the source cluster to a successfully migrated state; if the verification result indicates that the verification has failed, updating the data status of the source data in the source cluster to a failed migration state.

[0011] Optionally, after acquiring the source data from the source cluster, the method further includes: acquiring device information, wherein the device information includes one or more of CPU performance, memory space, and transmission rate; determining the number of concurrently executed data migration tasks based on the device information, and generating data migration tasks based on the number of tasks, wherein the CPU performance is positively correlated with the number of tasks, the memory space is positively correlated with the number of tasks, and the transmission rate is positively correlated with the number of tasks.

[0012] Optionally, after migrating the incremental data to the target cluster, the method further includes: constructing a data migration parameter configuration table, wherein the data migration parameter configuration table includes one or more of the following: serial number, data table name, source cluster address, target cluster address, source file path, target file path, migration status, full snapshot migration start time, full snapshot migration end time, incremental migration start time, incremental migration end time, data verification start time, and data verification end time; and displaying the data migration parameter configuration table in a display interface so that the target object can manage data migration according to the data migration parameter configuration table.

[0013] According to another aspect of this application, a data migration apparatus is provided, comprising: a first acquisition unit for acquiring source data in a source cluster; a creation unit for creating a snapshot of the source data at the current moment to obtain snapshot data, wherein the snapshot data includes at least one or more of a data source, snapshot time, data size, and data partition; a first migration unit for sending the snapshot data to a target cluster, loading data in the target cluster according to the snapshot data to obtain restored source data, wherein the services of the source cluster are not interrupted during the data migration process based on the snapshot data; a second acquisition unit for acquiring newly added data in the source cluster from the start time of the data migration based on the snapshot data to the end time of the data migration based on the snapshot data to obtain incremental data; and a second migration unit for migrating the incremental data to the target cluster, wherein the services of the source cluster are temporarily interrupted during the incremental data migration process.

[0014] According to another aspect of this application, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the steps of any of the data migration methods.

[0015] According to another aspect of this application, a data migration system is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including methods for performing any of the data migration methods described above.

[0016] By applying the technical solution of this application, the use of snapshot technology allows for "taking a picture" of the source data at any point in time. The captured data is completely static at that moment, which means that a "snapshot" containing all data and its metadata at the current moment can be created without pausing the source cluster's services. The snapshot data is then sent to the target cluster, and the target cluster loads the data based on the snapshot data to obtain the restored source data. This process is carried out without interrupting the source cluster's services, avoiding service interruptions during full migration. After the snapshot migration, the data added to the source cluster from the start to the completion of the migration is captured as incremental data. The process of migrating the incremental data to the target cluster takes place within a short period of time, during which the source cluster's services are temporarily interrupted. This ensures that the data in the target cluster includes both the data at the time of the snapshot and the new data after the snapshot, thereby minimizing service interruption time and improving the efficiency of the migration method. Attached Figure Description

[0017] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0018] Figure 1 A hardware structure block diagram of a mobile terminal for performing a data migration method according to an embodiment of this application is shown;

[0019] Figure 2 A schematic flowchart of a data migration method according to an embodiment of this application is shown;

[0020] Figure 3 A flowchart illustrating another data migration method is shown;

[0021] Figure 4 This diagram illustrates the communication and workflow between services.

[0022] Figure 5 A flowchart illustrating yet another specific data migration method is shown;

[0023] Figure 6 A structural block diagram of a data migration apparatus provided according to an embodiment of this application is shown.

[0024] The above figures include the following reference numerals:

[0025] 102. Processor; 104. Memory; 106. Transmission device; 108. Input / output device. Detailed Implementation

[0026] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0027] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0028] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0029] For ease of description, the following explains some of the nouns or terms used in the embodiments of this application:

[0030] Region: A Region is a horizontal partition of a data table based on the range of key values. Each Region is associated with a range of key values, and each Region belongs to only one RegionServer. The Region is the most basic unit of distributed storage.

[0031] Cloud-native data lake: A cloud-native data lake is a new data processing architecture that combines cloud computing and data lake technology. It aims to achieve efficient and flexible data storage and analysis through cloud-native technology. (1) Cloud-native architecture: Based on technologies such as containerization and microservices, it supports elastic scaling and distributed processing and is compatible with multi-tenant environments. (2) Data lake capabilities: It centrally stores structured / unstructured data, supports PB-level data storage and cold / hot tiered management, and is compatible with multiple data sources (such as OSS, databases, logs, etc.). (3) Analysis service integration: It provides serverless analysis services (such as Presto, Spark), simplifies the ETL process, and supports real-time querying and machine learning.

[0032] Snapshot data migration: This method freezes data by recording a set of data in a table at a specific moment (including file paths, sizes, region distribution, etc.), and its key feature is that it does not directly copy data files. This method supports cross-version and cross-cluster data migration, and its snapshot metadata uses a lightweight generation mechanism, making it particularly suitable for full migration scenarios, and it does not require interrupting business operations during the migration process.

[0033] Export / Import data migration method: This method uses MapReduce technology to achieve data migration. Specifically, the export process scans the source table using MapReduce and converts the data into SequenceFile format; the import process reads the SequenceFile file and uses MapReduce to write the data into the target table. The characteristic of this approach is that it requires MapReduce resources on both the source and target sides, necessitating a short downtime during the migration process. This method is suitable for incremental migration or small-scale migration scenarios.

[0034] The main problems with the proposed solutions are: using the traditional DistCp method for data migration requires a long-term interruption of services in the source cluster, and manual data table repair is required in the target cluster, which requires a lot of manual intervention and poses a significant operational risk. This also limits the scope and application scenarios of the solutions.

[0035] With the advent of the big data era, data volumes across industries are growing exponentially, necessitating a comprehensive upgrade and transformation of big data technologies. Currently, cloud-native data lakes are gradually being adopted across various industries, providing leading big data solutions for relevant enterprises through a novel data processing architecture that combines cloud-native and data lake technologies. Cloud-native data lakes typically employ high-version big data components, while the lower-version components previously used by various industries represent relatively outdated versions. Migrating big data from these lower-version components to cloud-native data lakes presents several technical challenges:

[0036] (1) Cross-version compatibility: file format, metadata structure compatibility, serialization compatibility, etc. between low and high versions of big data components.

[0037] (2) Zero data loss: It is necessary to ensure that incremental data is completely synchronized during the migration.

[0038] (3) Business continuity: business is uninterrupted or the business downtime window is controllable.

[0039] As described in the background section, prior art often involves prolonged business interruptions during data migration. To address these issues, embodiments of this application provide a data migration method, a data migration apparatus, a computer program product, and a data migration system.

[0040] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0041] The methods and embodiments provided in this application can be executed on a mobile terminal, computer terminal, or similar computing device. Taking running on a mobile terminal as an example, Figure 1 This is a hardware structure block diagram of a mobile terminal for a data migration method according to an embodiment of the present invention. Figure 1 As shown, a mobile terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The mobile terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the mobile terminal described above. For example, the mobile terminal may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0042] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the data migration method in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the aforementioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. The transmission device 106 is used to receive or send data via a network. Specific examples of the aforementioned networks may include wireless networks provided by the mobile terminal's communication provider. In one instance, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0043] This embodiment provides a method for migrating data running on a mobile terminal, computer terminal, or similar computing device. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Also, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0044] Figure 2 This is a flowchart illustrating a data migration method according to an embodiment of this application. Figure 2 As shown, the method includes the following steps:

[0045] Step S201: Obtain source data from the source cluster;

[0046] Specifically, the process involves retrieving source data from the source cluster. Here, source data refers to data tables or files that need to be migrated to the target cluster, which involves reading data from the source cluster and preparing for the migration operation.

[0047] Before data migration begins, it's crucial to identify which data needs to be migrated. This step involves identifying the source cluster address, table names, and specific data locations, laying the foundation for subsequent data snapshots and incremental migrations. Accurately identifying the source data avoids migrating unnecessary data, reduces network load, and improves migration efficiency.

[0048] Step S202: Create a snapshot of the source data at the current moment to obtain snapshot data, wherein the snapshot data includes at least one or more of the following: data source, snapshot time, data size, and data partition.

[0049] Specifically, the process of creating a snapshot is actually freezing the state of data at a certain moment and recording the data attributes at that moment, such as the source of the data, timestamp, size and distribution information, so as to facilitate the subsequent full migration of the data.

[0050] Snapshot mechanisms allow data to be captured at a specific moment while read and write operations are in progress. By using lightweight metadata records, they avoid the business interruption and network resource consumption caused by directly copying large amounts of data files. With snapshot mechanisms, data migration can be performed during business operations without interrupting data read and write operations on the source cluster, thereby reducing business downtime and improving the availability and efficiency of migration.

[0051] Step S203: Send the snapshot data to the target cluster, load the data in the target cluster according to the snapshot data, and obtain the restored source data. During the data migration process based on the snapshot data, the service of the source cluster is not interrupted.

[0052] Specifically, the snapshot metadata is sent to the target cluster. The target cluster then rebuilds and loads the data based on the received snapshot metadata to restore the data table to the state at the time the snapshot was created. The source cluster can continue to provide services during this process.

[0053] By transmitting snapshot metadata, the target cluster can understand the structure and storage location of the source data, thereby reconstructing the same data state on the target cluster without interrupting the service operation of the source cluster. The transmission and loading of snapshot data achieves lossless data migration while ensuring the continuity of services in the source cluster, avoiding prolonged downtime during full data migration, and improving business continuity and user experience.

[0054] Step S204: Obtain the data added to the source cluster from the start time of data migration based on the snapshot to the end time of data migration based on the snapshot, and obtain the incremental data;

[0055] Specifically, during the full data migration process, new data may be generated on the source cluster. This data is called incremental data and needs to be captured and migrated subsequently.

[0056] Incremental data refers to new data generated on the source cluster after a full snapshot. It is typically captured and migrated using Export / Import tools through filtering and version merging strategies. By migrating incremental data, the data on the target cluster is ensured to be up-to-date and complete, achieving the goal of zero data loss during migration. Furthermore, because the volume of incremental data is relatively small, it can be migrated quickly, further reducing the overall duration of business interruptions.

[0057] Step S205: The incremental data is migrated to the target cluster. During the incremental data migration process, the services of the source cluster are temporarily interrupted.

[0058] Specifically, incremental data is sent to the target cluster for final migration. At this time, in order to ensure the integrity and consistency of the incremental data, the services of the source cluster will be temporarily suspended to avoid generating new data during the migration process.

[0059] During the incremental migration phase, although write operations on the source cluster need to be temporarily interrupted, the interruption time is much shorter than that of a full data migration. This ensures that all incremental data is accurately captured and migrated, achieving eventual data consistency. Through precise capture and migration of incremental data, even in the event of brief business interruptions, the integrity and accuracy of data migration are guaranteed, thus achieving low-interruption big data migration and improving the overall efficiency of data migration.

[0060] This embodiment demonstrates that snapshot technology allows for "taking a picture" of the source data at any point in time. The captured data is completely static at that moment, meaning a snapshot containing all data and its metadata can be created without interrupting the source cluster's operations. The snapshot data is then sent to the target cluster, and the target cluster loads the data based on the snapshot data to obtain the restored source data. This process is performed without interrupting the source cluster's operations, avoiding the business interruption that occurs during a full migration. After the snapshot migration, new data added to the source cluster from the start to the completion of the migration is captured as incremental data. The process of migrating this incremental data to the target cluster takes place over a short period, during which the source cluster's operations are temporarily interrupted. This ensures that the data in the target cluster includes both the data at the time of the snapshot and the new data after the snapshot, thereby minimizing business interruption time and improving the efficiency of the migration method.

[0061] The information collected is authorized by the user or fully authorized by all parties. The collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data all comply with the relevant laws, regulations and standards of the relevant countries and regions, and necessary confidentiality measures have been taken. The information does not violate public order and good morals, and corresponding operation entry points are provided for users to choose to authorize or refuse. Users are provided with corresponding operation entry points to choose to agree to or refuse the automated decision results. If the user chooses to refuse, the process will proceed to the expert decision-making process.

[0062] Specifically, this invention employs a layered migration and verification scheme combining full migration based on snapshots and incremental migration via import / export. During the full migration phase, the snapshot mechanism is utilized to ensure data consistency while avoiding production environment write interruptions through snapshot creation, cross-cluster transmission, and data table recovery. During the incremental migration phase, Export / Import tools are used to achieve precise synchronization of incremental data through timestamp filtering and version merging strategies. After migration, data consistency verification is performed using field-by-field verification and row-level checksum calculation to ensure data integrity and consistency. During full migration, business operations remain uninterrupted, without affecting read / write operations on the source side. During incremental migration, incremental data generated during the full migration process is migrated, resulting in a short-term business downtime to ensure complete incremental migration. This invention implements the complex data migration process between different versions of big data components using a configuration-based approach. The overall process for low-interruption big data migration is as follows: Figure 3 As shown.

[0063] Specifically, the basic scheme of this application is as follows: (1) Set up a "data migration parameter configuration" module to complete the configuration of parameters such as data table name, source cluster address, target cluster address, source file path, and target file path. (2) Set up a "data migration task implementation" device to implement full migration based on snapshots and incremental migration of import and export according to the data migration parameter configuration, and migrate the data table from the source cluster to the target cluster. (3) Set up a "data migration performance tuning" module to continuously optimize the concurrent migration performance of the data table by analyzing statistical logs. The following will be described in detail.

[0064] Specifically, creating a snapshot involves the following steps: 1. Determining snapshot metadata information; 2. Creating the snapshot; 3. Transferring snapshot data.

[0065] For determining snapshot metadata information, the following are included:

[0066] Data Source: Determine the source of the data, i.e., the address of the source cluster and the name of the data table. For example, the source cluster address is src.cluster.com, and the data table is TableA.

[0067] Snapshot Time: Records the precise time the snapshot was created, such as 10:00:00 on April 15, 2023.

[0068] Data size: The total data volume of the statistical source data table, including the total file size, such as 200GB.

[0069] Data partitioning: Records the distribution of data in the source cluster, including the number of Regions, the size of each Region, and the relationship between Regions and RegionServers. For example, TableA contains 100 Regions, with each Region averaging 2GB in size.

[0070] For snapshot creation, including:

[0071] Issue a snapshot command: Use administrator commands or the corresponding API to send a snapshot creation command to the source cluster, specifying the name of the snapshot and the Region information it contains.

[0072] Freeze data state: When the snapshot command is triggered, the source cluster will freeze the state of the specified data table at the current point in time, record all relevant metadata, but will not directly copy the data file.

[0073] Generate snapshot metadata: When creating a snapshot, the system generates snapshot metadata containing information such as data source, snapshot time, data size, and data partition, and stores this metadata.

[0074] For snapshot data transfer, including:

[0075] Snapshot metadata data transmission: The generated snapshot metadata is transmitted over the network to a predetermined location on the target cluster, such as the HDFS directory / snapshots on the target cluster, and the start and end times of the transmission are recorded.

[0076] Specifically, restoring the source data includes the following:

[0077] 1. Prepare the target cluster environment: On the target cluster, prepare in advance the space for receiving and storing snapshot data, as well as the necessary data table or file system structure.

[0078] 2. Snapshot metadata parsing: After the target cluster receives the snapshot metadata, it parses the metadata to understand the structure and distribution of the source data, including information such as data source, snapshot time, data size, and data partitions.

[0079] 3. Data Reconstruction: Based on the parsed snapshot information, the target cluster begins to rebuild the structure of data tables or files locally. This process includes, but is not limited to, creating new table structures and assigning regions.

[0080] 4. Data transmission and loading: Although the snapshot data itself does not directly transmit data files, the target cluster can obtain the actual data files from the source cluster based on the source file path and target file path indicated by the snapshot metadata, or simulate the data loading process in the local environment based on the snapshot information.

[0081] Specifically, the determination of an increase in data includes the following:

[0082] 1. Determine the baseline time point: Before the full data migration, a static image of the data has been created on the source cluster through a snapshot mechanism. This time point is called the baseline time point.

[0083] 2. Monitor the source data table: From the creation of the snapshot until the full data migration is completed, continuously monitor the write operations of the source data table and record the data and timestamp of each write.

[0084] 3. Incremental Data Identification: Since data writing is usually accompanied by timestamp or version number updates, it can be determined whether the data is incremental by comparing the timestamp of the current write operation with the timestamp of the baseline time. If so, it is marked as incremental data.

[0085] 4. Collect incremental data: After the full data migration is completed, immediately stop the write operation of the source data table. At this time, collect all data marked as incremental, including newly inserted records and updated records.

[0086] 5. Incremental data export: Use a data export tool to export incremental data as a file, which facilitates subsequent data migration.

[0087] In the specific implementation process, after obtaining the source data in the source cluster, the above method further includes the following steps: Sending data scheduling instructions to the source cluster and the target cluster using a data migration scheduling service, wherein the data migration scheduling service is used to control the execution of data migration tasks, and the data scheduling instructions are used to instruct the source cluster and the target cluster to start data migration; Sending data access instructions to the source cluster and the target cluster using a data access control service, wherein the data access control service is used to monitor the read / write status of the data tables in the source cluster, control the uninterrupted service of the source cluster when using snapshot data migration, and control the source cluster to pause new data writing when using incremental data migration, wherein the data migration scheduling service and the data access control service interact, and the data access instructions are used to instruct the source cluster and the target cluster to start data migration and instruct the source cluster to pause data writing.

[0088] This solution effectively controls the behavior of the source and target clusters during the data migration process through the synergistic action of the data migration scheduling service and the data access control service, achieving high efficiency and intelligence in data migration. The scheduling service ensures the smooth progress of data migration and the rational allocation of resources, while the access control service guarantees the integrity and accuracy of data migration. Especially during incremental data migration, precise control of write operations on the source cluster avoids data omissions and redundancy, ensuring the accuracy of data migration and greatly improving the overall efficiency and quality of data migration.

[0089] Specifically, a data migration scheduling service is used to send data scheduling instructions to both the source and target clusters. This service plays a crucial scheduling role during the migration process, coordinating resource allocation and task execution order between the two clusters. During full data snapshot migration, the scheduling service sends data scheduling instructions, instructing the source cluster to begin preparing snapshot data and sending it to the target cluster. Simultaneously, it instructs the target cluster to receive and load the data. Throughout this process, the source cluster's services remain uninterrupted, ensuring business continuity.

[0090] Specifically, a data access control service sends data access commands to both the source and target clusters. This service focuses on monitoring the read / write status of the data tables in the source cluster and pausing write operations as needed to ensure accurate capture of incremental data. During snapshot migration, the access control service allows continuous read / write operations in the source cluster without interrupting business operations; however, during incremental data migration, the access control service pauses new data writes to the source cluster to guarantee the complete migration of incremental data.

[0091] Specifically, the data migration scheduling service and the data access control service interact with each other. Through their cooperation, the automated control of data migration is achieved, ensuring a seamless connection between full snapshot migration and incremental data migration. This not only guarantees the efficiency of data migration but also maintains business continuity.

[0092] Specifically, a "Data Migration Task Implementation" device is set up to perform full migration based on snapshots and incremental migration via import / export, migrating data tables from the source cluster to the target cluster, according to the configured data migration parameters. The "Data Migration Task Implementation" device includes modules such as data migration scheduling service and data access control service. The "Data Migration Task Implementation" device, for example... Figure 4 As shown.

[0093] The data migration scheduling service sends data scheduling instructions to low-version big data clusters and cloud-native data lake big data clusters to execute data migration tasks.

[0094] The data access control service sends data access commands to older big data clusters or cloud-native data lake big data clusters to execute data access tasks.

[0095] The data migration scheduling service and the data access control service communicate with each other to ensure seamless switching between full migration based on snapshots and incremental migration based on import and export, achieving low business interruption.

[0096] The "Data Migration Task Implementation" device uses the "Data Migration Parameter Configuration Table" to achieve seamless switching between full migration based on snapshots and incremental migration based on import and export, thus achieving low business interruption.

[0097] In some embodiments, after sending data scheduling instructions to the source cluster and the target cluster using a data migration scheduling service, the method further includes the following steps: obtaining the number of running snapshot migration tasks, wherein the snapshot migration tasks are data migration tasks generated based on snapshots; if the number is greater than or equal to a preset number threshold, waiting for a first preset time, wherein the next snapshot migration task is not executed during the waiting period; if the number is less than the preset number threshold, running the snapshot migration task.

[0098] In this solution, by dynamically monitoring the number of concurrent snapshot migration tasks and setting reasonable thresholds, the scheduling service can minimize the impact on the resource consumption of the source and target clusters while ensuring the quality and efficiency of data migration, thus achieving refined management and optimization of the data migration process. Furthermore, through a waiting mechanism, when the number of concurrent tasks reaches a certain level, the scheduling service can proactively adjust the task execution rhythm to prevent sudden shortages of resources in the source or target clusters during data migration, ensuring the stability of the data migration.

[0099] Specifically, when controlling the execution of data migration tasks, the data migration scheduling service needs to monitor the number of currently running snapshot migration tasks in real time to effectively control concurrent migration tasks. When the number of running snapshot migration tasks reaches or exceeds a preset threshold, the scheduling service will pause the execution of new snapshot migration tasks until a first preset time period is elapsed before checking again. This ensures that the system is not overloaded by too many concurrent tasks and avoids excessive impact on business continuity. When the number of running snapshot migration tasks falls below the preset threshold, the scheduling service will start new snapshot migration tasks to ensure efficient data migration.

[0100] By setting a preset threshold (e.g., 10 tasks), the startup of too many concurrent tasks can be prevented, thus avoiding reduced migration efficiency and increased system burden caused by excessive resource consumption.

[0101] The preset first duration (e.g., 5 minutes) serves as a buffer. When concurrent tasks reach the threshold, the waiting mechanism effectively prevents instability caused by resource contention and enhances the reliability of data migration.

[0102] In the specific implementation process, after sending data access instructions to the source cluster and the target cluster using the data access control service, the method further includes the following steps: using the data access control service to determine whether there is a data table currently performing a write operation; if there is a data table currently performing a write operation, waiting for a second preset time period, wherein the incremental migration task is not executed during the waiting period, wherein the incremental migration task is a data migration task generated based on the incremental data; if there is no data table currently performing a write operation, pausing new write operations on the data tables of the source cluster and executing the incremental migration task.

[0103] In this solution, the dynamic monitoring and waiting mechanism of the data access control service can ensure the accurate capture of incremental data, avoid data omissions or conflicts during the data migration process, minimize business interruption time, ensure business continuity, optimize resource scheduling, and achieve efficient data migration without interrupting business, significantly improving the reliability and efficiency of data migration.

[0104] Specifically, the data access control service not only monitors the read and write status of data tables in the source cluster, but also dynamically detects the real-time status of write operations to determine when to execute the incremental migration task. When monitoring detects a data table undergoing write operations, the service waits for a second preset duration to ensure all ongoing write operations are completed, avoiding data omissions during incremental data capture. After the wait is complete, the service checks for new write operations. If none are found, it pauses new write operations and begins executing the incremental migration task, capturing and migrating all incremental data generated after the full snapshot migration is complete from the source cluster.

[0105] The preset duration can be set to 10 minutes, 20 groups, 1 hour, 2 hours, or any other feasible duration.

[0106] In some embodiments, after performing the incremental migration task described above, the method further includes the following steps: verifying the source data in the source cluster and the migrated data in the target cluster to obtain a verification result, wherein the verification includes verifying data consistency and verifying data integrity; if the verification result indicates that the verification has passed, updating the data status of the source data in the source cluster to a successfully migrated state; if the verification result indicates that the verification has failed, updating the data status of the source data in the source cluster to a failed migration state.

[0107] This solution ensures complete consistency and integrity of data between the target cluster and the source cluster after migration by verifying data consistency and integrity, thus avoiding the impact on business operations due to data quality issues. The mechanism for updating data status provides closed-loop control for the entire migration process. Data is marked as successfully migrated only when all verifications pass; otherwise, it is marked as unsuccessful. This promotes the standardization of the data migration process and the automation of troubleshooting, significantly improving the efficiency and quality assurance system of data migration projects.

[0108] To ensure the accuracy of data migration, detailed verification of both the source and target data is required after migration. Data consistency verification involves comparing the structure, content, and update time of the source and target data to ensure that the target data completely replicates the state of the source data. Data integrity verification focuses on checking for any missing or corrupted target data, including but not limited to comparing the total data volume, the number and type of fields in the data tables, and calculating and comparing specific data checksums (e.g., checksums).

[0109] Once verification is complete, the data migration status is updated based on the verification results. If the verification results indicate that the data is consistent and complete, the data status is updated to "Migration Successful," signifying that the migration process was successfully completed and the target data is safe to use. Conversely, if the verification results reveal data problems, such as inconsistencies or compromised integrity, the data status will be updated to "Migration Unsuccessful," indicating the need for further measures, such as redoing the migration or repairing the target data, to ensure the quality of the data migration.

[0110] The specific process for "data migration task implementation" is as follows: Figure 5 As shown:

[0111] S1: Data migration scheduling begins;

[0112] S2: Does a table with a "migration status" of 0 exist? If it does, proceed to step S3; otherwise, proceed to step S13.

[0113] S3: Has the number of concurrent schedulers reached the upper limit? If the number of concurrent schedulers has reached the upper limit, proceed to step S4; otherwise, proceed to step S5.

[0114] S4: Wait XX minutes;

[0115] S5: This table undergoes a full snapshot migration;

[0116] S6: In the "Data Access Control Service", is the original cluster currently accessing this table? If the original cluster is currently accessing this table, proceed to step S7; otherwise, proceed to step S8.

[0117] S7: Wait XX minutes;

[0118] S8: In the "Data Access Control Service", the source cluster suspends the startup of new access tasks for this table and performs incremental data migration;

[0119] S9: Data validation;

[0120] S10: Whether the "data validation" of this table passes. If the validation passes, proceed to step S11; otherwise, proceed to step S12.

[0121] S11: Set the "Access Status" of this table to 6, and start the target cluster to access this table in the "Data Access Control Service";

[0122] S12: Set the "Access Status" of this table to 0;

[0123] S13: Data migration scheduling completed.

[0124] In the specific implementation process, after obtaining the source data in the source cluster, the above method also includes the following steps: obtaining device information, wherein the device information includes one or more of CPU performance, memory space, and transmission rate; determining the number of concurrent data migration tasks based on the device information, and generating data migration tasks based on the number of tasks, wherein the CPU performance is positively correlated with the number of tasks, the memory space is positively correlated with the number of tasks, and the transmission rate is positively correlated with the number of tasks.

[0125] This solution, through the evaluation of the aforementioned device information and the adjustment of task concurrency, enables adaptive optimization of the data migration process, effectively avoiding resource bottlenecks while ensuring the efficiency and stability of data migration. This dynamic adjustment mechanism not only fully leverages hardware performance but also maintains the optimal execution state of data migration under different device environments.

[0126] Acquiring and analyzing device information is a crucial step in optimizing data migration performance in this solution. By collecting key metrics such as CPU performance, memory space, and transmission rate from both the source and target clusters, the data migration performance tuning module can assess the cluster's capacity and transmission efficiency. Based on these assessments, the module dynamically adjusts the number of concurrently executed data migration tasks to ensure a highly efficient and stable data migration process. Specifically, higher CPU performance translates to stronger processing power, supporting more concurrent tasks; larger memory space allows for caching more data and reducing disk access, also contributing to increased task concurrency; and faster transmission rates directly impact data transfer efficiency, allowing for the parallel execution of larger-scale data migration tasks.

[0127] Specifically, a "Data Migration Performance Tuning" module is set up to continuously optimize the concurrent migration performance of data tables by analyzing statistical logs. This module performs statistical analysis on the duration of multiple data migration tasks based on the data migration logs. Data migration performance is affected by hardware environment, software environment, network transmission rate, and data migration program design. Given consistent hardware, software, and network transmission rates, the data migration program design can be optimized to minimize the overall data migration time. Regardless of whether a snapshot-based full migration or an Export / Import incremental migration method is used, assuming other conditions remain constant, the overall data migration time is related to the number of Maps and threads in the data migration program, as detailed below:

[0128] The number of Maps is positively correlated with the migration speed. That is, the more Maps (data processing units) there are, the faster the migration speed is, and vice versa. The upper limit of the number of Map tasks is determined by the number of data table shards.

[0129] The number of threads is also positively correlated with the migration speed; that is, the more threads there are, the faster the migration speed, and vice versa. The upper limit of the number of threads for I / O-intensive data migration tasks can be set to twice the number of CPU cores.

[0130] In some embodiments, after migrating the incremental data to the target cluster, the method further includes the following steps: constructing a data migration parameter configuration table, wherein the data migration parameter configuration table includes one or more of the following: serial number, data table name, source cluster address, target cluster address, source file path, target file path, migration status, full snapshot migration start time, full snapshot migration end time, incremental migration start time, incremental migration end time, data verification start time, and data verification end time; displaying the data migration parameter configuration table in a display interface so that the target object can manage data migration according to the data migration parameter configuration table.

[0131] This solution simplifies the data migration management process by constructing and displaying a data migration parameter configuration table, improving the transparency and manageability of migration tasks. Administrators can clearly see the current status of migration tasks and the operation points that need to be managed, effectively avoiding human error, reducing the workload of migration management, and thus improving the overall efficiency and success rate of data migration projects.

[0132] The data migration parameter configuration table provides a centralized information recording and monitoring tool for managing data migration tasks. This table meticulously records all key parameters for data migration, including table identification information, source and target cluster addresses, file paths, and migration timelines. It also includes a migration status field that dynamically updates as the migration task progresses, reflecting the current real-time migration status. Visualizing the data migration parameter configuration table in the display interface allows administrators to easily monitor migration progress, promptly identify and resolve issues, and ensure the smooth execution of the migration task.

[0133] Specifically, the "Data Migration Parameter Configuration" module is configured to set up parameters such as data table name, source cluster address, target cluster address, source file path, and target file path. Within this module, the data migration process is controlled and managed through the "Data Migration Parameter Configuration Table," which includes parameters such as sequence number, data table name, source cluster address, target cluster address, source file path, target file path, migration status, full snapshot migration start time, full snapshot migration end time, incremental migration start time, incremental migration end time, data verification start time, and data verification end time. The "Data Migration Parameter Configuration Table" is shown in Table 1.

[0134] Table 1

[0135]

[0136] The "Migration Status" field has the following meanings: 0 - Waiting to execute, 1 - Full snapshot migration in progress, 2 - Full snapshot migration completed, 3 - Incremental migration in progress, 4 - Incremental migration completed, 5 - Data verification in progress, 6 - Data verification passed, 7 - Full snapshot migration error, 8 - Incremental migration error, 9 - Data verification failed. The six fields, "Full Snapshot Migration Start Time," "Full Snapshot Migration End Time," "Incremental Migration Start Time," "Incremental Migration End Time," "Data Verification Start Time," and "Data Verification End Time," are all accurate to the year, month, day, hour, minute, and second.

[0137] Currently, cloud-native data lakes generally employ high-version big data components, while various industries previously used lower-version big data components, which are relatively outdated. Migrating big data from lower-version components to cloud-native data lakes presents several technical challenges, including ensuring data consistency, minimizing business interruption, and achieving efficient verification. This invention aims to research a method and apparatus for low-interruption big data migration. Through a layered migration and verification scheme combining snapshot-based full migration and import / export incremental migration, it achieves efficient and reliable data migration and quality control, possessing significant reference value and promotional potential in the field of big data migration.

[0138] Traditional DistCp-based big data migration requires prolonged service interruption in the source cluster and manual data table repair in the target cluster, necessitating significant human intervention and operational risks, thus severely limiting its application scope and scenarios. This invention employs a layered migration and verification scheme combining snapshot-based full migration and import / export incremental migration. During full migration, service remains uninterrupted, maintaining read and write operations on the source side. During incremental migration, incremental data generated during the full migration is migrated, resulting in brief service downtime to ensure complete incremental migration. This invention provides a highly efficient, reliable, and low-interruption big data migration solution with significant reference value and potential for widespread adoption.

[0139] This invention implements a low-interruption big data migration method and apparatus by setting up modules and devices such as "data migration parameter configuration", "data migration task implementation" and "data migration performance optimization". It achieves a high-efficiency, reliable and low-interruption data migration solution by setting up modules and devices such as "data migration parameter configuration", "data migration task implementation" and "data migration performance optimization".

[0140] This application also provides a data migration apparatus. It should be noted that the data migration apparatus of this application can be used to execute the data migration method provided in this application. This apparatus is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0141] The following describes the data migration apparatus provided in the embodiments of this application.

[0142] Figure 6 This is a structural block diagram of a data migration apparatus according to an embodiment of this application. Figure 6 As shown, the device includes:

[0143] The first acquisition unit 10 is used to acquire source data from the source cluster;

[0144] The creation unit 20 is used to create a snapshot of the source data at the current moment to obtain snapshot data, wherein the snapshot data includes at least one or more of the following: data source, snapshot time, data size, and data partition.

[0145] The first migration unit 30 is used to send the snapshot data to the target cluster, load data in the target cluster according to the snapshot data, and obtain the restored source data. During the data migration process based on the snapshot data, the service of the source cluster is not interrupted.

[0146] The second acquisition unit 40 is used to acquire the data added to the source cluster from the start time of data migration based on the snapshot to the end time of data migration based on the snapshot, and obtain incremental data.

[0147] The second migration unit 50 is used to migrate the incremental data to the target cluster, wherein the services of the source cluster are temporarily interrupted during the incremental data migration process.

[0148] This embodiment demonstrates that snapshot technology allows for "taking a picture" of the source data at any point in time. The captured data is completely static at that moment, meaning a snapshot containing all data and its metadata can be created without interrupting the source cluster's operations. The snapshot data is then sent to the target cluster, and the target cluster loads the data based on the snapshot data to obtain the restored source data. This process is performed without interrupting the source cluster's operations, avoiding the business interruption that occurs during a full migration. After the snapshot migration, new data added to the source cluster from the start to the completion of the migration is captured as incremental data. The process of migrating this incremental data to the target cluster takes place over a short period, during which the source cluster's operations are temporarily interrupted. This ensures that the data in the target cluster includes both the data at the time of the snapshot and the new data after the snapshot, thereby minimizing business interruption time and improving the efficiency of the migration method.

[0149] In specific implementation, the above-mentioned device further includes a first service unit and a second service unit. The first service unit is used to send data scheduling instructions to the source cluster and the target cluster using a data migration scheduling service after acquiring source data from the source cluster. The data migration scheduling service is used to control the execution of data migration tasks, and the data scheduling instructions are used to instruct the source cluster and the target cluster to start data migration. The second service unit is used to send data access instructions to the source cluster and the target cluster using a data access control service. The data access control service is used to monitor the read and write status of the data tables in the source cluster, control the uninterrupted service of the source cluster when using snapshot data migration, and control the source cluster to pause new data writing when using incremental data migration. The data migration scheduling service and the data access control service interact, and the data access instructions are used to instruct the source cluster and the target cluster to start data migration and instruct the source cluster to pause data writing.

[0150] This solution effectively controls the behavior of the source and target clusters during the data migration process through the synergistic action of the data migration scheduling service and the data access control service, achieving high efficiency and intelligence in data migration. The scheduling service ensures the smooth progress of data migration and the rational allocation of resources, while the access control service guarantees the integrity and accuracy of data migration. Especially during incremental data migration, precise control of write operations on the source cluster avoids data omissions and redundancy, ensuring the accuracy of data migration and greatly improving the overall efficiency and quality of data migration.

[0151] In some embodiments, the above-described apparatus further includes a third acquisition unit, a first waiting unit, and a running unit. The third acquisition unit is used to acquire the number of running snapshot migration tasks after sending data scheduling instructions to the source cluster and the target cluster using a data migration scheduling service, wherein the snapshot migration tasks are data migration tasks generated based on snapshots. The first waiting unit is used to wait for a first preset time when the number is greater than or equal to a preset number threshold, wherein the next snapshot migration task is not executed during the waiting process. The running unit is used to run the snapshot migration tasks when the number is less than the preset number threshold.

[0152] In this solution, by dynamically monitoring the number of concurrent snapshot migration tasks and setting reasonable thresholds, the scheduling service can minimize the impact on the resource consumption of the source and target clusters while ensuring the quality and efficiency of data migration, thus achieving refined management and optimization of the data migration process. Furthermore, through a waiting mechanism, when the number of concurrent tasks reaches a certain level, the scheduling service can proactively adjust the task execution rhythm to prevent sudden shortages of resources in the source or target clusters during data migration, ensuring the stability of the data migration.

[0153] In specific implementation, the above-mentioned device further includes a determining unit, a second waiting unit, and a first processing unit. The determining unit is used to determine whether there is a data table performing a write operation after sending a data access instruction to the source cluster and the target cluster using the data access control service. The second waiting unit is used to wait for a second preset time if there is a data table performing a write operation, wherein the incremental migration task is not executed during the waiting period, wherein the incremental migration task is a data migration task generated based on the incremental data. The first processing unit is used to suspend new write operations on the data tables of the source cluster and execute the incremental migration task if there is no data table performing a write operation.

[0154] In this solution, the dynamic monitoring and waiting mechanism of the data access control service can ensure the accurate capture of incremental data, avoid data omissions or conflicts during the data migration process, minimize business interruption time, ensure business continuity, optimize resource scheduling, and achieve efficient data migration without interrupting business, significantly improving the reliability and efficiency of data migration.

[0155] In some embodiments, the above-described apparatus further includes a verification unit, a first update unit, and a second update unit. The verification unit is used to verify the source data in the source cluster and the migrated data in the target cluster after performing the incremental migration task, and obtain a verification result. The verification includes verifying data consistency and verifying data integrity. The first update unit is used to update the data status of the source data in the source cluster to a successfully migrated state if the verification result indicates that the verification has passed. The second update unit is used to update the data status of the source data in the source cluster to a failed migration state if the verification result indicates that the verification has failed.

[0156] This solution ensures complete consistency and integrity of data between the target cluster and the source cluster after migration by verifying data consistency and integrity, thus avoiding the impact on business operations due to data quality issues. The mechanism for updating data status provides closed-loop control for the entire migration process. Data is marked as successfully migrated only when all verifications pass; otherwise, it is marked as unsuccessful. This promotes the standardization of the data migration process and the automation of troubleshooting, significantly improving the efficiency and quality assurance system of data migration projects.

[0157] In the specific implementation process, the above-mentioned device further includes a fourth acquisition unit and a second processing unit. The fourth acquisition unit is used to acquire device information after acquiring source data in the source cluster. The device information includes one or more of CPU performance, memory space, and transmission rate. The second processing unit is used to determine the number of concurrent data migration tasks based on the device information, and generate data migration tasks based on the number of tasks. The CPU performance is positively correlated with the number of tasks, the memory space is positively correlated with the number of tasks, and the transmission rate is positively correlated with the number of tasks.

[0158] This solution, through the evaluation of the aforementioned device information and the adjustment of task concurrency, enables adaptive optimization of the data migration process, effectively avoiding resource bottlenecks while ensuring the efficiency and stability of data migration. This dynamic adjustment mechanism not only fully leverages hardware performance but also maintains the optimal execution state of data migration under different device environments.

[0159] In some embodiments, the above-described apparatus further includes a construction unit and a display unit. The construction unit is used to construct a data migration parameter configuration table after migrating the incremental data to the target cluster. The data migration parameter configuration table includes one or more of the following: serial number, data table name, source cluster address, target cluster address, source file path, target file path, migration status, full snapshot migration start time, full snapshot migration end time, incremental migration start time, incremental migration end time, data verification start time, and data verification end time. The display unit is used to display the data migration parameter configuration table in a display interface so that the target object can manage the data migration according to the data migration parameter configuration table.

[0160] This solution simplifies the data migration management process by constructing and displaying a data migration parameter configuration table, improving the transparency and manageability of migration tasks. Administrators can clearly see the current status of migration tasks and the operation points that need to be managed, effectively avoiding human error, reducing the workload of migration management, and thus improving the overall efficiency and success rate of data migration projects.

[0161] The data migration device includes a processor and a memory. The first acquisition unit, creation unit, first migration unit, second acquisition unit, and second migration unit are all stored as program units in the memory. The processor executes the program units stored in the memory to achieve the corresponding functions. All of the above modules are located in the same processor; alternatively, the modules may be located in different processors in any combination.

[0162] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and adjusting kernel parameters can address the problem of prolonged service interruptions during data migration in existing technologies.

[0163] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0164] This invention provides a computer-readable storage medium including a stored program, wherein, when the program is executed, it controls the device containing the computer-readable storage medium to perform the data migration method.

[0165] This invention provides a processor for running a program, wherein the program executes the data migration method during runtime.

[0166] This invention provides a device including a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements at least the steps of a data migration method. The device described herein may be a server, PC, PAD, mobile phone, etc.

[0167] This application also provides a computer program product that, when executed on a data processing device, is adapted to perform the steps of an initialization method for a migration of at least data.

[0168] This application also provides a data migration system, including: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include a method for performing any of the above-described data migration methods.

[0169] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those described herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0170] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0171] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0172] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0173] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0174] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0175] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0176] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0177] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0178] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0179] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A data migration method, characterized in that, include: Retrieve source data from the source cluster; A snapshot is created for the source data at the current moment to obtain snapshot data, wherein the snapshot data includes at least one or more of the following: data source, snapshot time, data size, and data partition. The snapshot data is sent to the target cluster, where data is loaded based on the snapshot data to obtain the restored source data. During the data migration process based on the snapshot data, the services of the source cluster are not interrupted. The incremental data is obtained by acquiring the data added to the source cluster from the start time of data migration based on the snapshot to the end time of data migration based on the snapshot. The incremental data is migrated to the target cluster, wherein the services of the source cluster are temporarily interrupted during the incremental data migration process.

2. The method according to claim 1, characterized in that, After obtaining the source data from the source cluster, the method further includes: A data migration scheduling service is used to send data scheduling instructions to the source cluster and the target cluster. The data migration scheduling service is used to control the execution of data migration tasks. The data scheduling instructions are used to instruct the source cluster and the target cluster to start data migration. A data access control service is used to send data access instructions to the source cluster and the target cluster. The data access control service is used to monitor the read and write status of the data tables in the source cluster, control the source cluster to ensure uninterrupted service when using snapshot data migration, and control the source cluster to pause new data writing when using incremental data migration. The data migration scheduling service interacts with the data access control service, and the data access instructions are used to instruct the source cluster and the target cluster to start data migration and instruct the source cluster to pause data writing.

3. The method according to claim 2, characterized in that, After sending data scheduling instructions to the source cluster and the target cluster using the data migration scheduling service, the method further includes: Get the number of snapshot migration tasks that are currently running, wherein the snapshot migration task is a data migration task based on snapshots; If the number is greater than or equal to a preset number threshold, wait for a first preset time, wherein the next snapshot migration task is not executed during the waiting period; If the number is less than the preset number threshold, the snapshot migration task is executed.

4. The method according to claim 2, characterized in that, After sending data access commands to the source cluster and the target cluster using the data access control service, the method further includes: The data access control service is used to determine whether there is a data table currently undergoing a write operation; If there is a data table that is currently performing a write operation, wait for a second preset time period, wherein the incremental migration task is not executed during the waiting period, wherein the incremental migration task is a data migration task generated based on the incremental data; If no data table is currently performing write operations, pause new write operations on the data tables in the source cluster and execute the incremental migration task.

5. The method according to claim 4, characterized in that, After performing the incremental migration task, the method further includes: The source data in the source cluster and the migrated data in the target cluster are verified to obtain the verification result. The verification includes verifying data consistency and verifying data integrity. If the verification result indicates that the verification has passed, the data status of the source data in the source cluster will be updated to a state of successful migration. If the verification result indicates that the verification failed, the data status of the source data in the source cluster will be updated to a state of unsuccessful migration.

6. The method according to any one of claims 2 to 5, characterized in that, After obtaining the source data from the source cluster, the method further includes: Obtain device information, wherein the device information includes one or more of CPU performance, memory space, and transmission rate; Based on the device information, the number of concurrent data migration tasks is determined, and data migration tasks are generated based on the number of tasks. The performance of the CPU is positively correlated with the number of tasks, the size of the memory space is positively correlated with the number of tasks, and the transmission rate is positively correlated with the number of tasks.

7. The method according to any one of claims 2 to 5, characterized in that, After migrating the incremental data to the target cluster, the method further includes: Construct a data migration parameter configuration table, wherein the data migration parameter configuration table includes one or more of the following: serial number, data table name, source cluster address, target cluster address, source file path, target file path, migration status, full snapshot migration start time, full snapshot migration end time, incremental migration start time, incremental migration end time, data verification start time, and data verification end time. The data migration parameter configuration table is displayed in the display interface so that the target object can manage data migration according to the data migration parameter configuration table.

8. A data migration device, characterized in that, include: The first acquisition unit is used to acquire source data from the source cluster; A creation unit is used to create a snapshot of the source data at the current moment to obtain snapshot data, wherein the snapshot data includes at least one or more of the following: data source, snapshot time, data size, and data partition. The first migration unit is used to send the snapshot data to the target cluster, load data in the target cluster according to the snapshot data, and obtain the restored source data. During the data migration process based on the snapshot data, the service of the source cluster is not interrupted. The second acquisition unit is used to acquire the data added to the source cluster from the start time of data migration based on the snapshot to the end time of data migration based on the snapshot, and obtain incremental data. The second migration unit is used to migrate the incremental data to the target cluster, wherein the source cluster's services are temporarily interrupted during the incremental data migration process.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the data migration method according to any one of claims 1 to 7.

10. A data migration system, characterized in that, include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including a method for performing data migration as described in any one of claims 1 to 7.

Images