Data migration method and device, equipment, storage medium and program product

Abstract

The invention relates to a data migration method and device, equipment, a storage medium and a program product. The method is used for target data migration equipment in a data migration system, the data migration system comprises a plurality of pieces of data migration equipment, the plurality of pieces of data migration equipment form a plurality of migration clusters, and the method comprises the steps that a data migration request is received, and the data migration request comprises information of a source host, information of a target host and a migration type; determining a migration task according to the migration request, and determining a target migration cluster according to the migration type; sending the migration task to a target migration cluster, so that the target migration cluster migrates the data from the source host to the target host by executing the migration task; wherein the target migration cluster comprises a master node and a plurality of slave nodes, the master node is used for dividing the migration task into a plurality of sub-migration tasks and sending the sub-migration tasks to the slave nodes, and the slave nodes are used for executing the sub-migration tasks. By adopting the method, the data migration efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of data migration, and in particular to a data migration method, device, equipment, storage medium and program product. Background Art

[0002] With the explosive growth of data and the diversification of business needs, cloud migration has become a key step in the digital transformation of enterprises. Currently, enterprises often need to migrate data between different cloud service providers for many reasons, such as improving operational efficiency, reducing costs, accelerating innovation, and implementing multi-cloud strategies.

[0003] However, there are many types of cloud storage. Currently, domestic cloud vendors have different methods for migrating data of different types. For enterprises, before data migration, staff must be familiar with the migration methods of various vendors and various data types, and the efficiency of data migration is low. Summary of the invention

[0004] Based on this, it is necessary to provide a data migration method, device, equipment, storage medium and program product that can improve data migration efficiency in response to the above technical problems.

[0005] In a first aspect, the present application provides a data migration method, which is used for a target data migration device in a data migration system, wherein the data migration system includes multiple data migration devices, the target data migration device is any one of the multiple data migration devices, and the multiple data migration devices form multiple migration clusters, and the method includes:

[0006] receiving a data migration request, the data migration request including information of a source host, information of a target host, and a migration type;

[0007] Determine the migration task based on the migration request, and determine the target migration cluster based on the migration type;

[0008] The migration task is sent to the target migration cluster so that the target migration cluster migrates the data from the source host to the target host by executing the migration task; wherein the target migration cluster includes a master node and multiple slave nodes, the master node is used to divide the migration task into multiple sub-migration tasks, and send each sub-migration task to each slave node, and each slave node is used to execute the sub-migration task.

[0009] In one embodiment, determining a target migration cluster according to a migration type includes:

[0010] At least one candidate migration cluster corresponding to the migration type is determined according to the migration type; and a target migration cluster is determined according to the working status of the candidate migration cluster.

[0011] In one embodiment, determining a target migration cluster according to a working status of a candidate migration cluster includes:

[0012] If there is an idle migration cluster in at least one candidate migration cluster, the idle migration cluster is determined as the target migration cluster.

[0013] In one embodiment, the data migration method further includes:

[0014] If there is no idle migration cluster in at least one candidate migration cluster, a target migration cluster is generated according to the migration type and the preset migration cluster image.

[0015] In one embodiment, sending a migration task to a target migration cluster so that the target migration cluster migrates data from a source host to a target host by executing the migration task includes:

[0016] The migration task is sent to the master node of the target migration cluster, so that the master node obtains the target migration data according to the migration task, and determines multiple sub-migration tasks according to the target migration data, and sends each sub-migration task to each slave node. Each slave node is used to download the target migration data from the source host according to the sub-migration task and upload the target migration data to the target host.

[0017] In one of the embodiments, the migration type includes host migration, object storage migration, file storage migration or database migration, and the migration cluster includes a host migration cluster, an object storage migration cluster, a file storage migration cluster and a database migration cluster.

[0018] In a second aspect, the present application further provides a data migration device, which is a target data migration device set in a data migration system, the data migration system includes multiple data migration devices, the target data migration device is any one of the multiple data migration devices, and the multiple data migration devices form multiple migration clusters. The device includes:

[0019] A receiving module, used to receive a data migration request, the data migration request includes information of a source host, information of a target host and a migration type;

[0020] A determination module is used to determine a migration task according to a migration request and determine a target migration cluster according to a migration type;

[0021] The migration module is used to send the migration task to the target migration cluster so that the target migration cluster migrates the data from the source host to the target host by executing the migration task; wherein, the target migration cluster includes a master node and multiple slave nodes, the master node is used to divide the migration task into multiple sub-migration tasks, and send each sub-migration task to each slave node, and each slave node is used to execute the sub-migration task.

[0022] In one of the embodiments, the determination module is specifically configured to determine at least one candidate migration cluster corresponding to the migration type according to the migration type; and determine the target migration cluster according to the working status of the candidate migration cluster.

[0023] In one of the embodiments, the determination module is specifically configured to determine the migration cluster in the idle state as the target migration cluster if there is a migration cluster in the idle state in at least one candidate migration cluster.

[0024] In one of the embodiments, the determination module is further configured to generate a target migration cluster according to the migration type and a preset migration cluster image if there is no migration cluster in an idle state in at least one candidate migration cluster.

[0025] In one of the embodiments, the migration module is specifically used to send the migration task to the master node of the target migration cluster, so that the master node obtains the target migration data according to the migration task, and determines multiple sub-migration tasks according to the target migration data, and sends each sub-migration task to each slave node. Each slave node is used to download the target migration data from the source host according to the sub-migration task, and upload the target migration data to the target host.

[0026] In one of the embodiments, the migration type includes host migration, object storage migration, file storage migration or database migration, and the migration cluster includes a host migration cluster, an object storage migration cluster, a file storage migration cluster and a database migration cluster.

[0027] In a third aspect, the present application further provides a data migration device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0028] receiving a data migration request, the data migration request including information of a source host, information of a target host, and a migration type;

[0029] Determine the migration task based on the migration request, and determine the target migration cluster based on the migration type;

[0030] The migration task is sent to the target migration cluster so that the target migration cluster migrates the data from the source host to the target host by executing the migration task; wherein the target migration cluster includes a master node and multiple slave nodes, the master node is used to divide the migration task into multiple sub-migration tasks, and send each sub-migration task to each slave node, and each slave node is used to execute the sub-migration task.

[0031] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0032] receiving a data migration request, the data migration request including information of a source host, information of a target host, and a migration type;

[0033] Determine the migration task based on the migration request, and determine the target migration cluster based on the migration type;

[0034] The migration task is sent to the target migration cluster so that the target migration cluster migrates the data from the source host to the target host by executing the migration task; wherein the target migration cluster includes a master node and multiple slave nodes, the master node is used to divide the migration task into multiple sub-migration tasks, and send each sub-migration task to each slave node, and each slave node is used to execute the sub-migration task.

[0035] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0036] receiving a data migration request, the data migration request including information of a source host, information of a target host, and a migration type;

[0037] Determine the migration task based on the migration request, and determine the target migration cluster based on the migration type;

[0038] The migration task is sent to the target migration cluster so that the target migration cluster migrates the data from the source host to the target host by executing the migration task; wherein the target migration cluster includes a master node and multiple slave nodes, the master node is used to divide the migration task into multiple sub-migration tasks, and send each sub-migration task to each slave node, and each slave node is used to execute the sub-migration task.

[0039] The above-mentioned data migration method, device, equipment, storage medium and program product receive a data migration request, which includes information of the source host, information of the target host and the migration type; determine the migration task according to the migration request, and determine the target migration cluster according to the migration type; send the migration task to the target migration cluster, so that the target migration cluster migrates data from the source host to the target host by executing the migration task; wherein the target migration cluster includes a master node and multiple slave nodes, the master node is used to divide the migration task into multiple sub-migration tasks, and send each sub-migration task to each slave node, each slave node is used to execute the sub-migration task. The migration process of multiple data storage types can be adapted through multiple migration clusters. In this way, during the data migration process, it is only necessary to determine the corresponding target data migration cluster according to the migration type in the data migration request, and then complete the data migration through the target migration cluster. There is no need for the staff to be familiar with the migration tool before the data migration, which can effectively improve the data migration efficiency. At the same time, the migration cluster includes multiple slave nodes, and multiple slave nodes execute the data migration task at the same time, which can further improve the efficiency of data migration. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0041] FIG1 is an application environment diagram of a data migration method in one embodiment;

[0042] FIG2 is a schematic diagram of a flow chart of a data migration method in one embodiment;

[0043] FIG3 is a schematic diagram of a flow chart of steps for determining a target migration cluster in one embodiment;

[0044] FIG4 is a schematic diagram of a process flow of migrating a cluster in one embodiment;

[0045] FIG5 is a schematic diagram of a flow chart of a data migration method in another embodiment;

[0046] FIG6 is a structural block diagram of a data migration method device in one embodiment;

[0047] FIG. 7 is a diagram showing the internal structure of a data migration device in one embodiment. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0049] With the explosive growth of data and the diversification of business needs, cloud migration has become a key step in the digital transformation of enterprises. Currently, enterprises often need to migrate data between different cloud service providers for many reasons, such as improving operational efficiency, reducing costs, accelerating innovation, and implementing multi-cloud strategies.

[0050] However, there are many types of cloud storage. Currently, domestic cloud vendors have different methods for migrating data of different types. For enterprises, before data migration, staff must be familiar with the migration methods of various vendors and various data types, and the efficiency of data migration is low.

[0051] In view of this, the present application provides a data migration method that can improve the efficiency of data migration. The data migration method provided in the embodiment of the present application, its execution subject can be a data migration device, and the data migration device can be implemented by software, hardware, or a combination of software and hardware. It can be embedded in or independent of the processor in the data migration device in the form of hardware, or it can be stored in the memory of the data migration device in the form of software. In the following method embodiments, the execution subject is taken as an example of a data migration device, where the data migration device can be a server, and the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0052] First, the professional terms involved in the embodiments of the present application are explained:

[0053] Cloud storage: Cloud storage is a service that stores data on remote servers. Users can access the data through the Internet without the need for local storage devices. Cloud storage can be divided into block storage, file storage, object storage, etc. according to storage type. It has the advantages of strong scalability, low cost, and can be accessed anywhere.

[0054] Data migration: This can be the process of moving data from one storage environment or location to another storage environment or cloud service. This process may involve migrating from one resource pool to another, or from one cloud service provider to another.

[0055] The data migration method provided in the embodiment of the present application can be applied to the application environment as shown in Figure 1. The data migration system 100 includes multiple data migration devices 101. The target data migration device is any one of the multiple data migration devices. The multiple data migration devices form multiple migration clusters 102. Optionally, the master node 103 (master) and multiple slave nodes 104 (workers) in the migration cluster can be data migration devices or virtual devices deployed on the data migration device. At the same time, when the master node and the slave node are virtual devices, the master node and the slave node can be deployed on the same data migration device or on different data migration devices respectively. It can be understood that the data migration method of the present application is executed by the data migration device, and the number of slave nodes can be selected according to the amount of migration tasks.

[0056] In an exemplary embodiment, optionally, as shown in FIG2 , the data migration method includes the following steps 201 to 203. Among them:

[0057] S201: Receive a data migration request.

[0058] The data migration request includes information of the source host, information of the target host and the migration type.

[0059] In one possible implementation, the data migration request may be sent by the terminal, and the data migration request may be a REST request. The REST request is a request method based on the HTTP protocol for accessing and operating network resources. It uses methods of the HTTP protocol (such as GET, POST, PUT, DELETE, etc.) to operate resources (data or services) on the network, making the interaction between the terminal and the server simpler, more flexible and scalable.

[0060] In another possible implementation, the data migration request may be sent through a set scheduled task. The scheduled task is pre-set by the staff in the data migration device. By comparing the current time with the task time preset in the scheduled task, when the specified time is reached, the corresponding task will be triggered, the data migration request will be sent, and the data migration task will be executed according to the request. Among them, the scheduled task can be a task that is automatically executed at a preset specific time point or at a certain time interval.

[0061] Optionally, the information of the source host may include identification information of the source host (for example, an IP address, host name, server serial number or MAC address, etc. that can uniquely identify the device) and authentication credentials for accessing the source host, where the authentication credentials may be a user name and password, a digital certificate or a token, etc.; similarly, the information of the target host may also include identification information of the target host (for example, an IP address, host name, server serial number or MAC address, etc. that can uniquely identify the device) and authentication credentials for accessing the target host.

[0062] Optionally, when receiving a data migration request, each parameter in the request is verified to ensure the validity of each parameter.

[0063] Optionally, the source host and the target host can be data migration devices, and can be physical servers or cloud servers. It can be understood that the source host and the target host can both be cloud servers, or both be physical servers, or one of them can be a physical server and the other can be a cloud server. The embodiments of the present application do not limit this.

[0064] Optionally, the source host may be a host where the data is originally stored, and the target host may be a host to which the data is to be migrated.

[0065] S202: Determine a migration task according to the migration request, and determine a target migration cluster according to the migration type.

[0066] Optionally, when determining the migration task according to the migration request, the identification information of the source host, the identification information of the target host, and the migration type may be combined in a preset combination manner to obtain the migration task.

[0067] Optionally, the migration type includes host migration, object storage migration, file storage migration, or database migration, and the migration cluster includes a host migration cluster, an object storage migration cluster, a file storage migration cluster, and a database migration cluster.

[0068] Optionally, object storage migration can be to migrate objects stored in the source host to the target host. Object storage can achieve efficient storage by dividing unstructured data into independent data blocks, namely objects. These objects are stored in a flat environment without a hierarchical structure, making data management more direct and simple. It has the advantages of high scalability, simplified data management, high flexibility, high reliability and low cost; host migration can be to migrate all data stored in the source host to the target host; file storage migration can be to migrate files stored in the source host to the target host; database migration can be to migrate the database in the source host to the target host.

[0069] Optionally, when executing a migration task, different migration types may have different migration steps or migration methods. The target migration cluster for executing the migration task may be determined according to the migration type.

[0070] In a possible implementation, a preset mapping table may be queried according to the migration type, and a target migration cluster may be determined according to the query result. The preset mapping table stores the correspondence between different migration types and migration clusters.

[0071] In another possible implementation, both the migration type and the migration cluster have type identification information. Different types of migration clusters have different type identification information. A migration cluster corresponding to the type identification can be selected from multiple migration clusters according to the type identification information of the migration type as the target migration cluster.

[0072] Exemplarily, if the migration type is object storage migration, then an object storage migration cluster may be determined from a variety of migration clusters as a target migration cluster according to the migration type.

[0073] S203: Send the migration task to the target migration cluster, so that the target migration cluster migrates the data from the source host to the target host by executing the migration task.

[0074] The target migration cluster includes a master node and multiple slave nodes. The master node is used to divide the migration task into multiple sub-migration tasks and send each sub-migration task to each slave node. Each slave node is used to execute the sub-migration task.

[0075] Optionally, the master node load divides the migration task into multiple sub-migration tasks.

[0076] Optionally, the migration clusters corresponding to different migration types have an abstract common structure when executing the migration task, that is, the migration task is divided into multiple sub-migration tasks through the master node, and the sub-migration tasks are executed by multiple slave nodes. Therefore, for any migration scenario, if the migration type of the migration scenario does not belong to the existing migration type, it is possible to obtain the target migration cluster corresponding to the migration type by determining the task division method and migration implementation method corresponding to the migration type of the migration scenario on the basis of the common structure of the migration cluster to complete the data migration. It can be understood that the data migration method in the embodiment of the present application has strong scalability, and can process currently unsupported migration scenarios through an abstract common structure.

[0077] Exemplarily, the data type and data structure of the data to be migrated can be analyzed to obtain migration records with a data type or data structure similar to that of the data to be migrated from historical migration records, and the task division method and migration implementation method corresponding to the migration type can be determined based on the task division method and migration implementation method in the migration record.

[0078] In one possible implementation, when dividing the migration task, the number of sub-migration tasks obtained can be the same as the number of slave nodes, or the number of sub-migration tasks can be less than the number of slave nodes. It can be understood that the master node divides the migration task once and sends each sub-migration task to each slave node. After receiving the completion information from the slave node, the migration task can be completed.

[0079] In another possible implementation, when dividing the migration task, the number of sub-migration tasks obtained is greater than the number of slave nodes. It can be understood that after the master node divides the migration task once, the sub-migration tasks obtained are only a part of the migration task. Each sub-migration task is sent to each slave node. After receiving the completion information from the slave node, the remaining part of the migration task can be divided again until the migration task is completed.

[0080] The above data migration method receives a data migration request, which includes information of the source host, information of the target host, and a migration type; determines a migration task according to the migration request, and determines a target migration cluster according to the migration type; sends the migration task to the target migration cluster, so that the target migration cluster migrates data from the source host to the target host by executing the migration task; wherein the target migration cluster includes a master node and multiple slave nodes, the master node is used to divide the migration task into multiple sub-migration tasks, and sends each sub-migration task to each slave node, and each slave node is used to execute the sub-migration task. The migration process of multiple data storage types can be adapted through multiple migration clusters. In this way, during the data migration process, it is only necessary to determine the corresponding target data migration cluster according to the migration type in the data migration request, and then complete the data migration through the target migration cluster. There is no need for the staff to be familiar with the migration tool before the data migration, which can effectively improve the data migration efficiency. At the same time, the migration cluster includes multiple slave nodes, and multiple slave nodes execute data migration tasks at the same time, which can further improve the efficiency of data migration.

[0081] In an exemplary embodiment, optionally, as shown in FIG3 , determining the target migration cluster according to the migration type includes the following steps 301 to 302 . Among them:

[0082] S301: Determine, according to a migration type, at least one candidate migration cluster corresponding to the migration type.

[0083] Optionally, in the data migration system, one migration type corresponds to at least one migration cluster. It is understandable that the type identification information of the same type of migration clusters is the same. In the correspondence between the migration type and the migration cluster in the preset mapping table, one migration type corresponds to at least one migration cluster.

[0084] S302: Determine a target migration cluster according to the working status of the candidate migration cluster.

[0085] Optionally, the target migration device monitors the heartbeat of each migration cluster and the working status of each migration cluster, issues an alarm for a migration cluster with an abnormal status, and can also dynamically release and create resources. For example, if there are multiple migration clusters under a migration type and all migration clusters are in idle state, only one migration cluster can be retained and the other migration clusters can be released; or when a migration cluster under a migration type is in working state, a new migration cluster can be created.

[0086] In a possible implementation, determining the target migration cluster according to the working status of the candidate migration clusters includes: if there is an idle migration cluster in at least one candidate migration cluster, determining the idle migration cluster as the target migration cluster.

[0087] Optionally, if there are multiple idle migration clusters in at least one candidate migration cluster, one migration cluster can be randomly selected as the target migration cluster according to a random algorithm, or the target migration cluster can be selected from multiple idle migration clusters according to a load balancing algorithm. The embodiment of the present application does not limit the method of determining the target migration cluster from multiple idle migration clusters.

[0088] In another possible implementation, determining the target migration cluster according to the working status of the candidate migration clusters further includes: if there is no idle migration cluster in at least one candidate migration cluster, generating the target migration cluster according to the migration type and a preset migration cluster image.

[0089] Optionally, different migration types correspond to different types of migration clusters. Therefore, there are multiple preset migration cluster images corresponding to different migration types.

[0090] Optionally, the preset migration cluster image may be a collection of files or data, including complete configuration information of the migration cluster, and may be a "snapshot" of the migration cluster, which can be restored in the data migration device so as to quickly create a migration cluster identical to other migration clusters.

[0091] Optionally, elastic scaling of migration resources can be achieved through migration cluster mirroring. Since data migration is often a long-term but infrequent scenario, there is no need to pre-set a large number of migration clusters in the data migration system. When a migration task is determined, a corresponding migration cluster can be temporarily created according to the migration type to achieve elastic scaling of resources.

[0092] By determining at least one candidate migration cluster corresponding to the migration type according to the migration type and determining the target migration cluster according to the working status of the candidate migration cluster, the specific migration cluster for executing the migration task can be determined according to the migration type, thereby completing the data migration task and integrating multiple migration types to effectively improve the efficiency of data migration.

[0093] In an exemplary embodiment, optionally, sending the migration task to the target migration cluster so that the target migration cluster migrates data from the source host to the target host by executing the migration task includes:

[0094] The migration task is sent to the master node of the target migration cluster, so that the master node obtains the target migration data according to the migration task, and determines multiple sub-migration tasks according to the target migration data, and sends each sub-migration task to each slave node. Each slave node is used to download the target migration data from the source host according to the sub-migration task and upload the target migration data to the target host.

[0095] Optionally, after receiving the migration task, the master node of the target migration cluster determines the data to be migrated and the size of the data to be migrated according to the migration type in the migration task, and divides the migration task into multiple sub-migration tasks according to the division rules corresponding to the migration object. It can be understood that different data to be migrated corresponds to different division rules.

[0096] Exemplarily, if the migration type is object storage migration, the source storage bucket storing the object in the source host can be determined to determine the data to be migrated and the size of the data to be migrated, and then the migration object can be downloaded from the source storage bucket and uploaded to the target storage bucket; if the migration type is database migration, the size of the data to be migrated can be determined based on the database in the source host. The migration process includes obtaining the creation script of the database in the source host, creating a corresponding database in the target host according to the creation script, generating an insert statement based on the data in the database in the source host, and executing the insert statement in the database in the target host to complete the database migration.

[0097] Optionally, the master node may receive the heartbeat, monitoring-related instructions, and migration tasks sent by the target migration device, and return the processing result to the target migration device after performing corresponding processing.

[0098] Optionally, there are multiple sub-migration task processes in the slave node. After receiving a sub-migration task, a slave node stores the sub-migration task in a cache, and multiple idle sub-migration task processes execute the sub-migration task concurrently. After the migration is completed, each sub-migration process sends the migration result to the master node.

[0099] Optionally, as shown in FIG4 , which is a schematic diagram of the operation of the migration cluster, the target migration device 101 sends the migration task to the master node 103 (master), and performs a status check and a heartbeat check on the master node 103. The master node 103 includes a task partitioning module 401 (task partitioning), a monitoring module 402 (monitor), and a result summarizing module 403 (result summarizing), wherein the task partitioning module 401 divides the entire migration task according to certain rules, and the master node 103 sends the task to multiple slave nodes (worker) 104; the monitoring module 402 monitors the working status of each slave node and the execution status of the sub-migration task; the result summarizing module 403 serially summarizes the execution results of the sub-migration tasks of multiple slave nodes 104; the slave node 104 includes a task scheduling module 404 (subtask scheduling) and a migration process pool 405 (migrateprocess poll), wherein the task scheduling module 404 is responsible for the scheduling of sub-tasks within the slave node; the migration process pool 405 is a migration execution module, which is mainly responsible for executing migration sub-tasks.

[0100] By way of example, the migration type is object storage migration, and the process of the above data migration is described as follows:

[0101] 1) After receiving the data migration request, the request is verified for parameters, and a migration task is generated based on the data migration request. At the same time, it is determined that the migration type is object storage migration, and whether there is an idle object storage migration cluster in the current migration cluster. If not, the relevant function is called to create one based on the amount of migration task data. After the object storage migration cluster is created, the migration task is sent to the object migration cluster.

[0102] 2) After receiving the migration task, the object storage migration cluster initializes the master node and the slave node. In the object storage migration sub-cluster, the sub-task division module of the master node is implemented by listing the objects to be migrated from the object storage of the source host, dividing the migration objects into multiple ones, and sending the sub-migration tasks to the slave node through remote procedure call (RPC).

[0103] 3) After receiving the sub-migration task from the node, the sub-task is added to the cache, and the process function in the migration process pool is implemented to download the object from the source host and upload it to the target host.

[0104] 4) The task scheduling module of the slave node is responsible for sending the sub-migration tasks in the cache to the idle migration process. The migration process is responsible for executing the migration. After the migration is completed, the migration execution result of the sub-migration task is sent back to the master through RPC in the process callback.

[0105] 5) After receiving the subtask results, the master sends the results to the result summary module to serially process the migration results, count relevant information, such as the total migration amount, the number of failures, etc., and decides whether to wake up the subtask division module based on the cache size of the worker carried by the RPC, and send the next batch of tasks to the worker.

[0106] 6) The master result summary module regularly processes the subtask execution results, persists them, and determines whether the migration task has been completed.

[0107] As an optional implementation, as shown in FIG5 , the data migration method provided in the embodiment of the present application may include the following specific steps:

[0108] S501: Receive a data migration request.

[0109] The data migration request includes information of the source host, information of the target host and the migration type.

[0110] S502: Determine a migration task according to the migration request.

[0111] S503: Determine, according to the migration type, at least one candidate migration cluster corresponding to the migration type.

[0112] S504: If there is an idle migration cluster in at least one candidate migration cluster, determine the idle migration cluster as a target migration cluster.

[0113] S505: If there is no idle migration cluster in at least one candidate migration cluster, generate a target migration cluster according to the migration type and a preset migration cluster image.

[0114] S506, sending the migration task to the master node of the target migration cluster, so that the master node obtains the target migration data according to the migration task, and determines multiple sub-migration tasks according to the target migration data, and sends each sub-migration task to each slave node. Each slave node is used to download the target migration data from the source host according to the sub-migration task, and upload the target migration data to the target host.

[0115] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0116] Based on the same inventive concept, the embodiment of the present application also provides a data migration device for implementing the data migration method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more data migration device embodiments provided below can refer to the limitations on the data migration method above, and will not be repeated here.

[0117] In an exemplary embodiment, as shown in FIG6 , a data migration device 600 is provided, which is used for a target data migration device in a data migration system. The data migration system includes multiple data migration devices, the target data migration device is any one of the multiple data migration devices, and the multiple data migration devices form multiple migration clusters. The data migration device includes a receiving module 601, a determining module 602, and a migration module 603. Among them:

[0118] The receiving module 601 is used to receive a data migration request, where the data migration request includes information of a source host, information of a target host, and a migration type;

[0119] A determination module 602 is used to determine a migration task according to a migration request and determine a target migration cluster according to a migration type;

[0120] Migration module 603 is used to send the migration task to the target migration cluster so that the target migration cluster migrates data from the source host to the target host by executing the migration task; wherein, the target migration cluster includes a master node and multiple slave nodes, the master node is used to divide the migration task into multiple sub-migration tasks, and send each sub-migration task to each slave node, and each slave node is used to execute the sub-migration task.

[0121] In an exemplary embodiment, the determination module 602 is specifically configured to determine at least one candidate migration cluster corresponding to the migration type according to the migration type; and determine the target migration cluster according to the working status of the candidate migration cluster.

[0122] In an exemplary embodiment, the determination module 602 is specifically configured to determine the migration cluster in the idle state as the target migration cluster if there is a migration cluster in the idle state in at least one candidate migration cluster.

[0123] In an exemplary embodiment, the determination module 602 is further configured to generate a target migration cluster according to the migration type and a preset migration cluster image if there is no migration cluster in an idle state in at least one candidate migration cluster.

[0124] In an exemplary embodiment, the migration module 603 is specifically used to send the migration task to the master node of the target migration cluster, so that the master node obtains the target migration data according to the migration task, and determines multiple sub-migration tasks according to the target migration data, and sends each sub-migration task to each slave node. Each slave node is used to download the target migration data from the source host according to the sub-migration task, and upload the target migration data to the target host.

[0125] In an exemplary embodiment, the migration type includes host migration, object storage migration, file storage migration or database migration, and the migration cluster includes a host migration cluster, an object storage migration cluster, a file storage migration cluster and a database migration cluster.

[0126] Each module in the above data migration device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of the processor in the data migration device in the form of hardware, or can be stored in the memory in the data migration device in the form of software, so that the processor can call and execute the operations corresponding to each module above.

[0127] In an exemplary embodiment, a data migration device is provided, which may be a cloud server, and its internal structure diagram may be shown in FIG7. The data migration device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the data migration device is used to provide computing and control capabilities. The memory of the data migration device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the data migration device is used to store data related to data migration. The input / output interface of the data migration device is used to exchange information between the processor and an external device. The communication interface of the data migration device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data migration method is implemented.

[0128] Those skilled in the art will understand that the structure shown in FIG. 7 is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the data migration device to which the scheme of the present application is applied. The specific data migration device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0129] In an exemplary embodiment, a data migration device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0130] receiving a data migration request, the data migration request including information of a source host, information of a target host, and a migration type;

[0131] Determine the migration task based on the migration request, and determine the target migration cluster based on the migration type;

[0132] The migration task is sent to the target migration cluster so that the target migration cluster migrates the data from the source host to the target host by executing the migration task; wherein the target migration cluster includes a master node and multiple slave nodes, the master node is used to divide the migration task into multiple sub-migration tasks, and send each sub-migration task to each slave node, and each slave node is used to execute the sub-migration task.

[0133] In one embodiment, when the processor executes the computer program, the following steps are further implemented: determining at least one candidate migration cluster corresponding to the migration type according to the migration type; and determining a target migration cluster according to the working status of the candidate migration cluster.

[0134] In one embodiment, when the processor executes the computer program, the following steps are further implemented: if there is an idle migration cluster in at least one candidate migration cluster, the idle migration cluster is determined as a target migration cluster.

[0135] In one embodiment, when the processor executes the computer program, the following steps are further implemented: if there is no idle migration cluster in at least one candidate migration cluster, a target migration cluster is generated according to the migration type and a preset migration cluster image.

[0136] In one embodiment, when the processor executes the computer program, the following steps are also implemented: sending the migration task to the master node of the target migration cluster so that the master node obtains the target migration data according to the migration task, and determines multiple sub-migration tasks according to the target migration data, and sends each sub-migration task to each slave node, and each slave node is used to download the target migration data from the source host according to the sub-migration task, and upload the target migration data to the target host.

[0137] In one embodiment, the migration type includes host migration, object storage migration, file storage migration or database migration, and the migration cluster includes a host migration cluster, an object storage migration cluster, a file storage migration cluster and a database migration cluster.

[0138] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0139] receiving a data migration request, the data migration request including information of a source host, information of a target host, and a migration type;

[0140] Determine the migration task based on the migration request, and determine the target migration cluster based on the migration type;

[0141] The migration task is sent to the target migration cluster so that the target migration cluster migrates the data from the source host to the target host by executing the migration task; wherein the target migration cluster includes a master node and multiple slave nodes, the master node is used to divide the migration task into multiple sub-migration tasks, and send each sub-migration task to each slave node, and each slave node is used to execute the sub-migration task.

[0142] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: determining at least one candidate migration cluster corresponding to the migration type according to the migration type; and determining a target migration cluster according to the working status of the candidate migration cluster.

[0143] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: if there is an idle migration cluster in at least one candidate migration cluster, the idle migration cluster is determined as a target migration cluster.

[0144] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: if there is no idle migration cluster in at least one candidate migration cluster, generating a target migration cluster according to the migration type and a preset migration cluster image.

[0145] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: sending the migration task to the master node of the target migration cluster so that the master node obtains the target migration data according to the migration task, and determines multiple sub-migration tasks according to the target migration data, and sends each sub-migration task to each slave node, and each slave node is used to download the target migration data from the source host according to the sub-migration task, and upload the target migration data to the target host.

[0146] In one embodiment, the migration type includes host migration, object storage migration, file storage migration or database migration, and the migration cluster includes a host migration cluster, an object storage migration cluster, a file storage migration cluster and a database migration cluster.

[0147] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0148] receiving a data migration request, the data migration request including information of a source host, information of a target host, and a migration type;

[0149] Determine the migration task based on the migration request, and determine the target migration cluster based on the migration type;

[0150] The migration task is sent to the target migration cluster so that the target migration cluster migrates the data from the source host to the target host by executing the migration task; wherein the target migration cluster includes a master node and multiple slave nodes, the master node is used to divide the migration task into multiple sub-migration tasks, and send each sub-migration task to each slave node, and each slave node is used to execute the sub-migration task.

[0151] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: determining at least one candidate migration cluster corresponding to the migration type according to the migration type; and determining a target migration cluster according to the working status of the candidate migration cluster.

[0152] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: if there is an idle migration cluster in at least one candidate migration cluster, the idle migration cluster is determined as a target migration cluster.

[0153] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: if there is no idle migration cluster in at least one candidate migration cluster, generating a target migration cluster according to the migration type and a preset migration cluster image.

[0154] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: sending the migration task to the master node of the target migration cluster so that the master node obtains the target migration data according to the migration task, and determines multiple sub-migration tasks according to the target migration data, and sends each sub-migration task to each slave node, and each slave node is used to download the target migration data from the source host according to the sub-migration task, and upload the target migration data to the target host.

[0155] In one embodiment, the migration type includes host migration, object storage migration, file storage migration or database migration, and the migration cluster includes a host migration cluster, an object storage migration cluster, a file storage migration cluster and a database migration cluster.

[0156] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0157] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0158] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data migration method, characterized in that: A target data migration device in a data migration system, the data migration system comprising a plurality of data migration devices, the target data migration device being any one of the plurality of data migration devices, the plurality of data migration devices forming a plurality of migration clusters, the method comprising: receiving a data migration request, wherein the data migration request includes information of a source host, information of a target host, and a migration type; Determine a migration task according to the migration request, and determine a target migration cluster according to the migration type; The migration task is sent to a target migration cluster so that the target migration cluster migrates data from the source host to the target host by executing the migration task; wherein the target migration cluster includes a master node and multiple slave nodes, the master node is used to divide the migration task into multiple sub-migration tasks, and send each of the sub-migration tasks to each slave node, and each of the slave nodes is used to execute the sub-migration task.

2. The method according to claim 1, characterized in that The determining a target migration cluster according to the migration type includes: Determine, according to the migration type, at least one candidate migration cluster corresponding to the migration type; A target migration cluster is determined according to the working status of the candidate migration cluster.

3. The method according to claim 2, characterized in that The determining the target migration cluster according to the working status of the candidate migration cluster includes: If there is an idle migration cluster in the at least one candidate migration cluster, the idle migration cluster is determined as the target migration cluster.

4. The method according to claim 3, characterized in that The method further comprises: If there is no migration cluster in an idle state in the at least one candidate migration cluster, the target migration cluster is generated according to the migration type and a preset migration cluster image.

5. The method according to claim 1, characterized in that The step of sending the migration task to a target migration cluster so that the target migration cluster migrates data from the source host to the target host by executing the migration task includes: The migration task is sent to the master node of the target migration cluster, so that the master node obtains the target migration data according to the migration task, and determines multiple sub-migration tasks according to the target migration data, and sends each sub-migration task to each slave node. Each slave node is used to download the target migration data from the source host according to the sub-migration task, and upload the target migration data to the target host.

6. The method according to claim 1, characterized in that The migration type includes host migration, object storage migration, file storage migration or database migration, and the migration cluster includes a host migration cluster, an object storage migration cluster, a file storage migration cluster and a database migration cluster.

7. A data migration device, characterized in that: A target data migration device is provided in a data migration system, the data migration system includes a plurality of data migration devices, the target data migration device is any one of the plurality of data migration devices, the plurality of data migration devices form a plurality of migration clusters, and the apparatus includes: A receiving module, configured to receive a data migration request, wherein the data migration request includes information of a source host, information of a target host, and a migration type; A determination module, configured to determine a migration task according to the migration request, and determine a target migration cluster according to the migration type; A migration module, used for sending the migration task to a target migration cluster, so that the target migration cluster migrates data from the source host to the target host by executing the migration task; wherein, the target migration cluster includes a master node and multiple slave nodes, the master node is used for dividing the migration task into multiple sub-migration tasks, and sending each of the sub-migration tasks to each slave node, and each of the slave nodes is used for executing the sub-migration tasks.

8. A data migration device, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images