Data migration method and device, electronic equipment and readable storage medium

Abstract

The present disclosure provides a data migration method and device, electronic equipment and readable storage medium, specifically relates to the technical field of cloud computing and cloud storage, and can be applied to an intelligent cloud scene. A specific implementation scheme is as follows: in the process of executing a data migration task, the working state of a master node is detected according to a preset detection period, and the data migration task is sent by the master node; in response to detecting that the working state of the master node is abnormal, the execution of the data migration task is suspended; and in response to receiving an execution request of the data migration task, the execution of the data migration task is continued, and the execution request is sent by the master node to a slave node in response to the working state of the master node being normal. In the scheme, the slave node can suspend the data migration task after the master node is abnormal, and continue to execute the data migration task after the master node recovers to normal, so that the data migration task can be quickly and effectively completed, and the processing efficiency of the data migration task is ensured.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, specifically to the fields of cloud computing and cloud storage technology, and can be applied in intelligent cloud scenarios. Specifically, this disclosure relates to a data migration method, apparatus, electronic device, and readable storage medium. Background Technology

[0002] Cloud storage is a network storage technology that facilitates resource management by storing resources in the cloud.

[0003] In cloud storage, data stored in different clouds is generally isolated from each other, creating a need to migrate data stored in different clouds.

[0004] In related technologies, a master-worker model is generally used for data migration, where the master node manages and distributes data migration tasks, and the worker nodes execute these tasks. However, the master node may fail in this scenario. Therefore, how to quickly and effectively complete data migration tasks in the event of a master node failure has become a significant technical challenge. Summary of the Invention

[0005] To address at least one of the aforementioned deficiencies, this disclosure provides a data migration method, apparatus, electronic device, and readable storage medium.

[0006] According to a first aspect of this disclosure, a data migration method is provided, the method comprising:

[0007] During the execution of the data migration task, the working status of the master node is detected according to the preset detection cycle. The data migration task is sent by the master node.

[0008] In response to the detection that the master node's working status is abnormal, the data migration task is suspended.

[0009] In response to receiving an execution request for the data migration task, the data migration task continues to be executed. The execution request is sent from the master node to the slave node in response to the master node's normal working status.

[0010] According to a second aspect of this disclosure, another data migration method is provided, the method comprising:

[0011] In response to the master node's working status changing from abnormal to normal, if it is determined that the data migration tasks in the slave nodes were in progress when the master node's working status was abnormal, the data migration tasks will be suspended by the slave nodes.

[0012] Send an execution request for the data migration task to the slave node so that the slave node can continue to execute the data migration task.

[0013] According to a third aspect of this disclosure, a data migration apparatus is provided, the apparatus comprising:

[0014] The master node working status determination module is used to detect the working status of the master node according to a preset detection cycle during the execution of the data migration task. The data migration task is sent by the master node.

[0015] The task pause module is used to pause the execution of data migration tasks in response to the detection of an abnormal working status of the master node;

[0016] The task execution module is used to continue executing the data migration task in response to a received execution request. The execution request is sent from the master node to the slave node in response to the master node's normal working status.

[0017] According to a fourth aspect of this disclosure, another data migration apparatus is provided, the apparatus comprising:

[0018] The task determination module is used to respond to the master node's working status changing from abnormal to normal, and to determine the data migration tasks that are in the process of execution on each slave node when the master node's working status is abnormal. The data migration tasks are then suspended by the slave nodes.

[0019] The task restart module is used to send execution requests for the data migration task to the slave nodes, so that the slave nodes can continue to execute the data migration task.

[0020] According to a fifth aspect of this disclosure, an electronic device is provided, the electronic device comprising:

[0021] At least one processor; and

[0022] A memory communicatively connected to at least one of the aforementioned processors; wherein,

[0023] The memory stores instructions that can be executed by at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the data migration method.

[0024] According to a sixth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause a computer to perform the data migration method described above.

[0025] According to a seventh aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the data migration method described above.

[0026] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0027] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0028] Figure 1 This is a flowchart illustrating a data migration method provided in an embodiment of this disclosure;

[0029] Figure 2 This is a flowchart illustrating another data migration method provided in an embodiment of this disclosure;

[0030] Figure 3 This is a flowchart illustrating a specific implementation of the data migration method provided in this disclosure.

[0031] Figure 4 This is a schematic diagram of the structure of a data migration device provided in an embodiment of this disclosure;

[0032] Figure 5 This is a schematic diagram of another data migration device provided in an embodiment of this disclosure;

[0033] Figure 6 This is a block diagram of an electronic device used to implement the data migration method of the embodiments of this disclosure. Detailed Implementation

[0034] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0035] In cloud storage technology, data stored in different clouds is generally isolated from each other. However, when there are business needs such as data disaster recovery backup, business migration, and architecture adjustment, it is necessary to migrate the data stored in different clouds.

[0036] In related technologies, a master-worker model is generally used for data migration, where the master node manages and distributes data migration tasks, and the worker nodes execute these tasks. However, the master node may fail in this scenario. Therefore, how to quickly and effectively complete data migration tasks in the event of a master node failure has become a significant technical challenge.

[0037] If the data migration tasks being executed on the slave node are discarded when the master node is in an abnormal state, and then redistributed after the master node returns to normal, this will result in a loss of workload for the slave node and a waste of processing resources.

[0038] The data migration methods, apparatuses, electronic devices, and computer-readable storage media provided in this disclosure are intended to solve at least one of the above-mentioned technical problems of the prior art.

[0039] Figure 1 A flowchart illustrating a data migration method provided in an embodiment of this disclosure is shown, as follows: Figure 1 As shown, the method can mainly include:

[0040] Step S110: During the execution of the data migration task, the working status of the master node is detected according to the preset detection cycle. The data migration task is sent by the master node.

[0041] Step S120: In response to the detection that the master node's working status is abnormal, suspend the execution of the data migration task;

[0042] Step S130: In response to the detection that the master node's working status is abnormal, the data migration task is suspended.

[0043] Among them, the data migration task can be used to migrate files from the source database to the target database. The source database and the target database can be stored on different clouds.

[0044] The task information of a data migration task can include the file identifier of the file to be migrated and the storage address of the file to be migrated in the source database, so that the slave node can obtain the file to be migrated based on the task information and send it to the target database for storage.

[0045] The solution provided in this embodiment employs a master-worker model for data migration. The master node manages and distributes data migration tasks and records their progress in real time. The slave nodes execute the data migration tasks and provide timely feedback on their progress. When data migration is required, the master node establishes a communication connection with the worker nodes, sending the data migration task to them for execution.

[0046] The master node may experience abnormal operation due to network issues, abnormal process termination, or other reasons. During data migration tasks, slave nodes can check the master node's status at a preset detection period to obtain real-time information. For example, the detection period could be every 2 seconds, meaning the slave node checks the master node's status every 2 seconds.

[0047] When a slave node detects an abnormal operating status of the master node, it can pause the data migration task and wait for the master node to return to normal. Once the master node returns to normal, it can identify the data migration task that was in progress but was paused, and then send an execution request to the slave node, enabling the slave node to resume the data migration task.

[0048] The method provided in this disclosure, during the execution of a data migration task distributed by a master node by a slave node, detects the working status of the master node according to a preset detection cycle; in response to detecting an abnormal working status of the master node, suspends the execution of the data migration task; and in response to receiving an execution request sent by the master node after its working status returns to normal, resumes the execution of the data migration task. In this solution, the slave node can pause the data migration task after the master node experiences an abnormality and resume the data migration task after the master node recovers, enabling the data migration task to be completed quickly and effectively, ensuring the processing efficiency of the data migration task.

[0049] In this embodiment, the master node typically manages multiple slave nodes. The master node can distribute data migration tasks to these slave nodes, enabling distributed execution of the data migration tasks. When the master node's working state becomes abnormal, it cannot manage the data migration tasks being executed by the slave nodes. If the data migration tasks currently being executed on the slave nodes are discarded, and the master node's working state returns to normal before re-selecting a slave node to distribute the data migration tasks, this would result in a loss of workload for the slave nodes due to the discarded data migration tasks, leading to a waste of processing resources. In this solution, the tasks being processed by the slave nodes are paused. After the master node's working state returns to normal, the paused data migration tasks are identified, and an execution request is sent to the original slave node, allowing the slave node to continue executing the data migration tasks. Based on this solution, the discarding of currently executing data migration tasks can be avoided, preventing a loss of workload for the slave nodes and avoiding a waste of processing resources.

[0050] In one alternative embodiment of this disclosure, after pausing the data migration task, the method further includes:

[0051] Determine the execution schedule of the data migration task;

[0052] Continue performing the data migration task, including:

[0053] The data migration task will continue to be executed based on the progress.

[0054] In this embodiment of the disclosure, after the slave node suspends the execution of the data migration task, it can determine the execution progress of the data migration task and record the execution progress so that when the receiving end receives the execution request, it can continue to execute the data migration task based on the execution progress. This helps to quickly restore the execution status of the data migration task and ensure that the data migration task is completed quickly.

[0055] In one optional embodiment of this disclosure, the data migration task includes sub-data migration tasks for at least two files to be migrated, each sub-data migration task forming a sub-task queue according to its execution order, and determining the execution progress of the data migration task, including:

[0056] Identify the target sub-data migration tasks that have not been completed within the sub-data migration tasks;

[0057] The data migration task will continue to be executed based on the progress, including:

[0058] The target sub-data migration tasks are executed sequentially according to the order in the sub-task queue.

[0059] In this embodiment of the disclosure, the data migration task may generally include sub-data migration tasks for multiple files to be migrated, namely, obtaining the files to be migrated from the source database and sending the files to be migrated to the target database for storage.

[0060] The migration order of each file to be migrated can be configured according to actual needs. The migration order of the files to be migrated is the execution order of each sub-data migration task. For example, the files to be migrated can be sorted by their file names.

[0061] In this embodiment, the slave node can record target sub-data migration tasks that have not been completed, that is, record the execution progress by recording the target sub-data migration tasks that have not been completed. After receiving an execution request, the target sub-data migration tasks are executed sequentially according to the order in the sub-task queue. As an example, since the sub-data migration tasks in the sub-task queue are arranged in execution order, the sub-data migration task that is being executed when execution is paused can be recorded, and the sub-data migration task can be used as the starting point when restarting the execution task.

[0062] As an example, the main steps in the execution process of each sub-data migration task may include: obtaining the file to be migrated from the source database, verifying the file to be migrated, and sending the file to be migrated to the target database for storage. The execution progress can also be the status of each step in the current sub-data migration task. After restarting the execution task, each unexecuted step will be executed in sequence.

[0063] As an example, when the amount of data in the file to be migrated is large, the file to be migrated can be split into multiple file blocks, and sub-migration tasks can be created for each file block. At this time, the sub-task queue can include sub-migration tasks for each file block.

[0064] In one optional method of this disclosure, the working status of the master node is detected according to a preset detection cycle, including:

[0065] Send heartbeat information to the master node according to the preset detection cycle;

[0066] The master node's working status is determined based on whether the received receipt information meets the preset working status judgment conditions. The receipt information is returned by the master node in response to receiving the heartbeat information.

[0067] In this embodiment of the disclosure, the slave node can periodically send heartbeat information to the master node, and after receiving the heartbeat information, the master node can return receipt information to the slave node.

[0068] In this embodiment of the disclosure, working status judgment conditions corresponding to different working states can be pre-configured so as to determine the corresponding working state based on the working status judgment conditions satisfied by the received receipt information.

[0069] As an example, the working status judgment condition when the working status is abnormal is that no acknowledgment information is received within a preset time period. That is, if the master node does not return acknowledgment information within the preset time period, the working status of the current master node can be judged to be abnormal.

[0070] In one alternative embodiment of this disclosure, the method further includes the following steps before performing the data migration task:

[0071] Send heartbeat information to the master node and receive data migration tasks. The data migration task is the response from the master node in response to receiving the heartbeat information.

[0072] In this embodiment of the disclosure, the slave node and the master node can maintain their communication connection through heartbeat information during the data migration process.

[0073] When data migration is required, the slave node can establish a communication connection with the master node by sending heartbeat information to the master node. After receiving the heartbeat information, the master node can assign data migration tasks to the slave node.

[0074] In this embodiment, the slave node actively establishes a communication connection with the master node and then obtains the data migration task. The slave node does not need to register with the master node, and the master node does not need to maintain the slave node's information, which facilitates the expansion of more slave nodes to perform data transfer tasks.

[0075] In this embodiment of the disclosure, the slave node and the master node maintain a communication connection during the data migration process. After the data migration is completed, the slave node and the master node no longer need to maintain a communication connection, and the master node does not need to maintain the slave node's information, thus reducing the master node's maintenance cost for the slave node.

[0076] In one alternative embodiment of this disclosure, after the data migration task is completed, the method further includes:

[0077] In response to the completion of the data migration task, a first notification message is sent to the master node so that the master node can update the migration progress of the data migration task based on the first notification message.

[0078] In this embodiment of the disclosure, after the data migration task is completed, the slave node can return a first notification message to the master node. By maintaining the migration progress of each data migration task, the master node can effectively manage the data migration tasks.

[0079] As an example, after the master node distributes a data migration task, it can record that the task is in a distributed state and that the task has been distributed to the slave nodes. Once the slave node completes the task, it can return a first notification message, and the master node can record the migration progress as complete. The master node can also send migration progress information to the task record database so that it can retrieve the migration progress of each data migration task after an abnormal restart.

[0080] In one alternative embodiment of this disclosure, the method further includes:

[0081] In response to the failure of the data migration task, a second notification message is sent to the master node so that the master node can redistribute the data migration task based on the second notification message. The second notification message includes information related to the failure description.

[0082] In this embodiment of the disclosure, when the data migration task fails, the slave node can generate a second notification message and send the second notification message to the master node. The second notification message may include failure description information, which can reflect the reason for the task failure. The master node can determine the redistribution strategy of the data migration task based on the reason for the task failure.

[0083] As an example, failure description information may include, but is not limited to, failure type identifier, failure reason description information, etc.

[0084] As an example, the failure description information may include reasons such as: recoverable failure of the slave node, unrecoverable failure of the slave node, source database failure, and target database failure. The master node can assign appropriate redistribution strategies for different failure reasons. Specifically, for recoverable failure of the slave node, the redistribution strategy is to resend the data migration task to the slave node with the recoverable failure. For unrecoverable failure of the slave node, the redistribution strategy is to select a new slave node to send the data migration task. For source database failure and target database failure, these failure reasons may not be overcome by re-executing the data migration task. In this case, the data migration task can be stopped from being sent, and the migration progress of the data migration task can be recorded as a migration failure.

[0085] In one alternative embodiment of this disclosure, the method further includes:

[0086] The file to be migrated and its characteristic information are obtained from the source database. The characteristic information is used to verify the file to be migrated.

[0087] The files to be migrated and their feature information are sent to the data writing node, so that the data writing node can verify the files to be migrated based on the feature information, and write the files to be migrated to the target database after the verification is successful.

[0088] In this embodiment of the disclosure, when obtaining the file to be migrated from the source database, the feature information of the file to be migrated can also be obtained at the same time. The feature information is used to verify the file to be migrated. The feature information may include, but is not limited to, the Message-Digest Algorithm 5 (MD5) value.

[0089] The data write node receives files to be migrated from the slave node and writes them to the target database. Before writing the files to be migrated, the data write node can verify the files based on characteristic information to ensure the consistency of the migrated files before and after the migration.

[0090] In this embodiment of the disclosure, the task information of the data migration task may further include permission verification information of the source database, which is used to verify the permissions when obtaining the file to be migrated from the source database, so as to have the permission to obtain the file to be migrated. By providing permission verification information in the task information of the data migration task, when the data migration task is assigned to different slave nodes, each slave node can effectively obtain the file to be migrated from the source database, which facilitates the expansion of slave nodes.

[0091] Figure 2 A flowchart illustrating another data migration method provided in an embodiment of this disclosure is shown, such as... Figure 2As shown, the method can mainly include:

[0092] Step S210: In response to the master node's working status changing from abnormal to normal, determine the data migration tasks that are in the process of execution in each slave node when the master node's working status is abnormal, and suspend the execution of the data migration tasks by the slave nodes.

[0093] Step S220: Send an execution request for the data migration task to the slave node so that the slave node can continue to execute the data migration task.

[0094] Among them, the data migration task can be used to migrate files from the source database to the target database. The source database and the target database can be stored on different clouds.

[0095] The task information of a data migration task can include the file identifier of the file to be migrated and the storage address of the file to be migrated in the source database, so that the slave node can obtain the file to be migrated based on the task information and send it to the target database for storage.

[0096] The solution provided in this embodiment employs a master-slave mode for data migration. The master node manages and distributes data migration tasks and records their progress in real time. The slave node executes the data migration tasks and provides timely feedback on their progress. When data migration is required, the master node establishes a communication connection with the slave node, sending the data migration task to it for execution.

[0097] The master node may experience abnormal operation due to network issues, abnormal process termination, or other reasons. During data migration tasks, slave nodes can check the master node's status at a preset detection period to obtain real-time information. For example, the detection period could be every 2 seconds, meaning the slave node checks the master node's status every 2 seconds.

[0098] When a slave node detects an abnormal operating status of the master node, it can pause the data migration task and wait for the master node to return to normal. Once the master node returns to normal, it can identify the data migration task that was in progress but was paused, and then send an execution request to the slave node, enabling the slave node to resume the data migration task.

[0099] The method provided in this disclosure, in response to the master node's working state changing from abnormal to normal, identifies data migration tasks in progress on each slave node when the master node's working state was abnormal. The data migration tasks are then suspended by the slave nodes, and an execution request is sent to each slave node to enable it to resume execution. In this solution, slave nodes can pause data migration tasks after the master node experiences an abnormality and resume execution after the master node recovers, enabling rapid and efficient completion of data migration tasks and ensuring processing efficiency.

[0100] In one optional embodiment of this disclosure, the method further includes:

[0101] In response to receiving a data migration request initiated by a user, obtain the task information carried in the data migration request;

[0102] Generate data migration tasks based on task information;

[0103] In response to receiving a heartbeat message from a slave node, a data migration task is returned to the slave node.

[0104] In this embodiment of the disclosure, the user can initiate a data migration request and set the task information in the data migration request.

[0105] As an example, task information may include the file identifier of the file to be migrated and the storage address of the file to be migrated in the source database, enabling the slave node to obtain the file to be migrated based on the task information and send it to the target database for storage.

[0106] As an example, the task information for data migration requests can be stored in a task record database, and the master node can retrieve the task information for data migration requests from the task record database.

[0107] The master node can generate data migration tasks based on information. After the data migration task is generated, it can be in a pending distribution state, waiting for the slave node to establish a communication connection with the master node through heartbeat information, and then send the data migration task to the slave node.

[0108] As an example, during the data migration process, users can further select the files to be migrated, for instance, choosing not to migrate certain specified files. The master node can filter the data migration tasks corresponding to these files and prevent them from being distributed, thus preventing these files from being migrated.

[0109] As an example, the master node can receive data migration tasks initiated by multiple different users. Different users have different migration priorities. Data migration tasks initiated by users with higher migration priorities can be processed first. For example, a slave node with better processing capabilities can be selected to process data migration tasks initiated by users with higher priority, or more slave nodes can be assigned to process data migration tasks initiated by users with higher priority.

[0110] In this embodiment of the disclosure, the task information of the data migration task may further include permission verification information of the source database, which is used to verify the permissions when obtaining the file to be migrated from the source database, so as to have the permission to obtain the file to be migrated. By providing permission verification information in the task information of the data migration task, when the data migration task is assigned to different slave nodes, each slave node can effectively obtain the file to be migrated from the source database, which facilitates the expansion of slave nodes.

[0111] In one optional embodiment of this disclosure, after returning the data migration task to the slave node, the method further includes:

[0112] In response to receiving the first notification message, the migration progress of the data migration task is updated based on the first notification message, which is sent by the slave node after the data migration task is completed.

[0113] In this embodiment of the disclosure, after the data migration task is completed, the slave node can return a first notification message to the master node. Upon receiving the first notification message, the master node can know that the data migration task has been completed. By maintaining the migration progress of each data migration task, the master node can effectively manage the data migration tasks.

[0114] As an example, after the master node distributes a data migration task, it can record that the task is in a distributed state and that the task has been distributed to the slave nodes. Once the slave node completes the task, it can return a first notification message, and the master node can record the migration progress as complete. The master node can also send migration progress information to the task record database so that it can retrieve the migration progress of each data migration task after an abnormal restart.

[0115] In one optional embodiment of this disclosure, the migration progress is stored in a task record database, and the data migration tasks in progress on each slave node when the master node's working status is abnormal are identified, including:

[0116] Identify the data migration tasks that are in progress on each slave node when the master node's working status is abnormal from the task record database.

[0117] In this embodiment of the disclosure, the migration progress of each data migration task can be stored in a task record database. As an example, the task record database can be a relational database, and the migration progress of the data migration tasks can be stored in relational data, ensuring Atomicity-Consistency-Isolation-Durability (ACID).

[0118] When the master node is in a normal working state, the migration progress of the data migration task can be stored in the master node's memory. However, when the master node is in an abnormal working state, it may lose the migration progress of the data migration task. When the master node's working state returns to normal, it can retrieve the task progress of the data migration task from the task record database, thereby obtaining the data migration task that is in progress. At the same time, it can also obtain the slave nodes used to execute these data migration tasks, so as to send execution requests to these slave nodes.

[0119] In this embodiment of the disclosure, the master node in an abnormal working state may not be able to be restarted. In this case, it is necessary to enable a new master node, which can be selected from the slave nodes.

[0120] As an example, you can select a slave node that meets your requirements in terms of processing performance, storage performance, and other metrics as the master node.

[0121] In this embodiment of the disclosure, after a slave node is selected as the new master node, the new master node can obtain the migration progress of the data migration task from the task record database. Specifically, the new master node can initiate a request to obtain the migration progress by carrying the node identifier of the original master node, so as to filter the migration progress of the data migration task managed by the original master node from the task record database based on the node identifier of the original master node, and return it to the new master node.

[0122] In this embodiment of the disclosure, when a slave node is selected as a new master node, during the execution of a data migration task that the slave node may be handling, after the slave node becomes the new master node, it can send the task information of the data migration task in the slave node, the execution progress of the data migration task when it is paused, and other information to another slave node (referred to as the target slave node). This allows the target slave node to continue executing the task based on the obtained task information of the data migration task and the execution progress of the data migration task when it is paused.

[0123] In one optional embodiment of this disclosure, after returning the data migration task to the slave node, the method further includes:

[0124] Receive a second notification message, which is sent by the slave node after the data migration task fails. The second notification message includes information related to the failure description.

[0125] The redistribution strategy for data migration tasks is determined based on information related to the failure description.

[0126] In this embodiment of the disclosure, when the data migration task fails, the slave node can generate a second notification message and send the second notification message to the master node. The second notification message may include failure description information, which can reflect the reason for the task failure. The master node can determine the redistribution strategy of the data migration task based on the reason for the task failure.

[0127] As an example, failure description information may include, but is not limited to, failure type identifier, failure reason description information, etc.

[0128] In one optional implementation of this disclosure, a redistribution strategy for the data migration task is determined based on failure description information, including:

[0129] If the failure description information meets the first preset distribution condition, the data migration task will be resent to the slave node;

[0130] If the failure description information meets the second preset distribution condition, the data migration task is resent to the new slave node.

[0131] In this embodiment of the disclosure, the failure reasons reflected in the failure description information can mainly include the following types: recoverable failure of the slave node, unrecoverable failure of the slave node, source database failure, target database failure, etc.

[0132] The master node can assign appropriate redistribution strategies for different failure reasons. Specifically, the slave node recoverable exception can be specified as the first preset type. When the failure reason of the data migration task belongs to the slave node recoverable exception, the redistribution strategy is to resend the data migration task to the slave node with the recoverable exception.

[0133] The slave node unrecoverable exception can be specified as the second preset type. When the failure reason of the data migration task is the slave node unrecoverable exception, the redistribution strategy is to select a new slave node to send the data migration task.

[0134] If the source database or the target database fails, and these reasons cannot be overcome by re-executing the data migration task, then the data migration task can be stopped and its progress recorded as a migration failure.

[0135] As an example, Figure 3 This is a flowchart illustrating a specific implementation of the data migration method provided in this disclosure.

[0136] like Figure 3 As shown, the database is the task record database. The Master module is the master node. The main process of this specific implementation is as follows:

[0137] (1) Users create data migration tasks and set the source database, destination database, files to be migrated, and storage paths for the files to be migrated.

[0138] (2) Task permission verification, that is, to verify the permissions of the user who initiates the data migration task. After the permission verification is passed, the task record is written to the task record database.

[0139] (3) The master node retrieves the data migration task created by the user from the task record database. After the permission verification is passed, it lists the files to be migrated in the source database from the task information and saves the list results in the local file of the master node for long-term storage.

[0140] (4) After listing the files to be migrated, the master node enters the task distribution state, sequentially and batch-reads the files stored in the list, parses out the file information to be transferred, and saves it in memory according to the specified format. The master node can perform task flow control management based on user priority to manage the number of slave nodes allocated to each data migration task. The master node can also check the task status in a timely manner.

[0141] (5) The slave node acts as a Remote Procedure Call (RPC) client, requesting data migration tasks from the master node in batches via RPC. The slave node parses the task information from the data migration tasks, downloads the files to be migrated into memory by calling the download interface of the source database, and after verifying that the data is correct, calls the data upload interface of the target database to upload the data to the specified location.

[0142] (6) After the data migration task is completed, the master node can be notified. The master node will record the migration progress of each data migration task and update the migration progress based on the notification information. At the same time, it will record the data migration tasks that failed to migrate so that the failure can be queried.

[0143] Based on and Figure 1 The method shown follows the same principle. Figure 4 A schematic diagram of the structure of a data migration apparatus provided in an embodiment of this disclosure is shown, as follows: Figure 4 As shown, the data migration device 40 may include:

[0144] The master node working status determination module 410 is used to detect the working status of the master node according to a preset detection cycle during the execution of the data migration task. The data migration task is sent by the master node.

[0145] The task pause module 420 is used to pause the execution of the data migration task in response to the detection that the working status of the master node is abnormal.

[0146] The task execution module 430 is used to continue executing the data migration task in response to receiving an execution request for the data migration task. The execution request is sent from the master node to the slave node in response to the master node's normal working status.

[0147] The apparatus provided in this disclosure, during the execution of a data migration task distributed by a master node by a slave node, detects the working status of the master node according to a preset detection cycle; in response to detecting an abnormal working status of the master node, it suspends the execution of the data migration task; and in response to receiving an execution request sent by the master node after its working status returns to normal, it resumes the execution of the data migration task. In this solution, the slave node can suspend the data migration task after the master node experiences an abnormality and resume the data migration task after the master node recovers, enabling the data migration task to be completed quickly and effectively, ensuring the processing efficiency of the data migration task.

[0148] Optionally, the task pause module is also used for:

[0149] Determine the execution schedule of the data migration task;

[0150] When the task execution module continues to execute the data migration task, it is specifically used for:

[0151] The data migration task will continue to be executed based on the progress.

[0152] Optionally, the data migration task includes sub-data migration tasks for at least two files to be migrated. Each sub-data migration task forms a sub-task queue according to its execution order. The task pause module, when determining the execution progress of the data migration task, is specifically used for:

[0153] Identify the target sub-data migration tasks that have not been completed within the sub-data migration tasks;

[0154] When the task execution module continues to execute the data migration task based on the execution progress, it is specifically used for:

[0155] The target sub-data migration tasks are executed sequentially according to the order in the sub-task queue.

[0156] Optionally, when the master node working status determination module detects the working status of the master node according to a preset detection cycle, it is specifically used for:

[0157] Send heartbeat information to the master node according to the preset detection cycle;

[0158] The master node's working status is determined based on whether the received receipt information meets the preset working status judgment conditions. The receipt information is returned by the master node in response to receiving the heartbeat information.

[0159] Optionally, the above-mentioned device further includes:

[0160] The data migration task receiving module is used to send heartbeat information to the master node before executing the data migration task and to receive the data migration task. The data migration task is the response returned by the master node in response to receiving the heartbeat information.

[0161] Optionally, the above-mentioned device further includes:

[0162] The first notification message sending module sends a first notification message to the master node in response to the completion of the data migration task, so that the master node can update the migration progress of the data migration task based on the first notification message.

[0163] Optionally, the above-mentioned device further includes:

[0164] The second notification information sending module is used to send a second notification information to the master node in response to the failure of the data migration task, so that the master node can redistribute the data migration task based on the second notification information. The second notification information includes relevant information describing the failure.

[0165] Optionally, when executing a data migration task, the task execution module is specifically used for:

[0166] The file to be migrated and its characteristic information are obtained from the source database. The characteristic information is used to verify the file to be migrated.

[0167] The files to be migrated and their feature information are sent to the data writing node, so that the data writing node can verify the files to be migrated based on the feature information, and write the files to be migrated to the target database after the verification is successful.

[0168] It is understood that the above-described modules of the data migration apparatus in the embodiments of this disclosure have the ability to implement... Figure 1 The embodiments shown illustrate the functionality of corresponding steps in the data migration method. This functionality can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-described functions. These modules can be software and / or hardware, and each module can be implemented individually or multiple modules can be integrated. For a detailed description of the functions of each module in the above-described data migration device, please refer to [link to relevant documentation]. Figure 1 The corresponding descriptions of the data migration methods in the embodiments shown are not repeated here.

[0169] Based on and Figure 2 The method shown follows the same principle. Figure 5 A schematic diagram of another data migration apparatus provided in an embodiment of this disclosure is shown, such as... Figure 5 As shown, the data migration device 50 may include:

[0170] The task determination module 510 is used to determine the data migration tasks that are in the process of execution in each slave node when the working status of the master node changes from abnormal to normal in response to the master node's working status being abnormal. The data migration tasks are suspended by the slave nodes.

[0171] The task restart module 520 is used to send an execution request for the data migration task to the slave node so that the slave node can continue to execute the data migration task.

[0172] The apparatus provided in this disclosure, in response to the master node's working state changing from abnormal to normal, determines the data migration tasks in progress on each slave node when the master node's working state was abnormal, and suspends the data migration tasks on the slave nodes; it then sends execution requests for the data migration tasks to the slave nodes so that the slave nodes can continue executing the data migration tasks. In this solution, the slave nodes can suspend the data migration tasks after the master node experiences an abnormality and resume executing the data migration tasks after the master node recovers to normal, enabling the data migration tasks to be completed quickly and effectively, ensuring the processing efficiency of the data migration tasks.

[0173] Optionally, the above apparatus further includes a data migration task sending module, which is used for:

[0174] In response to receiving a data migration request initiated by a user, obtain the task information carried in the data migration request;

[0175] Generate data migration tasks based on task information;

[0176] In response to receiving a heartbeat message from a slave node, a data migration task is returned to the slave node.

[0177] Optionally, the above-mentioned device further includes:

[0178] The migration progress update module is used to update the migration progress of the data migration task based on the first notification information received after returning the data migration task to the slave node. The first notification information is sent by the slave node after the data migration task is completed.

[0179] Optionally, the migration progress is stored in the task record database. When the task determination module determines that a data migration task is in progress on each slave node when the master node's working status is abnormal, it is specifically used for:

[0180] Identify the data migration tasks that are in progress on each slave node when the master node's working status is abnormal from the task record database.

[0181] Optionally, the above apparatus further includes a redistribution strategy determination module, which is used to:

[0182] After returning the data migration task to the slave node, a second notification message is received. The second notification message is sent by the slave node after the data migration task fails. The second notification message includes information related to the failure description.

[0183] The redistribution strategy for data migration tasks is determined based on information related to the failure description.

[0184] Optionally, when determining the redistribution strategy for the data migration task based on failure description information, the redistribution strategy determination module is specifically used for:

[0185] The cause of failure is determined by responding to information related to the failure description;

[0186] If the failure reason falls under the first preset type, the data migration task will be resent to the slave node;

[0187] If the failure reason falls under the second preset type, the data migration task will be resent to a new slave node.

[0188] It is understood that the above-described modules of the data migration apparatus in the embodiments of this disclosure have the ability to implement... Figure 2 The embodiments shown illustrate the functionality of corresponding steps in the data migration method. This functionality can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-described functions. These modules can be software and / or hardware, and each module can be implemented individually or multiple modules can be integrated. For a detailed description of the functions of each module in the above-described data migration device, please refer to [link to relevant documentation]. Figure 2 The corresponding descriptions of the data migration methods in the embodiments shown are not repeated here.

[0189] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0190] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0191] The electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a data migration method as provided in the embodiments of this disclosure.

[0192] Compared with existing technologies, this electronic device, during the execution of data migration tasks distributed by the master node by the slave node, detects the working status of the master node according to a preset detection cycle; in response to detecting an abnormal working status of the master node, it suspends the execution of the data migration task; in response to receiving an execution request sent by the master node after its working status returns to normal, it resumes the execution of the data migration task. In this solution, the slave node can suspend the data migration task after the master node malfunctions and resume the data migration task after the master node recovers, enabling the data migration task to be completed quickly and effectively, ensuring the processing efficiency of the data migration task.

[0193] The readable storage medium is a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to perform a data migration method as provided in the embodiments of this disclosure.

[0194] Compared with existing technologies, this readable storage medium detects the working status of the master node according to a preset detection cycle during the execution of data migration tasks distributed by the master node by the slave node; in response to detecting an abnormal working status of the master node, it suspends the execution of the data migration task; and in response to receiving an execution request sent by the master node after its working status returns to normal, it resumes the execution of the data migration task. In this solution, the slave node can suspend the data migration task after the master node encounters an abnormality and resume the data migration task after the master node recovers, which can quickly and effectively complete the data migration task and ensure the processing efficiency of the data migration task.

[0195] The computer program product includes a computer program that, when executed by a processor, implements the data migration method provided in the embodiments of this disclosure.

[0196] Compared with existing technologies, this computer program product, during the execution of data migration tasks distributed by the master node by the slave node, detects the working status of the master node according to a preset detection cycle; in response to detecting an abnormal working status of the master node, it suspends the execution of the data migration task; in response to receiving an execution request sent by the master node after its working status returns to normal, it resumes the execution of the data migration task. In this solution, the slave node can suspend the data migration task after the master node malfunctions and resume the data migration task after the master node recovers, enabling the data migration task to be completed quickly and effectively, ensuring the processing efficiency of the data migration task.

[0197] Figure 6A schematic block diagram of an example electronic device 60 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0198] like Figure 6 As shown, the electronic device 60 includes a computing unit 610, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 620 or a computer program loaded from a storage unit 680 into a random access memory (RAM) 630. The RAM 630 may also store various programs and data required for the operation of the device 60. The computing unit 610, ROM 620, and RAM 630 are interconnected via a bus 640. An input / output (I / O) interface 650 is also connected to the bus 640.

[0199] Multiple components in device 60 are connected to I / O interface 650, including: input unit 660, such as keyboard, mouse, etc.; output unit 670, such as various types of monitors, speakers, etc.; storage unit 680, such as disk, optical disk, etc.; and communication unit 690, such as network card, modem, wireless transceiver, etc. Communication unit 690 allows device 60 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0200] The computing unit 610 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 610 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 610 performs the data migration method provided in the embodiments of this disclosure. For example, in some embodiments, performing the data migration method provided in the embodiments of this disclosure can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as storage unit 680. In some embodiments, part or all of the computer program can be loaded and / or installed on device 60 via ROM 620 and / or communication unit 690. When the computer program is loaded into RAM 630 and executed by computing unit 610, one or more steps of the data migration method provided in the embodiments of this disclosure can be performed. Alternatively, in other embodiments, the computing unit 610 can be configured to perform the data migration method provided in the embodiments of this disclosure by any other suitable means (e.g., by means of firmware).

[0201] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0202] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0203] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0204] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0205] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0206] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0207] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0208] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A data migration method, comprising: During the execution of the data migration task, the working status of the master node is detected according to a preset detection cycle. The data migration task is sent by the master node. In response to the detection that the master node's working status is abnormal, the slave node suspends the execution of the data migration task and determines the execution progress of the data migration task; In response to receiving an execution request for the data migration task, the slave node continues to execute the data migration task based on the execution progress. The execution request is sent by the master node to the slave node in response to the master node's normal working status. The master node is used to manage and distribute the data migration tasks and record the migration progress of the data migration tasks in real time. The slave node is used to execute the data migration tasks and provide feedback on the migration progress of the data migration tasks.

2. The method according to claim 1, wherein, The data migration task includes sub-data migration tasks for at least two files to be migrated. Each sub-data migration task forms a sub-task queue according to its execution order. Determining the execution progress of the data migration task includes: Identify the target sub-data migration tasks that have not been completed in the sub-data migration tasks; The step of continuing to execute the data migration task based on the execution progress includes: The target sub-data migration tasks are executed sequentially according to the order in the sub-task queue.

3. The method according to claim 1 or 2, wherein, The step of detecting the working status of the master node according to a preset detection cycle includes: Send heartbeat information to the master node according to the preset detection cycle; The working status of the master node is determined based on whether the received receipt information meets the preset working status judgment conditions. The receipt information is returned by the master node in response to receiving the heartbeat information.

4. The method according to any one of claims 1-3, wherein, Before performing the data migration task, the method further includes: Send heartbeat information to the master node and receive the data migration task, wherein the data migration task is returned by the master node in response to receiving the heartbeat information.

5. The method according to any one of claims 1-4, wherein, After the data migration task is completed, the method further includes: In response to the completion of the data migration task, a first notification message is sent to the master node so that the master node updates the migration progress of the data migration task based on the first notification message.

6. The method according to any one of claims 1-5, further comprising: In response to the failure of the data migration task, a second notification message is sent to the master node so that the master node can determine the redistribution strategy of the data migration task based on the second notification message. The second notification message includes information related to the failure description.

7. The method according to any one of claims 1-6, wherein, The execution of the data migration task includes: The file to be migrated and its characteristic information are obtained from the source database. The characteristic information is used for the verification of the file to be migrated. The file to be migrated and the feature information are sent to the data writing node, so that the data writing node can verify the file to be migrated based on the feature information, and write the file to be migrated into the target database after the verification is successful.

8. A data migration method, comprising: In response to the master node's working status changing from abnormal to normal, if the master node's working status is abnormal, the data migration tasks that are in the process of execution in each slave node are determined, and the data migration tasks are suspended by the slave nodes and their execution progress is determined. Send an execution request for the data migration task to the slave node so that the slave node continues to execute the data migration task based on the execution progress; The master node is used to manage and distribute the data migration tasks and record the migration progress of the data migration tasks in real time. The slave node is used to execute the data migration tasks and provide feedback on the migration progress of the data migration tasks.

9. The method according to claim 8, wherein, Also includes: In response to receiving a data migration request initiated by a user, the task information carried in the data migration request is obtained; A data migration task is generated based on the task information; In response to receiving the heartbeat information sent by the slave node, a data migration task is returned to the slave node.

10. The method according to claim 9, wherein, After returning the data migration task to the slave node, the method further includes: In response to receiving a first notification message, the migration progress of the data migration task is updated based on the first notification message, wherein the first notification message is sent by the slave node after the data migration task is completed.

11. The method according to claim 10, wherein, The migration progress is stored in the task record database. The process of determining the data migration tasks in progress on each slave node when the master node's working state is abnormal includes: The task record database is used to determine the data migration tasks that are in progress on each slave node when the master node's working status is abnormal.

12. The method according to any one of claims 9-11, wherein, After returning the data migration task to the slave node, the method further includes: Receive a second notification message, which is sent by the slave node after the data migration task fails to execute, and the second notification message includes information related to the failure description; The redistribution strategy for the data migration task is determined based on the information related to the failure description.

13. The method according to claim 12, wherein, The step of determining the redistribution strategy for the data migration task based on the failure description information includes: The cause of failure is determined in response to the information related to the failure description; If the failure reason belongs to the first preset type, the data migration task is resent to the slave node; If the failure reason falls under the second preset type, the data migration task is resent to a new slave node.

14. A data migration apparatus, comprising: The master node working status determination module is used to detect the working status of the master node according to a preset detection cycle during the execution of the data migration task, wherein the data migration task is sent by the master node. The task pause module is used to pause the execution of the data migration task on the slave node in response to the detection that the working status of the master node is abnormal, and to determine the execution progress of the data migration task. The task execution module is used to respond to receiving an execution request for the data migration task, and the slave node continues to execute the data migration task based on the execution progress. The execution request is sent by the master node to the slave node in response to the master node's normal working status. The master node is used to manage and distribute the data migration tasks and record the migration progress of the data migration tasks in real time. The slave node is used to execute the data migration tasks and provide feedback on the migration progress of the data migration tasks.

15. A data migration apparatus, comprising: The task determination module is used to determine the data migration tasks that are in the process of execution in each slave node when the working status of the master node changes from abnormal to normal in response to the master node's working status being abnormal. The data migration tasks are suspended by the slave nodes and their execution progress is determined. The task restart module is used to send an execution request for the data migration task to the slave node, so that the slave node can continue to execute the data migration task based on the execution progress; The master node is used to manage and distribute the data migration tasks and record the migration progress of the data migration tasks in real time. The slave node is used to execute the data migration tasks and provide feedback on the migration progress of the data migration tasks.

16. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-13.

17. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-13.

18. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-13.

Images