Data migration method, data migration device, data migration equipment and storage medium

Abstract

The application relates to the technical field of storage, and specifically discloses a data migration method, a data migration device, a data migration equipment and a storage medium. When performing a full-data migration task from a source storage cluster to a target storage cluster, a data change log of the source storage cluster compared with the target storage cluster is recorded. After performing the full-data migration task, the incremental data of the source storage cluster compared with the target storage cluster can be determined according to the data change log, and then an incremental data migration task of migrating the incremental data from the source storage cluster to the target storage cluster is performed. Compared with an existing incremental migration scheme in which a migration tool reads all data in the source storage cluster and all data in the target storage cluster into the memory to obtain the incremental data, the incremental data can be determined more quickly, the efficiency of the incremental data migration task is significantly improved, and the overall efficiency of data migration between storage clusters is improved.

Description

Technical Field

[0001] This application relates to the field of storage technology, and in particular to a data migration method, data migration apparatus, data migration device, and storage medium. Background Technology

[0002] When performing master-slave synchronization or failover tasks between storage clusters, migration tools are needed to perform a full migration from the source storage cluster to the target storage cluster. Since full migration tasks are time-consuming, stopping service on the source storage cluster would severely impact user experience and could even cause destructive consequences. Therefore, currently, the source storage cluster continues to provide service during a full migration. This means that users will not stop adding, deleting, modifying, or querying files on the source storage cluster during the full migration, leading to data changes. In this case, incremental data migration between the source and target storage clusters is necessary to ensure complete data consistency.

[0003] When performing incremental data migration tasks, the migration tool first needs to acquire all data in the source storage cluster and all data in the target storage cluster into memory for comparison one by one. When inconsistent data is found, it is confirmed as incremental data. After obtaining all incremental data, the tool starts the task of migrating the incremental data from the source storage cluster to the target storage cluster.

[0004] However, due to the large amount of data in the storage cluster, the time required to compare all the data in the source storage cluster with all the data in the target storage cluster to determine the incremental data when performing incremental data migration tasks between storage clusters can be very long, possibly taking weeks.

[0005] Providing an optimization and improvement scheme for migration tools when performing incremental data migration tasks between storage clusters to improve the efficiency of incremental migration is a technical problem that needs to be solved by those skilled in the art. Summary of the Invention

[0006] The purpose of this application is to provide a data migration method, data migration apparatus, data migration device, and storage medium to improve the efficiency of migration tools in performing incremental migrations between storage clusters.

[0007] To address the aforementioned technical problems, this application provides a data migration method, comprising:

[0008] When performing a full data migration task from the source storage cluster to the target storage cluster, record the data change log of the source storage cluster relative to the target storage cluster;

[0009] After the full data migration task is completed, the incremental data of the source storage cluster compared to the target storage cluster is determined based on the data change log;

[0010] Execute an incremental data migration task to migrate the incremental data from the source storage cluster to the target storage cluster.

[0011] Optionally, the full data migration task from the source storage cluster to the target storage cluster specifically includes:

[0012] The migration tool is invoked to traverse the entire data of the source storage cluster, and this data is designated as the first data to be migrated.

[0013] The migration tool is invoked to perform the full data migration task that migrates the first data to be migrated from the source storage cluster to the target storage cluster.

[0014] Optionally, recording the data change log of the source storage cluster compared to the target storage cluster specifically includes:

[0015] Starting from the first point in time when the migration tool begins to traverse the full data of the source storage cluster, the data change log of the source storage cluster compared to the target storage cluster is recorded.

[0016] Optionally, the incremental data migration task, which involves migrating the incremental data from the source storage cluster to the target storage cluster, specifically includes:

[0017] For the first data change log between the first time point and the second time point when the migration tool starts executing the full data migration task for the first data to be migrated, determine whether the target storage cluster contains incremental data corresponding to the first data change log.

[0018] If the target storage cluster contains all incremental data corresponding to the first data change log, then skip the first data change log;

[0019] If the target storage cluster does not contain all incremental data corresponding to the first data change log, then execute the first incremental data migration task for the incremental data corresponding to the first data change log.

[0020] After determining the incremental data corresponding to the second data change log after the second time point, directly execute the second incremental data migration task for the incremental data corresponding to the second data change log.

[0021] Optionally, determining whether the target storage cluster contains incremental data corresponding to the first data change log specifically includes:

[0022] Among the filenames of the first data to be migrated that the migration tool traverses, determine the filename to be checked that corresponds to the first data change log;

[0023] The file to be checked corresponding to the file name to be checked is retrieved in the target storage cluster. Based on the first data change log, it is determined whether each of the files to be checked has performed the operation corresponding to the first data change log.

[0024] If all the files to be checked have performed the operation corresponding to the first data change log, then it is confirmed that the target storage cluster contains all the incremental data corresponding to the first data change log;

[0025] If not all of the files to be checked have performed the operations corresponding to the first data change log, then it is confirmed that the target storage cluster does not contain all the incremental data corresponding to the first data change log.

[0026] Optional, also includes:

[0027] When the migration tool is invoked to perform the full data migration task of migrating the first data to be migrated from the source storage cluster to the target storage cluster, an error log is generated for the first data to be migrated that cannot be queried.

[0028] The step of determining the incremental data of the source storage cluster relative to the target storage cluster based on the data change log specifically involves:

[0029] The incremental data of the source storage cluster compared to the target storage cluster is determined based on the error log and the data change log.

[0030] Optionally, recording the data change log of the source storage cluster compared to the target storage cluster specifically includes:

[0031] Record the first data change log of the source storage cluster, and use the first data change log as the data change log of the source storage cluster compared with the target storage cluster.

[0032] Optionally, recording the data change log of the source storage cluster relative to the target storage cluster specifically includes:

[0033] Record the second data change log of the source storage cluster, and simultaneously record the third data change log of the target storage cluster;

[0034] The data change logs of the source storage cluster relative to the target storage cluster are determined based on the second data change log and the third data change log.

[0035] Optional, also includes:

[0036] When performing the incremental data migration task, the incremental data change log of the source storage cluster compared to the target storage cluster is recorded;

[0037] After the incremental data migration task is completed, the secondary incremental data of the source storage cluster compared with the target storage cluster is determined according to the incremental data change log;

[0038] Perform a secondary incremental data migration task to migrate the secondary incremental data from the source storage cluster to the target storage cluster.

[0039] Optionally, both the source storage cluster and the target storage cluster are specifically object storage clusters.

[0040] To address the aforementioned technical problems, this application also provides a data migration apparatus, comprising:

[0041] The first recording unit is used to record the data change log of the source storage cluster relative to the target storage cluster when performing a full data migration task from the source storage cluster to the target storage cluster;

[0042] The first determining unit is used to determine the incremental data of the source storage cluster relative to the target storage cluster based on the data change log after the full data migration task is completed.

[0043] The first incremental execution unit is used to execute an incremental data migration task that migrates the incremental data from the source storage cluster to the target storage cluster.

[0044] To address the aforementioned technical problems, this application also provides a data migration device, comprising:

[0045] Memory, used to store computer programs;

[0046] A processor for executing the computer program, which, when executed by the processor, implements the steps of the data migration method as described in any of the preceding descriptions.

[0047] To address the aforementioned technical problems, this application also provides a storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the data migration method as described in any of the preceding claims.

[0048] The data migration method provided in this application records data change logs between the source and target storage clusters during the full data migration task from the source to the target storage cluster. This allows the method to determine incremental data between the source and target storage clusters based on the data change logs after the full data migration task. Then, an incremental data migration task is performed to transfer this incremental data from the source to the target storage cluster. By determining incremental data based on the data change logs, this method significantly improves the efficiency of incremental data migration tasks compared to existing incremental migration schemes that compare all data in both the source and target storage clusters in memory. This approach improves the overall efficiency of data migration between storage clusters.

[0049] This application also provides a data migration apparatus, data migration device, and storage medium, which have the aforementioned beneficial effects, and will not be elaborated further here. Attached Figure Description

[0050] To more clearly illustrate the technical solutions of the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 A flowchart illustrating a data migration method provided in this application embodiment;

[0052] Figure 2 This is a schematic diagram of the structure of a data migration device provided in an embodiment of this application;

[0053] Figure 3 This is a schematic diagram of the structure of a data migration device provided in an embodiment of this application. Detailed Implementation

[0054] The core of this application is to provide a data migration method, data migration apparatus, data migration device, and storage medium to improve the efficiency of migration tools in performing incremental migrations between storage clusters.

[0055] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0056] Example 1

[0057] Figure 1 This is a flowchart of a data migration method provided in an embodiment of this application.

[0058] like Figure 1 As shown, the data migration method provided in this application includes:

[0059] S101: When performing a full data migration task from the source storage cluster to the target storage cluster, record the data change log of the source storage cluster compared to the target storage cluster.

[0060] S102: After performing the full data migration task, determine the incremental data of the source storage cluster relative to the target storage cluster based on the data change log.

[0061] S103: Execute an incremental data migration task that migrates incremental data from the source storage cluster to the target storage cluster.

[0062] In specific implementations, the source storage cluster and target storage cluster in the embodiments of this application can be any type of storage cluster. In one implementation, both the source storage cluster and the target storage cluster are specifically object storage clusters, in which case the source storage cluster is the source bucket, the target storage cluster is the target bucket, and the data to be migrated is an object.

[0063] For S101, pre-deployed migration tools can be invoked to perform a full data migration task from the source storage cluster to the target storage cluster. Specifically, the full data migration task from the source storage cluster to the target storage cluster performed by S101 may include:

[0064] The migration tool is invoked to traverse the entire data of the source storage cluster, and this data is designated as the first data to be migrated.

[0065] The migration tool is invoked to perform a full data migration task that migrates the first data to be migrated from the source storage cluster to the target storage cluster.

[0066] Existing migration strategies for object storage tools involve sending requests to the source bucket to iterate through its contents before each migration. The number of requests sent to the source bucket corresponds to the number of objects already present, continuing until all data in the source bucket has been migrated. The migration tool records these objects in an internal array to initiate a full data migration from the source bucket to the target bucket. After all objects in the source bucket have been traversed, the same process is repeated for the target bucket. Once both buckets have been fully traversed, the objects in both buckets are compared, and the differing objects are then incrementally migrated. This incremental migration method is extremely inefficient; it's possible for the source bucket to only have one new object added, yet the incremental migration can take a week or even longer, which is unacceptable.

[0067] Therefore, the data migration method provided in this application optimizes the existing migration tools. In addition to using the migration tool to traverse the data of the source storage cluster to determine the first data to be migrated, it adds a data operation logging mechanism. Specifically, it only needs to record the data change logs during the execution of the full data migration task and record the generated data change logs in a specified file.

[0068] The data operations involved can include adding, deleting, modifying, and querying data. Query operations typically do not cause data changes, so the corresponding data change logs for query operations can be omitted.

[0069] To quickly and easily determine the incremental data between the source storage cluster and the target storage cluster, a data operation logging mechanism is added. Specifically, when recording data change logs between the source and target storage clusters, the first data change log of the source storage cluster is recorded, and this first data change log is used as the data change log between the source and target storage clusters. In other words, only the data change logs of the source storage cluster that have undergone data changes can be recorded to determine the incremental data between the source and target storage clusters.

[0070] Alternatively, to more accurately determine the incremental data between the source and target storage clusters, an additional data operation logging mechanism can be implemented. Specifically, recording data change logs between the source and target storage clusters can include: recording a second data change log for the source cluster and a third data change log for the target cluster; and determining the data change logs between the source and target storage clusters based on the second and third data change logs. That is, during a full data migration task, both the second and third data change logs of the source and target storage clusters can be recorded simultaneously. The third data change log of the target storage cluster can determine if the full data migration task was successfully executed. Combined with the first data to be migrated identified by the migration tool, data that was not successfully migrated can be determined. By combining the second data change log and the data that was not successfully migrated, the incremental data between the source and target storage clusters can be defined, allowing for a more accurate determination of the incremental data. In practical applications, if there is data that was not successfully migrated to the target storage cluster during a full data migration task, but the second data change log indicates that this data was deleted from the source storage cluster, the corresponding data change log can be omitted.

[0071] For S102, after executing the full data migration task, the incremental data of the source storage cluster relative to the target storage cluster is determined based on the data change log of the source storage cluster relative to the target storage cluster. For details, please refer to the above steps and perform the corresponding operation to determine the incremental data in conjunction with the data change log generation method.

[0072] For S103, once the incremental data is determined, the migration tool can be invoked to perform the incremental data migration task.

[0073] If both the source and target storage clusters are object storage clusters, then the incremental data refers to the objects whose data has changed, and the incremental data migration task is a migration task for the objects whose data has changed.

[0074] If the source and target storage clusters are other types of storage clusters, the migration unit of the incremental data migration task can be determined according to the specific storage structure. For example, the migration can be carried out in units of the smallest file, or in units of pages, tables, etc., and the corresponding pages and tables in the target storage cluster can be replaced.

[0075] The data migration method provided in this application records data change logs between the source and target storage clusters when performing a full data migration task from the source storage cluster to the target storage cluster. This allows the method to determine incremental data between the source and target storage clusters based on the data change logs after the full data migration task. Then, an incremental data migration task is performed to migrate the incremental data from the source to the target storage cluster. By determining incremental data based on the data change logs, compared to existing incremental migration schemes where migration tools read all data from both the source and target storage clusters into memory for comparison, this method can determine incremental data more quickly, significantly improving the efficiency of incremental data migration tasks and consequently improving the overall efficiency of data migration between storage clusters.

[0076] Example 2

[0077] Based on the above embodiments, the starting point for recording the data change log of the source storage cluster compared to the target storage cluster can be either the time when the migration tool begins to traverse the full data of the source storage cluster, or the time when the full data migration task begins. The migration tool typically only needs a short time to traverse the full data of the source storage cluster to determine the first data to be migrated, but it cannot be ruled out that data changes may occur in the source storage cluster during this time. Therefore, recording the data change log of the source storage cluster compared to the target storage cluster in S101 can specifically be: starting from the first time point when the migration tool begins to traverse the full data of the source storage cluster, recording the data change log of the source storage cluster compared to the target storage cluster.

[0078] Since it is impossible to determine whether the data changes in the source storage cluster will be traversed by the migration tool during the process of traversing the full data of the source storage cluster, it is necessary to redefine the incremental data corresponding to this part of the data change log during the process of determining the incremental data of the source storage cluster relative to the target storage cluster based on the data change log in S102, or during the process of executing the incremental data migration task of migrating the incremental data from the source storage cluster to the target storage cluster in S103.

[0079] S103 then executes an incremental data migration task, which involves migrating incremental data from the source storage cluster to the target storage cluster. Specifically, this may include:

[0080] For the first data change log between the first time point when the self-migration tool starts traversing the full data of the source storage cluster and the second time point when the migration tool starts executing the full data migration task for the first data to be migrated, determine whether the target storage cluster contains incremental data corresponding to the first data change log.

[0081] If the target storage cluster contains all incremental data corresponding to the first data change log, then skip the first data change log;

[0082] If the target storage cluster does not contain all the incremental data corresponding to the first data change log, then execute the first incremental data migration task for the incremental data corresponding to the first data change log.

[0083] After determining the incremental data corresponding to the second data change log after the second time point, directly execute the second incremental data migration task for the incremental data corresponding to the second data change log.

[0084] Since incremental data may be generated during the process of traversing and confirming the full data, and it is uncertain whether these incremental data have been traversed, incremental data migration tasks are performed on the incremental data during and after the traversal. Incremental migration can be performed on the incremental data generated after the traversal, while for the incremental data generated during the traversal, the data that has been migrated by the full data migration task is filtered out before incremental migration is performed, thereby reducing unnecessary incremental migration operations.

[0085] In practical applications, the changes that occur when the migration tool traverses the first data to be migrated during a full data migration task include adding, deleting, modifying, and querying. Excluding query operations, it's possible that, according to the storage structure of the source storage cluster and the traversal order of the migration tool, a new file has been added to a file already traversed by the migration tool. In this case, the newly added file will not be recorded in the specified file of the migration tool. It's also possible that data in the first data to be migrated that has already been traversed and recorded in the specified file by the migration tool has been deleted. In this case, the migration tool will generate an error when performing a full data migration task because it cannot find the data in the source storage cluster. Therefore, the data migration method provided in this application embodiment may further include: when calling the migration tool to perform a full data migration task to migrate the first data to be migrated from the source storage cluster to the target storage cluster, generating an error log for the first data to be migrated that cannot be queried; correspondingly, in S102, determining the incremental data of the source storage cluster relative to the target storage cluster based on the data change log can specifically be: determining the incremental data of the source storage cluster relative to the target storage cluster based on the error log and the data change log. When performing a full data migration task, the migration tool records the error log of the first data to be migrated that has been deleted. This log is then compared with the data change log of the source storage cluster compared with the target storage cluster to identify the data that has been deleted from the first data to be migrated and eliminate the error from the migration tool.

[0086] Specifically, determining whether the target storage cluster contains incremental data corresponding to the first data change log may include:

[0087] In the filenames of the first data to be migrated that the migration tool traverses, determine the filenames to be checked that correspond to the first data change logs;

[0088] The files to be checked are retrieved from the target storage cluster according to the filenames to be checked. Based on the first data change log, it is determined whether the operations corresponding to the first data change log have been performed on each file to be checked.

[0089] If all files to be checked have performed the operations corresponding to the first data change log, then it is confirmed that the target storage cluster contains all incremental data corresponding to the first data change log;

[0090] If not all files to be checked have performed the operations corresponding to the first data change log, then it is confirmed that the target storage cluster does not contain all the incremental data corresponding to the first data change log.

[0091] Based on the filename of the first data to be migrated encountered by the migration tool, the incremental data involved in the first data change log generated during the traversal process is checked. If the target storage cluster contains all the incremental data involved in the first data change log, then incremental migration is not required for the incremental data corresponding to the first data change log. If the target storage cluster does not contain all the incremental data involved in the first data change log, i.e., data not migrated in the full data migration task has been generated, then incremental migration is performed only on the unmigrated data.

[0092] Example 3

[0093] Because the source storage cluster may generate new incremental data during the incremental data migration process, inconsistencies may still exist between the source and target storage clusters after the first incremental data migration task is completed. Therefore, based on the above embodiments, the data migration method provided in this application may further include:

[0094] When performing incremental data migration tasks, log the incremental data changes of the source storage cluster compared to the target storage cluster.

[0095] After the incremental data migration task is completed, the secondary incremental data of the source storage cluster compared with the target storage cluster is determined based on the incremental data change log;

[0096] Perform a secondary incremental data migration task to migrate the secondary incremental data from the source storage cluster to the target storage cluster.

[0097] In practice, during the full data migration from the source storage cluster to the target storage cluster, data change logs are recorded. This is used to execute the first incremental data migration task after the full migration. During the first incremental data migration, incremental data change logs are recorded in the same way as those recording data changes between the source and target storage clusters. This is the second data change log. After the first incremental data migration, a second incremental data migration task corresponding to the second data change log is executed. Similarly, during the second incremental data migration, a third incremental data migration task is recorded in the same way as those recording data changes between the source and target storage clusters. This is the third data change log. And so on.

[0098] Data migration tasks between storage clusters are sometimes for data synchronization between primary and standby storage clusters, and sometimes for storage cluster failover. When performing data migration between storage clusters for data synchronization, after executing a full data migration task and an incremental data migration task, and confirming data consistency between the source and target storage clusters, an incremental data migration task from the source storage cluster to the target storage cluster is executed every time incremental data is generated in the source storage cluster. When performing data migration between storage clusters for storage cluster failover, a full data migration task and an incremental data migration task are executed, and even a second, third, or even a third incremental data migration task is executed, until the incremental data volume drops below a preset threshold. At this point, the service of the source storage cluster is paused, the remaining incremental data is migrated, and then the service of the source storage cluster is restarted on the target storage cluster.

[0099] The various embodiments of the data migration method have been described in detail above. Based on this, this application also discloses a data migration apparatus, a data migration device, and a storage medium corresponding to the above method.

[0100] Example 4

[0101] Figure 2 This is a schematic diagram of a data migration device provided in an embodiment of this application.

[0102] like Figure 2 As shown, the data migration apparatus provided in this application embodiment includes:

[0103] The first recording unit 201 is used to record the data change log of the source storage cluster relative to the target storage cluster when performing a full data migration task from the source storage cluster to the target storage cluster.

[0104] The first determining unit 202 is used to determine the incremental data of the source storage cluster relative to the target storage cluster based on the data change log after the full data migration task is performed.

[0105] The first incremental execution unit 203 is used to execute the incremental data migration task that migrates incremental data from the source storage cluster to the target storage cluster.

[0106] Furthermore, the first recording unit 201 performs a full data migration task from the source storage cluster to the target storage cluster, specifically including:

[0107] The migration tool is invoked to traverse the entire data of the source storage cluster, and this data is designated as the first data to be migrated.

[0108] The migration tool is invoked to perform a full data migration task that migrates the first data to be migrated from the source storage cluster to the target storage cluster.

[0109] Furthermore, the first recording unit 201 records the data change logs of the source storage cluster compared to the target storage cluster, specifically as follows:

[0110] Starting from the first point in time when the migration tool begins to traverse the full data of the source storage cluster, a data change log is recorded between the source storage cluster and the target storage cluster.

[0111] Furthermore, the first incremental execution unit 203 specifically includes:

[0112] The first incremental execution subunit is used to check the first data change log between the first time point and the second time point when the migration tool starts executing the full data migration task for the first data to be migrated, and determine whether the target storage cluster contains incremental data corresponding to the first data change log. If the target storage cluster contains all incremental data corresponding to the first data change log, the first data change log is skipped. If the target storage cluster does not contain all incremental data corresponding to the first data change log, the first incremental data migration task for the incremental data corresponding to the first data change log is executed.

[0113] The second incremental execution subunit is used to determine the incremental data corresponding to the second data change log after the second time point, and then directly execute the second incremental data migration task for the incremental data corresponding to the second data change log.

[0114] Furthermore, the first incremental execution subunit determines whether the target storage cluster contains incremental data corresponding to the first data change log, specifically including:

[0115] In the filenames of the first data to be migrated that the migration tool traverses, determine the filenames to be checked that correspond to the first data change logs;

[0116] The files to be checked are retrieved from the target storage cluster according to the filenames to be checked. Based on the first data change log, it is determined whether the operations corresponding to the first data change log have been performed on each file to be checked.

[0117] If all files to be checked have performed the operations corresponding to the first data change log, then it is confirmed that the target storage cluster contains all incremental data corresponding to the first data change log;

[0118] If not all files to be checked have performed the operations corresponding to the first data change log, then it is confirmed that the target storage cluster does not contain all the incremental data corresponding to the first data change log.

[0119] Furthermore, the data migration apparatus provided in this application embodiment may further include:

[0120] The error reporting unit is used to generate error logs for the first data to be migrated that cannot be found when the migration tool is called to perform a full data migration task to migrate the first data to be migrated from the source storage cluster to the target storage cluster.

[0121] The first determining unit 202 determines the incremental data of the source storage cluster relative to the target storage cluster based on the data change log, specifically:

[0122] Determine the incremental data of the source storage cluster relative to the target storage cluster based on the error logs and data change logs.

[0123] Furthermore, the first recording unit 201 records the data change logs of the source storage cluster compared to the target storage cluster, specifically as follows:

[0124] Record the first data change log of the source storage cluster, and use the first data change log as the data change log of the source storage cluster compared to the target storage cluster.

[0125] Alternatively, the first recording unit 201 records the data change log of the source storage cluster relative to the target storage cluster, specifically including:

[0126] Record the second data change log of the source storage cluster, and simultaneously record the third data change log of the target storage cluster;

[0127] The data change logs of the source storage cluster relative to the target storage cluster are determined based on the second and third data change logs.

[0128] Furthermore, the data migration apparatus provided in this application embodiment may further include:

[0129] The second recording unit is used to record the incremental data change log of the source storage cluster compared to the target storage cluster when performing incremental data migration tasks.

[0130] The second determining unit is used to determine the secondary incremental data of the source storage cluster relative to the target storage cluster based on the incremental data change log after the incremental data migration task is completed.

[0131] The second incremental execution unit is used to perform the secondary incremental data migration task, which migrates the secondary incremental data from the source storage cluster to the target storage cluster.

[0132] Furthermore, both the source storage cluster and the target storage cluster are specifically object storage clusters.

[0133] Since the embodiments of the apparatus and the embodiments of the method correspond to each other, please refer to the description of the embodiments of the method for the embodiments of the apparatus, which will not be repeated here.

[0134] Example 5

[0135] Figure 3 This is a schematic diagram of the structure of a data migration device provided in an embodiment of this application.

[0136] like Figure 3 As shown, the data migration device provided in this application embodiment includes:

[0137] Memory 310 is used to store computer program 311;

[0138] Processor 320 is configured to execute computer program 311, which, when executed by processor 320, implements the steps of the data migration method as described in any of the above embodiments.

[0139] The processor 320 may include one or more processing cores, such as a 3-core processor or an 8-core processor. The processor 320 may be implemented using at least one hardware form selected from Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor 320 may also include a main processor and a coprocessor. The main processor, also known as a Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 320 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 320 may also include an Artificial Intelligence (AI) processor, which handles computational operations related to machine learning.

[0140] The memory 310 may include one or more storage media, which may be non-transitory. The memory 310 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory 310 is used to store at least the following computer program 311, wherein, after being loaded and executed by the processor 320, the computer program 311 is able to implement the relevant steps in the data migration method disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 310 may also include an operating system 312 and data 313, and the storage method may be temporary storage or permanent storage. The operating system 312 may be Windows. The data 313 may include, but is not limited to, the data involved in the above methods.

[0141] In some embodiments, the data migration device may further include a display screen 330, a power supply 340, a communication interface 350, an input / output interface 360, a sensor 370, and a communication bus 380.

[0142] Those skilled in the art will understand that Figure 3 The structure shown does not constitute a limitation on the data migration device and may include more or fewer components than illustrated.

[0143] The data migration device provided in this application includes a memory and a processor. When the processor executes the program stored in the memory, it can implement the data migration method described above, with the same effect.

[0144] Example 6

[0145] It should be noted that the device and equipment embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or modules may be electrical, mechanical, or other forms. Modules described as separate components may or may not be physically separate, and components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0146] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0147] If the integrated modules are implemented as software functional modules and sold or used as independent products, they can be stored in a storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods described in the various embodiments of this application.

[0148] Therefore, embodiments of this application also provide a storage medium storing a computer program, which, when executed by a processor, implements steps such as the data migration method.

[0149] The storage medium can include various media that can store program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0150] The computer program contained in the storage medium provided in this embodiment can implement the steps of the data migration method described above when executed by a processor, with the same effect.

[0151] The foregoing has provided a detailed description of a data migration method, data migration apparatus, data migration device, and storage medium provided in this application. The various embodiments in the specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus, device, and storage medium disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and relevant parts can be referred to in the method section. It should be noted that those skilled in the art can make several improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of the claims of this application.

[0152] It should also be noted that, in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A data migration method, characterized in that, include: When performing a full data migration task from the source storage cluster to the target storage cluster, starting from the first time point when the migration tool begins to traverse the full data of the source storage cluster, a data change log of the source storage cluster compared to the target storage cluster is recorded. After the full data migration task is completed, the incremental data of the source storage cluster compared to the target storage cluster is determined based on the data change log; Execute an incremental data migration task to migrate the incremental data from the source storage cluster to the target storage cluster; The incremental data migration task, which involves migrating the incremental data from the source storage cluster to the target storage cluster, specifically includes: For the first data change log between the first time point and the second time point when the migration tool starts executing the full data migration task for the first data to be migrated, determine whether the target storage cluster contains incremental data corresponding to the first data change log. If the target storage cluster contains all incremental data corresponding to the first data change log, then skip the first data change log; If the target storage cluster does not contain all incremental data corresponding to the first data change log, then execute the first incremental data migration task for the incremental data corresponding to the first data change log. After determining the incremental data corresponding to the second data change log after the second time point, directly execute the second incremental data migration task for the incremental data corresponding to the second data change log; Data migration methods also include: When the migration tool is invoked to perform the full data migration task of migrating the first data to be migrated from the source storage cluster to the target storage cluster, an error log is generated for the first data to be migrated that cannot be found. The step of determining the incremental data of the source storage cluster relative to the target storage cluster based on the data change log specifically involves: The incremental data of the source storage cluster compared to the target storage cluster is determined based on the error log and the data change log.

2. The data migration method according to claim 1, characterized in that, The full data migration task from the source storage cluster to the target storage cluster specifically includes: The migration tool is invoked to traverse the entire data of the source storage cluster, and this data is designated as the first data to be migrated. The migration tool is invoked to perform the full data migration task that migrates the first data to be migrated from the source storage cluster to the target storage cluster.

3. The data migration method according to claim 1, characterized in that, The step of determining whether the target storage cluster contains incremental data corresponding to the first data change log specifically includes: Among the filenames of the first data to be migrated that the migration tool traverses, determine the filename to be checked that corresponds to the first data change log; The file to be checked corresponding to the file name to be checked is retrieved in the target storage cluster. Based on the first data change log, it is determined whether each of the files to be checked has performed the operation corresponding to the first data change log. If all the files to be checked have performed the operation corresponding to the first data change log, then it is confirmed that the target storage cluster contains all the incremental data corresponding to the first data change log; If not all of the files to be checked have performed the operations corresponding to the first data change log, then it is confirmed that the target storage cluster does not contain all the incremental data corresponding to the first data change log.

4. The data migration method according to claim 1, characterized in that, The recording of data change logs between the source storage cluster and the target storage cluster specifically includes: Record the first data change log of the source storage cluster, and use the first data change log as the data change log of the source storage cluster compared with the target storage cluster.

5. The data migration method according to claim 1, characterized in that, The data change log recording of the source storage cluster relative to the target storage cluster specifically includes: Record the second data change log of the source storage cluster, and simultaneously record the third data change log of the target storage cluster; The data change logs of the source storage cluster relative to the target storage cluster are determined based on the second data change log and the third data change log.

6. The data migration method according to claim 1, characterized in that, Also includes: When performing the incremental data migration task, the incremental data change log of the source storage cluster compared to the target storage cluster is recorded; After the incremental data migration task is completed, the secondary incremental data of the source storage cluster compared with the target storage cluster is determined according to the incremental data change log; Perform a secondary incremental data migration task to migrate the secondary incremental data from the source storage cluster to the target storage cluster.

7. The data migration method according to claim 1, characterized in that, Both the source storage cluster and the target storage cluster are specifically object storage clusters.

8. A data migration device, characterized in that, include: The first recording unit is used to record the data change log of the source storage cluster relative to the target storage cluster, starting from the first time point when the migration tool begins to traverse the full data of the source storage cluster during the execution of the full data migration task from the source storage cluster to the target storage cluster. The first determining unit is used to determine the incremental data of the source storage cluster relative to the target storage cluster based on the data change log after the full data migration task is completed. The first incremental execution unit is used to execute the incremental data migration task that migrates the incremental data from the source storage cluster to the target storage cluster; The incremental data migration task, which involves migrating the incremental data from the source storage cluster to the target storage cluster, specifically includes: For the first data change log between the first time point and the second time point when the migration tool starts executing the full data migration task for the first data to be migrated, determine whether the target storage cluster contains incremental data corresponding to the first data change log. If the target storage cluster contains all incremental data corresponding to the first data change log, then skip the first data change log; If the target storage cluster does not contain all incremental data corresponding to the first data change log, then execute the first incremental data migration task for the incremental data corresponding to the first data change log. After determining the incremental data corresponding to the second data change log after the second time point, directly execute the second incremental data migration task for the incremental data corresponding to the second data change log; The data migration device also includes: The error reporting unit is used to generate an error log for the first data to be migrated that cannot be found when the migration tool is called to perform the full data migration task of migrating the first data to be migrated from the source storage cluster to the target storage cluster. The step of determining the incremental data of the source storage cluster relative to the target storage cluster based on the data change log specifically involves: The incremental data of the source storage cluster compared to the target storage cluster is determined based on the error log and the data change log.

9. A data migration device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program, which, when executed by the processor, implements the steps of the data migration method as described in any one of claims 1 to 7.

10. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the data migration method as described in any one of claims 1 to 7.

Images