Data migration method, device and equipment

Abstract

The present application discloses a data migration method, including: determining historical data in a source data storage system according to a set time; migrating the historical data to a target data storage system; obtaining new data in the source data storage system after the set time, wherein the new data is obtained by parsing data increment notifications in a message queue; and migrating the new data to the target data storage system. The above method can realize rapid migration of new data from the source data storage system to the target data storage system, significantly improving the migration speed of new data.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data migration method, device, electronic device and storage device. Background Art

[0002] With the penetration of cloud computing in various industries, cloud migration has become a general trend, especially the migration of source data storage systems to target data storage systems (cloud storage). Specifically, cloud migration requires the migration of historical data and new data from the source data storage system to the target data storage system. The amount of data migration during cloud migration has transitioned from TB level to PB level, and the number of migrated files has reached tens of billions. Seamless migration of object storage applications to the cloud, with minimal impact on applications, has become a core requirement for cloud migration.

[0003] In the prior art, when migrating new data from a source data storage system to a target data storage system, the existing data is generally scanned to discover the new data. When the number of scanned files (e.g., tens of billions) is too large, the scanning time will be too long.

[0004] Therefore, when migrating newly added data from a source data storage system to a target data storage system, improving the migration speed of the newly added data is an issue that needs to be addressed urgently. Summary of the invention

[0005] The present application provides a data migration method, apparatus, electronic device and storage device, which realize rapid migration of new data from a source data storage system to a target data storage system, and significantly improve the migration speed of the new data.

[0006] This application provides a data migration method, including:

[0007] Determine historical data in the source data storage system according to a set time;

[0008] Migrating the historical data to a target data storage system;

[0009] Obtaining newly added data in the source data storage system after the set time, wherein the newly added data is obtained by parsing the data increment notification in the message queue;

[0010] Migrate the newly added data to the target data storage system.

[0011] Optionally, migrating the historical data to a target data storage system includes:

[0012] Generate a historical data migration task according to the file size of the historical data and the number of files of the historical data;

[0013] Execute the historical data migration task to migrate the historical data to the target data storage system.

[0014] Optionally, the obtaining of new data in the source data storage system after the set time, wherein the new data is obtained by parsing a data increment notification in a message queue, includes:

[0015] Configuring data increment notification in the source data storage system;

[0016] Get the message arrival notification of the message queue;

[0017] Obtaining a data increment notification in the message queue according to the message arrival notification;

[0018] The data increment notification in the message queue is parsed to obtain the newly added data in the source data storage system after the set time.

[0019] Optionally, the message queue includes at least one of the following: an unbuffered message queue; a buffered message queue.

[0020] Optionally, also include:

[0021] When the historical data migration is completed, the unmigrated new data will be migrated to the target data storage system by mirroring back to the source.

[0022] Optionally, the step of migrating the unmigrated newly added data to the target storage system by mirroring back to the source includes:

[0023] When the specified newly added data cannot be found in the target data storage system, the return source address corresponding to the specified newly added data is obtained according to the specified mirror return source configuration basis;

[0024] According to the return-to-source address, the unmigrated newly added data is migrated to the target storage system.

[0025] Optionally, migrating the unmigrated newly added data to the target storage system according to the return-to-source address includes:

[0026] According to the return-to-source address, the unmigrated new data is migrated to the target storage system by using a dedicated line.

[0027] Optionally, migrating the unmigrated new data to the target storage system by using a dedicated line according to the return-to-source address includes:

[0028] Get a dedicated customer line;

[0029] Authenticating the customer dedicated line to determine whether the customer dedicated line has access rights;

[0030] If yes, the client dedicated line is used to access the source address to migrate the unmigrated new data to the target storage system.

[0031] Optionally, also include:

[0032] Performing data verification on the data in the target data storage system and the data in the source data storage system to determine whether the data are consistent;

[0033] If they are consistent, stop mirroring back to the source.

[0034] The present application also provides a data migration device, comprising:

[0035] A historical data determination unit, used to determine the historical data in the source data storage system according to a set time;

[0036] A historical data migration unit, used to migrate the historical data to a target data storage system;

[0037] A newly added data obtaining unit, configured to obtain newly added data in the source data storage system after the set time, wherein the newly added data is obtained by parsing the data increment notification in the message queue;

[0038] The newly added data migration unit is used to migrate the newly added data to the target data storage system.

[0039] The present application also provides an electronic device, comprising:

[0040] Processor; and

[0041] The memory is used to store a program of the data migration method. After the device is powered on and the program of the data migration method is run by the processor, the following steps are performed:

[0042] Determine historical data in the source data storage system according to a set time;

[0043] Migrating the historical data to a target data storage system;

[0044] Obtaining newly added data in the source data storage system after the set time, wherein the newly added data is obtained by parsing the data increment notification in the message queue;

[0045] Migrate the newly added data to the target data storage system.

[0046] The present application also provides a storage device storing a program of a data migration method, wherein the program is executed by a processor to perform the following steps:

[0047] Determine historical data in the source data storage system according to a set time;

[0048] Migrating the historical data to a target data storage system;

[0049] Obtaining newly added data in the source data storage system after the set time, wherein the newly added data is obtained by parsing the data increment notification in the message queue;

[0050] Migrating the newly added data to the target data storage system, including:

[0051] Compared with the prior art, this application has the following advantages:

[0052] The present application provides a data migration method, including: determining historical data in a source data storage system according to a set time; migrating the historical data to a target data storage system; obtaining new data in the source data storage system after the set time, wherein the new data is obtained by parsing data increment notifications in a message queue; and migrating the new data to the target data storage system. The data migration method provided by the present application can quickly migrate new data from a source data storage system to a target data storage system based on the implementation of a message queue, significantly improving the migration speed of new data. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is a flow chart of a data migration method provided in the first embodiment of the present application.

[0054] Figure 2 This is a schematic diagram of migrating historical data from a source data storage system to a target data storage system provided by the first embodiment of the present application.

[0055] Figure 3 It is a schematic diagram of an incremental migration system provided in the first embodiment of the present application.

[0056] Figure 4 This is a schematic diagram of the principle of mirroring back to the source.

[0057] Figure 5 It is a schematic diagram of a mirroring back-to-source system provided in the first embodiment of the present application.

[0058] Figure 6 This is a flowchart of another data migration method provided in the first embodiment of the present application.

[0059] Figure 7 It is a schematic diagram of a data migration device provided in the second embodiment of the present application. DETAILED DESCRIPTION

[0060] Many specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present invention, so the present invention is not limited to the specific implementation disclosed below.

[0061] The first embodiment of the present application provides a data migration method. The implementation subject of the first embodiment of the present application may be a migration system, which is used to migrate data from a source data storage system to a target storage system. Figure 1 Provide detailed explanation.

[0062] like Figure 1 As shown, in step S101, historical data in the source data storage system is determined according to a set time.

[0063] The source data storage system includes NAS, FTP, HDFS, OpenStack, cloud storage, etc.

[0064] The historical data refers to a set of data before a set time.

[0065] For example, customer A is an Internet service company, whose main application is set up at a cloud computing service provider B, providing its users with online editing services such as pictures and videos. Customer A has about 100 million historical data files stored at B, totaling about 320TB in size, with about 20GB of data added every day.

[0066] Due to the company's development needs, we are considering switching the application to a cloud computing service provider C. When switching, the original data and newly added data need to be migrated to cloud computing service provider C.

[0067] For example, if it is planned to seamlessly migrate the applications of Internet company A on cloud computing service B to cloud computing service C on June 12, 2020, then the storage system of Internet company A on cloud computing service B is the source data storage system, and the storage system of Internet company A on cloud computing service C is the target data storage system. June 5, 2020 can be determined as the set time, and the data stored by Internet company A on cloud computing service B before this time can be determined as historical data.

[0068] like Figure 1 As shown, in step S102, the historical data is migrated to a target data storage system.

[0069] The target data storage system is an object-based data storage system, such as object storage OSS, AWS's S3.

[0070] Migrating the historical data to the target data storage system includes:

[0071] Generate a historical data migration task according to the file size of the historical data and the number of files of the historical data;

[0072] Execute the historical data migration task to migrate the historical data to the target data storage system.

[0073] In specific implementation, the historical data migration task is executed to migrate the historical data to the target data storage system. The master-slave operation framework can be used to execute the historical data migration task to migrate the historical data in the source data storage system to the target data storage system.

[0074] Please refer to Figure 2 . Figure 2 It is a schematic diagram of migrating historical data from a source data storage system to a target data storage system. Figure 2 The following modules are included:

[0075] Job Scheduling (JobDispatch): Generates migration tasks based on the source storage data distribution, such as file size and quantity.

[0076] ImportCluser: executes migration tasks, handles migration errors, and generates migration reports. It is based on the Master-Worker framework, where the Master is responsible for scheduling migration tasks and the Worker is responsible for executing migration tasks.

[0077] Source data storage system (Source): It is the source storage, which can be NAS, FTP, HDFS, OpenStack, cloud storage, etc.

[0078] Destnation: Destination object storage.

[0079] Figure 2 In the process, first, in JobDispatch, migration tasks are generated based on the distribution of historical data, such as file size and quantity. Then, in ImportCluser: migration tasks are executed, migration errors are handled, and migration reports are generated. Based on the Master-Worker framework, the Master is responsible for scheduling migration tasks, and the Worker is responsible for executing migration tasks.

[0080] like Figure 1 As shown, in step S103, newly added data in the source data storage system after the set time is obtained, wherein the newly added data is obtained by parsing the data increment notification in the message queue.

[0081] Please refer to Figure 3, which is a schematic diagram of an incremental migration system. Figure 3 The module description is as follows:

[0082] Message Queue Service (QueueService): It can be an unbuffered message queue, such as AWS SQS.

[0083] Notification Service: It can be a buffered message queue, such as AWS SNS, AliCloud MNS.

[0084] Message Driver (MsgDriver): Message format adaptation, converting messages from various message queues into a unified format.

[0085] Job Scheduling (JobDispatch): Generate migration tasks in groups based on the size, number, and last modification time of incremental files.

[0086] ImportCluser: executes migration tasks, handles migration errors, and generates migration reports.

[0087] Source data storage system (Source): It is the source storage, which can be NAS, FTP, HDFS, OpenStack, cloud storage, etc.

[0088] Destination data storage system (Destnation): It is the target object storage.

[0089] The obtaining of the newly added data in the source data storage system after the set time, wherein the newly added data is obtained by parsing the data increment notification in the message queue, comprises:

[0090] Configuring data increment notification in the source data storage system;

[0091] Get the message arrival notification of the message queue;

[0092] Obtaining a data increment notification in the message queue according to the message arrival notification;

[0093] The data increment notification in the message queue is parsed to obtain the newly added data in the source data storage system after the set time.

[0094] like Figure 3As shown, the incremental migration system configures message notifications in the source data storage system. After the data in the source data storage system is updated, the incremental migration system is notified in the form of messages. The incremental migration system receives notifications of data increments from the source data storage system by maintaining one or more message queues. When the incremental migration system receives notifications of data increments, it first gets a message arrival notification from the message queue. According to the message arrival notification, it searches the message queue to obtain the data increment notification. Then, the data increment notification in the message queue is parsed to obtain the newly added data in the source data storage system after the set time.

[0095] Based on the implementation of the message queue, incremental data can be quickly migrated, and its speed is unattainable by the incremental migration of the scanning method in the prior art. The message queue includes an unbuffered message queue and a buffered message queue. Since the unbuffered message queue and the buffered message queue are prior art, they will not be described here.

[0096] like Figure 1 As shown, in step S104, the newly added data is migrated to the target data storage system.

[0097] like Figure 3 As shown, the incremental migration system receives data increment notifications from the source data storage system through an unbuffered message queue and / or a buffered message queue. Then, after adapting the message format, the messages of various message queues are converted into a unified format. Next, the incremental migration system parses the messages in the unified format to obtain incremental file information such as the size, number, and last modification time of the incremental files. Then, based on the above incremental file information, migration tasks are generated in groups. Finally, the above migration tasks are executed, migration errors in the migration process are handled, and a migration report is generated.

[0098] As an implementation mode, the first embodiment of the present application may further include:

[0099] When the historical data migration is completed, the unmigrated new data will be migrated to the target data storage system by mirroring back to the source.

[0100] The step of migrating the unmigrated newly added data to the target storage system by mirroring back to the source includes:

[0101] When the specified newly added data cannot be found in the target data storage system, the return source address corresponding to the specified newly added data is obtained according to the specified mirror return source configuration basis;

[0102] According to the return-to-source address, the unmigrated newly added data is migrated to the target storage system.

[0103] Image back-to-source is mainly used to seamlessly migrate data to OSS. That is, if the service is already running on the source station you have established or on other cloud products and needs to be migrated to OSS, but the service cannot be stopped, the image back-to-source function can be used to achieve this. OSS (Object Storage Service) is a massive, secure and highly reliable cloud storage service provided by cloud service providers. Figure 4 shown.

[0104] Specific scenarios for mirroring back to the source include the following:

[0105] (1) The source station has a batch of cold data and is constantly generating new hot data.

[0106] You can first migrate cold data to OSS through the migration tool, configure mirror back to source, and configure the address of the source site to OSS. After switching the domain name to OSS (or the cloud service provider's CDN, back to source to OSS), even if some of the newly generated data has not been migrated, it can still be accessed normally on OSS, and the file will be stored in OSS after one access. After the domain name is switched, the source site has no new data generated. At this time, scan it again, import the data that has not been imported to OSS at one time, and then delete the mirror back to source configuration.

[0107] If the configured source site is an IP address, you can continue to mirror to the source site after migrating the domain name to OSS. However, if you configure a domain name, since the domain name itself will resolve to OSS or CDN, the mirror will lose its function. In this case, you can apply for another domain name as the source site of the mirror. This domain name resolves to the same IP address as the domain name being served, so that you can continue to mirror to the source site when the service domain name is migrated.

[0108] (2) Only part of the traffic from the source station is switched to OSS or CDN. The source station itself is still generating data.

[0109] The migration method is similar to the above method, except that after the traffic is switched to OSS, do not delete the mirror back to source configuration. This ensures that the traffic switched to OSS or CDN can still retrieve the data from the source station.

[0110] The step of migrating the unmigrated newly added data to the target storage system by mirroring back to the source includes:

[0111] When the specified newly added data cannot be found in the target data storage system, the return source address corresponding to the specified newly added data is obtained according to the specified mirror return source configuration basis;

[0112] According to the return-to-source address, the unmigrated newly added data is migrated to the target storage system.

[0113] The step of migrating the unmigrated newly added data to the target storage system according to the return-to-source address includes:

[0114] According to the return-to-source address, the unmigrated new data is migrated to the target storage system using the public network or dedicated line.

[0115] The step of migrating the unmigrated new data to the target storage system by using a dedicated line according to the return-to-source address includes:

[0116] Get a dedicated customer line;

[0117] Authenticating the customer dedicated line to determine whether the customer dedicated line has access rights;

[0118] If yes, the client dedicated line is used to access the source address to migrate the unmigrated new data to the target storage system.

[0119] Please refer to Figure 5 The mirrored source system is shown.

[0120] Figure 5 The module description is as follows:

[0121] System access control: security configuration of the operating environment of the image back-to-source service system, such as VPC (Virtual Private Cloud, customer dedicated line) configuration, login authentication, data encryption, etc.

[0122] Storage authentication: Authentication of access rights to the source data storage system.

[0123] File mapping: The file correspondence between the target data storage system and the source data storage system.

[0124] File access service: file access and management of source data storage system, such as range read (Get withRange), file prefetch, etc.

[0125] As an implementation mode, the first embodiment of the present application may further include:

[0126] Performing data verification on the data in the target data storage system and the data in the source data storage system to determine whether the data are consistent;

[0127] If they are consistent, stop mirroring back to the source.

[0128] Figure 6 Schematic diagram of a migration system according to the method provided in the first embodiment of the present application. Figure 6 The workflow of the migration system is described below.

[0129] Step S601: Start the migration of new data and migrate the new data. The migration of new data is based on the message queue. The implementation principle is as follows: Figure 3 As shown in the figure. The source storage configures message notifications. When the source data is updated, the incremental migration system is notified. Based on the implementation of the message queue, incremental data can be quickly migrated, which is unattainable by incremental migration using scanning methods. Figure 3 The following modules are included.

[0130] Message Queue Service (QueueService): It can be an unbuffered message queue, such as AWS SQS.

[0131] Notification Service: It can be a buffered message queue, such as AWS SNS, AliCloud MNS.

[0132] Message Driver (MsgDriver): Message format adaptation, converting messages from various message queues into a unified format.

[0133] Job Scheduling (JobDispatch): Generate migration tasks in groups based on the size, number, and last modification time of incremental files.

[0134] ImportCluser: executes migration tasks, handles migration errors, and generates migration reports.

[0135] Source data storage system (Source): It is the source storage, which can be NAS, FTP, HDFS, OpenStack, cloud storage, etc.

[0136] Destination data storage system (Destnation): It is the target object storage.

[0137] Step S602: Start historical data migration and migrate historical data. Figure 2 , Figure 2 The following modules are included:

[0138] Job Scheduling (JobDispatch): Generates migration tasks based on the source storage data distribution, such as file size and quantity.

[0139] ImportCluser: executes migration tasks, handles migration errors, and generates migration reports. It is based on the Master-Worker framework, where the Master is responsible for scheduling migration tasks and the Worker is responsible for executing migration tasks.

[0140] Source data storage system (Source): It is the source storage, which can be NAS, FTP, HDFS, OpenStack, cloud storage, etc.

[0141] Destnation: Destination object storage.

[0142] Step S603: The source historical data migration is completed.

[0143] Step S604: Configure mirroring back to the source in the target storage space (bucket). Applications are constantly updating storage, and the RPO and RTO of incremental migration cannot be 0. When the application is switched, incremental migration may still be in progress. There is incremental data that has not yet been migrated. This part of data can be accessed through the storage mirroring function to achieve the purpose of seamless application migration. The implementation principle of the mirroring back to the source service is detailed in Figure 5 , Figure 5 The following modules are included:

[0144] System access control: security configuration of the operating environment of the image back-to-source service system, such as VPC (Virtual Private Cloud, customer dedicated line) configuration, login authentication, data encryption, etc.

[0145] Storage authentication: Authentication of access rights to the source data storage system.

[0146] File mapping: The file correspondence between the target data storage system and the source data storage system.

[0147] File access service: file access and management of source data storage system, such as range read (Get withRange), file prefetch, etc.

[0148] Step S605: Switch the application to the target storage system. The application is switched from the source storage system to the target storage system, and the application runs based on the target storage system.

[0149] Step S606: Stop source incremental migration. After the application is switched to the target storage system, the source storage no longer has incremental data. The message processing of the source message queue is completed, and the incremental migration is stopped.

[0150] Step S607: Compare source and target data. Compare the data in the source and target storage systems to ensure that the source and target data are consistent.

[0151] Step S608: Stop mirroring back to the source. After the source and target storage data are consistent, stop mirroring back to the source, and the migration is completed.

[0152] Figure 6The migration system shown forms a complete migration solution through historical data migration, incremental migration, and mirroring back to the source of the object storage system. The migration solution applies switching without interrupting the application, achieving seamless switching of the application and minimizing the impact on the application. Corresponding to the data processing method provided in the first embodiment of the present application, the second embodiment of the present application also provides a data processing device.

[0153] like Figure 7 As shown, the data migration device includes:

[0154] A historical data determination unit 701 is used to determine historical data in a source data storage system according to a set time;

[0155] A historical data migration unit 702, configured to migrate the historical data to a target data storage system;

[0156] A newly added data obtaining unit 703 is used to obtain newly added data in the source data storage system after the set time, wherein the newly added data is obtained by parsing the data increment notification in the message queue;

[0157] The newly added data migration unit 704 is used to migrate the newly added data to the target data storage system.

[0158] Optionally, the historical data migration unit includes:

[0159] Generate a historical data migration task according to the file size of the historical data and the number of files of the historical data;

[0160] Execute the historical data migration task to migrate the historical data to the target data storage system.

[0161] Optionally, the newly added data obtaining unit includes:

[0162] Configuring data increment notification in the source data storage system;

[0163] Get the message arrival notification of the message queue;

[0164] Obtaining a data increment notification in the message queue according to the message arrival notification;

[0165] The data increment notification in the message queue is parsed to obtain the newly added data in the source data storage system after the set time.

[0166] Optionally, the message queue includes at least one of the following: an unbuffered message queue; a buffered message queue.

[0167] Optionally, the device further comprises:

[0168] The migration unit is used to migrate the unmigrated new data to the target data storage system by mirroring back to the source after the historical data migration is completed.

[0169] Optionally, the migration unit is specifically used to:

[0170] When the specified newly added data cannot be found in the target data storage system, the return source address corresponding to the specified newly added data is obtained according to the specified mirror return source configuration basis;

[0171] According to the return-to-source address, the unmigrated newly added data is migrated to the target storage system.

[0172] Optionally, the migration unit is specifically used to:

[0173] According to the return-to-source address, the unmigrated new data is migrated to the target storage system by using a dedicated line.

[0174] Optionally, the migration unit is specifically used to:

[0175] Get a dedicated customer line;

[0176] Authenticating the customer dedicated line to determine whether the customer dedicated line has access rights;

[0177] If yes, the client dedicated line is used to access the source address to migrate the unmigrated new data to the target storage system.

[0178] Optionally, the device further comprises:

[0179] The data verification unit is used to verify the data in the target data storage system with the data in the source data storage system to determine whether the data are consistent; if they are consistent, the mirroring back to the source is stopped.

[0180] It should be noted that for the detailed description of the device provided in the second embodiment of the present application, reference can be made to the relevant description of the first embodiment of the present application, which will not be repeated here.

[0181] Corresponding to the data processing method provided in the first embodiment of the present application, the third embodiment of the present application provides an electronic device, including:

[0182] Processor; and

[0183] The memory is used to store a program of the data processing method. After the device is powered on and the program of the data processing method is run by the processor, the following steps are performed:

[0184] Determine historical data in the source data storage system according to a set time;

[0185] Migrating the historical data to a target data storage system;

[0186] Obtaining newly added data in the source data storage system after the set time, wherein the newly added data is obtained by parsing the data increment notification in the message queue;

[0187] Migrate the newly added data to the target data storage system.

[0188] Optionally, migrating the historical data to a target data storage system includes:

[0189] Generate a historical data migration task according to the file size of the historical data and the number of files of the historical data;

[0190] Execute the historical data migration task to migrate the historical data to the target data storage system.

[0191] Optionally, the obtaining of new data in the source data storage system after the set time, wherein the new data is obtained by parsing a data increment notification in a message queue, includes:

[0192] Configuring data increment notification in the source data storage system;

[0193] Get the message arrival notification of the message queue;

[0194] Obtaining a data increment notification in the message queue according to the message arrival notification;

[0195] The data increment notification in the message queue is parsed to obtain the newly added data in the source data storage system after the set time.

[0196] Optionally, the message queue includes at least one of the following: an unbuffered message queue; a buffered message queue.

[0197] Optionally, the electronic device further performs the following steps:

[0198] When the historical data migration is completed, the unmigrated new data will be migrated to the target data storage system by mirroring back to the source.

[0199] Optionally, the step of migrating the unmigrated newly added data to the target storage system by mirroring back to the source includes:

[0200] When the specified newly added data cannot be found in the target data storage system, the return source address corresponding to the specified newly added data is obtained according to the specified mirror return source configuration basis;

[0201] According to the return-to-source address, the unmigrated newly added data is migrated to the target storage system.

[0202] Optionally, migrating the unmigrated newly added data to the target storage system according to the return-to-source address includes:

[0203] According to the return-to-source address, the unmigrated new data is migrated to the target storage system by using a dedicated line.

[0204] Optionally, migrating the unmigrated new data to the target storage system by using a dedicated line according to the return-to-source address includes:

[0205] Get a dedicated customer line;

[0206] Authenticating the customer dedicated line to determine whether the customer dedicated line has access rights;

[0207] If yes, the client dedicated line is used to access the source address to migrate the unmigrated new data to the target storage system.

[0208] Optionally, the electronic device further performs the following steps:

[0209] Performing data verification on the data in the target data storage system and the data in the source data storage system to determine whether the data are consistent;

[0210] If they are consistent, stop mirroring back to the source.

[0211] It should be noted that for the detailed description of the electronic device provided in the third embodiment of the present application, reference can be made to the relevant description of the first embodiment of the present application, and will not be repeated here.

[0212] Corresponding to the data migration method provided in the first embodiment of the present application, the fourth embodiment of the present application provides a storage device storing a program of the data migration method, the program is executed by a processor, and the following steps are performed:

[0213] Determine historical data in the source data storage system according to a set time;

[0214] Migrating the historical data to a target data storage system;

[0215] Obtaining newly added data in the source data storage system after the set time, wherein the newly added data is obtained by parsing the data increment notification in the message queue;

[0216] Migrate the newly added data to the target data storage system.

[0217] It should be noted that for the detailed description of the storage device provided in the fourth embodiment of the present application, reference can be made to the relevant description of the first embodiment of the present application, and will not be repeated here.

[0218] Although the present application is disclosed as above in the form of a preferred embodiment, it is not intended to limit the present application. Any technical personnel in this field may make possible changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be based on the scope defined by the claims of the present application.

[0219] In a typical configuration, a computing device includes one or more processors (CPU), memory-mapped input / output interfaces, network interfaces, and memory.

[0220] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0221] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include non-transitory media such as modulated data signals and carrier waves.

[0222] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

Claims

1. A data migration method, comprising: Determine historical data in the source data storage system according to a set time; Migrating the historical data to a target data storage system; Obtaining newly added data in the source data storage system after the set time, wherein the newly added data is obtained by parsing the data increment notification in the message queue; the data increment notification comes from the source data storage system; Migrate the newly added data to the target data storage system.

2. The method according to claim 1, wherein migrating the historical data to a target data storage system comprises: Generate a historical data migration task according to the file size of the historical data and the number of files of the historical data; Execute the historical data migration task to migrate the historical data to the target data storage system.

3. The method according to claim 1, wherein the newly added data in the source data storage system after the set time is obtained, wherein: The newly added data is obtained by parsing the data increment notification in the message queue, including: Configuring data increment notification in the source data storage system; Get the message arrival notification of the message queue; Obtaining a data increment notification in the message queue according to the message arrival notification; The data increment notification in the message queue is parsed to obtain the newly added data in the source data storage system after the set time.

4. According to the method of claim 1, the message queue comprises at least one of the following: an unbuffered message queue; a buffered message queue.

5. The method according to claim 1, further comprising: When the historical data migration is completed, the unmigrated new data will be migrated to the target data storage system by mirroring back to the source.

6. The method according to claim 5, wherein migrating the unmigrated new data to the target storage system by mirroring back to the source comprises: When the specified newly added data cannot be found in the target data storage system, the return source address corresponding to the specified newly added data is obtained according to the specified mirror return source configuration basis; According to the return-to-source address, the unmigrated newly added data is migrated to the target storage system.

7. The method according to claim 6, wherein the step of migrating the unmigrated newly added data to the target storage system according to the return-to-source address comprises: According to the return-to-source address, the unmigrated new data is migrated to the target storage system by using a dedicated line.

8. The method according to claim 7, wherein the step of migrating the unmigrated new data to the target storage system by using a dedicated line according to the return-to-source address comprises: Get a dedicated customer line; Authenticating the customer dedicated line to determine whether the customer dedicated line has access rights; If yes, the client dedicated line is used to access the source address to migrate the unmigrated new data to the target storage system.

9. The method according to claim 5, further comprising: Verify the data in the target data storage system and the data in the source data storage system to determine whether the data are consistent; If they are consistent, stop mirroring back to the source.

10. A data migration device, comprising: A historical data determination unit, used to determine the historical data in the source data storage system according to a set time; A historical data migration unit, used to migrate the historical data to a target data storage system; A newly added data obtaining unit, used to obtain newly added data in the source data storage system after the set time, wherein the newly added data is obtained by parsing the data increment notification in the message queue; the data increment notification comes from the source data storage system; The newly added data migration unit is used to migrate the newly added data to the target data storage system.

11. An electronic device, comprising: processor; as well as The memory is used to store a program of the data migration method. After the device is powered on and the program of the data migration method is run by the processor, the following steps are performed: Determine historical data in the source data storage system according to a set time; Migrating the historical data to a target data storage system; Obtaining newly added data in the source data storage system after the set time, wherein the newly added data is obtained by parsing the data increment notification in the message queue; the data increment notification comes from the source data storage system; Migrate the newly added data to the target data storage system.

12. A storage device storing a program of a data migration method, wherein the program is executed by a processor to perform the following steps: Determine historical data in the source data storage system according to a set time; Migrating the historical data to a target data storage system; Obtaining newly added data in the source data storage system after the set time, wherein the newly added data is obtained by parsing the data increment notification in the message queue; the data increment notification comes from the source data storage system; Migrate the newly added data to the target data storage system.

Images