Data migration method and device, electronic equipment and storage medium

Abstract

The invention discloses a data migration method and device, electronic equipment and a storage medium, and relates to the technical field of data storage, and the data migration method comprises the steps that source storage equipment determines a second storage area in target storage equipment according to a root storage path of at least one piece of data, and then maps the second storage area to the source storage equipment to obtain a first storage area, the data stored in the first storage area is the same as the data stored in the second storage area, and after the data stored in the first storage area is modified, the storage area in the second storage area is correspondingly changed. And then, according to the data stored in the first storage area, determining target data which is not stored in the target storage equipment in the at least one piece of data, and finally migrating the target data which is not stored in the target storage equipment from the source storage equipment to the target storage equipment. According to the data migration method provided by the embodiment of the invention, the data migration efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of data storage technology, and in particular to a data migration method, device, electronic device and storage medium. Background Art

[0002] Data migration refers to the process of transferring data from one storage device to another, or from one system to another.

[0003] In related technologies, data migration solutions often rely on third-party servers or centralized storage gateways. Specifically, using a third-party server as an example, data migration can proceed as follows: upon receiving a data migration request, the source storage device invokes the relevant protocol for that request within its kernel module. Then, according to the protocol, the data to be migrated is sent to the third-party server. Upon receiving the data from the source storage device, the third-party server performs format conversion, data cleansing, and other processing before sending the processed data to the target storage device.

[0004] However, during the data migration process using the above method, the intervention of a third-party server or a centralized storage gateway prolongs the data migration transmission path, resulting in lower data migration efficiency. Summary of the Invention

[0005] The present application provides a data migration method, device, electronic device and storage medium, which improve the efficiency of data migration.

[0006] This application provides a data migration method, including:

[0007] Receive a data migration request; the data migration request is used to instruct to migrate at least one data from a source storage device to a target storage device;

[0008] Generate a first storage area based on a root storage path of at least one data; wherein the first storage area is an area of ​​the source storage device mapped to the second storage area in the target storage device, the data stored in the first storage area is the same as the data stored in the second storage area, and the root storage path of the second storage area is the same as the root storage path of the at least one data;

[0009] determining at least one target data from at least one data according to the data stored in the first storage area;

[0010] Migrate at least one target data from a source storage device to a target storage device.

[0011] The present application provides a data migration device, comprising:

[0012] A receiving module, configured to receive a data migration request; the data migration request is used to instruct to migrate at least one data from a source storage device to a target storage device;

[0013] a generating module, configured to generate a first storage area based on a root storage path of at least one data; wherein the first storage area is an area of ​​the source storage device mapped to the second storage area of ​​the target storage device, the data stored in the first storage area is the same as the data stored in the second storage area, and the root storage path of the second storage area is the same as the root storage path of the at least one data;

[0014] a determination module, configured to determine at least one target data from at least one data according to the data stored in the first storage area;

[0015] The migration module is configured to migrate at least one target data from a source storage device to a target storage device.

[0016] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned data migration methods when executing the computer program.

[0017] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, any of the above-mentioned data migration steps is implemented.

[0018] The data migration method, apparatus, electronic device, and storage medium provided by the embodiments of the present application are as follows: a source storage device determines a second storage area in a target storage device based on a root storage path of at least one data, and then maps the second storage area to the source storage device to obtain a first storage area. The data stored in the first storage area is identical to the data stored in the second storage area, and after the data stored in the first storage area is modified, the storage area in the second storage area will also change accordingly. Then, based on the data stored in the first storage area, data not stored in the target storage device is determined from at least one data, and finally, the data not stored in the target storage device is migrated from the source storage device to the target storage device.

[0019] During the above-described data migration process, since the first storage area is the storage area mapped to the second storage area in the source storage device, the data stored in the first storage area is identical to the data stored in the second storage area. Therefore, data migration can be performed based on the first storage area, thereby improving the efficiency of data migration. In addition, when migrating at least one data item, by determining data not stored in the target storage device from the at least one data item, and then migrating the data not stored in the target storage device from the source storage device to the target storage device, repeated migration of data already stored in the target storage device is avoided, further improving the efficiency of data migration and saving data migration resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0021] Figure 1 Schematic diagram of application scenarios provided by embodiments of the present application;

[0022] Figure 2 A flowchart of a data migration method provided in an embodiment of the present application;

[0023] Figure 3 A schematic diagram of a process for generating a first storage area provided in an embodiment of the present application;

[0024] Figure 4 A schematic diagram of a process for determining at least one target data provided in an embodiment of the present application;

[0025] Figure 5 A schematic diagram of a process for migrating at least one target data from a source storage device to a target storage device provided in an embodiment of the present application;

[0026] Figure 6 A schematic diagram of a data migration method provided in an embodiment of the present application;

[0027] Figure 7 A schematic diagram of the structure of a NAS container migration engine module provided in an embodiment of the present application;

[0028] Figure 8 A schematic diagram of determining the migration status of target data provided in an embodiment of the present application;

[0029] Figure 9 An architectural diagram of a user-mode hybrid access module provided in an embodiment of the present application;

[0030] Figure 10 A schematic diagram of the structure of a data migration device provided in an embodiment of the present application;

[0031] Figure 11 This is a schematic diagram of the structure of the electronic device provided in this application. DETAILED DESCRIPTION

[0032] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0033] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0034] First, let’s explain the terms involved in this application:

[0035] Kernel state: Kernel state is a privileged mode provided by the operating system. In a computer system, kernel state has the highest permissions and is responsible for implementing the operating system's core functions, such as process management, file system management, and device drivers. Kernel state also stores the computer system's key resources and protocols. When user state needs to access kernel state resources or protocols, it must apply to kernel state. Only after kernel state verification is the application program in user state allowed to access kernel state resources and protocols, thus ensuring the security and stability of the computer system.

[0036] User state: User state is an operating environment for applications provided by the operating system. In some embodiments, for data migration in user state, the source storage device runs in user state when executing the data migration task. To ensure the independence and security of each task, resource access in user state is restricted, that is, some key resources or protocols cannot be directly accessed in the user state of the data migration task. When the user state needs to access these key resources or protocols, it needs to send a request to the kernel state. After verifying the legitimacy of the request, the kernel state allows the user state of the data migration to access these key resources or protocols.

[0037] Data migration refers to the process of transferring data from a source storage device to a target storage device. In related technologies, data migration can be achieved through a third-party server or a centralized storage gateway. Taking a third-party server as an example, the data migration method can be as follows: the source storage device receives a data migration request, and the data migration request is used to migrate at least one data from the source storage device to the target storage device. Then, the relevant protocol of the data migration request is called in the kernel module. The relevant protocol may include, for example, the Network File System (NFS) protocol and the Server Message Block (SMB) protocol. Then, data migration is performed on at least one data according to the called protocol. Specifically, the source storage device sends at least one data to the third-party server through a preset resource quota. After receiving the at least one data, the third-party server performs format conversion, data cleaning and other processing on the at least one data, and sends the processed at least one data to the target storage device.

[0038] However, in the process of relying on a third-party server or a centralized storage gateway to achieve data migration, the data migration transmission path is extended, and the risk of single point failure during the data migration process is increased. For each data migration request, after receiving the data migration request, the source storage device needs to call the relevant data migration protocol in the kernel module, and then perform data migration, resulting in a large consumption of time resources and performance resources required for data migration. In addition, for each data in at least one data, the data is migrated according to a preset resource quota. In summary, when performing data migration through the above method, the efficiency of data migration is low, and the resource utilization rate during the data migration process is low.

[0039] In some embodiments, a decentralized architecture can be used to make the data migration process independent of third-party servers or centralized storage gateways. Specifically, by implementing a point-to-point direct connection and a distributed coordination mechanism between the source and target storage devices, the collaborative access capabilities between the source and target storage devices are improved. However, in the process of implementing data migration through a decentralized structure, the efficiency of data migration is low, and fixed resources are allocated for data migration, resulting in low resource utilization of the source storage device.

[0040] An embodiment of the present application provides a data migration method, in which a source storage device determines a second storage area in a target storage device based on a root storage path of at least one data, and then maps the second storage area to the source storage device to obtain a first storage area. The data stored in the first storage area is the same as the data stored in the second storage area, and after the stored data in the first storage area is modified, the storage area in the second storage area will also change accordingly. In this way, data migration can be performed based on the first storage area, avoiding the need to perform data migration through a third-party server or a centralized storage gateway, thereby improving the efficiency of data migration. In addition, when migrating at least one data, data not stored in the target storage device is determined in at least one data, and then the data not stored in the target storage device is migrated from the source storage device to the target storage device, thereby avoiding repeated migration of data already stored in the target storage device, improving the efficiency of data migration, and saving resources for data migration.

[0041] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0042] In conjunction with the specific application environment architecture or specific hardware architecture on which the execution of the data migration method depends, the specific application environment architecture or specific hardware architecture is described here. Figure 1 , Figure 1 This is a schematic diagram of an application scenario provided by an embodiment of the present application. Figure 1 As shown, the system includes a client 11, a source storage device 12, and a target storage device 13. The client 11 may be, for example, a server or other device; the source storage device 12 may include, for example, a storage server, a storage system, or other device with data storage capabilities; and the target storage device 13 may include, for example, a storage server, a storage system, or other device with data storage capabilities.

[0043] In actual application, the client 11 can communicate with the source storage device 12. For example, the client 11 can send a data migration request to the source storage device 12. The source storage device 12 can communicate with the target storage device 13. For example, the source storage device 12 can send a first request message to the target storage device 13, and the target storage device 13 can send a first response message to the source storage device 12.

[0044] It should be noted that Figure 1 This is just an example to illustrate an application scenario, and is not intended to limit the application scenario.

[0045] The following specific embodiments describe in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0046] Figure 2 A flow chart of a data migration method provided in an embodiment of the present application is shown as follows: Figure 2 As shown, an embodiment of the present application provides a data migration method, which is described in detail as follows:

[0047] S201. Receive a data migration request. The data migration request is used to instruct to migrate at least one data from a source storage device to a target storage device.

[0048] The execution subject of the present application may be a source storage device or a data migration device provided in the source storage device. The data migration device may be implemented by software or by a combination of software and hardware.

[0049] The source storage device is a storage device where at least one data is located, and the target storage device is a storage device where at least one data is located after data migration is performed on the at least one data.

[0050] In some embodiments, a data migration request is sent by a client to a source storage device, and the data migration request includes path information of at least one data item and address information of a target storage device. After receiving the data migration request, the source storage device determines to migrate the at least one data item to the target storage device based on the path information of the at least one data item in the data migration request and the address information of the target storage device.

[0051] The path information of at least one data may be, for example, a storage path of at least one data in a source storage device; and the address information of a target storage device may be, for example, an Internet Protocol (IP) address of the target storage device.

[0052] S202. Generate a first storage area based on a root storage path of at least one data; wherein the first storage area is an area of ​​the target storage device mapped to the source storage device by the second storage area, the data stored in the first storage area is the same as the data stored in the second storage area, and the root storage path of the second storage area is the same as the root storage path of at least one data.

[0053] For each data item, the storage path of the data item is used to indicate the specific storage location of the data item in the source storage device. For example, for data item 1, the storage path of data item 1 in the source storage device is A / B, which means that data item 1 is stored in folder B in the source storage device, and folder B is located in folder A.

[0054] The root storage path of at least one data item indicates the storage location of the at least one data item on the source storage device. For example, assume that the at least one data item includes data 1 and data 2, where data 1 has storage paths A / B on the source storage device; and data 2 has storage paths A / C on the source storage device. Then, the root storage path of the at least one data item is A, indicating that the at least one data item is stored in folder A on the source storage device.

[0055] The second storage area is a storage area in the target storage device, and the storage path of the second storage area in the target storage device is the same as the root storage path of the at least one data item. For example, assuming that the root storage path of the at least one data item is A / B, the storage path of the second storage area in the target storage device is A / B.

[0056] After mapping the second storage area in the target storage area to the source storage device, a first storage area is obtained. At this point, the data stored in the first storage area is the same as the data stored in the second storage area. Modifying the data stored in the first storage area will modify the data stored in the second storage area accordingly. Modifying the data stored in the second storage area will modify the storage path in the first storage area accordingly.

[0057] S203: Determine at least one target data from at least one data according to the data stored in the first storage area.

[0058] Since the data stored in the first storage area is identical to the data stored in the second storage area, for each data item in the at least one data item, it is possible to determine whether the data item is stored in the second storage area based on the data item stored in the first storage area. If the data item is stored in the second storage area, it indicates that the target storage device already has the data item stored in it, and in this case, data migration of the data item is not required. If the data item is not stored in the second storage area, it indicates that the target storage device does not have the data item stored in it, and in this case, data migration of the data item is required.

[0059] For example, assuming that at least one data includes data 1, data 2 and data 3, and according to the data stored in the first storage area, data 1 is in the second storage area, then data 1 is already stored in the target storage device, and data 1 does not need to be migrated at this time; according to the data stored in the first storage area, it is determined that data 2 is not in the second storage area, then data 2 is not stored in the target storage device, and data migration of data 2 is required at this time; according to the data stored in the first storage area, it is determined that data 3 is not in the second storage area, then data 3 is not stored in the target storage device, and data migration of data 3 is required at this time.

[0060] At least one target data set includes data requiring data migration. For example, assume that the at least one data set includes data 1, data 2, and data 3. Data 1 does not require data migration, data 2 requires data migration, and data 3 requires data migration. Therefore, in the at least one data set, it is determined that the at least one target data set includes data 2 and data 3.

[0061] S304: Migrate at least one target data from the source storage device to the target storage device.

[0062] For each target data item, since the target data item is not stored in the target storage device, the target data item is migrated from the source storage device to the target storage device. Thus, after the migration of the at least one target data item from the source storage device to the target storage device is completed, the target storage device stores the at least one data item, thereby completing the migration of the at least one data item from the source storage device to the target storage device.

[0063] exist Figure 2 In the illustrated embodiment, after receiving a data migration request, the source storage device determines a second storage area in the target storage device based on the root storage path of at least one data item. The second storage area is then mapped to the source storage device to obtain a first storage area. The data stored in the first storage area is identical to the data stored in the second storage area. Then, based on the data stored in the first storage area, at least one target data item is determined from the at least one data item, and the at least one target data item is migrated from the source storage device to the target storage device.

[0064] During the above-mentioned data migration process, since the first storage area is the storage area in the source storage device mapped to the second storage area, the data stored in the first storage area is the same as the data stored in the second storage area, and after the stored data in the first storage area is modified, the storage area in the second storage area will also change accordingly. In this way, the data migration process does not rely on a third-party server or a centralized storage gateway, thereby improving the efficiency of data migration. In addition, based on the data stored in the first storage area, data not stored in the target storage device can be determined in at least one data, and then the data not stored in the target storage device can be migrated from the source storage device to the target storage device, thereby avoiding repeated migration of data already stored in the target storage device, improving the efficiency of data migration, and saving data migration resources.

[0065] exist Figure 2 Based on the embodiment shown below, combined with Figure 3 The method of generating the first storage area according to the root storage path of at least one data in the embodiment of the present application is further explained.

[0066] Figure 3 A schematic diagram of a process for generating a first storage area is provided in an embodiment of the present application, such as Figure 3 As shown, the process may include the following steps:

[0067] S301: Create a virtual storage space in a source storage device.

[0068] After receiving the data migration request, the source storage device establishes a data migration task according to the data migration request, creates a task process for the data migration task, and then creates a virtual storage space in the task process for the data migration task.

[0069] In some embodiments, the virtual storage space is used to store at least one data item, related data generated during data migration of the at least one data item, and state information during data migration of the at least one data item. The related data generated during data migration of the at least one data item may include, for example, verification data and a hash value of the at least one data item; and the state information during data migration of the at least one data item may include, for example, the migration state of the at least one data item.

[0070] In some embodiments, before migrating at least one data, the source storage device reads the at least one data into the virtual storage space. This facilitates centralized management of the at least one data to be migrated and improves the efficiency and accuracy of data migration.

[0071] S302: Send a first request message to the target storage device according to the NFS protocol, where the first request message is used to request to determine a second storage area in the target storage device.

[0072] The NFS protocol is a distributed file system protocol. The source storage device can remotely access the file system on the target storage device based on the NFS protocol.

[0073] The NFS protocol includes an NFS access terminal and an NFS share terminal. The NFS access terminal is deployed on the source storage device and is implemented through software.

[0074] It should be noted that the NFS access end is deployed in the user state of the source storage device.

[0075] After deploying an NFS share client on the target storage device, the source storage device may send a first request message to the target storage device via the NFS access client. In some embodiments, the first request message includes a root storage path of at least one data item. The source storage device sends the first request message to the target storage device, requesting that a second storage area be determined on the target storage device, where the storage path of the second storage area on the target storage device is the same as the root storage path of the at least one data item.

[0076] S303: Receive a first response message sent by the target storage device according to the NFS protocol, where the first response message includes the data stored in the second storage area.

[0077] After receiving the first request message, the NFS sharing end in the target storage device determines whether a second storage area exists in the target storage device according to a root storage path of at least one data.

[0078] In the case that the target storage device has a second storage area, the data stored in the second storage area is sent to the source storage device through the NFS sharing end.

[0079] If the second storage area does not exist on the target storage device, a second storage area is created on the target storage device based on the root storage path of the at least one data item. The data stored in the second storage area is then sent to the source storage device via the NFS share. It should be noted that at this point, the data stored in the second storage area is empty, i.e., the data stored in the second storage area included in the first response message is empty.

[0080] S304: Mount the data stored in the second storage area to the virtual storage space according to the NFS protocol to obtain the first storage area.

[0081] After receiving the first response message, the source storage device mounts the data stored in the second storage area into the virtual storage space by calling an NFS protocol function, thereby obtaining the first storage area. Therefore, the data stored in the first storage area is consistent with the data stored in the second storage area. The NFS protocol function called at this time may be, for example, nfs_mount (nfs_mount is used to obtain the data stored in the second storage area).

[0082] In some embodiments, the data stored in the second storage area can be mounted into the virtual storage space based on the address information of the second storage area. Specifically, the IP address of the target storage device and the storage path of the second storage area in the target storage device are obtained. Then, the NFS protocol function is called to parse the IP address of the target storage device and the storage path of the second storage area in the target storage device to obtain the address information of the second storage area. The NFS protocol function called at this time may be, for example, nfs_parse_url_full (used to parse the IP address, shared directory, etc. of the target storage device to obtain the key address information required for NFS sharing), and the address information of the second storage area may include, for example, the IP address of the target storage device, the port number, the storage path of the second storage area in the target storage device, and other information.

[0083] Finally, the NFS protocol function nfs_mount (used to mount shared data on a mount point) is called to mount the data stored in the second storage area into the virtual storage space, resulting in the first storage area. It should be noted that the first storage area is a shared area between the source storage device and the target storage device. Both the source storage device and the target storage device can read and write to the first storage area, and when the source storage device accesses the data stored in the first storage area, it is equivalent to remotely accessing the data stored in the second storage area.

[0084] exist Figure 3 In the illustrated embodiment, the source storage device deploys an NFS access terminal in user mode, and the target storage device deploys an NFS sharing terminal. After receiving a data migration request, the source storage device creates a virtual storage space, and then sends a first request message to the target storage device based on the NFS access terminal, requesting the target storage device to determine a second storage area. After receiving the first request message, the NFS sharing terminal of the target storage device determines the second storage area and sends the content stored in the second storage area to the source storage device. After receiving the content stored in the second storage area, the source storage device mounts the content stored in the second storage area to the virtual storage area based on the address information of the NFS access terminal and the second storage area, thereby obtaining the first storage area.

[0085] In the above method, the first storage area is a shared area between the source storage device and the target storage device, and the data stored in the first storage area is consistent with the data stored in the second storage area. The data stored in the first storage area can be modified to modify the data stored in the second storage area, thereby avoiding the need to rely on a third-party server or a centralized storage gateway for data migration, improving the efficiency of data migration, and reducing the possibility of single point failures during data migration. In addition, the NFS access terminal is deployed in the user state of the source storage device. In this way, the NFS protocol can be called directly without requesting to call the NFS protocol in the kernel state, saving the process of requesting to call the NFS protocol in the kernel state, and further improving the efficiency of data migration.

[0086] exist Figure 3 In the embodiment shown, a method of generating a first storage area in a source storage device is introduced. Figure 4 , further illustrating a method for determining at least one target data in at least one data according to the first storage area in an embodiment of the present application.

[0087] Figure 4 This is a flow chart of determining at least one target data provided in an embodiment of the present application. Figure 4 ,include:

[0088] S401. For any first data among at least one data, determine whether second data is stored in a first storage area according to a first storage path of the first data in a source storage device, and the second storage path of the second data in a target storage device is the same as the first storage path.

[0089] The first storage path is used to indicate the location where the first data is stored in the source storage device. For example, assuming that the first storage path of the first data in the source storage device is A / B / C, it means that the first data is stored in folder C in the source storage device, where folder C is located in folder B in the source storage device, and folder B is located in folder A in the source storage device.

[0090] The method for determining whether the second data is stored in the first storage area based on the first storage path can be as follows: determining whether there is data in the first storage area whose storage path in the target storage device is consistent with the first storage path. If so, the data is determined to be the second data, and it is determined that the second data is stored in the first storage area; if not, it is determined that the second data is not stored in the second storage area.

[0091] In some embodiments, if the second data is not stored in the second storage area, a folder can be created in the first storage area by calling a local protocol function. The storage path of the folder in the target storage device is consistent with the storage path of the folder containing the first data in the metastorage device. The local protocol function called in this case may include, for example, icfs_mkdir (used to create a new directory file in a distributed file system) or icfs_create (used to create a new directory file in a distributed file system).

[0092] For example, assuming the first storage path of the first data is A / B / C, the folder where the first data is located is folder C, and the storage path of folder C on the source storage device is A / B. If the second data is not stored in the second storage area, icfs_mkdir or icfs_create is called to create a folder D in the first storage area. The storage path of folder D on the target storage device is A / B.

[0093] S402: When the second data is stored in the first storage area, determine whether the first data belongs to the target data according to the first data and the second data.

[0094] When the first storage area stores the second data, the first data and the second data may be the same or different. Therefore, it is necessary to further determine whether the first data and the second data are the same. If the first data and the second data are the same, it means that the first data is already stored in the target storage device and there is no need to migrate the first data. In other words, it is determined that the first data does not belong to the target data. If the first data and the second data are different, it means that the first data is not stored in the target storage device and it is necessary to migrate the first data. In other words, it is determined that the first data belongs to the target data.

[0095] In some embodiments, the method of determining whether the first data belongs to the data target can be as follows: determine whether the attribute information of the first data is the same as the attribute information of the second data; determine whether the storage information of the first data is the same as the storage information of the second data; when the attribute information of the first data is the same as the attribute information of the second data, and the storage information of the first data is the same as the storage information of the second data, determine that the first data does not belong to the target data; when the attribute information of the first data is different from the attribute information of the second data, and / or the storage information of the first data is different from the storage information of the second data, determine that the first data belongs to the target data.

[0096] The attribute information of the first data may include, for example, the data size of the first data, the modification time of the first data, etc., and the attribute information of the second data may include, for example, the data size of the second data, the modification time of the second data, etc. It should be noted that, if the attribute information of the first data includes the data size of the first data and the modification time of the first data, the attribute information of the second data also includes the data size of the second data and the modification time of the second data; and if the attribute information of the first data includes the data size of the first data, the attribute information of the second data also includes the data size of the second data.

[0097] Taking the example of attribute information including data size and modification time, the method for determining whether the attribute information of the first data is the same as the attribute information of the second data can be as follows: determine whether the data size of the first data is the same as the data size of the second data; determine whether the modification time of the first data is the same as the modification time of the second data. If the data size of the first data is the same as the data size of the second data, and the modification time of the first data is the same as the modification time of the second data, the attribute information of the first data is determined to be the same as the attribute information of the second data; if the data size of the first data is different from the data size of the second data, and / or the modification time of the first data is different from the modification time of the second data, the attribute information of the first data is determined to be different from the attribute information of the second data.

[0098] In some embodiments, the first data and the second data are both stored in block storage, the storage information of the first data is used to indicate the block storage where the first data is located, and the storage information of the second data is used to indicate the block storage where the second data is located.

[0099] In some embodiments, a method for determining whether the storage information of the first data is the same as the storage information of the second data can be as follows: determine the first storage block of the first data in the source storage device, and the second storage block of the second data in the target storage device; shard the first storage block to obtain M first shards, where M is a positive integer; shard the second storage block to obtain M second shards; and determine whether the storage information of the first data is the same as the storage information of the second data based on the hash values ​​of the M first shards and the hash values ​​of the M second shards.

[0100] The first storage block is used to store the first data, and the second storage block is used to store the second data. Whether the storage information of the first data is consistent with the storage information of the second data can be determined by determining whether the first storage block and the second storage block are consistent. If the first storage block and the second storage block are consistent, the storage information of the first data is determined to be consistent with the storage information of the second data; if the first storage block and the second storage block are inconsistent, the storage information of the first data is determined to be inconsistent with the storage information of the second data.

[0101] The first shard is the shard obtained by sharding the first storage block, and the second shard is the shard obtained by sharding the second storage block. It should be noted that the number of first shards is the same as the number of second shards, and there is a one-to-one correspondence between the first shards and the second shards. That is, for each first shard in the M first shards, there is a corresponding second shard in the M second shards.

[0102] For each first shard, a hash value of the first shard and a hash value of the second shard corresponding to the first shard are determined, and then whether the hash value of the first shard and the hash value of the second shard corresponding to the first shard are consistent. If the hash value of the first shard and the hash value of the second shard corresponding to the first shard are inconsistent, the first storage block is determined to be inconsistent with the second storage block; if the hash value of the first shard and the hash value of the second shard corresponding to the first shard are consistent, the first storage block is determined to be consistent with the second storage block.

[0103] In some embodiments, for any shard, the hash value of the shard can be determined by a hash function. For example, assuming that the hash value of the shard is determined by a 64-bit hash function, the formula for determining the hash value of the shard can be shown as formula (1):

[0104] H=Hash64(F) (1)

[0105] Where F represents the shard, Hash64() represents a 64-bit hash function, and H represents the hash value of the shard.

[0106] In some embodiments, it is possible to determine whether the hash value of the first shard and the hash value of the second shard corresponding to the first shard are consistent in a chain-association manner. Specifically, for each first shard, determine whether the hash value of the first shard and the hash value of the second shard corresponding to the first shard are consistent. In the case that the hash value of the first shard and the hash value of the second shard corresponding to the first shard are consistent, determine whether the hash value of the next first shard and the hash value of the second shard corresponding to the next first shard are consistent; in the case that the hash value of the first shard and the hash value of the second shard corresponding to the first shard are inconsistent, end the hash value comparison of the first shard and the second shard, and confirm that the first storage block and the second storage block are inconsistent. Exemplarily, the chain-association formula can be shown as formula (2):

[0107] H i+1 =Hash64(H i &F i+1 ) (2)

[0108] Where i is the index value, i is less than or equal to M. H iis the hash value of the first shard of the i-th shard, F i+1 Indicates the next first fragment, H i+1 The hash value of the next first shard.

[0109] Through the above chain association method, by calculating the hash values ​​of the data blocks and linking them together, a hash chain is formed. For the i-th second shard, it can be determined whether the hash value of the i-th second shard is consistent with the hash value of the i-th first shard. If the hash value of the i-th second shard is inconsistent with the hash value of the i-th first shard, it is directly determined that the storage information of the first data is inconsistent with the storage information of the second data, and the calculation of the hash value of the i+1-th first shard is notified.

[0110] In summary, if the attribute information of the first data is the same as the attribute information of the second data, and the storage information of the first data is the same as the storage information of the second data, this means that the file size of the first data is the same as the file size of the second data, the modification time of the first data is the same as the modification time of the second data, and the block storage of the first data is the same as the block storage of the second data. In this case, it can be determined that the first data is consistent with the second data, and further determined that the first data does not belong to the target data.

[0111] When the attribute information of the first data is different from the attribute information of the second data, and / or the storage information of the first data is different from the storage information of the second data, it can be determined that the first data is inconsistent with the second data, and further determined that the first data belongs to the target data.

[0112] S403: If the target storage device does not store the second data, determine the first data as the target data.

[0113] When the target storage device does not store the second data, it indicates that the first data needs to be migrated, and therefore the first data is determined as the target data.

[0114] exist Figure 4 In the illustrated embodiment, for each first data item, a determination is made as to whether the target storage device stores the second data item based on the first storage path of the first data item in the source storage device. If the target storage device does not store the second data item, the first data item is determined to belong to the target data item. If the target storage device stores the second data item, the first data item is determined to be identical to the second data item by determining whether the attribute information of the first data item is consistent with the attribute information of the second data item and whether the storage information of the first data item is consistent with the storage information of the second data item. If the first data item is identical to the second data item, the first data item is determined not to belong to the target data item. If the first data item is different from the second data item, the first data item is determined to belong to the target data item.

[0115] In the above process of determining at least one target data, it is first determined whether the target storage device stores second data, and the storage path of the second data in the target storage device is the same as the first storage path. If the second data is not stored in the target storage device, the first data is determined to be the target data. If the second data is stored in the target storage device, whether the first data and the second data are identical is further determined based on the attribute information and storage information. Based on whether the first data and the second data are identical, whether the first data is the target data is determined. This allows accurate identification of at least one target data in the at least one data.

[0116] exist Figure 4 The embodiment shown introduces a method for determining at least one target data in at least one data. Figure 5 In the embodiment of the present application, a method of migrating at least one target data from a source storage device to a target storage device is further described.

[0117] Figure 5 The present invention provides a flow chart of migrating at least one target data from a source storage device to a target storage device. Figure 5 Shown, including:

[0118] S501 : For each target data in at least one target data, determine a writing method for the target data.

[0119] In some embodiments, the write method includes a write method by a source storage device and a write method by a target storage device. The write method by the source storage device refers to writing the target data to the first storage area via the source storage device. Specifically, the target data can be written to the first storage area by calling an NFS access terminal. The write method by the target storage device refers to writing the target data to the first storage area via the target storage device. Specifically, the target data can be written to the first storage area by calling an NFS share terminal.

[0120] Since the first storage area is mapped to the source storage device by the second storage area, after the data stored in the first storage area is modified, the data stored in the second storage area will also be modified accordingly. Therefore, after the target data is written to the first storage area through the source storage device, the target data is also stored in the second storage area, thereby completing the migration of the target data from the source storage device to the target storage device.

[0121] In some embodiments, the writing method of the target data can be determined according to the first read / write efficiency of the source storage device and the second read / write efficiency of the target storage device. The first read / write efficiency is the read / write efficiency of the source storage device, which is used to indicate the speed and performance of the source storage device for reading and writing data. The greater the first read / write efficiency, the faster the source storage device reads and writes data and the better its performance. The second read / write efficiency is the read / write efficiency of the target storage device, which is used to indicate the speed and performance of the target storage device for reading and writing data. The greater the second read / write efficiency, the faster the target storage device reads and writes data and the better its performance.

[0122] Therefore, when the first read / write efficiency is greater than or equal to the second read / write efficiency, the writing method is determined to be the writing method of the source storage device; when the first read / write efficiency is less than the second read / write efficiency, the writing method is determined to be the writing method of the target storage device.

[0123] In some embodiments, writing by the source storage device can be preferentially selected. Specifically, when the first read / write efficiency is greater than or equal to the product of the second read / write efficiency and a preset parameter, the writing method is determined to be the writing method of the source storage device; when the first read / write efficiency is less than the product of the second read / write efficiency and the preset parameter, the writing method is determined to be the writing method of the target storage device.

[0124] Exemplarily, assuming that the preset parameter is 1.2, the first read / write efficiency is P1, and the second read / write efficiency is P2, then when P1 ≥ P2 * 1.2, the writing method is determined to be the writing method of the source storage device; when P1 < P2 * 1.2, the writing method is determined to be the writing method of the target storage device.

[0125] In some embodiments, the write latency and data access frequency of the source storage device, and the write latency and data access frequency of the target storage device can be obtained; the first read / write efficiency of the source storage device can be determined according to the write latency and data access frequency of the source storage device; the second read / write efficiency of the target storage device can be determined according to the write latency and data access frequency of the target storage device.

[0126] Among them, taking the write latency and data access frequency of the source storage device as an example, the write latency of the source storage device refers to the time from the issuance of the instruction to write data to the successful writing of the data to the storage device. The smaller the write latency of the source storage device, the higher the write efficiency and the better the write performance of the meta-storage device. The data access frequency of the source storage device refers to the number of times the data stored in the source storage device is accessed (read and written) within a certain time period. The data access frequency of the source storage device reflects the read and write performance of the source storage device. The smaller the data access frequency of the source storage device, the better the read and write performance of the source storage device. In some embodiments, a least recently used (LRU) algorithm can be used to determine the data access frequency of the source storage device.

[0127] Specifically, based on write latency and data access frequency, the formula for determining read and write efficiency can be shown as formula (3):

[0128] P=α·Latency +β·Hotness (3)

[0129] Where P represents read and write efficiency, Latency represents write latency, Hotness represents data access frequency, α is the weight coefficient of write latency, and β is the weight coefficient of data access frequency.

[0130] S502: Determine a resource quota for each target data in at least one target data.

[0131] For each target data item, the resource quota for that target data refers to the amount of resources allocated to it during the data migration process. These resources may include, for example, computing resources and bandwidth resources. Sufficient computing resources and bandwidth resources ensure efficient data migration for that target data. Therefore, by properly allocating resource quotas, data migration efficiency and resource utilization on the source storage device can be improved.

[0132] In some embodiments, the resource quota of the target data can be reasonably allocated according to the load of the source storage device. Specifically, the maximum load and current load of the source storage device can be obtained; and the resource quota of the target data can be determined according to the difference between the maximum load and the current load. Among them, the maximum load of the source storage device is used to indicate the maximum load limit that the source storage device can bear; the current load of the source storage device is used to indicate the workload of the source storage device at the current moment. The difference between the maximum load and the current load is used to indicate the load margin of the source storage device at the current moment; if the difference between the maximum load and the current load is larger, it means that the source storage device has a larger load margin at the current moment, and in this case, a larger resource quota can be allocated to the target data; if the difference between the maximum load and the current load is smaller, it means that the source storage device has a smaller load margin at the current moment, and in this case, a smaller resource quota can be allocated to the target data.

[0133] In some embodiments, the resource quota of the target data can be determined by a Proportion-Integration-Differentiation (PID) algorithm. Specifically, the formula for determining the resource quota by the PID algorithm can be shown as formula (4):

[0134]

[0135] Among them, R represents the resource quota of the target data, e(t) represents the difference between the maximum load and the current load of the source storage device, and K p is the first tuning parameter, K i is the second tuning parameter, K d is the third tuning parameter, and t represents the current time.

[0136] In some embodiments, ∫e(t)dt in formula (4) can be Wherein, 0 represents the time when the source storage device starts data migration, and t represents the current time. S503: For each target data in at least the target data, migrate the target data from the source storage device to the target storage device according to the writing mode and resource quota of the target data.

[0137] For each target data, after determining the writing method and resource quota of the target data, the target data is written to the target storage device according to the writing method and resource quota of the target data, thereby completing the migration of the target data from the source storage device to the target storage device.

[0138] In some embodiments, for each target data, the target data may be written to the target storage device as follows: calling the icfs_read (used to read data in the file system) and icfs_lseek (used to set the file's read / write offset, indicating the starting position of the next read / write operation) protocol functions to read the target data, and then calling nfs_write (used to write data in the first storage area) and nfs_lseek (used to set the file's read / write offset, indicating the actual position of the next read / write operation) to write the target data to the first storage area. After writing the target data to the first storage area, the NFS access end sends the target data to the NFS sharing end. After receiving the target data, the NFS sharing end synchronously writes the target data to the second storage area, thereby migrating the target data from the source storage device to the target storage device.

[0139] In some embodiments, for each target data, after completing the data migration of the target data, nfs_close (used to close the opened NFS shared file) is called to close the folder of the target data in the target storage device, and icfs_close (used to close the opened local file) is called to close the folder of the target data in the source storage device, and the modification time of the target data in the source storage device is determined as the modification time of the target data in the target storage device.

[0140] In some embodiments, for each target data, after the data migration task of the target data is terminated, nfs_destroy_url (used to release system resources) is called to release the resources occupied by the data migration task of the target data, and then the nfs_close protocol function is called to close the folder of the target data in the target storage device, and the icfs_close protocol function is called to close the folder of the target data in the source storage device.

[0141] exist Figure 5In the illustrated embodiment, a first read / write efficiency of the source storage device and a second read / write efficiency of the target storage device are determined, and a target data writing method is determined based on the first read / write efficiency and the second read / write efficiency. Furthermore, the load margin of the source storage device at the current moment can be determined based on the maximum load and current load of the source storage device, and the resource quota of the target data can be determined based on the load margin of the source storage device at the current moment. Finally, the target data is written to the target storage device based on the target data writing method and the resource quota. In the above method, based on the first read / write efficiency of the source storage device and the second read / write efficiency of the target storage device, if the first read / write efficiency is greater, the target data writing method is determined to be written by the source storage device; if the second read / write efficiency is greater, the target data writing method is determined to be written by the target storage device, thereby improving the efficiency of data migration of the target data. Furthermore, based on the load margin of the source storage device at the current moment, the resource quota of the target data can be flexibly adjusted, thereby improving the efficiency of data migration of the target data and also improving the resource utilization of the source storage device.

[0142] Based on any of the above embodiments, the data migration method provided in the embodiments of the present application is further described below in conjunction with specific embodiments.

[0143] Figure 6 A schematic diagram of a data migration method provided in an embodiment of the present application is shown as follows: Figure 6 As shown, it includes a client 11, a source storage device 12 and a target storage device 13.

[0144] In some embodiments, the source storage device 12 includes a command line interface module, a network attached storage (NAS) container migration engine module, a dynamic resource scheduling module, and a user-mode hybrid access module.

[0145] The command line interface module is used to receive a data migration request sent by the client 11 .

[0146] A NAS container is installed in the source storage device, and a local user library is deployed in the NAS container. The local user library refers to a collection of library files in the local system for use by user programs. For example, the local user library includes local protocol functions such as icfs_mkdir and icfs_create.

[0147] The NAS container migration engine module is a NAS container installed in the source storage device. The NAS container migration engine module includes a task management submodule and an incremental synchronization submodule. Figure 7 Further understand the NAS container migration engine module, Figure 7This is a structural diagram of a NAS container migration engine module provided in an embodiment of the present application. Figure 7 As shown, the NAS container migration engine module includes an engine initialization submodule, a task management submodule, and an incremental synchronization submodule.

[0148] The engine initialization submodule is used to initialize the NAS container migration engine module, for example, to deploy a local user library in the NAS container migration engine module.

[0149] The task management submodule includes a task status management submodule, a task logic processing submodule and a task persistence submodule. The task status management submodule is used to determine the status of each target data during the data migration process. Specifically, a timer can be deployed in the task status management submodule. For each target data, the task status management submodule determines the migration status of the target data within the time interval preset by the timer. The migration status may include, for example, a completed state, an unfinished state and an abnormal state. The completed state is used to indicate that the target data has completed data migration; the unfinished state is used to indicate that the target data has not completed data migration. At this time, the data migration progress of the target data at the current moment can be determined; the abnormal state is used to indicate that the data migration of the target data is abnormal. It should be noted that when the migration status is an abnormal state, the data migration of the target data is terminated, and the reason why the migration status is an abnormal state is determined.

[0150] In some embodiments, for each target data in at least one target data, Figure 8 Understand the determination of the migration status of the target data. Figure 8 A schematic diagram of determining the migration status of target data provided in an embodiment of the present application is shown as follows: Figure 8 As shown, when the data migration of the target data begins, the migration status of the target data is in an unfinished state. When the user controls to temporarily suspend the data migration of the target data, or when the target data generates an exception during the data migration process, the migration status of the target data changes from an unfinished state to an abnormal state; correspondingly, when the user controls to continue the data migration of the target data, the migration status of the target data changes from an abnormal state to an unfinished state; for target data with an unfinished or abnormal migration status, when the user controls to terminate the data migration of the target data, the migration status of the target data changes from an unfinished state to a completed state, or from an abnormal state to a completed state. For target data with an unfinished migration status, after the data migration of the target data is completed, the migration status of the target data changes from an unfinished state to a completed state.

[0151] For each target data, when the migration status of the target data is completed, the task persistence submodule stores the target data in the historical task database; when the migration status of the target data is incomplete, the task persistence submodule stores the target data in the real-time task database.

[0152] For each target data whose migration status is incomplete, the task logic processing submodule generates a data migration task thread for the target data and calls the incremental synchronization submodule to perform data migration on the target data.

[0153] In some embodiments, the incremental synchronization submodule includes a file synchronization transfer submodule and a layered difference detection submodule. The layered difference detection submodule is configured to detect whether first data and second data are identical. If the first data and second data are identical, the first data is determined to belong to the target data. If the first data and second data are different, the first data is determined not to belong to the target data. For at least one target data, the file synchronization transfer submodule is configured to migrate the at least one target data to a target storage device.

[0154] The dynamic resource scheduling module is used to determine the access method and resource quota of target data.

[0155] The user-mode hybrid access module includes a local storage access submodule and a remote storage access submodule, which are used to manage local storage access and remote storage access. Figure 9 To understand, Figure 9 This is an architectural diagram of a user-mode hybrid access module provided in an embodiment of the present application, such as Figure 9 As shown, the user-state hybrid access module includes a local storage access submodule and a remote storage access submodule, wherein the local storage access submodule includes a local user-state submodule and a local file system; the remote storage access submodule includes a user-state NFS access terminal, a mounting submodule and a parsing submodule.

[0156] The local user state is the interface through which applications running in user state interact with the local storage of the source storage device. By calling the functions or protocols provided by the local user state, various operations on the local storage can be performed. The local file system is used to organize and manage the data stored on the source storage device.

[0157] In some embodiments, a local user state can be configured in the local storage access submodule to create a local virtual storage space. The nfs read function is then called to read at least one data item from the local storage into a user state buffer, and a memory-mapped file (MMAP) is used to create a readable and writable storage area in the local virtual storage space. The at least one data item in the user buffer is then copied to the storage area. In this way, during data migration of at least one data item, the at least one data item can be directly read from the storage area, eliminating the need for the source storage device to request the server to read the at least one data item in the local storage. This improves data access efficiency, thereby improving data migration efficiency.

[0158] The user-mode NFS access terminal is used to obtain the data stored in the second storage area; the parsing submodule is used to parse the address information of the second storage area, and the mounting submodule is used to mount the data stored in the second storage area to the virtual storage space according to the address information of the second storage area and the NFS protocol to obtain the first storage area.

[0159] exist Figure 6 In the illustrated embodiment, the source storage device 12 includes a command line interface module, a NAS container migration engine module, a dynamic resource scheduling module, and a user-mode hybrid access module. After the command line interface module receives a data migration request from the client 11, the user-mode hybrid access module maps the second storage area in the target storage device 13 to the source storage device 12 through the remote storage access submodule, thereby achieving a point-to-point direct connection between the source storage device 12 and the target storage device 13 and avoiding dependence on a third-party server or centralized storage gateway. Furthermore, the user-mode hybrid access module maps at least one data in the data migration request from local storage to a virtual storage space in the local user mode through the local storage access submodule, thereby improving data access efficiency. The task management module then determines the migration status of each data in the at least one data. For each data, if the migration status of the data is incomplete, the incremental synchronization submodule is used to determine whether the data is the target data. If the data is the target data, the dynamic resource scheduling module is used to flexibly adjust the write mode and resource quota of the target data. Finally, according to the writing method and resource quota of the target data, the target data is migrated from the source storage device to the target storage device. In summary, the data migration method provided in the embodiment of the present application improves the efficiency of data migration.

[0160] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0161] Figure 10 This is a structural diagram of a data migration device provided in an embodiment of the present application. Figure 10 As shown, an embodiment of the present application further provides a data migration device 100, which includes a receiving module 101, a generating module 102, a determining module 103 and a migration module 103, wherein:

[0162] The receiving module 101 is configured to receive a data migration request; the data migration request is used to instruct to migrate at least one data from a source storage device to a target storage device;

[0163] A generating module 102 is configured to generate a first storage area based on a root storage path of at least one data item; wherein the first storage area is an area of ​​the source storage device mapped from a second storage area in the target storage device, the data stored in the first storage area is the same as the data stored in the second storage area, and the root storage path of the second storage area is the same as the root storage path of the at least one data item;

[0164] A determination module 103 is configured to determine at least one target data from at least one data according to the data stored in the first storage area;

[0165] The migration module 104 is configured to migrate at least one target data from a source storage device to a target storage device.

[0166] In a possible implementation, the generation module 102 is specifically configured to:

[0167] Create a virtual storage space in the source storage device;

[0168] Sending a first request message to the target storage device according to a network file system protocol, the first request message being used to request determination of a second storage area in the target storage device, the network file system protocol being configured in user mode;

[0169] receiving a first response message sent by the target storage device according to a network file system protocol, wherein the first response message includes data stored in the second storage area;

[0170] The data stored in the second storage area is mounted to the virtual storage space according to the network file system protocol to obtain the first storage area.

[0171] In a possible implementation, the determining module 103 is specifically configured to:

[0172] For any first data in the at least one data, perform the following operations:

[0173] determining, based on a first storage path of the first data in the source storage device, whether second data is stored in the first storage area, wherein a second storage path of the second data in the target storage device is the same as the first storage path;

[0174] In a case where the second data is stored in the first storage area, determining whether the first data belongs to the target data based on the first data and the second data;

[0175] When the second data is not stored in the first storage area, the first data is determined as the target data.

[0176] In a possible implementation, the determining module 103 is specifically configured to:

[0177] determining whether the attribute information of the first data is the same as the attribute information of the second data;

[0178] determining whether the storage information of the first data is the same as the storage information of the second data;

[0179] When the attribute information of the first data is the same as the attribute information of the second data, and the storage information of the first data is the same as the storage information of the second data, determining that the first data does not belong to the target data;

[0180] When the attribute information of the first data is different from the attribute information of the second data, and / or the storage information of the first data is different from the storage information of the second data, it is determined that the first data belongs to the target data.

[0181] In a possible implementation, the determining module 103 is specifically configured to:

[0182] Determine a first storage block of the first data in the source storage device and a second storage block of the second folder in the target storage device;

[0183] Slice the first storage block to obtain M first slicing blocks, where M is a positive integer.

[0184] Slice the second storage block to obtain M second slicings;

[0185] Determine whether storage information of the first data is the same as storage information of the second data according to the hash values ​​of the M first shards and the hash values ​​of the M second shards.

[0186] In a possible implementation, the migration module 104 is specifically configured to:

[0187] For each target data in the at least one target data, perform the following operations:

[0188] Obtaining the write latency and data access frequency of the source storage device and the write latency and data access frequency of the target storage device;

[0189] determining a first read / write efficiency of the source storage device according to a write latency and a data access frequency of the source storage device;

[0190] determining a second read / write efficiency of the target storage device according to a write latency and a data access frequency of the target storage device;

[0191] Determining a writing mode of the target data according to the first read / write efficiency and the second read / write efficiency; the writing mode includes a writing mode by a source storage device and a writing mode by a target storage device;

[0192] Get the maximum load and current load of the source storage device;

[0193] Determine the resource quota of the target data based on the difference between the maximum load and the current load;

[0194] Write the target data to the target storage device according to the writing method and resource quota;

[0195] The formula for determining resource quota is Among them, R represents resource quota, e(t) represents difference value, K p is the first tuning parameter, K i is the second tuning parameter, K d is the third tuning parameter, and t represents the current time.

[0196] In a possible implementation, the data migration apparatus 100 further includes a processing module, which is specifically configured to:

[0197] For each target data, determine the migration status of the target data within a preset time interval; the migration status includes completed status, uncompleted status and abnormal status;

[0198] If the migration status is abnormal, data migration of the target data is terminated, and a reason why the migration status is abnormal is determined.

[0199] For descriptions of features in the embodiment corresponding to the data migration apparatus 100 , reference may be made to the relevant descriptions of the embodiment corresponding to the data migration method, which will not be detailed here.

[0200] Figure 11 This is a schematic diagram of the structure of the electronic device provided in this application. Figure 11As shown, the electronic device 110 provided in this embodiment includes: at least one processor 111 and a memory 112. Optionally, the electronic device 110 further includes a communication component 113. The processor 111, the memory 112 and the communication component 113 are connected via a bus.

[0201] During the specific implementation process, at least one processor 111 executes the computer-executable instructions stored in the memory 112 , so that the at least one processor 111 executes the above-mentioned data migration method embodiment.

[0202] The specific implementation process of the processor 111 can be found in the above-mentioned method embodiment. Its implementation principle and technical effects are similar, and will not be repeated here in this embodiment.

[0203] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the application may be directly implemented by a hardware processor or implemented by a combination of hardware and software modules in the processor.

[0204] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (NVM), such as at least one disk memory.

[0205] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Buses can be classified into address buses, data buses, and control buses. For ease of illustration, the buses in the drawings of this application are not limited to just one bus or just one type of bus.

[0206] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. The computer program is configured to execute the steps of any of the above-mentioned data migration method embodiments when run.

[0207] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0208] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any of the above-mentioned data migration method embodiments are implemented.

[0209] An embodiment of the present application also provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above-mentioned data migration method embodiments are implemented.

[0210] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0211] The above is a detailed introduction to a data migration method, device, electronic device and storage medium provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A data migration method, characterized in that: The method comprises: Receive a data migration request; the data migration request is used to instruct to migrate at least one data from a source storage device to a target storage device; generating a first storage area according to the root storage path of the at least one data; wherein the first storage area is an area of ​​the source storage device mapped to the second storage area in the target storage device, the data stored in the first storage area is the same as the data stored in the second storage area, and the root storage path of the second storage area is the same as the root storage path of the at least one data; determining at least one target data from the at least one data according to the data stored in the first storage area; The at least one target data is migrated from the source storage device to the target storage device.

2. The method according to claim 1, characterized in that Generating a first storage area according to the root storage path of the at least one data includes: Creating a virtual storage space in the source storage device; sending a first request message to the target storage device according to a network file system protocol, wherein the first request message is used to request determination of the second storage area in the target storage device; receiving a first response message sent by the target storage device according to the network file system protocol, wherein the first response message includes data stored in the second storage area; The data stored in the second storage area is mounted to the virtual storage space according to the network file system protocol to obtain the first storage area.

3. The method according to claim 2, characterized in that The determining, based on the data stored in the first storage area, at least one target data from the at least one data, includes: For any first data among the at least one data, perform the following operations: determining, based on a first storage path of the first data in the source storage device, whether second data is stored in the first storage area, wherein a second storage path of the second data in the target storage device is the same as the first storage path; In a case where the second data is stored in the first storage area, determining whether the first data belongs to target data based on the first data and the second data; If the second data is not stored in the first storage area, the first data is determined as target data.

4. The method according to claim 3, characterized in that The determining, based on the first data and the second data, whether the first data belongs to target data includes: determining whether the attribute information of the first data is the same as the attribute information of the second data; determining whether the storage information of the first data is the same as the storage information of the second data; When the attribute information of the first data is the same as the attribute information of the second data, and the storage information of the first data is the same as the storage information of the second data, determining that the first data does not belong to the target data; When the attribute information of the first data is different from the attribute information of the second data, and / or the storage information of the first data is different from the storage information of the second data, it is determined that the first data belongs to target data.

5. The method according to claim 4, characterized in that The determining whether the storage information of the first data is the same as the storage information of the second data includes: Determine a first storage block of the first data in the source storage device and a second storage block of the second folder in the target storage device; Slice the first storage block to obtain M first slices, where M is a positive integer; Slice the second storage block to obtain M second slices; Determine whether the storage information of the first data is the same as the storage information of the second data according to the hash values ​​of the M first shards and the hash values ​​of the M second shards.

6. The method according to any one of claims 1 to 5, characterized in that Migrating the at least one target data from the source storage device to the target storage device includes: For each target data in the at least one target data, perform the following operations: Obtaining the write latency and data access frequency of the source storage device and the write latency and data access frequency of the target storage device; determining a first read / write efficiency of the source storage device according to a write latency and a data access frequency of the source storage device; determining a second read / write efficiency of the target storage device according to a write latency and a data access frequency of the target storage device; Determining a writing method of the target data according to the first read / write efficiency and the second read / write efficiency; the writing method includes a writing method by a source storage device or a writing method by a target storage device; Obtaining the maximum load and current load of the source storage device; determining a resource quota for the target data according to a difference between the maximum load and the current load; Writing the target data into the target storage device according to the writing mode and the resource quota; The resource quota satisfies: Where R represents the resource quota, e(t) represents the difference value, and K p is the first tuning parameter, K i is the second tuning parameter, K d is the third tuning parameter, and t represents the current time.

7. The method according to any one of claims 1 to 5, characterized in that The method further comprises: For each target data, determining the migration status of the target data within a preset time interval; the migration status includes a completed state, an uncompleted state, and an abnormal state; If the migration status is an abnormal state, the migration of the target data to the target storage device is stopped.

8. A data migration device, characterized in that: The device comprises: A receiving module, configured to receive a data migration request; the data migration request is used to instruct to migrate at least one data from a source storage device to a target storage device; a generating module, configured to generate a first storage area based on a root storage path of the at least one data; wherein the first storage area is an area of ​​the source storage device mapped to the second storage area in the target storage device, the data stored in the first storage area is the same as the data stored in the second storage area, and the root storage path of the second storage area is the same as the root storage path of the at least one data; a determination module, configured to determine at least one target data in the at least one data according to the data stored in the first storage area; A migration module is configured to migrate the at least one target data from the source storage device to the target storage device.

9. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the data migration method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the data migration method according to any one of claims 1 to 7.

Images