Methods, apparatus, electronic devices and storage media for data migration from storage pools

Abstract

This invention provides a method, apparatus, electronic device, and storage medium for data migration from a storage pool. The storage pool includes a metadata server and an object storage server. The metadata server stores object metadata, including the storage location of the object data. The object storage server stores the object data. The method involves deleting versions of the target object's metadata from the first metadata server, determining the versions to be deleted based on deletion indicators in the target object's metadata from the second metadata server, and then migrating the remaining target object metadata to the second object storage server. Additionally, the method involves updating the storage location in the remaining target object's metadata and writing it in batches to the second metadata server. This achieves the full migration of data from the first storage pool to the second storage pool, avoiding data consistency issues and improving user experience during the full data migration process.

Description

Technical Field

[0001] This invention relates to the field of data migration technology, and in particular to a method, apparatus, electronic device, and storage medium for data migration of a storage pool. Background Technology

[0002] When handling massive data storage, multiple storage pools are typically built, and new pools are flexibly added as data volume grows. Due to differences in construction time, equipment configuration, and service life among different storage pools, significant imbalances exist in overall capacity, remaining available capacity, equipment failure rate, and data read / write performance. Specifically, older storage pools tend to have higher failure rates and potentially lower data read / write performance due to prolonged operation. To ensure data storage stability and efficiency, these older storage pools must be updated or decommissioned. Updating some older equipment requires data reconstruction within the storage pool to ensure data integrity and continuity during the update process. Decommissioning an entire older storage pool necessitates the safe and efficient migration of all data to other storage pools to prevent data loss or service interruption.

[0003] Since the data volume of a single storage pool can reach the petabyte level, a full migration of the entire storage pool will face many challenges, such as data consistency issues during the migration process, which will affect the user experience during the data migration process. Summary of the Invention

[0004] In view of the above problems, embodiments of the present invention are proposed to provide a method, apparatus, electronic device and storage medium for data migration of a storage pool to overcome or at least partially solve the above problems.

[0005] To address the aforementioned problems, this invention discloses a method for data migration in a storage pool. The storage pool includes a metadata server and an object storage server. The metadata server stores object metadata; each object has at least one version of its metadata, with the version number identified by a timestamp indicating the creation time of the version metadata. The metadata includes the storage location of the object's data. The object storage server stores the object's data. The method includes:

[0006] The metadata of the target object is queried from the first metadata server of the first storage pool and the second metadata server of the second storage pool, respectively, to obtain the metadata of at least one first version and at least one second version.

[0007] Determine if a version with a deletion identifier exists in the metadata of the second at least one version;

[0008] From the metadata of the first at least one version, remove the metadata of the second at least one version that contains a deletion flag to obtain candidate metadata of at least one version;

[0009] Based on the candidate metadata of at least one version, the data of each version is migrated from the first object storage server of the first storage pool to the second object storage server of the second storage pool, and the target storage location of the data of each version is obtained.

[0010] Replace the storage location of each version of the candidate metadata with the target storage location of each version of the data to obtain the target metadata of at least one version.

[0011] The target metadata of at least one version is written in batches to the second metadata server.

[0012] Optionally, the method further includes:

[0013] If the metadata of the second at least one version does not exist, the target metadata of the at least one version is written to the second metadata server in batches through atomic operations.

[0014] Optionally, migrating the data of each version from the first object storage server to the second object storage server based on the candidate metadata of the at least one version includes:

[0015] Based on the candidate metadata of at least one version, the storage location of the data for each version is obtained;

[0016] Based on the storage location of the data for each version, the data for each version is migrated from the first object storage server to the second object storage server.

[0017] Optionally, the first storage pool further includes a first interface server, and the method further includes:

[0018] Upon receiving a write request for the target object, the first interface server queries the metadata of the target object from the second metadata server.

[0019] If the target object's metadata exists in the second metadata server, then the metadata of the current version of the target object is written to the second metadata server;

[0020] If the target object's metadata does not exist in the second metadata server, then the target object's metadata is queried from the first metadata server;

[0021] If the target object's metadata does not exist in the first metadata server, then the metadata of the current version of the target object is written to the second metadata server;

[0022] If the target object's metadata exists in the first metadata server, then the target object's current version metadata and the target object's metadata existing in the first metadata server are written to the second metadata server in batches through atomic operations.

[0023] Optionally, the method further includes:

[0024] Upon receiving a deletion request for a target version of the target object, the first interface server queries the metadata of the target object from the second metadata server.

[0025] If the target version of the target object exists in the second metadata server, then add a deletion flag to the target version of the target object's metadata;

[0026] If the target object's metadata does not exist in the second metadata server, then the target object's metadata is queried from the first metadata server;

[0027] If the target version of the target object exists in the first metadata server, then the target version of the target object's metadata is removed from the target object's metadata in the first metadata server to obtain the removed metadata;

[0028] If the removed metadata is not empty, then the removed metadata is written to the second metadata server through an atomic operation;

[0029] If the removed metadata is empty, then based on the metadata of the target version of the target object, a first version of the metadata of the target object is created, and the first version of the metadata contains a deletion identifier;

[0030] The metadata of the first version is written to the second metadata server through atomic operations.

[0031] Optionally, the method further includes:

[0032] Upon receiving a read request for the target version of the target object, the first interface server queries the metadata of the target version of the target object from the second metadata server.

[0033] If the target version of the target object exists in the second metadata server, and there is no deletion identifier in the target version of the target object's metadata, then the data of the target version of the target object is read according to the storage location in the target version of the target object's metadata;

[0034] If the target version of the target object does not exist in the second metadata server, or if the target version of the target object has a deletion identifier in its metadata, then the target version of the target object is queried from the first metadata server through the first interface server.

[0035] If the target version of the target object exists in the first metadata server, then the target version of the target object is read from the first object storage server according to the storage location in the target version of the target object's metadata.

[0036] Optionally, the method further includes:

[0037] After all the data of all objects in the first storage pool has been migrated, the second metadata server is traversed to determine the versions of metadata that have deletion marks.

[0038] Delete versions of metadata that have deletion flags.

[0039] Accordingly, this invention discloses an apparatus for data migration of a storage pool, wherein the storage pool includes a metadata server and an object storage server; the metadata server is used to store the metadata of objects, an object has at least one version of metadata, the version number of the version is identified by a timestamp of the creation time of the version metadata, and the metadata includes the storage location of the object's data; the object storage server is used to store the data of the object; the apparatus includes:

[0040] The metadata query module is used to query the metadata of the target object from the first metadata server of the first storage pool and the second metadata server of the second storage pool, respectively, to obtain the metadata of at least one first version and at least one second version.

[0041] The deletion identifier version determination module is used to determine the version in the metadata of the second at least one version that contains a deletion identifier;

[0042] The elimination module is used to remove the metadata of the second at least one version from the metadata of the first at least one version, and obtain candidate metadata of at least one version;

[0043] The migration module is used to migrate the data of each version from the first object storage server of the first storage pool to the second object storage server of the second storage pool based on the candidate metadata of the at least one version, and to obtain the target storage location of the data of each version.

[0044] The target metadata determination module is used to replace the storage location of each version of the candidate metadata of at least one version with the target storage location of the data of each version, so as to obtain the target metadata of at least one version.

[0045] The target metadata writing module is used to write the target metadata of at least one version to the second metadata server in batches.

[0046] Optionally, the device further includes:

[0047] The first atomic operation batch write module is used to write the target metadata of the at least one version to the second metadata server in batches through atomic operations when the metadata of the second at least one version does not exist.

[0048] Optionally, the migration module includes:

[0049] The storage location determination submodule is used to determine the storage location of each version of the data based on the candidate metadata of the at least one version;

[0050] The migration submodule is used to migrate the data of each version from the first object storage server to the second object storage server according to the storage location of the data of each version.

[0051] Optionally, the first storage pool further includes a first interface server, and the device further includes:

[0052] The first query module is used to query the metadata of the target object from the second metadata server through the first interface server when a write request for the target object is received.

[0053] The first writing module is configured to write the metadata of the current version of the target object into the second metadata server if the target object's metadata exists in the second metadata server.

[0054] The second query module is used to query the metadata of the target object from the first metadata server if the metadata of the target object does not exist in the second metadata server.

[0055] The second writing module is used to write the metadata of the current version of the target object into the second metadata server if the metadata of the target object does not exist in the first metadata server.

[0056] The second atomic operation batch write module is used to write the current version metadata of the target object and the metadata of the target object existing in the first metadata server to the second metadata server in batches through atomic operations if the target object's metadata exists in the first metadata server.

[0057] Optionally, the device further includes:

[0058] The third query module is used to query the metadata of the target object from the second metadata server through the first interface server when a deletion request for the target version of the target object is received.

[0059] The deletion identifier addition module is used to add a deletion identifier to the metadata of the target version of the target object if the metadata of the target version of the target object exists in the second metadata server.

[0060] The fourth query module is used to query the metadata of the target object from the first metadata server if the metadata of the target object does not exist in the second metadata server.

[0061] The metadata removal determination module is used to remove the metadata of the target version of the target object from the metadata of the target object in the first metadata server if the metadata of the target version of the target object exists in the first metadata server, so as to obtain the metadata after removal;

[0062] The third atomic operation batch write module is used to write the removed metadata to the second metadata server through atomic operations if the removed metadata is not empty.

[0063] The metadata creation module is used to create a first version of the metadata of the target object based on the metadata of the target version of the target object if the removed metadata is empty. The first version of the metadata contains a deletion identifier.

[0064] The fourth atomic operation batch write module is used to write the metadata of the first version to the second metadata server through atomic operations.

[0065] Optionally, the device further includes:

[0066] The fifth query module is used to query the metadata of the target version of the target object from the second metadata server through the first interface server when a read request for the target version of the target object is received.

[0067] The first reading module is used to read the data of the target version of the target object according to the storage location in the metadata of the target version of the target object if the metadata of the target version of the target object exists in the second metadata server and there is no deletion identifier in the metadata of the target version of the target object;

[0068] The sixth query module is used to query the target version metadata of the target object from the first metadata server through the first interface server if the target version metadata of the target object does not exist in the second metadata server, or if the target version metadata of the target object contains a deletion identifier.

[0069] The second reading module is used to read the target version data of the target object from the first object storage server according to the storage location in the target version metadata of the target object if the target version metadata of the target object exists in the first metadata server.

[0070] Optionally, the device further includes:

[0071] The traversal module is used to traverse the second metadata server after all the data of all objects in the first storage pool has been migrated, to determine the version of metadata that has a deletion mark;

[0072] The "Tag Version Deletion" module is used to delete versions whose metadata contains deletion tags.

[0073] Accordingly, embodiments of the present invention disclose an electronic device, including: a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, it implements the various steps of the above-described data migration method embodiments for the storage pool.

[0074] Accordingly, embodiments of the present invention disclose a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the various steps of the data migration method embodiments of the storage pool described above.

[0075] The embodiments of the present invention have the following advantages:

[0076] This invention discloses a data migration method for a storage pool, comprising a metadata server and an object storage server. The metadata server stores the metadata of objects, where each object has at least one version of metadata. The version number is identified by a timestamp indicating the creation time of the version metadata, and the metadata includes the storage location of the object's data. The object storage server stores the object's data. First, it queries the metadata of the target object from the first metadata server of a first storage pool and the second metadata server of a second storage pool to obtain the metadata of at least one first version and at least one second version. Then, it determines the version with a deletion flag in the metadata of the at least one second version. Next, it removes the metadata of the version with the deletion flag from the metadata of the at least one first version, obtaining candidate metadata for at least one version. Then, based on the candidate metadata for at least one version, it migrates the data of each version from the first object storage server of the first storage pool to the second object storage server of the second storage pool, obtaining the target storage location of the data for each version. It replaces the storage locations of the data for each version in the candidate metadata for at least one version with the corresponding target storage locations of the data for each version, obtaining the target metadata for at least one version. Finally, it writes the target metadata for at least one version in batches to the second metadata server. This invention deletes the versions of the target object's metadata that need to be deleted from the first metadata server. It then determines the versions to be deleted by identifying the deletion representations in the target object's metadata from the second metadata server. Finally, it migrates the data corresponding to the remaining target object's metadata to the second object storage server and updates the storage location in the remaining target object's metadata before writing it in batches to the second metadata server. This achieves the migration of all data from the first storage pool to the second storage pool, avoiding data consistency issues and improving the user experience during the full data migration process. Attached Figure Description

[0077] Figure 1 This is a flowchart illustrating the steps of a data migration method for a storage pool according to an embodiment of the present invention;

[0078] Figure 2 This is a scenario diagram of data migration in a storage pool according to an embodiment of the present invention;

[0079] Figure 3 This is a structural block diagram of a data migration device for a storage pool according to an embodiment of the present invention. Detailed Implementation

[0080] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0081] Since the data volume of a single storage pool can reach the petabyte level, a full migration of the entire storage pool will face many challenges, such as data consistency issues during the migration process, which will affect the user experience during the data migration process.

[0082] One of the core concepts of this invention is that it deletes the version of the target object's metadata that needs to be deleted from the first metadata server, determines the version that needs to be deleted by the deletion indication in the target object's metadata in the second metadata server, then migrates the data corresponding to the metadata of the remaining target objects to the second object storage server, and updates the storage location in the metadata of the remaining target objects and writes it in batches to the second metadata server. This achieves the migration of all data from the first storage pool to the second storage pool, avoids data consistency issues, and improves the user experience during the full data migration process.

[0083] Reference Figure 1 This diagram illustrates a flowchart of a data migration method for a storage pool according to an embodiment of the present invention. The storage pool includes a metadata server and an object storage server. The metadata server stores the metadata of objects. An object has at least one version of metadata, and the version number is identified by a timestamp indicating the creation time of the version metadata. The metadata includes the storage location of the object's data. The object storage server stores the object's data and may specifically include the following steps:

[0084] Step 101: Query the metadata of the target object from the first metadata server of the first storage pool and the second metadata server of the second storage pool respectively to obtain the metadata of at least one version of the first object and at least one version of the second object.

[0085] Specifically, the first storage pool can be the source storage pool, and the second storage pool can be the destination storage pool. All data in the first storage pool needs to be migrated to the second storage pool. The metadata server is used to store the metadata of objects, exemplarily in key-value pairs.<k,v> This stores the metadata of an object, where k is the object's identifier and v is the content of the metadata. Different versions of an object share the same k, but each version has a different v. Each version's version number is identified by its creation timestamp (ts).<k,ts,v> For example, the first metadata server contains metadata for object a, and a has three versions.<a,t1,v1> ,<a,t2,v2> ,<a,t3,v3> The metadata for object 'a' also exists in the second metadata server; 'a' has three versions.<a,t1,v1> ,<a,t2,v2^> ,<a,t3,v3> The metadata of target object 'a' is queried from the first metadata server of the first storage pool and the second metadata server of the second storage pool, respectively, to obtain the metadata of at least one version of 'a', that is, all versions of 'a' on the first metadata server.<a,t1,v1> ,<a,t2,v2> ,<a,t3,v3>}, and at least one second version of metadata, i.e., a in all versions of the second metadata server {<a,t1,v1> ,<a,t2,v2^> ,<a,t3,v3>}

[0086] Step 102: Determine the version whose metadata contains a deletion identifier.

[0087] Specifically, it determines the version whose metadata contains a deletion identifier, for example, the ^ symbol in v2^ is the deletion identifier, i.e.<a,t2,v2^> This refers to versions with a deletion flag, meaning that version number t2 is a version whose metadata contains a deletion flag at least in the second or first version.

[0088] Step 103: Remove the metadata of the version with a deletion flag from the metadata of the first at least one version, and obtain candidate metadata of at least one version.

[0089] Specifically, the metadata of the first at least one version is {<a,t1,v1> ,<a,t2,v2> ,<a,t3,v3>}, where version number t2 is a version whose metadata contains a deletion flag in at least the second version. After removing the metadata of version number t2 from the metadata of at least the first version, we get {<a,t1,v1> ,<a,t3,v3>}, that is, at least one version of candidate metadata.

[0090] Step 104: Based on the candidate metadata of at least one version, migrate the data of each version from the first object storage server of the first storage pool to the second object storage server of the second storage pool, and obtain the target storage location of the data of each version.

[0091] Specifically, based on at least one version of candidate metadata {<a,t1,v1> ,<a,t3,v3> The data of target object a, versions t1 and t3, are migrated from the first object storage server of the first storage pool to the second object storage server of the second storage pool. After migration to the second object storage server of the second storage pool, the target storage locations of the data of versions t1 and t3 are obtained on the second object storage server.

[0092] Step 105: Replace the storage location of each version of the candidate metadata in the at least one version with the target storage location of the data of each version to obtain the target metadata of at least one version.

[0093] Specifically, at least one version of the candidate metadata {<a,t1,v1> ,<a,t3,v3> The storage location in v1 is replaced with the data of version t1 in the target storage location of the second object storage server, and the storage location in v3 is replaced with the data of version t3 in the target storage location of the second object storage server, thus obtaining at least one version of the target metadata.<a,t1,v1'> ,<a,t3,v3'>}

[0094] Step 106: Write the target metadata of at least one version into the second metadata server in batches.

[0095] Specifically, at least one version of the target metadata is obtained.<a,t1,v1'> ,<a,t3,v3'>},Will<a,t1,v1'> and<a,t3,v3'> Batch write to the second metadata server.

[0096] In this embodiment of the invention, the method further includes:

[0097] If the metadata of the second at least one version does not exist, the target metadata of the at least one version is written to the second metadata server in batches through atomic operations.

[0098] Specifically, in the absence of metadata for the second at least one version, for example, metadata for object a exists in the first metadata server, and a has three versions.<a,t1,v1> ,<a,t2,v2> ,<a,t3,v3> The second metadata server does not contain metadata for object 'a', and at least one version of the candidate metadata is {<a,t1,v1> ,<a,t2,v2> ,<a,t3,v3> After migrating data for versions t1, t2, and t3 from the first object storage server to the second object storage server, the target storage locations of the data for versions t1, t2, and t3 on the second object storage server are obtained respectively. The storage locations in v1, v2, and v3 are then replaced with the target storage locations of the data for versions t1, t2, and t3 on the second object storage server, resulting in target metadata for at least one version.<a,t1,v1'> ,<a,t2,v2'> ,<a,t3,v3'>},Will{<a,t1,v1'> ,<a,t2,v2'> ,<a,t3,v3'> The process involves batch writing to the second metadata server using atomic operations. If the write fails, the process repeats from the first metadata server to the second metadata server, starting with the step of querying the target object's metadata from both servers, until success is achieved.

[0099] By using atomic operations to batch write to the second metadata server, the atomicity of metadata update operations can be ensured during data migration, thereby guaranteeing data consistency and integrity. Atomic operations mean that these operations either all succeed or none fail, preventing situations where some operations succeed while others fail.

[0100] In this embodiment of the invention, step 104 may specifically include the following sub-steps:

[0101] Sub-step S11: Based on the candidate metadata of the at least one version, obtain the storage location of the data for each version.

[0102] Specifically, continuing the previous example, based on at least one version of candidate metadata {<a,t1,v1> ,<a,t3,v3> In the}, v1 and v3, we obtain the storage location of the data of version t1 in the first object storage server, and the storage location of the data of version t3 in the first object storage server.

[0103] Sub-step S12: Based on the storage location of the data for each version, migrate the data for each version from the first object storage server to the second object storage server.

[0104] Specifically, based on the storage location of the t1 version data in the first object storage server, the t1 version data is migrated from the first object storage server to the second object storage server, and based on the storage location of the t3 version data in the first object storage server, the t3 version data is migrated from the first object storage server to the second object storage server.

[0105] In this embodiment of the invention, the first storage pool further includes a first interface server, and the method further includes:

[0106] Upon receiving a write request for the target object, the first interface server queries the metadata of the target object from the second metadata server.

[0107] Specifically, read / write redirection occurs first. Before data migration is ready, i.e., before time T0, the first interface server only retrieves the object's metadata from the first metadata server and reads / writes the deleted object's data from the first object storage server. Once data migration begins, i.e., from time T0, objects in the first storage pool are read-only, and all newly written object data is written to the second storage pool. (Refer to...) Figure 2 The diagram illustrates a scenario of data migration in a storage pool according to an embodiment of the present invention. Starting from time T1, data migration begins using a migration tool.

[0108] Starting from time T0, a user can initiate a write request for a target object through the client. Upon receiving a write request for a target object, the first interface server queries the target object's metadata from the second metadata server. For example, the first metadata server contains metadata for object a and object b, where a has three versions.<a,t1,v1> ,<a,t2,v2> ,<a,t3,v3> b has only one version<b,t0,v0> Neither of these two objects exists on the second metadata server. Upon receiving a write request for target object k, the system queries the second metadata server for a list of all versions of metadata corresponding to target object k through the first interface server.<k,ts,v> .

[0109] If the target object's metadata exists in the second metadata server, then the metadata of the current version of the target object is written to the second metadata server.

[0110] Specifically, if the second metadata server contains metadata for the target object—that is, if the List is not empty and contains metadata for other versions of the target object k—then the data for the target object k in the write request for the target object k is written to the second object storage server. After the write, corresponding metadata will be generated.<k,t,v> Then, the metadata of the target object k<k,t,v> Write a second metadata server.

[0111] If the target object's metadata is not present in the second metadata server, then the target object's metadata is queried from the first metadata server.

[0112] Specifically, if the second metadata server contains the metadata for the target object, meaning List is empty, then the first metadata server is consulted to retrieve all versions of List' corresponding to the target object k.<k,ts,v> .

[0113] If the target object's metadata does not exist in the first metadata server, then the metadata of the current version of the target object is written to the second metadata server.

[0114] Specifically, if the target object's metadata does not exist in the first metadata server (i.e., List' is empty), then the target object k has no metadata in either the first or second metadata server. The data for target object k in the write request is then written to the second object storage server, generating corresponding metadata after the write.<k,t,v> Then, the metadata of the target object k<k,t,v> Write a second metadata server.

[0115] If the target object's metadata exists in the first metadata server, then the target object's current version metadata and the target object's metadata existing in the first metadata server are written to the second metadata server in batches through atomic operations.

[0116] Specifically, if the target object's metadata exists in the first metadata server (meaning List' is not empty), then the target object k has metadata in the first metadata server but not in the second metadata server. The current version of the target object k's metadata will then be retrieved.<k,t,v> Add to List' to get List', then write List' to the second metadata server in batches using atomic operations. Continuing the previous example, if k == a, then retrieve the metadata List' corresponding to a from the first metadata server.<a,t1,v1> ,<a,t2,v2> ,<a,t3,v3>}, retrieve the metadata of the current version of the target object k.<k,t,v> Add to List', get List"{<a,t1,v1> ,<a,t2,v2> ,<a,t3,v3> ,<k,t,v>}, the List″ is written to the second metadata server in batches using atomic operations (if List is empty, then List″ is written in batches). If the write fails, the step of retrieving the target object's metadata from the second metadata server is retried until successful. For example, if List″{<a,t1,v1> ,<a,t2,v2> ,<a,t3,v3> ,<k,t,v> If writing to the second metadata server by atomic operation fails, then...<k,t,v> When writing to the second metadata server, it's possible that the atomic operation might fail because two operations simultaneously target the target object's metadata. In this case, retrying will reveal that the target object's metadata no longer exists on the first metadata server. Therefore, the metadata will be directly...<k,t,v> Write it to the second metadata server.

[0117] In this embodiment of the invention, the method further includes:

[0118] Upon receiving a deletion request for a target version of the target object, the first interface server queries the metadata of the target object from the second metadata server.

[0119] Specifically, upon receiving the target version for target object k<k,ts> When a deletion request is made, the metadata list of all versions corresponding to k is retrieved from the second metadata server.<k,ts,v> .

[0120] If the target version of the target object exists in the second metadata server, then a deletion flag is added to the target version of the target object's metadata.

[0121] Specifically, if the List is not empty, but does not exist...<k,ts> If the list is not empty, return directly; otherwise, if the list contains an empty list, return the contents of the list.<k,ts> In the second metadata server<k,ts> Add a special column DELETE to the metadata value v, which serves as a deletion identifier, resulting in v', to indicate that this version has been deleted.<k,ts,v^> .

[0122] If the target object's metadata is not present in the second metadata server, then the target object's metadata is queried from the first metadata server.

[0123] Specifically, if List is empty, then it is necessary to query all versions of List' corresponding to the target object k from the first metadata server.<k,ts,v> .

[0124] If the target version metadata of the target object exists in the first metadata server, then the target version metadata of the target object is removed from the target object's metadata in the first metadata server to obtain the removed metadata.

[0125] Specifically, if List' contains<k,ts> At that time, then<k,ts> Remove from List' to get List''<k,ts,v> Continuing with the previous example, taking k == a and t == t2 as an example, retrieve the metadata List'{ corresponding to a from the first metadata server.<a,t1,v1> ,<a,t2,v2> ,<a,t3,v3>}, Remove the metadata of version t2 to get List″{<a,t1,v1> ,<a,t3,v3>}

[0126] If the removed metadata is not empty, then the removed metadata is written to the second metadata server through an atomic operation.

[0127] Specifically, List″{<a,t1,v1> ,<a,t3,v3> The process involves atomically writing to the second metadata server (if List is empty, write to List" in batches). If the write is successful, the data stored in the metadata v2 of version t2 is deleted from the first object storage server. If the write fails, the process retryes starting from the step of querying the target object's metadata from the second metadata server until successful. In practice, it's possible that the atomic write operation might fail because two operations simultaneously target the target object's metadata. In this case, retrying simply requires adding a special column `DELETE` to v2, resulting in v2^.<a,t2,v2^> Write the data to the second metadata server, and then delete the corresponding data from the second object storage server.

[0128] If the removed metadata is empty, then based on the metadata of the target version of the target object, a first version of the metadata of the target object is created, and the first version of the metadata contains a deletion identifier.

[0129] Specifically, if the removed metadata is empty, that is, List″ is empty, meaning List′ only contains<k,ts> In removal<k,ts> Afterwards, if "List" is empty, a special version needs to be constructed.<k,ts',v^> Added to the List, this special version indicates that all versions of the object have been deleted.

[0130] The metadata of the first version is written to the second metadata server through atomic operations.

[0131] Specifically, the List is written to the second metadata server through atomic operations.

[0132] In this embodiment of the invention, the method further includes:

[0133] Upon receiving a read request for the target version of the target object, the first interface server queries the metadata of the target version of the target object from the second metadata server.

[0134] Specifically, upon receiving a read request for target version ts of target object k, the system first queries the second metadata server through the first interface server.<k,ts> Corresponding metadata<k,ts,v> .

[0135] If the target version of the target object exists in the second metadata server, and there is no deletion identifier in the target version of the target object's metadata, then the target version of the target object's data is read according to the storage location in the target version of the target object's metadata.

[0136] Specifically, if the second metadata server contains<k,ts,v> If v does not contain the special column DELETE, then data is read from either the first object storage server or the second object storage server based on the storage location of the record in v.

[0137] If the target version of the target object does not exist in the second metadata server, or if the target version of the target object has a deletion identifier in its metadata, then the target version of the target object is queried from the first metadata server through the first interface server.

[0138] Specifically, if the second metadata server does not exist<k,ts,v> Then, it is obtained from the first metadata server through the first interface server.<k,ts> Corresponding metadata<k,ts,v'> .

[0139] If the target version of the target object exists in the first metadata server, then the target version of the target object is read from the first object storage server according to the storage location in the target version of the target object's metadata.

[0140] Specifically, if the first metadata server contains<k,ts,v'> Then, data is read from the first object storage server.

[0141] In this embodiment of the invention, the method further includes:

[0142] After all the data of all objects in the first storage pool has been migrated, the second metadata server is traversed to determine the versions of metadata that have deletion marks.

[0143] Specifically, it iterates through the first metadata server to obtain a List of all versions of each object.<k,ts,v> Using the above method, the data of each object is migrated using a migration tool. At time T2, the migration of all object data is complete. The second storage pool also includes a second interface server. At time T3, all requests destined for the first interface server are redirected to the second interface server at the network layer. From time T3 onwards, the first interface server no longer receives any requests; all requests are processed by the second interface server according to normal logic. However, for read requests, if v contains the special column DELETE, it is assumed that version does not exist.

[0144] At time T3, the second metadata server is traversed using a cleanup tool to determine the versions of metadata marked for deletion.

[0145] Delete versions of metadata that have deletion flags.

[0146] Specifically, for all versions of v that contain the special column DELETE<k,ts,v^> The deletion is performed. At time T4, the deletion of all object versions containing the special column DELETE is completed. At time T5, all requests are processed by the various components in the second storage pool according to the general logic.

[0147] The migration process in this embodiment of the invention first performs read / write redirection on the first metadata server, then migrates the data, and finally cleans up special markers. The read / write redirection method is as follows: before migration is complete, requests to the first storage pool are still processed by the first metadata server; after migration, requests to the first storage pool are processed by the second metadata server. The method for ensuring data consistency during migration uses atomic operations and special columns to achieve metadata consistency processing. It has wide applicability, suitable for full data migration between most object storage pools, and is low-cost. The migration process does not affect service availability or data reliability, truly achieving the replacement of old storage pools without the user's awareness, eliminating the risks of long-term operation of old equipment. During the migration of object data from the source storage pool to the destination storage pool, users can read, write, and delete old objects from the source storage pool and read, write, and delete new objects from the destination resource pool. After migration, users do not need to make any modifications and can read, write, and delete any objects from the new storage pool.

[0148] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0149] Reference Figure 3 This diagram illustrates a structural block diagram of a data migration device for a storage pool according to an embodiment of the present invention. The storage pool includes a metadata server and an object storage server. The metadata server stores the metadata of objects. An object has at least one version of metadata, and the version number is identified by a timestamp indicating the creation time of the version metadata. The metadata includes the storage location of the object's data. The object storage server stores the object's data and may specifically include the following modules:

[0150] Metadata query module 201 is used to query the metadata of the target object from the first metadata server of the first storage pool and the second metadata server of the second storage pool respectively, and obtain the metadata of at least a first version and at least a second version of the metadata.

[0151] The deletion identifier version determination module 202 is used to determine the version in the metadata of the second at least one version that contains a deletion identifier;

[0152] The elimination module 203 is used to eliminate the metadata of the second at least one version from the metadata of the first at least one version, and remove the metadata of the version with the deletion identifier, so as to obtain candidate metadata of at least one version;

[0153] Migration module 204 is used to migrate data of each version from the first object storage server of the first storage pool to the second object storage server of the second storage pool according to the candidate metadata of at least one version, and to obtain the target storage location of the data of each version.

[0154] The target metadata determination module 205 is used to replace the storage location of each version of the candidate metadata of at least one version with the target storage location of the data of each version, so as to obtain the target metadata of at least one version.

[0155] The target metadata writing module 206 is used to write the target metadata of at least one version in batches to the second metadata server.

[0156] The apparatus of this invention deletes the version that needs to be deleted from the metadata of the target object in the first metadata server, determines the version that needs to be deleted by the deletion indication in the metadata of the target object in the second metadata server, then migrates the data corresponding to the metadata of the remaining target objects to the second object storage server, and updates the storage location in the metadata of the remaining target objects and writes it in batches to the second metadata server, thereby realizing the full migration of data from the first storage pool to the second storage pool, avoiding data consistency problems, and improving the user experience during the full data migration process.

[0157] In this embodiment of the invention, the device further includes:

[0158] The first atomic operation batch write module is used to write the target metadata of the at least one version to the second metadata server in batches through atomic operations when the metadata of the second at least one version does not exist.

[0159] In this embodiment of the invention, the migration module includes:

[0160] The storage location determination submodule is used to determine the storage location of each version of the data based on the candidate metadata of the at least one version;

[0161] The migration submodule is used to migrate the data of each version from the first object storage server to the second object storage server according to the storage location of the data of each version.

[0162] In this embodiment of the invention, the first storage pool further includes a first interface server, and the device further includes:

[0163] The first query module is used to query the metadata of the target object from the second metadata server through the first interface server when a write request for the target object is received.

[0164] The first writing module is configured to write the metadata of the current version of the target object into the second metadata server if the target object's metadata exists in the second metadata server.

[0165] The second query module is used to query the metadata of the target object from the first metadata server if the metadata of the target object does not exist in the second metadata server.

[0166] The second writing module is used to write the metadata of the current version of the target object into the second metadata server if the metadata of the target object does not exist in the first metadata server.

[0167] The second atomic operation batch write module is used to write the current version metadata of the target object and the metadata of the target object existing in the first metadata server to the second metadata server in batches through atomic operations if the target object's metadata exists in the first metadata server.

[0168] In this embodiment of the invention, the device further includes:

[0169] The third query module is used to query the metadata of the target object from the second metadata server through the first interface server when a deletion request for the target version of the target object is received.

[0170] The deletion identifier addition module is used to add a deletion identifier to the metadata of the target version of the target object if the metadata of the target version of the target object exists in the second metadata server.

[0171] The fourth query module is used to query the metadata of the target object from the first metadata server if the metadata of the target object does not exist in the second metadata server.

[0172] The metadata removal determination module is used to remove the metadata of the target version of the target object from the metadata of the target object in the first metadata server if the metadata of the target version of the target object exists in the first metadata server, so as to obtain the metadata after removal;

[0173] The third atomic operation batch write module is used to write the removed metadata to the second metadata server through atomic operations if the removed metadata is not empty.

[0174] The metadata creation module is used to create a first version of the metadata of the target object based on the metadata of the target version of the target object if the removed metadata is empty. The first version of the metadata contains a deletion identifier.

[0175] The fourth atomic operation batch write module is used to write the metadata of the first version to the second metadata server through atomic operations.

[0176] In this embodiment of the invention, the device further includes:

[0177] The fifth query module is used to query the metadata of the target version of the target object from the second metadata server through the first interface server when a read request for the target version of the target object is received.

[0178] The first reading module is used to read the data of the target version of the target object according to the storage location in the metadata of the target version of the target object if the metadata of the target version of the target object exists in the second metadata server and there is no deletion identifier in the metadata of the target version of the target object;

[0179] The sixth query module is used to query the target version metadata of the target object from the first metadata server through the first interface server if the target version metadata of the target object does not exist in the second metadata server, or if the target version metadata of the target object contains a deletion identifier.

[0180] The second reading module is used to read the target version data of the target object from the first object storage server according to the storage location in the target version metadata of the target object if the target version metadata of the target object exists in the first metadata server.

[0181] In this embodiment of the invention, the device further includes:

[0182] The traversal module is used to traverse the second metadata server after all the data of all objects in the first storage pool has been migrated, to determine the version of metadata that has a deletion mark;

[0183] The "Tag Version Deletion" module is used to delete versions whose metadata contains deletion tags.

[0184] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0185] This invention also provides an electronic device, comprising:

[0186] It includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, it implements the various processes of the above-described method embodiment for data migration of a storage pool and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0187] This invention also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described method embodiment for data migration of a storage pool and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0188] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0189] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0190] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0191] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0192] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0193] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.

[0194] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0195] The present invention has provided a detailed description of a data migration method, apparatus, electronic device, and storage medium for a storage pool. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, those skilled in the art will recognize that there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for data migration from a storage pool, characterized in that, The storage pool includes a metadata server and an object storage server; the metadata server is used to store the metadata of objects. An object has at least one version of metadata. The version number of the version is identified by the timestamp of the creation time of the version metadata. The metadata includes the storage location of the object's data. An object storage server is used to store the data of the object; the method includes: The metadata of the target object is queried from the first metadata server of the first storage pool and the second metadata server of the second storage pool, respectively, to obtain the metadata of at least one first version and at least one second version. Determine if a version with a deletion identifier exists in the metadata of the second at least one version; From the metadata of the first at least one version, remove the metadata of the second at least one version that contains a deletion flag to obtain candidate metadata of at least one version; Based on the candidate metadata of at least one version, the data of each version is migrated from the first object storage server of the first storage pool to the second object storage server of the second storage pool, and the target storage location of the data of each version is obtained. Replace the storage location of each version of the candidate metadata with the target storage location of each version of the data to obtain the target metadata of at least one version. The target metadata of at least one version is written in batches to the second metadata server.

2. The method according to claim 1, characterized in that, The method further includes: If the metadata of the second at least one version does not exist, the target metadata of the at least one version is written to the second metadata server in batches through atomic operations.

3. The method according to claim 1, characterized in that, The step of migrating data of each version from the first object storage server to the second object storage server based on the candidate metadata of at least one version includes: Based on the candidate metadata of at least one version, the storage location of the data for each version is obtained; Based on the storage location of the data for each version, the data for each version is migrated from the first object storage server to the second object storage server.

4. The method according to claim 1, characterized in that, The first storage pool further includes a first interface server, and the method further includes: Upon receiving a write request for the target object, the first interface server queries the metadata of the target object from the second metadata server. If the target object's metadata exists in the second metadata server, then the metadata of the current version of the target object is written to the second metadata server; If the target object's metadata does not exist in the second metadata server, then the target object's metadata is queried from the first metadata server; If the target object's metadata does not exist in the first metadata server, then the metadata of the current version of the target object is written to the second metadata server; If the target object's metadata exists in the first metadata server, then the target object's current version metadata and the target object's metadata existing in the first metadata server are written to the second metadata server in batches through atomic operations.

5. The method according to claim 4, characterized in that, The method further includes: Upon receiving a deletion request for a target version of the target object, the first interface server queries the metadata of the target object from the second metadata server. If the target version of the target object exists in the second metadata server, then add a deletion flag to the target version of the target object's metadata; If the target object's metadata does not exist in the second metadata server, then the target object's metadata is queried from the first metadata server; If the target version of the target object exists in the first metadata server, then the target version of the target object's metadata is removed from the target object's metadata in the first metadata server to obtain the removed metadata; If the removed metadata is not empty, then the removed metadata is written to the second metadata server through an atomic operation; If the removed metadata is empty, then based on the metadata of the target version of the target object, a first version of the metadata of the target object is created, and the first version of the metadata contains a deletion identifier; The metadata of the first version is written to the second metadata server through atomic operations.

6. The method according to claim 4, characterized in that, The method further includes: Upon receiving a read request for the target version of the target object, the first interface server queries the metadata of the target version of the target object from the second metadata server. If the target version of the target object exists in the second metadata server, and there is no deletion identifier in the target version of the target object's metadata, then the data of the target version of the target object is read according to the storage location in the target version of the target object's metadata; If the target version of the target object does not exist in the second metadata server, or if the target version of the target object has a deletion identifier in its metadata, then the target version of the target object is queried from the first metadata server through the first interface server. If the target version of the target object exists in the first metadata server, then the target version of the target object is read from the first object storage server according to the storage location in the target version of the target object's metadata.

7. The method according to claim 1, characterized in that, The method further includes: After all the data of all objects in the first storage pool has been migrated, the second metadata server is traversed to determine the versions of metadata that have deletion marks. Delete versions of metadata that have deletion flags.

8. An apparatus for data migration from a storage pool, characterized in that, The storage pool includes a metadata server and an object storage server; the metadata server is used to store the metadata of objects. An object has at least one version of metadata. The version number of the version is identified by the timestamp of the creation time of the version metadata. The metadata includes the storage location of the object's data. An object storage server is used to store data of the objects; the apparatus includes: The metadata query module is used to query the metadata of the target object from the first metadata server of the first storage pool and the second metadata server of the second storage pool, respectively, to obtain the metadata of at least one first version and at least one second version. The deletion identifier version determination module is used to determine the version in the metadata of the second at least one version that contains a deletion identifier; The elimination module is used to remove the metadata of the second at least one version from the metadata of the first at least one version, and obtain candidate metadata of at least one version; The migration module is used to migrate the data of each version from the first object storage server of the first storage pool to the second object storage server of the second storage pool based on the candidate metadata of the at least one version, and to obtain the target storage location of the data of each version. The target metadata determination module is used to replace the storage location of each version of the candidate metadata of at least one version with the target storage location of the data of each version, so as to obtain the target metadata of at least one version. The target metadata writing module is used to write the target metadata of at least one version to the second metadata server in batches.

9. An electronic device, characterized in that, include: A processor, a memory, and a computer program stored on the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the steps of the method for data migration of the storage pool as claimed in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the steps of the method for data migration of the storage pool as claimed in any one of claims 1 to 7.

Images