Metadata migration method and device, computer equipment and storage medium

Abstract

The invention relates to the technical field of distributed file systems, and discloses a metadata migration method and device, computer equipment and a storage medium, and the method comprises the steps: obtaining the load information of each metadata service, and determining a to-be-migrated metadata service based on the load information, the to-be-migrated metadata service comprising metadata of a plurality of sub-volumes; determining metadata of the to-be-migrated sub-volume based on the popularity information of the sub-volume in the to-be-migrated metadata service; determining a plurality of clients of tenants bound with the to-be-migrated sub-volumes based on the metadata of the to-be-migrated sub-volumes; to-be-recycled metadata matched with the to-be-migrated sub-volumes are screened out from the multiple clients; releasing the to-be-recycled metadata in the metadata of the to-be-migrated sub-volume to obtain target metadata of the to-be-migrated sub-volume; and migrating the target metadata of the to-be-migrated sub-volume to the target metadata service. According to the invention, the metadata migration time of the sub-volume and the waiting time for the client to access the sub-volume are reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of distributed file systems, and in particular to a metadata migration method, device, computer equipment and storage medium. Background Art

[0002] With the advent of the information age, distributed storage architecture has become crucial in emerging fields such as big data and cloud computing. Distributed file systems are designed to meet this demand. In distributed file systems, data is organized into files for storage and access, providing scalable, high-performance and distributed file storage services.

[0003] In the related art, there are many metadata services in the distributed file system to provide metadata processing capabilities for the distributed file system. However, there is often a large load difference between metadata services, and the load is unbalanced. At this time, it is necessary to migrate some metadata from the high-load metadata service to the low-load metadata service to solve the load imbalance problem. Among them, when performing metadata migration, the subvolume corresponding to the metadata needs to be locked.

[0004] However, in the related art, the method of performing metadata migration locks the sub-volume for too long, resulting in a long waiting time for the client to access the sub-volume and access timeout. Summary of the invention

[0005] In view of this, the present invention provides a metadata migration method, apparatus, computer device and storage medium to solve the problem that the metadata migration method in the related art locks the sub-volume for too long, resulting in a long waiting time for the client to access the sub-volume and access timeout.

[0006] In a first aspect, the present invention provides a metadata migration method, the method comprising:

[0007] Get the load information of each metadata service;

[0008] Determine, based on the load information, a metadata service to be migrated, wherein the metadata service to be migrated includes metadata of a plurality of subvolumes;

[0009] Determining metadata of the subvolume to be migrated based on the popularity information of the subvolume in the metadata service to be migrated;

[0010] Based on the metadata of the subvolume to be migrated, determining multiple clients of tenants bound to the subvolume to be migrated;

[0011] Filtering metadata to be recycled that matches the subvolume to be migrated from multiple clients;

[0012] Releasing the metadata to be recycled in the metadata of the subvolume to be migrated, and obtaining target metadata of the subvolume to be migrated;

[0013] The target metadata of the subvolume to be migrated is migrated to the target metadata service.

[0014] The metadata migration method provided in the present embodiment determines the metadata of the subvolume to be migrated, determines multiple clients of tenants bound to the subvolume to be migrated based on the metadata of the subvolume to be migrated, filters out the metadata to be recycled that matches the subvolume to be migrated from the multiple clients, releases the metadata to be recycled in the metadata of the subvolume to be migrated, obtains the target metadata of the subvolume to be migrated, and migrates the target metadata of the subvolume to be migrated to the target metadata service, thereby reducing the metadata information to be migrated of the subvolume to be migrated, and further when migrating the metadata information of the subvolume to be migrated, reducing the locking time of the subvolume to be migrated, and shortening the waiting time for the client to access the subvolume to be migrated.

[0015] In an optional implementation, determining the metadata service to be migrated based on the load information includes:

[0016] For any metadata service, if the load information of the metadata service exceeds a load threshold, the metadata service is determined to be a metadata service to be migrated.

[0017] The metadata migration method provided in this embodiment realizes load balancing of metadata services in a distributed file system and improves the service capability of metadata services by determining metadata services whose load information exceeds a load threshold as metadata services to be migrated.

[0018] In an optional implementation, the determining the metadata of the subvolume to be migrated based on the heat information of the subvolume in the metadata service to be migrated includes:

[0019] For any sub-volume in the metadata service to be migrated, if the heat information of the sub-volume exceeds a heat threshold, the sub-volume is determined to be a sub-volume to be migrated.

[0020] The metadata migration method provided in this embodiment effectively reduces the load of the metadata service to be migrated by determining the subvolume whose heat information exceeds the heat threshold as the subvolume to be migrated, and realizes the load balancing of the metadata service in the distributed file system.

[0021] In an optional implementation manner, the step of screening out metadata to be recycled that matches the subvolume to be migrated from multiple clients includes:

[0022] According to the least used strategy, target files matching the subvolume to be migrated are screened from multiple clients;

[0023] If the number of the target files screened out reaches a preset number threshold, and the screening time does not reach a preset time threshold, the screening is stopped, and the metadata of the target files screened out is determined to be metadata to be recycled;

[0024] If the number of the screened target files does not reach a preset number threshold, and the screening time reaches a preset time threshold, the screening is stopped, and the metadata of the screened target files is determined to be metadata to be recycled.

[0025] The metadata migration method provided in this embodiment recycles the metadata information of the target file that is least used and releases it as metadata to be recycled, thereby reducing the metadata information to be migrated of the sub-volume to be migrated. Furthermore, when performing metadata migration of the sub-volume to be migrated, the locking time of the sub-volume to be migrated is reduced, and the waiting time for the client to access the sub-volume to be migrated is shortened.

[0026] In an optional implementation manner, migrating the target metadata of the subvolume to be migrated to the target metadata service includes:

[0027] Based on the tenant bound to the subvolume to be migrated, target metadata of the subvolume to be migrated whose bound tenant is the same tenant are migrated as a batch to the target metadata service;

[0028] Migrate the target metadata of the subvolumes to be migrated that are bound to different tenants to the target metadata service in batches.

[0029] The metadata migration method provided in this embodiment reduces data security risks and ensures metadata security of tenants by isolating tenants when migrating target metadata of the subvolume to be migrated.

[0030] In an optional implementation, before releasing the metadata to be recycled in the metadata of the subvolume to be migrated, the method further includes:

[0031] The metadata of the subvolume to be migrated is persistently stored, so that when an unexpected power failure occurs during the process of migrating the target metadata of the subvolume to be migrated to the target metadata service, the persistently stored metadata of the subvolume to be migrated can be used for recovery.

[0032] The metadata migration method provided in this embodiment ensures the security of the metadata of the subvolume to be migrated during migration by persistently storing the metadata of the subvolume to be migrated. Even when the metadata service loses power unexpectedly, the security of the metadata of the subvolume to be migrated can be protected.

[0033] In a second aspect, the present invention provides a metadata migration device, the device comprising:

[0034] A first acquisition module is used to acquire load information of each metadata service;

[0035] A first determining module, configured to determine a metadata service to be migrated based on the load information, wherein the metadata service to be migrated includes metadata of a plurality of subvolumes;

[0036] A second determination module, configured to determine metadata of the subvolume to be migrated based on the heat information of the subvolume in the metadata service to be migrated;

[0037] A third determination module, configured to determine, based on the metadata of the subvolume to be migrated, a plurality of clients of the tenant bound to the subvolume to be migrated;

[0038] A screening module, used for screening out metadata to be recycled that matches the subvolume to be migrated from multiple clients;

[0039] A second acquisition module is used to release the metadata to be recycled in the metadata of the subvolume to be migrated, and obtain the target metadata of the subvolume to be migrated;

[0040] The migration module is used to migrate the target metadata of the subvolume to be migrated to the target metadata service.

[0041] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the metadata migration method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0042] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the metadata migration method of the first aspect or any corresponding embodiment thereof.

[0043] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions for causing a computer to execute the metadata migration method of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the related technologies, the drawings required for use in the specific embodiments or the related technical descriptions will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0045] Figure 1is a schematic diagram of performing metadata service load balancing according to an embodiment of the present invention;

[0046] Figure 2 is a flowchart of a metadata migration method according to an embodiment of the present invention;

[0047] Figure 3 is a flowchart of another metadata migration method according to an embodiment of the present invention;

[0048] Figure 4 It is a schematic diagram of isolating target metadata of different sub-volumes to be migrated of bound tenants according to an embodiment of the present invention;

[0049] Figure 5 is a flowchart of yet another metadata migration method according to an embodiment of the present invention;

[0050] Figure 6 is a structural block diagram of a metadata migration device according to an embodiment of the present invention;

[0051] Figure 7 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0053] Before introducing the embodiments of the present application, some terms or concepts involved in the embodiments of the present application are first explained. It should be understood that the present application does not specifically limit the naming of the following terms. The following terms may have other names. The renamed terms still meet the following related term explanations.

[0054] 1. Tenant (lessee)

[0055] A user of a distributed file system may have one or more clients that mount the file system, but all of these mounted resources belong to this tenant. Tenants are usually bound to some resources, such as subvolume resources.

[0056] 2. Subvolume

[0057] An independent sub-file system under a large file system can set capacity independently. A sub-volume is often bound to a tenant. For the tenant, it is an independent file system.

[0058] 3. Distributed File System (dfs)

[0059] The physical storage resources managed by the file system are not necessarily directly connected to the local node, but are connected to the node through a computer network, or are a complete hierarchical file system formed by combining several different logical disk partitions or volumes. DFS provides a logical tree-like file system structure for resources distributed anywhere on the network, making it easier for users to access shared files distributed on the network.

[0060] 4. Metadata

[0061] The file system stores the metadata of files, including the file name, file size, file creation time, file modification time, file permission group, directory structure information where the file is located, etc. The size of metadata is generally small compared to the file data.

[0062] 5. Metadata server (mds)

[0063] In a distributed file system, a service that specifically processes file metadata. It processes the structural relationship between files, the creation, deletion, and information update of file metadata, etc. It does not include the data part of the file.

[0064] 6. Metadata service load balancing

[0065] In a large distributed file system, there are many mds to provide elastic expansion capabilities of mds and improve the metadata processing capabilities of the file system. However, there is often a large load gap and load imbalance between metadata services. In this case, it is necessary to migrate some metadata from the high-load mds to the low-load mds, and this part of the metadata will be handled by the new mds to achieve the purpose of load balancing among all mds and improve the stability of the system.

[0066] 7. Metadata Migration

[0067] The operation performed during metadata service load balancing is the process of transferring part of the metadata from the high-loaded MDS to the low-loaded MDS. This process is called metadata migration.

[0068] 8. Subvolume Locking

[0069] During the migration process, the subvolume is migrated according to its metadata. During this process, the subvolume needs to be locked. During the locking time, files cannot be deleted, created, or read under the subvolume.

[0070] With the advent of the information age, emerging fields such as big data and cloud computing have emerged and developed rapidly. Distributed storage architecture is crucial in these fields. Distributed file system is a kind of distributed storage architecture. In the distributed file system, data is organized into files for storage and access, providing scalable, high-performance, distributed file storage services.

[0071] In order to ensure the processing efficiency of metadata services and prevent metadata services from becoming a system bottleneck, in related technologies, distributed file systems not only provide multiple data services, but also provide elastic expansion of metadata services, as well as load balancing and metadata migration services between metadata services.

[0072] In a metadata service cluster, when an MDS has an excessively high load, some metadata in the MDS needs to be migrated to other MDSs with lower loads for metadata processing to achieve load balancing of the distributed file system's metadata service.

[0073] Specifically, by counting the load of each mds and the heat of each subvolume, the metadata of the hot subvolume on the mds with high load is migrated to the mds with low load, so that the new mds can provide metadata services for the migrated subvolume. It should be noted that the metadata of a subvolume includes the metadata of the subvolume itself and the metadata of the files under it, that is, the metadata of the files it includes.

[0074] For example, Figure 1 FIG. 1 is a schematic diagram of performing metadata service load balancing according to an embodiment of the present invention. Figure 1 As shown in the figure, the load of mds1 is high, and the load of mds2 is low. It is necessary to migrate the metadata of some tenant subvolumes of mds1 to mds2, and mds2 will manage the metadata of the migrated tenant subvolumes. The source mds is mds1, and the destination mds is mds2. The tenant subvolume is the subvolume.

[0075] Metadata service load balancing can be used to improve the horizontal scalability of metadata services and the service capabilities of distributed file systems.

[0076] However, in the related art, when performing metadata migration, there are the following problems:

[0077] 1. When metadata service load balancing is performed, the subvolume corresponding to the metadata to be migrated needs to be locked. If the locking time is too long, the subvolume will be inaccessible during the locking period. If the locking time is too long, it will trigger client access timeout, affect client use, and even cause client data loss.

[0078] For example, when a tenant's client uses the file system, if a new file is created with a timeout, a timeout or error may be returned if the file is not created successfully within seconds, resulting in a very poor user experience. If the customer creates files asynchronously, the timeout may affect the security of the customer's files and may even lead to file loss.

[0079] 2. When the metadata corresponding to a subvolume with a large number of files is migrated, the total migration time reaches tens of seconds or even minutes.

[0080] 3. When performing metadata migration, if the tenants bound to the subvolume corresponding to the metadata are not the same tenant, tenant isolation cannot be performed, and metadata of different tenants are migrated together, posing a data security risk.

[0081] 4. During the metadata migration process, if the migration fails due to an unexpected power outage of the MDS, the tenant's metadata will be lost.

[0082] An embodiment of the present invention provides a metadata migration method, which determines the metadata of a subvolume to be migrated, determines multiple clients of tenants bound to the subvolume to be migrated based on the metadata of the subvolume to be migrated, filters out metadata to be recycled that matches the subvolume to be migrated from the multiple clients, releases the metadata to be recycled in the metadata of the subvolume to be migrated, obtains target metadata of the subvolume to be migrated, and migrates the target metadata of the subvolume to be migrated to a target metadata service to reduce the metadata information to be migrated of the subvolume to be migrated, and further reduces the locking time of the subvolume to be migrated when migrating the metadata information of the subvolume to be migrated, thereby shortening the waiting time for the client to access the subvolume to be migrated.

[0083] According to an embodiment of the present invention, an embodiment of a metadata migration method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0084] In this embodiment, a metadata migration method is provided, which can be used in a distributed file system. Figure 2 is a flowchart of a metadata migration method according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0085] Step S201, obtaining load information of each metadata service.

[0086] First, the load information of each metadata service in the distributed file system is obtained. The load information may include the utilization rate of the central processing unit CPU, the memory utilization rate, etc.

[0087] Step S202: determining metadata services to be migrated based on the load information, where the metadata services to be migrated include metadata of multiple sub-volumes.

[0088] After the load information of each metadata service is obtained, the metadata service to be migrated is determined according to the load information of the metadata service.

[0089] It should be noted that each metadata service is used to manage metadata of multiple subvolumes.

[0090] Step S203: determining metadata of the subvolume to be migrated based on the popularity information of the subvolume in the metadata service to be migrated.

[0091] After the metadata service to be migrated is determined, metadata of the subvolume to be migrated is determined according to the heat information of multiple subvolumes managed by the metadata service to be migrated.

[0092] It can be understood that the metadata of the subvolume to be migrated is metadata to be migrated.

[0093] The popularity information may include the access frequency, the number of files under the subvolume, etc.

[0094] Step S204: Based on the metadata of the subvolume to be migrated, multiple clients of the tenant bound to the subvolume to be migrated are determined.

[0095] After the metadata of the sub-volume to be migrated is determined, multiple clients of tenants bound to the sub-volume to be migrated are determined based on the metadata of the sub-volume to be migrated.

[0096] Step S205 , filtering out metadata to be recycled that matches the subvolume to be migrated from multiple clients.

[0097] After determining multiple clients of tenants bound to the subvolume to be migrated, the metadata to be recycled that matches the subvolume to be migrated is screened out from the multiple clients.

[0098] Step S206: releasing the metadata to be recycled in the metadata of the sub-volume to be migrated, and obtaining the target metadata of the sub-volume to be migrated.

[0099] After the metadata to be recycled is obtained, the metadata matching the metadata to be recycled in the metadata of the subvolume to be migrated is released, that is, the metadata to be recycled in the memory is cleared to obtain the target metadata of the subvolume to be migrated.

[0100] Step S207: Migrate the target metadata of the subvolume to be migrated to the target metadata service.

[0101] After the target metadata of the subvolume to be migrated is obtained, the target metadata is migrated to the target metadata service.

[0102] It should be noted that before migrating the target metadata of the subvolume to be migrated to the target metadata service, the subvolume to be migrated needs to be locked and cannot be accessed.

[0103] The metadata service to be migrated packages the target metadata, obtains a target metadata package, and sends the target metadata package to the target metadata service.

[0104] After receiving the target metadata package, the target metadata service parses the target metadata package to obtain the metadata of the subvolume to be migrated, restores the file structure based on the metadata of the subvolume to be migrated, unlocks the subvolume to be migrated, and then all operation requests under this subvolume will be performed by the target metadata service.

[0105] It should be noted that, according to the load information of each metadata service, the metadata service with the lowest load information is determined as the target metadata service.

[0106] The metadata migration method provided in the present embodiment determines the metadata of the subvolume to be migrated, determines multiple clients of tenants bound to the subvolume to be migrated based on the metadata of the subvolume to be migrated, filters out the metadata to be recycled that matches the subvolume to be migrated from the multiple clients, releases the metadata to be recycled in the metadata of the subvolume to be migrated, obtains the target metadata of the subvolume to be migrated, and migrates the target metadata of the subvolume to be migrated to the target metadata service, thereby reducing the metadata information to be migrated of the subvolume to be migrated, and further, when migrating the metadata information of the subvolume to be migrated, reduces the locking time of the subvolume to be migrated, reduces the migration time of the metadata of the subvolume to be migrated, and shortens the waiting time for the client to access the subvolume to be migrated.

[0107] In this embodiment, a metadata migration method is provided, which can be used in a distributed file system. Figure 3 is a flowchart of a metadata migration method according to an embodiment of the present invention. Figure 3 As shown, the process includes the following steps:

[0108] Step S301: Obtain load information of each metadata service.

[0109] Step S302: determining metadata services to be migrated based on the load information, where the metadata services to be migrated include metadata of multiple sub-volumes.

[0110] Specifically, the above step S302 includes:

[0111] Step S3021: for any metadata service, if the load information of the metadata service exceeds a load threshold, the metadata service is determined to be a metadata service to be migrated.

[0112] The load threshold is determined by technicians based on experience.

[0113] It should be noted that, when the load information includes CPU usage and memory usage, the same or different weights are assigned to the CPU usage and memory usage. For any metadata service, the CPU usage and memory usage of the metadata service are weighted and summed to obtain the load value, which is compared with the load threshold to determine the metadata service whose load value exceeds the load threshold as the metadata service to be migrated.

[0114] Step S303: determining metadata of the subvolume to be migrated based on the popularity information of the subvolume in the metadata service to be migrated.

[0115] Specifically, the above step S303 includes:

[0116] Step S3031: for any sub-volume in the metadata service to be migrated, if the heat information of the sub-volume exceeds a heat threshold, the sub-volume is determined to be a sub-volume to be migrated.

[0117] The heat threshold is determined by technicians based on experience.

[0118] It should be noted that, when the heat information includes the access frequency and the number of files under the subvolume, the same or different weights are assigned to the access frequency and the number of files under the subvolume. For any subvolume, the access frequency of the subvolume and the number of files under the subvolume are weighted and summed to obtain a heat value, which is compared with a heat threshold to determine the subvolume whose heat value exceeds the heat threshold as the subvolume to be migrated.

[0119] Step S304: Based on the metadata of the subvolume to be migrated, multiple clients of the tenants bound to the subvolume to be migrated are determined. Figure 2 Step S204 of the illustrated embodiment will not be described in detail here.

[0120] Step S305 , filtering out metadata to be recycled that matches the subvolume to be migrated from multiple clients.

[0121] Specifically, the above step S305 includes:

[0122] Step S3051: According to the least used policy, target files matching the subvolume to be migrated are screened from multiple clients.

[0123] That is, the least used target files matching the subvolume to be migrated are filtered from multiple clients.

[0124] It should be noted that the target files matching the subvolume to be migrated may also be screened from multiple clients in the order of usage from low to high.

[0125] The target file that matches the subvolume to be migrated is the target file that belongs to the subvolume to be migrated.

[0126] Step S3052: If the number of the screened target files reaches a preset number threshold, and the screening time does not reach a preset time threshold, the screening is stopped, and the metadata of the screened target files is determined to be metadata to be recycled.

[0127] Among them, in the process of screening target files matching the subvolume to be migrated from multiple clients according to the least used strategy or the order of usage from low to high, if the number of screened target files reaches a preset number threshold and the screening time does not reach a preset time threshold, the screening is stopped and the metadata of the screened target files is determined to be the metadata to be recycled.

[0128] The preset quantity threshold and the preset time threshold are set by a technician.

[0129] Step S3053: If the number of the screened target files does not reach the preset number threshold, and the screening time reaches the preset time threshold, the screening is stopped, and the metadata of the screened target files is determined to be metadata to be recycled.

[0130] In the process of screening target files matching the subvolume to be migrated from multiple clients according to the least used strategy or the order of usage from low to high, if the number of screened target files does not reach a preset number threshold and the screening time reaches a preset time threshold, the screening is stopped and the metadata of the screened target files is determined to be the metadata to be recycled.

[0131] It can be understood that the number of screened target files does not reach the preset number threshold, and the screening time reaches the preset time threshold. In order to prevent unlimited waiting due to network problems, the screening is forcibly stopped.

[0132] It should be noted that when the metadata of the same subvolume is migrated, the screening action for metadata to be recycled will only be performed once, and the screening will not be repeated.

[0133] Step S306: Release the metadata to be recycled in the metadata of the subvolume to be migrated, and obtain the target metadata of the subvolume to be migrated. Figure 2 Step S206 of the illustrated embodiment will not be described in detail here.

[0134] Step S307: Migrate the target metadata of the subvolume to be migrated to the target metadata service.

[0135] Specifically, the above step S307 includes:

[0136] Step S3071: Based on the tenant bound to the sub-volume to be migrated, target metadata of the sub-volume to be migrated whose bound tenant is the same tenant are migrated to the target metadata service as a batch.

[0137] In order to protect data security during metadata migration, target metadata of different to-be-migrated subvolumes of bound tenants are isolated.

[0138] Figure 4 FIG. 1 is a schematic diagram of isolating target metadata of different subvolumes to be migrated of bound tenants according to an embodiment of the present invention. Figure 4 As shown, according to the tenant bound to the subvolume to be migrated in the metadata service, the subvolumes to be migrated with different bound tenants are isolated, that is, the target metadata of the tenant subvolumes with different bound tenants are isolated.

[0139] The sub-volumes to be migrated with different bound tenants are isolated. Specifically, when performing metadata migration, the target metadata of the sub-volumes to be migrated with the same bound tenant are migrated to the target metadata service as a batch.

[0140] Step S3072: Migrate the target metadata of the to-be-migrated subvolumes whose bound tenants are different tenants to the target metadata service in batches.

[0141] If the tenants bound to the subvolumes to be migrated are different, the target metadata of the subvolumes to be migrated are migrated to the target metadata service in different batches to achieve isolation of metadata of subvolumes of different tenants.

[0142] The metadata migration method provided in this embodiment realizes load balancing of metadata services in a distributed file system and improves the service capability of metadata services by determining metadata services whose load information exceeds a load threshold as metadata services to be migrated.

[0143] By determining the subvolume whose heat information exceeds the heat threshold as the subvolume to be migrated, the load of the metadata service to be migrated is effectively reduced, and the load balancing of the metadata service in the distributed file system is achieved.

[0144] By recycling the metadata information of the least used target file and releasing it as metadata to be recycled, the metadata information to be migrated of the subvolume to be migrated is reduced. Furthermore, when performing metadata migration of the subvolume to be migrated, the locking time of the subvolume to be migrated is reduced, and the waiting time for the client to access the subvolume to be migrated is shortened.

[0145] By isolating tenants when migrating the target metadata of the subvolume to be migrated, data security risks are reduced and the metadata security of tenants is ensured.

[0146] In some optional implementations, before releasing the metadata to be recycled in the metadata of the subvolume to be migrated, the metadata migration method further includes:

[0147] Step a1, persistently storing the metadata of the subvolume to be migrated, so that in case of an unexpected power failure during the process of migrating the target metadata of the subvolume to be migrated to the target metadata service, the persistently stored metadata of the subvolume to be migrated can be used for recovery.

[0148] Among them, in order to ensure data security during the metadata migration process, the metadata of the subvolume to be migrated is persistently stored before the metadata to be recycled is released, so that in the event of an unexpected power outage during the migration of the target metadata of the subvolume to be migrated to the target metadata service, the persistently stored metadata of the subvolume to be migrated can be used to restore it.

[0149] The metadata migration method provided in this embodiment ensures the security of the metadata of the subvolume to be migrated during migration by persistently storing the metadata of the subvolume to be migrated. Even when the metadata service loses power unexpectedly, the security of the metadata of the subvolume to be migrated can be protected.

[0150] In this embodiment, a metadata migration method is provided. Figure 5 is a flowchart of a metadata migration method according to an embodiment of the present invention. Figure 5 As shown, the process includes the following steps:

[0151] Step 1: Start the pre-migration process and obtain the tenant subvolumes that need to be migrated.

[0152] It can be understood that what is actually obtained is the metadata of a tenant subvolume that needs to be migrated. Please refer to the description of the above steps S301 to S303 for details, which will not be repeated here.

[0153] Isolate the tenant subvolumes that need to be migrated and obtain the isolated tenant subvolumes. Specifically, isolate the tenant subvolumes that need to be migrated by the tenants bound to the tenant subvolumes to ensure that the tenant subvolumes that need to be migrated are isolated from each other and the metadata of the tenant subvolumes that need to be migrated are not related.

[0154] It is understandable that the isolation of the tenant subvolumes that need to be migrated can also be performed before the metadata migration, as described in the aforementioned steps S3071 and S3072, which will not be repeated here.

[0155] Step 2: Determine whether to start subvolume optimization, that is, determine whether to start acquiring metadata to be recycled for the subvolume to be migrated.

[0156] If subvolume optimization has not yet started, start subvolume optimization, that is, start subvolume metadata optimization, to optimize and recycle the tenant's unused metadata. MDS sends messages to all tenant clients used by the tenant to recycle some metadata under this tenant. Among them, the metadata is recycled according to the least used and low usage policies.

[0157] After receiving the message, the tenant client releases the metadata resources that are not used but occupied by the tenant client. After the MDS successfully reclaims the file metadata, it can clean up this part of the metadata in the memory, freeing up the MDS memory and reducing the CPU usage. When the MDS performs subvolume migration, the metadata to be migrated under the subvolume will be reduced.

[0158] The total set of metadata to be migrated under the subvolume is U = {c 1 ,c 2 ,c 3 ,…,c n}, c is the metadata resource owned by the tenant client, c n The subscript number indicates the client number of this tenant. If the tenant client occupies metadata, the corresponding mds will have corresponding resource consumption, including memory resource consumption and CPU resource consumption.

[0159] The file system volume set is V = {sv 1 ,sv 2 ,sv 3 ,…,sv n}, where sv represents each subvolume, and all subvolume resources are combined into a volume. n The subscript number indicates the subvolume number. If there is a target subvolume to be migrated, it will be optimized, and the total resources occupied by V will also be reduced. The metadata of the optimized subvolume is recycled from the tenant side, not from other MDSs.

[0160] Step 3: Subvolume optimization has started. Check whether the subvolume has been optimized to the specified number of metadata. If so, the optimization is complete. If the optimization has not reached the specified number of metadata, check whether the specified upper limit of optimization time has been reached. If so, the optimization is forced to complete to prevent unlimited waiting due to network and other issues. The same subvolume will only be optimized once when it starts migration, and will not be optimized repeatedly.

[0161] Please refer to the corresponding descriptions of the aforementioned steps S3051 to S3053 for details, which will not be repeated here.

[0162] Step 4: When optimizing the subvolume, the metadata recovered from the tenant is stored persistently to ensure the security and reliability of the tenant's metadata. Then, the metadata is cleaned up and released on the metadata service to reduce the memory and CPU usage of data resources on the metadata service. It should be noted that metadata other than the recovered metadata in the subvolume metadata is also stored persistently.

[0163] Persistent storage ensures that the tenant's subvolume metadata remains safe and reliable in the event of an unexpected power outage during the migration optimization process. After the MDS is restored, all the tenant's metadata is fully restored, and unexpected situations during the migration process do not affect the tenant's data security.

[0164] Step 5: After subvolume optimization is completed, the remaining data set to be migrated is

[0165] UR = {cr 1 ,cr 2 ,cr 3 ,…,cr n}, cr is the metadata used by tenants after subvolume optimization, where cr<=c. After subvolume optimization, the metadata to be migrated on mds has been greatly reduced. Then the subvolume is locked and inaccessible. The metadata of the target subvolume is packaged from the source mds. At this time, the amount of metadata of the subvolume to be migrated on the source mds is reduced, and the packaging and sending time is also reduced, and then it is sent to the destination mds.

[0166] Step 6: The destination mds receives the metadata package, parses the metadata of the target subvolume, restores the file structure, and unlocks the subvolume. After that, all operation requests under this subvolume will be provided by the destination mds. At this time, the subvolume is restored to normal and can receive normal file operations, such as creating files, deleting files, and opening files. Because the amount of metadata sent by the source mds is reduced, the time it takes for the destination mds to restore the subvolume is also reduced, and the overall time of subvolume locking is reduced.

[0167] The metadata migration method provided in this embodiment recycles metadata according to the tenant-isolated subvolume level, accurately locates the tenant client of the metadata, and isolates the metadata of each tenant from each other when the metadata service is load balanced, ensuring the security and independence of data between tenants. When balancing MDS, migrating a subvolume has no impact on other subvolumes.

[0168] In the tenant's subvolume optimization, the recovered metadata is persisted on the mds. Even if an abnormal situation such as mds power failure occurs during the metadata balancing process, the tenant's data is still safe and reliable. All metadata of the subvolume can be completely restored after the mds is powered on and restored, ensuring that the tenant's metadata security is still protected even if an accident occurs during the migration process.

[0169] After the tenant's subvolume is optimized, the metadata packaged in the source mds during migration is reduced, and the subvolume locking time can be reduced for the first time. Since the amount of subvolume data to be migrated is reduced, the time for the source mds to send the metadata package to the destination mds is also reduced accordingly, and the subvolume locking time can be reduced for the second time. Finally, at the destination mds, the metadata package sent by the source mds is received, and all the structures of the subvolume are restored. At this time, the subvolume locking time is reduced for the third time. Compared with related technologies, the subvolume locking time is greatly reduced, from the original seconds to hundreds of milliseconds, reducing the time by more than 90%. Finally, the impact of migration on the tenant's client file operations is greatly reduced and can even be ignored.

[0170] In this embodiment, a metadata migration device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0171] This embodiment provides a metadata migration device, such as Figure 6 As shown, including:

[0172] The first acquisition module 601 is used to acquire load information of each metadata service.

[0173] The first determining module 602 is used to determine the metadata service to be migrated based on the load information, where the metadata service to be migrated includes metadata of multiple sub-volumes.

[0174] The second determination module 603 is used to determine the metadata of the sub-volume to be migrated based on the popularity information of the sub-volume in the metadata service to be migrated.

[0175] The third determination module 604 is configured to determine, based on the metadata of the sub-volume to be migrated, a plurality of clients of the tenant bound to the sub-volume to be migrated.

[0176] The screening module 605 is used to screen out metadata to be recycled that matches the subvolume to be migrated from multiple clients.

[0177] The second acquisition module 606 is used to release the metadata to be recycled in the metadata of the sub-volume to be migrated, and obtain the target metadata of the sub-volume to be migrated.

[0178] The migration module 607 is used to migrate the target metadata of the subvolume to be migrated to the target metadata service.

[0179] In some optional implementations, the first determining module 602 includes:

[0180] The first determining subunit is configured to determine, for any metadata service, if the load information of the metadata service exceeds a load threshold, that the metadata service is a metadata service to be migrated.

[0181] In some optional implementations, the second determining module 603 includes:

[0182] The second determining sub-unit is configured to determine, for any sub-volume in the metadata service to be migrated, that the sub-volume is a sub-volume to be migrated if the heat information of the sub-volume exceeds a heat threshold.

[0183] In some optional implementations, the screening module 605 includes:

[0184] The first screening sub-unit is used to screen target files matching the sub-volume to be migrated from multiple clients according to the least used policy.

[0185] The first determining unit is used to stop screening if the number of the screened target files reaches a preset number threshold and the screening time does not reach a preset time threshold, and determine that the metadata of the screened target files are metadata to be recycled.

[0186] The second determining unit is used to stop screening if the number of the screened target files does not reach a preset number threshold and the screening time reaches a preset time threshold, and determine that the metadata of the screened target files are metadata to be recycled.

[0187] In some optional implementations, the migration module 607 includes:

[0188] The first migration sub-unit is used to migrate the target metadata of the sub-volumes to be migrated that are bound to the same tenant as the tenant to be migrated to the target metadata service as a batch based on the tenant bound to the sub-volumes to be migrated.

[0189] The second migration sub-unit is used to migrate the target metadata of the to-be-migrated sub-volumes whose bound tenants are different tenants to the target metadata service in batches.

[0190] In some optional implementations, before releasing the metadata to be recycled in the metadata of the subvolume to be migrated, the metadata migration device further includes:

[0191] The storage module is used to persistently store the metadata of the subvolume to be migrated, so that in the event of an unexpected power failure during the process of migrating the target metadata of the subvolume to be migrated to the target metadata service, the persistently stored metadata of the subvolume to be migrated can be used for recovery.

[0192] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0193] The metadata migration device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0194] The embodiment of the present invention also provides a computer device having the above Figure 6 The metadata migration apparatus is shown.

[0195] See also Figure 7 , Figure 7 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 7 As shown, the computer device includes: one or more processors 701, memory 702, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Each component utilizes different buses to communicate with each other, and can be installed on a common mainboard or installed in other ways as required. The processor can process the instructions executed in the computer device, including instructions stored in the memory or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides part of the necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 7 A processor 701 is taken as an example.

[0196] The processor 701 may be a central processing unit, a network processor or a combination thereof. The processor 701 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable logic gate array, a general purpose array logic or any combination thereof.

[0197] The memory 702 stores instructions executable by at least one processor 701, so that the at least one processor 701 executes the method shown in the above embodiment.

[0198] The memory 702 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 702 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 702 may optionally include a memory remotely arranged relative to the processor 701, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0199] The memory 702 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 702 may also include a combination of the above types of memory.

[0200] The computer device also includes a communication interface 703, which is used for the computer device to communicate with other devices or a communication network.

[0201] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0202] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0203] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A metadata migration method, characterized in that: The method comprises: Get the load information of each metadata service; Determine, based on the load information, a metadata service to be migrated, wherein the metadata service to be migrated includes metadata of a plurality of subvolumes; Determining metadata of the subvolume to be migrated based on the popularity information of the subvolume in the metadata service to be migrated; Based on the metadata of the subvolume to be migrated, determining multiple clients of tenants bound to the subvolume to be migrated; Filtering metadata to be recycled that matches the subvolume to be migrated from multiple clients; Releasing the metadata to be recycled in the metadata of the subvolume to be migrated, and obtaining target metadata of the subvolume to be migrated; The target metadata of the subvolume to be migrated is migrated to the target metadata service.

2. The method according to claim 1, characterized in that The determining, based on the load information, the metadata service to be migrated includes: For any metadata service, if the load information of the metadata service exceeds a load threshold, the metadata service is determined to be a metadata service to be migrated.

3. The method according to claim 1, characterized in that The determining the metadata of the subvolume to be migrated based on the heat information of the subvolume in the metadata service to be migrated includes: For any sub-volume in the metadata service to be migrated, if the heat information of the sub-volume exceeds a heat threshold, the sub-volume is determined to be a sub-volume to be migrated.

4. The method according to claim 1, characterized in that: The step of screening out metadata to be recycled that matches the subvolume to be migrated from multiple clients includes: According to the least used strategy, target files matching the subvolume to be migrated are screened from multiple clients; If the number of the target files screened out reaches a preset number threshold, and the screening time does not reach a preset time threshold, the screening is stopped, and the metadata of the target files screened out is determined to be metadata to be recycled; If the number of the screened target files does not reach a preset number threshold, and the screening time reaches a preset time threshold, the screening is stopped, and the metadata of the screened target files is determined to be metadata to be recycled.

5. The method according to claim 1, characterized in that: The step of migrating the target metadata of the subvolume to be migrated to the target metadata service includes: Based on the tenant bound to the subvolume to be migrated, target metadata of the subvolume to be migrated whose bound tenant is the same tenant are migrated as a batch to the target metadata service; Migrate the target metadata of the subvolumes to be migrated that are bound to different tenants to the target metadata service in batches.

6. The method according to claim 1, characterized in that Before releasing the metadata to be recycled in the metadata of the subvolume to be migrated, the method further includes: The metadata of the subvolume to be migrated is persistently stored, so that when an unexpected power failure occurs during the process of migrating the target metadata of the subvolume to be migrated to the target metadata service, the persistently stored metadata of the subvolume to be migrated can be used for recovery.

7. A metadata migration device, characterized in that: The device comprises: A first acquisition module is used to acquire load information of each metadata service; A first determining module, configured to determine a metadata service to be migrated based on the load information, wherein the metadata service to be migrated includes metadata of a plurality of subvolumes; A second determination module, configured to determine metadata of the subvolume to be migrated based on the heat information of the subvolume in the metadata service to be migrated; A third determination module, configured to determine, based on the metadata of the subvolume to be migrated, a plurality of clients of the tenant bound to the subvolume to be migrated; A screening module, used for screening out metadata to be recycled that matches the subvolume to be migrated from multiple clients; A second acquisition module is used to release the metadata to be recycled in the metadata of the subvolume to be migrated, and obtain the target metadata of the subvolume to be migrated; The migration module is used to migrate the target metadata of the subvolume to be migrated to the target metadata service.

8. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the metadata migration method according to any one of claims 1 to 6 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the metadata migration method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The method comprises computer instructions for causing a computer to execute the metadata migration method according to any one of claims 1 to 6.

Images