Metadata balance management method and device, equipment and medium

Abstract

The invention discloses a metadata balance management method and device, equipment and a medium, and relates to the technical field of data management, and the method comprises the steps: carrying out the fragmentation of a local preset file sharing directory based on a received configuration instruction, so as to obtain a plurality of corresponding fragmentation directories and a plurality of corresponding fragmentation values; performing Hash calculation on the plurality of fragment values and directory numbers corresponding to a preset file sharing directory, so as to determine metadata server daemon process numbers corresponding to the plurality of fragment directories through the obtained plurality of Hash values; respectively distributing a target daemon process and a standby daemon process for the plurality of fragment directories; and if the target daemon process is abnormal, starting a standby daemon process corresponding to the target daemon process. Therefore, the load balancing of the metadata server can be efficiently realized, so that the performance and the stability of the distributed file system are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data management, and in particular to a metadata balanced management method, device, equipment and medium. Background Art

[0002] CephFS (Ceph File System) is a high-performance, scalable distributed file system provided by the Ceph distributed storage system. The metadata of the file system is managed by the metadata server (MDS). Each CephFS file system is configured with an active MDS daemon by default. When the client performs a large number of metadata operations in parallel, the huge OPS (Operations Per Second) pressure may reach the performance bottleneck of a single MDS, and may even cause the MDS daemon to crash or metadata performance to degrade. There is a problem that a single metadata server has a performance bottleneck, and the performance needs to be improved. In addition, in order to expand the performance of metadata, multiple active MDS daemons can be configured to share the metadata load. The existing static subtree partitioning method relies on manual work and has limited applicable scenarios. The dynamic subtree partitioning method can automatically balance the load, but during the subtree migration process, it is necessary to block requests from the client. Therefore, frequent subtree migration will have a negative impact on the business, affecting the performance and stability of the MDS, and it is difficult to efficiently achieve balanced distribution of metadata load.

[0003] In summary, how to efficiently implement load balancing of metadata servers to improve the performance and stability of distributed file systems is a problem that needs to be solved urgently. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a metadata balance management method, device, equipment and medium, which can efficiently realize the load balancing of metadata servers to improve the performance and stability of distributed file systems. The specific scheme is as follows:

[0005] In a first aspect, the present application provides a metadata balanced management method, which is applied to a distributed file system, comprising:

[0006] Slice the local preset file sharing directory based on the received configuration instruction to obtain a plurality of slice directories corresponding to the preset file sharing directory and a plurality of slice values ​​corresponding to the slice directories;

[0007] Performing hash calculations on the plurality of shard values ​​and the directory numbers corresponding to the preset file sharing directories respectively to obtain a plurality of corresponding hash values, and determining metadata server daemon process numbers corresponding to the plurality of shard directories respectively through the plurality of hash values;

[0008] Allocate target daemons corresponding to the metadata server daemon process numbers to the plurality of shard directories respectively, and allocate standby daemons to the plurality of shard directories respectively;

[0009] If the target daemon process is abnormal, the backup daemon process corresponding to the target daemon process is enabled.

[0010] Optionally, before slicing the local preset file sharing directory based on the received configuration instruction, the method further includes:

[0011] Creating a preset file sharing directory locally, and creating several sharing service subdirectories in the preset file sharing directory;

[0012] Based on the received configuration instructions, the several shared service sub-directories are written into a preset path configuration to provide external file system services through the several shared service sub-directories; the preset path configuration is used to use the several shared service sub-directories as a file system provided externally.

[0013] Optionally, slicing the local preset file sharing directory based on the received configuration instruction to obtain a plurality of slicing directories corresponding to the preset file sharing directory and a plurality of slicing values ​​corresponding to the slicing directories includes:

[0014] Determine whether a configuration instruction is received. If the configuration instruction is received, shard the preset file sharing directory according to a preset hash partitioning algorithm to obtain a number of shard directories and a number of shard values ​​corresponding to the number of shard directories; the number of shard values ​​are identifiers generated when the sharding of the preset file sharing directory is completed according to the preset hash partitioning algorithm and are used to uniquely identify the number of shards.

[0015] Optionally, before performing hash calculations on the plurality of shard values ​​and the directory numbers corresponding to the preset file sharing directories to obtain corresponding plurality of hash values, and determining the metadata server daemon process numbers corresponding to the plurality of shard directories respectively through the plurality of hash values, the method further includes:

[0016] Determine whether a preset daemon process quantity control parameter value is greater than the current quantity value of the target daemon process;

[0017] If the preset daemon process quantity control parameter value is not less than the current target daemon process quantity value, the generation of new metadata server daemon process numbers is stopped.

[0018] Optionally, performing hash calculations on the plurality of shard values ​​and directory numbers corresponding to the preset file sharing directories to obtain corresponding plurality of hash values, and determining metadata server daemon numbers corresponding to the plurality of shard directories respectively through the plurality of hash values ​​includes:

[0019] Determine the directory number corresponding to the preset file sharing directory;

[0020] Hash calculations are performed on the plurality of shard values ​​and the directory numbers respectively to obtain a plurality of corresponding hash values, and metadata server daemon process numbers corresponding to the plurality of shard directories are determined according to the plurality of hash values.

[0021] Optionally, the step of respectively allocating target daemons corresponding to the metadata server daemon process numbers to the plurality of shard directories, and respectively allocating standby daemons to the plurality of shard directories comprises:

[0022] Allocating the target daemon process to a plurality of shard directories corresponding to the metadata server daemon process number according to the metadata server daemon process number respectively;

[0023] The plurality of shard directories and the corresponding target daemon processes are bound respectively, and corresponding standby daemon processes are allocated respectively to the plurality of shard directories.

[0024] Optionally, if the target daemon process is abnormal, enabling a standby daemon process corresponding to the target daemon process includes:

[0025] If the target daemon process is abnormal, the service of the target daemon process is taken over by the standby daemon process corresponding to the target daemon process based on the metadata cache of the target daemon process synchronized in real time.

[0026] In a second aspect, the present application provides a metadata balancing management device, which is applied to a distributed file system, including:

[0027] A directory calculation module, used for slicing a local preset file sharing directory based on the received configuration instruction to obtain a plurality of slicing directories corresponding to the preset file sharing directory and a plurality of slicing values ​​corresponding to the slicing directories;

[0028] A number determination module is used to perform hash calculations on the plurality of shard values ​​and the directory numbers corresponding to the preset file sharing directories to obtain corresponding plurality of hash values, and determine the metadata server daemon process numbers corresponding to the plurality of shard directories respectively through the plurality of hash values;

[0029] A process allocation module, used to allocate target daemon processes corresponding to the metadata server daemon process numbers to the plurality of shard directories respectively, and allocate standby daemon processes to the plurality of shard directories respectively;

[0030] The process switching module is used to enable the backup daemon process corresponding to the target daemon process if an abnormality occurs in the target daemon process.

[0031] In a third aspect, the present application provides an electronic device, including:

[0032] Memory, used to store computer programs;

[0033] The processor is used to execute the computer program to implement the aforementioned metadata balance management method.

[0034] In a fourth aspect, the present application provides a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the aforementioned metadata balancing management method is implemented.

[0035] In the present application, a local preset file sharing directory is sharded based on a received configuration instruction to obtain a number of shard directories corresponding to the preset file sharing directory and a number of shard values ​​corresponding to the shard directories; hash calculations are performed on the number of shard values ​​and the directory number corresponding to the preset file sharing directory to obtain a number of corresponding hash values, and metadata server daemon numbers corresponding to the number of shard directories are determined through the number of hash values; target daemons corresponding to the metadata server daemon numbers are assigned to the number of shard directories, and standby daemons are assigned to the number of shard directories; if an abnormality occurs in the target daemon, the standby daemon corresponding to the target daemon is enabled. As can be seen from the above, the present application first slices the local preset file sharing directory based on the received configuration instructions to obtain several slice values ​​corresponding to the several slice directories, and then respectively performs hash calculations on the several slice values ​​and the directory numbers corresponding to the preset file sharing directories, and then determines the metadata server daemon numbers corresponding to the several slice directories according to the obtained several hash values, and assigns the corresponding target daemon and backup daemon to the slice directory according to the metadata server daemon number, and starts the corresponding backup daemon when the target daemon is abnormal. In this way, through the above process of the present application, multiple target daemons are configured, the high availability of the metadata server is achieved, and the target daemons corresponding to the metadata server daemon numbers are respectively assigned to several slice directories, so as to linearly expand the metadata server cluster's ability to process metadata, and then efficiently realize the load balancing of the metadata server, so as to improve the performance and stability of the distributed file system. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0037] Figure 1 A flowchart of a metadata balance management method disclosed in this application;

[0038] Figure 2 A timing diagram of a hash calculation partitioning strategy disclosed in this application;

[0039] Figure 3 A schematic diagram of a metadata server cluster architecture disclosed in this application;

[0040] Figure 4A flowchart of a specific metadata balance management method disclosed in this application;

[0041] Figure 5 A schematic diagram of the structure of a metadata balance management device disclosed in this application;

[0042] Figure 6 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] Currently, for multiple active MDS daemons, the subtree partitioning method is usually used to achieve metadata load balancing. However, the existing static subtree partitioning method relies on manual work and has limited applicable scenarios. The dynamic subtree partitioning method can automatically perform load balancing, but during the subtree migration process, it is necessary to block requests from the client. Therefore, frequent subtree migrations will have a negative impact on the business, affecting the performance and stability of MDS, and it is difficult to efficiently achieve balanced distribution of metadata load.

[0045] In order to overcome the above technical problems, the present application provides a metadata balancing management method to efficiently achieve load balancing of metadata servers to improve the performance and stability of the distributed file system.

[0046] See also Figure 1 As shown, an embodiment of the present invention discloses a metadata balance management method, which is applied to a distributed file system, including:

[0047] Step S11: slice the local preset file sharing directory based on the received configuration instruction to obtain a plurality of slice directories corresponding to the preset file sharing directory and a plurality of slice values ​​corresponding to the slice directories.

[0048] In this embodiment, the distributed file system first receives a configuration instruction to slice the local preset file sharing directory to obtain a number of slice directories and a number of slice values ​​corresponding to the number of slice directories. The distributed file system is a CephFS distributed file system (a file system based on the Ceph distributed storage system), the configuration instruction is an instruction for configuration initiated by a user through a preset configuration interface; the preset file sharing directory is a shared directory created for a file server integrated with the distributed file system; the number of slice values ​​are identifiers generated when the preset file sharing directory is sliced ​​according to a preset hash partitioning algorithm and used to uniquely identify the number of slices.

[0049] It should be pointed out that since the size of the several shard directories is related to the effect of load balancing, if the shard directory is too large, it may cause uneven load, and if the shard directory is too small, it may cause too many migration operations. Therefore, the size setting of the several shard directories is extremely important. In this embodiment, the size of the several shard directories can be set by the user, and the size setting can be packaged into the configuration instruction. Specifically, after receiving the configuration instruction, the present embodiment can parse the configuration instruction to obtain the preset directory shard size therein, and then partition and shard the several preset files in the preset file sharing directory according to the preset directory shard size of the preset hash partitioning algorithm to obtain the corresponding several shard directories containing several preset files and several shard values ​​corresponding to the several shard directories, and the metadata is dispersed in the distributed file system in the form of a directory subtree, and the metadata server (i.e., MetaData Server, MDS) daemon process can be allocated to the several shard directories based on the obtained shard values. The preset hash partitioning algorithm includes but is not limited to the node remainder partitioning algorithm, the consistent hash partitioning algorithm, the virtual slot partitioning algorithm, etc. In this embodiment, the jump consistent hash partitioning algorithm is used for sharding.

[0050] It can be understood that before sharding the preset file sharing directory, it is necessary to first create the preset file sharing directory, and the processing flow is as follows: create a preset file sharing directory locally, and create several shared service subdirectories in the preset file sharing directory; write the several shared service subdirectories into the preset path configuration based on the received configuration instructions, so as to provide external file system services through the several shared service subdirectories; the preset path configuration is used to use the several shared service subdirectories as the file system provided to the outside. That is, after using the distributed file system as the storage backend through the file server, create a preset file sharing directory locally for the file server, and create several shared service subdirectories in the preset file sharing directory, so as to enable the file server to write the several shared service subdirectories into the path configuration exported by the file server based on the configuration instructions, and provide the network file system service of the file server to the outside. Among them, the file server is used to map a specific storage backend such as CephFS to the standard file system interface of the operating system, including but not limited to NFS-Ganesha (a file server in user mode that supports file sharing protocol (i.e., Network File System, NFS)) and the like. For example, when the file server is NFS-Ganesha, an NFS directory is created locally, and a subdirectory for providing NFS services to the outside is created in the directory, so that NFS-Ganesha writes the subdirectory of the NFS directory into its exported path configuration based on the received configuration instruction, and provides file system services to the outside. Figure 2 The figure shows a timing diagram of performing a hash calculation partition strategy on the NFS directory after the NFS directory is created. First, a hash calculation partition strategy is set for the NFS directory, and then partitioning and sharding are performed according to the set hash calculation partition strategy to evenly distribute the NFS directory in the metadata server. In this way, the present embodiment shards the preset file sharing directory based on the received configuration instruction, and performs hash calculation on the obtained several shard directories based on the preset hash partition algorithm, so as to determine the metadata server daemon process numbers corresponding to the several shard directories based on the obtained several hash shard values, thereby improving the efficiency of realizing metadata load balancing.

[0051] Step S12: perform hash calculations on the plurality of shard values ​​and the directory numbers corresponding to the preset file sharing directories respectively to obtain a plurality of corresponding hash values, and determine the metadata server daemon process numbers corresponding to the plurality of shard directories respectively through the plurality of hash values.

[0052] In this embodiment, the distributed file system performs hash calculations on the obtained several shard values ​​and the directory numbers corresponding to the preset file sharing directories to obtain the corresponding several hash values, and determines the metadata server daemon numbers corresponding to the several shard directories respectively through the several hash values. Among them, the metadata server daemon number is the unique position identifier of each metadata server daemon in the metadata server active cluster, which is used to indicate the relative position of the metadata server in the metadata server active cluster. It can be understood that the operation of sharding the preset file sharing directory is essentially to partition the metadata contained in the preset file sharing directory, and the several shared service subdirectories are the carriers of the metadata. Therefore, after sharding, the several shard directories contain the target shared service subdirectories. In this embodiment, the directory number corresponding to the preset file sharing directory and the several shard values ​​can be used to determine the metadata server daemon numbers corresponding to the several shard directories.

[0053] It should be pointed out that the distributed file system has a daemon number control parameter, namely the max_mds parameter. The daemon number control parameter is used to control the number of metadata server daemons created, that is, to control the number of metadata server daemons in the metadata server active cluster. For example, if the value of the daemon number control parameter in the active cluster is 3, then the corresponding number of metadata server daemons is 3. However, it should be further pointed out that the daemon number control parameter is not always effective. Specifically, before respectively determining the target shared service subdirectories contained in the several shard directories, it may also include: determining whether the preset daemon number control parameter value is greater than the current number value of the target daemons; if the preset daemon number control parameter value is not less than the current number value of the target daemons, then stopping generating new metadata server daemon numbers. That is, the daemon process number control parameter takes effect only when the actual number of metadata servers is greater than or equal to the daemon process number control parameter value, that is, the actual number of metadata server daemon process numbers in the distributed file system can only be increased when there is a spare metadata server daemon process that can be used to assume the new metadata server daemon process number. For example, if only one metadata server daemon process is running and the max_mds parameter value is set to 2, the second metadata server daemon process number will not be created. In this way, this embodiment uses the jump consistency hashing algorithm to implement the automatic static partitioning strategy, and determines the metadata server daemon process numbers corresponding to the several shard directories based on the directory number corresponding to the preset file sharing directory and the several shard values, so that the several directory shards can be respectively delegated to the metadata server daemon process cluster, and the metadata server daemon process numbers corresponding to the several shard directories can be determined more efficiently and with low memory loss.

[0054] Step S13: assign target daemons corresponding to the metadata server daemon numbers to the plurality of shard directories respectively, and assign standby daemons to the plurality of shard directories respectively.

[0055] In this embodiment, the distributed file system allocates corresponding target daemons to the plurality of shard directories according to the determined metadata server daemon process numbers, and allocates standby daemons to the plurality of shard directories, wherein the target daemon process refers to an active metadata server daemon process, which is used to manage metadata in the distributed file system.

[0056] It should be pointed out that the present embodiment uses a jump consistent hashing algorithm to allocate the corresponding target daemons to the several shard directories, which can minimize the reallocation when the number of the target daemons is reduced. That is, when the number of the target daemons is reduced, the present embodiment can only repartition the metadata corresponding to the reduced target daemons without affecting the business execution of the remaining other target daemons. The method of subtree partitioning using the jump consistent hashing algorithm is different from the static subtree partitioning method that requires manual negotiation and the dynamic subtree partitioning method that causes business jitter. The metadata of the reduced target daemons is reallocated without affecting the work of the remaining currently running target daemons. The subtrees are only migrated when the directory shards are allocated, which reduces the number of subtree migrations and reduces the impact of business interruptions.

[0057] It is understandable that when the client executes a large number of metadata operations in parallel, if only one metadata server daemon is configured, the huge metadata operation pressure may reach the performance bottleneck of a single metadata server, and may even cause the metadata server daemon to crash or reduce the performance of the metadata. In order to expand the performance of metadata, this embodiment can configure multiple active metadata server daemons to form an active metadata server cluster to jointly bear the metadata load and linearly expand the performance of the metadata server active cluster in processing metadata. At the same time, in order to ensure that when an active metadata server daemon fails, other daemons can promptly take over the service of the active metadata server daemon to avoid interruption of metadata processing services, this embodiment can also configure one or more backup daemons for each of the active metadata server daemons. Figure 3 The figure shows a schematic diagram of a metadata server cluster architecture provided by this application. Figure 3The metadata server cluster in the embodiment has four metadata server daemons, of which two are active metadata server daemons and two are configured as standby metadata server daemons, which not only expands the performance of metadata but also ensures the high availability of the metadata server cluster. It should be pointed out that each active metadata server daemon has a unique metadata server daemon number, and the number of the standby daemon corresponding to the active metadata server daemon is consistent with the metadata server daemon number of the active metadata server daemon. In this way, the present embodiment allocates target daemons corresponding to the metadata server daemon numbers corresponding to the several shard directories, thereby realizing load balancing and parallel processing of metadata. At the same time, the target daemons corresponding to the several shard directories are allocated corresponding standby daemons, so as to ensure that when the target daemon fails, the standby daemon corresponding to it can quickly take over, thereby ensuring the high availability of the metadata server cluster.

[0058] Step S14: If the target daemon process is abnormal, the backup daemon process corresponding to the target daemon process is enabled.

[0059] In this embodiment, if the target daemon process in the distributed file system is abnormal, the backup daemon process corresponding to the target daemon process is started so that the backup daemon process can take over the service of the target daemon process. The abnormal situation of the target daemon process includes but is not limited to the situation that the target daemon process crashes or becomes unresponsive due to internal errors, resource exhaustion, etc., the situation that the network connection of the distributed file system fails, and the situation that the target daemon process cannot work normally due to configuration errors such as improper permission settings and configuration parameter errors.

[0060] It can be understood that the metadata server active-standby switching mode is divided into a cold standby switching mode and a hot standby switching mode. Among them, the present embodiment can select a standby daemon process in the cold standby mode. Specifically, if the target daemon process is abnormal, the standby daemon process as the process backup is enabled, and the operation of replaying the log is performed based on the heartbeat communication maintained between the standby daemon process and the target daemon process to restore the metadata to the cache and take over the service of the target daemon process. In addition, the present embodiment can also select a standby daemon process in the hot standby mode, and its specific processing flow is as follows: if the target daemon process is abnormal, the standby daemon process corresponding to the target daemon process and based on the metadata cache of the target daemon process synchronized in real time takes over the service of the target daemon process. That is, if the standby daemon process in the hot standby mode is selected, since the standby daemon process in the hot standby mode will continuously synchronize the metadata cache of the target daemon process when the target daemon process is running, when the target daemon process is abnormal, the standby daemon process can quickly switch to the active state, and take over the service of the target daemon process based on the metadata cache, without the need for additional log replay operations. In this way, when an abnormality occurs in the target daemon process, this embodiment promptly enables the backup daemon process corresponding to the target daemon process to take over the service of the target daemon process, thereby ensuring the high availability of the metadata server cluster. At the same time, the backup daemon process in hot standby mode can enable the backup daemon process to quickly switch to an active state without performing additional operations, thereby improving the efficiency of active-standby switching.

[0061] As can be seen from the above, the embodiment of the present application first shards the local preset file sharing directory based on the received configuration instructions to obtain a number of shard values ​​corresponding to the said several shard directories, and then performs hash calculations on the several shard values ​​and the directory numbers corresponding to the said preset file sharing directories, and then determines the metadata server daemon numbers corresponding to the several shard directories according to the obtained several hash values, and allocates corresponding target daemon processes and backup daemon processes to the shard directories according to the metadata server daemon process numbers, and starts the corresponding backup daemon process when an abnormality occurs in the target daemon process. In this way, through the above process of the embodiment of the present application, on the one hand, multiple target daemons are configured, which not only expands the performance of metadata, but also ensures the high availability of the metadata server cluster; on the one hand, the preset file sharing directory is first sharded to pave the way for the subsequent allocation of metadata server daemons to the several shard directories, thereby improving the efficiency of metadata load balancing; on the one hand, the preset file sharing directory is partitioned using the jump consistency hashing algorithm, which can more efficiently and with low memory loss determine the metadata server daemon numbers corresponding to the several shard directories, thereby reducing the number of subtree migrations and the impact of business interruptions; on the one hand, target daemons corresponding to the metadata server daemon numbers are allocated to several shard directories, respectively, which can linearly expand the ability of the metadata server cluster to process metadata; on the other hand, a hot standby backup daemon is configured for the target daemon to ensure that when the target daemon fails, the corresponding backup daemon can quickly take over, thereby improving the efficiency of the primary-standby switching, and then efficiently realizing the load balancing of the metadata server, thereby improving the performance and stability of the distributed file system.

[0062] Based on the above embodiments, it can be known that through the method of the present application, after receiving the configuration instruction, the preset file sharing directory can be first sharded to determine the metadata server daemon process numbers corresponding to the obtained several shard directories, thereby further realizing the balanced management of metadata efficiently. For this reason, this embodiment describes in detail how to shard the preset file sharing directory and determine the metadata server daemon process numbers corresponding to the obtained several shard directories. Figure 4 As shown, an embodiment of the present invention discloses a metadata balance management method, which is applied to a distributed file system, including:

[0063] Step S21: Slice the local preset file sharing directory based on the received configuration instruction to obtain a plurality of slice directories corresponding to the preset file sharing directory and a plurality of slice values ​​corresponding to the slice directories.

[0064] Step S22: Determine the directory number corresponding to the preset file sharing directory.

[0065] Step S23: perform hash calculations on the plurality of shard values ​​and the directory numbers respectively to obtain a plurality of corresponding hash values, and determine the metadata server daemon process numbers corresponding to the plurality of shard directories respectively according to the plurality of hash values.

[0066] In this embodiment, the distributed file system first calculates the directory number corresponding to the preset file sharing directory and the plurality of hash values ​​corresponding to the plurality of shard values, respectively, to determine the metadata server daemon process numbers corresponding to the plurality of shard directories according to the plurality of hash values. That is, this embodiment uses a jump consistent hashing algorithm to determine the metadata server daemon process numbers corresponding to the plurality of shard directories.

[0067] It can be understood that since the jump consistent hashing algorithm can map keys to a set of ordered buckets when processing key-value pair data, thereby ensuring that the mapping of keys to buckets can remain as stable as possible when the number of buckets changes, therefore when the scale of the metadata server active cluster grows or shrinks, the jump consistent hashing algorithm can ensure that the several shard directories remain as stable as possible, without the need to continuously migrate subtrees based on the metadata popularity of each metadata server in the later stage as required by the dynamic subtree partitioning method, and only needs to migrate subtrees when allocating directory shards.

[0068] Step S24: assign the target daemon process to the plurality of shard directories corresponding to the metadata server daemon process number according to the metadata server daemon process number.

[0069] Step S25: Bind the plurality of shard directories and the corresponding target daemon processes respectively, and allocate corresponding standby daemon processes to the plurality of shard directories respectively.

[0070] In this embodiment, after the preset file sharing directory is sliced ​​and the daemon process is assigned, the distributed file system can bind the several slice directories obtained after the preset file sharing directory is sliced ​​to the target daemon processes corresponding to the several slice directories. Specifically, the temporary lock attribute of the preset file sharing directory can be set based on the received configuration instruction to start the distributed strategy. When the temporary lock attribute of the preset file sharing directory is set, the preset file sharing directory will be forced to perform directory sharding to obtain several slice directories. After the metadata server daemon process number is assigned to the slice directory, the slice directory and the target daemon process corresponding to the corresponding metadata server daemon process number can be bound, and the several slice directories can be assigned to all current metadata server daemons. In this way, the load of the preset file sharing directory can be dispersed to prevent the directory from being overloaded and causing the system to crash when a large number of metadata operations are performed on the directory.

[0071] Step S26: If the target daemon process is abnormal, the backup daemon process corresponding to the target daemon process is enabled.

[0072] Among them, the specific implementation process of steps S21, S22, S24 and S26 can refer to the contents of the aforementioned embodiments, and will not be repeated here.

[0073] As can be seen from the above, the embodiment of the present application first slices the local preset file sharing directory based on the received configuration instructions to obtain a number of slice directories corresponding to the preset file sharing directory and a number of slice values ​​corresponding to the slice directory, and then determines the directory number corresponding to the preset file sharing directory, and respectively performs hash calculations on the several slice values ​​and the directory number to obtain a number of corresponding hash values, and determines the metadata server daemon numbers corresponding to the several slice directories according to the several hash values, and then allocates the target daemon to the several slice directories corresponding to the metadata server daemon number according to the metadata server daemon number, and binds the several slice directories and the corresponding target daemon respectively, and allocates corresponding backup daemons to the several slice directories, and finally, if the target daemon is abnormal, the backup daemon corresponding to the target daemon is enabled. In this way, through the above process of the embodiment of the present application, on the one hand, the preset file sharing directory is sharded to disperse the load of the preset file sharing directory, thereby preventing the directory load value from being too high and causing the system to crash when a large number of metadata operations are performed on the directory; on the other hand, the jump consistency hashing algorithm is used to partition the preset file sharing directory to minimize reallocation, and only migrate subtrees when allocating directory shards, reducing the number of subtree migrations, thereby reducing the cost of data migration, reducing the impact of business interruptions, and further efficiently realizing balanced management of metadata.

[0074] Accordingly, see Figure 5 As shown, the embodiment of the present application also provides a metadata balancing management device, which is applied to a distributed file system, including:

[0075] A directory calculation module 11, configured to slice a local preset file sharing directory based on the received configuration instruction to obtain a plurality of slice directories corresponding to the preset file sharing directory and a plurality of slice values ​​corresponding to the slice directories;

[0076] The number determination module 12 is used to perform hash calculations on the plurality of shard values ​​and the directory numbers corresponding to the preset file sharing directories to obtain corresponding plurality of hash values, and determine the metadata server daemon process numbers corresponding to the plurality of shard directories respectively through the plurality of hash values;

[0077] The process allocation module 13 is used to allocate target daemon processes corresponding to the metadata server daemon process numbers to the plurality of shard directories respectively, and allocate standby daemon processes to the plurality of shard directories respectively;

[0078] The process switching module 14 is used to enable the backup daemon process corresponding to the target daemon process if an abnormality occurs in the target daemon process.

[0079] As can be seen from the above, the present application firstly slices the local preset file sharing directory based on the received configuration instructions to obtain a number of slice directories corresponding to the preset file sharing directory and a number of slice values ​​corresponding to the slice directory, and then determines the directory number corresponding to the preset file sharing directory, and respectively performs hash calculations on the several slice values ​​and the directory number to obtain a number of corresponding hash values, and determines the metadata server daemon numbers corresponding to the several slice directories according to the several hash values, and then respectively allocates the target daemon to the several slice directories corresponding to the metadata server daemon number according to the metadata server daemon number, and respectively binds the several slice directories and the corresponding target daemon, and respectively allocates corresponding backup daemons to the several slice directories, and finally, if the target daemon has an abnormality, the backup daemon corresponding to the target daemon is enabled. In this way, through the above process of the embodiment of the present application, multiple target daemons are configured to achieve high availability of the metadata server, and target daemons corresponding to the metadata server daemon numbers are assigned to several shard directories respectively, so as to linearly expand the ability of the metadata server cluster to process metadata, and then efficiently achieve load balancing of the metadata server to improve the performance and stability of the distributed file system.

[0080] In some specific implementations, the metadata balance management device may further include:

[0081] A subdirectory creation unit, used to create a preset file sharing directory locally, and create a plurality of sharing service subdirectories in the preset file sharing directory;

[0082] The path configuration unit is used to write the several shared service sub-directories into a preset path configuration based on the received configuration instructions, so as to provide external file system services through the several shared service sub-directories; the preset path configuration is used to use the several shared service sub-directories as file systems provided externally.

[0083] In some specific implementations, the directory calculation module 11 may specifically include:

[0084] A directory sharding unit is used to determine whether a configuration instruction is received. If the configuration instruction is received, the preset file sharing directory is sharded according to a preset hash partitioning algorithm to obtain a plurality of shard directories and a plurality of shard values ​​corresponding to the plurality of shard directories; the plurality of shard values ​​are identifiers generated when the sharding of the preset file sharing directory is completed according to the preset hash partitioning algorithm and used to uniquely identify the plurality of shards.

[0085] In some specific implementations, the metadata balance management device may further include:

[0086] A conditional judgment unit, used to judge whether a preset daemon process quantity control parameter value is greater than the current quantity value of the target daemon process;

[0087] A stop unit is generated, which is used to stop generating a new metadata server daemon process number if the preset daemon process number control parameter value is not less than the current number value of the target daemon process.

[0088] In some specific implementations, the number determination module 12 may specifically include:

[0089] A first number determination unit, used to determine the directory number corresponding to the preset file sharing directory;

[0090] The second number determination unit is used to perform hash calculations on the plurality of shard values ​​and the directory numbers respectively to obtain a plurality of corresponding hash values, and determine the metadata server daemon process numbers corresponding to the plurality of shard directories respectively according to the plurality of hash values.

[0091] In some specific implementations, the process allocation module 13 may further include:

[0092] A first process allocation unit is used to allocate the target daemon process to a plurality of shard directories corresponding to the metadata server daemon process number according to the metadata server daemon process number respectively;

[0093] The second process allocation unit is used to bind the plurality of shard directories and the corresponding target daemon processes respectively, and allocate corresponding standby daemon processes to the plurality of shard directories respectively.

[0094] In some specific implementations, the process switching module 14 may specifically include:

[0095] The service takeover unit is used to take over the service of the target daemon process through the backup daemon process corresponding to the target daemon process and based on the metadata cache of the target daemon process synchronized in real time if the target daemon process is abnormal.

[0096] Furthermore, the present application also discloses an electronic device. Figure 6 This is a structural diagram of an electronic device 20 according to an exemplary embodiment, and the content in the figure cannot be regarded as any limitation on the scope of use of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the metadata balancing management method disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0097] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0098] In addition, the memory 22 as a carrier for resource storage may be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon may include an operating system 221, a computer program 222, etc., and the storage method may be temporary storage or permanent storage.

[0099] The operating system 221 is used to manage and control the hardware devices on the electronic device 20 and the computer program 222, which can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program that can be used to complete the metadata balancing management method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to complete other specific tasks.

[0100] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the metadata balancing management method disclosed above. The specific steps of the method can refer to the corresponding contents disclosed in the above embodiments, and will not be repeated here.

[0101] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0102] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0103] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0104] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0105] The technical solution provided by the present application is introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technicians in this field, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A metadata balance management method, characterized in that: Applicable to distributed file systems, including: Slice the local preset file sharing directory based on the received configuration instruction to obtain a plurality of slice directories corresponding to the preset file sharing directory and a plurality of slice values ​​corresponding to the slice directories; Performing hash calculations on the plurality of shard values ​​and the directory numbers corresponding to the preset file sharing directories respectively to obtain a plurality of corresponding hash values, and determining metadata server daemon process numbers corresponding to the plurality of shard directories respectively through the plurality of hash values; Allocate target daemons corresponding to the metadata server daemon process numbers to the plurality of shard directories respectively, and allocate standby daemons to the plurality of shard directories respectively; If the target daemon process is abnormal, the backup daemon process corresponding to the target daemon process is enabled.

2. The metadata balance management method according to claim 1, characterized in that: Before slicing the local preset file sharing directory based on the received configuration instruction, the method further includes: Creating a preset file sharing directory locally, and creating several sharing service subdirectories in the preset file sharing directory; Based on the received configuration instructions, the several shared service sub-directories are written into a preset path configuration to provide external file system services through the several shared service sub-directories; the preset path configuration is used to use the several shared service sub-directories as a file system provided externally.

3. The metadata balance management method according to claim 1, characterized in that: The slicing of the local preset file sharing directory based on the received configuration instruction to obtain a plurality of slicing directories corresponding to the preset file sharing directory and a plurality of slicing values ​​corresponding to the slicing directories includes: Determine whether a configuration instruction is received. If the configuration instruction is received, shard the preset file sharing directory according to a preset hash partitioning algorithm to obtain a number of shard directories and a number of shard values ​​corresponding to the number of shard directories; the number of shard values ​​are identifiers generated when the sharding of the preset file sharing directory is completed according to the preset hash partitioning algorithm and are used to uniquely identify the number of shards.

4. The metadata balance management method according to claim 1, characterized in that: The method further includes: performing hash calculations on the plurality of shard values ​​and the directory numbers corresponding to the preset file sharing directories to obtain corresponding plurality of hash values, and determining the metadata server daemon process numbers corresponding to the plurality of shard directories respectively through the plurality of hash values. Determine whether a preset daemon process quantity control parameter value is greater than the current quantity value of the target daemon process; If the preset daemon process quantity control parameter value is not less than the current target daemon process quantity value, the generation of new metadata server daemon process numbers is stopped.

5. The metadata balance management method according to claim 1, characterized in that: The method of performing hash calculations on the plurality of shard values ​​and the directory numbers corresponding to the preset file sharing directories to obtain corresponding plurality of hash values, and determining metadata server daemon process numbers corresponding to the plurality of shard directories respectively through the plurality of hash values, comprises: Determine the directory number corresponding to the preset file sharing directory; Hash calculations are performed on the plurality of shard values ​​and the directory numbers respectively to obtain a plurality of corresponding hash values, and metadata server daemon process numbers corresponding to the plurality of shard directories are determined according to the plurality of hash values.

6. The metadata balance management method according to claim 1, characterized in that: The step of respectively allocating target daemons corresponding to the metadata server daemon process numbers to the plurality of shard directories, and respectively allocating standby daemons to the plurality of shard directories comprises: Allocating the target daemon process to a plurality of shard directories corresponding to the metadata server daemon process number according to the metadata server daemon process number respectively; The plurality of shard directories and the corresponding target daemon processes are bound respectively, and corresponding standby daemon processes are allocated respectively to the plurality of shard directories.

7. The metadata balance management method according to any one of claims 1 to 6, characterized in that: If the target daemon process is abnormal, enabling the standby daemon process corresponding to the target daemon process includes: If the target daemon process is abnormal, the service of the target daemon process is taken over by the standby daemon process corresponding to the target daemon process based on the metadata cache of the target daemon process synchronized in real time.

8. A metadata balance management device, characterized in that: Applicable to distributed file systems, including: A directory calculation module, used for slicing a local preset file sharing directory based on the received configuration instruction to obtain a plurality of slicing directories corresponding to the preset file sharing directory and a plurality of slicing values ​​corresponding to the slicing directories; A number determination module is used to perform hash calculations on the plurality of shard values ​​and the directory numbers corresponding to the preset file sharing directories to obtain corresponding plurality of hash values, and determine the metadata server daemon process numbers corresponding to the plurality of shard directories respectively through the plurality of hash values; A process allocation module, used to allocate target daemon processes corresponding to the metadata server daemon process numbers to the plurality of shard directories respectively, and allocate standby daemon processes to the plurality of shard directories respectively; The process switching module is used to enable the backup daemon process corresponding to the target daemon process if an abnormality occurs in the target daemon process.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the metadata balancing management method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Used to store a computer program; wherein, when the computer program is executed by a processor, the metadata balancing management method as described in any one of claims 1 to 7 is implemented.

Images