Metadata balancing method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a metadata balancing method and device, computer equipment, a computer readable storage medium and a computer program product. Comprising the steps of obtaining a directory depth threshold value of a to-be-created directory; if the depth of the parent directory is smaller than or equal to a directory depth threshold value, selecting a metadata server from a metadata server list by adopting a polling strategy; if the depth of the parent directory is greater than a directory depth threshold value, selecting the metadata server with the highest weight according to weight information of each metadata server in the metadata server list; the weight information is information given according to the current running state of each metadata server; and sending a directory creation request of a to-be-created directory to the selected metadata server, and executing a directory creation operation. According to the method, different metadata servers are selected based on a fine strategy to store directory information, so that uniform distribution of metadata is realized, the load of each metadata server is reduced, and load balance and efficient response of the parallel file system are realized.

Description

Technical Field

[0001] This application relates to the technical field of distributed file storage, and in particular to a metadata balancing method, device, computer device, computer-readable storage medium, and computer program product. Background Art

[0002] In the fields of cloud computing and high performance computing (HPC), with the explosive growth of data volume and the increasing complexity of computing requirements, the traditional single-point metadata server (MDS) architecture has been difficult to meet the requirements of modern data centers for high availability, scalability, and performance. The concept of multi-MDS services emerged, aiming to build a distributed metadata management system to solve the bottleneck problem in large-scale parallel file systems. The core idea of multi-MDS services is to deploy multiple MDS instances in a cluster, and each MDS is responsible for managing and storing a part of the metadata, so as to achieve load balancing and fault redundancy. This architecture not only improves the overall throughput of the system, but also enhances the fault tolerance and dynamic adjustment capabilities of the system. The introduction of multi-MDS services has greatly improved the performance and reliability of file systems in cloud computing and high performance computing environments, providing a solid foundation for large-scale data processing and scientific computing. However, the implementation of multi-MDS services also faces many challenges, such as how to design an efficient metadata distribution strategy, how to optimize the data migration and synchronization mechanism across MDSs, and how to reduce latency while ensuring consistency. These are all problems that need to be solved urgently. Summary of the Invention

[0003] Based on this, in view of the above technical problems, it is necessary to provide a metadata balancing method, device, computer device, computer-readable storage medium, and computer program product, which can achieve uniform distribution of metadata, reduce the load of each metadata server, and achieve load balancing and efficient response of the parallel file system.

[0004] In a first aspect, this application provides a metadata balancing method, which is applied to a parallel file system including a plurality of metadata servers. The method includes:

[0005] Obtain the directory depth threshold of the directory to be created;

[0006] When the depth of the parent directory of the directory to be created is less than or equal to the directory depth threshold, select a metadata server from the metadata server list using a polling strategy;

[0007] When the depth of the parent directory of the directory to be created is greater than the directory depth threshold, obtain the weight information of each metadata server in the metadata server list; according to the weight information, select the metadata server with the highest weight; the weight information is the information assigned to the corresponding metadata server according to the current running state of each metadata server;

[0008] Send the directory creation request of the directory to be created to the selected metadata server, and perform the directory creation operation on the selected metadata server.

[0009] In one embodiment, obtaining the directory depth threshold of the directory to be created includes:

[0010] Obtain the directory depth threshold from the extended attributes of the directory to be created; or, obtain the directory depth threshold of the directory to be created from the in-memory directory object; the in-memory directory object is used to cache at least one directory depth threshold.

[0011] In one embodiment, the method further includes:

[0012] Receive a directory access request;

[0013] According to the directory access request, read the extended attributes of each directory, and parse out the directory depth threshold of each directory;

[0014] Cache the parsed directory depth threshold into the in-memory directory object.

[0015] In one embodiment, the method further includes:

[0016] Read the layout attributes of the parent directory; the layout attributes include associated metadata server information;

[0017] According to the associated metadata server information, determine the target metadata server where the parent directory is located;

[0018] When the current running state of the target metadata server meets the set conditions, perform a file creation operation on the target metadata server.

[0019] In one embodiment, the method further includes:

[0020] Check whether the file creation operation is successful;

[0021] If the file creation operation is successful, communicate with the target metadata server to confirm the consistency of the file metadata.

[0022] In one embodiment, the method further includes:

[0023] Receive the directory depth threshold update information of the target parent directory;

[0024] Synchronize the directory depth threshold update information to the metadata server associated with the target parent directory.

[0025] In a second aspect, the present application further provides a metadata balancing device, which is applied to a parallel file system. The parallel file system includes a plurality of metadata servers, and the device includes:

[0026] An acquisition module, configured to acquire the directory depth threshold of the directory to be created;

[0027] A selection module, configured to select a metadata server from the metadata server list by using a polling strategy when the depth of the parent directory of the directory to be created is less than or equal to the directory depth threshold;

[0028] The selection module is further configured to, when the depth of the parent directory of the directory to be created is greater than the directory depth threshold, acquire the weight information of each metadata server in the metadata server list; and select the metadata server with the highest weight according to the weight information; the weight information is information assigned to the corresponding metadata server according to the current operating status of each metadata server;

[0029] A directory creation module, configured to send a directory creation request of the directory to be created to the selected metadata server and perform a directory creation operation on the selected metadata server.

[0030] In a third aspect, the present application further provides a computer device, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the steps of the method described in the first aspect above are implemented.

[0031] In a fourth aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in the first aspect above are implemented.

[0032] In a fifth aspect, the present application further provides a computer program product, including a computer program. When the computer program is executed by a processor, the steps of the method described in the first aspect above are implemented.

[0033] The above metadata balancing method, device, computer device, computer-readable storage medium, and computer program product obtain a directory depth threshold for a directory to be created; when the depth of the parent directory of the directory to be created is less than or equal to the directory depth threshold, a metadata server is selected from the list of metadata servers using a polling strategy; when the depth of the parent directory of the directory to be created is greater than the directory depth threshold, weight information of each metadata server in the list of metadata servers is obtained; according to the weight information, the metadata server with the highest weight is selected; the weight information is information assigned to the corresponding metadata server according to the current operating state of each metadata server; a directory creation request for the directory to be created is sent to the selected metadata server, and a directory creation operation is performed on the selected metadata server. Through the above method, during the directory creation process, different metadata servers are selected based on a refined strategy to store directory information, so as to achieve uniform distribution of metadata, reduce the load on each metadata server, and achieve load balancing and efficient response of the parallel file system. Reasonably allocating the resources of the metadata server according to the current operating state of each metadata server can avoid waste of resources. Furthermore, it can improve the overall performance and stability of the parallel file system and meet the high requirements for large-scale data storage and access in the public cloud environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following will briefly introduce the drawings required for use in the description of the embodiments of the present application or related technologies. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other related drawings can also be obtained based on these drawings.

[0035] Figure 1 It is an application environment diagram of the metadata balancing method in an embodiment;

[0036] Figure 2 It is a flowchart of the metadata balancing method in an embodiment;

[0037] Figure 3 It is a flowchart of directory creation in an embodiment;

[0038] Figure 4 It is a flowchart of file creation in an embodiment;

[0039] Figure 5 It is a flowchart of directory depth threshold configuration in an embodiment;

[0040] Figure 6 It is a structural block diagram of the metadata balancing device in an embodiment;

[0041] Figure 7Internal structure diagram of a computer device in an embodiment. Detailed implementation

[0042] In order to make the purpose, technical solution and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0043] To facilitate the understanding of the embodiments provided by the present application, some terms that appear in the specific description process are introduced below:

[0044] HPFS (High-Performance File System): A file system designed specifically to meet the requirements of high-performance computing environments. The design goals focus on high performance, high reliability, large storage capacity, parallel data access, metadata optimization, etc. It aims to provide a stable, efficient, and scalable file storage solution for high-performance computing environments to meet the stringent requirements of large-scale data processing and high-performance computing.

[0045] MDS (Metadata Server): Plays a key role in a distributed file system. It is mainly responsible for managing and storing the metadata of the file system, including the attributes, permissions, location information of files and directories, and the namespace of the file system, etc.

[0046] HPC (High-Performance Computing): Plays a core role in fields such as scientific research, engineering analysis, and big data processing. It mainly relies on powerful computing capabilities, efficient parallel algorithms, and high-speed network communications to achieve the rapid processing of large-scale data sets and the efficient execution of complex computing tasks.

[0047] The metadata balancing method provided by the embodiments of the present application can be applied to an application environment as Figure 1 shown. Among them, the parallel file system 102 includes several metadata servers 104, and the parallel file system 102 is communicatively connected to several metadata servers 104. The parallel file system 102 selects a metadata server 104 for directory creation operations based on the metadata balancing method provided by the present application to achieve the uniform distribution of metadata. The metadata server 104 is used to store and manage the metadata of the parallel file system 102. The parallel file system 102 provided by the embodiments of the present application can be an HPFS system, or other parallel file systems, such as Lustre. This embodiment is described by taking the HPFS system as an example.

[0048] In an exemplary embodiment, as Figure 2As shown, a metadata balancing method is provided. Taking the parallel file system 102 in Figure 1 as an example for illustration, the parallel file system includes a plurality of metadata servers, and the method includes:

[0049] Step 202, obtain the directory depth threshold of the directory to be created.

[0050] Wherein, the directory to be created refers to the directory structure planned to be created. Optionally, the parallel file system receives a directory creation request and determines the directory to be created based on this directory creation request. The directory depth threshold refers to the limit value allowed for the hierarchical depth of the parent directory of the directory to be created. Optionally, according to system capacity and performance requirements, configure the directory depth threshold of each parent directory.

[0051] Exemplarily, the parallel file system is provided with a target storage area, which is used to store the directory depth threshold corresponding to each parent directory. Step 202 includes: determining the parent directory identifier of the directory to be created, and searching for the corresponding directory depth threshold in the target storage area based on this parent directory identifier.

[0052] Step 204, when the depth of the parent directory of the directory to be created is less than or equal to the directory depth threshold, select a metadata server from the list of metadata servers by using a polling strategy.

[0053] Wherein, before creating a directory, evaluate whether the depth of the parent directory of the directory to be created is lower than the directory depth threshold to obtain an evaluation result, and this evaluation result is used to determine the subsequent MDS selection strategy, and the MDS selection strategies corresponding to different evaluation results are different.

[0054] If the depth of the parent directory of the directory to be created is less than or equal to the directory depth threshold, it indicates that the depth of the parent directory of the directory to be created is relatively low. The HPFS system uses a polling strategy to select an MDS, which means that the HPFS system will sequentially traverse the list of available metadata servers and evenly distribute the directory creation requests to each MDS, so as to balance the load between MDSs and avoid a single MDS being overloaded (i.e., processing too many requests) and becoming a performance bottleneck.

[0055] Step 206, when the depth of the parent directory of the directory to be created is greater than the directory depth threshold, obtain the weight information of each metadata server in the list of metadata servers; select the metadata server with the highest weight according to the weight information; the weight information is the information assigned to the corresponding metadata server according to the current running state of each metadata server.

[0056] Among them, if the depth of the parent directory of the directory to be created is greater than the directory depth threshold, it indicates that the depth of the parent directory of the directory to be created is relatively high. Processing the request for the directory to be created will consume the resources of the MDS and bring a large load to the MDS. The HPFS system uses a weighted random strategy to select the MDS. In this strategy, a weight information is assigned to each MDS according to its current running state. When the HPFS system selects an MDS, it will determine the selection probability according to the weight information of each MDS, so as to be more inclined to select the MDS with a better current running state and achieve a more refined load balancing. The current running state includes at least one of the current load information, historical performance information, running stability, and running priority. Optionally, the weight information of each MDS is dynamically adjusted according to its current running state. Optionally, the higher the current load of the MDS, the lower its corresponding weight; the lower the current load of the MDS, the higher its corresponding weight.

[0057] Step 208: Send the directory creation request of the directory to be created to the selected metadata server and perform the directory creation operation on the selected metadata server.

[0058] Among them, the directory creation request of the directory to be created is sent to the metadata server selected in Step 204 or Step 206, and the HPFS system performs the directory creation operation on this metadata server. The directory creation operation includes storing the relevant metadata of the directory to be created on the selected MDS to ensure the correctness and integrity of the directory information.

[0059] Exemplarily, referring to Figure 3 , the directory creation process includes: the HPFS system determines whether the directory depth of the parent directory is greater than the directory depth threshold; if so, it uses the weighted random strategy to select the metadata server; if not, it uses the polling strategy to select the metadata server; and creates a directory on the selected metadata server.

[0060] In the above metadata balancing method, obtain the directory depth threshold of the directory to be created; when the depth of the parent directory of the directory to be created is less than or equal to the directory depth threshold, select a metadata server from the list of metadata servers using a polling strategy; when the depth of the parent directory of the directory to be created is greater than the directory depth threshold, obtain the weight information of each metadata server in the list of metadata servers; according to the weight information, select the metadata server with the highest weight; the weight information is the information assigned to the corresponding metadata server according to the current running state of each metadata server; send the directory creation request of the directory to be created to the selected metadata server, and perform the directory creation operation on the selected metadata server. In the above manner, during the directory creation process, different metadata servers are selected based on a refined strategy to store directory information, so as to achieve the uniform distribution of metadata, reduce the load of each metadata server, and achieve the load balancing and efficient response of the parallel file system. Reasonably allocating the resources of the metadata server according to the current running state of each metadata server can avoid resource waste. Furthermore, it can improve the overall performance and stability of the parallel file system and meet the high requirements for large-scale data storage and access in the public cloud environment.

[0061] In an exemplary embodiment, step 202 includes: obtaining the directory depth threshold from the extended attributes of the directory to be created; or, obtaining the directory depth threshold of the directory to be created from the in-memory directory object; the in-memory directory object is used to cache at least one directory depth threshold.

[0062] Among them, in the HPFS system, the directory depth threshold is an important parameter attached to the directory as an extended attribute. Before creating a directory, it is necessary to first determine the directory depth threshold of its parent directory and save it in the form of an extended attribute. Ensure that each directory carries information about its depth, which is convenient for subsequent selection of the MDS based on this.

[0063] Optionally, for the read directory depth threshold, the HPFS system caches it in the in-memory directory object. During the subsequent directory creation process, the directory depth threshold of the directory to be created can be read from this in-memory directory object.

[0064] Exemplarily, obtain the directory depth threshold of the directory to be created from the in-memory directory object; if not hit, obtain the directory depth threshold from the extended attributes of the directory to be created.

[0065] In an exemplary embodiment, the method further includes: receiving a directory access request; according to the directory access request, reading the extended attributes of each directory, and parsing out the directory depth threshold of each directory; caching the parsed directory depth threshold into the in-memory directory object.

[0066] Among them, when a user accesses a directory, the HPFS system reads the extended attributes of each directory during the directory lookup process, parses out the directory depth threshold from them, and caches it in the in-memory directory object for quick access when creating subdirectories or performing other operations, reducing disk I / O operations and improving the system response speed.

[0067] In an exemplary embodiment, the method further includes: reading the layout attributes of the parent directory; the layout attributes include associated metadata server information; determining the target metadata server where the parent directory is located according to the associated metadata server information; and performing a file creation operation on the target metadata server when the current running state of the target metadata server meets the set conditions.

[0068] Among them, at the beginning of file creation, the HPFS system first reads the layout attributes of the parent directory, which include the directory depth information and the metadata server information associated with the parent directory, providing data support for subsequent operations. Based on the layout information of the parent directory, determine the target metadata server where the parent directory is located, and judge whether the current running state of the target metadata server meets the set conditions. Among them, the preset conditions are used to determine whether the target metadata server is available. Specifically, check whether the target metadata server is online and determine the load situation of the target metadata server. If the target metadata server is online and the current load of the target metadata server is lower than the set threshold, it is determined that the current running state of the target metadata server meets the set conditions, that is, the target metadata server is available.

[0069] If it is detected that the current running state of the target metadata server meets the set conditions, the HPFS system will perform a file creation operation on the target metadata server. The file creation operation includes registering the metadata of the file on the target metadata server, such as file name, size, permissions, creation time, etc. information, to ensure that the metadata of the file is accurately recorded.

[0070] In an exemplary embodiment, the method further includes: checking whether the file creation operation is successful; if the file creation operation is successful, communicating with the target metadata server to confirm the consistency of the file metadata.

[0071] Among them, after the file creation is completed, the HPFS system will perform the final confirmation step: check whether the file creation operation is successful to ensure that there are no permission issues or insufficient disk space. Communicate with the relevant MDS to confirm the consistency of the file metadata to prevent data inconsistency caused by network latency or MDS failure.

[0072] In an alternative implementation, refer to Figure 4, the file creation process includes: obtaining the layout attributes of the parent directory; selecting the MDS where the parent directory is located; creating a file on the selected MDS; and confirming the file creation status.

[0073] In an exemplary embodiment, the method further includes: receiving directory depth threshold update information of a target parent directory; and synchronizing the directory depth threshold update information to a metadata server associated with the target parent directory.

[0074] Among them, as the directory hierarchy deepens, it may be necessary to adjust the directory depth threshold to adapt to system changes. The HPFS system supports dynamic updates of the directory depth threshold, that is, when the directory structure changes or the system load changes, the directory depth threshold can be reconfigured. In addition, to ensure the consistency of system data and the consistency of global policies, any modification to the directory depth threshold will be synchronized to all relevant MDSs through an effective communication mechanism.

[0075] In an alternative implementation, with reference to Figure 5 , the directory depth threshold configuration process includes: 1. Initializing and storing the directory depth threshold. That is, in the HPFS system, the directory depth threshold is an important parameter attached to the directory as an extended attribute. Before creating a directory, it is necessary to first determine the directory depth threshold of its parent directory and save it in the form of an extended attribute. Ensure that each directory carries information about its depth, which is convenient for subsequent selection of MDSs based on this. 2. Threshold acquisition and caching during directory access. That is, when a user accesses a directory, the HPFS system reads the extended attributes of each directory during the directory lookup process, parses the directory depth threshold from them, and caches it in the in-memory directory object for quick access when creating subdirectories or performing other operations. 3. Update and synchronization of the directory depth threshold. That is, the HPFS system supports dynamic updates of the directory depth threshold. When the directory structure changes or the system load changes, the directory depth threshold can be reconfigured and synchronized to all relevant MDSs through an effective communication mechanism to ensure the consistency of system data and the consistency of global policies.

[0076] Exemplarily, the HPFS parallel file system applying the method of the embodiments of the present application aims to provide fast and stable data storage and access services for scenarios such as large-scale scientific computing and data analysis. Among them, in order to effectively manage the huge file and directory structure, the HPFS parallel file system adopts a multi-element data server architecture, that is, multiple metadata servers (MDS) cooperate together to improve the scalability and reliability of the system. In the HPFS parallel file system, for the directory creation operation, a metadata server selection strategy based on directory depth is implemented. It not only solves the challenges of metadata management in large-scale file systems, but also provides strong support for high-performance computing environments. The implemented directory creation process includes: 1. Define the depth threshold: According to the system capacity and performance requirements, define the directory depth threshold. For example, the defined directory depth threshold is 5. 2. Implement the polling strategy: For directory creation requests with a depth less than or equal to 5, the HPFS parallel file system selects the MDS in a polling manner. This means that each request will be allocated to different MDSs in turn, thus ensuring the load balance among the MDSs. 3. Implement the weighted random strategy: When the directory depth exceeds 5, the HPFS parallel file system will select the MDS in a weighted random manner. Optionally, the weights used are dynamically calculated according to the real-time load conditions of each MDS. The lower the load of an MDS, the higher the weight it obtains, so it is more likely to be selected to receive new requests.

[0077] By implementing the metadata balancing method of the embodiments of the present application, the metadata management ability of the HPFS parallel file system has been significantly improved: 1. Improve the system response speed: Load balancing enables the MDS to respond to client requests faster, reducing the waiting time. 2. Enhance the system stability: The dynamic MDS selection strategy effectively prevents single-point failures and improves the overall stability of the system. 3. Optimize the resource utilization efficiency: Through intelligent scheduling, the resources of the MDS are more reasonably allocated, avoiding resource waste and improving the overall performance.

[0078] It should be understood that although the steps in the flowcharts involved in the above-mentioned embodiments are shown in sequence according to the indication of the arrows, these steps do not necessarily have to be executed in the order indicated by the arrows. Unless there is a clear indication in this article, the execution of these steps does not have a strict order limit, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments may include multiple steps or multiple stages. These steps or stages do not necessarily have to be executed at the same time, but can be executed at different times. The execution order of these steps or stages does not necessarily have to be sequential, but can be executed alternately or in turn with at least a part of other steps or steps or stages in other steps.

[0079] Based on the same inventive concept, an embodiment of the present application further provides a metadata balancing device for implementing the above-mentioned metadata balancing method. The solution provided by this device for solving problems is similar to the solution described in the above method. Therefore, the specific limitations in one or more embodiments of the metadata balancing device provided below can refer to the limitations on the metadata balancing method in the above text and will not be repeated here.

[0080] In an exemplary embodiment, as Figure 6 shown, a metadata balancing device is provided. The device is applied to a parallel file system, and the parallel file system includes a plurality of metadata servers. The device includes:

[0081] An acquisition module 602, configured to acquire the directory depth threshold of the directory to be created.

[0082] A selection module 604, configured to, when the depth of the parent directory of the directory to be created is less than or equal to the directory depth threshold, select a metadata server from the metadata server list by using a polling strategy.

[0083] The selection module 604 is further configured to, when the depth of the parent directory of the directory to be created is greater than the directory depth threshold, acquire the weight information of each metadata server in the metadata server list; select the metadata server with the highest weight according to the weight information; the weight information is information assigned to the corresponding metadata server according to the current running state of each metadata server.

[0084] A directory creation module 606, configured to send the directory creation request of the directory to be created to the selected metadata server and perform a directory creation operation on the selected metadata server.

[0085] In the above metadata balancing device, a directory depth threshold of a directory to be created is obtained; when the depth of the parent directory of the directory to be created is less than or equal to the directory depth threshold, a metadata server is selected from the list of metadata servers by using a polling strategy; when the depth of the parent directory of the directory to be created is greater than the directory depth threshold, weight information of each metadata server in the list of metadata servers is obtained; according to the weight information, the metadata server with the highest weight is selected; the weight information is information assigned to the corresponding metadata server according to the current running state of each metadata server; a directory creation request of the directory to be created is sent to the selected metadata server, and a directory creation operation is performed on the selected metadata server. By the above method, during the directory creation process, different metadata servers are selected based on a fine-grained strategy to store directory information, so as to achieve uniform distribution of metadata, reduce the load of each metadata server, and achieve load balancing and efficient response of the parallel file system. Reasonably allocating the resources of the metadata server according to the current running state of each metadata server can avoid waste of resources. Furthermore, the overall performance and stability of the parallel file system are improved, meeting the high requirements for large-scale data storage and access in the public cloud environment.

[0086] In an exemplary embodiment, the obtaining module 602 is further configured to obtain the directory depth threshold from the extended attributes of the directory to be created; or obtain the directory depth threshold of the directory to be created from the in-memory directory object; the in-memory directory object is used to cache at least one directory depth threshold.

[0087] In an exemplary embodiment, the metadata balancing device further includes a directory access module, configured to receive a directory access request; according to the directory access request, read the extended attributes of each directory, and parse out the directory depth threshold of each directory; cache the parsed directory depth threshold into the in-memory directory object.

[0088] In an exemplary embodiment, the metadata balancing device further includes a file creation module, configured to read the layout attributes of the parent directory; the layout attributes include associated metadata server information; according to the associated metadata server information, determine the target metadata server where the parent directory is located; when the current running state of the target metadata server meets the set conditions, perform a file creation operation on the target metadata server.

[0089] In an exemplary embodiment, the file creation module is further configured to check whether the file creation operation is successful; if the file creation operation is successful, communicate with the target metadata server to confirm the consistency of the file metadata.

[0090] In an exemplary embodiment, the metadata balancing device further includes a threshold update module, which is configured to receive directory depth threshold update information of a target parent directory; and synchronize the directory depth threshold update information to a metadata server associated with the target parent directory.

[0091] Each module in the above metadata balancing device can be implemented in whole or in part by software, hardware, and their combination. The above modules can be embedded in the processor of the computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so as to facilitate the processor to call and execute the operations corresponding to the above respective modules.

[0092] In an exemplary embodiment, a computer device is provided. The computer device can be a server, and its internal structure diagram can be as Figure 7 shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O), and a communication interface. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used for exchanging information between the processor and external devices. The communication interface of the computer device is used for communicating with external terminals through a network connection. When the computer program is executed by the processor, a metadata balancing method is implemented.

[0093] Those skilled in the art can understand that Figure 7 the structure shown in

[0094] In an exemplary embodiment, a computer device is provided, including a memory and a processor. A computer program is stored in the memory. When the processor executes the computer program, the following steps are implemented: obtaining a directory depth threshold of a directory to be created; when the depth of the parent directory of the directory to be created is less than or equal to the directory depth threshold, selecting a metadata server from a list of metadata servers by using a polling strategy; when the depth of the parent directory of the directory to be created is greater than the directory depth threshold, obtaining weight information of each metadata server in the list of metadata servers; selecting the metadata server with the highest weight according to the weight information; the weight information is information assigned to the corresponding metadata server according to the current running state of each metadata server; sending a directory creation request of the directory to be created to the selected metadata server, and performing a directory creation operation on the selected metadata server.

[0095] In an embodiment, when the processor executes the computer program, the following steps are further implemented: obtaining the directory depth threshold from the extended attributes of the directory to be created; or, obtaining the directory depth threshold of the directory to be created from a memory directory object; the memory directory object is used to cache at least one directory depth threshold.

[0096] In an embodiment, when the processor executes the computer program, the following steps are further implemented: receiving a directory access request; according to the directory access request, reading the extended attributes of each directory, and parsing out the directory depth threshold of each directory; caching the parsed directory depth threshold into the memory directory object.

[0097] In an embodiment, when the processor executes the computer program, the following steps are further implemented: reading the layout attribute of the parent directory; the layout attribute includes associated metadata server information; determining the target metadata server where the parent directory is located according to the associated metadata server information; when the current running state of the target metadata server meets the set conditions, performing a file creation operation on the target metadata server.

[0098] In an embodiment, when the processor executes the computer program, the following steps are further implemented: checking whether the file creation operation is successful; if the file creation operation is successful, communicating with the target metadata server to confirm the consistency of the file metadata.

[0099] In an embodiment, when the processor executes the computer program, the following steps are further implemented: receiving directory depth threshold update information of a target parent directory; synchronizing the directory depth threshold update information to the metadata servers related to the target parent directory.

[0100] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented: obtaining a directory depth threshold of a directory to be created; when the depth of the parent directory of the directory to be created is less than or equal to the directory depth threshold, selecting a metadata server from a list of metadata servers by using a polling strategy; when the depth of the parent directory of the directory to be created is greater than the directory depth threshold, obtaining weight information of each metadata server in the list of metadata servers; selecting the metadata server with the highest weight according to the weight information; the weight information is information assigned to the corresponding metadata server according to the current running state of each metadata server; sending a directory creation request of the directory to be created to the selected metadata server, and performing a directory creation operation on the selected metadata server.

[0101] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented: obtaining the directory depth threshold from the extended attributes of the directory to be created; or, obtaining the directory depth threshold of the directory to be created from a memory directory object; the memory directory object is used to cache at least one directory depth threshold.

[0102] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented: receiving a directory access request; according to the directory access request, reading the extended attributes of each directory, and parsing out the directory depth threshold of each directory; caching the parsed directory depth threshold into the memory directory object.

[0103] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented: reading the layout attribute of the parent directory; the layout attribute includes associated metadata server information; determining the target metadata server where the parent directory is located according to the associated metadata server information; when the current running state of the target metadata server meets a set condition, performing a file creation operation on the target metadata server.

[0104] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented: checking whether the file creation operation is successful; if the file creation operation is successful, communicating with the target metadata server to confirm the consistency of the file metadata.

[0105] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented: receiving directory depth threshold update information of a target parent directory; synchronizing the directory depth threshold update information to the metadata servers related to the target parent directory.

[0106] In one embodiment, a computer program product is provided, including a computer program which, when executed by a processor, implements the following steps: obtaining a directory depth threshold of a directory to be created; when the depth of the parent directory of the directory to be created is less than or equal to the directory depth threshold, selecting a metadata server from a list of metadata servers using a polling strategy; when the depth of the parent directory of the directory to be created is greater than the directory depth threshold, obtaining weight information of each metadata server in the list of metadata servers; selecting the metadata server with the highest weight according to the weight information; the weight information is information assigned to the corresponding metadata server according to the current running state of each metadata server; sending a directory creation request of the directory to be created to the selected metadata server and performing a directory creation operation on the selected metadata server.

[0107] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: obtaining the directory depth threshold from the extended attributes of the directory to be created; or obtaining the directory depth threshold of the directory to be created from a memory directory object; the memory directory object is used to cache at least one directory depth threshold.

[0108] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: receiving a directory access request; according to the directory access request, reading the extended attributes of each directory and parsing out the directory depth threshold of each directory; caching the parsed directory depth threshold into the memory directory object.

[0109] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: reading the layout attribute of the parent directory; the layout attribute includes associated metadata server information; determining the target metadata server where the parent directory is located according to the associated metadata server information; when the current running state of the target metadata server meets the set conditions, performing a file creation operation on the target metadata server.

[0110] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: checking whether the file creation operation is successful; if the file creation operation is successful, communicating with the target metadata server to confirm the consistency of the file metadata.

[0111] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: receiving directory depth threshold update information of a target parent directory; synchronizing the directory depth threshold update information to the metadata servers related to the target parent directory.

[0112] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data need to comply with relevant regulations.

[0113] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, the memories, databases, or other media mentioned in the various embodiments provided in this application can all include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the various embodiments provided in this application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., without limitation. The processors involved in the various embodiments provided in this application can be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, data processing logics based on quantum computing, artificial intelligence (AI) processors, etc., without limitation.

[0114] The technical features of the above embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in this application.

[0115] The above-described embodiments merely represent several implementation manners of this application. The description is relatively specific and detailed, but it should not be construed as a limitation on the patent scope of this application. It should be noted that for those of ordinary skill in the art, without departing from the concept of this application, several modifications and improvements can still be made, and these all belong to the protection scope of this application. Therefore, the protection scope of this application shall be subject to the appended claims.

Claims

1. A metadata balancing method, characterized in that, The method is applied to a parallel file system, which includes a plurality of metadata servers. The method includes: Obtaining a directory depth threshold of a directory to be created; When the depth of the parent directory of the directory to be created is less than or equal to the directory depth threshold, selecting a metadata server from the metadata server list by using a polling strategy; When the depth of the parent directory of the directory to be created is greater than the directory depth threshold, obtaining weight information of each metadata server in the metadata server list; and selecting the metadata server with the highest weight according to the weight information; the weight information is information assigned to the corresponding metadata server according to the current running state of each metadata server; Sending a directory creation request of the directory to be created to the selected metadata server, and performing a directory creation operation on the selected metadata server.

2. The method according to claim 1, wherein The obtaining of the directory depth threshold of the directory to be created includes: Obtaining the directory depth threshold from the extended attributes of the directory to be created; or obtaining the directory depth threshold of the directory to be created from a memory directory object; the memory directory object is used to cache at least one directory depth threshold.

3. The method according to claim 2, wherein The method further includes: Receiving a directory access request; According to the directory access request, reading the extended attributes of each directory and parsing out the directory depth threshold of each directory; Caching the parsed directory depth threshold into the memory directory object.

4. The method according to claim 1, wherein The method further includes: Reading the layout attribute of the parent directory; the layout attribute includes associated metadata server information; Determining the target metadata server where the parent directory is located according to the associated metadata server information; When the current running state of the target metadata server meets the set conditions, performing a file creation operation on the target metadata server.

5. The method according to claim 4, wherein The method further includes: Checking whether the file creation operation is successful; If the file creation operation is successful, communicating with the target metadata server to confirm the consistency of the file metadata.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: Receiving directory depth threshold update information of a target parent directory; Synchronizing the directory depth threshold update information to the metadata servers related to the target parent directory.

7. A metadata balancing device, characterized in that The apparatus is applied to a parallel file system, which includes a plurality of metadata servers. The apparatus includes: An obtaining module, configured to obtain a directory depth threshold of a directory to be created; A selection module, configured to, when the depth of the parent directory of the directory to be created is less than or equal to the directory depth threshold, select a metadata server from the metadata server list by using a polling strategy; The selection module is further configured to, when the depth of the parent directory of the directory to be created is greater than the directory depth threshold, obtain weight information of each metadata server in the metadata server list; and select the metadata server with the highest weight according to the weight information; the weight information is information assigned to the corresponding metadata server according to the current running state of each metadata server; A directory creation module, configured to send a directory creation request for the directory to be created to a selected metadata server and perform a directory creation operation on the selected metadata server.

8. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images