Implementation of directory-level QoS functionality in distributed file systems

Abstract

This application provides a method, apparatus, device, and medium for implementing directory-level QoS functionality in a distributed file system. The method uses the dmclock algorithm to implement distributed QoS, avoiding the complexity and performance issues introduced by a unified QoS service. It supports QoS functionality for both IOPS and bandwidth, achieving directory-level QoS for the distributed file system. This facilitates multi-tenant isolation and ensures service quality for each tenant. It eliminates the need for additional QoS service components, reducing system complexity; it also reduces the impact of additional services on IO requests, increasing service stability; and it minimizes IO request latency because service IO requests do not require prior QoS service requests.

Description

Technical Field

[0001] This application relates to the field of distributed file storage technology, and in particular to a method, apparatus, device and medium for implementing QoS functionality at the directory level of a distributed file system. Background Technology

[0002] QoS is an abbreviation for Quality of Service. It originates from network technology and is used to solve problems such as network latency and congestion, enabling the provision of better service capabilities for specified network communications.

[0003] In storage systems, QoS is used to ensure a certain level of storage service quality, and its main functions are as follows:

[0004] Provide higher quality of service (such as IOPS and bandwidth) for high-priority services;

[0005] Control resource contention to prevent multiple users in the same storage cluster from competing for IOPS and bandwidth resources, ensuring that users' services are not affected by resource contention.

[0006] Currently, common QoS algorithms include the token bucket algorithm, the leaky bucket algorithm, and the mclock algorithm. However, these algorithms can only implement QoS functionality on a single machine. In distributed file storage scenarios with multiple servers, they can only limit the resource usage of a single server.

[0007] In distributed file storage scenarios, a file system contains multiple servers, and these servers need to provide QoS functionality to different clients, requiring the implementation of distributed QoS.

[0008] Currently, the main method for implementing distributed QoS is a unified QoS service. The principle of a QoS service is that before serving a client's request, the storage service sends a request to a unified QoS service. The QoS service then uses a single-machine QoS algorithm to determine if it can provide the service and returns the result to the storage service. The disadvantages of using a QoS service to implement distributed QoS are as follows:

[0009] The introduction of additional QoS service components increases the complexity of the system;

[0010] The server needs to access the QoS service before processing IO requests, and its strong dependence on the QoS service has a certain impact on the stability of the storage service.

[0011] The server adds a QoS service network request before processing IO, which affects IO latency.

[0012] Therefore, how to implement directory-level QoS functionality in a distributed file system has become an urgent technical problem to be solved. Summary of the Invention

[0013] This application provides a method, apparatus, device, and storage medium for implementing directory-level QoS functionality in a distributed file system, aiming to achieve directory-level QoS functionality in a distributed file system.

[0014] Firstly, this application provides a method for implementing QoS functionality at the directory level in a distributed file system, the method comprising:

[0015] When the client sends an IO request to the server, the request order corresponding to the IO request is obtained;

[0016] Based on the dmclock algorithm and the request order, calculate the QoS metric corresponding to the IO request and obtain the QoS limitation scheme of the current distributed file system.

[0017] Obtain the file system subdirectories currently mounted by the distributed file system;

[0018] Based on the current QoS limiting scheme of the distributed file system, QoS limiting indicators are set for the subdirectories of the file system to implement a directory-level QoS scheme for the distributed file system.

[0019] Secondly, this application also provides a distributed file system directory-level QoS function implementation device, the distributed file system directory-level QoS function implementation device comprising:

[0020] The request order acquisition module is used to acquire the request order corresponding to the IO request when the client sends an IO request to the server.

[0021] The metric calculation module is used to calculate the QoS metric corresponding to the IO request based on the dmclock algorithm and the request order, and to obtain the QoS limitation scheme of the current distributed file system.

[0022] The subdirectory acquisition module is used to acquire the subdirectories of the file system currently mounted by the distributed file system;

[0023] The directory-level QoS scheme implementation module is used to set QoS limit indicators for the subdirectories of the file system based on the current QoS limit scheme of the distributed file system, so as to implement a directory-level QoS scheme for the distributed file system.

[0024] Thirdly, this application also provides a computer device, the computer device including a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, it implements the steps of the distributed file system directory-level QoS function implementation method as described above.

[0025] Fourthly, this application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps of the distributed file system directory-level QoS function implementation method described above.

[0026] This application provides a method, system, device, equipment, and storage medium for implementing directory-level QoS functionality in a distributed file system. The method includes: obtaining the request order corresponding to the IO request when the client sends an IO request to the server; calculating the QoS metric corresponding to the IO request based on the dmclock algorithm and the request order to obtain the QoS constraint scheme for the current distributed file system; obtaining the file system subdirectories mounted by the current distributed file system; and setting QoS constraint metrics for the file system subdirectories based on the current distributed file system QoS constraint scheme to implement a directory-level QoS scheme for the distributed file system. Through the above method, this application implements distributed QoS functionality using the dmclock algorithm, avoiding the complexity and performance issues introduced by a unified QoS service. It also supports IOPS and bandwidth QoS functionality, implementing directory-level QoS functionality for the distributed file system. This facilitates multi-tenant isolation of the distributed file system and ensures the service quality of each tenant. It eliminates the need for additional QoS service components, reducing system complexity; it also reduces the impact of additional services on IO requests, increasing service stability; and it reduces IO request latency because service IO requests do not require prior requests to the QoS service. Attached Figure Description

[0027] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 A flowchart illustrating the first embodiment of a distributed file system directory-level QoS functionality implementation method provided in this application;

[0029] Figure 2This is a schematic diagram of the enqueue process for the mclock algorithm provided in the embodiments of this application;

[0030] Figure 3 This is a schematic diagram of the dequeue process of the mclock algorithm provided in the embodiments of this application;

[0031] Figure 4 This is a schematic diagram illustrating the order in which Client1 sends requests according to an embodiment of this application;

[0032] Figure 5 A flowchart illustrating a second embodiment of a distributed file system directory-level QoS functionality implementation method provided in this application;

[0033] Figure 6 This is a schematic diagram of the mounting process of a distributed file system provided in an embodiment of this application;

[0034] Figure 7 A schematic diagram illustrating the mounting process of a distributed file system self-directory provided in an embodiment of this application;

[0035] Figure 8 A schematic diagram illustrating the implementation process of the subdirectory QoS scheme provided in this application embodiment;

[0036] Figure 9 This is a schematic diagram of the structure of a first embodiment of a distributed file system directory-level QoS function implementation device provided in this application;

[0037] Figure 10 This is a schematic block diagram of the structure of a computer device provided in an embodiment of this application.

[0038] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0039] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0040] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0041] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0042] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of a distributed file system directory-level QoS function implementation method provided in this application.

[0043] like Figure 1 As shown, the method for implementing QoS functionality at the directory level of this distributed file system includes steps S101 to S104.

[0044] S101. When the client sends an IO request to the server, obtain the request order corresponding to the IO request;

[0045] Generally, the mclock algorithm tags each request upon arrival. These tags include three types: Reservation tag, Proportional tag (also called Weight tag), and Limit tag. Below is the formula for tag calculation:

[0046]

[0047]

[0048]

[0049] The meanings of each data point are as follows:

[0050]

[0051] The mclock algorithm consists of two processes: enqueueing and dequeueing.

[0052] In one embodiment, the enqueue process of the mclock algorithm is as follows: Figure 2As shown, when a new request arrives, first determine whether the client where this request is located is in an idle state (a newly connected client or a client that has not accessed for a long time) and becomes an active state; if the idle state becomes an active state, it is necessary to adjust the existing P tag (weight ratio) tags of unprocessed requests so that the P tags of existing requests and the P tags of new requests are processed from the same starting point: traverse all requests to obtain the minPtag; traverse and adjust the P tags of all requests: P tag -= (minPtag – t); calculate the P tag, L tag, and R tag of the current request; and place the request into P_heap, L_heap, and R_heap.

[0053] In one embodiment, the dequeue process of the mclock algorithm is as Figure 3 shown. When the algorithm schedules a request, it first enters the Reservation-based scheduling phase and gives priority to meeting the Reservation settings, obtaining the request with the smallest R tag from the R_heap; if the R tag of the request < t, it can be immediately scheduled. If this condition cannot be met, it means that the Reservation conditions have been satisfied and all clients have reached their Reservation settings, and it enters step c); at this time, it enters the Weight-based scheduling strategy, traverses the L_heap to find all requests with L tag < now, sets the ready state of the request to true, and adjusts the P_heap (since the ready state has changed, it needs to be readjusted); obtains the ready request with the smallest P tag from the P_heap for scheduling; for requests scheduled through the Weight-based scheduling strategy, it is necessary to adjust the R tags of all remaining requests: R tag = R tag – 1 / Rate; if no request meets the requirement of L tag < t, schedule after delaying for a time of t - L tag.

[0054] Understandably, in scenarios like distributed file systems where requests are executed across multiple servers, it's impossible to set a uniform limit rate. That is, requests originate from multiple clients and are sent to multiple servers. In this scenario, mClock is no longer suitable because it's a single-machine algorithm. If an mClock algorithm runs on each server, these algorithm instances are independent of each other. For example, suppose request A has a limit of La, and request A is evenly distributed across three servers S1, S2, and S3. Each of S1, S2, and S3 will satisfy La, and the final result is that S1, S2, and S3 collectively satisfy 3 * La, which is three times the specified limit.

[0055] To achieve distributed QoS and allow clients to set a unified Limit for the three servers, the formulas for calculating R, L, and P tags were adjusted as follows:

[0056]

[0057]

[0058]

[0059] The formula uses delta instead of 1. Here, delta can be understood as meaning that between the last request received by Server S1 and the current request received, this client sent delta-1 requests to other Servers S2 and S3.

[0060] As mentioned above, the difference between dmclock and mclock is that the 1 in the formula is replaced with delta, where delta is the number of IO requests between the last request received by Server S1 and the current request received. This client sent delta-1 requests to other Servers S2 and S3.

[0061] Furthermore, the total number of IO requests sent by the client and the amount of IO sent when the client sends the IO request to each object storage service OSS are counted; based on the total amount of IO data and the amount of IO sent, the request order corresponding to the IO request sent by each object storage service OSS is calculated.

[0062] In one embodiment, the client can collect IO status info, calculate delta using the statistical information, and then pass delta into the IO request to the server.

[0063] For example, suppose that client1 sends requests in the following order: Figure 4 As shown, between the first request sent to OSS1 and the second request sent to OSS1, one request was sent to OSS2 and another to OSS3. Therefore, the delta of the second request sent to OSS1 is 2+1=3.

[0064] Specifically, the statistical method is as follows:

[0065] 1) Record the following IO Status Info:

[0066] a) The total number of IOs (ios_total) is updated and incremented when processing res;

[0067] b) Record the total number of IOs sent when each OSS sends a request (ios_oss1, ios_oss2, ios_oss3), and record them separately when sending a request;

[0068] 2) When sending a request, use the following formula to calculate delta. For example, when sending a request to OSS1: delta = ios_total - ios_oss1 + 1.

[0069] S102. Based on the dmclock algorithm and the request order, calculate the QoS metric corresponding to the IO request to obtain the QoS limitation scheme of the current distributed file system.

[0070] In one embodiment, the algorithm formula for dmclock is modified as follows:

[0071]

[0072]

[0073]

[0074] The meanings of each data point are as follows:

[0075]

[0076] Therefore, it is possible to calculate the QoS limiting scheme for bandwidth resources.

[0077] In one embodiment, the bandwidth QoS formula implemented by dmclock above and the IOPS QoS limiting scheme implemented by the file system are used. In order to achieve the QoS of file system bandwidth limiting, the main task is to calculate bd_delta for each IO request on the client side.

[0078] For example, the scheme for calculating bd_delta is as follows:

[0079] 1) Record the following IO Status Info:

[0080] a) The total number of IO operations (data_total) is updated and incremented when processing res;

[0081] b) Record the total amount of IO data (data_oss1, data_oss2, data_oss3) when each OSS sends a request, and record it separately when sending the request;

[0082] When sending a request, the following formula is used to calculate data_delta. For example, when sending a request to OSS1: bd_delta = data_total - data_oss1 + size of current io.

[0083] Furthermore, based on the dmclock algorithm and the request order, the IOPS (Input / Output Count per Second) tag and bandwidth tag corresponding to the IO request are calculated respectively; based on the maximum value function, the IOPS tag and the bandwidth tag are compared to obtain the QoS metric corresponding to the IO request.

[0084] In one embodiment, the IOPS tag includes an IOPS reservation time tag, an IOPS weight time tag, and an IOPS upper limit time tag, and the bandwidth tag includes a bandwidth reservation time tag, a bandwidth weight time tag, and a bandwidth upper limit time tag.

[0085] Furthermore, based on the maximum value function, the IOPS reserved time tag and the bandwidth reserved time tag, the IOPS weighted time tag and the bandwidth weighted time tag, the IOPS upper limit time tag and the bandwidth upper limit time tag are compared respectively, and the reserved time tag, weighted time tag and upper limit time tag with the largest value are selected as the QoS metric corresponding to the IO request.

[0086] In one embodiment, when limiting resources for users of the file system, it is usually necessary to limit both IOPS and bandwidth simultaneously. The scheme for limiting IOPS and bandwidth simultaneously is as follows, mainly by modifying the dmclock algorithm formula as follows:

[0087]

[0088]

[0089]

[0090]

[0091]

[0092]

[0093]

[0094]

[0095]

[0096] In one embodiment, Set to and The larger values, and Similarly, by modifying the formula of the dmclock algorithm, the dmclock algorithm will only schedule requests that meet both the bandwidth and IOPS limits when scheduling requests, thus simultaneously fulfilling the QoS limits for both bandwidth and IOPS.

[0097] S103. Obtain the file system subdirectories mounted by the currently distributed file system;

[0098] In one embodiment, the file system typically supports a multi-tenant strategy, where a single file system can be accessed by multiple different users. User data needs to be isolated to prevent users from accessing other users' data. Furthermore, all tenants need to share the IOPS resources of the entire file system. To ensure the quality of service for tenants, limits, weights, and reservations need to be set for the directories where multiple users reside; that is, a directory-level QoS scheme needs to be implemented.

[0099] S104. Based on the current distributed file system's QoS restriction scheme, set QoS restriction indicators for the file system's subdirectories to implement a directory-level QoS scheme for the distributed file system.

[0100] In one embodiment, the server-side OSS sets limits, weights, and reservations for different directories, and retrieves the corresponding limits, weights, and reservations for each subdirectory from these settings. Then, it follows the dmclock algorithm to perform enqueue and dequeue processes. This completes the QoS restrictions for different subdirectories.

[0101] This embodiment provides a method for implementing directory-level QoS functionality in a distributed file system. This method uses the dmclock algorithm to achieve distributed QoS, avoiding the complexity and performance issues introduced by a unified QoS service. While the dmclock algorithm can only limit IOPS, this method supports both IOPS and bandwidth QoS. Furthermore, it achieves directory-level QoS functionality, facilitating multi-tenant isolation and ensuring service quality for each tenant, whereas the dmclock algorithm can only achieve overall file system IOPS QoS. This eliminates the need for additional QoS service components, reducing system complexity; it also reduces the impact of additional services on IO requests, increasing service stability; and since IO requests do not require prior QoS service requests, IO request latency is low.

[0102] Please refer to Figure 5 , Figure 5 This is a flowchart illustrating a second embodiment of a distributed file system directory-level QoS functionality implementation method provided in this application.

[0103] like Figure 5 As shown, based on the above Figure 1 In the illustrated embodiment, prior to step S103, the method further includes:

[0104] S201. Based on the file system mount command, the client performs a mount operation on the current distributed file system to obtain the file root directory of the current distributed file system;

[0105] Further, the client connects to the metadata node MDS, and the MDS node returns an acknowledgment character ACK; the client sends the root directory's inode to the MDS; the MDS obtains the root node of the root directory and returns it to the client; the client places the root node in (sb->root_inode) in the superblock, so that sb->root_inode becomes the file root directory of the current distributed file system.

[0106] In one embodiment, the current mount command for the distributed file system is as follows: [root@localhost~]#mount 192.128.0.100: / / mnt / local_path.

[0107] like Figure 6As shown, the client connects to the MDS metadata node, and the MDS node returns ACK; the client sends GET_ROOT_MSG to the MDS, or the inode of the root directory; the MDS obtains the inode of the root directory (inode num=2) and returns it to the client; the client puts the root_inode into the super block (sb->root_inode), and then all client accesses will start from sb->root_inode as the root directory of the current file system.

[0108] S202. Based on the subdirectory mount command, mount the file system subdirectory to the file root directory.

[0109] In one embodiment, the mount command for the current distributed file system subdirectory is as follows: [root@localhost~]#mount 192.128.0.100: / sub_dir / mnt / local_path.

[0110] like Figure 7 As shown, the difference between mounting a subdirectory and mounting a file system is that when sending a GET_ROOT_MGS message to MDS, path=' / sub_dir' transmits the subdirectory to be mounted. MDS returns the inode of the subdirectory ' / sub_dir' to the client. The client then places it as the root inode in the super block (sb->root_inode). All client accesses will then use sb->root_inode as the root directory of the current file system. In this way, the client can only access files within the subdirectory.

[0111] Furthermore, the client mounts the file system subdirectory to its local machine; when the client reads or writes files in the subdirectory, it sends the subdirectory information to the server; the server sets QoS limit indicators for the file system subdirectory based on the current distributed file system's QoS limit scheme, so as to implement a directory-level QoS scheme for the distributed file system.

[0112] In one embodiment, such as Figure 8 As shown, the implementation scheme of QoS in the subdirectory is as follows:

[0113] When the client mounts a subdirectory to the local machine, `sb->root_inode` points to the inode where `sub_dir` is located. Simultaneously, a field `sb->root_dir = " / subdir1"` is added to the super block: `[root@localhost~]#mount 192.128.0.100: / subdir1 / mnt / local_path`. When the client reads or writes files in the subdirectory (e.g., ` / subdir1 / file1`), it also adds `sb->root_dir` to the I / O request: `req->root_dir = " / subdir1"`.

[0114] In one embodiment, the server-side OSS sets limits, weights, and reservations for different directories, and retrieves the corresponding limits, weights, and reservations for each subdirectory from these settings. Then, it follows the dmclock algorithm to perform enqueue and dequeue processes. This completes the QoS restrictions for different subdirectories.

[0115] Please see Figure 9 , Figure 9 This is a schematic diagram of the first embodiment of a distributed file system directory-level QoS function implementation device provided in this application. This distributed file system directory-level QoS function implementation device is used to execute the aforementioned distributed file system directory-level QoS function implementation method. The distributed file system directory-level QoS function implementation device can be configured in a server.

[0116] like Figure 9 As shown, the distributed file system directory-level QoS function implementation device 300 includes: a request order acquisition module 301, an indicator calculation module 302, a subdirectory acquisition module 303, and a directory-level QoS scheme implementation module 304.

[0117] The request order acquisition module 301 is used to acquire the request order corresponding to the IO request when the client sends an IO request to the server.

[0118] The metric calculation module 302 is used to calculate the QoS metric corresponding to the IO request based on the dmclock algorithm and the request order, and obtain the QoS limitation scheme of the current distributed file system.

[0119] The subdirectory acquisition module 303 is used to acquire the file system subdirectories mounted by the currently distributed file system;

[0120] The directory-level QoS scheme implementation module 304 is used to set QoS restriction indicators for the subdirectories of the file system based on the current QoS restriction scheme of the distributed file system, so as to implement the directory-level QoS scheme of the distributed file system.

[0121] In one embodiment, the request order acquisition module 301 is further configured to count the total number of IO requests sent by the client and the amount of IO sent when the client sends the IO request to each object storage service OSS; and calculate the request order corresponding to the IO request sent by each object storage service OSS based on the total amount of IO data and the amount of IO sent.

[0122] In one embodiment, the metric calculation module 302 is further configured to calculate the IOPS tag and bandwidth tag corresponding to the IO request based on the dmclock algorithm and the request order, respectively; and to obtain the QoS metric corresponding to the IO request by comparing the IOPS tag and the bandwidth tag based on the maximum value function.

[0123] In one embodiment, the IOPS tag includes an IOPS reservation time tag, an IOPS weight time tag, and an IOPS upper limit time tag, and the bandwidth tag includes a bandwidth reservation time tag, a bandwidth weight time tag, and a bandwidth upper limit time tag;

[0124] The metric calculation module 302 is further configured to, based on the maximum value function, compare the IOPS reserved time tag and the bandwidth reserved time tag, the IOPS weighted time tag and the bandwidth weighted time tag, the IOPS upper limit time tag and the bandwidth upper limit time tag respectively, and filter the reserved time tag, weighted time tag and upper limit time tag with the largest value as the QoS metric corresponding to the IO request.

[0125] In one embodiment, the distributed file system directory-level QoS function implementation device 300 further includes a directory mounting module, which is used to perform a mounting operation on the current distributed file system through the client based on the file system mount command to obtain the file root directory of the current distributed file system; and to mount the file system subdirectories to the file root directory based on the subdirectory mount command.

[0126] In one embodiment, the directory mounting module is further configured to allow the client to connect to the metadata node MDS, and the MDS node to return an acknowledgment character ACK; the client sends the root directory's index node to the MDS; the MDS obtains the root node of the root directory and returns it to the client; the client places the root node in (sb->root_inode) in the superblock, so that sb->root_inode becomes the file root directory of the current distributed file system.

[0127] In one embodiment, the directory mounting module is further configured to have the client mount the file system subdirectory to the local machine; when the client reads or writes files in the subdirectory, it sends the subdirectory information to the server; the server sets QoS limit indicators for the file system subdirectory based on the current distributed file system's QoS limit scheme, so as to implement a directory-level QoS scheme for the distributed file system.

[0128] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the above-described device and modules can be referred to the corresponding processes in the aforementioned implementation method of the distributed file system directory-level QoS function, and will not be repeated here.

[0129] The apparatus provided in the above embodiments can be implemented as a computer program, which can be used in, for example... Figure 10 It runs on the computer device shown.

[0130] Please see Figure 10 , Figure 10 This is a schematic block diagram illustrating the structure of a computer device according to an embodiment of this application. The computer device may be a server.

[0131] See Figure 10 The computer device includes a processor, memory, and network interface connected via a system bus, wherein the memory may include non-volatile storage media and internal memory.

[0132] Non-volatile storage media can store operating systems and computer programs. These computer programs include program instructions that, when executed, cause the processor to perform any distributed file system directory-level QoS implementation.

[0133] The processor provides computing and control capabilities, supporting the operation of the entire computer device.

[0134] Internal memory provides an environment for the execution of computer programs in non-volatile storage media. When the computer program is executed by the processor, it enables the processor to implement any distributed file system directory-level QoS functionality.

[0135] This network interface is used for network communication, such as sending assigned tasks. Those skilled in the art will understand that... Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0136] It should be understood that the processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, a general-purpose processor can be a microprocessor or any conventional processor.

[0137] In one embodiment, the processor is configured to run a computer program stored in memory to perform the following steps:

[0138] When the client sends an IO request to the server, the request order corresponding to the IO request is obtained;

[0139] Based on the dmclock algorithm and the request order, calculate the QoS metric corresponding to the IO request and obtain the QoS limitation scheme of the current distributed file system.

[0140] Obtain the file system subdirectories currently mounted by the distributed file system;

[0141] Based on the current QoS limiting scheme of the distributed file system, QoS limiting indicators are set for the subdirectories of the file system to implement a directory-level QoS scheme for the distributed file system.

[0142] When the processor implements the step of obtaining the request order corresponding to the IO request, it is used to:

[0143] The total number of IO requests sent by the client and the amount of IO sent when the client sends the IO request to each object storage service OSS are counted.

[0144] Based on the total amount of IO data and the amount of IO data sent, calculate the request order corresponding to the IO requests sent by each of the object storage services OSS.

[0145] When the processor calculates the QoS metric corresponding to the IO request based on the dmclock algorithm and the request order, it is used to implement:

[0146] Based on the dmclock algorithm and the request order, calculate the IOPS tag and bandwidth tag corresponding to the IO request respectively;

[0147] Based on the maximum value function, the QoS metric corresponding to the IO request is obtained by comparing the IOPS label and the bandwidth label.

[0148] In one embodiment, the IOPS tag includes an IOPS reservation time tag, an IOPS weight time tag, and an IOPS upper limit time tag, and the bandwidth tag includes a bandwidth reservation time tag, a bandwidth weight time tag, and a bandwidth upper limit time tag;

[0149] When the processor implements the method of comparing the IOPS label and the bandwidth label based on the maximum value function to obtain the QoS metric corresponding to the IO request, it is used to:

[0150] Based on the maximum value function, the IOPS reserved time tag and the bandwidth reserved time tag, the IOPS weighted time tag and the bandwidth weighted time tag, the IOPS upper limit time tag and the bandwidth upper limit time tag are compared respectively, and the reserved time tag, weighted time tag and upper limit time tag with the largest value are selected as the QoS metric corresponding to the IO request.

[0151] In one embodiment, before implementing the acquisition of the file system subdirectory mounted by the currently distributed file system, the processor is further configured to implement:

[0152] Based on the file system mount command, the client performs a mount operation on the current distributed file system to obtain the file root directory of the current distributed file system;

[0153] The subdirectory of the file system is mounted to the root directory of the file system using the subdirectory mount command.

[0154] In one embodiment, when the processor implements the file system mount command to perform a mount operation on the current distributed file system through the client and obtain the file root directory of the current distributed file system, it is configured to:

[0155] The client connects to the metadata node MDS, and the MDS node returns an acknowledgment character ACK.

[0156] The client sends the root directory index node to the MDS;

[0157] The MDS obtains the root node of the root directory and returns it to the client;

[0158] The client places the root node in (sb->root_inode) within the superblock, so that sb->root_inode becomes the file root directory of the current distributed file system.

[0159] In one embodiment, when the processor implements the QoS limiting scheme based on the current distributed file system and sets QoS limiting indicators for the subdirectories of the file system to implement a directory-level QoS scheme for the distributed file system, it is configured to:

[0160] The client mounts the file system subdirectory to its local machine;

[0161] When the client reads or writes files in a subdirectory, it sends the subdirectory information to the server.

[0162] The server sets QoS limit indicators for the subdirectories of the file system based on the current QoS limit scheme of the distributed file system, so as to implement a directory-level QoS scheme for the distributed file system.

[0163] The embodiments of this application also provide a computer-readable storage medium storing a computer program, the computer program including program instructions, and the processor executing the program instructions to implement any of the distributed file system directory-level QoS function implementation methods provided in the embodiments of this application.

[0164] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiments, such as the hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, SmartMediaCard (SMC), SecureDigital (SD) card, or FlashCard equipped on the computer device.

[0165] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for implementing directory-level QoS functionality in a distributed file system, characterized in that, The method includes: When the client sends an IO request to the server, the request order corresponding to the IO request is obtained; Based on the dmclock algorithm and the request order, calculate the QoS metric corresponding to the IO request and obtain the QoS limitation scheme of the current distributed file system. Obtain the file system subdirectories currently mounted by the distributed file system; Based on the current distributed file system's QoS limiting scheme, QoS limiting indicators are set for the file system's subdirectories to implement a directory-level QoS scheme for the distributed file system. The step of calculating the QoS metric corresponding to the IO request based on the dmclock algorithm and the request order includes: Based on the dmclock algorithm and the request order, calculate the IOPS tag and bandwidth tag corresponding to the IO request respectively; Based on the maximum value function, the IOPS label and the bandwidth label are compared to obtain the QoS metric corresponding to the IO request; The IOPS tag includes an IOPS reservation time tag, an IOPS weight time tag, and an IOPS upper limit time tag; the bandwidth tag includes a bandwidth reservation time tag, a bandwidth weight time tag, and a bandwidth upper limit time tag. The step of obtaining the QoS metric corresponding to the IO request by comparing the IOPS tag and the bandwidth tag based on a maximum value function includes: Based on the maximum value function, the IOPS reserved time tag and the bandwidth reserved time tag, the IOPS weighted time tag and the bandwidth weighted time tag, the IOPS upper limit time tag and the bandwidth upper limit time tag are compared respectively, and the reserved time tag, weighted time tag and upper limit time tag with the largest value are selected as the QoS metric corresponding to the IO request.

2. The method for implementing directory-level QoS functionality in a distributed file system according to claim 1, characterized in that, Obtaining the request order corresponding to the IO request includes: The total number of IO requests sent by the client and the amount of IO sent when the client sends the IO request to each object storage service OSS are counted. Based on the total amount of IO data and the amount of IO data sent, calculate the request order corresponding to the IO requests sent by each of the object storage services OSS.

3. The method for implementing directory-level QoS functionality in a distributed file system according to claim 1, characterized in that, Before obtaining the subdirectories of the file system currently mounted by the distributed file system, the process also includes: Based on the file system mount command, the client performs a mount operation on the current distributed file system to obtain the file root directory of the current distributed file system; The subdirectory of the file system is mounted to the root directory of the file system using the subdirectory mount command.

4. The method for implementing directory-level QoS functionality in a distributed file system according to claim 3, characterized in that, The file system mount command, performed by the client, mounts the current distributed file system to obtain its root directory, including: The client connects to the metadata node MDS, and the MDS node returns an acknowledgment character ACK. The client sends the root directory index node to the MDS; The MDS obtains the root node of the root directory and returns it to the client; The client places the root node in sb->root_inode in the superblock, so that sb->root_inode serves as the root directory of the current distributed file system.

5. The method for implementing directory-level QoS functionality in a distributed file system according to claim 3, characterized in that, The QoS limiting scheme based on the current distributed file system sets QoS limiting indicators for subdirectories of the file system to implement a directory-level QoS scheme for the distributed file system, including: The client mounts the file system subdirectory to its local machine; When the client reads or writes files in a subdirectory, it sends the subdirectory information to the server. The server sets QoS limit indicators for the subdirectories of the file system based on the current QoS limit scheme of the distributed file system, so as to implement a directory-level QoS scheme for the distributed file system.

6. A device for implementing directory-level QoS functionality in a distributed file system, characterized in that, The distributed file system directory-level QoS functionality implementation device includes: The request order acquisition module is used to acquire the request order corresponding to the IO request when the client sends an IO request to the server. The metric calculation module is used to calculate the QoS metric corresponding to the IO request based on the dmclock algorithm and the request order, and to obtain the QoS limitation scheme of the current distributed file system. The subdirectory acquisition module is used to acquire the subdirectories of the file system currently mounted by the distributed file system; The directory-level QoS scheme implementation module is used to set QoS limit indicators for the subdirectories of the file system based on the current QoS limit scheme of the distributed file system, so as to implement the directory-level QoS scheme of the distributed file system. The indicator calculation module is also used for: Based on the dmclock algorithm and the request order, calculate the IOPS tag and bandwidth tag corresponding to the IO request respectively; Based on the maximum value function, the IOPS label and the bandwidth label are compared to obtain the QoS metric corresponding to the IO request; The IOPS tag includes an IOPS reservation time tag, an IOPS weight time tag, and an IOPS upper limit time tag; the bandwidth tag includes a bandwidth reservation time tag, a bandwidth weight time tag, and a bandwidth upper limit time tag; the metric calculation module is also used for: Based on the maximum value function, the IOPS reserved time tag and the bandwidth reserved time tag, the IOPS weighted time tag and the bandwidth weighted time tag, the IOPS upper limit time tag and the bandwidth upper limit time tag are compared respectively, and the reserved time tag, weighted time tag and upper limit time tag with the largest value are selected as the QoS metric corresponding to the IO request.

7. A computer device, characterized in that, The computer device includes a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, it implements the steps of the distributed file system directory-level QoS function implementation method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, it implements the steps of the distributed file system directory-level QoS function implementation method as described in any one of claims 1 to 5.

Images