Phased adaptive request scheduling method and scheduler for distributed systems

Abstract

The present invention discloses a distributed system-oriented phased adaptive request scheduling method and scheduler, belonging to the field of distributed storage systems, including: at the storage node end, constructing an IO request queue and a corresponding token bucket for each user, wherein the limit parameter and the reservation parameter are respectively initialized to the QoS target upper limit and lower limit, and the token generation rate actual_rate=limit; traversing the IO request queue to schedule user requests, and when the user performance does not meet the QoS target lower limit, executing a segmented adaptive adjustment strategy, including the following steps executed in sequence: fast balancing stage: punishing the token generation rate until the performance of each user meets the QoS target lower limit; slow growth stage: rewarding the token generation rate until the user performance does not meet the QoS target lower limit; rebalancing stage: punishing the token generation rate. The present invention can improve the system resource utilization rate when the actual operating performance of each user is met.

Description

Technical Field

[0001] The present invention belongs to the field of distributed storage systems, and more specifically, relates to a distributed system-oriented phased adaptive request scheduling method and a scheduler. Background Art

[0002] With the explosive growth of data volume, distributed storage clusters have gradually become an important choice for large-scale data centers. For example, Ceph, as one of the typical representatives of decentralized distributed storage systems, aims to provide reliable and efficient storage services. According to the 2021 OpenStack survey report, in the actual production environment of OpenStack, nearly 71% of systems use Ceph block storage services, and it has shown a certain growth trend.

[0003] However, it is difficult to accurately estimate the service capacity of the storage end of a distributed system. It is affected by many factors. First, the number of concurrent user requests in different time periods varies. Especially when encountering scenarios with multiple users and high traffic concurrency such as Double Eleven, there may be a situation where storage resources are in short supply. Second, the distributed cluster uses multiple copies or erasure code redundancy strategies to ensure data reliability. When a user initiates a write request, the redundancy mechanism will come into play. The primary copy will synchronize the data object to the secondary copy, and the redundancy mechanism will also occupy part of the bandwidth of the distributed cluster. Third, when a storage node fails and data recovery is required, the data recovery request will also compete for the bandwidth of the cluster. Fourth, the unpredictability of the storage end processing capacity caused by the diverse I / O request characteristics. Read and write requests have different scheduling modes in the system. Coupled with the influence of the system cache strategy, read and write requests often have different access costs.

[0004] Therefore, personalized application service requirements, a variety of underlying storage media, and the large and complex functional system of distributed storage systems all mean that it is full of challenges to build an effective service quality control system for actual distributed storage systems. However, the existing decentralized distributed storage system service quality technology cannot provide users with the upper and lower limit control capabilities of I / O throughput and I / O bandwidth at the same time, and when there is high concurrent traffic in the distributed cluster and insufficient storage resources, or when most of the request objects with similar data are concentrated in a few service nodes, the performance of some tenants is more severely impacted. Summary of the invention

[0005] In view of the defects of the prior art and the need for improvement, the present invention provides a phased adaptive request scheduling method and scheduler for distributed systems, which aims to allocate resources to user requests in a balanced manner and improve the utilization of system resources while satisfying the actual operating performance of each user.

[0006] To achieve the above object, according to one aspect of the present invention, a distributed system-oriented phased adaptive request scheduling method is provided, comprising the following steps performed at a storage node:

[0007] According to the QoS target upper limit and QoS target lower limit carried in the user request, an IO request queue and a corresponding token bucket are constructed for each user. The limit parameter and reservation parameter of the token bucket are the QoS target upper limit and QoS target lower limit of the corresponding user respectively. The initial value of the token generation rate actual_rate is equal to the limit parameter;

[0008] Traverse the IO request queue, schedule the user request according to the token bucket parameters and the tokensNeeded parameter carried in the request, and execute the segmented adaptive adjustment strategy when the user performance does not meet the QoS target lower limit; the segmented adaptive adjustment strategy includes:

[0009] Fast balance stage: the token generation rate is continuously penalized until the performance of each user meets the lower limit of the QoS target;

[0010] Slow growth phase: After the fast balance phase, the token generation rate is continuously rewarded until the user performance does not meet the QoS target lower limit;

[0011] Rebalancing phase: after the slow growth phase, the token generation rate is penalized;

[0012] Among them, the QoS target is throughput or bandwidth; the tokensNeeded parameter is the number of tokens required to execute the request; the token generation rate is punished each time, including: reducing the actual_rate of the token generation rate of each token bucket in the storage node within the range of [reservation, actual_rate]; the token generation rate is rewarded each time, including: increasing the actual_rate of the token generation rate of each token bucket in the storage node within the range of [actual_rate, limit]; the reward degree in the slow growth stage and the penalty degree in the rebalancing stage are both smaller than the penalty degree in the fast balancing stage.

[0013] Furthermore, the judgment method of whether the user performance does not meet the QoS target lower limit includes:

[0014] If the remaining number of tokens in the token bucket corresponding to a certain IO request queue is greater than limit-reservation for N consecutive seconds, it is determined that the user performance corresponding to the IO request does not meet the QoS target lower limit;

[0015] Among them, N>0.

[0016] Furthermore, in the fast balance phase, the token generation rate is penalized each time, and the degree of penalty is: (actual_rate-reservation) / N 1 .

[0017] Furthermore, during the slow growth phase, the token generation rate is rewarded each time, and the reward level is: (limit-reservation) / N 2 and limit–actual_rate, whichever is smaller; N 2 <N 1 .

[0018] Furthermore, during the rebalancing phase, the token generation rate is penalized each time, and the degree of penalty is: (limit-reservation) / N 3 The smaller value of actual_rate–reservation; N 3 <N 1 .

[0019] Furthermore, the segmented adaptive adjustment strategy also includes: when all requests of a certain user are executed, the token generation rate of each token bucket in the storage node is reset to the limit parameter.

[0020] Furthermore, the phased adaptive request scheduling method for distributed systems provided by the present invention also includes: after receiving a user request at the storage node end and parsing the QoS target upper limit value and the QoS target lower limit value therefrom, if the user's IO request queue already exists in the storage node, and the limit parameter and reservation parameter of the corresponding token bucket are inconsistent with the parsed QoS target upper limit value and the QoS target lower limit value, then the limit parameter and reservation parameter of the token bucket are adjusted to the parsed QoS target upper limit value and the QoS target lower limit value respectively.

[0021] Furthermore, the distributed system-oriented phased adaptive request scheduling method provided by the present invention further includes the following steps performed at the sending end:

[0022] Calculate the tokensNeeded parameter of the user request, set the tokensNeeded parameter, and the QoS target upper limit and QoS target lower limit specified by the user in the message sent to the user request, and then send the message to the storage node.

[0023] Furthermore, the distributed system-oriented phased adaptive request scheduling method provided by the present invention further includes: maintaining a throughput coordination hash table and a bandwidth coordination hash table at the sending end;

[0024] The throughput coordination hash table stores the information corresponding to each request in the form of key-value pairs. In each key-value pair, the ID of the storage node corresponding to the current request is used as the key, and the sequence label of the previous request that arrived at the storage node and the sequence label of the current request are used as the value.

[0025] The bandwidth coordination hash table stores the information corresponding to each request in the form of key-value pairs. In each key-value pair, the sequence tag of the current request is used as the key, and the sum of the request size sent by the user and the current request size is used as the value;

[0026] Furthermore, if the QoS target specified by the user is throughput, the tokensNeeded parameter of the user request is calculated as follows: after determining the storage node id corresponding to the user request, find the corresponding key-value pair in the throughput coordination hash table, extract the sequence label of the current request and the sequence label of the previous request, and then make the difference, and use the difference as the tokensNeeded parameter of the request;

[0027] If the QoS target specified by the user is bandwidth, the calculation method of the tokensNeeded parameter of the user request is as follows: determine the storage node id corresponding to the user request, search the throughput coordination hash table and the bandwidth coordination hash table respectively according to the storage node id and the sequence label of the current request, extract the value therein, subtract it from the value corresponding to the sequence label of the previous request to the same storage node, and use the difference as the tokensNeeded parameter of the request.

[0028] According to another aspect of the present invention, there is provided a distributed system-oriented phased adaptive request scheduler, comprising: a computer-readable storage medium and a processor;

[0029] The computer readable storage medium stores a computer program;

[0030] The processor is used to read the computer program in the computer-readable storage medium and execute the distributed system-oriented phased adaptive request scheduling method provided by the present invention.

[0031] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects:

[0032] (1) The present invention provides a method for real-time monitoring of the performance satisfaction of each user when performing request scheduling at the storage node end, and executing a segmented adaptive adjustment strategy when the user performance does not meet the QoS target lower limit value, specifically including: in the fast balancing stage, by gradually penalizing the token generation rate of the token bucket, the lower limit level of all user performance requirements is preferentially met, thereby avoiding the situation where the performance of some tenants is greatly impacted; in the slow growth stage after the fast balancing stage, by gradually rewarding the token generation rate of the token bucket, the utilization rate of system resources can be increased while ensuring that the performance of each user is met, so that the system resources are more fully utilized; in the rebalancing stage after the slow growth stage, when the user performance is not met again, the token generation rate is gradually penalized to rebalance the allocation of system resources, and finally ensure that the utilization rate of system resources is maximized while the performance requirements of each user are met. In general, the present invention can control the actual operating performance of users to be near their QoS upper limit target value when cluster node resources are sufficient; when cluster node resources are insufficient, the present invention can ensure that the actual operating performance of users is not lower than their QoS lower limit target value on the basis of avoiding single point bottlenecks and not introducing additional network traffic.

[0033] (2) At the sending end, the present invention calculates the number of tokens that the user request should consume, i.e., the tokensNeeded parameter, before sending a request message to the storage node end through the network based on the throughput rate or bandwidth requirement specified by the user, and adds the tokensNeeded parameter and the user's QoS target metadata to the message, thereby realizing user-oriented multi-node resource scheduling. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A schematic diagram of a distributed system-oriented phased adaptive request scheduling method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0036] In the present invention, the terms "first", "second", etc. (if any) in the present invention and the accompanying drawings are used to distinguish similar objects but not necessarily to describe a specific order or sequence.

[0037] In order to solve the problem that when there is high concurrent traffic and insufficient storage resources in the existing distributed storage system, or when most request objects with similar data are gathered in a few service nodes, the performance of some users will be greatly impacted, the present invention provides a distributed system-oriented phased adaptive request scheduling method and scheduler, the overall idea of ​​which is to implement user-oriented multi-node resource scheduling based on a token bucket algorithm, monitor the satisfaction of each user's performance requirements in real time at the storage node end, and when the user's performance requirements are not met, use a phased adaptive adjustment strategy to adjust the token bucket generation rate, so that when the cluster node resources are sufficient, the user's actual operating performance is controlled to be close to its QoS upper limit target value; when the cluster node resources are insufficient, on the basis of avoiding single point bottlenecks and not introducing additional network traffic, the user's actual operating performance is guaranteed to be not lower than its QoS lower limit target value.

[0038] The present invention can be applied to any decentralized distributed storage system. Without loss of generality, the Ceph system is used as an example for description in the following embodiments.

[0039] The following are examples.

[0040] Embodiment 1:

[0041] A phased adaptive request scheduling method for distributed systems, such as Figure 1 shown.

[0042] Distributed storage systems use multiple storage nodes to store data; a single-node resource scheduling strategy refers to the strategy adopted by a single node when scheduling its own resources; however, in a distributed system, fair scheduling of single-node resources does not mean that scheduling of multiple nodes will be fair, so it is also necessary to coordinate the fairness of resource scheduling of multiple nodes.

[0043] In this embodiment, in order to implement a user-oriented single-node resource scheduling strategy, a set of control interfaces are provided for the user to configure QoS target metadata, namely, a QoS target upper limit value and a QoS target lower limit value. The QoS target is specifically an I / O throughput rate or an I / O bandwidth. For ease of description, in this embodiment, the throughput upper limit value and the throughput lower limit value are represented by symbols Li and Ri, respectively, and the bandwidth upper limit value and the bandwidth lower limit value are represented by symbols Lb and Rb, respectively. At the same time, a limit parameter and a reservation parameter are added to the message; after receiving the user-specified throughput demand or bandwidth demand through the control interface, the limit parameter and the reservation parameter of the current message are initialized to the QoS target upper limit value and the QoS target lower limit value, respectively, so that the QoS target metadata will be transmitted to the storage node end through the network for use by the storage end user IO queue scheduling.

[0044] The Ceph distributed storage system uses the CRUSH decentralized data layout algorithm to calculate the address of the target storage node.

[0045] In order to implement a user-oriented multi-node resource scheduling strategy, in this embodiment, the number of tokens that should be consumed by the user request, that is, the tokensNeeded parameter, will be calculated at the sending end; specifically, this implementation will number the requests according to the queuing order of the requests at the sending end as the order label of the requests, and maintain two hash tables at the sending end, namely the throughput coordination hash table and the bandwidth coordination hash table;

[0046] The throughput coordination hash table is constructed as follows:

[0047] First, we use atomic operations to solve the high-concurrency conflict problem of requests. Then, we use the decentralized idea of ​​distributed storage systems to obtain the ID of the target storage node through calculation. We use the ID of the target storage node as the key value, and use the sequence label of the previous request that arrived at the target storage node and the sequence label of the current request as the value to build a hash table, namely the throughput coordination hash table. Therefore, when the QoS target specified by the user is throughput, the number of tokens that the current user request should consume is the difference between the previous and next two requests that arrive at the same target node.

[0048] Since "bandwidth = throughput * IO size", bandwidth performance guarantee needs to be based on throughput performance guarantee; therefore, the bandwidth coordination hash table is constructed as follows:

[0049] On the basis of throughput guarantee, another hash data table, namely the bandwidth coordination hash table, is constructed with the sequential label of the request as the key value and the sum of the request size sent by the user and the current request size as the value value. The throughput coordination hash table is queried according to the ID of the current target storage node to obtain the sequential label value of the previous request that reaches the current target storage node, and then the value size corresponding to the label value is queried from the bandwidth coordination hash table. Therefore, in the bandwidth coordination hash table, the value corresponding to the label value of the current request and the value corresponding to the label value of the previous request that reaches the same storage node are the difference between them, which is the required number of tokens that needs to be calculated. In addition, the key-value pair information related to the previous request that reaches the current target storage node in the bandwidth coordination hash table is the key-value pair information that has been used and will not be used again in the future, and can be directly erased to save memory resources.

[0050] In order to make the above hash table construction process and the calculation method of the tokensNeeded parameter clearer, a specific example is further explained below.

[0051] In a certain scenario, the current distributed storage system has 3 storage nodes, marked as A, B and C respectively; suppose the user sends 8 requests to the cluster, first, according to the order of sending the requests, these 8 requests are numbered from 1 to 8. For example, the 1st, 2nd and 8th requests are sent to node A, the 3rd and 6th requests are sent to node B, and the 4th, 5th and 7th requests are sent to node C. Here, the throughput coordination hash table is used for recording. The throughput coordination hash table only needs to record the request numbers of the two requests sent to the same node. According to the records in data table 1, the request numbered 1 is sent to node A, and no request has arrived at node A before. Therefore, the number of tokens that the request numbered 1 needs to consume is 1-0=1; the request numbered 2 is sent to node A. According to the records in the data table, the number of the previous request sent to node A is 1, so the number of tokens that the request numbered 2 needs to consume is 2-1=1; similarly, the number of tokens that the remaining requests should consume can be calculated. When the user needs bandwidth control capability, further calculation is required. On the basis of the throughput coordination hash table, the bandwidth coordination hash table is introduced. Assuming that the request size is 4KB, the bandwidth coordination hash table will record the total size of the requests sent up to the current request. For example, the request numbered 1 records 4KB. At this time, the number of tokens that the request numbered 1 should consume is 4KB-0=4KB; the request numbered 2 records 4KB plus the record of request 1 4KB, that is, 8KB. Since both requests 1 and 2 are sent to node A, the number of tokens that request 2 should consume is 8KB-4KB=4KB. At this time, node A does not need to use request 1 anymore and can erase the information related to request 1 in the data table; similarly, the request numbered 3 records 4KB plus the record of request 2 8KB, that is, 12KB. Request 3 is sent to node B, so the tokens that request 3 should consume are 12KB-0=12KB; the number of tokens that the remaining requests should consume is calculated in the same way.

[0052] After the sender obtains the user's QoS upper and lower limit target values, as well as the number of tokens that the current request should consume, it will send them to the storage node through the network along with the request.

[0053] On the storage node side, the following steps are performed:

[0054] According to the QoS target metadata carried in the user request, namely the QoS target upper limit and QoS target lower limit, an IO request queue and a corresponding token bucket are constructed for each user. The limit parameter and reservation parameter of the token bucket are the QoS target upper limit and QoS target lower limit of the corresponding user, respectively. The initial value of the token generation rate actual_rate is equal to the limit parameter;

[0055] It is easy to understand that after parsing the received request, if the storage node already has an IO request queue for sending the request, the request can be directly added to the end of the queue. If the storage node does not have an IO request queue for sending the request, a new IO request queue, i.e. the corresponding token bucket, is created for the user, and the request is added to the created IO request queue. The limit parameter and reservation parameter of the token bucket are respectively initialized to the QoS target upper limit value and QoS target lower limit value carried by the request.

[0056] Since in this embodiment, the control interface provided by the sending end allows the user to dynamically adjust the QoS target metadata, it may happen that the limit parameter and reservation parameter of the token bucket are inconsistent with the QoS target metadata carried by the request. In this case, this embodiment will reset the limit parameter and reservation parameter of the token bucket to the limit parameter and reservation parameter carried by the request to adapt to the user's latest performance requirements.

[0057] To ensure the validity of the token bucket clock, when an IO request queue is empty, the queue will be temporarily maintained according to the scheduling clock of the corresponding token bucket. When the IO request queue is empty for more than the scheduling time interval of the token bucket, the user queue will be erased.

[0058] Based on the maintained IO request queue, this embodiment will traverse the IO request queue in the storage node segment by fair polling, and schedule the user request according to the token bucket parameter and the tokensNeeded parameter carried in the request; specifically, for the currently traversed IO request queue, if the request at the head of the queue can obtain sufficient tokens, the corresponding number of tokens will be consumed according to the tokensNeeded parameter carried by the request, and the request will be dequeued, and then the next IO request queue will be traversed; if the request at the head of the currently traversed IO request queue cannot obtain sufficient tokens, the next IO request queue will be traversed directly;

[0059] During the traversal process, if the head request of all IO request queues of the current storage node cannot obtain sufficient tokens, the dequeue thread will release the queue resources and hand them over to the enqueue thread to start the next round of competition for queue resources. After the scheduling is completed, the user IO request will be passed to the object storage layer for access operations.

[0060] In order to avoid the situation where the performance of some users will be greatly impacted when there is high concurrent traffic and insufficient storage resources, or when most request objects with similar data are concentrated in a few service nodes, this embodiment will monitor the performance requirements of each user in real time while scheduling IO requests, and execute a segmented adaptive adjustment strategy when the user performance does not meet the QoS target lower limit;

[0061] The segmented adaptive adjustment strategy includes three stages, namely the fast balance stage, the slow growth stage and the rebalancing stage. The specific operations performed in each stage are as follows:

[0062] The fast balancing stage aims to give priority to meeting the lower limit of all users' performance requirements when the performance of some users does not meet the standard. If the performance of some users begins to be affected as the number of users accessing the cluster increases, the system resources can no longer meet the minimum performance requirements of these users, and it is difficult to determine the balance position of resource allocation. At this time, a fast balancing process is required to give priority to ensuring the minimum performance requirements of all users. Otherwise, if the performance of all users is up to standard at this time, no resource reallocation operation is performed. The actual generation rate of tokens, actual_rate, is initially initialized to the upper limit (limit) target value of the user's performance requirement. In the fast balancing process, once it is detected that the performance of a user does not meet the standard, the actual_rate will be punished. Accordingly, the fast balancing stage includes: continuously punishing the token generation rate until the performance of each user meets the lower limit of the QoS target. Optionally, in this embodiment, in the fast balancing stage, the degree of punishment of the actual generation rate of tokens, actual_rate, is (actual_rate-reservation) / 2 each time.

[0063] The slow growth phase is intended to make more effective use of system resources after the fast balance phase occurs. After the fast balance process, the minimum performance requirements of all users have been met. However, the fast balance process may cause underutilization of system resources, so it is necessary to reward users with resources to make full use of system resources to further improve the user's service performance experience; accordingly, the slow growth phase includes: after the fast balance phase, continuously rewarding the token generation rate until the user performance does not meet the QoS target lower limit; optionally, in this embodiment, the degree of rewarding the actual generation rate of the token bucket actual_rate each time is the smaller of (limit-reservation) / 20 and limit-actual_rate.

[0064] The rebalancing phase is designed to rebalance the allocation of system resources when the slow growth phase occurs and user performance does not meet the standard. After going through the resource reward process, there is still a certain possibility that the system cannot meet the minimum performance requirements of some users. At this time, it is necessary to rebalance the allocation of resources, that is, to penalize the token generation rate of the user queue and return to the balance state of the previous round; accordingly, the rebalancing phase includes: after the slow growth phase, the token generation rate is penalized; optionally, in this embodiment, in the rebalancing phase, the token generation rate is penalized each time, and the degree of penalty is: (limit-reservation) / 20 and the smaller value of actual_rate-reservation.

[0065] When all requests of a certain user are executed, resources need to be reallocated. Therefore, in this embodiment, the segmented adaptive adjustment strategy also includes: when all requests of a certain user are executed, the token generation rate of each token bucket in the storage node is reset to the limit parameter.

[0066] Considering that when the actual token generation rate actual_rate is greater than (limit-reservation) for a continuous period of time, it means that the tokens actually consumed by the IO request queue are less than reservation, that is to say, the user's QoS target lower limit is not reached. Therefore, in this embodiment, the method for determining whether the user performance does not meet the QoS target lower limit is as follows: if the remaining tokens in the token bucket corresponding to a certain IO request queue are greater than limit-reservation for 8 consecutive seconds, it is determined that the user performance corresponding to the IO request does not meet the QoS target lower limit.

[0067] It should be noted that 8 seconds is only an optional duration setting. In actual applications, as long as the performance obtained can be guaranteed to be relatively accurate and to be responded to in a relatively timely manner, it can be flexibly adjusted to other durations.

[0068] In general, this embodiment controls the user's actual operating performance to be near its QoS upper limit target value when the cluster node resources are sufficient; when the cluster node resources are insufficient, the user's actual operating performance is guaranteed to be no lower than its QoS lower limit target value on the basis of avoiding single point bottlenecks and not introducing additional network traffic.

[0069] Embodiment 2:

[0070] A phased adaptive request scheduler for a distributed system, comprising: a computer-readable storage medium and a processor;

[0071] The computer readable storage medium stores a computer program;

[0072] The processor is used to read the computer program in the computer-readable storage medium and execute the distributed system-oriented phased adaptive request scheduling method provided in the above-mentioned embodiment 1.

[0073] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A phased adaptive request scheduling method for distributed systems, characterized in that: The following steps are performed on the storage node: According to the QoS target upper limit and QoS target lower limit carried in the user request, an IO request queue and a corresponding token bucket are constructed for each user. The limit parameter and reservation parameter of the token bucket are the QoS target upper limit and QoS target lower limit of the corresponding user respectively. The initial value of the token generation rate actual_rate is equal to the limit parameter; Traverse the IO request queue, schedule user requests according to the token bucket parameters and the tokensNeeded parameter carried in the request, and execute the segmented adaptive adjustment strategy when the user performance does not meet the QoS target lower limit; The segmented adaptive adjustment strategy includes: Fast balance stage: the token generation rate is continuously penalized until the performance of each user meets the lower limit of the QoS target; Slow growth phase: After the fast balance phase, the token generation rate is continuously rewarded until the user performance does not meet the QoS target lower limit; Rebalancing phase: after the slow growth phase, the token generation rate is penalized; Among them, the QoS target is throughput or bandwidth; the tokensNeeded parameter is the number of tokens required to execute the request; the token generation rate is punished each time, including: reducing the actual_rate of the token generation rate of each token bucket in the storage node within the range of [reservation, actual_rate]; the token generation rate is rewarded each time, including: increasing the actual_rate of the token generation rate of each token bucket in the storage node within the range of [actual_rate, limit]; the reward degree in the slow growth stage and the penalty degree in the rebalancing stage are both smaller than the penalty degree in the fast balancing stage.

2. The distributed system-oriented phased adaptive request scheduling method according to claim 1, characterized in that: The judgment method for the user performance not meeting the QoS target lower limit includes: If the remaining number of tokens in the token bucket corresponding to a certain IO request queue is greater than limit-reservation for N consecutive seconds, it is determined that the user performance corresponding to the IO request does not meet the QoS target lower limit; Among them, N>0.

3. The distributed system-oriented phased adaptive request scheduling method according to claim 1, characterized in that: In the fast balancing phase, the token generation rate is penalized each time, and the degree of penalty is: (actual_rate-reservation) / N1; N1=2.

4. The distributed system-oriented phased adaptive request scheduling method according to claim 3, characterized in that: In the slow growth phase, the token generation rate is rewarded each time, and the reward level is: the smaller one of (limit-reservation) / N2 and limit-actual_rate; N2=20.

5. The distributed system-oriented phased adaptive request scheduling method according to claim 4, characterized in that: In the rebalancing phase, the token generation rate is penalized each time, and the degree of penalty is the smaller value of (limit-reservation) / N3 and actual_rate-reservation; N3=20.

6. The distributed system-oriented phased adaptive request scheduling method according to any one of claims 1 to 5, characterized in that: The segmented adaptive adjustment strategy also includes: when all requests of a certain user are executed, the token generation rate of each token bucket in the storage node is reset to the limit parameter.

7. The distributed system-oriented phased adaptive request scheduling method according to any one of claims 1 to 5, characterized in that: Also includes: After receiving the user request at the storage node end and parsing the QoS target upper limit value and QoS target lower limit value therefrom, if the user's IO request queue already exists in the storage node, and the limit parameter and reservation parameter of the corresponding token bucket are inconsistent with the parsed QoS target upper limit value and QoS target lower limit value, then the limit parameter and reservation parameter of the token bucket are adjusted to the parsed QoS target upper limit value and QoS target lower limit value respectively.

8. The distributed system-oriented phased adaptive request scheduling method according to any one of claims 1 to 5, characterized in that: Also includes: The following steps are performed on the sending side: Calculate the tokensNeeded parameter of the user request, set the tokensNeeded parameter, and the QoS target upper limit and QoS target lower limit specified by the user in the message sent to the user request, and then send the message to the storage node.

9. The distributed system-oriented phased adaptive request scheduling method according to claim 8, characterized in that: Also includes: Maintaining a throughput coordination hash table and a bandwidth coordination hash table at the sending end; The throughput coordination hash table stores information corresponding to each request in the form of a key-value pair, wherein each key-value pair uses the ID of the storage node corresponding to the current request as the key, and uses the sequence label of the previous request that arrived at the storage node and the sequence label of the current request as the value; The bandwidth coordination hash table stores information corresponding to each request in the form of a key-value pair, wherein in each key-value pair, the sequence tag of the current request is used as the key, and the sum of the request size sent by the user and the current request size is used as the value; Furthermore, if the QoS target specified by the user is throughput, the tokensNeeded parameter of the user request is calculated as follows: after determining the storage node id corresponding to the user request, find the corresponding key-value pair in the throughput coordination hash table, extract the sequence tag of the current request and the sequence tag of the previous request, and then make a difference, and use the difference as the tokensNeeded parameter of the request; If the QoS target specified by the user is bandwidth, the calculation method of the tokensNeeded parameter of the user request is as follows: determine the storage node id corresponding to the user request, search the throughput coordination hash table and the bandwidth coordination hash table respectively according to the storage node id and the sequence label of the current request, extract the value therein, subtract it from the value corresponding to the sequence label of the previous request to the same storage node, and use the difference as the tokensNeeded parameter of the request.

10. A phased adaptive request scheduler for distributed systems, characterized in that: include: A computer readable storage medium and a processor; The computer readable storage medium stores a computer program; The processor is used to read the computer program in the computer-readable storage medium and execute the distributed system-oriented phased adaptive request scheduling method according to any one of claims 1 to 9.

Images