MClock data flow control method and device of file system

Abstract

The invention relates to the technical field of cloud computing storage, and particularly provides an mClock data flow control method and device for a file system, and the method comprises the following steps: S1, declaring an mClock structure, i.e., defining a client-side related configuration and a data structure about mClock, including declaring related parameters of mClock, declaring an mClock queue and a computing request tag; s2, declaring an mClock flow control verification logic, namely, realizing the modified mClock flow control logic, which comprises a Constraint-based stage and a Weight-based stage; and S3, adding the flow control logic in the read-write operation, namely determining whether to continue to execute the subsequent read-write logic according to the mClock flow control logic verification result, including the flow control verification success logic and the flow control verification failure logic. Compared with the prior art, the method has the advantages that the situation that the use pressure of a single client is too large and the use of other clients is influenced can be reduced, the service life of the storage cluster is prolonged to a certain extent, and the probability that problems possibly occur in the later period is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of cloud computing storage, and specifically provides a method and device for controlling mClock data flow of a file system. Background Art

[0002] In current cloud storage usage scenarios, multiple clients often use the server resources of the same storage cluster. Since the server resources of the storage cluster are limited, flow control is usually used to limit the usage pressure of a single client, such as read and write iops, throughput, etc., so as to maintain the resource utilization rate of the storage cluster and prevent it from being too high and causing competition and waste of resources.

[0003] Flow control is a solution to limit the communication traffic between the client and the server. In a broad sense, it can be understood as limiting the request frequency and size of the client. The use of flow control can control and manage the transmission traffic in the network, thereby reducing network congestion and other problems caused by network fluctuations. At the same time, while reducing network pressure, it also allows the server pressure to be initially controlled, thereby extending the working life of the server.

[0004] When the usage pressure of a certain client is too high, it will have a negative impact on the usage of other clients. The resulting resource competition and resource occupation will affect the client's usage performance and user experience.

[0005] Being under excessively high pressure load for a long time will shorten the service life of servers and disks, thereby increasing the probability of problems with the storage cluster in the future. Summary of the invention

[0006] The present invention aims at solving the above-mentioned deficiencies of the prior art and provides a mClock data flow control method for a file system with strong practicability.

[0007] A further technical task of the present invention is to provide an mClock data flow control device for a file system that is reasonably designed, safe and applicable.

[0008] The technical solution adopted by the present invention to solve the technical problem is:

[0009] A mClock data flow control method for a file system comprises the following steps:

[0010] S1. Declare the mClock structure, that is, define the client's mClock-related configuration and data structure, including declaring mClock-related parameters, declaring mClock queues, and calculating request tags;

[0011] S2. Declare the mClock flow control verification logic, that is, implement the modified mClock flow control logic, including defining the Constraint-based stage and the Weight-based stage;

[0012] S3. Add flow control logic to the read and write operations, that is, determine whether to continue to execute subsequent read and write logic based on the mClock flow control logic verification result, including flow control verification success logic and flow control verification failure logic.

[0013] Further, in step S1, mClock related parameters in the read and write scenarios, that is, the reserved value, upper limit value and weight of the two types of read and write are declared. The specific meanings of the parameters are as follows:

[0014] Reserved value: the minimum limit for this type of operation, that is, the minimum read and write IOPS of a single client, with a value range greater than 0;

[0015] Upper limit: the maximum limit of this type of operation, that is, the maximum read and write IOPS of a single client, with a value range greater than 0;

[0016] Weight: The proportion of this type of operation in all operations, that is, the read and write ratio of a single client. The value range is greater than 0.

[0017] Furthermore, the calculation request tag, that is, the tag is calculated when the client is ready to read or write according to the parameters in the previous step. The specific tag calculation method is:

[0018]

[0019] Where r, w, l are the reserved value, upper limit value, and weight parameter set above, t is the current request time, r is the number of requests, and the label calculation formula for a request. The default label of the 0th request is 0.

[0020] Specifically, on the libcephfs client, add the minimum and maximum values ​​of read and write iops and the parameters of read and write request weights, and calculate the reservation label, upper limit label, and weight label when preparing to read and write.

[0021] Furthermore, the mClock queue is declared, that is, the reservation queue, upper limit queue and weight queue are declared in the client, and the reservation tag, upper limit tag and weight tag of each request are generated into a tag, wherein the request number of the request tag is the number of requests of the current client, and then the request tags are added to the corresponding reservation queue, upper limit queue and weight queue in order of the tag size;

[0022] In the libcephfs client, define the reserved queue, upper limit queue, and weight queue. The queue contains the 0th request label by default, and the label value is 0.

[0023] Further, in step S2, the constraint-based stage is defined, that is, checking in the reservation queue whether the reservation tag corresponding to the current request is less than or equal to the current time. If the condition is met, the verification is successful and the flow control verification logic is ended. Otherwise, the weight-based stage is executed;

[0024] Specifically, in the libcephfs client, the corresponding reservation label is found according to the request number of the request label. If the reservation label is less than or equal to the current time, verification success is returned, and the reservation label of the request label of the subsequent reservation queue is recalculated.

[0025] Furthermore, the Weight-based stage is defined, that is, checking in the upper limit queue whether the upper limit tag corresponding to this request is less than or equal to the current time. If the condition is met, the verification is successful, otherwise the verification fails, and both end the flow control verification logic.

[0026] Specifically, in the libcephfs client, the corresponding upper limit label is found according to the number of requests of the request label. If the upper limit label is less than or equal to the current time, verification success is returned.

[0027] Further, in step S3, after the flow control verification success logic, i.e., the read-write flow control verification succeeds, the reservation tag, upper limit tag, and weight tag of the previous request corresponding to the request are removed from the reservation queue, upper limit queue, and weight queue, and then the subsequent read-write logic is executed;

[0028] Specifically, when the flow control verification succeeds, the request label is removed from the reserved queue, upper limit queue, and weight queue of the libcephfs client, and the subsequent upper limit label and weight label are recalculated.

[0029] Furthermore, the flow control verification failure processing, that is, after the read-write flow control verification fails, wait for the time interval corresponding to the upper limit value and perform the flow control verification again until the verification succeeds;

[0030] Specifically, when flow control verification fails, the libcephfs client waits for a time interval corresponding to the upper limit value and then re-verifies until the verification succeeds. During the waiting period, the libcephfs exclusive lock is released to allow other requests to execute. Before verification, the libcephfs exclusive lock is acquired to strictly limit the serialization of flow control verification.

[0031] A mClock data flow control device for a file system, comprising: at least one memory and at least one processor;

[0032] The at least one memory is used to store a machine-readable program;

[0033] The at least one processor is used to call the machine-readable program to execute a mClock data flow control method for a file system.

[0034] Compared with the prior art, the mClock data flow control method and device of the file system of the present invention has the following outstanding beneficial effects:

[0035] The present invention reduces the impact on other clients due to excessive usage pressure on a single client by limiting the maximum resources that can be used by a single client.

[0036] At the same time, since an upper limit is set on the resources that can be used by a single client, the pressure on the storage cluster can be gradually controlled, thereby extending the service life of the storage cluster to a certain extent and reducing the probability of problems that may occur in the future. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0038] Attached Figure 1 The present invention is a flowchart of the mClock data flow control method of the file system. DETAILED DESCRIPTION

[0039] In order to enable those skilled in the art to better understand the solution of the present invention, the present invention is further described in detail below in conjunction with specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] A best embodiment is given below:

[0041] like Figure 1 As shown, in this embodiment, a mClock data flow control method for a file system uses the mClock algorithm on the client to perform flow control restrictions on data operations such as file system reading and writing, and the specific restrictions are the IOPS of read and write operations. In the scenario where there is a request to read and write file data, the read and write operation needs to be verified by the mClock algorithm whether it can be executed. If the verification is successful, the subsequent read and write logic will continue to be executed, otherwise the request lock will be released to wait for the next round of verification until the verification is successful.

[0042] Taking the cephfs file system as an example, the specific implementation steps of this solution are as follows:

[0043] S1. Declare the mClock structure, that is, define the client's mClock-related configuration and data structure, including declaring mClock-related parameters, declaring mClock queues, and calculating request tags;

[0044] (1) When reading and writing mClock related parameters in the scenario, the reserved value, upper limit value and weight of the two types of reading and writing are declared. The specific meanings of the parameters are as follows:

[0045] Reserved value: the minimum limit for this type of operation, that is, the minimum read and write IOPS of a single client, with a value range greater than 0;

[0046] Upper limit: the maximum limit of this type of operation, that is, the maximum read and write IOPS of a single client, with a value range greater than 0. The expected effect is that in a single-client scenario, the read and write IOPS is maintained between the reserved value and the upper limit;

[0047] Weight: The proportion of this type of operation in all operations, that is, the read and write ratio of a single client, with a value range greater than 0. Due to different read or write requirements in different business scenarios, the read and write weights will also be adjusted accordingly.

[0048] In this solution, the read-write ratio is tentatively set to 1:1. For some applications with high read demand, such as web browsing and database queries, the read-write ratio can be adjusted to 9:1.

[0049] Specifically, add the reserved value, upper limit value, and weight to the configuration of the libcephfs client. The default reserved value for read and write operations is 1, the upper limit value is 1000, and the weight is 1, which means that the read or write IOPS is limited to 1000. When a large number of read and write requests are processed, the total read and write IOPS is within 2000.

[0050] (2) Calculate the request tag, that is, calculate the tag when the client is ready to read or write based on the parameters in the previous step. The specific tag calculation method is:

[0051]

[0052] Where r, w, l are the reserved value, upper limit value, and weight parameter set above, t is the current request time, and r is the number of requests. For the label calculation formula of a request, the default value of the 0th request label is 0.

[0053] Specifically, on the libcephfs client, add the minimum and maximum values ​​of read and write iops and the parameters of read and write request weights, and calculate the reservation label, upper limit label, and weight label when preparing to read and write.

[0054] For example, if the reservation tag of the previous request is 10 and the current time is 11, the reservation tag of this request is the maximum value of 10.001 and 11, that is, 11.

[0055] (3) Declaring the mClock queue, that is, declaring the reservation queue, upper limit queue and weight queue in the client, and generating labels with the reservation label, upper limit label and weight label of each request, wherein the request number of the request label is the number of requests of the current client, and then adding the request labels to the corresponding reservation queue, upper limit queue and weight queue in order of label size;

[0056] In the libcephfs client, define the reserved queue, upper limit queue, and weight queue. The queue contains the 0th request label by default, and the label value is 0.

[0057] S2. Declare the mClock flow control verification logic, that is, implement the modified mClock flow control logic, including defining the Constraint-based stage and the Weight-based stage;

[0058] When defining the Constraint-based stage, check in the reservation queue whether the reservation tag corresponding to this request is less than or equal to the current time. If the condition is met, the verification is successful and the flow control verification logic ends. Otherwise, the Weight-based stage is executed.

[0059] Specifically, in the libcephfs client, the corresponding reservation label is found according to the request number of the request label. If the reservation label is less than or equal to the current time, verification success is returned, and the reservation label of the request label of the subsequent reservation queue is recalculated.

[0060] For example, assuming the current time is 12, when the reserved tag is 12, the verification is successful, and when the reserved tag is 13, it enters the Weight-based phase.

[0061] When defining the Weight-based phase, the upper limit tag corresponding to this request is checked in the upper limit queue to see if it is less than or equal to the current time. If the condition is met, the verification is successful, otherwise the verification fails. Both of them end the flow control verification logic.

[0062] Specifically, in the libcephfs client, the corresponding upper limit label is found according to the number of requests of the request label. If the upper limit label is less than or equal to the current time, verification success is returned.

[0063] For example, assuming the current time is 12, the verification is successful if the current upper limit tag is 12, and the verification fails if the upper limit tag is 13.

[0064] S3, add flow control logic to the read and write operations, that is, determine whether to continue to execute subsequent read and write logic according to the mClock flow control logic verification result, including flow control verification success logic and flow control verification failure logic;

[0065] When the flow control verification logic is successful, that is, after the read and write flow control verification is successful, the reservation label, upper limit label, and weight label of the previous request corresponding to the request are removed from the reservation queue, upper limit queue, and weight queue. Then the subsequent read and write logic is executed.

[0066] Specifically, when the flow control verification succeeds, the request label is removed from the reserved queue, upper limit queue, and weight queue of the libcephfs client, and the subsequent upper limit label and weight label are recalculated.

[0067] When processing flow control verification failure, the libcephfs client waits for the time interval corresponding to the upper limit value and then re-verifies until the verification succeeds. During the waiting period, the libcephfs exclusive lock is released to allow other requests to be executed. The libcephfs exclusive lock is obtained before verification to strictly limit the serialization of flow control verification.

[0068] For example, assuming that the time is 12, the upper limit tag is 13, and the upper limit value is 10, then after multiple cycles of verification, the time is 13, the upper limit is 13, the verification is successful, and subsequent read and write operations are performed.

[0069] Based on the above method, a mClock data flow control device of a file system in this embodiment includes: at least one memory and at least one processor;

[0070] The at least one memory is used to store a machine-readable program;

[0071] The at least one processor is used to call the machine-readable program to execute a mClock data flow control method for a file system.

[0072] The above-mentioned specific implementations are only specific cases of the present invention. The patent protection scope of the present invention includes but is not limited to the above-mentioned specific implementations. Any technical solutions that conform to the above-mentioned specific implementations of the present invention and any appropriate changes or substitutions made by ordinary technicians in the relevant technical field shall fall within the patent protection scope of the present invention.

[0073] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A mClock data flow control method for a file system, characterized in that: The steps are as follows: S1. Declare the mClock structure, that is, define the client's mClock-related configuration and data structure, including declaring mClock-related parameters, declaring mClock queues, and calculating request tags; S2. Declare the mClock flow control verification logic, that is, implement the modified mClock flow control logic, including defining the Constraint-based stage and the Weight-based stage; S3. Add flow control logic to the read and write operations, that is, determine whether to continue to execute subsequent read and write logic based on the mClock flow control logic verification result, including flow control verification success logic and flow control verification failure logic.

2. The mClock data flow control method of a file system according to claim 1, characterized in that: In step S1, mClock related parameters in the read and write scenarios, that is, the reserved value, upper limit value and weight of the two types of read and write are declared. The specific meanings of the parameters are as follows: Reserved value: the minimum limit for this type of operation, that is, the minimum read and write IOPS of a single client, with a value range greater than 0; Upper limit: the maximum limit of this type of operation, that is, the maximum read and write IOPS of a single client, with a value range greater than 0; Weight: The proportion of this type of operation in all operations, that is, the read and write ratio of a single client. The value range is greater than 0.

3. The mClock data flow control method of a file system according to claim 2, characterized in that: The calculation request tag is to calculate the tag when the client is ready to read or write according to the parameters in the previous step. The specific tag calculation method is: Where r, w, l are the reserved value, upper limit value, and weight parameter set above, t is the current request time, r is the number of requests, and the label calculation formula for a request. The default label of the 0th request is 0. Specifically, on the libcephfs client, the minimum and maximum values ​​of iops and the parameters of the read and write request weights are read and written respectively, and the reservation label, upper limit label, and weight label are calculated when preparing to read and write.

4. The mClock data flow control method of a file system according to claim 3, characterized in that: Declaring the mClock queue, that is, declaring the reservation queue, upper limit queue and weight queue in the client, and generating labels with the reservation label, upper limit label and weight label of each request, wherein the request number of the request label is the number of requests of the current client, and then adding the request labels to the corresponding reservation queue, upper limit queue and weight queue in order of label size; In the libcephfs client, define the reserved queue, upper limit queue, and weight queue. The queue contains the 0th request label by default, and the label value is 0.

5. The mClock data flow control method of a file system according to claim 4, characterized in that: In step S2, the constraint-based stage is defined, that is, checking in the reservation queue whether the reservation tag corresponding to the current request is less than or equal to the current time. If the condition is met, the verification is successful and the flow control verification logic ends. Otherwise, the weight-based stage is executed; Specifically, in the libcephfs client, the corresponding reservation label is found according to the request number of the request label. If the reservation label is less than or equal to the current time, verification success is returned, and the reservation label of the request label of the subsequent reservation queue is recalculated.

6. The mClock data flow control method of a file system according to claim 5, characterized in that: The Weight-based stage is defined, that is, checking in the upper limit queue whether the upper limit tag corresponding to this request is less than or equal to the current time. If the condition is met, the verification is successful, otherwise the verification fails, and both end the flow control verification logic. Specifically, in the libcephfs client, the corresponding upper limit label is found according to the number of requests of the request label. If the upper limit label is less than or equal to the current time, verification success is returned.

7. The mClock data flow control method of a file system according to claim 6, characterized in that: In step S3, after the flow control verification is successful, that is, the read-write flow control verification is successful, the reservation tag, upper limit tag and weight tag of the previous request corresponding to the request are removed from the reservation queue, upper limit queue and weight queue, and then the subsequent read-write logic is executed; Specifically, when the flow control verification succeeds, the request label is removed from the reserved queue, upper limit queue, and weight queue of the libcephfs client, and the subsequent upper limit label and weight label are recalculated.

8. The mClock data flow control method of a file system according to claim 6, characterized in that: The flow control verification failure processing, that is, after the read-write flow control verification fails, wait for the time interval corresponding to the upper limit value, and perform the flow control verification again until the verification succeeds; Specifically, when flow control verification fails, the libcephfs client waits for a time interval corresponding to the upper limit value and then re-verifies until the verification succeeds. During the waiting period, the libcephfs exclusive lock is released to allow other requests to execute. Before verification, the libcephfs exclusive lock is acquired to strictly limit the serialization of flow control verification.

9. A mClock data flow control device for a file system, characterized in that: include: at least one memory and at least one processor; The at least one memory is used to store a machine-readable program; The at least one processor is configured to call the machine-readable program to execute the method according to any one of claims 1 to 8.

Images