Distributed object storage cluster and service quality limitation method, device and medium

Abstract

The present invention discloses a distributed object storage cluster and a service quality limitation method, device, and medium, which relate to the field of distributed storage technology and are used to achieve cluster-level service quality limitation. Aiming at the problem that the traditional solution is inefficient and will affect the cluster performance, a distributed object storage service quality limitation method is provided, which selects a master node in the cluster, and the master node uniformly allocates corresponding bandwidth limitation values ​​and TPS limitation values ​​to all nodes in the cluster based on the pre-configured total bandwidth limitation value and total TPS limitation value. In this way, QOS limitation based on the cluster can be achieved, and only one communication between the master node and each slave node is required, and no two-to-two communication between all nodes in the cluster is required, which greatly reduces the occupation of cluster communication resources, thereby reducing the impact on cluster performance, and making the cluster QOS limitation based on this method can bring higher efficiency.

Description

Technical Field

[0001] The present invention relates to the field of distributed storage technology, and in particular to a distributed object storage cluster and a service quality limitation method, device and medium. Background Art

[0002] In the field of distributed storage, in order to ensure the stable operation of the cluster so that all users can normally access the cluster and use the services provided by the cluster, it is necessary to avoid a large number of requests from a single user occupying too many cluster resources, resulting in a decrease in the quality of service (QOS) of other users in the cluster.

[0003] Currently, the service quality of other users is mainly guaranteed by limiting the number of requests processed per second (Transaction PerSecond, TPS) and bandwidth of users and buckets in the cluster. Common TPS and bandwidth limitation schemes include limiting a single node and limiting a cluster. Among them, limiting the QOS of the cluster depends on frequent communication between nodes in the cluster to obtain the access volume of other nodes, and then calculate the restriction conditions corresponding to the node and impose restrictions. However, this scheme requires high-frequency node communication. When the cluster is large, a large number of node-to-node communications will occupy too many communication resources, which will be inefficient and affect the performance of the cluster.

[0004] Therefore, technicians in this field are in urgent need of a distributed object storage service quality limitation method to solve the problem that the current QOS limitation based on clusters is inefficient and affects cluster performance. Summary of the invention

[0005] The purpose of the present invention is to provide a distributed object storage cluster and a service quality restriction method, device, and medium to solve the problem that the current QOS restriction is inefficient and affects the cluster performance when the cluster is used as an object.

[0006] In order to solve the above technical problems, the present invention provides a method for limiting the quality of service of distributed object storage, which is applied to the master node side, and includes:

[0007] Get the total value of pre-configured bandwidth limit and total value of request limit per second;

[0008] Determine the bandwidth limit value and the request number limit value corresponding to each slave node according to the total bandwidth limit value and the total request number limit value per second;

[0009] Each group of the bandwidth limit value and the request processing number limit value per second is sent to each corresponding slave node, so that the slave node can schedule and process the request according to the corresponding bandwidth limit value and the request processing number limit value per second when receiving a request sent by a client.

[0010] In a possible embodiment, the method further includes:

[0011] Obtaining the actual value of bandwidth and the actual value of the number of requests processed per second returned by each slave node;

[0012] For each slave node, determine the difference between the corresponding bandwidth limit value and the returned bandwidth actual value as a first difference value; determine the difference between the corresponding request number limit value and the returned request number actual value as a second difference value;

[0013] If the first difference is greater than a preset first threshold, the first difference is subtracted from the original bandwidth limit value corresponding to the slave node to obtain a new bandwidth limit value, and the new bandwidth limit value is sent to the slave node; and any slave node whose returned bandwidth actual value is equal to the corresponding bandwidth limit value is determined, the first difference is added to the original bandwidth limit value of the slave node to obtain a new bandwidth limit value, and the new bandwidth limit value is sent to the slave node;

[0014] If the second difference is greater than a preset second threshold, the second difference is subtracted from the original limit value of the number of requests processed per second corresponding to the slave node to obtain a new limit value of the number of requests processed per second, and the new limit value is sent to the slave node; and any slave node whose actual value of the number of requests processed per second returned is equal to the corresponding limit value of the number of requests processed per second is determined, and the second difference is added to the original limit value of the number of requests processed per second of the slave node to obtain a new limit value of the number of requests processed per second, and the new limit value is sent to the slave node.

[0015] In a possible embodiment, determining the bandwidth limit value and the request processing number limit value corresponding to each slave node according to the bandwidth limit total value and the request processing number limit value per second includes:

[0016] Get the total number of slave nodes;

[0017] Taking the total number of slave nodes as the denominator, the total bandwidth limit value is evenly divided to obtain the bandwidth limit value corresponding to each slave node, and the total request processing per second limit value is evenly divided to obtain the request processing per second limit value corresponding to each slave node.

[0018] In a possible embodiment, after obtaining the actual value of the bandwidth and the actual value of the number of requests processed per second returned by each slave node, the method further includes:

[0019] If the actual value of the bandwidth and the actual value of the number of requests processed per second returned by a certain slave node are not obtained, it is determined that the slave node is faulty, and the total number of slave nodes is updated.

[0020] In order to solve the above technical problems, the present invention also provides a method for limiting the quality of service of distributed object storage, which is applied to the slave node side, comprising:

[0021] Receiving a bandwidth limit value and a request processing per second limit value sent by a master node; wherein the bandwidth limit value and the request processing per second limit value are determined by the master node for each slave node according to the bandwidth limit total value and the request processing per second limit total value after obtaining a pre-configured bandwidth limit total value and a pre-configured request processing per second limit total value;

[0022] When a request sent by a client is received, the request is scheduled and processed according to the corresponding bandwidth limit value and the limit value of the number of requests processed per second.

[0023] In a possible embodiment, the scheduling and processing of the request according to the corresponding bandwidth limit value and the request number limit value per second includes:

[0024] Adding the request to the end of a pre-established scheduling queue; wherein the request carries arrival time information;

[0025] For the request at the head of the scheduling queue, judging whether the request is waiting in the scheduling queue or being directly output according to the arrival time information of the request, the bandwidth limit value, and the limit value of the number of requests processed per second;

[0026] The request output by the scheduling queue is processed.

[0027] In a possible embodiment, the method further includes:

[0028] If the duration of not receiving the heartbeat information sent by the master node exceeds the preset timeout duration, a master node election is initiated, and an election voting request is sent to other slave nodes; wherein the master node sends heartbeat information to each slave node according to a preset time period, and the preset time period is less than the timeout duration corresponding to any slave node; the timeout durations corresponding to different slave nodes are different; when any slave node receives the election voting request, if it has not cast a vote during this master node election cycle, the vote is cast for the requester of the election voting request;

[0029] During this master node election cycle, if the vote is not cast for other slave nodes, the vote will be cast for oneself;

[0030] One of the slave nodes is elected as the new master node according to the voting result.

[0031] In a possible embodiment, selecting one of the slave nodes as the new master node according to the voting result includes:

[0032] When there is a slave node that obtains more than a preset proportion of votes from all nodes in the cluster, the slave node is determined to be the new master node;

[0033] If no slave node obtains votes from all nodes in the cluster exceeding the preset ratio, the election of the current master node is deemed to have failed, and the next time a slave node initiates the election of the master node is the time to wait;

[0034] The method further comprises:

[0035] If the number of consecutive master node election failures exceeds the preset warning threshold, an alarm message will be returned.

[0036] In order to solve the above technical problems, the present invention further provides a distributed object storage service quality limitation device, which is applied to the master node side and includes:

[0037] An acquisition module is used to obtain a pre-configured total value of bandwidth limit and a total value of the limit on the number of requests processed per second;

[0038] An allocation module, configured to determine a bandwidth limit value and a request processing number limit value corresponding to each slave node according to the total bandwidth limit value and the total request processing number limit value per second;

[0039] The communication module is used to send each group of the bandwidth limit value and the limit value of the number of requests processed per second to the corresponding slave nodes, so that the slave nodes can schedule and process the requests according to the corresponding bandwidth limit value and the limit value of the number of requests processed per second when receiving the requests sent by the client.

[0040] In order to solve the above technical problems, the present invention further provides a distributed object storage service quality limitation device, which is applied to a slave node side and includes:

[0041] A receiving module, configured to receive a bandwidth limit value and a request processing number limit value per second sent by a master node; wherein the bandwidth limit value and the request processing number limit value per second are determined by the master node for each slave node according to the bandwidth limit total value and the request processing number limit total value per second after obtaining a pre-configured bandwidth limit total value and a pre-configured bandwidth limit total value and a pre-configured bandwidth limit total value per second;

[0042] The scheduling module is used to schedule and process the request according to the corresponding bandwidth limit value and the request number limit value per second when receiving the request sent by the client.

[0043] To solve the above technical problems, the present invention further provides a distributed object storage cluster, comprising: a plurality of nodes; wherein one of the nodes is a master node, and all the other nodes are slave nodes;

[0044] When the node is the master node, steps for executing the method for limiting the quality of service of the distributed object storage applied to the master node side as described above;

[0045] When the node is the slave node, it is used to execute the steps of the service quality limitation method applied to the distributed object storage on the slave node side as described above.

[0046] To solve the above technical problem, the present invention also provides a non-volatile storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the service quality limitation method for distributed object storage as described above are implemented.

[0047] The present invention provides a method for limiting the quality of service of distributed object storage, which selects a master node in a cluster, and the master node uniformly allocates corresponding bandwidth limit values ​​and TPS limit values ​​to all nodes in the cluster based on the pre-configured total bandwidth limit value and total TPS limit value. In this way, QOS limit based on the cluster can be realized, and only one communication between the master node and each slave node is required, and no two-to-two communication between all nodes in the cluster is required, which greatly reduces the occupation of cluster communication resources, thereby reducing the impact on cluster performance, and making the cluster QOS limit based on this method can bring higher efficiency.

[0048] The distributed object storage cluster, distributed object storage service quality limitation device, and non-volatile storage medium provided by the present invention correspond to the above method and have the same effect as above. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0050] Figure 1 A flowchart of a method for limiting the quality of service of a distributed object storage applied to a master node side provided by an embodiment of the present invention;

[0051] Figure 2 A flowchart of a method for limiting the quality of service of a distributed object storage applied to a slave node side provided by an embodiment of the present invention;

[0052] Figure 3 A flow chart of a method for limiting and dynamically adjusting quality of service provided by an embodiment of the present invention;

[0053] Figure 4 A flow diagram of a node based on limiting service quality is provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] The core of the present invention is to provide a distributed object storage cluster and a service quality limitation method, device and medium.

[0056] In order to enable those skilled in the art to better understand the scheme of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0057] In related technologies, if you want to implement cluster-level Quality of Service (QOS) restrictions, each node in the cluster needs to frequently obtain the access volume of users and buckets of other nodes, so as to calculate the access restrictions required by this node. However, it is not difficult to see that this solution requires a large amount of frequent communication between nodes. When the cluster scale is large, it will occupy a large amount of communication resources in the cluster, thereby affecting the cluster performance. And a large amount of communication takes a long time, resulting in high latency for nodes to determine access restrictions.

[0058] To solve the above problems, the present invention provides a method for limiting the quality of service of distributed object storage, by electing a master node (leader) in the cluster, and the other nodes are slave nodes (followers). The master node implements the access restriction allocation of each node, thereby completing the QOS restriction at the cluster level. It is particularly important to note that the master node and slave node mentioned in the present invention are only a master-slave relationship between cluster nodes required when implementing this method. If there is a master-slave relationship based on other needs in the cluster, the master-slave relationship in this embodiment and the master-slave relationship may be the same or different, and the present invention does not limit this.

[0059] Since the method involves both the master node and the slave node in the cluster, the method provided by the present invention is also applied to both the master node and the slave node. In order to clearly illustrate the service quality limitation method of a distributed object storage provided by the present invention, the following embodiments will be described from both the master node and the slave node.

[0060] First, apply it to the master node side, such as Figure 1 As shown, the method includes:

[0061] S11: Obtain a pre-configured total value of bandwidth limit and total value of request number limit per second.

[0062] S12: Determine the bandwidth limit value and the request number limit value per second corresponding to each slave node according to the total bandwidth limit value and the request number limit value per second.

[0063] S13: Send each group of bandwidth limit values ​​and request processing number limit values ​​per second to each corresponding slave node.

[0064] So that the slave node: when receiving a request sent by a client, schedules the request according to the corresponding bandwidth limit value and the limit value of the number of requests processed per second.

[0065] Correspondingly, it is applied to the slave node side, such as Figure 2 As shown, the method includes:

[0066] S21: Receive the bandwidth limit value and the request number limit value per second sent by the master node.

[0067] The bandwidth limit value and the request limit value per second are determined by the master node for each slave node according to the bandwidth limit value and the request limit value per second after the master node obtains the pre-configured bandwidth limit value and the request limit value per second.

[0068] S22: When a request sent by a client is received, the request is scheduled and processed according to the corresponding bandwidth limit value and the limit value of the number of requests processed per second.

[0069] As can be seen from the above, this method limits the QOS of the cluster by limiting the bandwidth of the node and the number of requests processed per second (Transaction Per Second, TPS). Specifically, the user or relevant personnel can pre-configure the total bandwidth limit value and the total TPS limit value, that is, give an overall limit requirement for the QOS limit of the cluster. Then, after receiving the total bandwidth limit value and the total TPS limit value, the master node allocates the bandwidth limit value and TPS limit value corresponding to each slave node according to these two total limit values. After each slave node receives the bandwidth limit value and TPS limit value of the node assigned by the master node, it schedules and processes the requests received by the node from the client according to these two limit values, thereby achieving QOS restriction.

[0070] It is not difficult to see that the implementation of cluster-level QOS restrictions in this method only requires communication between the master node and the slave node. That is, only one node in the cluster needs to communicate with all other nodes. Compared with the traditional solution in which all nodes in the cluster need to communicate with other nodes, the number of communications between nodes can be greatly reduced. Especially when the cluster scale is large, the effect of this method in reducing the number of communications between nodes is more significant, that is, it can better guarantee the cluster performance, as well as the timeliness of each node determining its own QOS limit value, to achieve a better QOS limit effect.

[0071] Furthermore, considering that the method is implemented based on both the master node and the slave node, the present invention will also be described based on the master node and the slave node when providing other possible embodiments in the following. First, based on the master node side, this embodiment provides a possible implementation scheme, such as Figure 3 As shown, the above method also includes:

[0072] S14: Obtain the actual value of bandwidth and the actual value of the number of requests processed per second returned by each slave node.

[0073] For each slave node, there is:

[0074] S15: Determine the difference between the corresponding bandwidth limit value and the returned bandwidth actual value as the first difference value; determine the difference between the corresponding request number limit value per second and the returned request number actual value per second as the second difference value.

[0075] S16: If the first difference is greater than a preset first threshold, the first difference is subtracted from the original bandwidth limit value corresponding to the slave node to obtain a new bandwidth limit value, and the new bandwidth limit value is sent to the slave node; and any slave node whose returned bandwidth actual value is equal to the corresponding bandwidth limit value is determined, the first difference is added to the original bandwidth limit value of the slave node to obtain a new bandwidth limit value, and the new bandwidth limit value is sent to the slave node.

[0076] S17: If the second difference is greater than the preset second threshold, the second difference is subtracted from the original limit value of the number of requests processed per second corresponding to the slave node to obtain a new limit value of the number of requests processed per second, and the new limit value is sent to the slave node; and any slave node whose actual value of the number of requests processed per second returned is equal to the corresponding limit value of the number of requests processed per second is determined, and the second difference is added to the original limit value of the number of requests processed per second of the slave node to obtain a new limit value of the number of requests processed per second, and the new limit value is sent to the slave node.

[0077] That is, this embodiment provides a solution for dynamically adjusting the QOS limit value (i.e., bandwidth limit value and TPS limit value) assigned to the slave node according to the actual access situation of the slave node. For each slave node, it is necessary to feed back the actual value of its own bandwidth and TPS to the master node in real time so that the master node can know the real-time access situation of each slave node. If the actual bandwidth value and TPS actual value of the slave node are closer to the bandwidth limit value and TPS limit value assigned to the slave node, it means that the limit value assigned by the master node to the slave node is more suitable for the actual access situation of the slave node.

[0078] However, when the actual values ​​of the bandwidth and TPS of the slave node are significantly different from the corresponding limit values ​​(i.e., the first difference is greater than the first threshold, or the second difference is greater than the second threshold), it means that the bandwidth and TPS limit values ​​currently allocated by the master node to the slave node are not suitable for the actual service processing capacity of the slave node. The actual service processing capacity of the slave node is far from reaching the QOS limit value. In other words, for the slave node, the QOS limit value setting does not match the actual service processing capacity, and the slave node does not need such a high QOS limit value.

[0079] In addition, there is another situation where the limit value of the slave node bandwidth and TPS is equal to the actual value. At this time, there are two situations, namely, the actual service processing capacity of the slave node is exactly equal to the assigned QOS limit value. However, this situation is less likely in actual applications. It is more likely that the actual service processing capacity of the slave node is greater than the QOS limit value. It is just that because the QOS of the slave node is limited, the upper limit of the actual value of the slave node's bandwidth and TPS can only reach the corresponding limit value.

[0080] At this time, since the user has set a certain limit on the overall QOS of the cluster (i.e., the total limit of bandwidth and TPS is certain), if the actual business processing capacity of some slave nodes is far from the limit value, it means that the QOS limit in other slave nodes is too strict. In other words, when the QOS limit is set for each node in the cluster, the appropriate QOS limit value is not adaptively allocated to each slave node according to the actual pressure distribution in the cluster.

[0081] Based on this, this embodiment determines whether the actual bandwidth value and TPS actual value of the slave node are too far away from the corresponding allocated limit value based on the first threshold and the second threshold. If too far away, the allocation share of the limit value is allocated to other slave nodes whose actual business processing capacity is limited. This is to achieve dynamic adaptive adjustment of the node QOS limit value according to the pressure distribution in the cluster.

[0082] For example, suppose there are slave nodes A and B in the cluster, and the bandwidth limit values ​​of the two slave nodes in the initial state are both 50 (the same applies to the TPS limit value). After the initial limit value allocation is completed, the actual bandwidth value of node A is 25, and the actual bandwidth value of node B is 50. Assuming that the first threshold is 10, then for node A, 50-25=25>10, so for node A, the initially allocated bandwidth limit value is not accurate enough and needs further adjustment. At this time, based on the method provided in this embodiment, since the actual bandwidth value of node B is equal to the bandwidth upper limit value, the limited share of node A can be allocated to node B. That is, the newly allocated bandwidth limit value of node B = the original bandwidth limit value of node B + (the bandwidth limit value of node A - the actual bandwidth value of node A) = 50 + (50-25) = 75.

[0083] That is, based on the above adjustment, the sum of the bandwidth limit values ​​of all nodes in the cluster is guaranteed to remain unchanged and is still the total bandwidth limit value. However, for node A, the bandwidth limit value is more in line with the actual business processing capability of node A. As for node B, since its actual bandwidth value is equal to the bandwidth limit value. That is, for node B, it is more likely that its actual business processing capability has been limited. The adjustment based on this method can increase the limit value of node B, thereby releasing the actual business processing capability of node B, which is beneficial to improving the overall performance of the cluster.

[0084] It should also be noted that if the actual business processing capacity of node B is not as high as expected above. Then after the limit value of node B is adjusted, the difference between the actual bandwidth value and the bandwidth limit value of node B may exceed the first threshold. At this time, execute the above steps for node B to perform a new round of limit value adjustment. Based on multiple rounds of limit value adjustment, the limit values ​​of all slave nodes in the cluster can be made to fit their actual business processing capabilities (that is, the difference between the actual value and the limit value is less than the corresponding threshold). Thereby realizing a QOS limit value dynamic adjustment scheme that adapts to the dynamically changing pressure distribution of the cluster.

[0085] Furthermore, based on the QOS limit value dynamic adjustment scheme provided in the above embodiment, the accuracy of the cluster QOS limit is less affected by the accuracy of the QOS limit value that the master node first allocates to each slave node. In other words, even if the QOS limit value allocated by the master node for the first time is inaccurate, it can be gradually adjusted to be accurate through the dynamic adjustment scheme provided in the above embodiment. Therefore, when the master node allocates the QOS limit value for the first time, the corresponding allocation scheme can be designed with the highest efficiency as the first priority.

[0086] Based on this, this embodiment provides an implementation scheme for the primary node to initially allocate a limit value, that is, the above-mentioned step S12 is specifically as follows:

[0087] S121-A: Get the total number of slave nodes.

[0088] S122-A: Taking the total number of slave nodes as the denominator, the total bandwidth limit value is evenly divided to obtain the bandwidth limit value corresponding to each slave node, and the total request processing per second limit value is evenly divided to obtain the request processing per second limit value corresponding to each slave node.

[0089] That is, this embodiment provides a limit value allocation scheme for evenly dividing the total limit value. When the master node first allocates the corresponding QOS limit value to each slave node, it directly divides the total QOS limit value evenly to obtain the QOS limit value corresponding to each slave node. This scheme has low computational difficulty and high computational efficiency, that is, the master node is fast in initially allocating QOS limit values, and can start QOS restriction for all slave nodes in the cluster more quickly, which improves the timeliness and efficiency of this method. In addition, this evenly divided scheme is also a QOS restriction scheme that meets the restriction requirements and has better cluster performance when the pressure distribution in the cluster is unknown.

[0090] On the other hand, this embodiment also provides another possible implementation scheme applied to the master node side, and the above method further includes:

[0091] S123-A: If the actual value of the bandwidth and the actual value of the number of requests processed per second returned by a certain slave node are not obtained, it is determined that the slave node is faulty, and the total number of slave nodes is updated.

[0092] That is, this embodiment provides a scheme for a master node to detect the health status of a slave node. In the above embodiment, the slave node needs to feedback the actual QOS value in real time and also for the master node to dynamically adjust the QOS limit value. Therefore, when the slave node fails, the slave node cannot feedback the actual QOS value to the master node. Based on this, the master node can also determine whether the slave node is faulty by whether the actual QOS value is received.

[0093] In addition, when the master node determines that the slave node is faulty, it is necessary to readjust the QOS limit value. The faulty node cannot provide services, that is, there is no need to impose QOS restrictions on it. Its QOS share can be distributed to other nodes to free up the business processing capabilities of other nodes, thereby ensuring the overall performance of the cluster. This is reflected in the present embodiment, that is, the number of slave nodes is updated in a timely manner. Therefore, when the QOS limit value is evenly distributed in the above embodiment, more QOS shares can be allocated to each slave node.

[0094] In addition, this embodiment provides another possible implementation scheme for the master node to initially allocate the QOS limit value, and the above step S12 is specifically as follows:

[0095] S121-B: Obtain the bandwidth historical data and the number of requests processed per second historical data of each slave node.

[0096] The bandwidth historical data is the actual bandwidth value data recorded by the slave node during the historical operation process, and the number of requests processed per second historical data is the actual number of requests processed per second data recorded by the slave node during the historical operation process.

[0097] S122-B: Determine a bandwidth prediction value and a request processing number prediction value of each slave node according to the bandwidth historical data and the request processing number historical data.

[0098] S123-B: Determine the bandwidth limit value of each slave node according to the proportional relationship between the corresponding bandwidth prediction values ​​of each slave node; determine the limit value of the number of requests processed per second of each slave node according to the proportional relationship between the corresponding predicted values ​​of the number of requests processed per second of each slave node.

[0099] The proportional relationship between the bandwidth limit values ​​corresponding to each slave node is equal to the proportional relationship between the bandwidth prediction values ​​of each slave node. The proportional relationship between the request number limit values ​​corresponding to each slave node per second is equal to the proportional relationship between the request number prediction values ​​of each slave node per second.

[0100] It should be noted that the bandwidth prediction value and TPS prediction value are predicted based on the changes in the corresponding historical data. Or in a possible embodiment, in order to reduce the difficulty of calculation and improve efficiency, the bandwidth prediction value and TPS prediction value can select the most recent bandwidth value and TPS value in history.

[0101] Based on the above, this embodiment provides another implementation scheme for the master node to initially allocate QOS limit values ​​to each slave node. Compared with the allocation scheme provided in the previous embodiment, this scheme allocates limit values ​​with higher accuracy, and the calculation difficulty does not increase much. In fact, since this method is more accurate when initially allocating QOS limit values, when the limit values ​​of each node are dynamically adjusted through the limit value dynamic adjustment scheme, the number of adjustments or the adjustment time required may be less. That is, compared with the previous scheme, this scheme may make the cluster limit value enter a stable state more quickly.

[0102] In summary, the two allocation schemes provided in the above embodiment have their own advantages and disadvantages. In practical applications, different allocation schemes can be selected according to actual needs, and this embodiment does not limit this.

[0103] On the other hand, for the slave node side, this embodiment also provides a possible embodiment. The above step S22 applied to the slave node side specifically includes:

[0104] S221: Add the request to the end of the pre-established scheduling queue.

[0105] The request carries arrival time information.

[0106] S222: For the request at the head of the scheduling queue, determine whether the request is waiting in the scheduling queue or is directly output according to the arrival time information of the request, the bandwidth limit value, and the limit value of the number of requests processed per second.

[0107] S223: Process the request output by the scheduling queue.

[0108] Specifically, in actual applications, the management of the scheduling queue in the above steps can be implemented through a functional module, such as the Mclock module. Mclock is an input-output (IO) scheduling algorithm that can calculate whether a request is directly processed or needs to wait in the scheduling queue for a period of time before being processed based on the time when the request arrives and the pre-configured bandwidth and TPS limit value. That is, the Mclock algorithm can be used to implement the above step S222.

[0109] Furthermore, the actual QOS restriction implementation process and the hardware architecture involved are as follows: Figure 4As shown. The user sends a request to the cluster through the client. This request is generally a network request, which is implemented based on the Hypertext Transfer Protocol (HTTP) or Hypertext Transfer Protocol Secure (HTTPS). When the request enters the object storage part of the cluster (Rados Gateway, RGW), it is first received by the network (World Wide Web, Web) server inside the RGW. When the Web server receives the request, it will start a thread to process the request, that is, add the request to the scheduling queue (Mclock queue). Since the request carries the arrival time information, the Mclock algorithm can calculate whether the request is directly dequeued or needs to continue waiting in the scheduling queue according to the request arrival time and the pre-configured bandwidth limit value and TPS limit value. If the request is dequeued, the request is processed. Since the requests for the distributed storage cluster generally involve data stored in the cluster or writing data to the cluster. Therefore, these requests generally need to interact with the storage in the distributed storage system (the component or device responsible for storing and managing data in the distributed storage system, which can be called a storage device).

[0110] On the other hand, it is not difficult to see from the above that this method relies on the master node to implement the allocation of QOS limit values. In actual applications, the master node may have problems such as failures. In view of this, this embodiment also provides a possible implementation plan, and the above method also includes:

[0111] S231: If the duration of not receiving the heartbeat information sent by the master node exceeds the preset timeout duration, a master node election is initiated and an election voting request is sent to other slave nodes.

[0112] The master node sends heartbeat information to each slave node according to a preset period, which is less than the timeout period corresponding to any slave node. The timeout periods corresponding to different slave nodes are different. When any slave node receives an election voting request, if it has not voted in this master node election cycle, it will vote for the requester of the election voting request.

[0113] S232: During this master node election cycle, if you do not vote for other slave nodes, you will vote for yourself.

[0114] S233: A slave node is elected as a new master node according to the voting result.

[0115] That is, in this embodiment, the master node periodically sends heartbeat information to other slave nodes to indicate its own health status. If the master node fails, the slave node cannot receive the heartbeat information. When a slave node does not receive the heartbeat information sent by the master node for a continuous timeout period, it can be considered that the master node is faulty. At this time, a new master node needs to be re-elected to continue the allocation or dynamic adjustment of the QOS limit value.

[0116] Specifically, when any slave node discovers that the master node has failed, it initiates a master node election, lists itself as a candidate, casts a vote for itself, and sends an election request to other slave nodes. When a slave node receives an election request, if the slave node has not cast a vote in this master node election cycle, it will vote for the requester. Since the timeout duration of each slave node is different, the time when each slave node discovers the master node failure is also different. Therefore, there is always a slave node that first discovers the master node failure and initiates an election. At this time, if other slave nodes have not discovered the master node failure, they will vote for the slave node to elect this slave node as the new master node and continue to complete the QOS limit value allocation task.

[0117] Furthermore, it is easy to know that communication between nodes takes time, so even if the timeout durations of different nodes are different, it is not ideal that all slave nodes will vote for the first slave node that discovers the failure of the master node. Therefore, in a possible implementation scheme, step S223 can determine the new master node according to the number of votes obtained by each slave node. The slave node with the most votes can be elected as the new master node. If multiple slave nodes obtain the same number of votes and the number is the largest, it can be regarded as a failure of this election, and a new election will be held to select a single slave node as the new master node.

[0118] However, this embodiment also provides another possible solution for electing a master node, and the above step S223 specifically includes:

[0119] When a slave node obtains votes from all nodes in the cluster exceeding a preset ratio, the slave node is determined to be the new master node.

[0120] If no slave node obtains more than the preset proportion of votes from all nodes in the cluster, the current master node election is considered to have failed, and the next slave node will initiate the master node election.

[0121] Accordingly, the method further comprises:

[0122] S234: If the number of consecutive master node election failures exceeds a preset warning threshold, an alarm message is returned.

[0123] The specific value of the above-mentioned preset ratio is not limited in this embodiment. A suitable value can be selected according to actual needs. The purpose of setting the preset ratio is to ensure that the elected master node is recognized by a certain number of nodes in the cluster. It can be ensured that the number of available nodes in the cluster is greater than the preset ratio of the total number of clusters. In other words, when the number of nodes in the cluster drops to a certain ratio of the original total number of nodes, the cluster is considered to have a fault, and the above-mentioned preset ratio can be set to the ratio value.

[0124] Based on the above, it can be seen that this embodiment also incorporates a cluster health status judgment mechanism in the master node election. By determining the condition for the success of the master node election as obtaining votes from slave nodes above a preset proportion in the cluster, the number of available nodes in the cluster can be detected while electing the master node. If the proportion of the number of available nodes in the cluster to the total number of nodes in the cluster is less than the preset proportion, no new master node can be elected no matter how many times the master node election is repeated. Therefore, based on step S234 of this embodiment, an alarm message is returned at this time to inform relevant personnel of the fault condition, thereby facilitating the operation and maintenance personnel to take timely measures to ensure the availability of the cluster.

[0125] In addition to the embodiment of a distributed object storage service quality limitation method provided in the above embodiment, the present invention also provides an embodiment corresponding to a computer program product. A computer program product includes a computer program / instruction, and when the computer program / instruction is executed by a processor, the steps of the distributed object storage service quality limitation method described in any of the above embodiments can be implemented.

[0126] Since the embodiments of the computer program product part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the computer program product part, which will not be repeated here.

[0127] In the above embodiment, a distributed object storage service quality restriction method is described in detail, and the present invention also provides a corresponding embodiment of a distributed object storage service quality restriction device. It should be noted that the above method is described based on both the master node and the slave node, so when this embodiment provides an embodiment of the device part, it is also described from both the master node and the slave node.

[0128] Applied to the master node side, this embodiment provides a distributed object storage service quality limitation device, including:

[0129] An acquisition module is used to obtain a pre-configured total value of bandwidth limit and a total value of the limit on the number of requests processed per second;

[0130] An allocation module, used for determining a bandwidth limit value and a request processing number limit value corresponding to each slave node according to a total bandwidth limit value and a request processing number limit value per second;

[0131] The communication module is used to send each group of bandwidth limit values ​​and the limit value of the number of requests processed per second to the corresponding slave nodes, so that the slave nodes can schedule and process the requests according to the corresponding bandwidth limit value and the limit value of the number of requests processed per second when receiving the requests sent by the client.

[0132] Applied to the slave node side, this embodiment provides a distributed object storage service quality limitation device, including:

[0133] A receiving module, used for receiving the bandwidth limit value and the request number limit value per second sent by the master node; wherein the bandwidth limit value and the request number limit value per second are determined by the master node for each slave node according to the bandwidth limit total value and the request number limit value per second after obtaining the pre-configured bandwidth limit total value and the request number limit value per second;

[0134] The scheduling module is used to schedule and process the request according to the corresponding bandwidth limit value and the limit value of the number of requests processed per second when receiving the request sent by the client.

[0135] Since the embodiments of the apparatus part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the apparatus part, which will not be repeated here.

[0136] In addition, this embodiment also provides an embodiment corresponding to a distributed object storage cluster. A distributed object storage cluster provided in this embodiment includes multiple nodes, wherein one node is a master node and all other nodes are slave nodes.

[0137] When the node is a master node, it is used to execute the service quality limitation method for distributed object storage applied to the master node side as provided in the above embodiment.

[0138] When the node is a slave node, it is used to execute the service quality limitation method for distributed object storage applied to the slave node side as provided in the above embodiment.

[0139] Since the embodiments of the cluster part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the cluster part. Specifically, the embodiments of the master node in the cluster can refer to the above-mentioned method embodiments applied to the master node side, and the embodiments of the slave nodes in the cluster can refer to the above-mentioned method embodiments applied to the slave node side, which will not be repeated here.

[0140] Finally, the present invention also provides an embodiment corresponding to a non-volatile storage medium. The non-volatile storage medium stores a computer program, and when the computer program is executed by a processor, the steps recorded in the above method embodiment are implemented.

[0141] It is understandable that if the method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc. Various media that can store program codes.

[0142] The above is a detailed introduction to a distributed object storage cluster and a method, device, and medium for limiting quality of service provided by the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referenced to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, the present invention can also be improved and modified in several ways, and these improvements and modifications also fall within the scope of protection of the present invention.

[0143] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

Claims

1. A method for limiting the quality of service of a distributed object storage, characterized in that: Applied to the master node side, including: Get the total value of pre-configured bandwidth limit and total value of request limit per second; Determine the bandwidth limit value and the request number limit value corresponding to each slave node according to the total bandwidth limit value and the total request number limit value per second; Each group of the bandwidth limit value and the request processing number limit value per second is sent to each corresponding slave node, so that the slave node can schedule and process the request according to the corresponding bandwidth limit value and the request processing number limit value per second when receiving a request sent by a client.

2. The method for limiting the quality of service of distributed object storage according to claim 1, characterized in that: The method further comprises: Obtaining the actual value of bandwidth and the actual value of the number of requests processed per second returned by each slave node; For each slave node, determine the difference between the corresponding bandwidth limit value and the returned bandwidth actual value as a first difference value; determine the difference between the corresponding request number limit value and the returned request number actual value as a second difference value; If the first difference is greater than a preset first threshold, the first difference is subtracted from the original bandwidth limit value corresponding to the slave node to obtain a new bandwidth limit value, and the new bandwidth limit value is sent to the slave node; and any slave node whose returned bandwidth actual value is equal to the corresponding bandwidth limit value is determined, the first difference is added to the original bandwidth limit value of the slave node to obtain a new bandwidth limit value, and the new bandwidth limit value is sent to the slave node; If the second difference is greater than a preset second threshold, the second difference is subtracted from the original limit value of the number of requests processed per second corresponding to the slave node to obtain a new limit value of the number of requests processed per second, and the new limit value is sent to the slave node; and any slave node whose actual value of the number of requests processed per second returned is equal to the corresponding limit value of the number of requests processed per second is determined, and the second difference is added to the original limit value of the number of requests processed per second of the slave node to obtain a new limit value of the number of requests processed per second, and the new limit value is sent to the slave node.

3. The method for limiting the quality of service of distributed object storage according to claim 2, characterized in that: Determining the bandwidth limit value and the request number limit value corresponding to each slave node according to the bandwidth limit total value and the request number limit value per second includes: Get the total number of slave nodes; Taking the total number of slave nodes as the denominator, the total bandwidth limit value is evenly divided to obtain the bandwidth limit value corresponding to each slave node, and the total request processing per second limit value is evenly divided to obtain the request processing per second limit value corresponding to each slave node.

4. The method for limiting the quality of service of distributed object storage according to claim 3, characterized in that: After obtaining the actual value of bandwidth and the actual value of number of requests processed per second returned by each slave node, the method further includes: If the actual value of the bandwidth and the actual value of the number of requests processed per second returned by a certain slave node are not obtained, it is determined that the slave node is faulty, and the total number of slave nodes is updated.

5. A method for limiting the quality of service of a distributed object storage, characterized in that: Applied to the slave node side, including: Receiving a bandwidth limit value and a request processing per second limit value sent by a master node; wherein the bandwidth limit value and the request processing per second limit value are determined by the master node for each slave node according to the bandwidth limit total value and the request processing per second limit total value after obtaining a pre-configured bandwidth limit total value and a pre-configured request processing per second limit total value; When a request sent by a client is received, the request is scheduled and processed according to the corresponding bandwidth limit value and the limit value of the number of requests processed per second.

6. The method for limiting the quality of service of distributed object storage according to claim 5, characterized in that: The scheduling of the request according to the corresponding bandwidth limit value and the request number limit value per second includes: Adding the request to the end of a pre-established scheduling queue; wherein the request carries arrival time information; For the request at the head of the scheduling queue, judging whether the request is waiting in the scheduling queue or being directly output according to the arrival time information of the request, the bandwidth limit value, and the limit value of the number of requests processed per second; The request output by the scheduling queue is processed.

7. The method for limiting the quality of service of distributed object storage according to claim 5, characterized in that: The method further comprises: If the duration of not receiving the heartbeat information sent by the master node exceeds the preset timeout duration, a master node election is initiated, and an election voting request is sent to other slave nodes; wherein the master node sends heartbeat information to each slave node according to a preset time period, and the preset time period is less than the timeout duration corresponding to any slave node; the timeout durations corresponding to different slave nodes are different; when any slave node receives the election voting request, if it has not cast a vote during this master node election cycle, the vote is cast for the requester of the election voting request; During this master node election cycle, if the vote is not cast for other slave nodes, the vote will be cast for oneself; One of the slave nodes is elected as the new master node according to the voting result.

8. The method for limiting the quality of service of distributed object storage according to claim 5, characterized in that: The selecting one of the slave nodes as the new master node according to the voting result comprises: When there is a slave node that obtains more than a preset proportion of votes from all nodes in the cluster, the slave node is determined to be the new master node; If no slave node obtains votes from all nodes in the cluster exceeding the preset ratio, the election of the current master node is deemed to have failed, and the next time a slave node initiates the election of the master node is the time to wait; The method further comprises: If the number of consecutive master node election failures exceeds the preset warning threshold, an alarm message will be returned.

9. A distributed object storage service quality limiting device, characterized in that: Applied to the master node side, including: An acquisition module is used to obtain a pre-configured total value of bandwidth limit and total value of request processing limit per second; An allocation module, configured to determine a bandwidth limit value and a request processing number limit value corresponding to each slave node according to the total bandwidth limit value and the total request processing number limit value per second; The communication module is used to send each group of the bandwidth limit value and the limit value of the number of requests processed per second to the corresponding slave nodes, so that the slave nodes can schedule and process the requests according to the corresponding bandwidth limit value and the limit value of the number of requests processed per second when receiving the requests sent by the client.

10. A distributed object storage service quality limitation device, characterized in that: Applied to the slave node side, including: A receiving module, configured to receive a bandwidth limit value and a request processing number limit value per second sent by a master node; wherein the bandwidth limit value and the request processing number limit value per second are determined by the master node for each slave node according to the bandwidth limit total value and the request processing number limit total value per second after obtaining a pre-configured bandwidth limit total value and a pre-configured bandwidth limit total value and a pre-configured bandwidth limit total value per second; The scheduling module is used to schedule and process the request according to the corresponding bandwidth limit value and the request number limit value per second when receiving the request sent by the client.

11. A distributed object storage cluster, characterized in that: include: Multiple nodes; one of the nodes is a master node, and all the other nodes are slave nodes; When the node is the master node, the method is used to execute the steps of the method for limiting the quality of service of distributed object storage according to any one of claims 1 to 4; When the node is the slave node, it is used to execute the steps of the distributed object storage service quality limitation method as described in any one of claims 5 to 8.

12. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a computer program, which, when executed by a processor, implements the steps of the distributed object storage quality of service limitation method as described in any one of claims 1 to 8.

Images