A flow control method and device, electronic equipment and storage medium

Abstract

The application discloses a flow control method and device, an electronic device and a computer readable storage medium. The method comprises the following steps: determining a residual capacity ratio of a cluster, and adjusting a garbage recovery flow control value, a service flow control value and a flow control ratio of garbage recovery and service of the cluster according to the residual capacity ratio; wherein the flow control ratio is negatively correlated with the residual capacity ratio; distributing the service flow control value to nodes in the cluster based on service pressure values of the nodes; wherein the nodes comprise storage nodes and computing nodes, and the service quota distributed to the nodes is positively correlated with the service pressure of the nodes; and distributing the garbage recovery flow control value to the storage nodes based on service pressure values of the storage nodes; wherein the garbage recovery quota distributed to the storage nodes is negatively correlated with the service pressure of the storage nodes. The flow control method provided by the application realizes the balance between service and garbage recovery, and does not affect the service performance and cause the storage space to be exhausted.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, more particularly, to a traffic control method and device, an electronic device and a computer readable storage medium. BACKGROUND

[0002] The application of big data era has higher and higher requirements on storage performance, and the traditional storage system based on mechanical disk is difficult to meet the demand of high performance of these applications. The log-structured storage system based on flash memory makes use of the characteristics of fast sequential access of flash memory, which can greatly improve the performance of the storage system, and thus begins to be widely used. The log-structured storage system is a storage system that sequentially appends data and establishes address mapping relationship through write-time redirection. The garbage collection mechanism is used to recycle invalid data space not pointed by the address mapping to ensure that the storage space is not exhausted by garbage. However, since there is a direct competition relationship between business and garbage collection for system resources, too much resource occupation of business will lead to low efficiency of garbage collection, thus causing storage space exhaustion, and even causing business to be stuck; too much resource occupation of garbage collection will significantly affect business performance and service quality.

[0003] Therefore, how to balance business and garbage collection, without affecting business performance and causing storage space exhaustion, is a technical problem to be solved by those skilled in the art. SUMMARY

[0004] The purpose of the present application is to provide a traffic control method and device, an electronic device and a computer readable storage medium, which balance business and garbage collection, without affecting business performance and causing storage space exhaustion.

[0005] To achieve the above purpose, the present application provides a traffic control method, comprising:

[0006] determining a remaining capacity ratio of a cluster, and adjusting a garbage collection flow control value, a business flow control value and a flow control ratio of garbage collection and business of the cluster according to the remaining capacity ratio; wherein the flow control ratio is negatively correlated with the remaining capacity ratio;

[0007] allocating the business flow control value to nodes in the cluster based on a business pressure value of the nodes; wherein the nodes include storage nodes and computing nodes, and the business quota allocated to the nodes is positively correlated with the business pressure of the nodes;

[0008] allocating the garbage collection flow control value to the storage nodes in the cluster based on a business pressure value of the storage nodes; wherein the garbage collection quota allocated to the storage nodes is negatively correlated with the business pressure of the storage nodes.

[0009] The determining the remaining capacity ratio of the cluster comprises:

[0010] The capacity information of the nodes in the cluster is acquired every first preset time; wherein, the capacity information comprises total capacity, garbage capacity and remaining capacity;

[0011] After the second preset time is accumulated, the total capacity of the cluster and the remaining capacity of the cluster are calculated; wherein, the second preset time is an integer multiple of the first preset time;

[0012] The remaining capacity ratio of the cluster is determined according to the total capacity of the cluster and the remaining capacity of the cluster.

[0013] The adjusting the garbage recovery flow control value, the business flow control value and the garbage recovery and business flow control ratio of the cluster according to the remaining capacity ratio comprises:

[0014] If the remaining capacity ratio is greater than a first preset value and less than 1, the garbage recovery flow control value of the cluster is adjusted to 0, the business flow control value is adjusted to infinity, and the garbage recovery and business flow control ratio is adjusted to 0;

[0015] If the remaining capacity ratio is greater than a second preset value and less than or equal to the first preset value, the garbage recovery and business flow control ratio is adjusted to a first target value according to the level to which the remaining capacity ratio belongs; wherein, the level to which the remaining capacity ratio belongs is negatively correlated with the remaining capacity ratio, the first target value is negatively correlated with the level to which the remaining capacity ratio belongs, and the first target value corresponding to different levels is all greater than 0 and less than 1;

[0016] If the remaining capacity ratio is greater than a third preset value and less than or equal to the second preset value, the garbage recovery and business flow control ratio is adjusted to a second target value; wherein, the second target value is greater than 1;

[0017] If the remaining capacity ratio is greater than 0 and less than or equal to a fourth preset value, the garbage recovery flow control value of the cluster is adjusted to infinity, the business flow control value is adjusted to a fixed value, and the garbage recovery and business flow control ratio is adjusted to infinity.

[0018] The method further comprises:

[0019] The token bucket algorithm is used to control the traffic of the node business based on the business quota allocated to the node;

[0020] The timestamp algorithm is used to control the traffic of the garbage recovery of the storage node based on the garbage recovery quota allocated to the storage node.

[0021] The token bucket algorithm is used to control the traffic of the node business based on the business quota allocated to the node, comprising:

[0022] complement the token quantity in the token bucket based on the service quota allocated by the node;

[0023] determine whether the contribution value of the current service task is less than or equal to the current token quantity in the token bucket;

[0024] if yes, issue the current service task and update the current token quantity;

[0025] if no, determine the scheduling time of the current service task according to the contribution value of the current service task, the current token quantity and the service quota allocated by the node.

[0026] wherein the time stamp algorithm is used to control the flow of the garbage collection of the storage node based on the garbage collection quota allocated by the storage node, comprising:

[0027] calculate the time stamp mark of the current garbage collection task according to the contribution value of the previous garbage collection task, the time stamp mark and the garbage collection quota allocated by the storage node;

[0028] determine whether the time stamp mark of the current garbage collection task is less than or equal to the current scheduling time;

[0029] if yes, issue the current garbage collection task;

[0030] if no, issue the current garbage collection task when the current scheduling time reaches the time stamp mark of the current garbage collection task.

[0031] wherein the calculation of the time stamp mark of the current garbage collection task according to the contribution value of the previous garbage collection task, the time stamp mark and the garbage collection quota allocated by the storage node comprises:

[0032] calculate the execution time of the previous garbage collection task according to the contribution value of the previous garbage collection task and the garbage collection quota allocated by the storage node;

[0033] set the sum of the time stamp mark of the previous garbage collection task and the execution time as the time stamp mark of the current garbage collection task.

[0034] To achieve the above object, the application provides a flow control device, comprising:

[0035] a determination module configured to determine the remaining capacity proportion of a cluster and adjust the garbage collection flow control value, the service flow control value and the flow control proportion of garbage collection and service of the cluster according to the remaining capacity proportion; wherein the flow control proportion is negatively correlated with the remaining capacity proportion;

[0036] a first distribution module configured to distribute the service flow control value to nodes in the cluster based on service pressure values of the nodes; wherein the nodes include storage nodes and computing nodes, and the service quota allocated to the nodes is positively correlated with the service pressure of the nodes;

[0037] a second distribution module configured to distribute the garbage collection flow control value to the storage nodes in the cluster based on service pressure values of the storage nodes; wherein the garbage collection quota allocated to the storage nodes is negatively correlated with the service pressure of the storage nodes.

[0038] To achieve the above object, the present application provides an electronic device, comprising:

[0039] a memory configured to store a computer program;

[0040] a processor configured to execute the computer program to implement the steps of the traffic control method.

[0041] To achieve the above object, the present application provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the traffic control method.

[0042] According to the above scheme, the traffic control method provided by the present application comprises: determining a residual capacity ratio of a cluster, and adjusting a garbage collection flow control value, a service flow control value and a flow control ratio of garbage collection and service of the cluster according to the residual capacity ratio; wherein the flow control ratio is negatively correlated with the residual capacity ratio; distributing the service flow control value to nodes in the cluster based on service pressure values of the nodes; wherein the nodes include storage nodes and computing nodes, and the service quota allocated to the nodes is positively correlated with the service pressure of the nodes; and distributing the garbage collection flow control value to the storage nodes in the cluster based on service pressure values of the storage nodes; wherein the garbage collection quota allocated to the storage nodes is negatively correlated with the service pressure of the storage nodes.

[0043] The traffic control method provided in the application first adaptively calculates the flow control ratio of garbage collection and business according to the residual capacity ratio of the cluster. The flow control ratio is negatively correlated with the residual capacity ratio. When the residual capacity ratio is large, there is no risk of space exhaustion, and the business can not be controlled. The garbage collection is executed in depth on the basis of meeting the triggering condition, so as to reduce the influence on the business as much as possible. When the residual capacity of the cluster is small, the flow control value ratio of garbage collection and business gradually increases, so as to ensure that the space is not exhausted and reduce the influence on the business as much as possible. Secondly, the business flow control value is distributed to each node and the garbage collection flow control value is allocated to the storage node according to the business pressure of each node, so as to ensure the load balancing of the business of each node. The application also discloses a traffic control device, an electronic equipment and a computer readable storage medium, which can also achieve the above technical effects.

[0044] It should be understood that the above general description and the following detailed description are only exemplary and cannot limit the application. BRIEF DESCRIPTION OF DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings. The drawings are used to provide further understanding of the disclosure and constitute a part of the specification, and are used to explain the disclosure together with the following specific embodiments, but do not constitute a limitation on the disclosure. In the drawings:

[0046] Figure 1 The architecture diagram of a traffic control system according to an exemplary embodiment is shown.

[0047] Figure 2 The flowchart of a traffic control method according to an exemplary embodiment is shown.

[0048] Figure 3 The flowchart of another traffic control method according to an exemplary embodiment is shown.

[0049] Figure 4 The flowchart of a token bucket flow control algorithm of business IO according to an exemplary embodiment is shown.

[0050] Figure 5 The flowchart of a timestamp flow control algorithm of garbage collection according to an exemplary embodiment is shown.

[0051] Figure 6 The structure diagram of a traffic control device according to an exemplary embodiment is shown.

[0052] Figure 7 FIG. 1 is a structural diagram of an electronic device according to an example embodiment. DETAILED DESCRIPTION

[0053] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the scope of protection of the present application. In addition, in the embodiments of the present application, "first", "second", and the like are used to distinguish similar objects, not necessarily to describe a specific order or sequence.

[0054] In order to facilitate understanding of the flow control method provided by the present application, the system used by the present application will be introduced below. Referring to Figure 1 , which is an architecture diagram of a flow control system according to an example embodiment, as shown in Figure 1 , including a storage server, a storage client and a global manager module.

[0055] Among them, the storage server is used to respond to the request of the storage client, and is responsible for the functions of data distribution, reading, disk landing, etc., including Server0, Server1 and Server3.

[0056] The storage client contains business and garbage collection processes, wherein the storage nodes include node0, node1 and node2, and the computing nodes include node3, node4 and node5. Each storage node adapts the flow control function at the entrance of the business IO and the garbage collection process. The computing node only needs to adapt the business flow control function because it does not perform garbage collection. VMi represents the business process on the ith node, VM_FCi represents the business flow control process on the ith node, GCi represents the garbage collection process on the ith node, and GC_FCi represents the garbage collection flow control process on the ith node. In the present application, different flow control technologies are applied to the properties of business and garbage collection. Considering that the business needs burst capability to ensure burst performance, the business flow control uses token bucket algorithm, while the burst of garbage collection will cause the business performance to be temporarily affected, and the burst flow control algorithm is not suitable for garbage collection. Moreover, in order to improve the accuracy of flow control, the garbage collection flow control uses timestamp algorithm.

[0057] The global manager module is responsible for collecting the capacity, traffic and state information of each node from the storage client at a fixed time, and after a certain time, according to different cluster residual capacity ratios, adaptively calculates different garbage collection and business flow control value ratios, and according to the traffic and state information of each node, adjusts the quota to each node.

[0058] The embodiment of the application discloses a flow control method, which realizes the balance of business and garbage collection, and does not affect the business performance and causes the storage space to be exhausted.

[0059] Referring to Figure 2 , a flow chart of a flow control method according to an exemplary embodiment is shown, as Figure 2 shown, comprising:

[0060] S101: determining the residual capacity ratio of the cluster, and adjusting the garbage collection flow control value, the business flow control value and the flow control ratio of garbage collection and business of the cluster according to the residual capacity ratio; wherein the flow control ratio is negatively correlated with the residual capacity ratio;

[0061] The embodiment is applied to a flash-based log-structured storage cluster. In this step, first, the residual capacity ratio of the cluster is determined. As a feasible implementation manner, the determination of the residual capacity ratio of the cluster comprises: acquiring the capacity information of the nodes in the cluster every first preset time; wherein the capacity information comprises total capacity, garbage capacity and residual capacity; after accumulating for a second preset time, the total capacity of the cluster and the residual capacity of the cluster are calculated; wherein the second preset time is an integer multiple of the first preset time; the residual capacity ratio of the cluster is determined according to the total capacity of the cluster and the residual capacity of the cluster. In a specific implementation, the capacity information of the nodes in the cluster, including total capacity, garbage capacity and residual capacity, is acquired every first preset time, and the first preset time is not limited here, for example, 1 second, 5 seconds, 10 seconds, etc. The smaller the first preset time is, the higher the accuracy of flow control adjustment is, but the greater the pressure of network communication is, and the first preset time can be flexibly configured according to actual conditions. After accumulating for a second preset time, the total capacity of the cluster and the residual capacity of the cluster are calculated. The second preset time is also not limited here, for example, 20 seconds, 30 seconds, 60 seconds, etc. The second preset time needs to be greater than the first preset time and an integer multiple of the first preset time. The shorter the second preset time is, the higher the real-time performance of flow control adjustment is, but the greater the calculation pressure is.

[0062] Secondly, the flow control ratio of garbage collection and business is adaptively calculated according to the residual capacity ratio of the cluster, and the flow control ratio is negatively correlated with the residual capacity ratio. On the premise that the garbage collection speed can follow, part of the business is controlled, part of the resources is given to garbage collection for use, and at the same time, the speed of garbage collection is also controlled to avoid occupying too many resources and excessively affecting the business.

[0063] As a feasible implementation, the adjusting the garbage collection flow control value, the service flow control value and the garbage collection and service flow control ratio of the cluster according to the remaining capacity ratio comprises: if the remaining capacity ratio is greater than a first preset value and less than 1, adjusting the garbage collection flow control value of the cluster to 0, the service flow control value to infinity, and the garbage collection and service flow control ratio to 0; if the remaining capacity ratio is greater than a second preset value and less than or equal to the first preset value, adjusting the garbage collection and service flow control ratio to a first target value according to the level to which the remaining capacity ratio belongs; wherein the level to which the remaining capacity ratio belongs is negatively correlated with the remaining capacity ratio, and the first target value is negatively correlated with the level to which the remaining capacity ratio belongs, and the first target value corresponding to different levels are all greater than 0 and less than 1; if the remaining capacity ratio is greater than a third preset value and less than or equal to the second preset value, adjusting the garbage collection and service flow control ratio to a second target value; wherein the second target value is greater than 1; if the remaining capacity ratio is greater than 0 and less than or equal to a fourth preset value, adjusting the garbage collection flow control value of the cluster to infinity, the service flow control value to a fixed value, and the garbage collection and service flow control ratio to infinity.

[0064] In a specific implementation, the threshold values A1, A2, A3 and A4 correspond to the first target value, the second target value, the third target value and the fourth target value respectively, and the following relationship exists: 0 < A4 < A3 < A2 < A1 < 1. When A1 < the remaining capacity ratio < 1, it is considered that the cluster capacity is sufficient, and there is no risk of space depletion at all. A1 can be 0.3, 0.4, 0.5, etc. At this time, the business can be executed at full speed, and the garbage collection can be executed at a single depth under the condition of meeting the triggering condition to avoid causing a large impact on the business. The triggering condition is related to the garbage amount, which can be flexibly configured according to the actual situation, and is not specifically limited, for example, the garbage capacity is greater than 30% of the cluster vertical capacity to trigger garbage collection. The calculated garbage collection and business flow control ratio is 0, the garbage collection flow control value is 0, and the business flow control value is infinite. When A2 < the remaining capacity ratio ≤ A1, it is considered that the cluster space is not sufficient, but there is basically no risk of space depletion. A2 can be 0.2, 0.3, 0.4, etc. At this time, although the business flow control value can still be greater than the garbage collection flow control value, part of the resources will be given to garbage collection. The flow control ratio of garbage collection and business can be adjusted according to the level to which the remaining capacity ratio belongs, and the number of levels can be flexibly set according to the actual situation. For example, four levels are set, and it is assumed that A2 < B3 < B2 < B1 ≤ A1 is established. If B1 < the remaining capacity ratio ≤ A1, the flow control ratio of garbage collection and business is calculated as C1, that is, the garbage collection flow only needs to run to C1 times the business flow control. If B2 < the remaining capacity ratio ≤ B1, the flow control ratio of garbage collection and business is calculated as C2, that is, the garbage collection flow only needs to run to C2 times the business flow control. If B3 < the remaining capacity ratio ≤ B2, the flow control ratio of garbage collection and business is calculated as C3, that is, the garbage collection flow only needs to run to C3 times the business flow control. If A2 < the remaining capacity ratio ≤ B3, the flow control ratio of garbage collection and business is calculated as C4, that is, the garbage collection flow only needs to run to C4 times the business flow control. C1, C2, C3 and C4 satisfy 0 < C1 < C2 < C3 < C4 < 1. When A3 < the remaining capacity ratio ≤ A2, it is considered that the cluster space is insufficient, and there is a risk of space depletion. A3 can be 0.1, 0.2, 0.3, etc. At this time, the flow of garbage collection needs to be controlled to be greater than or equal to the flow of business to recover the cluster space, that is, the flow control ratio of garbage collection and business is calculated as C5, and C5 satisfies C5 >= 1. The greater the value of C5, the more resources the garbage collection occupies, and the faster the garbage collection, but the greater the impact on the business. In order to reduce the impact of garbage collection on the business, the value of C5 will be slightly greater than 1, such as 1.1, 1.2, 1.3, etc. When 0 < the remaining capacity ratio ≤ A4, it is considered that the cluster space is severely insufficient, and the business may be deadlocked at any time due to space depletion. A4 can be 0.05, 0.1, 0.15, etc.At this time, full-speed garbage collection is required, the business control is the minimum quota value, and the ratio of the garbage collection to the business flow control value is calculated as infinity, that is, the flow control value of the garbage collection is infinity, and the flow control value of the business is a fixed value Q, which can be 5MB / s, 10MB / s, 20MB / s, etc.

[0065] S102: allocate the business flow control value to the nodes in the cluster based on the business pressure values of the nodes; wherein the nodes include storage nodes and computing nodes, and the business quota allocated to the nodes is positively correlated with the business pressure of the nodes;

[0066] S103: allocate the garbage collection flow control value to the storage nodes in the cluster based on the business pressure values of the storage nodes; wherein the garbage collection quota allocated to the storage nodes is negatively correlated with the business pressure of the storage nodes.

[0067] In a specific implementation, the business flow control value is allocated to the nodes in the cluster, including the storage nodes and the computing nodes, and the garbage collection flow control value is allocated to the storage nodes. For the storage nodes, the smaller the business pressure, the larger the allocated garbage collection flow control value quota, and for all the nodes, the larger the business pressure, the larger the allocated business flow control value.

[0068] It should be noted that in the allocation process of the business flow control value and the garbage collection flow control value, the traffic and the state of the nodes can also be referred to. For a node whose resources have run out, the flow control value still needs to be allocated to the node when the calculated ratio of the garbage collection to the business flow control is not met, but after the calculated ratio of the garbage collection to the business flow control is met, the flow control value quota is no longer occupied, and the flow control value is allocated to other nodes whose resources have not run out according to the business pressure.

[0069] The traffic control method provided by the embodiments of the present application first adaptively calculates the ratio of the garbage collection to the business flow control according to the remaining capacity ratio of the cluster, and the ratio of the flow control is negatively correlated with the remaining capacity ratio. When the remaining capacity ratio is large, there is no risk of space exhaustion, and the business can not be controlled, and the garbage collection can be executed in depth on the basis of meeting the triggering condition to minimize the impact on the business. When the remaining capacity of the cluster is small, the ratio of the garbage collection to the business flow control value gradually increases, ensuring that the space will not be exhausted while minimizing the impact on the business. Secondly, the business flow control value is distributed to each node according to the business pressure of each node, and the garbage collection flow control value is allocated to the storage nodes, to ensure the load balancing of the business of each node.

[0070] The embodiments introduce the flow control strategy of the nodes, specifically:

[0071] Referring to Figure 3, according to an exemplary embodiment, a flow chart of another traffic control method is shown, as shown in Figure 3 , comprising:

[0072] S201: using a token bucket algorithm to control the traffic of the node service based on the service quota allocated to the node;

[0073] In this embodiment, the token bucket flow control algorithm is adapted for services with burst requirements. As a feasible implementation, this step includes: supplementing the number of tokens in the token bucket based on the service quota allocated to the node; determining whether the contribution value of the current service task is less than or equal to the current token number in the token bucket; if yes, issuing the current service task and updating the current token number; if no, determining the scheduling time of the current service task according to the contribution value of the current service task, the current token number and the service quota allocated to the node.

[0074] In a specific implementation, the token bucket flow control algorithm process of service IO is as shown in Figure 4 . Calculate the contribution value of the current task. If the flow control target is service speed, the contribution value of each task is service IO (Input Output) size. If the flow control target is service IOPS (Input / Output Operations Per Second), the contribution value of each task is 1. Add the current service task to the flow control task queue and update the token number in the token bucket. The token number in the token bucket is supplemented by tokens according to the allocated service quota, but cannot exceed the maximum capacity of the bucket. Check whether the number of tasks in the flow control task queue is 0. If yes, end this round of scheduling; if no, take out the current service task from the head of the task queue, and determine whether the task is issued according to the contribution value of the current service task and the current token number in the bucket, that is, whether the contribution value of the current service task is less than or equal to the current token number. If the contribution value of the current service task is less than or equal to the current token number, update the current token number, that is, subtract the token number consumed by the issued task, issue the task, remove the task from the task queue, and the number of currently issued and executed tasks + 1. After the task is completed, the number of currently issued and executed tasks - 1, and the step of checking whether the number of tasks in the flow control task queue is 0 is re-entered, so as to issue all tasks in the task queue that meet the conditions. If the contribution value of the current service task is greater than the current token number, set a timer task scheduling according to the time required to supplement the tokens of the contribution value of the current service task based on the current token number and according to the allocated service quota.

[0075] S202: using a timestamp algorithm to control the traffic of the storage node garbage collection based on the garbage collection quota allocated to the storage node.

[0076] In the embodiment, the timestamp flow control algorithm is adapted for garbage collection without burst demand. As a feasible implementation, the step comprises: calculating a timestamp mark of a current garbage collection task according to a contribution value of a previous garbage collection task, the timestamp mark of the previous garbage collection task and a garbage collection quota allocated by the storage node; determining whether the timestamp mark of the current garbage collection task is less than or equal to a current scheduling time; if yes, issuing the current garbage collection task; if no, issuing the current garbage collection task when the current scheduling time reaches the timestamp mark of the current garbage collection task.

[0077] In a specific implementation, the timestamp flow control algorithm process of garbage collection is as shown in FIG. 4. Figure 5 When the current garbage collection task arrives, first, a timestamp mark of the current garbage collection task is calculated according to a garbage collection quota allocated by the storage node, and the current garbage collection task is added to the flow control task queue. As a feasible implementation, the calculating of the timestamp mark of the current garbage collection task according to the contribution value of the previous garbage collection task, the timestamp mark of the previous garbage collection task and the garbage collection quota allocated by the storage node comprises: calculating an execution time of the previous garbage collection task according to the contribution value of the previous garbage collection task and the garbage collection quota allocated by the storage node; and setting a sum of the timestamp mark of the previous garbage collection task and the execution time as the timestamp mark of the current garbage collection task. Wherein, the timestamp mark calculation formula of the current garbage collection task can be expressed as:

[0078]

[0079] Wherein, represents the timestamp mark of the current task, represents the timestamp mark of the previous task, w i-1 represents the contribution value of the previous task, fc_value represents the garbage collection quota allocated by the storage node, and current_time represents the current scheduling time. If the contribution value of the previous task is large, the calculated time can be larger than current_time, and the task needs to be suspended until the calculated timestamp is issued. Early task issuance can cause the traffic to exceed the flow control value; if the contribution value of the previous task is small, the calculated time can be smaller than current_time, and the task is immediately issued, and the traffic will not exceed the set flow control value.

[0080] Secondly, it is checked whether the number of tasks in the flow control task queue is 0, if yes, the current round of scheduling is ended, if no, a current garbage collection task is taken out from the head of the task queue, and it is determined whether the task needs to be issued according to the size relationship between the time stamp flag and the current scheduling time, that is, whether the time stamp flag is less than or equal to the current scheduling time. If the time stamp flag is less than or equal to the current scheduling time, the current garbage collection task is issued, the current garbage collection task taken out is removed from the task queue, and the number of tasks currently issued and executed is incremented by 1. After the task is completed, the number of tasks currently issued and executed is decremented by 1, and the step of checking whether the number of tasks in the flow control task queue is 0 is re-entered, so as to issue all the tasks in the task queue which meet the conditions. If the time stamp flag is greater than the current scheduling time, the time difference between the time stamp flag and the current scheduling time is calculated, and a timer task is scheduled.

[0081] It can be seen that the application adaptively applies different flow control technologies according to the properties of the business and garbage collection. The burst ability is considered to ensure the burst performance of the business, and the token bucket algorithm is used for business flow control. The burst of garbage collection will cause the business performance to be temporarily affected, and the burst flow control algorithm is not suitable for garbage collection. In order to improve the accuracy of flow control, the time stamp algorithm is used for garbage collection flow control.

[0082] Next, a flow control device provided by an embodiment of the application is introduced, and the flow control device described below can be referred to with the flow control method described above.

[0083] Referring to Figure 6 , a structural diagram of a flow control device according to an exemplary embodiment is shown, as Figure 6 shown, comprising:

[0084] The determining module 601 is configured to determine a remaining capacity ratio of the cluster, and adjust a garbage collection flow control value, a business flow control value and a garbage collection and business flow control ratio of the cluster according to the remaining capacity ratio, wherein the flow control ratio is negatively correlated with the remaining capacity ratio.

[0085] The first distribution module 602 is configured to distribute the business flow control value to the nodes in the cluster based on a business pressure value of the nodes, wherein the nodes include storage nodes and computing nodes, and the business quota obtained by the node distribution is positively correlated with the business pressure of the node.

[0086] The second distribution module 603 is configured to distribute the garbage collection flow control value to the storage nodes in the cluster based on a business pressure value of the storage nodes, wherein the garbage collection quota obtained by the storage node distribution is negatively correlated with the business pressure of the storage node.

[0087] The traffic control device provided by the embodiments of the present application firstly adaptively calculates the flow control ratio of garbage collection and service according to the residual capacity ratio of the cluster, and the flow control ratio is negatively correlated with the residual capacity ratio. When the residual capacity ratio is relatively large, there is no risk of space exhaustion, and the service can not be controlled, and the garbage collection is executed in deepness on the basis of meeting the triggering condition, so as to reduce the influence on the service as much as possible. When the residual capacity of the cluster is relatively small, the flow control value ratio of garbage collection and service is gradually increased, so as to ensure that the space is not exhausted and reduce the influence on the service as much as possible. Secondly, the service flow control value is distributed to each node and the garbage collection flow control value is distributed to the storage node according to the service pressure of each node, so as to ensure the load balancing of the service of each node.

[0088] On the basis of the above-mentioned embodiments, as a preferred embodiment, the determination module 601 comprises:

[0089] The acquisition unit is configured to acquire the capacity information of the nodes in the cluster every first preset time, wherein the capacity information comprises total capacity, garbage capacity and residual capacity.

[0090] The first calculation unit is configured to calculate the total capacity of the cluster and the residual capacity of the cluster after accumulating for a second preset time, wherein the second preset time is an integer multiple of the first preset time.

[0091] The first determination unit is configured to determine the residual capacity ratio of the cluster according to the total capacity of the cluster and the residual capacity of the cluster.

[0092] On the basis of the above-mentioned embodiments, as a preferred embodiment, the determination module 601 comprises:

[0093] The first adjustment unit is configured to, if the residual capacity ratio is greater than a first preset value and less than 1, adjust the garbage collection flow control value of the cluster to 0, the service flow control value to infinity and the flow control ratio of garbage collection and service to 0.

[0094] The second adjustment unit is configured to, if the residual capacity ratio is greater than a second preset value and less than or equal to the first preset value, adjust the flow control ratio of garbage collection and service to a first target value according to the level to which the residual capacity ratio belongs, wherein the level to which the residual capacity ratio belongs is negatively correlated with the residual capacity ratio, the first target value is negatively correlated with the level to which the residual capacity ratio belongs, and the first target value corresponding to different levels is greater than 0 and less than 1.

[0095] The third adjustment unit is configured to, if the residual capacity ratio is greater than a third preset value and less than or equal to the second preset value, adjust the flow control ratio of garbage collection and service to a second target value, wherein the second target value is greater than 1.

[0096] A fourth adjusting unit is configured to, if the remaining capacity ratio is greater than 0 and less than or equal to a fourth preset value, adjust a garbage collection flow control value of the cluster to infinity, a service flow control value to a fixed value, and a garbage collection and service flow control ratio to infinity.

[0097] On the basis of the above-mentioned embodiments, as a preferred embodiment, the method further comprises:

[0098] A first flow control module is configured to perform flow control of the node service based on the service quota allocated to the node by using a token bucket algorithm.

[0099] A second flow control module is configured to perform flow control of garbage collection of the storage node based on the garbage collection quota allocated to the storage node by using a timestamp algorithm.

[0100] On the basis of the above-mentioned embodiments, as a preferred embodiment, the first flow control module comprises:

[0101] A supplement unit is configured to supplement the number of tokens in the token bucket based on the service quota allocated to the node.

[0102] A first judging unit is configured to judge whether a contribution value of a current service task is less than or equal to a current number of tokens in the token bucket; if yes, a work flow of a first issuing unit is started; if no, a work flow of a second determining unit is started.

[0103] The first issuing unit is configured to issue the current service task and update the current number of tokens.

[0104] The second determining unit is configured to determine a scheduling time of the current service task according to the contribution value of the current service task, the current number of tokens, and the service quota allocated to the node.

[0105] On the basis of the above-mentioned embodiments, as a preferred embodiment, the second flow control module comprises:

[0106] A second calculating unit is configured to calculate a timestamp mark of a current garbage collection task according to a contribution value of a previous garbage collection task, the timestamp mark, and the garbage collection quota allocated to the storage node.

[0107] A second judging unit is configured to judge whether the timestamp mark of the current garbage collection task is less than or equal to a current scheduling time; if yes, a work flow of a first issuing unit is started; if no, a work flow of a third issuing unit is started.

[0108] The second issuing unit is configured to issue the current garbage collection task.

[0109] The third issuing unit is configured to issue the current garbage collection task when the current scheduling time reaches the timestamp mark of the current garbage collection task.

[0110] Based on the above-mentioned embodiments, as a preferred implementation, the second calculating unit is specifically configured to calculate the execution time of the previous garbage collection task according to the contribution value of the previous garbage collection task and the garbage collection quota allocated by the storage node, and set the sum of the timestamp mark of the previous garbage collection task and the execution time as the timestamp mark of the current garbage collection task.

[0111] As to the apparatus in the above-mentioned embodiments, the specific manners in which various modules perform operations have been described in detail in the embodiments of the method, and thus will not be described in detail here.

[0112] Based on the hardware implementation of the above-mentioned program modules, and in order to implement the method of the embodiments of the present application, the embodiments of the present application further provide an electronic device, Figure 7 As shown in FIG. 1, the electronic device according to an example embodiment includes: Figure 7 As shown in FIG. 1, the electronic device according to an example embodiment includes:

[0113] The communication interface 1 is capable of information interaction with other devices such as network devices and the like.

[0114] The processor 2 is connected with the communication interface 1 to realize information interaction with other devices, and is used to execute the flow control method provided by one or more technical solutions when running a computer program. The computer program is stored on the memory 3.

[0115] Of course, in actual application, various components in the electronic device are coupled together through a bus system 4. It can be understood that the bus system 4 is used to realize the connection and communication between the components. The bus system 4 includes not only a data bus, but also a power bus, a control bus and a status signal bus. However, in order to clearly illustrate, all kinds of buses are marked as the bus system 4 in the Figure 7 .

[0116] The memory 3 in the embodiments of the present application is used to store various types of data to support the operation of the electronic device. Examples of these data include: any computer program used to operate on the electronic device.

[0117] It can be understood that the memory 3 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a ferromagnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM). The magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a random access memory (RAM) used as an external cache. By way of example but not limitation, many forms of RAM can be used, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), sync link dynamic random access memory (SLDRAM), and direct rambus random access memory (DRRAM).The memory 2 described in the embodiments of the present application is intended to include, but not limited to, these and any other suitable types of memories.

[0118] The method disclosed in the embodiments of the present application can be applied to or implemented by the processor 2. The processor 2 can be an integrated circuit chip having a processing capability of signals. In the implementation, each step of the method can be completed by integrated logic circuits of hardware in the processor 2 or instructions in the form of software. The processor 2 described above can be a general processor, a DSP, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 2 can implement or execute the disclosed methods, steps and logic block diagrams in the embodiments of the present application. The general processor can be a microprocessor or any conventional processor, etc. In combination with the steps of the method disclosed in the embodiments of the present application, the execution can be directly completed by a hardware decoding processor or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in the memory 3. The processor 2 reads the program in the memory 3 and combines the hardware to complete the steps of the foregoing method.

[0119] The processor 2 implements the corresponding flow in each method of the embodiments of the present application when executing the program. For brevity, the foregoing will not be repeated here.

[0120] In the exemplary embodiments, the embodiments of the present application also provide a storage medium, i.e., a computer storage medium, specifically a computer readable storage medium, such as the memory 3 storing a computer program, which can be executed by the processor 2 to complete the steps of the foregoing method. The computer readable storage medium can be an FRAM, a ROM, a PROM, an EPROM, an EEPROM, a Flash Memory, a magnetic surface memory, an optical disc, or a CD-ROM, etc.

[0121] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction related hardware, and the foregoing program can be stored in a computer readable storage medium. When the program is executed, the steps of the method embodiments are executed. The storage medium includes a mobile storage device, a ROM, a RAM, a magnetic disc or an optical disc, etc. various program code storage media.

[0122] Alternatively, the above-mentioned integrated units of the present application, if realized in the form of software function modules and sold or used as independent products, can also be stored in a computer-readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions for causing an electronic device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in the embodiments of the present application. The aforementioned storage medium includes: mobile storage devices, ROM, RAM, magnetic disks or optical disks, and various media that can store program codes.

[0123] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A flow control method, characterized by, include: Determine the remaining capacity ratio of the cluster, and adjust the garbage collection flow control value, service flow control value, and garbage collection and service flow control ratio of the cluster according to the remaining capacity ratio; wherein the flow control ratio is negatively correlated with the remaining capacity ratio; The service flow control value is allocated to the nodes based on the service pressure value of the nodes in the cluster; wherein the nodes include storage nodes and computing nodes, and the service quota allocated to the nodes is positively correlated with the service pressure of the nodes; The garbage collection flow control value is allocated to the storage nodes based on their service pressure values ​​in the cluster; wherein, the garbage collection quota allocated to the storage nodes is negatively correlated with the service pressure of the storage nodes; The determination of the remaining capacity ratio of the cluster includes: Capacity information of nodes in the cluster is obtained every first preset time interval; wherein, the capacity information includes total capacity, garbage capacity and remaining capacity; After accumulating the second preset time, calculate the total cluster capacity and the remaining cluster capacity; wherein, the second preset time is an integer multiple of the first preset time; The remaining capacity ratio of the cluster is determined based on the total cluster capacity and the remaining cluster capacity. The step of adjusting the cluster's garbage collection flow control value, service flow control value, and garbage collection to service flow control ratio according to the remaining capacity ratio includes: If the remaining capacity ratio is greater than the first preset value and less than 1, then the cluster's garbage collection flow control value is adjusted to 0, the service flow control value is adjusted to infinity, and the ratio of garbage collection to service flow control is adjusted to 0. If the remaining capacity ratio is greater than the second preset value and less than or equal to the first preset value, then the flow control ratio of waste collection and business is adjusted to the first target value according to the level to which the remaining capacity ratio belongs; wherein, the level to which the remaining capacity ratio belongs is negatively correlated with the remaining capacity ratio, the first target value is negatively correlated with the level to which the remaining capacity ratio belongs, and the first target value corresponding to different levels is greater than 0 and less than 1; If the remaining capacity ratio is greater than the third preset value and less than or equal to the second preset value, then the ratio of waste collection to business flow control will be adjusted to the second target value; wherein the second target value is greater than 1. If the remaining capacity ratio is greater than 0 and less than or equal to the fourth preset value, then the cluster's garbage collection flow control value will be adjusted to infinity, the service flow control value will be adjusted to a fixed value, and the ratio of garbage collection to service flow control will be adjusted to infinity.

2. The flow control method of claim 1, wherein, Also includes: The token bucket algorithm is used to control the traffic of the node's services based on the service quota allocated to the node. Based on the garbage collection quota allocated to the storage nodes, a timestamp algorithm is used to control the flow of garbage collection on the storage nodes.

3. The flow control method of claim 2, wherein, The service quota allocated based on the node uses a token bucket algorithm for traffic control of the node's services, including: The number of tokens in the token bucket is supplemented based on the business quota allocated by the node; Determine whether the contribution value of the current business task is less than or equal to the current number of tokens in the token bucket; If yes, the current service task is issued, and the current token quantity is updated. If no, a scheduling time of the current service task is determined according to a contribution value of the current service task, the current token quantity and a service quota obtained by the node allocation.

4. The flow control method of claim 2, wherein, The garbage collection of the storage node is controlled by using a timestamp algorithm based on the garbage collection quota obtained by the storage node allocation, and the method comprises the steps of: calculating a timestamp mark of a current garbage collection task according to a contribution value of a previous garbage collection task, the timestamp mark and a garbage collection quota obtained by the storage node allocation; determining whether the timestamp mark of the current garbage collection task is less than or equal to a current scheduling time; if yes, issuing the current garbage collection task; if no, issuing the current garbage collection task when the current scheduling time reaches the timestamp mark of the current garbage collection task.

5. The flow control method of claim 4, wherein, The method for calculating the timestamp mark of the current garbage collection task according to the contribution value of the previous garbage collection task, the timestamp mark and the garbage collection quota obtained by the storage node allocation comprises the steps of: calculating an execution time of the previous garbage collection task according to the contribution value of the previous garbage collection task and the garbage collection quota obtained by the storage node allocation; setting a sum of the timestamp mark of the previous garbage collection task and the execution time as the timestamp mark of the current garbage collection task.

6. A flow control device, characterized by The method comprises the steps of: determining a residual capacity proportion of a cluster, and adjusting a garbage collection flow control value, a service flow control value and a flow control proportion of garbage collection and service of the cluster according to the residual capacity proportion; wherein the flow control proportion is negatively correlated with the residual capacity proportion; a first allocation module is configured to allocate the service flow control value to nodes in the cluster based on service pressure values of the nodes; wherein the nodes comprise storage nodes and computing nodes, and a service quota obtained by the node allocation is positively correlated with the service pressure of the nodes; a second allocation module is configured to allocate the garbage collection flow control value to the storage nodes in the cluster based on service pressure values of the storage nodes; wherein a garbage collection quota obtained by the storage node allocation is negatively correlated with the service pressure of the storage nodes. The determination module is specifically configured to: acquire capacity information of nodes in the cluster every first preset time; the capacity information includes total capacity, garbage capacity, and remaining capacity; calculate total capacity of the cluster and remaining capacity of the cluster after a second preset time; the second preset time is an integer multiple of the first preset time; determine a remaining capacity ratio of the cluster according to the total capacity of the cluster and the remaining capacity of the cluster; if the remaining capacity ratio is greater than a first preset value and less than 1, adjust a garbage recovery flow control value of the cluster to 0, a business flow control value to infinity, and a garbage recovery and business flow control ratio to 0; if the remaining capacity ratio is greater than a second preset value and less than or equal to the first preset value, adjust the garbage recovery and business flow control ratio to a first target value according to a level to which the remaining capacity ratio belongs; the level to which the remaining capacity ratio belongs is negatively correlated with the remaining capacity ratio, the first target value is negatively correlated with the level to which the remaining capacity ratio belongs, and the first target value corresponding to different levels is all greater than 0 and less than 1; if the remaining capacity ratio is greater than a third preset value and less than or equal to the second preset value, adjust the garbage recovery and business flow control ratio to a second target value; the second target value is greater than 1; if the remaining capacity ratio is greater than 0 and less than or equal to a fourth preset value, adjust the garbage recovery flow control value of the cluster to infinity, the business flow control value to a fixed value, and the garbage recovery and business flow control ratio to infinity.

7. An electronic device, comprising: The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the flow control method in any one of claims 1 to 5. The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the flow control method in any one of claims 1 to 5. ​ 8. A computer-readable storage medium, characterized in that, ​

Images