Network cluster and tenant-aware hierarchical processing system for distributed quality of service

Abstract

The present invention relates to quality of service (QoS) of services. The present invention provides a QoS handling system for one or more services, the QoS handling system comprising a plurality of QoS handling nodes arranged in a hierarchical tree structure having at least two levels of hierarchy. The highest level of hierarchy comprises one or more root nodes, each root node being associated with a set of service instances of the services. The lowest level of hierarchy comprises a plurality of leaf nodes, each leaf node being associated with one service instance, each leaf node being derived from at least one root node, and the service instances of the set of each root node being associated with the leaf nodes derived from the root node. Each leaf node can apply a local QoS policy to the associated service instance of each leaf node, and each root node can apply a first common QoS policy to the set of service instances of each root node.

Description

TECHNICAL FIELD

[0001] The present invention relates to quality of service (QoS) policies for services. The invention provides a QoS processing system for one or more services. The QoS processing system is based on a hierarchical tree structure of QoS processing nodes. The QoS processing system can be used to provide distributed QoS on service instances of one or more services in a network or for different tenants. BACKGROUND

[0002] With the use of microservices, applications can be scaled horizontally by simply adding more server instances to the production fleet. However, horizontal scaling poses several challenges to the service infrastructure, such as how to apply QoS policies (e.g., global policies) to these instances without impacting the overall service performance.

[0003] Currently, QoS schemes typically rely on a distributed store to make collective QoS decisions based on the aggregated information of all active nodes. This approach works well in large-scale multi-region applications, but it is not suitable for multi-tenant service deployments where QoS policies need to be applied at both tenant and global levels. The noisy neighbor problem is acute when all these policies share a distributed store for decision making. In large service deployments, multiple distributed store systems are used to mitigate this problem. Therefore, this model has its own cost and management overhead.

[0004] Most traditional rate limiter (QoS parameter) schemes are designed specifically for rate limiting queries to a particular service or service instance. One traditional approach is to deploy multiple instances of the rate limiter scheme (e.g., based on a distributed architecture) to distribute the rate limiter load. However, managing such deployments in large-scale shared cloud systems is quite cumbersome. Therefore, the main drawbacks of traditional rate limiter schemes are: (1) dependency on third-party distributed store schemes; (2) scalability; (3) suitability for multi-tenant QoS use cases, such as rate limiting for OBS; (4) maintenance, as most schemes require maintaining a series of different components for QoS processing. SUMMARY

[0005] It is an object of the present invention to design a QoS processing system that overcomes the above-mentioned drawbacks. For example, there is a need for a QoS processing system that can handle ultra-large multi-tenant workloads without using multiple QoS service deployments.

[0006] These and other objects are achieved by the present invention, as specified in the independent claims. Advantageous implementations are further described in the dependent claims.

[0007] A first aspect of the present invention provides a QoS processing system for one or more services, the QoS processing system comprising: a plurality of QoS processing nodes arranged in a hierarchical tree structure comprising at least two hierarchy levels; a highest hierarchy level of the at least two hierarchy levels comprising one or more root nodes of the tree structure of the QoS processing nodes, each root node being associated with a set of service instances of the one or more services; a lowest hierarchy level of the at least two hierarchy levels comprising a plurality of leaf nodes of the tree structure of the QoS processing nodes, each leaf node being associated with one service instance of the one or more services, each leaf node being derived from at least one root node, and the service instances of the set of service instances of each root node being associated with the leaf nodes derived from the root node; each leaf node being configured to apply a local QoS policy to the associated service instance of each leaf node; each root node being configured to apply a first common QoS policy to the set of service instances of each root node.

[0008] Depending on how the hierarchical tree structure is specifically designed to handle the processing nodes and the connections between the processing nodes, each single root node can be associated with multiple leaf nodes, and thus with multiple service instances (as each leaf node is associated with one service instance).

[0009] Using the hierarchical tree structure of the QoS processing system of the first aspect to provide local and / or common QoS policies to service instances of the one or more services provides several advantages. The QoS processing system of the present invention does not rely on third-party distributed storage solutions. Furthermore, due to the hierarchical approach, the QoS processing system has good scalability. It is possible to add and remove hierarchy levels as needed. The QoS processing system is also suitable for multi-tenant QoS use cases, such as rate limiting for object storage (OBS). The QoS processing system is easy to maintain, as it does not require maintaining a collection of different components for QoS processing. The QoS processing system can handle hyper-scale multi-tenant workloads without using multiple QoS service deployments.

[0010] In an implementation form of the first aspect, the hierarchical tree structure comprises at least three hierarchy levels, and the QoS processing system further comprises: at least one intermediate hierarchy level arranged between the highest hierarchy level and the lowest hierarchy level and comprising a plurality of intermediate nodes of the QoS processing nodes, each intermediate node being derived from at least one root node, each intermediate node being associated with a subset of service instances of the one or more services, and the service instances of the subset of service instances of each intermediate node being associated with leaf nodes derived from the intermediate node; each intermediate node being configured to apply a second common QoS policy to the subset of service instances of each intermediate node.

[0011] The hierarchical tree structure can also comprise 4 or 5 hierarchy levels, each hierarchy level can comprise a plurality of intermediate nodes. The number of hierarchy levels is not limited, such that the QoS processing system has good scalability. Different intermediate hierarchy levels can be associated with hosts, clusters, AZs, etc. Depending on the number of hierarchy levels, more than two common QoS policies can be applied. At each higher hierarchy level, a common QoS policy for a larger set of service instances can be applied.

[0012] In an implementation form of the first aspect, each particular QoS processing node of the QoS processing nodes is configured to: collect at least one QoS metric of at least one service instance from each child node of the QoS processing nodes, each child node being directly derived from the particular QoS processing node; aggregate the collected QoS metrics; provide the aggregated QoS metrics to each parent node of the QoS processing nodes, the particular QoS processing node being directly derived from each parent node.

[0013] In this way, relevant QoS metrics can be distributed to all QoS processing nodes of all hierarchy levels. The root nodes can receive all QoS metrics and can make QoS policy decisions based on these QoS metrics.

[0014] In an implementation form of the first aspect, each particular QoS processing node of the QoS processing nodes is configured to: receive a QoS configuration of at least one service instance from each parent node of the QoS processing nodes, the particular QoS processing node being directly derived from each parent node; and / or provide the received QoS configuration and / or a QoS configuration of at least one service instance of its own to each child node of the QoS processing nodes, each child node being directly derived from the particular QoS processing node.

[0015] Thus, each QoS processing node on each hierarchy level can receive a QoS configuration and can apply a local or common QoS policy, e.g., depending on the QoS configuration.

[0016] In an implementation form of the first aspect, each root node is configured to distribute at least one rate limiting parameter of at least one service instance to each QoS processing node derived from the root node.

[0017] Thus, a rate limiting scheme can be implemented by the QoS processing system.

[0018] In an implementation form of the first aspect, each QoS policy comprising each local QoS policy, each first common QoS policy and / or each second common QoS policy is applied based on at least one of: one or more of the QoS indicators of the one or more service instances to which the QoS policy is applied; one or more of the QoS configurations of the one or more service instances to which the QoS policy is applied; one or more of the rate limit parameters of the one or more service instances to which the QoS policy is applied.

[0019] The QoS indicators and QoS configurations can be propagated and distributed through the hierarchy levels as described above, and the QoS policies can be applied accordingly in a hierarchical manner.

[0020] In an implementation form of the first aspect, the local QoS policy, the first common QoS policy and / or the second common QoS policy comprises a rate limit policy.

[0021] In an implementation form of the first aspect, the QoS processing system further comprises a control node configured to allocate and / or reallocate one or more of the plurality of QoS processing nodes in the hierarchical tree structure.

[0022] Thus, the control node can select the optimal QoS processing node based on using some suitable algorithm.

[0023] In an implementation form of the first aspect, the control node is one of the QoS processing nodes of the hierarchical tree structure.

[0024] In an implementation form of the first aspect, the control node is configured to perform the allocation and / or the reallocation based on at least one of: a load on currently allocated QoS processing nodes, QoS policies applied by the currently allocated QoS processing nodes, a communication cost between the currently allocated QoS processing nodes, a topology of the service instances and QoS configurations of each of the service instances.

[0025] In an implementation form of the first aspect, each QoS processing node in the hierarchical tree structure is aware of its position in the tree structure and / or comprises information describing the entire hierarchical tree structure.

[0026] In an implementation form of the first aspect, the QoS processing nodes in the hierarchical tree structure are configured to perform a consensus protocol to determine which QoS processing node is responsible for which QoS policy and / or to share one or more QoS policies using the hierarchical tree structure.

[0027] In an implementation form of the first aspect, the control node is further configured to report the allocation and / or reallocation of the one or more QoS processing nodes to all of the QoS processing nodes.

[0028] The QoS processing nodes can further apply their local and / or common QoS policies based on their position in the tree structure, and / or based on the result of the consensus protocol, and / or the report of the allocation and / or reallocation.

[0029] In an implementation form of the first aspect, the hierarchical tree structure comprises four hierarchy levels, wherein: each QoS processing node of a lowest hierarchy level is associated with one service instance of one or more services of one host, each host executing one or more service instances of the one or more services; each QoS processing node of a first intermediate hierarchy level above the lowest hierarchy level is associated with all the service instances of one host; each QoS processing node of a second intermediate hierarchy level above the first intermediate hierarchy level is associated with all the service instances of all hosts in one host cluster; each QoS processing node of a highest hierarchy level above the second intermediate hierarchy level is associated with all the service instances of all clusters in one cluster region.

[0030] A second aspect of the present application provides a method for QoS processing of one or more services, the method comprising: allocating or reallocating a plurality of QoS processing nodes in a hierarchical tree structure comprising at least two hierarchy levels; a highest hierarchy level of the at least two hierarchy levels comprising one or more root nodes of the tree structure of the QoS processing nodes, each root node being associated with a set of service instances of the one or more services; a lowest hierarchy level of the at least two hierarchy levels comprising a plurality of leaf nodes of the tree structure of the QoS processing nodes, each leaf node being associated with one service instance of the one or more services, each leaf node being derived from at least one root node, and a service instance of the set of service instances of each root node being associated with a leaf node derived from the root node; each leaf node applying a local QoS policy to the service instance associated with the leaf node; each root node applying a first common QoS policy to the set of service instances associated with the root node.

[0031] In an implementation form of the second aspect, the hierarchical tree structure comprises at least three hierarchy levels, and the method further comprises: allocating or re-allocating a plurality of intermediate nodes of the QoS processing nodes in at least one intermediate hierarchy level arranged between the highest hierarchy level and the lowest hierarchy level, each intermediate node being derived from at least one root node, each intermediate node being associated with a subset of service instances of the one or more services, and the service instances in the subset of service instances of each intermediate node being associated with leaf nodes derived from the intermediate node; each intermediate node being configured to apply a second common QoS policy to the subset of service instances of each intermediate node.

[0032] In an implementation form of the second aspect, the method comprises: with each particular QoS processing node of the QoS processing nodes, collecting at least one QoS indicator of at least one service instance from each child node of the QoS processing node, each child node being directly derived from the particular QoS processing node; aggregating the collected QoS indicators; providing the aggregated QoS indicators to each parent node of the QoS processing nodes, the particular QoS processing node being directly derived from each parent node.

[0033] In an implementation form of the second aspect, the method comprises: with each particular QoS processing node of the QoS processing nodes, receiving a QoS configuration of at least one service instance from each parent node of the QoS processing nodes, the particular QoS processing node being directly derived from each parent node; and / or providing the received QoS configuration and / or a QoS configuration of at least one service instance of the particular QoS processing node to each child node of the QoS processing node, each child node being directly derived from the particular QoS processing node.

[0034] In an implementation form of the second aspect, each root node distributes at least one rate limit parameter of at least one service instance to each QoS processing node derived from the root node.

[0035] In an implementation form of the second aspect, each QoS policy comprising each local QoS policy, each first common QoS policy and / or each second common QoS policy is applied based on at least one of: one or more QoS indicators of the one or more service instances to which the QoS policy is applied; one or more QoS configurations of the one or more service instances to which the QoS policy is applied; one or more rate limit parameters of the one or more service instances to which the QoS policy is applied.

[0036] In an implementation form of the second aspect, the local QoS policy, the first common QoS policy and / or the second common QoS policy comprises a rate limiting policy.

[0037] In an implementation form of the second aspect, the method further comprises allocating and / or reallocating one or more of the plurality of QoS processing nodes in the hierarchical tree structure by a control node.

[0038] In an implementation form of the second aspect, the control node is one of the QoS processing nodes of the hierarchical tree structure.

[0039] In an implementation form of the second aspect, the control node performs the allocating and / or the reallocating based on at least one of: a load on currently allocated QoS processing nodes, QoS policies applied by the currently allocated QoS processing nodes, a communication cost between the currently allocated QoS processing nodes, a topology of the service instances and a QoS configuration of each of the service instances.

[0040] In an implementation form of the second aspect, each QoS processing node in the hierarchical tree structure knows its position in the tree structure and / or comprises information describing the whole hierarchical tree structure.

[0041] In an implementation form of the second aspect, the QoS processing nodes in the hierarchical tree structure perform a consensus protocol to determine which QoS processing node is responsible for which QoS policy and / or share one or more QoS policies using the hierarchical tree structure.

[0042] In an implementation form of the second aspect, the control node reports the allocation and / or the reallocation of the one or more QoS processing nodes to all the QoS processing nodes.

[0043] In an implementation form of the second aspect, the hierarchical tree structure comprises four hierarchy levels, wherein: each QoS processing node of the lowest hierarchy level is associated with one service instance of one or more services of one host, each host executing one or more service instances of the one or more services; each QoS processing node of a first intermediate hierarchy level above the lowest hierarchy level is associated with all the service instances of one host; each QoS processing node of a second intermediate hierarchy level above the first intermediate hierarchy level is associated with all the service instances of all hosts in one host cluster; each QoS processing node of the highest hierarchy level above the second intermediate hierarchy level is associated with all the service instances of all clusters in one cluster region.

[0044] The method according to the second aspect and its implementation forms achieves the same advantages as the QoS processing system according to the first aspect and its corresponding implementation forms.

[0045] A third aspect of the present application provides a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method according to the second aspect or any of its implementation forms.

[0046] A fourth aspect of the present application provides a non-transitory storage medium storing executable program code which, when executed by a processor, performs the method according to the second aspect or any of its implementation forms.

[0047] In summary, the present application proposes a hierarchical QoS processing system using QoS processing nodes in a hierarchical tree architecture, and using aggregation on different hierarchy levels for QoS policy decision. Furthermore, the present application proposes an algorithm to allocate or re-allocate QoS processing nodes to handle QoS policies of different service instances. Various factors (e.g. load on QoS processing nodes, QoS policies currently handled by QoS processing nodes, communication cost with a given QoS processing node) are considered to determine which QoS processing nodes should be responsible for handling specific QoS policies of services belonging to different tenants. The present application also proposes an aggregation pattern in each hierarchy level of the tree structure to better utilize resources.

[0048] It should be noted that all devices, elements, units and means described in the present application can be implemented in software or hardware elements or any kind of combination thereof. All steps which are performed by the various entities described in the present application as well as the functionalities described to be performed by the various entities are intended to be implemented by a respective entity for performing the respective steps and functionalities. Although in the following description of specific embodiments, a specific functionality or step to be performed by external entities is not reflected in the description of a specific detailed element of the entity performing the specific step or functionality, it should be clear for the skilled in the art that these methods and functionalities can be implemented in respective software or hardware elements, or any kind of combination thereof. BRIEF DESCRIPTION OF DRAWINGS

[0049] The above aspects and their realizations are explained in greater detail below with respect to specific embodiments in connection with the attached drawings, in which:

[0050] Figure 1 A QoS processing system with at least two hierarchy levels is shown.

[0051] Figure 2 A QoS processing system with five hierarchy levels is shown.

[0052] Figure 3 A QoS processing system with four hierarchy levels is shown.

[0053] Figure 4 A local QoS policy handling at a service instance in the QoS processing system of Figure 3 is shown.

[0054] Figure 5 A host level QoS policy handling in the QoS processing system of Figure 3 is shown.

[0055] Figure 6 A cluster level QoS policy handling in the QoS processing system of Figure 3 is shown.

[0056] Figure 7 A zone level QoS policy handling in the QoS processing system of Figure 3 is shown.

[0057] Figure 8 A method for QoS processing of one or more services is shown. DETAILED DESCRIPTION

[0058] Figure 1A QoS processing system 100 according to the present application is shown. The QoS processing system 100 is configured to process QoS for one or more services, in particular by applying one or more QoS policies to service instances 105 of the one or more services. The QoS processing system 100 comprises a plurality of QoS processing nodes 101, 102 arranged in a hierarchical tree structure comprising at least two hierarchy levels 103, 104. In Figure 1 particular, two hierarchy levels 103, 104 are shown. However, this is merely an example, the hierarchical tree structure can comprise three hierarchy levels, or four hierarchy levels (as shown in Figures 3 to 7 ), or five hierarchy levels (as shown in Figure 2 ), or even more hierarchy levels.

[0059] As shown in Figure 1 , the highest hierarchy level 103 of the at least two hierarchy levels 103, 104 comprises one or more root nodes 101 of the hierarchical tree structure of QoS processing nodes 101, 102. Each root node 101 is associated with a set of service instances 105 of one or more services. Further, the lowest hierarchy level 104 of the at least two hierarchy levels 103, 104 comprises a plurality of leaf nodes 102 of the tree structure of QoS processing nodes 101, 102. Each leaf node 102 is associated with a service instance 105 of one or more services, each leaf node 102 is derived from at least one root node 101, and each service instance 105 of the set of service instances 105 of each root node 101 is associated with a leaf node 102 derived from this root node 101.

[0060] In the hierarchical tree structure of the QoS processing system, each leaf node 102 is configured to apply a local QoS policy to the associated service instance 105 of each leaf node 102. Further, each root node 101 is configured to apply a first common QoS policy to the set of service instances 105 of each root node 101. Thus, the local QoS policy is applied only to the associated service instance 105, while the common QoS policy is applied in a common way to a plurality of service instances 105.

[0061] Each QoS processing node 101, 102 can comprise a processor or processing circuitry (not shown) for performing, conducting or initiating the various operations of the QoS processing node 101, 102 described herein. The processing circuitry can comprise hardware and / or the processing circuitry can be controlled by software. The hardware can comprise analog circuitry or digital circuitry, or both. The digital circuitry can comprise components such as application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), digital signal processors (DSPs), or multi-purpose processors, etc. Each QoS processing node 101, 102 can also comprise a memory circuitry storing one or more instructions that can be executed by the processor or processing circuitry (in particular, under the control of software). For example, the memory circuitry can comprise a non-transitory storage medium storing executable software code that, when executed by the processor or processing circuitry, causes the various operations of the QoS processing node 101, 102 to be performed. In one embodiment, the processing circuitry comprises one or more processors and a non-transitory memory connected to the one or more processors. The non-transitory memory can carry executable program code that, when executed by the one or more processors, causes the respective QoS processing node 101, 102 to perform, conduct or initiate the operations or methods described herein.

[0062] According to the present application, two main principles can be used. First is the multi-level hierarchical QoS processing node 101, 102. The QoS processing node 101, 102 can be used for rate limiter processing. Second, dynamic sharding can be implemented to achieve shared-nothing (SN) for example in multi-tenant deployments.

[0063] For example, the QoS processing system 100 can be a rate limiter system having a series of QoS processing nodes 101, 102 connected in a hierarchical order, i.e. in a hierarchical tree structure. The QoS processing nodes 101, 102 at each level 103, 104 can only communicate with their respective parent node (a parent node is a QoS processing node arranged on a higher level, from which the QoS processing nodes 101, 102 are derived) or their respective child node (a child node is a QoS processing node on a lower level, from which the QoS processing nodes 101, 102 are derived), which communication can be generally referred to as "north-south traffic". Adjacent QoS processing nodes 101, 102 on the same level 103, 104 can not interact with each other for any QoS policy decisions. Instead, they can communicate with their parent node or child node, respectively.

[0064] Each QoS processing node 101, 102 in each level 103, 104 of the hierarchical tree structure can perform one or more of the following operations:

[0065] Collect relevant QoS metrics from its one or more child nodes.

[0066] Aggregate the collected QoS metrics, e.g. based on one or more QoS policies (such as rate limiter policies).

[0067] Report the aggregated QoS metrics to its one or more parent nodes, e.g. as configured by the QoS policies.

[0068] Poll one or more QoS configurations from its one or more parent nodes.

[0069] Prepare a subset of the polled one or more QoS configurations and push the subset to its one or more child nodes.

[0070] Leaf nodes 102 of the hierarchical tree structure are arranged at service instances 105, where local QoS policies can be applied. The QoS policies (e.g. rate limiter policies) can be determined, e.g. by a customer, at the root node 101. These QoS policies (e.g. rate limiter policies) can be propagated down to the leaf nodes 102 (associated with the service instances 105) at regular intervals for local QoS decisions. Each QoS processing node 101, 102 in the hierarchical tree structure can be aware of its (in the hierarchy) position and can always have a local copy of the hierarchical tree structure. When the hierarchical tree structure is modified, the modification can be reported to all QoS processing nodes 101, 102 on all levels 103, 104.

[0071] When starting or stopping one or more service instances 105, this information can be propagated from one or more leaf nodes up to one or more root nodes 101, e.g. by intermediate QoS processing nodes in the hierarchical tree structure. The one or more root nodes 101 can then identify the QoS policy for a given one or more services or service instances 105 and can propagate this QoS policy down to the QoS processing nodes 101, 102 in each hierarchy level 103, 104 including the leaf nodes 102. For example, the scheduler in each hierarchy level 103, 104 can determine who (e.g. which QoS processing node 101, 102) is responsible for the given service or service instance 105 and can assign this particular QoS processing node 101, 102 to the given service or service instance 105 for QoS policy processing, i.e. applying the QoS policy. Next, the QoS processing nodes 101, 102 in each hierarchy level 103, 104 can update with their new list of parent nodes they have to interact with. This can be determined by the scheduler using the list of services the QoS processing nodes 101, 102 are responsible for and the QoS policy being configured for this service. After updating the node list, the QoS processing can be distributed over multiple QoS processing nodes 101, 102 using sharding.

[0072] The QoS processing system 100 can comprise at least three hierarchy levels 103, 104. For example, Figure 2 An example of a QoS processing system 100 provided by the present invention with five hierarchy levels 103, 204, 205, 206, 104 is shown.

[0073] Thus, the QoS processing system 100 can further comprise at least one intermediate hierarchy level 204, 205, 206 (e.g. Figure 2 three intermediate hierarchy levels) arranged between the highest hierarchy level 103 and the lowest hierarchy level 104 and comprising a plurality of intermediate nodes 201, 202, 203 of QoS processing nodes 101, 102, 202, 202, 203. Each intermediate node 201, 202, 203 is derived from at least one root node 101. Each intermediate node 201, 202, 203 is further associated with a subset of service instances 105 of one or more services and the service instances 105 of the subset of service instances 105 of each intermediate node 201, 202, 203 are associated with leaf nodes 102 derived from this intermediate node 201, 202, 203. Each intermediate node 201, 202, 203 is configured to apply a second common QoS policy to the subset of service instances 105 of each intermediate node 201, 202, 203.

[0074] As in Figure 2As can be seen, QoS processing nodes 101, 102, 201, 202, 203 can be deployed at instance 105 (leaf node 102 of hierarchy level 104), host (intermediate node 203 of hierarchy level 206), cluster (intermediate node 202 of hierarchy level 205), availability zone (AZ) (intermediate node 204 of hierarchy level 204), and region (root node 101 of the highest hierarchy level 103). One or more instance level QoS processing nodes 102 can run as part of the application itself rather than a separate binary. Depending on the configured QoS policies, the particular allocated nodes 101, 201, 202, 203 can participate in the aggregation and processing of QoS metrics and policies for a given service.

[0075] Figures 3 to 7 Another example of a QoS processing system 100 with four hierarchy levels 103, 205, 206, 104 is shown. In this example, 5 cloud services (DB service 1, OBS service 2, analytics service 3, DNS service 4, and alert service 5) are shared by the QoS processing system 100 deployed for a single region. From Figure 3 As can be seen, there is a QoS processing node 102 (leaf node, which can be implemented as an embedded library) inside each service instance 105. In addition, each host has its own intermediate QoS processing node 203 for local QoS processing. At the cluster level, there is a set of intermediate QoS processing nodes 202 deployed for QoS processing. The number of QoS processing nodes 202 in each cluster can vary based on the workload (e.g., the number of services and QoS policies to be processed within the cluster). At the region level, there is another set of QoS processing root nodes 101 for region level QoS processing.

[0076] Local QoS policy processing at the service instance 105 of the lowest hierarchy level 104 can work as described below with reference to Figure 4 For example, in this example, the domain name system (DNS) service exists only as a single instance 105 throughout the cloud. This means that QoS processing can be done locally within the service instance 105 itself without any distributed computation. The QoS processing system 100 can be aware of the service topology to determine the QoS processing nodes 102 to be processed.

[0077] Similarly, the alert service can do QoS processing locally, even though it has more than one instance 105 in the cluster. This is because when the QoS policy is local (e.g., it does not require any distributed computation), only the local QoS processing node 102 can be used. In this example, the alert service has all the policies needed to process QoS locally, so it is treated as local processing.

[0078] Host level QoS policy processing can work as described below with reference to Figure 5 The host level is the next level 205 in the hierarchical tree structure for QoS processing. A host level QoS processing node 203 can be responsible for QoS processing for services in the same host. Alternatively, it can be responsible for QoS policies configured for a particular host. The OBS service can have policies configured for a particular host, so it can be processed locally by the host level QoS processing node 203 using data from the service instance QoS processing node 102.

[0079] Cluster level QoS policy processing can work as described below with reference to Figure 6 The next level 205 of QoS processing is the cluster level. At the cluster level, the QoS processing system 100 can use more than one QoS processing node 202, and the workload can be sharded across these intermediate QoS processing nodes 202. The sharding algorithm can run at the cluster level and can ensure that the intermediate QoS processing nodes 202 are not overloaded and that there are no noisy neighbor situations. The sharding algorithm used can be completely up to the application and its deployment. Examples of available algorithms are:

[0080] Service sticky sharding algorithm: always process all QoS policies for a given service in a single QoS processing node 202.

[0081] Staggered sharding algorithm: stagger the load across QoS processing nodes 202 based on policies, services, or even topology.

[0082] Random sharding algorithm: randomly assign service instances 105 across cluster QoS processing nodes 202.

[0083] In this example, the database (DB) service spans a single cluster, so any QoS policy applied across the DB service would be handled at the cluster-level QoS processing node 202. For example, when the total number of active DB connections from a customer is greater than 100, new DB connections from the customer can be blocked. To apply such a QoS policy, the QoS processing system 100 can need to collect the number of DB connections per customer from all active DB instances and aggregate these values to determine if the connections need to be blocked. Afterwards, the decision should be published to the QoS processing node that is making the request. For the hierarchical model of this example, the aggregation of the first level metrics would happen at the host level and then the results would be forwarded to the cluster QoS processing node 202. At the cluster level, these aggregated QoS metrics can be aggregated again to find the final value.

[0084] The following references are made to Figure 7 Region-level QoS policy processing is described. Similar to cluster-level QoS processing, there can be a set of region QoS processing nodes 101 for region-level policy processing. The region-level QoS processing nodes 101 of a hierarchy level 103 can be sharded across different clusters based on various factors (e.g., service topology, configured policies, network topology, etc.).

[0085] As can be seen, the OBS service can span multiple clusters within a single region, and applying any policy across the OBS service can require policy management at the region level. For example, when the overall utilization of OBS CPU instances is greater than 70%, queries from a customer can be blocked. To apply such a QoS policy, each service can have to report its CPU utilization to the next level QoS processing node. At each hierarchy level, the data can be aggregated and the aggregated data can be forwarded all the way to the region-level QoS processing node 101. At the region level, the final aggregation can occur and the QoS policy decision can be made. The decision can then be pushed down to the service QoS processing nodes to properly handle the queries.

[0086] Figure 8 A method 800 provided by the present disclosure is shown. The method 800 is for QoS processing of one or more services and can be performed by Figures 1 to 7 the QoS processing system 100 shown.

[0087] The method 800 comprises a step 801 of allocating (801) or re-allocating a plurality of QoS processing nodes 101, 102, 201, 202, 203 in a hierarchical tree structure comprising at least two hierarchy levels 103, 104, 204, 205, 206. A highest hierarchy level 103 of the at least two hierarchy levels comprises one or more root nodes 101 of the hierarchical tree structure of QoS processing nodes, each root node 101 being associated with a set of service instances 105 of one or more services. A lowest hierarchy level 104 of the at least two hierarchy levels comprises a plurality of leaf nodes 102 of the hierarchical tree structure of QoS processing nodes, each leaf node 102 being associated with one service instance 105 of one or more services, each leaf node 102 being derived from at least one root node 101, and each service instance 105 of the set of service instances 105 of each root node 101 being associated with a leaf node 102 derived from this root node 101. The method 800 further comprises a step 802 of each leaf node 102 applying a local QoS policy to the service instance 105 associated with this leaf node 102. The method 800 further comprises a step 803 of each root node 101 applying a first common QoS policy to the set of service instances 105 associated with this root node 101.

[0088] The present solution provides at least the following advantages. Scalability, existing rate limiting solutions are not scalable at all or cannot perform QoS across regions, or even across AZs. The present solution solves this problem by handling QoS (e.g. rate limiting) in a hierarchical model. Multi-tenancy rate limiting support, as QoS processing nodes (e.g. rate limiting nodes) are allocated among service instances 105 based on QoS policies and service instance topology. This can guarantee that QoS processing does not interfere with each other. It can also ensure that there is no noisy neighbor situation when using this solution.

[0089] The present solution is applicable to large scale microservice deployments, where one wants to apply rate limiting policies on a large number of instances. The present solution is also most useful for large scale applications that are shared among multiple users or tenants that want to use the service in isolated mode. The present solution is also very useful for some large scale deployments.

[0090] With a single hierarchical QoS processing solution, QoS processing can be performed across all these instances without interfering with each other. Each service instance 105 is associated with a leaf node 102 of the hierarchical tree structure. The total number of hierarchy levels 103, 104, 204, 205, 206 and the number of QoS processing nodes 101, 102, 201, 202, 203 in each hierarchy level can be completely based on the type of QoS policy and the scale of one or more applications.

[0091] The application has been described in connection with various embodiments, as examples and implementations. However, alterations, modifications, and improvements to the application will become apparent to those of ordinary skill in the art upon reading the foregoing description, and it is intended to include all such alterations, modifications, and improvements within the scope of the claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. A single element or other unit can fulfill the purposes of several entities or items recited in the claims. Reference to an item or an element does not exclude that the item or element is part of a larger whole. Reference to an item or an element does not exclude that the item or element is part of a larger whole.

Claims

1. A quality of service (QoS) handling system (100) for one or more services, characterized by, The QoS processing system (100) comprises: a plurality of QoS processing nodes (101, 102, 201, 202, 203) arranged in a hierarchical tree structure comprising at least two level ranks (103, 104, 204, 205, 206); a highest level rank (103) of the at least two level ranks (103, 104, 204, 205, 206) comprising one or more root nodes (101) of the hierarchical tree structure of the QoS processing nodes (101, 102, 201, 202, 203), each root node (101) being associated with a set of service instances (105) of the one or more services; a lowest level rank (104) of the at least two level ranks (103, 104, 204, 205, 206) comprising a plurality of leaf nodes (102) of the hierarchical tree structure of the QoS processing nodes (101, 102, 201, 202, 203), each leaf node (102) being associated with one service instance (105) of the one or more services, each leaf node (102) being derived from at least one root node (101), and the service instances (105) of the set of service instances (105) of each root node (101) being associated with the leaf nodes (102) derived from the root node (101); each leaf node (102) being configured to apply a local QoS policy to the associated service instance (105) of each leaf node (102); each root node (101) being configured to apply a first common QoS policy to the set of service instances (105) of each root node (101); each particular QoS processing node (101, 102, 201, 202, 203) of the QoS processing nodes being configured to: collect at least one QoS indicator of at least one service instance (105) from each child node (102, 201, 202, 203) of the QoS processing nodes, each child node (102, 201, 202, 203) being directly derived from the particular QoS processing node (101, 102, 201, 202, 203); aggregate the collected QoS indicators; provide the aggregated QoS indicators to each parent node (101, 201, 202, 203) of the QoS processing nodes, the particular QoS processing node (101, 102, 201, 202, 203) being directly derived from each parent node (101, 201, 202, 203).

2. The QoS processing system (100) according to claim 1, characterized in that the hierarchical tree structure comprising at least three level ranks (103, 104, 204, 205, 206), the QoS processing system (100) further comprising: at least one intermediate hierarchical level (204, 205, 206) arranged between the highest hierarchical level (103) and the lowest hierarchical level (104) and comprising a plurality of intermediate nodes (201, 202, 203) of the QoS processing nodes (101, 102, 201, 202, 203), each intermediate node (201, 202, 203) being derived from at least one root node (101), each intermediate node (201, 202, 203) being associated with a subset of service instances (105) of the one or more services, and the service instances (105) of the subset of service instances (105) of each intermediate node (201, 202, 203) being associated with leaf nodes (102) derived from the intermediate node (201, 202, 203); each intermediate node (201, 202, 203) is configured to apply a second common QoS policy to the subset of service instances (105) of each intermediate node (201, 202, 203).

3. The QoS processing system (100) according to claim 2, characterized in that, each particular QoS processing node (101, 102, 201, 202, 203) of the QoS processing nodes is configured to: receive a QoS configuration of at least one service instance (105) from each parent node (101, 201, 202, 203) of the QoS processing nodes from which the particular QoS processing node (101, 102, 201, 202, 203) is directly derived; and / or provide the received QoS configuration and / or a QoS configuration of at least one service instance (105) of the particular QoS processing node (101, 102, 201, 202, 203) to each child node (102, 201, 202, 203) of the QoS processing nodes directly derived from the particular QoS processing node (101, 102, 201, 202, 203).

4. The QoS processing system (100) according to claim 3, characterized in that each root node (101) is configured to distribute at least one rate limit parameter of at least one service instance (105) to each QoS processing node (101, 102, 201, 202, 203) derived from the root node (101).

5. The QoS processing system (100) of claim 4, wherein: each QoS policy comprising each local QoS policy, each first common QoS policy and / or each second common QoS policy is applied based on at least one of: one or more QoS indicators of the one or more service instances (105) to which the QoS policy is applied; one or more QoS configurations of the one or more service instances (105) to which the QoS policy is applied; one or more rate limit parameters of the one or more service instances (105) to which the QoS policy is applied.

6. The QoS processing system (100) according to claim 2, characterized in that, The local QoS policy, the first common QoS policy and / or the second common QoS policy comprises a rate limiting policy.

7. The QoS processing system (100) according to claim 1, characterized in that, Further comprising a control node for allocating and / or re-allocating one or more of the plurality of QoS processing nodes (101, 102, 201, 202, 203) in the hierarchical tree structure.

8. The QoS processing system (100) according to claim 7, characterized by The control node is one of the QoS processing nodes (101, 102, 201, 202, 203) of the hierarchical tree structure.

9. The QoS processing system (100) according to claim 7, characterized by The control node is for performing the allocation and / or the re-allocation based on at least one of: a load on currently allocated QoS processing nodes (101, 102, 201, 202, 203), QoS policies applied by the currently allocated QoS processing nodes (101, 102, 201, 202, 203), a communication cost between the currently allocated QoS processing nodes (101, 102, 201, 202, 203), a topology of the service instances (105) and a QoS configuration of each of the service instances (105).

10. The QoS processing system (100) according to claim 1, characterized in that, Each QoS processing node (101, 102, 201, 202, 203) in the hierarchical tree structure knows its position in the hierarchical tree structure and / or comprises information describing the whole hierarchical tree structure.

11. The QoS processing system (100) according to claim 10, characterized in that, The QoS processing nodes (101, 102, 201, 202, 203) in the hierarchical tree structure are for performing a consensus protocol to determine which QoS processing node (101, 102, 201, 202, 203) is responsible for which QoS policy and / or to share one or more QoS policies using the hierarchical tree structure.

12. The QoS processing system (100) according to claim 7 or 9, characterized by, The control node is further for reporting the allocation and / or re-allocation of the one or more QoS processing nodes (101, 102, 201, 202, 203) to all the QoS processing nodes (101, 102, 201, 202, 203).

13. The QoS processing system (100) according to any one of claims 1 to 11, characterized in that, The hierarchical tree structure comprises four hierarchy levels (103, 104, 205, 206), wherein: Each QoS processing node (102) of the lowest hierarchy level (104) is associated with one service instance (105) of one or more services of one host, each host executing one or more service instances (105) of the one or more services; Each QoS processing node (203) of a first intermediate hierarchy level (206) above the lowest hierarchy level (104) is associated with all the service instances (105) of one host; Each QoS processing node (202) of a second intermediate hierarchy level (205) above the first intermediate hierarchy level (206) is associated with all the service instances (105) of all hosts in one host cluster; Each QoS processing node (201) of a third intermediate hierarchy level (103) above the second intermediate hierarchy level (205) is associated with all the service instances (105) of all hosts in a host cluster of a plurality of host clusters; and Each QoS processing node (101) of a top hierarchy level (103) above the third intermediate hierarchy level (103) is associated with all the service instances (105) of all hosts in all host clusters of a plurality of host clusters. Each QoS processing node (101) of the highest hierarchy level (103) above the second intermediate hierarchy level (205) is associated with all the service instances (105) of all clusters in one cluster area.

14. A method (800) for quality of service (QoS) handling of one or more services, characterized by, The method (800) comprises: allocating (801) or re-allocating a plurality of QoS processing nodes (101, 102, 201, 202, 203) in a hierarchical tree structure comprising at least two hierarchy levels (103, 104, 204, 205, 206); a highest hierarchy level (103) of the at least two hierarchy levels (103, 104, 204, 205, 206) comprises one or more root nodes (101) of the hierarchical tree structure of the QoS processing nodes (101, 102, 201, 202, 203), each root node (101) being associated with a set of service instances (105) of the one or more services; a lowest hierarchy level (104) of the at least two hierarchy levels (103, 104, 204, 205, 206) comprises a plurality of leaf nodes (102) of the hierarchical tree structure of the QoS processing nodes (101, 102, 201, 202, 203), each leaf node (102) being associated with one service instance (105) of the one or more services, each leaf node (102) being derived from at least one root node (101), and the service instances (105) of the set of service instances (105) of each root node (101) being associated with the leaf nodes (102) derived from the root node (101); each leaf node (102) applies (802) a local QoS policy to the service instance (105) associated with the leaf node (102); each root node (101) applies (803) a first common QoS policy to the set of service instances (105) associated with the root node (101); each particular QoS processing node (101, 102, 201, 202, 203) of the QoS processing nodes is configured to: collect at least one QoS indicator of at least one service instance (105) from each child node (102, 201, 202, 203) of the QoS processing nodes, each child node (102, 201, 202, 203) being directly derived from the particular QoS processing node (101, 102, 201, 202, 203); aggregate the collected QoS indicators; provide the aggregated QoS indicators to each parent node (101, 201, 202, 203) of the QoS processing nodes, the particular QoS processing node (101, 102, 201, 202, 203) being directly derived from each parent node (101, 201, 202, 203).

15. A computer program product comprising instructions, characterized in that, The instructions, when executed by a computer, cause the computer to perform the method (800) according to claim 14. The instructions, when executed by a computer, cause the computer to perform the method (800) according to claim 14.

Images