Edge device for distributed traffic engineering system with quality of service control of multiple flow groups

Abstract

The invention relates to quality of service control for multiple flow groups. The invention proposes an edge device for a distributed traffic engineering system having quality of service control of the plurality of flow groups routed over a set of overlay links, and a corresponding method for operating the edge device. The edge device is configured to: receive one or more service level agreement requirements for the plurality of flow groups from a network controller; receiving first control information from one or more other edge devices in the distributed traffic engineering system; acquiring monitoring information of the plurality of flow groups and the coverage link set; and adjusting a plurality of queue parameters of the plurality of flow groups based on at least one of the first control information, the monitoring information and the one or more SLA requirements.

Description

Technical Field

[0001] The present invention relates to service quality control of multiple flow groups.The present invention proposes an edge device and a corresponding method for operating the edge device. Background Art

[0002] Traditionally, traffic engineering is key to controlling how bandwidth is shared in a network, especially to meet the service level agreements (SLAs) of applications in terms of throughput, latency, loss, and jitter. Routing over multiple paths helps to better utilize network capacity and can dynamically select paths based on network conditions. Flows traversing common links may experience bandwidth contention, which can degrade QoS. QoS policies can help control bandwidth sharing between concurrent flows.

[0003] Multipath routing and QoS policies are commonly used in software-defined wide area networks (SD-WANs), where an enterprise's headquarters site and local branches can be interconnected through different networks, such as multi-label protocol switching (MLPS), the Internet, and / or long-term evolution (LTE). In such an overlay network, a controller entity is usually deployed at the headquarters site to manage the network, while the business premises are equipped with access routers (ARs). The controller is responsible for configuring the access routers and updating high-level routing policies. The access routers then route traffic over the available set of overlay links to be consistent with the policies defined by the controller and local knowledge of network conditions (e.g., average delay, loss, jitter of the overlay links).

[0004] Applications are often mapped to different "flow groups" with different QoS requirements, for example:

[0005] - Production (medium traffic, high SLA requirements)

[0006] -VoIP (medium traffic, high SLA requirements)

[0007] -Office (high traffic, medium SLA requirements)

[0008] -Data (medium traffic, low / no SLA requirements)

[0009] To meet the flow group requirements, traffic scheduling is traditionally performed using a class based queuing (CBQ) architecture. Each access router applies QoS policies on each of its outgoing ports before the traffic enters the WAN area. A strict priority (SP) scheduler is used to provide priority to high priority flow groups (e.g., real-time traffic such as IP voice or production traffic), and a weighted-fair queuing (WFQ) scheduler and shaper are used to control the amount of bandwidth allocated to low priority flow groups.

[0010] The shaping rate and WFQ weight of each flow group on the egress port of the access router can be determined. For example, one approach is to assign fixed values ​​based on the relative priority of the flow groups. However, performance can be improved through dynamic adaptation of parameters. When the network is not congested, the adaptation of the weights can be used to provide more bandwidth or "service rate" to the flow groups to reduce their latency. When the end-to-end delay requirements are already met or more stringent delay requirements can be coped with, the latency can be further improved. When the network is congested, the shaping can be adjusted to protect high priority flow groups, for example, leaving them enough service rate so that they can meet their delay requirements, and to ensure fairness among low priority flow groups, for example, proportionally reducing the bandwidth of all flow groups.

[0011] Adaptation of QoS parameters such as shaping and weights requires solving a global optimization problem and cannot be done by each access router in an independent manner. When multiple access routers send traffic on the same bottleneck link, it may not be possible to maximize overall SLA satisfaction and may not guarantee fairness in the event of congestion. For example, an enterprise branch office can be connected to two data centers through an MPLS virtual private network (VPN) and broadband Internet. When the two data centers send traffic from the first port and the second port of MPLS to the third port of the branch office, they will both compete on the same bottleneck. Once path selection is implemented to choose between MPLS and the Internet for a set of flow groups, the QoS policies applied on the first port and the second port can help protect high priority traffic and better share bandwidth within the low priority flow group to meet SLA requirements and ensure fairness in the event of congestion.

[0012] As traffic and network conditions change over time, QoS policies and routing can be adjusted to better use network resources and meet application requirements.

[0013] Many active queue management (AQM), traffic conditioning, and network scheduling techniques have been proposed to help maintain latency and throughput requirements. In general, traditional AQM is about implicitly, such as through packet dropping, or explicitly, such as explicit congestion notification (ECN) marking or bandwidth pricing, signaling several sources or destinations that share a bottleneck link to adjust queue parameters and globally optimize performance.

[0014] In addition, dynamic adaptation of scheduling parameters can be used, for example, adaptive weighted fair queuing (AWFQ), including:

[0015] Control the output queue of the access device.

[0016] Centralized approach using bandwidth proxies.

[0017] A distributed protocol that triggers queue adjustments. The proxy at the destination is notified of latency violations, causing the upstream proxy to adjust its weights.

[0018] An "adaptive QoS" approach can enable WAN interface shapers and per-tunnel shapers at the enterprise edge to adapt to the available WAN bandwidth. Local UP / DOWN decisions to increase or decrease shaping parameters can be made based on packet loss. However, this approach may only achieve a local optimum.

[0019] PfR (Performance Routing) / iWAN schemes have been proposed for dynamic selection of paths in WAN networks to meet SLA requirements. In this architecture, users define policies at the master controller (MC) level based on the SLA requirements of each application. The access router monitors the quality of the path and sends monitoring updates to the MC. The MC then compares the quality of the path with the application requirements and updates the path selection in the router when needed. This approach requires frequent communication between the device and the controller because the path is actually selected in the MC. Therefore, the MC is slow to react to changes in network conditions. In addition, this only controls the routing and does not optimize the QoS policy inside the device. Summary of the invention

[0020] In view of the above situation, an embodiment of the present invention aims to provide efficient queue parameter optimization inside an edge device.

[0021] These and other objects are achieved by embodiments of the invention as set out in the attached independent claims. Advantageous implementations of these embodiments are further defined in the dependent claims.

[0022] A first aspect of the present invention provides an edge device for a distributed traffic engineering system, the distributed traffic engineering system having quality of service (QoS) control for multiple flow groups routed on a set of overlay links, wherein the edge device, such as a QoS agent and / or a scheduling optimization module included in the edge device, is used to: receive one or more service level agreement (SLA) requirements of the multiple flow groups from a network controller; receive first control information from one or more other edge devices in the distributed traffic engineering system; obtain monitoring information of the multiple flow groups and the set of overlay links; and adjust multiple queue parameters of the multiple flow groups based on the first control information, the monitoring information and at least one of the one or more SLA requirements.

[0023] For example, the edge device may be configured to adjust multiple queue parameters of multiple flow groups based on the first control information, the monitoring information, and one or more SLA requirements.

[0024] If the edge device determines that certain SLAs cannot be met, the edge device can send a request for help.

[0025] A global framework can be defined to optimize queue parameters through cooperation among edge devices.

[0026] Edge devices can determine queue parameters based on global rate-based decisions based on SLA measurements and predictions.

[0027] In another implementation of the first aspect, the edge device includes a control plane, wherein the control plane includes a QoS agent, such as the QoS agent of the first aspect, and / or a routing agent; and / or wherein the QoS agent includes a scheduling optimization module, such as the scheduling optimization module of the first aspect, and / or an analysis module to support the generation of help requests; and / or the edge device includes a data plane, wherein the data plane includes a scheduling engine, a monitoring engine and / or a forwarding engine.

[0028] In another implementation of the first aspect, the edge device, such as the QoS agent, is further used to: determine whether a help request is required based on the monitoring information and the one or more SLA requirements; if a help request is required, provide a help request to the network controller.

[0029] In another implementation of the first aspect, the edge device, such as the routing agent, is further configured to: receive a routing policy from the network controller if at least one edge device in the distributed traffic engineering system provides a request for assistance to the network controller.

[0030] The routing policy may be a smart policy routing (SPR) policy.

[0031] In another implementation of the first aspect, the edge device, such as the QoS agent, is further used to: determine second control information based on multiple adjusted queue parameters and / or the monitoring information; and provide the second control information to the one or more other edge devices.

[0032] The edge device and one or more other edge devices may be configured to transmit control information between each other. Transmitting control information may include receiving first control information and / or providing second control information one or more times, respectively.

[0033] The edge device may be configured to receive the first control information and / or provide the second control information more frequently and / or at a higher repetition frequency than other steps that the edge device is configured to perform, respectively.

[0034] In another implementation of the first aspect, the edge device, such as the QoS agent, is further configured to receive one or more endpoints of the help request from the network controller.

[0035] In another implementation of the first aspect, the edge device, such as the QoS agent and / or the scheduling optimization module, is further configured to adjust the multiple queue parameters based on an SLA prediction model.

[0036] In another implementation of the first aspect, the edge device, such as the QoS agent and / or the scheduling optimization module, is also used to periodically and / or repeatedly perform the following operations: receive first control information from one or more other edge devices in the distributed traffic engineering system; obtain monitoring information of the multiple flow groups; iteratively adjust the multiple queue parameters of the multiple flow groups based on the first control information and / or the monitoring information; determine second control information based on the multiple adjusted queue parameters; and provide the second control information to the one or more other edge devices.

[0037] In another implementation of the first aspect, the edge device is also used to periodically and / or repeatedly perform the following operations: determine whether a help request is required based on the monitoring information and the one or more SLA requirements; if a help request is required, provide a help request to the network controller; and / or if at least one edge device in the distributed traffic engineering system provides a help request to the network controller, receive the routing policy from the network controller.

[0038] For example, the routing agent may be used to receive a routing policy from a network controller, and the QoS agent and / or the analysis module may be used to determine whether a help request is needed and provide the help request to the network controller.

[0039] In another implementation of the first aspect, the second control information includes the load of each overlay link in the overlay link set and / or a scalar for each flow group in the multiple flow groups, and / or the first control information includes the load of each overlay link in the second overlay link set corresponding to the one or more other edge devices and / or a scalar for each flow group in the second multiple flow groups corresponding to the one or more other edge devices.

[0040] In another implementation of the first aspect, the multiple queue parameters include a shaping rate for each flow group and / or a weighted-fair queuing (WFQ) weight for each flow group, and / or the one or more SLA requirements include at least one of the throughput, delay, and jitter and loss requirements for each of the multiple flow groups.

[0041] In another implementation of the first aspect, the scheduling optimization module is used to provide the multiple adjusted queue parameters to the scheduling engine, and / or the scheduling engine is used to enforce a bandwidth sharing policy to ensure the SLA requirements and / or fairness according to a fairness target.

[0042] In another implementation of the first aspect, the QoS agent, such as the analysis module, is used to obtain the monitoring information from the monitoring engine, and / or the monitoring engine is used to determine the monitoring information based on measurements of each of the multiple flow groups.

[0043] In another implementation of the first aspect,

[0044] The routing agent is used to: if at least one edge device in the distributed traffic engineering system provides a request for help to the network controller, receive the routing policy from the network controller and / or receive the request for help from the QoS agent, such as the analysis module; and provide the routing policy to the forwarding engine.

[0045] If the edge device provides a request for assistance to the network controller, the forwarding engine can expect to receive routing policies from the routing agent.

[0046] In another implementation of the first aspect,

[0047] The edge device, such as the forwarding engine, is configured to forward traffic of each flow group in the plurality of flow groups on the set of overlay links, for example, based on the routing policy.

[0048] In another implementation of the first aspect,

[0049] The analysis module is used to: receive the monitoring information from the monitoring engine; and / or receive the multiple adjusted queue parameters from the scheduling optimization module; determine whether a help request is required based on the monitoring information and the one or more SLA requirements and / or the multiple adjusted queue parameters; if a help request is required, provide the help request to the network controller and / or the routing agent.

[0050] In another implementation of the first aspect,

[0051] The edge device, such as the analysis module, is further configured to determine whether a help request is required based on the SLA prediction model and / or by determining and / or predicting whether the one or more SLA requirements are met.

[0052] In another implementation of the first aspect, the edge device, such as the scheduling optimization module, is used to determine multiple rate allocations, the multiple rate allocations including rate allocations on each overlay link in the overlay link set for each flow group in the multiple flow groups; adjusting the multiple queue parameters based on the multiple rate allocations, and / or wherein determining the multiple rate allocations is based on: the first control information, the monitoring information, the one or more SLA requirements, and optionally the SLA prediction model.

[0053] In another implementation of the first aspect,

[0054] The monitoring information includes, for each flow group in the plurality of flow groups and / or for each overlay link in the overlay link set: measured throughput, and / or QoS statistics, wherein the QoS statistics include at least one of delay, jitter and loss.

[0055] In another implementation of the first aspect,

[0056] The edge device, such as the analysis module, is used to: compare the measured throughput of each of the multiple flow groups with the one or more SLA requirements and / or with the multiple rate allocations; if the measured throughput is insufficient to meet at least one of the one or more SLA requirements, provide the help request to the routing agent, the network controller, and at least one of the one or more endpoints of the help request.

[0057] In another implementation of the first aspect, the edge device, such as the scheduling optimization module, is used to adjust the multiple queue parameters using a class-based queuing (CBQ) architecture.

[0058] The CBQ architecture can be based on a QoS optimization model and / or can include meeting SLA requirements for each of multiple flow groups, protecting high priority flow groups by, for example, ensuring that SLA requirements are met, and ensuring fairness for low priority flow groups according to fairness constraints by, for example, eliminating omissions of each low priority flow group.

[0059] The CBQ architecture can be based on a QoS optimization model and / or can include satisfying SLA requirements for all flow groups corresponding to an edge device and one or more other edge devices in a distributed traffic engineering system, protecting high priority flow groups by, for example, ensuring that SLA requirements are met, and ensuring fairness for low priority flow groups according to fairness constraints by, for example, eliminating omissions of each low priority flow group.

[0060] In another implementation of the first aspect, the edge device, such as the scheduling optimization module, is configured to determine the multiple rate allocations based on a QoS optimization model.

[0061] The QoS optimization model can minimize SLA violations and / or rate violations according to an objective function.

[0062] In another implementation of the first aspect, the QoS optimization model includes at least one of the following constraints: satisfying the capacity of each overlay link in the overlay link set, the target of rate allocation fairness, and / or wherein the edge device, such as the scheduling optimization module, is used to determine SLA violations, rate allocations and / or rate violations, and the QoS optimization model is based on the SLA violations, the rate allocations and / or the rate violations.

[0063] In another implementation of the first aspect, the constraint satisfying the capacity of each overlay link in the overlay link set is relaxed according to a Lagrangian relaxation model, and the edge device, such as the scheduling optimization module, is used to: use a Lagrangian multiplier Penalizes violations of relaxed constraints, which imposes a cost on constraint violations.

[0064] In another implementation of the first aspect,

[0065] The QoS optimization model includes at least one of the following inputs: one or more flow groups, one or more tunnels, wherein each tunnel is defined by an origin and a destination, the multiple flow groups, and / or wherein each flow group is defined by a traffic class and its origin and destination, the traffic demand of each flow group on each overlay link, the SLA prediction model of each flow group on each overlay link, the SLA requirement of each flow group, and a penalty for a demand violation of each flow group; and / or the QoS optimization model includes at least one of the following outputs: rate allocation of each flow group on each overlay link, SLA violation of each flow group on each overlay link, rate violation of each flow group on each overlay link, and intermediate variables of each flow group on each overlay link, used to optimize fairness according to a fairness target.

[0066] In another implementation of the first aspect,

[0067] The fairness target is based on at least one of: a minimum and / or maximum rate allocation for each of the plurality of flow groups, a proportional rate allocation constraint for each of the plurality of flow groups.

[0068] A second aspect of the present invention provides a network controller for a distributed traffic engineering system, the distributed traffic engineering system having quality of service (QoS) control for multiple flow groups routed on a set of overlay links, wherein the network controller is used to: provide SLA requirements for each of the multiple flow groups to two or more edge devices of the distributed traffic engineering system; receive a request for assistance from at least one of the two or more edge devices; and provide routing policies to at least one of the two or more edge devices.

[0069] The network controller may determine a routing strategy based on information about all flow groups and overlay link sets corresponding to two or more edge devices, wherein the information may be obtained from the two or more edge devices.

[0070] In another implementation of the second aspect, the network controller is further configured to provide one or more endpoints of a help request to the two or more edge devices.

[0071] A third aspect of the present invention provides a method for operating an edge device for a distributed traffic engineering system, wherein the distributed traffic engineering system has quality of service (QoS) control for multiple flow groups routed on an overlay link set, wherein the method includes: receiving one or more service level agreement (SLA) requirements of the multiple flow groups from a network controller; receiving first control information from one or more other edge devices in the distributed traffic engineering system; obtaining monitoring information of the multiple flow groups and the overlay link set; and adjusting multiple queue parameters of the multiple flow groups based on the first control information, the monitoring information and at least one of the one or more SLA requirements.

[0072] The method of the third aspect may have an implementation corresponding to the implementation of the device of the first aspect. The method of the second aspect and its implementation achieves the advantages and effects described above for the device of the first aspect and its corresponding implementation.

[0073] In the present invention, an “edge device” may also be referred to as an “access router” or a “switch”.

[0074] Furthermore, in the present invention, the analysis module may be referred to as a "rate and SLA analysis module".

[0075] It should be noted that all devices, elements, units and modules described in the present invention can be implemented in software or hardware elements or any kind of combination thereof. All steps performed by various entities described in the present invention and the functions to be performed by various entities described are intended to refer to the corresponding entities being suitable for or used to perform the corresponding steps and functions. Although in the description of the following specific embodiments, the specific functions or steps performed by the external entity are not reflected in the description of the specific elements of the entity that performs the specific steps or functions, it should be clear to the technician that these methods and functions can be implemented in the corresponding hardware elements or software elements or any kind of combination thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] In conjunction with the accompanying drawings, the following description of specific embodiments will illustrate the above aspects and their implementation methods, wherein:

[0077] Figure 1 An edge device provided by an embodiment of the present invention is shown;

[0078] Figure 2 A network controller provided by an embodiment of the present invention is shown;

[0079] Figure 3 An exemplary architecture of an edge device and a network controller is shown;

[0080] Figure 4 An exemplary architecture of an edge device and a network controller is shown;

[0081] Figure 5 An exemplary process performed by an edge device to adjust queue parameters provided by an embodiment of the present invention is shown;

[0082] Figure 6 The steps that an edge device provided by an embodiment of the present invention may perform to trigger a help request are shown;

[0083] Figure 7 The method provided by the embodiment of the present invention is shown. DETAILED DESCRIPTION

[0084] Figure 1 The edge device 100 provided by an embodiment of the present invention is shown. In addition, Figure 1 A distributed traffic engineering system 200 is shown, including an edge device 100, a network controller 201, and one or more other edge devices 202, wherein the edge device 100 and / or the distributed traffic engineering system has QoS control for multiple flow groups routed on a set of overlay links. The edge device 100 is used to receive one or more SLA requirements for the multiple flow groups from the network controller 201, and receive first control information 102 from one or more other edge devices 202 in the distributed traffic engineering system 200. In addition, the edge device 100 is used to obtain monitoring information 103 of the multiple flow groups and the set of overlay links, and adjust multiple queue parameters 104 of the multiple flow groups based on at least one of the first control information 102, the monitoring information 103, and the one or more SLA requirements 101.

[0085] In the present invention, the edge device 100 corresponding to a plurality of flow groups routed on the overlay link set may refer to a plurality of flow groups routed on the overlay link set managed by the edge device 100 .

[0086] The plurality of flow groups routed on the set of overlay links may correspond only to the edge device 100. Alternatively, the plurality of flow groups routed on the set of overlay links may correspond to the edge device 100 and one or more other edge devices 202.

[0087] The distributed traffic engineering system 200 can manage one or more flow groups routed on one or more overlay link sets. Each edge device 100 in the distributed traffic engineering system 200 can manage different multiple flow groups in the one or more flow groups routed on different overlay link sets in the one or more overlay link sets.

[0088] The communication between the network controller 201 and the edge device 100 may be minimized, and / or the communication between the edge device 100 and other edge devices may be minimized.

[0089] Figure 2 The network controller 201 provided by the embodiment of the present invention is shown. In addition, Figure 2 A distributed traffic engineering system 200 is shown, including a network controller 201, an edge device 100, and one or more other edge devices 202 forming two or more edge devices 100, 202, wherein the distributed traffic engineering system 200 has QoS control for multiple flow groups routed on an overlay link set. The network controller 201 is used to: provide SLA requirements 101 of each of the multiple flow groups to the two or more edge devices 100, 202 of the distributed traffic engineering system 200; receive a help request 105 from at least one edge device 100 of the two or more edge devices 100, 202; and provide a routing strategy 203 to at least one edge device 100 of the two or more edge devices 100, 202.

[0090] The network controller 201 may provide the routing policy 203 to an edge device 100 different from the edge device 100 from which the help request 105 was received. Alternatively, the network controller 201 may provide the routing policy 203 to the same edge device 100 from which the help request 105 was received.

[0091] Each edge device 100 in the distributed traffic engineering system 200 may include an intent-based “QoS agent” module. Figure 3 An exemplary architecture of an edge device 100 and a network controller 201 is shown, as well as how the edge device 100 may interact with the network controller 201 and one or more other edge devices 202 .

[0092] Figure 3 An exemplary edge device 100 equipped with a control plane and a data plane is shown. In the data plane, the monitoring module provides throughput information and QoS statistics 103, such as delay, jitter, and loss rate, for each flow group and each overlay link. The forwarding engine module forwards the traffic of each flow group on the overlay link set. The routing policy 203 is given by the local routing agent. For example, the routing agent may decide to load balance the traffic on the selected active link set using a weight that is inversely proportional to the capacity of the physical port, such as when implemented based on an SPR policy. The set of active / backup overlay links and / or paths is determined by a centralized controller (e.g., a network controller 201) based on SLA requirements 101 and measurements 103.

[0093] The QoS agent can continuously adjust multiple queue parameters 104, such as WFQ weights and / or shaping rates, based on SLA requirements 101, local measurements 103, and control information 102 received from other QoS agents at other edge devices. Two goals of the QoS agent can be: (1) meet the SLA requirements 101 of multiple flow groups, and (2) handle congestion efficiently and / or fairly. In addition, when the QoS agent detects an SLA violation, it can send a "help request" signal 105 to an external system (such as a routing system and / or a local routing agent running on the network controller 201) to request help, such as updating the SPR policy and / or turning on forward error encoding (FEC). The QoS agent can provide multiple adjusted queue parameters 104, such as WFQ weights and / or shaping rates, to the scheduling engine, which can enforce bandwidth sharing policies to ensure SLA requirements 101 and fairness, such as according to fairness targets. Due to traffic scheduling, such as WFQ and traffic shaping, minimum and maximum rate allocations can be configured for each flow group.

[0094] Figure 4 A more detailed exemplary architecture of the edge device 100 and the network controller 201 is shown. The QoS agent is divided into two modules: (1) a scheduling optimization module that works in conjunction with other QoS agents to optimize multiple queue parameters 104 in a distributed manner, and (2) a rate and SLA analysis module that verifies whether the SLA is met and requests assistance when needed.

[0095] The QoS proxy module may receive the following input data from the network controller 201:

[0096] SLA requirements 101, e.g., in terms of throughput, jitter, latency, loss for each flow group.

[0097] A list of endpoints to which the "Help Request 105" message is addressed when needed, such as local routing systems, local FEC / WOC modules, centralized routing controllers.

[0098] Based on the SLA requirements 101 and measurements 103 of a flow group, e.g., a flow group whose egress port is a local access router 100, the edge device 100 manages a scheduling optimization module of the edge device 100 to continuously adjust multiple queue parameters 104, e.g., shaping rate, WFQ weight. To coordinate with other QoS agents, the scheduling optimization module applies a distributed optimization method, which, for example, requires exchanging (1) the load of the outgoing coverage link and (2) a scalar for each flow group at each iteration of the algorithm. The method may or may not be embedded in an SLA prediction model to predict the impact of multiple adjusted queue parameters 104 on the flow group SLA.

[0099] Each update and / or adjustment of the queue parameters 104 can be provided to the rate and SLA analysis module in conjunction with an indication of an SLA prediction model that can be used by the scheduling optimization module. Based on this information and local measurements 103 of end-to-end QoS parameters from telemetry, the rate and SLA analysis module can continuously verify whether the SLAs for the corresponding flow groups are met. If the rate and SLA analysis module can determine that the scheduling optimization module cannot improve SLA satisfaction, the rate and SLA analysis module can decide to trigger a help request 105. For example, such a notification includes (1) a list of violating flow groups and (2) the level of violation. The rate and SLA analysis can adjust the load balancing by targeting the routing system locally or directly at the network controller 201 via the help request 105.

[0100] Figure 5 An exemplary process performed by the edge device 100 to adjust the plurality of queue parameters 104 is shown. The steps of the exemplary process may include:

[0101] Step 1: The edge device 100 receives updated information in the data receiving step:

[0102] - SLA requirement 101 from network controller 201

[0103] SLA description for each flow group (throughput, latency, jitter, packet loss)

[0104] Help request 105 endpoint

[0105] - Control information 102 from other edge QoS agents

[0106] - Information from monitoring 103 for each flow group

[0107] Step 2: Monitoring steps:

[0108] - Compare measured / predicted SLA 103 with SLA requirement 101

[0109] - If an SLA violation is detected, a notification is sent using the Help Request 105 endpoint

[0110] Step 3: Optimization steps:

[0111] - Solve QoS optimization problems to calculate new queue parameters based on, for example, SLA prediction models 104

[0112] - Prepare control information for other edge QoS agents, such as the load of the outgoing overlay link and a scalar for each flow group

[0113] Step 4: Enforce rate allocation and send control data in the configuration step:

[0114] - Enforces queue parameter 104 and provides it to the monitoring module

[0115] - Send control information 102 to one or more other QoS agents for coordination

[0116] Repeat steps 1 to 4 one or more times in the local control loop

[0117] All values ​​measured and / or received (eg link load) may be averaged over a moving time window to smooth out variations.

[0118] Iterations may be performed periodically and / or after a minimum number of updates are received from other QoS agents.

[0119] Multiple queue parameters 104 can be optimized in a collaborative and distributed manner among a set of access routers that share the same resources. Once the network controller 201 provides the target, the QoS agent can run at a high frequency without involving the network controller 201. However, if the situation cannot be improved and the SLA is still violated, the set of access routers can seek help.

[0120] QoS agents can use SLA predictions based on closed-form data-driven models.

[0121] Other scheduling architectures may be supported, such as hierarchical token bucket and weighted deficit round robin.

[0122] The exchanges between QoS agents can be based on existing protocols, such as Open Shortest Path First, Border Gateway Protocol.

[0123] The system for distributed intelligent queuing may be based on at least one of the following:

[0124] - Edge devices 100, 202 determined for each flow group, which manage queue parameters 104 by exchanging control information 102 with other edge devices.

[0125] - Decisions can be made iteratively to (1) maximize SLA satisfaction and (2) ensure fairness in the face of congestion. SLA prediction models can be used.

[0126] - Help may be requested from the network controller 201 based on SLA measurements.

[0127] Methods for optimal utility-based and fair allocation may include:

[0128] The edge devices 100 , 202 implement a distributed algorithm that receives small amounts of control information 102 from other edge devices and can adjust queue parameters 104 at each iteration.

[0129] Methods for triggering routing policy adjustment may include:

[0130] - Adjustments based on monitoring and prediction of the SLA for each flow group.

[0131] The following provides an example of a scheduling optimization module when using a class-based queuing (CBQ) architecture to adjust multiple queue parameters 104. The goal is to protect high priority flows, meet SLA requirements 101 for all flow groups, and ensure fairness for low priority flows.

[0132] Fairness can be ensured based on fairness objectives.

[0133] The edge device 100 may be configured to determine a plurality of rate allocations based on a QoS optimization model and / or an objective function.

[0134] The input to the QoS optimization model can be at least one of the following:

[0135] -H traffic classes

[0136] -N tunnels, each defined by a source and a destination

[0137] - K flow groups = H × N, each flow group is defined by a traffic class and its source and destination

[0138] - Traffic demand of flow group k on overlay link l

[0139] - SLA prediction model for flow group k on overlay link l,

[0140] -q k : SLA requirements for flow group k

[0141] -M k : Penalty for demand violation of flow group k

[0142] The output, e.g., decision variable, can be at least one of the following:

[0143] - Rate allocation for flow group k on overlay link l. The rate allocation can be greater than or equal to the demand.

[0144] - SLA violation for flow group k on overlay link l.

[0145] - Rate violation for flow group k on overlay link l.

[0146] - The intermediate variables of flow group k and coverage link l are used to optimize the fairness in the objective function.

[0147] Can be considered as the primary output and can be sent to the scheduling engine to redirect incoming flows to the appropriate overlay links.

[0148] The following nonlinear programming model calculates the rate allocation on each overlay link for each flow group. It penalizes SLA violations and rate violations in the objective function. The QoS optimization model and / or objective function can be defined as follows:

[0149]

[0150] in:

[0151]

[0152] Constraint (1) ensures that all link capacities are met. Constraint (2) supports the calculation of SLA violations. Constraint (3) supports the calculation of rate allocation and, when capacity is insufficient, the calculation of rate violations. Constraint (4) links variables x and z to ensure fairness, which can be called the fairness objective. The objective function can minimize rate and SLA violations.

[0153] Constraint (1) can be relaxed to obtain the following Lagrangian relaxation of the problem.

[0154] Lagrangian relaxation method uses Lagrangian multipliers The violation of the inequality constraint is penalized, which imposes a cost on the violation. After relaxing the capacity constraint (1), the following Lagrangian relaxation problem can be obtained:

[0155]

[0156] Given the Lagrange multiplier The Lagrangian relaxation problem is decomposed into |K| subproblems that can be solved independently, defined as follows:

[0157] The Lagrangian subproblem p associated with the flow group k∈K 1 :

[0158]

[0159] The Lagrangian subproblem P associated with a group of flows k∈K 2 :

[0160] P 1 The returned solution is not guaranteed to satisfy the capacity constraint (1). * To begin, the following ensures that the feasibility is understood.

[0161]

[0162] The following distributed algorithms can support solving Lagrangian relaxations and converge to the optimal solution:

[0163]

[0164] An example of how the rate and SLA analysis module may operate to trigger a help request message 105 is provided below.

[0165] Figure 6 Steps that the edge device 100 may perform to trigger the help request 105 are shown.

[0166] The rate and SLA analysis module may periodically measure the throughput of each flow group and compare it with the rate allocation given by the plurality of queue parameters 104. If the rate allocation is not sufficient to meet the SLA requirement 101, a help request 105 may be triggered, for example to an SPR module of the network controller 201 responsible for load balancing.

[0167] As mentioned above, is a rate used to adjust WFQ weights and shaping. The allocation is performed to allocate all available bandwidth, for example, in a fair manner and to meet SLA requirements 101.

[0168] The estimation of SLA satisfaction can be based on the SLA prediction model, for example, the end-to-end delay of flow group k on overlay link l Can be The function f, such as Figure 6 This function can be based on a closed-form expression from queueing theory or network calculus, or based on a data-driven approach using machine learning to approximate the delay.

[0169] Embodiments of the present invention combine some or all of the following advantages:

[0170] These decisions can be made locally. Hence, scalability is likely to be high.

[0171] Edge devices collaborate through a small data volume of control information 102. Therefore, the overhead may be low.

[0172] Intelligent queuing and routing may collaborate when needed.

[0173] The edge device 100 and / or the network controller 201 may include a processor. The processor may be configured to perform the method 300 together with the edge device 100 .

[0174] In general, a processor may be used to perform, implement, or initiate various operations of the edge device 100 and / or the network controller 201 described herein. The processor may include hardware and / or may be controlled by software. The hardware may include analog circuits or digital circuits, or both. The digital circuit may include components such as an application-specific integrated circuit (ASIC), a field-programmable array (FPGA), a digital signal processor (DSP), or a multi-purpose processor. The edge device 100 may also include a memory circuit that stores one or more instructions that may be executed by the processor or, in particular, under the control of software. For example, the memory circuit may include a non-transitory storage medium storing executable software code that, when executed by a processor, enables the execution of various operations of the edge device 100. In one embodiment, the edge device 100 may include one or more processors and a non-transitory memory connected to the one or more processors. The non-transitory memory may carry an executable program code that, when executed by one or more processors, enables the edge device 100 to perform, implement, or initiate the operations or methods described herein.

[0175] Figure 7 A method 300 provided by an embodiment of the present invention is shown. The method 300 can be performed by an edge device 100. The method 300 includes step 301: receiving one or more SLA requirements of multiple flow groups from a network controller 201. In addition, the method 300 includes step 302: receiving first control information 102 from one or more other edge devices 202 in the distributed traffic engineering system 200. In addition, the method 300 includes step 303: obtaining monitoring information 103 of multiple flow groups and coverage link sets. In addition, the method 300 includes step 304: adjusting multiple queuing parameters 104 of multiple flow groups based on at least one of the first control information 102, the monitoring information 103 and the one or more SLA requirements 101.

[0176] The invention has been described in conjunction with various embodiments and implementations as examples. However, those skilled in the art will be able to understand and implement other variations when implementing the claimed subject matter based on a study of the drawings, the invention and the independent claims. In the claims and in the specification, the word "comprising" does not exclude other elements or steps, and "a" or "an" does not exclude a plurality. A single element or other unit may fulfil the functions of several entities or items described in the claims. The enumeration of certain measures in mutually different dependent claims does not indicate that a combination of these measures cannot be used in an advantageous implementation.

Claims

1. An edge device (100) for a distributed traffic engineering system (200), characterized in that: The distributed traffic engineering system (200) has quality of service, QoS, control for a plurality of flow groups routed on a set of overlay links, wherein the edge device (100) is used to: receiving one or more service level agreement, SLA, requirements (101) for the plurality of flow groups from a network controller (201); Receiving first control information (102) from one or more other edge devices (202) in the distributed traffic engineering system (200); Acquire monitoring information of the plurality of flow groups and the overlay link set (103); Based on at least one of the first control information (102), the monitoring information (103) and the one or more SLA requirements (101), a plurality of queue parameters (104) of the plurality of flow groups are adjusted.

2. The edge device (100) according to claim 1, characterized in that: The edge device (100) comprises a control plane, Wherein, the control plane includes a QoS agent and / or a routing agent; and / or Wherein, the QoS agent includes a scheduling optimization module and / or an analysis module to support the generation of a help request (105); and / or The edge device (100) comprises a data plane, The data plane includes a scheduling engine, a monitoring engine and / or a forwarding engine.

3. The edge device (100) according to claim 1 or 2, characterized in that: The edge device is also used to: Determining whether a help request (105) is needed based on the monitoring information (103) and the one or more SLA requirements (101); If a help request (105) is required, the help request (105) is provided to the network controller (201).

4. The edge device (100) according to claim 3, characterized in that: The edge device is also used to: If at least one edge device (100) in the distributed traffic engineering system (200) provides a request for assistance (105) to the network controller (201), a routing policy is received from the network controller (201).

5. The edge device (100) according to any one of the preceding claims, characterized in that: The edge device (100) is also used for: Determining second control information based on the plurality of adjusted queue parameters (104) and / or the monitoring information (103); The second control information is provided to the one or more other edge devices (202).

6. The edge device (100) according to any one of the preceding claims, characterized in that: The edge device (100) is also configured to receive one or more endpoints of the help request (105) from the network controller (201).

7. The edge device (100) according to any one of the preceding claims, characterized in that: The edge device (100) is used for: The plurality of queue parameters (104) are further adjusted based on the SLA prediction model.

8. The edge device (100) according to any one of the preceding claims, characterized in that: The edge device (100) is used to periodically and / or repeatedly perform the following operations: Receiving first control information (102) from one or more other edge devices (202) in the distributed traffic engineering system (200); Obtaining monitoring information of the plurality of flow groups (103); Iteratively adjusting the plurality of queue parameters (104) of the plurality of flow groups based on the first control information (102) and / or the monitoring information (103); determining second control information based on the plurality of adjusted queue parameters (104); The second control information is provided to the one or more other edge devices (202).

9. The edge device (100) according to claim 8, characterized in that: The edge device is also used to periodically and / or repeatedly perform the following operations: Determining whether a help request (105) is needed based on the monitoring information (103) and the one or more SLA requirements (101); If a help request (105) is required, providing a help request (105) to the network controller (201); and / or If at least one edge device (100) in the distributed traffic engineering system (200) provides a request for assistance (105) to the network controller (201), the routing policy is received from the network controller (201).

10. The edge device (100) according to any one of the preceding claims, characterized in that: The second control information includes a load of each overlay link in the overlay link set and / or a scalar of each flow group in the plurality of flow groups, and / or The first control information (102) includes a load of each overlay link in a second set of overlay links corresponding to the one or more other edge devices (202) and / or a scalar for each flow group in a second plurality of flow groups corresponding to the one or more other edge devices (202).

11. The edge device (100) according to any one of the preceding claims, characterized in that: The plurality of queue parameters (104) include a shaping rate for each flow group and / or a weighted fair queue for each flow group, a WFQ weight, and / or The one or more SLA requirements (101) include at least one of throughput, delay, and jitter and loss requirements for each of the plurality of flow groups.

12. The edge device (100) according to any one of claims 2 to 11, characterized in that: The scheduling optimization module is used to provide the plurality of adjusted queue parameters (104) to the scheduling engine, and / or The scheduling engine is used to enforce a bandwidth sharing policy to ensure the SLA requirement (101) and / or fairness according to a fairness target.

13. The edge device (100) according to any one of claims 2 to 12, characterized in that: The QoS agent is used to obtain the monitoring information (103) from the monitoring engine, and / or The monitoring engine is configured to determine the monitoring information based on measurements of each of the plurality of flow groups (103).

14. The edge device (100) according to any one of claims 4 to 13, characterized in that: The routing agent is used to: If at least one edge device (100) in the distributed traffic engineering system (200) provides a help request (105) to the network controller (201), receiving the routing policy from the network controller (201) and / or receiving the help request (105) from the QoS agent; The routing policy is provided to the forwarding engine.

15. The edge device (100) according to any one of the preceding claims, characterized in that: The edge device (100) is used for forwarding traffic of each flow group in the plurality of flow groups on the set of overlay links.

16. The edge device (100) according to any one of claims 2 to 15, characterized in that: The analysis module is used to: receiving the monitoring information from the monitoring engine (103); and / or receiving the plurality of adjusted queue parameters from the scheduling optimization module (104); Determining whether a help request (105) is required based on the monitoring information (103) and the one or more SLA requirements (101) and / or the plurality of adjusted queue parameters (104); If a help request (105) is required, the help request (105) is provided to the network controller (201) and / or the routing agent.

17. The edge device (100) according to any one of claims 3 to 16, characterized in that: The edge device (100) is also used for: Further based on the SLA prediction model and / or by determining and / or predicting whether the one or more SLA requirements (101) are met, it is determined whether a help request is required (105).

18. The edge device (100) according to any one of the preceding claims, characterized in that: The edge device (100) is used for: determining a plurality of rate allocations, the plurality of rate allocations comprising a rate allocation on each overlay link in the set of overlay links for each flow group in the plurality of flow groups; adjusting the plurality of queue parameters (104) based on the plurality of rate allocations, and / or Wherein, determining the multiple rate allocations is based on: The first control information (102), The monitoring information (103), the one or more SLA requirements (101), The SLA prediction model is optional.

19. The edge device (100) according to any one of the preceding claims, characterized in that: The monitoring information (103) includes, for each flow group in the plurality of flow groups and / or for each overlay link in the overlay link set: Measured throughput, and / or QoS statistical information, wherein the QoS statistical information includes at least one of delay, jitter and loss.

20. The edge device (100) according to claim 19, characterized in that: The edge device (100) is used for: comparing the measured throughput of each of the plurality of flow groups with the one or more SLA requirements (101) and / or with the plurality of rate allocations; If the measured throughput is insufficient to meet at least one of the one or more SLA requirements (101), providing the help request (105) to at least one of the routing agent, the network controller (201), and the one or more endpoints of the help request (105).

21. The edge device (100) according to any one of the preceding claims, characterized in that: The edge device (100) is configured to adjust the plurality of queue parameters (104) using a class-based queuing (CBQ) architecture.

22. The edge device (100) according to any one of claims 18 to 21, characterized in that: The edge device (100) is used to determine the multiple rate allocations based on a QoS optimization model.

23. The edge device (100) according to claim 22, characterized in that: The QoS optimization model includes at least one of the following constraints: Satisfying the capacity of each overlay link in the overlay link set, rate allocation fairness goals, and / or The edge device (100) is used to determine SLA violations, rate allocations and / or rate violations, Furthermore, the QoS optimization model is based on The SLA is violated, The rate allocation and / or the rate violation.

24. The edge device (100) according to claim 23, characterized in that: The constraint satisfying the capacity of each overlay link in the set of overlay links is relaxed according to a Lagrangian relaxation model, The edge device (100) is used for: Using Lagrange multipliers Penalizes violations of relaxed constraints, which imposes a cost on constraint violations.

25. The edge device (100) according to any one of claims 22 to 24, characterized in that: The QoS optimization model includes at least one of the following inputs: One or more stream groups, one or more tunnels, where each tunnel is defined by an origin and a destination, the plurality of flow groups, and / or wherein each flow group is defined by a traffic class and its origin and destination, The traffic demand of each flow group on each overlay link, The SLA prediction model for each flow group on each overlay link, SLA requirements for each flow group, Penalties for demand violations for each flow group; and / or The QoS optimization model includes at least one of the following outputs: The rate allocation of each flow group on each overlay link, SLA violation per flow group on each overlay link, Rate violation per flow group on each overlay link, The intermediate variables of each flow group on each overlay link are used to optimize fairness according to the fairness target.

26. The edge device (100) according to any one of claims 12 to 25, characterized in that: The fairness objective is based on at least one of the following: minimum and / or maximum rate allocation for each of the plurality of flow groups, A proportional rate allocation constraint for each flow group in the plurality of flow groups.

27. A network controller (201) for a distributed traffic engineering system (200), characterized in that: The distributed traffic engineering system (200) has quality of service (QoS) control for multiple flow groups routed on a set of overlay links, wherein the network controller (201) is used to: Providing an SLA requirement (101) of each flow group in the plurality of flow groups to two or more edge devices (100, 202) of the distributed traffic engineering system (200); Receiving a help request (105) from at least one edge device (100) of the two or more edge devices (100, 202); A routing policy (203) is provided to at least one edge device (100) of the two or more edge devices (100, 202).

28. The edge device (100) according to any one of the preceding claims, characterized in that: The network controller (201) is also used for: One or more endpoints of a help request (105) are provided to the two or more edge devices (100, 202).

29. A method (300) of operating an edge device (100) for a distributed traffic engineering system (200), characterized in that: The distributed traffic engineering system (200) has quality of service, QoS, control for a plurality of flow groups routed on a set of overlay links, wherein the method (300) comprises: receiving (301) one or more service level agreement, SLA, requirements for the plurality of flow groups from a network controller (201); Receiving (302) first control information (102) from one or more other edge devices (202) in the distributed traffic engineering system (200); Acquiring (303) monitoring information (103) of the plurality of flow groups and the set of overlay links; Based on at least one of the first control information (102), the monitoring information (103) and the one or more SLA requirements (101), a plurality of queue parameters (104) of the plurality of flow groups are adjusted (304).

Images