Congestion Avoidance in Slice-Based Networks

Abstract

To provide a method for managing the latency in a slice-based network.SOLUTION: The method comprises: setting a flag in a packet header of a packet corresponding to a slice; receiving slice-specific packet timing information determined by an agent from multiple switches in a slice-based network; determining a latency value on the basis of the packet timing information from the multiple switches in a first slice path for the slice; selecting an alternative slice path for the slice on the basis of comparison of the latency value to a threshold, where the alternative slice path includes a third switch absent in the first slice path; and implementing the alternative slice path via at least the first switch and the third switch.SELECTED DRAWING: Figure 1B

Description

[Background technology]

[0001] Today's 3G, 4G, and LTE networks operate with multiple data centers ("DCs") that can be distributed across clouds. These networks are centrally managed by just a few Operational Support Systems ("OSSs") and Network Operations Centers ("NOCs"). 5G technology will vastly increase network connectivity for all kinds of devices that need to connect to Telco networks and share physical network resources. Current network architectures cannot scale to meet these demands.

[0002] Network slicing is a form of virtualization that allows multiple logical networks to operate on a shared physical network infrastructure. A distributed cloud network may share network resources with various slices to allow multiplexing of different users, called tenants, onto a single physical infrastructure. For example, Internet of Things ("IoT") devices, mobile broadband devices, and low-latency in-vehicle devices will all need to share a 5G network. These different use cases will have different transmission characteristics and requirements. For example, IoT typically has a large number of devices but very low throughput. Mobile broadband is the opposite, with each device sending and receiving high-bandwidth content. Network slicing may enable segmentation at the end-to-end level of the physical network to group traffic, isolate tenant traffic, and configure network resources at a macro level.

[0003] However, traditional approaches to quality of service ("QoS") and congestion avoidance do not easily scale to accommodate the significant increase in network usage enabled by slice-based networks. For example, when a network is congested, existing solutions throttle the source of congestion, effectively slowing data transmission to free up bandwidth. For example, existing flow control mechanisms, such as Priority Flow Control ("PFC"), limit congestion by notifying an upstream switch (e.g., a router) of the congestion and throttling traffic at the source of congestion, such as a host sending traffic through the switch. However, slice-based networks are inefficient because throttling can adversely affect other slices, causing the slice to perform less than possible under its software license agreement ("SLA"). A new slice-based solution is needed that does not violate SLA requirements.

[0004] Accurately monitoring key performance metrics such as latency and throughput is a particular challenge in slice-based networks. In traditional networks, latency can be calculated in software by sending probe packets such as pings between any two endpoints. However, software-based monitoring probes do not scale in large Telco networks with tens of millions of users. Detecting congestion at the virtual layer, such as with slice Virtual Network Functions (VNFs), is typically not scalable and quickly overwhelms the underlying physical hardware.

[0005] Without another way to mitigate congestion, critical slices could be adversely affected if the network infrastructure becomes overloaded. SLAs often require reliable transmission of critical traffic, especially 911 calls and communications with autonomous vehicles.

[0006] As a result, there is a need for a system that alleviates congestion in slice-based networks. Summary of the Invention

[0007] Examples described herein include systems and methods for alleviating congestion in slice-based networks. The system may include a monitoring module that communicates with agents on physical devices such as routers or servers. The network can be partitioned into slices to accommodate different use cases for one or more tenants. Each slice may have required SLA performance attributes such as latency, bandwidth, round trip time, and other threshold levels.

[0008] Each network slice in a Virtual Service Network ("VSN") can carry data from multiple terminal devices such as phones, vehicles, and IoT devices. For example, a 911 slice can carry calls from thousands of mobile devices at once. The same case can exist with a YOUTUBE slice, where hundreds of thousands of mobile users can watch YOUTUBE videos simultaneously. Each slice can have different VNFs for different specialized functions within the slice. However, monitoring some performance metrics by VNFs is not scalable.

[0009] In one example, some monitoring tasks are offloaded to the physical layer by running an agent on the underlying switch. The agent can analyze the traffic and report telematics data back to the monitoring module. In one example, the monitoring module can provide the agent to the switch, which can be programmable. In one example, the switch includes a programmable networking application specific integrated circuit ("ASIC"), such as a TOFINO chip, that can be programmed using P4 language or some other protocol.

[0010] A switch can be programmed to run an agent at the physical layer instead of the virtual layer. The agent on the switch can report telematics data to the monitoring module. The telematics data can be slice specific to indicate some performance characteristic of the slice. Each switch can report telematics data for the slices it handles. The telematics data can relate to any performance metric of the slice such as latency, bandwidth, and round trip time.

[0011] The monitoring module can track compiled telematics data for switches throughout the slice-based network by maintaining or retrieving a network connectivity graph. The monitoring module can also determine that a slice is not meeting SLA requirements based on one or more switches in the network connectivity graph. This determination can be based on telematics data for the entire slice path of the slice. In one example, the telematics data, also referred to as performance data, can be compiled over time, such as averaged, and compared to SLA thresholds.

[0012] Based on the SLA not being met, the monitoring module can reduce congestion by selecting an alternative slice path. For example, the network connectivity graph can indicate another possible route that has a better performance criterion. The alternative slice path can include at least one switch that is not in the existing slice path for the slice. By implementing the selected alternative slice path, the monitoring module can change the routing such that a first switch sends traffic to a third switch instead of a second switch in the original slice path. This can reduce congestion by distributing slice traffic across the network instead of simply routing all slices the same path.

[0013] In one example, an alternative slice path can be selected based on a latency value that is below an SLA requirement. For example, the monitoring module can receive packet timing information from multiple physical switches for multiple slices. An agent operating in the physical domain of the switches can send the packet timing information. The monitoring module can determine a latency value based on the packet timing information, such as by averaging multiple different instances of the packet timing information received from the switches. Based on a comparison of the latency value to a threshold, the monitoring module can select an alternative slice path for the slice, where the alternative slice path includes a switch that is not in the current slice path for the slice. The monitoring module can then implement the alternative slice path, which can include updating the switch with the new slice path so that the next hop is correct.

[0014] In one example, the alternative slice path can be selected based on a throughput below the SLA requirement. The monitoring module can receive data rate information from a switch in the slice path of the first slice. The data rate information can correspond to a slice identifier that enables the monitoring module to correlate the data rate with a particular slice. The monitoring module can determine an aggregate throughput for the first slice based on the received data rate information. Based on a comparison of the aggregate throughput to a threshold, the monitoring module can implement the alternative slice path for the first slice.

[0015] These stages may be performed in some examples by a monitoring module executing as part of an orchestration system. Alternatively, a non-transitory computer readable medium containing instructions may cause a processor to perform the stages when the processor executes the instructions. The term "switch" may broadly refer to any device that performs a network function, such as a server or a router.

[0016] Both the foregoing general description and the following detailed description are exemplary and explanatory only and are not limiting as set forth in the appended claims. [Brief description of the drawings]

[0017] [Figure 1A] 1 is a flowchart of an exemplary method for reducing congestion of a slice in a VSN. [Figure 1B] 1 is a flowchart of an example method for suppressing congestion based on slice latency in a VSN. [Figure 1C] 1 is a flowchart of an example method for reducing congestion based on slice throughput in a VSN. [Figure 2A] FIG. 13 is an exemplary sequence diagram for suppressing slice congestion in a VSN. [Figure 2B] 1 is an exemplary sequence diagram for suppressing slice congestion in a VSN based on latency. [Figure 2C] 1 is an exemplary sequence diagram for suppressing slice congestion in a VSN based on throughput. [Diagram 3] 1 is an exemplary flowchart of congestion suppression in a slice-based network. [Figure 4A] FIG. 1 is an exemplary system diagram illustrating multiple slices in a VSN. [Figure 4B] FIG. 1 is an exemplary system diagram illustrating multiple slices in a VSN. [Diagram 5] 1 is an exemplary flow chart for calculating slice throughput. [Figure 6] FIG. 1 is an exemplary system diagram of a topology of a VSN. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Reference will now be made in detail to the present examples, including examples illustrated in the accompanying drawings, in which, wherever possible, the same reference numbers are used throughout the drawings to refer to the same or like parts. In one example, a system includes a monitoring module that receives telematics data, such as performance data related to latency or throughput, from multiple switches in a slice-based network. Agents running at a physical layer on the switches can report the telematics data. The monitoring module can track performance over a network connectivity graph and determine when SLA requirements are not met for one or more slices.

[0019] Based on slice performance to the switches in the network connectivity graph, the monitoring module can select a new slice path for the slice to make the slice SLA compliant again. A slice can include elements in both the virtual and physical overlays of the network and potentially span multiple clouds. Slicing allows a provider to segment a Telco network for use by multiple tenants, each of which may have its own slice governed by SLA requirements.

[0020] A slice may span one or more clusters or clouds in a VSN. An orchestrator can manage multiple slices in a network that can be used for a specific purpose or rented to a tenant. For example, a slice can be reserved for a specific application, IoT device, or customer. Each slice includes a virtual network that runs on a shared physical network infrastructure distributed across one or more Telco clouds. A slice can include a service chain of VNFs that perform a specific network task. The required combination of VNFs can vary based on the intended use of the slice, such as video streaming or IoT device management. Slices can be reserved for specific types of traffic and prioritized for QoS purposes, etc. SLAs can specify the required performance criteria for a slice and can specify the required performance criteria for different slices. The performance criteria can vary depending on the intended use of a given slice. Also, the VNFs that make up the service chain can be specified by the SLA or by separate slice records.

[0021] To instantiate a slice, a VNF can be placed across a slice path. A slice path can represent a subset of a provider's distributed network and can span one or more switches. Different slices can route traffic differently, even if the traffic is heading to the same destination. Because physical switches and VNFs can span different areas in the network, slices can take different paths to their destination and the workload can be effectively distributed across the network.

[0022] The physical switch can send the telematics data to a monitoring module, which can be part of an orchestrator or a separate process. In one example, the orchestrator and the monitoring module can be hosted on a managed cluster of servers. In one example, the physical switch is programmed to run an agent that monitors packets at the hardware level. For example, the agent can run in the core of the switch, not in the virtual layer where the VNFs run. The agent can cause the switch to route traffic based on the slice. The agent can also report the telematics data to the monitoring module. In one example, the switch can use a programmable networking ASIC, such as a Tofino chip, that can be programmed using P4 language or some other protocol. In one example, the monitoring module can program the switch to use the agent.

[0023] The monitoring module can ensure that the network is functioning as expected for various slices. The monitoring module can also provide slice paths to the switch that allow the switch to send packets to different next hops based on the slice, which may include another switch in the slice path. This allows for preferred slices to be set across low-congestion routes in the network. Based on packet priority in slice path selection within the switch and across the network, congestion can be reduced.

[0024] 1 is an exemplary method including stages performed by a monitoring module. In stage 105, the monitoring module can receive slice-specific telematics data from a plurality of physical switches. The telematics data can include any performance data related to slices routed to the switch. In one example, the switch can be programmed to execute an agent that communicates with the monitoring module. The agent can periodically report or be periodically polled by the monitoring module for the slice-specific telematics data. The telematics data can include performance attributes such as switch latency, slice latency, total round trip time, slice bandwidth, and switch bandwidth.

[0025] At stage 110, the monitoring module can determine if the performance fails to meet the SLA requirements of the slice. The service provider's tenants can negotiate to exchange specific performance requirements outlined in the SLA. The service provider needs to ensure that these SLA requirements are met, especially since the SLA requirements may be the basis of a contractual relationship with the tenants.

[0026] In one example, the monitoring module can aggregate the telematics data by slice and then compare the data to slice-specific SLA requirements to ensure that SLA requirements are met. In one example, the telematics data can include a slice identifier, allowing the monitoring module to perform per-slice analysis. Also, in one example, the monitoring module can maintain a graph of switches in the physical layer that correspond to the slice. This can enable the monitoring module to determine the telematics data to use to determine the performance of the slice.

[0027] As an example, a first slice may have SLA requirements for latency and bandwidth. The monitoring module may collect performance metrics from a switch associated with the first slice, which may relate to latency, bandwidth, or both. The monitoring module may determine that the SLA is not being met if the performance metrics fall below the SLA requirements for a period of time.

[0028] As a result, in stage 115, the monitoring module can select an alternative slice path for the slice. The alternative slice path can include a new path through the physical layer, such as a new switch that is not part of the existing slice path. The monitoring module can select the alternative slice path based on a network connectivity graph. This graph can include telematics data of other switches that indicate more favorable performance with respect to SLA requirements that the slice does not currently meet. For example, telematics data from the new switch can indicate that latency and bandwidth will be improved if the slice is routed to the new switch instead of an existing switch in the slice path.

[0029] At stage 120, the monitoring module can implement an alternative slice path. This can include sending a message to the first switch to update slice path information at the first switch. The original slice path information can indicate a second switch as the next hop for the slice, while the updated slice path information can indicate a third switch as the next hop. The monitoring module or some other orchestration process can also instantiate one or more VNFs at the third switch as needed. Because slices can operate at both the physical layer and the virtual layer, the monitoring module can coordinate slice path switching at the physical layer with VNF availability at the virtual layer.

[0030] Slice routing changes can reduce network congestion. The monitoring module can help distribute slices across available parts of the physical network. The switches used can span one or more clusters or clouds to reduce the load on overloaded switches.

[0031] FIG. 1B illustrates an example method for suppressing congestion based on latency requirements. Latency can be an important SLA requirement for various customers. For example, a slice of 911 calls needs to have as low latency as possible. If the monitoring module detects a large latency time for that slice, in one example, the monitoring module can analyze the VSN topology to calculate an alternative low latency path.

[0032] At stage 125, the monitoring module can receive packet timing information from multiple switches. The packet timing information can be slice specific. In one example, the timing information is a timestamp collected by an agent when a packet makes a hop. In one example, the switch can track the request-response time of a packet at wire speed. A packet can be tracked across multiple switches, each reporting timing information for use in determining latency.

[0033] Table 1 below shows exemplary variations of requests and responses for various packet types.

[0034] [Table 1] Internet Control Message Protocol ("ICMP") and Address Resolution Protocol ("ARP") packets can be tracked based on the protocol headers. In one example, to track Transmission Control Protocol ("TCP") packets, a switch can set a flag in the packet, such as an urgent flag, and track acknowledgments from the packet.

[0035] Each switch can determine timing information for various unique flows corresponding to different slices. For example, a slice may have multiple different ingress points for different flows. YouTube traffic may join a slice, for example, in both San Francisco and Miami, resulting in two different flows. These flows can be determined based on packet headers. For example, the packet headers may include a source Internet Protocol ("IP") address, a destination IP address, a source port, and a destination port. The switches can use this information to identify unique flows corresponding to the slices. If packets of a flow are tracked, the timing information reported by each switch can be used by the monitoring module to determine latency.

[0036] The pseudocode in Table 2 below is one example of how an agent on a switch may determine timing information for a packet.

[0037] [Table 2] As shown in Table 2, for an ICMP or ARP packet, the request may cause an agent on the switch to store the current time of the packet. When a response comes back, the switch can capture that time as well. The switch can then calculate the time difference between the request and the response and return the result as the RTT for the slice ID.

[0038] The way the switch calculates the RTT may be different for various packet types. More specifically, for each packet P, the protocol type ("protoType") is checked. For ICMP, if the packet is part of a request (e.g., if "isRequest" is true), the switch may store the slice ID, source IP address ("sip"), destination IP address ("dip"), and current time. If the ICMP packet is a return, the switch may determine the difference between the current time and a previously stored time. This value may represent the RTT, and may be updated for the slice ID, sip, and dip. In one example, the RTT value may be sent to a monitoring module.

[0039] For an ARP packet, if a request is recognized (e.g., "isRequest" is true), the current time of the packet is stored in association with the slice ID and source MAC address ("smac"). If the ARP packet is not a request, this may mean that it is a response. In this case, the response may cause the agent to update the stored information with the round trip time ("RTT"). This can be reported to the monitoring module as the timing information of the slice.

[0040] For TCP packets, the agent can calculate the RTT based on the time difference between the forward flow packets and the return packets. For example, a forward flow may be created when a user opens a video and the flow extends toward the destination where the video was accessed. The video may then return from the destination, which includes the return packets.

[0041] In more detail, the agent may first check a table of recent flows to try to determine if the packet corresponds to a forward flow ("forFlow"). This table can store recent flows based on source IP address ("sip"), destination IP address ("dip"), source port ("sip"), and destination port ("dport"). Based on this information, the agent can check the flow table to make sure the flow has already been identified and stored with a timestamp from the forward flow. If a forward flow is recognized (e.g., "forFLow is enabled"), the agent can obtain the timestamp of the forward flow ("forFLow->timeStamp"). The agent can also obtain the current time ("getTime()") and subtract the timestamp to determine the RTT. The agent can use this value to update the RTT of the slice (e.g., "updateSliceRtt(sliceID,rtt)").

[0042] The agent can then clean up the forwarded flow so that a new timestamp is available for determining the RTT in the future. In one example, this can include removing the forwarded flow ("forFlow") from the flow table. In another example, a value is set in the flow table indicating that the RTT has already been calculated for the forwarded flow. This allows "forFlow" to be disabled in future lookups of that flow, allowing another timestamp to be generated for that flow in the flow table if the RTT is measured in the future.

[0043] If the monitoring module wants to measure RTT using TCP packets, it can set flags in the packet headers that are recognized by the agent on the switch. The agent can record forwarding flows based on these flags set in the TCP packets, for example. In one example, this can enable periodic RTT testing. As shown in the pseudo code in Table 1, if a forwarding flow is not valid, packet P is checked to make sure that a particular flag is not equal to zero. For example, an application can set an urgent flag ("URG"), a SYN flag, or a PSH flag. Using these flags, a new flow can be identified and added to the flow table for the purpose of calculating the RTT.

[0044] The URG flag in the TCP header may indicate that the packet needs to be delivered without delay. The SYN flag may be set when a TCP connection is initiated between a client and a server. The PSH flag may instruct the client and server to push buffered data to the application.

[0045] The switch can track these flags for purposes of calculating RTT. For example, a receiver can send a response immediately if any of these flags are set. If none of these flags are present in a TCP packet, the receiver may buffer the arriving data and respond later, but such a packet will not be eligible for RTT calculations. For example, if a sender sends a TCP packet one byte at a time, instead of immediately sending the one byte to the application, the receiver's TCP stack may buffer the data. The receiver may wait for a few more packets to arrive before notifying the application of the larger chunk of data. Thus, if data is buffered, there is a possibility of a delayed response. As a result, in one example, buffered packets cannot be used to accurately calculate latency because the buffering would skew the results. By checking the URG, SYN, and PSH flags, the system can keep buffering-related delays out of the latency calculation.

[0046] In Table 1, "createFlow" allows the storage of a new flow based on the dip, sip, dport, and sport of the packet. This new flow ("revFlow") can be given a timestamp based on a call such as getTime() that retrieves the current time. If a return packet is then received at the switch, the flow ("forFlow") can be obtained using the same packet information (sip, dip, sport, dport). This flow can be found as valid and the timestamp is subtracted from the current time to determine the RTT.

[0047] In one example, at each hop, each switch records this timing information and reports it to a monitoring module for use in determining latency. At stage 130, the monitoring module can determine a latency value. In one example, this can include summing timing information of packets from each hop on the slice to determine the slice RTT. In another example, the monitoring module averages or aggregates the RTT information for the slice over a period of time. In one example, this aggregate value can be used as the slice latency value.

[0048] At stage 135, the monitoring module can compare the latency value to the SLA threshold. The monitoring module can perform a comparison on slice-specific latency by comparing the latency value against the slice-specific threshold of the SLA. In one example, if the latency falls below the threshold, the monitoring module can first verify that the latency value continues to fall below the threshold for a period of time. This can help to verify that the slice path truly needs to change and prevent the slice path from constantly changing. However, if the SLA threshold continues to not be met for the threshold period, the monitoring module acts to change the slice path.

[0049] If the SLA latency requirements are not met, the monitoring module can select an alternative slice path for the slice. This can include referencing a network connectivity graph to determine another route with better performance characteristics. In one example, the network connectivity graph can store information corresponding to other switches reported from the physical or virtual layer of the network. The alternative path selected by the monitoring module can include switches that were not part of the original slice path, thus requiring a different physical and virtual path through the slice-based network.

[0050] At stage 140, the monitoring module can implement an alternative slice path through the new switch, which may include updating slice path information in the existing switch to route slice traffic to the third switch instead of the second switch, and in one example, instantiating an appropriate VNF in a virtual layer corresponding to the third switch.

[0051] Figure 1C illustrates an exemplary method for mitigating congestion based on throughput requirements, which may be important SLA requirements for various customers. Traditional approaches to calculating throughput include periodically sending large amounts of data through the network. However, this software-based approach does not scale in large VSNs that Telco providers may distribute across the country. Instead, physical network switches can collect telematics data, and a monitoring module can aggregate this data to calculate the overall slice throughput.

[0052] At stage 145, the orchestrator or monitoring module can provide the agent to a programmable switch in the VSN. This can include having the switch execute the agent in the switch hardware, such as in the core of the switch. The switch executing the agent can then calculate data rate information for multiple slices. In one example, the switch can maintain data rate information in the following format: Slice ID, Smac, Dmac, SIP, DIP, SPort, DPort, Pkt Rate. Slice ID can indicate the slice. Dmac can indicate the destination MAC address while Smac can indicate the source MAC address. SIP and DIP can correspond to the source IP address and destination IP address, respectively. SPort and DPort can correspond to the source and destination ports, and Pkt Rate can indicate the packet rate.

[0053] At stage 150, the monitoring module can receive data rate information from multiple switches, including the switches in the current slice path of the first slice. The data rate information can identify slices and packet rates. In one example, the monitoring module periodically collects this information by sending messages to the switches. In another example, the switches periodically query the monitoring module without having to receive a request from the monitoring module.

[0054] At stage 155, the monitoring module can determine the aggregate throughput of the slice and other slices in the VSN. This can include using the network topology and slice paths to filter out duplicate information. For example, if the switch is not the starting node of the flow, it may include duplicate information. The monitoring module can loop through the flows to update the slice throughput based on rate information from the first switch for each flow, as described in more detail below with respect to FIG. 5. In one example, other rate information can be ignored.

[0055] In stage 160, the monitoring module can implement a new slice path for the slice based on a comparison of the aggregate throughput to a threshold, such as a slice throughput threshold in the SLA. Similar to latency, the monitoring module can, in one example, determine that the throughput does not comply with the SLA for a period of time before changing the slice path. The orchestrator or monitoring module can select and implement an alternative slice path, as described above. The monitoring module can obtain a path that includes switches that do not experience low throughput. This can allow the slices to be distributed across the VSN to maintain SLA compliance.

[0056] FIG. 2A is an example sequence diagram for suppressing congestion based on telematics data. In stage 205, a monitoring module can program multiple switches S1, S2, S3 of a slice-based network. This can include remotely querying interfaces on the switches and sending packages to run on the switches. The packages can include agents. In one example, a P4 language script is used to launch the agents on the switches. Also, information such as slice paths can be sent to the switches for use in routing.

[0057] In stage 210, an agent can be executed on the switch. The agent can be used to collect telematics data, such as latency or data rate information, for each slice. The agent 210 can further include logic to look up the slice path and determine the next hop for the packet based on the slice ID. In this example, a slice can have a current slice path that includes switches S1 and S2. For that slice, the next hop from S1 includes S2.

[0058] At stage 220, the switches can periodically send telematics data to the monitoring module. In one example, the switches determine when to send the telematics data based on the agent. In another example, the monitoring module requests telematics data on an as-needed basis by querying individual switches.

[0059] At stage 225, the monitoring module can aggregate the telematics data for each slice and determine that the slice is not meeting one or more performance requirements of the SLA. This can be based on any performance criteria, of which latency and throughput are just two examples. The monitoring module can then determine an alternative slice path. This can include analysis of the network topology graph to reveal other underloaded switches. In this example, the third switch S3 can be less congested than the second switch S2.

[0060] In response, at stages 230 and 235, the monitoring module can implement a new slice path for the slice by modifying the path to include the third switch S3 and exclude the second switch S2. At stage 230, the monitoring module can verify that the appropriate VNFs for the slice are running in the virtual layer corresponding to the third switch S3. When the slice path is modified, the VNFs in the virtual layer corresponding to the second switch S2 are no longer used by the slice. In one example, placement of VNFs is handled by the orchestrator or some orchestrator process other than the monitoring module.

[0061] At stage 235, the monitoring module can inform the first switch S1 of the new slice path. In one example, this can be done after the VNFs are ready at stage 230. This can cause the switch S1 to change the next hop to S3 for that particular slice, resulting in a new slice path at stage 240. This can reduce congestion across the slice-based network.

[0062] 2B is an example sequence diagram for suppressing congestion based on slice latency. At stage 242, the monitoring module can receive timing information from switches in a slice-based network. As described above, agents on the switches can communicate this information to the monitoring module. At stage 244, a first slice path can include switches S1 and S2. Both of these switches can report timing information at stage 242.

[0063] At stage 246, the monitoring module can determine a latency value for a first slice based on timing information received from switches S1 and S2. At stage 248, the monitoring module can determine a latency value for a different slice utilizing switch S3 based on timing information from switch S3.

[0064] At stage 250, a comparison with an SLA threshold may reveal that the latency is too high in the first slice. In response, at stage 256, the monitoring module may change the slice path of the first slice to the second slice path. To do this, at stage 252, one or more required VNFs may be instantiated in the virtual layer of switch S3. Also, configuration information may be sent to switch S1 to change the next hop of the first slice to switch S3. This may relieve congestion at switch S2 since the first slice will no longer use it.

[0065] 2C is an example sequence diagram of suppressing congestion based on slice throughput. At stage 265, a first slice may have a current slice path that includes switches S1 and S2. An agent on the switch may collect data rate information for each slice. At stage 260, the switch may report slice-specific data rate information to a monitoring module.

[0066] At stage 270, the monitoring module can calculate slice throughput. The monitoring module can do this for the first slice and other slices in the network. This can include ignoring overlapping data rate information at stage 275. In this example, switch S2 reports overlapping data rate information for the first slice that has a flow that originates at switch S1 but not at switch S2. In some examples, only a portion of the data rate information provided by switch S2 overlaps with the data rate information received from switch S1, such as when switch S2 receives data from multiple switches including S1. In these examples, stage 275 can include ignoring only the overlapping data rate information that has been considered based on the data rate information associated with switch S1. A detailed description and related discussion are provided in FIG. 5.

[0067] At stage 280, the monitoring module can determine that the throughput of the first slice is below the SLA requirement. In response, at stage 285, a new slice path for the first slice can be implemented, which can improve throughput by using switch S3 instead of switch S2.

[0068] FIG. 3 is an example flow chart of congestion suppression in a slice-based network. In one example, each stage can be performed by a monitoring module. In stage 310, the VSN monitor receives telematics data from network elements. This can include receiving any performance data from any switches in the network, such as routers, servers, and hosts. The VSN monitor can include one or more processes running on a physical server. The server can be part of a management cluster that manages various operations on the slice-based network. In one example, the VSN monitor includes a virtual analysis engine, such as VMware®'s vRealize®, that monitors how the VNFs (e.g., virtual machines) in the VSN are operating. The VNFs can represent virtual controllers, virtual routers, virtual interfaces, virtual local area networks ("VLANs"), host virtual machines ("VMs"), or other virtualized network functions that run on physical hardware, such as servers connected by switches. In one example, the VSN monitor can also include a physical analysis engine that can act as a physical underlay to analyze the performance of hardware, such as switches, in the network. It is also possible for the monitoring module to include a physical analysis engine. In another example, the monitoring module may include the entire analysis engine.

[0069] The physical analysis engine may include a congestion check process that utilizes performance information (e.g., telematics data) from the physical layer. At stage 320, the congestion check process may determine if any of the telematics data indicates congestion. The congestion may be based on the performance data not meeting a threshold. For example, the bandwidth, RTT, latency, or throughput of a particular slice may fall below an SLA threshold.

[0070] At stage 330, the congestion removal process can access the fabric topology of the slice-based network to determine alternative slice paths. The fabric topology can include a graph of other available switches and associated performance metrics. The VSN hardware can report which VNFs are running on which devices and which switches are in communication with each other. With the discovery of both hardware and virtual components, the system can map them together to generate the fabric topology.

[0071] Congestion removal can determine one or more alternative slice paths based on the rest of the fabric topology and its current performance. At the physical layer, performance can be reported by programmable switches running agents. Switches can report performance information such as data rate, throughput, latency, and bandwidth. This performance information can be used to detect congestion.

[0072] Congestion avoidance based on performance data from the physical layer can be one reason for changing slice paths. Issues in the virtual layer can also contribute to this decision. Virtual components can report performance separately to the virtual analysis engine. In one example, analysis of both the virtual and physical layers determines when to change slice paths.

[0073] In one example, an orchestration process can manage a controller hierarchy. The controller hierarchy configures various entities in one or more datacenters to realize a virtual service network. A high-level VSN controller can coordinate a set of other controllers that configure entities in the datacenters where the VSN is implemented. In some embodiments, each datacenter has its own set of low-level controllers. These controllers can include compute controllers (e.g., that configure VMs that implement VNFs), network controllers (e.g., that configure forwarding switches to transmit data messages between slice selectors and network services), storage controllers, and software-defined networking ("SDN") controllers (e.g., that configure slice selectors and / or gateways that transmit data messages between datacenters).

[0074] At stage 340, the process can set up an alternative slice path in the network. The VSN controller hierarchy can work together to implement the new slice path. This can include instantiating VNFs at the virtual layer and querying new switches at the virtual or physical layer by reconfiguring the physical or virtual switches. The orchestrator process can manage the VNF instantiation based on the network topology. The monitoring module can send messages to one or more switches to change the slice path. The new slice path can then be monitored by future telematics data.

[0075] 4A is an example system diagram illustrating multiple slices in a VSN. Multiple terminal devices 410 (phones in this example) can communicate with the VSN 405 by transmitting and receiving network data at cell towers in San Francisco 420.

[0076] The cell tower can be communicatively coupled to a slice selector 425 that determines the correct slice for a packet from the terminal device 410. This can be done based on packet information such as packet type, source and destination IP addresses, source and destination ports, and source and destination MAC addresses. In one example, the slice selector 425 first processes and assigns the packet to one of the network slices of the VSN. The slice selector 425 can also handle service chaining operations so that the packet is processed by the correct set of network services for the assigned slice. In various examples, the slice selector 425 can be implemented by, among other things, a VM, a software forwarding element (e.g., a flow-based forwarding element) operating within the VM or within the virtualization software of the host computer, or a set of modules that run outside of the forwarding element (e.g., between the VM and a port of the forwarding element) within the virtualization software of the host computer.

[0077] In some cases, multiple slice selectors 425 are configured for a VSN. In a communications service provider example, a network slice selector can be configured for each cell tower, base station, or other aspect of an access network. The communications service provider's access network can include an edge cloud for each cell tower, and at least one slice selector 425 can be configured at each such edge cloud. In other examples (e.g., for SD-WAN traffic contained entirely within a set of connected data centers), a distributed network slice selector is configured such that network slice selection for a data message sent from a VM occurs on the same host computer as the source of the data message (albeit outside the source VM).

[0078] In this example, there is a first slice 430 in the VSN 405 for YOUTUBE streaming and a second slice 435 for 911 calls. Each of these slices 430, 435 may have different SLA requirements and therefore may span one or more switches in the VSN. In one example, these switches may span multiple clouds across the Internet. These same slices 430, 435 may also terminate in New York City 450. A different slice selector 440 at that point can route network traffic to the slice-based network.

[0079] 4B is an example system diagram showing multiple slices in a VSN. A first slice 460 may span switches R1, R3, R5, and R6. A second slice 465 may span switches R1, R2, R4, and R6. Both slices 460, 465 span from San Francisco 420 to New York City 450. The switches in this example include routers. Each of the switches calculates packet rate and timing information for each slice 460, 465.

[0080] In one example, the second slice initially has a slice path from San Francisco 420 to New York City 450. The slice path may be changed to use switches R2 and R4 instead of switches R3 and R5 based on congestion. Thus, the monitoring module can update the routing table in switch R1 to change the next hop to switch R2 instead of switch R3. Also, the routing tables in switches R2, R3, R4, and R5 can be updated to reflect the new slice path.

[0081] The monitoring module can use telematics data, such as packet rate and timing information, from the various switches mentioned above to detect congestion. Using this data, the monitoring module can attempt to aggregate the throughput of each slice 460, 465. In one example, only slice throughputs require non-overlapping data rate information to calculate the total slice throughput. For example, switches R3 and R5 may have overlapping packet rate data for the first slice 460 from San Francisco 420 to New York 450, but switch R5 will have non-overlapping associated packet rate data for the first slice 460 from Miami 470 to New York 450. Thus, in one example, data rate information from switch R3 is ignored, while data for Miami 470 at switch R5 is included in the calculation of throughput for the first slice 460. This is because the first slice 460 has two different throughput sources, first and second, in the same slice, which are flows from San Francisco 420 and Miami 470, respectively. Meanwhile, switch R3 only includes overlapping flow information from R1.

[0082] FIG. 5 is an example flow chart for calculating slice throughput and determining how to remove data from duplicate flows. Other methods of calculating throughput are possible, and FIG. 5 is merely one example approach. At stage 505, the monitoring module begins aggregating the throughput of the slice as described above. In one example, the calculation of slice throughput requires only data rate information from the first switch for each flow in the slice. Thus, in one example, the monitoring module can analyze which flows are starting at which switches in the slice path and calculate the throughput based on the data rate information from these switches.

[0083] At stage 510, the monitoring module collects telematics data from the next switch S in the slice path of the slice, which may include collecting packet rate data. At stage 515, the monitoring module obtains packet rate data for the next flow F.

[0084] If this switch is the first switch in the path of the flow, then the data rate information is relevant to the slice throughput determination, at stage 520. As a result, at stage 525, the monitoring module can update the slice throughput to include the data rate information of the flow.

[0085] If the switch is not the first path of the flow, the data rate information may be ignored for determining the slice throughput. Instead, the algorithm may again get the next flow F at stage 515 and repeat the process until there are no more flows to check at the switch. For FIG. 4B, switch R5 was associated with the throughput of the first slice for this reason, but R3 was not. In switch R3, there is a flow from San Francisco, but the flow originates at switch R1. In contrast, in switch R5, the first flow originates at switch R1 while the second flow originates at switch R5.

[0086] At stage 530, the monitoring module determines whether there are any outstanding flows. The monitoring module can do this based on access to the fabric topology, which can include a graph, which in one example, can represent the various flows based on the flow paths and physical switches. If there are outstanding flows, the algorithm can return to stage 515 to obtain rate statistics for the next flow.

[0087] If not, in stage 535, the monitoring module can determine whether there is another switch. If so, the algorithm can start again at stage 510. The next switch can be optionally identified as the first switch in any flow path and included in the slice throughput. However, if there are no more switches, the computation of the slice throughput is complete in stage 540.

[0088] 6 is an example diagram of system components in a VSN 600. The VSN 600 includes a distributed Telco cloud network that includes one or more clouds 620, 640. Slices 672, 678, 682 can be distributed across these clouds 620, 640.

[0089] Each cloud 620, 640 may have physical and virtual infrastructure for network function virtualization ("NFV") 642. For example, physical switches 644 such as routers and servers may run VMs 646 or microservices that provide VNF functionality. A slice may include a first VNF ​​running on the edge cloud 620. A VNF may utilize one or more vCPUs, in one example one or more VMs 624. However, the edge cloud 620 may run many VNFs, often for multiple tenants where the VNFs are part of different slices. Slices may keep separation from a functional perspective, with VNFs from different slices being unaware of each other's existence, even if they rely on VMs 624 running on shared physical hardware 622.

[0090] A first VNF ​​in a slice path can communicate with a second VNF that can be located in a different cloud 640. For example, the second VNF can include one or more VMs 646 running on physical hardware 644 of the core cloud 640. The second VNF can also communicate with yet another VNF in the slice path. In one example, one or more of these VNFs can act as an output to the Internet 660.

[0091] One or more user devices 602 can connect to a slice of the VSN 600, for example, using a 5G data connection. The user devices 602 can include any physical processor-enabled device that can connect to a Telco network. Examples include vehicles, phones, laptops, tablets, IoT devices, virtual reality devices, etc. A transceiver, such as a cell tower 605, can transmit to and receive from these user devices 602. At the input to the edge cloud 620, a slice selector 608 can receive data sent from the user devices 602 and determine a slice to apply. The slice selector 608 can operate as a VM 624 in the edge cloud or on a different hardware connected to the edge cloud 620. In one example, the slice selector can use information in a packet header to determine which slice a packet belongs to.

[0092] To manage a distributed virtual infrastructure, a provider may run a topology 665 of management processes, including an orchestrator 668 with monitoring modules. The orchestrator 668 can in turn communicate with monitoring modules running separately on different servers or in different virtual environments. In this example, the monitoring modules are part of the topology 665 that work with the orchestrator 668. An example framework for these processes is VCLOUD NFV by VMWARE, which can use VSPHERE for network virtualization and VREALIZE for virtual analytics. An exemplary orchestrator is CLOUDIFY.

[0093] In one example, the orchestrator may be responsible for managing slices and VNFs. This may include provisioning new slices or re-provisioning existing slices based on performance criteria and network load. The orchestrator may run on one or more physical servers located in one or more core clouds 620, 640 or separately from the clouds. The orchestrator 668 may provide tools to keep track of which clouds and VNFs are included in each slice. The orchestrator may also track slice performance for individual tenants 670, 680 and provide a management console. The orchestrator 668 may also receive performance criteria and load information to determine when the monitoring module should find new slice paths.

[0094] In this example, a first tenant 670 has multiple slices 672, 674. Each slice 672, 678 can be defined by a slice record that indicates the VNF requirements for that slice. Each VNF 674, 676 can provide a different function in a service chain.

[0095] The SLA may also specify various threshold performance requirements for the slices. These performance requirements may include latency, round trip time, bandwidth, etc. In one example, these may serve as per-slice QoS requirements.

[0096] The orchestrator 668 can rely on the monitoring module to receive telematics data from the switches 622, 644 and determine if the SLAs are being met. In one example, the monitoring module provides the switches 622, 644 with an agent 601. The switches 622, 644 can be programmed to run the agent 601. The monitoring module can also provide policing algorithms that the switches use to move packets from input ports 603 to output ports 606 and from output ports 606 to next hops in the network 600. The monitoring module can also provide slice path information that the switches 622, 644 use to determine next hops and output interfaces (e.g., ports) to use for those next hops.

[0097] The orchestrator 668 may also modify the settings of the slice selector 608 and the switches 622, 644 to correctly route traffic through the slice path. This may include modifying tables to which the devices compare packet information. For example, slice selection may be based on information in the packet header of the packet. For example, the switch or slice selector may use a combination of Layer 2-Layer 4 (L2-L4) headers or may perform deep packet inspection (e.g., to classify traffic based on data in the Layer 7 (L7) header). For example, slice selection may be based simply on the source device by using the source network layer (e.g., IP) address or based on traffic type or destination network region by examining the L7 header. In some embodiments, the network slice selector maintains state that maps connections to network slices so that deep packet inspection does not have to be performed on each data message of the connection. Also, for some connections, only certain data messages contain the L7 header information required to perform slice selection.

[0098] When slice selection is performed using deep packet inspection, the first data message of a connection may not contain the L7 header information required by the slice selector to correctly identify the slice. For example, a connection between an end device (e.g., a mobile device such as a smartphone or tablet, a laptop or desktop computer, an IoT device, a self-driving car, a smart camera belonging to a security system) and a network domain is often initiated by a set of connection initiation messages such as a TCP handshake. After the handshake is completed, the device sends, for example, an http get message that includes the network domain. Subsequent data messages sent between the device and the network domain may not contain such information.

[0099] Although examples have been described above with respect to a physical switch, these examples can alternatively be performed with a virtual switch. Also, although an orchestrator, a virtual management topology, and a monitoring module are mentioned separately, all of these processes may operate together. These examples are not intended to limit which process performs which steps. Instead, the monitoring module is considered to be any part of the virtual management topology that performs the above stages.

[0100] Other examples of the present disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the examples disclosed herein. Although some of the above methods have been presented as a sequence of steps, it will be understood that one or more steps may occur simultaneously, overlap, or occur in a different order. The order of steps presented is merely a possibility, and the steps may be performed or carried out in any suitable manner. Moreover, the various features of the examples described herein are not mutually exclusive. Rather, any feature of any example described herein may be incorporated into any other suitable example. The above specification and examples are considered to be merely illustrative, with the true scope and spirit of the present disclosure being indicated by the following claims.

Claims

1. 1. A method for managing latency in a slice-based network, comprising: The monitoring module sets a flag in the packet header of the packet corresponding to the slice; The monitoring module receives slice-specific packet timing information from a plurality of switches in a slice-based network, the packet timing information being determined by an agent executing on the switches, the agent comprising at least if the flag is recognized but the packet does not relate to a flow already stored in the switch, storing the flow in association with a current time; determining the slice-specific packet timing information by subtracting the stored current time from a new current time when the flow is recognized to be stored in an individual switch based on address information in the packet header; the monitoring module determines a latency value based on the packet timing information from a plurality of switches in a first slice path for the slice; the monitoring module selects an alternative slice path for the slice based on a comparison of the latency value to a threshold, the alternative slice path including a third switch not in the first slice path; The method includes the monitoring module implementing the alternative slice path through at least a first switch and a third switch.

2. The method of claim 1, wherein the agent further deletes the flow from storage in the switch.

3. 3. The method of claim 1 or 2, wherein the packet timing information is collected by the agent at each hop in the plurality of switches, and the packet timing information is received by the monitoring module and added as part of determining the latency value of the slice.

4. The method of any one of claims 1 to 3, wherein the threshold is established by a service level agreement ("SLA") that includes a maximum latency for the slice.

5. The method of any one of claims 1 to 4, wherein the plurality of switches track flags in Transmission Control Protocol ("TCP") packet headers to report the packet timing information.

6. 6. A method according to any preceding claim, wherein receiving the packet timing information comprises receiving source and destination addresses from first and second agents, and the monitoring module uses the source and destination addresses as part of calculating the latency value for the slice.

7. The method of any preceding claim, wherein calculating the latency value comprises averaging the latency of the slices over a period of time.

8. The method of any one of claims 1 to 7, wherein implementing the alternative slice path includes instantiating a new Virtual Network Function ("VNF") in the third switch for access by the slice traffic before configuring the first switch to route slice traffic to the third switch.

9. 1. A computer-readable medium comprising instructions that, when executed by a computer system, perform a method for managing latency in a slice-based network, the method comprising: Set a flag in the packet header of the packet corresponding to the slice, receiving slice-specific packet timing information from a plurality of switches in a slice-based network, the packet timing information being determined by an agent executing on the switches, the agent comprising at least if the flag is recognized but the packet does not relate to a flow already stored in the switch, storing the flow in association with a current time; determining the slice-specific packet timing information by subtracting the stored current time from a new current time when the flow is recognized to be stored in an individual switch based on address information in the packet header; determining a latency value based on the packet timing information from a plurality of switches in a first slice path for the slice; selecting an alternate slice path for the slice based on a comparison of the latency value to a threshold, the alternate slice path including a third switch not in the first slice path; implementing the alternate slice path through at least a first switch and a third switch.

10. A computer system configured to execute a method for managing latency in a slice-based network, the method comprising: Set a flag in the packet header of the packet corresponding to the slice, receiving slice-specific packet timing information from a plurality of switches in a slice-based network, the packet timing information being determined by an agent executing on the switches, the agent comprising at least if the flag is recognized but the packet does not relate to a flow already stored in the switch, storing the flow in association with a current time; determining the slice-specific packet timing information by subtracting the stored current time from a new current time when the flow is recognized to be stored in an individual switch based on address information in the packet header; determining a latency value based on the packet timing information from a plurality of switches in a first slice path for the slice; selecting an alternate slice path for the slice based on a comparison of the latency value to a threshold, the alternate slice path including a third switch not in the first slice path; implementing the alternate slice path through at least a first switch and a third switch.

Images