Tenant-driven dynamic resource allocation for virtual network functions

Abstract

Techniques for tenant-driven dynamic resource allocation in a network function virtualization infrastructure (NFVI). In one example, an orchestration system is operated by a data center provider of a data center, and the orchestration system includes processing circuitry coupled to a memory; logic stored in the memory and configured to be executed by the processing circuitry, wherein the logic is operable to: compute an aggregate bandwidth of a plurality of flows associated with a tenant of the data center provider and processed by a virtual network function assigned to the tenant that is executed on a server of the data center; and modify an allocation of computing resources of the server executing the virtual network function based on the aggregate bandwidth.

Description

[0001] This application claims the benefit of U.S. Application No. 16 / 888,280, filed May 29, 2020, which is incorporated by reference herein in its entirety. TECHNICAL FIELD

[0002] The present disclosure relates to computer networks, and more specifically to virtual network functions provided on computer networks. SUMMARY

[0003] Virtualized data centers are becoming the core foundation of modern information technology (IT) infrastructures. In particular, modern data centers have widely utilized virtualization environments in which virtual hosts, also referred to herein as virtual execution elements such as virtual machines or containers, are deployed and executed on an underlying computing platform of physical computing devices. Virtualization within a data center can provide several advantages. One advantage is that virtualization can provide significant improvements in efficiency. With the advent of multi-core microprocessor architectures in which each physical CPU has a large number of cores, the underlying physical computing devices (i.e., servers) have become increasingly powerful, and the benefits of virtualization have increased significantly. As another advantage, network functions of virtualization can be performed by servers, switches, storage devices, and cloud computing infrastructures, rather than having custom hardware devices for each network function.

[0004] Network operators can provide a network function virtualization (NFV) architecture or network function virtualization infrastructure (NFVI) that is capable of orchestrating and managing multiple virtualized network functions (VNFs), or physical devices that apply network functions (or “services”) in an ordered sequence to packet flows. Example VNF instances can provide firewalls, routing / switching, carrier grade network address translation (CG-NAT), performance enhancement proxies for video, transmission control protocol (TCP) optimization and header enrichment, caching, load balancing, or other network functions. To implement a service chain, one or more virtual networks are used to interconnect VNF instances, through which packet flows are forwarded along the ordered sequence for application of the network functions that make up the service chain. In response to increased load, a service in the service chain applied by one or more VNF instances can be scaled up by spawning more VNF instances for the service. Likewise, in response to decreased load, the service can be scaled down by deleting one or more VNF instances spawned for the service. One or more VNF instances for a single service can be hosted by independent computing devices (e.g., compute nodes or servers), but a computing device can host multiple VNF instances for one or more services. SUMMARY

[0006] In general, this disclosure describes techniques for a network service orchestrator for a network function virtualization infrastructure (NFVI) to dynamically allocate computing resources among virtual network functions assigned to a tenant based on tenant flow statistics. For example, an NFVI provider can deploy an NFVI hosting VNFs that are assigned to multiple different customer tenants (also more simply referred to as tenants or customers) of the NFVI provider. Each VNF performs its corresponding network function to process packet flows associated with one of the tenants of the NFVI provider to which the VNF is assigned. As described herein, a network service orchestrator for the NFVI dynamically distributes physical resources, such as multiple processing cores in servers of the NFVI, among VNFs executing on the servers based on actual utilization of the VNFs for processing flows associated with the tenants assigned to the VNFs.

[0007] The described techniques can provide one or more technical advantages that present at least one practical application. For example, the techniques described herein can improve computing resource utilization by enabling servers to execute more VNFs through on-demand resource allocation while still meeting tenant service level agreements (SLAs) for flow processing. For example, when actual bandwidth utilization for a tenant assigned to a VNF, as determined using tenant flow statistics, does not reach the bandwidth allocated to the VNF (and corresponding processing resource allocation under a static allocation scheme), there is an envelope of unused processing resources that can be dynamically reassigned to one or more other VNFs. As such, implementing the techniques described herein can increase the maximum number of VNFs executing at a time on a server.

[0008] As a further advantage, the techniques allow for tenant-driven dynamic resource allocation by an orchestration system in a data center, rather than from components within the same server as the tenant's VNFs. The orchestration system provides many benefits for resource allocation, including access to tenant flow statistics for multiple VNFs spread across the NFVI and assigned to multiple different tenants. For at least this reason, such dynamic resource allocation can be performed even when a tenant's customer equipment is not in the same data center environment (e.g., interconnection facility) as the NFVI on which the tenant VNFs are executing.

[0009] In one example, an orchestration system operated by a data center provider of a data center, the orchestration system comprising: processing circuitry coupled to a memory; logic stored in the memory and configured for execution by the processing circuitry, wherein the logic is operable to: compute an aggregate bandwidth for a plurality of flows associated with a tenant of the data center provider and processed by virtual network functions, the virtual network functions assigned to the tenant executing on servers of the data center; and modify an allocation of computing resources of the servers for executing the virtual network functions based on the aggregate bandwidth.

[0010] In one example, a method of an orchestration system operated by a data center provider of a data center, the method comprising: computing, by the orchestration system, an aggregate bandwidth of a plurality of flows associated with a tenant of the data center provider and processed by virtual network functions, the virtual network functions assigned to the tenant being executed on servers of the data center; and modifying, by the orchestration system, an allocation of computing resources of the servers executing the virtual network functions based on the aggregate bandwidth.

[0011] In one example, an interconnection system comprising: at least one interconnection facility comprising: a cluster comprising one or more computing devices to host virtual network functions (VNFs) for tenants, wherein each tenant corresponds to one or more VNFs forming a virtual router through which packet flows are exchanged between services and customers, and a programmable network platform configured to: compute an aggregate bandwidth of a plurality of flows associated with one of the tenants and processed by one of the VNFs, the VNFs assigned to the tenant being executed on the computing devices of the cluster; and modify an allocation of computing resources of the computing devices executing the virtual network functions based on the aggregate bandwidth.

[0012] The details of one or more examples are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1 is a block diagram illustrating a conceptual view of a network system with a metro-based cloud exchange providing a plurality of cloud exchange points and with a network function virtualization infrastructure in accordance with the techniques described herein.

[0014] Figure 2 is a block diagram illustrating an example data center providing an operating environment for VNFs with tenant-driven resource allocation in accordance with one or more aspects of the techniques described in this disclosure.

[0015] Figure 3 is a block diagram illustrating an example network function virtualization infrastructure with tenant-driven resource allocation in accordance with the techniques described herein.

[0016] Figure 4 is a flow diagram illustrating an example operation for tenant-driven resource allocation based on actual resource usage in accordance with the techniques described herein.

[0017] Figure 5 is a block diagram illustrating further details of one example of a computing device operating in accordance with one or more techniques of this disclosure.

[0018] Figure 6is a block diagram illustrating a conceptual view of bandwidth usage based processor core management in accordance with one or more techniques of this disclosure.

[0019] In all drawings and text, like reference designations will refer to like elements. DETAILED DESCRIPTION

[0020] Figure 1 is a block diagram illustrating a conceptual view of a metro-based cloud exchange with network function virtualization infrastructure and with multiple cloud exchange points provided in accordance with the techniques described herein. Each cloud-based service exchange point 120A-120C (hereinafter described as "cloud exchange points" and collectively as "cloud exchange points 120") of a cloud-based service exchange 100 ("cloud exchange 100") can represent different data centers (or interconnection facilities) geographically located within the same metropolitan area ("metro-based," e.g., New York City, New York; Silicon Valley, California; Seattle-Tacoma, Washington; Minneapolis-St. Paul, Minnesota; London, United Kingdom; etc.) to provide a resilient and independent cloud-based service exchange through which cloud-based service customers ("cloud customers") and cloud-based service providers ("cloud providers") connect to receive and provide cloud services, respectively. In various examples, the cloud exchange 100 can include more or fewer cloud exchange points 120. In some instances, the cloud exchange 100 includes only one cloud exchange point 120. As used herein, a reference to a "cloud exchange" or "cloud-based service exchange" can refer to a cloud exchange point. A cloud exchange provider can deploy instances of the cloud exchange 100 in multiple different metropolitan areas, each instance of the cloud exchange 100 having one or more cloud exchange points 120.

[0021] Any one or more of the cloud exchange points 120 can be used to at least partially implement the NFVI 102. Each cloud exchange point 120 includes network infrastructure and an operating environment through which the cloud customers 108A-108C (collectively "cloud customers 108") receive cloud services from the plurality of cloud service providers 110A-110N (collectively "cloud service providers 110"). The cloud service providers 110 can host one of a plurality of cloud services. As noted above, the cloud service providers 110 can be public or private cloud service providers.

[0022] The cloud exchange 100 provides secure, private, virtual connectivity to a plurality of cloud service providers (CSPs) in the aggregate to customers of the exchange (e.g., enterprises, network operators, network service providers, and SAAS customers). The plurality of CSPs participate in the cloud exchange by having at least one accessible port in the cloud exchange through which customers can respectively connect to one or more cloud services offered by the CSPs. The cloud exchange 100 allows the private networks of any customer to be directly cross-connected at a common point to any other customer, thereby allowing direct exchange of network traffic between the networks of customers.

[0023] The cloud customers 108 can receive cloud-based services directly via layer 3 peering and physical connectivity with one of the cloud exchange points 120, or indirectly via one of the network service providers 106A-106B (collectively, “NSPs 106,” or alternatively, “carriers 106”). The cloud customers 108 can include customers associated with the VNFs 306A as described above. For example, the cloud customers 108 can include systems used by any or all customer devices used by the cloud clients 118 to access cloud services via VNFs executing in the NFVI 102 at the cloud exchange points 120. The NSPs 106 provide “cloud transit” by maintaining a physical presence within one or more of the cloud exchange points 120 and aggregating layer 3 access from one or more customers 108. The NSPs 106 can directly peer at layer 3 with one or more of the cloud exchange points 120, and in doing so provide indirect layer 3 connectivity and peering to one or more customers 108 that can obtain cloud services from the cloud exchange 100 through the connectivity and peering. In Figure 1In the example above, each cloud exchange point 120 is assigned a different autonomous system number (ASN). For example, cloud exchange point 120A is assigned ASN 1, cloud exchange point 120B is assigned ASN 2, and so on. As a result, each cloud exchange point 120 is a next hop in a path vector routing protocol (e.g., BGP) path from a cloud service provider 110 to a customer 108. As a result, although each cloud exchange point 120 is not a transit network with one or more wide area network links and attendant Internet access and transit policies, each cloud exchange point 120 can peer with multiple different autonomous systems via external BGP (eBGP) or other external gateway routing protocols in order to exchange, aggregate, and route service traffic from one or more cloud service providers 110 to customers. In other words, cloud exchange points 120 can internalize eBGP peering relationships that customers 108 and cloud service providers 110 would otherwise maintain on a pairwise basis. Conversely, customers 108 can configure a single eBGP peering relationship with a cloud exchange point 120 and receive multiple cloud services from one or more cloud service providers 110 via the cloud exchange. While described herein primarily with respect to eBGP or other layer 3 routing protocol peering between cloud exchanges and customers, NSP or cloud service provider networks, cloud exchanges can learn routes from these networks in other ways, such as through static configuration, or via routing information protocol (RIP), open shortest path first (OSPF), intermediate system to intermediate system (IS-IS), or other route distribution protocols.

[0024] As an example of the above, customer 108C is illustrated as having contracted with a cloud exchange provider of cloud exchange 100 to access layer 3 cloud services directly via cloud exchange point 120C. In this way, customer 108C receives, for example, redundant layer 3 connectivity to cloud service provider 110A. In contrast, customer 108C is illustrated as having contracted with a cloud exchange provider of cloud exchange 100 to access layer 3 cloud services directly via cloud exchange point 120C and also contracted with NSP 106B to access layer 3 cloud services via the transit network of NSP 106B. Customer 108B is illustrated as having contracted with multiple NSPs 106A, NSP 106B to have redundant cloud access to cloud exchange point 120A, cloud exchange point 120B via respective transit networks of NSPs 106A, NSP 106B. The above contracts are instantiated in the network infrastructure of cloud exchange points 120 by L3 peering configurations within the exchange equipment of NSPs 106 and cloud exchange points 120 and L3 connections (e.g., layer 3 virtual circuits) established within cloud exchange points 120 to interconnect cloud service provider 110 networks to NSP 106 networks and customer 108 networks, all with at least one port providing connectivity within one or more cloud exchange points 120.

[0025] In some examples, the cloud exchange 100 allows any network service provider (NSP) or "carrier" 106A-106B (collectively, "carriers 106") to directly connect, via virtual layer 2 (L2) or layer 3 (L3) connections, a corresponding one of customers 108A, 108B or other cloud customers including customer 108C to any other customer network and / or CSP 110, allowing direct exchange of network traffic between customer networks and CSPs 110. The virtual L2 or L3 connections can be referred to as "virtual circuits."

[0026] Carriers 106 can each represent a network service provider associated with a transit network through which network subscribers of the carriers 106 can access cloud services provided by CSPs 110 via the cloud exchange 100. Generally, customers of CSPs 110 can include network carriers, large enterprises, managed service providers (MSPs), and software as a service (SaaS), platform as a service (PaaS), infrastructure as a service (IaaS), virtualization as a service (VaaS), and data storage as a service (DSaaS) customers for such cloud-based services provided by CSPs 110 via the cloud exchange 100.

[0027] In this manner, the cloud exchange 100 streamlines and simplifies the partnering process for CSPs 110 and customers in a transparent and neutral manner (via carriers 106 or directly). One example application of the cloud exchange 100 is co-location and interconnection of data centers, where CSPs 110 and carriers 106 and / or customers 108 can already have a network presence, such as by having one or more accessible ports available for interconnection within a data center, which can represent any cloud exchange point 120. This allows participating carriers, customers, and CSPs to have extensive interconnectivity options within the same facility. In this manner, carriers / customers can have the option to create many-to-many interconnections with only one hook-up to one or more cloud exchange points 120. In other words, the cloud exchange 100 allows customers to interconnect to multiple CSPs and cloud services, rather than having to establish separate connections over a transit network to access different cloud service providers or different cloud services of one or more cloud service providers.

[0028] Cloud exchange 100 includes a programmable network platform 104 for dynamically programming cloud exchange 100 to responsively and assuredly meet service requests encapsulating business requirements for services provided by cloud exchange 100 and / or cloud service providers 110 coupled to cloud exchange 100. Programmable network platform 104 can include a network service orchestrator 112 that processes requests for VNF deployment by tenants (e.g., cloud customers). For example, network service orchestrator 112 can organize, direct, and integrate underlying services, such as virtual machines (or containers), and other software and network subsystems, for managing various services (e.g., deployment of VNFs). As a result, programmable network platform 104 can orchestrate business-level services across heterogeneous cloud service providers 110 in accordance with well-defined service policies, quality-of-service policies, service-level agreements, and costs, and further in accordance with service topologies for business-level services.

[0029] Hardware and / or software components of NFVI 102 implement a network management and resource orchestration system, at least one of which generally performs at least one technique described herein. In one example, the network management and resource orchestration system forms an architecture with at least three functional blocks, one example functional block being an orchestration system that is responsible for onboarding of new network service (NS) and virtual network function (VNF) packages; NS lifecycle management; global and local resource management; and validation and authorization of network function virtualization infrastructure (NFVI) resource requests. In some examples, network service orchestrator 112 in programmable network platform 104 is an example of the orchestration system, while in other examples the orchestration system instructs (e.g., through function calls) network service orchestrator 112 to dynamically allocate resource (e.g., compute) resources. Other functional blocks, such as a management block, oversee lifecycle management of VNF instances; fill the role of fitting and adapting configuration and event reporting between the NFV infrastructure (NFVI) and element / network management systems, and control and manage NFVI compute, storage, and network resources. While shown separately as part of programmable network platform 104, these management blocks can reside in NFVI 102 and cooperate with the orchestration system in deploying VNFs.

[0030] NFVI 102 includes one or more servers 123A-123N (servers 123) for executing / hosting virtual network functions (VNFs) 124A-124N that apply network services to packet flows. Network service orchestrator 112 processes deployment and organization of these network services, for example, by instantiating VNFs on servers 123 to execute these network services.

[0031] The programmable network platform 104 enables the cloud exchange 100 to be dynamically configured and managed by the cloud service providers 110 that manage the cloud exchange 100 to, for example, facilitate virtual connections for cloud-based service delivery from multiple cloud service providers 110 to one or more cloud customers 108. The cloud exchange 100 can enable cloud customers 108 to bypass the public Internet to directly connect to cloud service providers 110, thereby improving performance, reducing costs, increasing the security and privacy of the connection, and leveraging cloud computing for additional applications. In this way, for example, enterprises, network operators, and SaaS customers can integrate cloud services with their internal applications in at least some respects as if such services were part of their own data center network or as if directly coupled to their own data center network.

[0032] In other examples, the programmable network platform 104 enables cloud service providers to configure the cloud exchange 100 with L3 instances requested by cloud customers 108, as described herein. For example, a customer 108 can request an L3 instance to link multiple cloud service providers through the L3 instance (e.g., for transferring the customer’s data between two cloud service providers, or for obtaining a service mesh from multiple cloud service providers).

[0033] The programmable network platform 104 can represent an application executing within one or more data centers of the cloud exchange 100, or alternatively can represent an application executing off-site at a back office or branch office of the cloud provider, for example. The programmable network platform 104 can be distributed in whole or in part among the data centers, each data center being associated with a different cloud exchange point 120 to make up the cloud exchange 100. While shown as managing a single cloud exchange 100, the programmable network platform 104 can control service provisioning for multiple different cloud exchanges. Alternatively or additionally, multiple separate instances of the programmable network platform 104 can control service provisioning for respective multiple different cloud exchanges.

[0034] In the illustrated example, the programmable network platform 104 includes a service interface (or “service API”) 114 that defines methods, fields, and / or other software primitives that applications 111, such as a customer portal, can invoke through to the programmable network platform 104. The service interface 114 can allow the operators 106, customers 108, cloud service providers 110, and / or cloud exchange providers programmable access to the capabilities and assets of the cloud exchange 100 in accordance with the techniques described herein.

[0035] For example, the service interface 114 can facilitate machine-to-machine communication to enable dynamic provisioning of virtual circuits in the cloud exchange for interconnecting customer and / or cloud service provider networks. In this manner, the programmable network platform 104 enables automation of various aspects of cloud service provisioning. For example, the service interface 114 can provide an automatic and seamless way for customers to establish, offload, and manage interconnections between multiple different cloud providers participating in the cloud exchange.

[0036] Further example details of a cloud-based services exchange can be found in U.S. Patent Application No. 15 / 099,407, filed April 14, 2006, entitled “CLOUD-BASED SERVICES EXCHANGE”; U.S. Patent Application No. 14 / 927,451, filed October 29, 2005, entitled “INTERCONNECTION PLATFORM FOR REAL-TIME CONFIGURATION AND MANAGEMENT OF A CLOUD-BASED SERVICES EXCHANGE”; and U.S. Patent Application No. 14 / 927,106, filed October 29, 2005, entitled “ORCHESTRATION ENGINE FOR REAL-TIME CONFIGURATION AND MANAGEMENT OF INTERCONNECTIONS WITHIN A CLOUD-BASED SERVICES EXCHANGE”; each of which is incorporated by reference herein in its respective entirety.

[0037] Customer 108B represents (or includes) a tenant network of the cloud exchange 100. The customer 108B exchanges packetized data in packet flows with one or more other networks, for example, via a virtual circuit or through other connections of the cloud exchange point 120A. In some cases, the NFVI 102 applies one or more network services to the packet flows on behalf of a tenant associated with the customer 108.

[0038] In the illustrated example, customer 108B exchanges packets of packet flow 125A with cloud service provider network 110A and exchanges packets of packet flow 125B with cloud service provider network 110N. A VNF of one or more VNFs 124A executed by server 123A applies a network function (in some cases as part of a network service) to packets of packet flows 125A-125B (collectively "packet flows 125"). This VNF can be referred to as a tenant VNF because it is assigned to or associated with a tenant of the NFVI 102 provider (here also the cloud exchange 100 provider). In other words, the tenant VNF handles packet flows 125 associated with a tenant. The tenant VNF can be leased or sold by the NFVI 102 provider to a tenant, or the tenant can deploy the tenant VNF to server 123A or a virtual execution element thereof. The virtual execution element can include a virtual machine or a container. The tenant can invoke service interface 114 or otherwise request a service chain in NFVI 102 that includes the tenant VNF for packet flows to and / or from customer 108B. Server 123A can execute multiple VNFs 124A associated with multiple different tenants, each of the VNFs 124A handling one or more packet flows associated with a tenant for that VNF.

[0039] According to the techniques of the present disclosure, network service orchestrator 112 for NFVI 102 dynamically allocates physical resources, such as multiple processing cores in each server 123, between VNFs 124 executed on any server 123 based on actual utilization of the VNFs 124 for handling flows associated with tenants associated with the VNFs 124. For example, network service orchestrator 112 generates flow statistics for server 123A indicating bandwidth for each flow 125 forwarded by server 123A. That is, server 123A performs flow monitoring of flows traversing server 123A and generates flow records 127 for the flows. Network service orchestrator 112 collects the flow records, which can be used to compute estimated bandwidth for each flow as well as other flow properties such as virtual network interfaces of server 123A used by the flow and source and destination network addresses for the flow. Using the collected flow records, network service orchestrator 112 computes estimated bandwidth for each flow 125. In addition, network service orchestrator 112 determines from the flow records that flow 125 is associated with customer 108B, which is a tenant of the NFVI provider. Network service orchestrator 112 can consolidate this computed data into tenant flow statistics.

[0040] In the example where packet flows are encapsulated in frames transported according to User Datagram Protocol (UDP), the following steps are performed to compute estimated bandwidth:

[0041] 1) Assuming a sFlow sample rate of 0.25%, 50,000 packets are the sample, which is 20,000,000 total packets per frame on the link.

[0042] 2) Assuming each packet carries 1518 bytes, the total number of bytes sent on the link is calculated as 20,000,000 * 1518 = 30,360,000,000.

[0043] 3) The bits are calculated as 30360,000,000 * 8 to convert the bytes to total bits.

[0044] 4) The total bits are divided by the average latency measured on the link to yield the total bandwidth utilization. For example, if the latency is sixty (60) seconds, then the total bandwidth utilization is calculated as (30360,000,000 * 8) / 60 or 4048000000 bits per second (i.e., 4,048 Megabits per second (Mbps)).

[0045] In the example of transmitting a packet stream according to the Transmission Control Protocol (TCP), the following steps are performed to calculate the estimated bandwidth:

[0046] 1) Convert the TCP window size from bytes to bits.

[0047] 2) Using a standard 64KB TCP window size, where 64KB = 65536 bytes, the TCP window size in bits will be calculated as 65536 * 8 = 524288 bits.

[0048] 3) After converting the TCP window to bits, divide the window in seconds by the round trip latency of the link. If the round trip latency is thirty (30) milliseconds, then the bandwidth utilization can be calculated as 524288 bits / 0.030 seconds = 17476266 bits per second throughput or 17.4 Mbps maximum possible throughput.

[0049] Tenant flow statistics generally include measured or computed packet flow information (e.g., packet size, packets per second, estimated bandwidth, etc.) for flows associated with each of one or more tenants of the NFVI 102 provider. As described above, the network service orchestrator 112 can map tenant information to flow records to determine which flows are associated with at least one particular tenant. As described herein, tenant flow statistics can indicate actual aggregate bandwidth handled by a tenant VNF for a tenant, and this information can be compared to resource allocation information to dynamically allocate resources. For example, in response to determining that the aggregate bandwidth handled by a tenant VNF of the VNF 124A does not satisfy bandwidth allocation / requirements, the techniques described herein can modify the allocation of computing resources of the server 123A, such as the number of processor cores or the amount of memory allocated for use by the tenant VNF.

[0050] In this way, by determining the rate at which the server 123A forwards each packet flow 125 associated with a tenant, the rate accordingly indicates the rate at which the tenant VNF handles the packet flow 125 (e.g., the aggregate bandwidth or throughput of the tenant VNF). These rates can be combined to obtain an aggregate bandwidth, and in response to determining the aggregate bandwidth of the packet flows 125 associated with a tenant and handled by the tenant VNF, the network service orchestrator 112 can compare the aggregate bandwidth to bandwidth requirements provided by resource allocation information for the VNF and produce a comparison result. The comparison result can indicate an expected aggregate bandwidth for the tenant VNF that indicates more or less computing resources are needed to handle the expected aggregate bandwidth as compared to the amount of computing resources currently allocated to the tenant VNF. As a result, and thus based on the aggregate bandwidth, the network service orchestrator 112 can output configuration data 129 to cause the server 123 to modify the allocation of computing resources of the server 123A to allocate more or less computing resources to execute the tenant VNF. In some examples, the network service orchestrator 112 periodically repeats the comparison and maintains comparison results for determining trends such that, over time, a correlation can be identified between the number of processor cores needed and actual bandwidth usage. This correlation can be a function or lookup table used by the network service orchestrator 112 to determine an appropriate number of processor cores to satisfy the desired bandwidth for executing the tenant VNF to continue handling the flows 125 (and in some cases other flows associated with the customer 108B). Although described herein as being performed primarily by the network service orchestrator, the techniques described in this disclosure can be performed by a separate computing system that has determined a modification to computing resources of a tenant VNF, such as a controller or application, calling the network service orchestrator 112 to request the modification to computing resources of the tenant VNF in the NFVI 102.

[0051] Figure 2is a block diagram illustrating an example data center for a VNF-provisioned operating environment with tenant-driven resource allocation in accordance with one or more aspects of the techniques described in this disclosure.

[0052] In this example data center, cloud exchange 100 allows any NSP 106A-106C or other customer's customer network 202A, 202B and corresponding one of NSP networks 204A-204C (collectively, "NSP or "carrier" networks 204) to be directly cross-connected to any other customer network via Layer 2 (L2) or Layer 3 (L3) connections, allowing for the exchange of service traffic between customer networks and CSPs 110. Data center 200 can be located entirely within a centralized region, such as a warehouse or localized data center consortium, and provide power, cabling, security, and other services to NSPs, customers, and cloud service providers that position their respective networks within data center 200 (e.g., for co-location) and / or connect to data center 200 through one or more external links.

[0053] Cloud exchange 100 includes network infrastructure 206 (e.g., for virtual networks) and an operating environment through which customer networks 202 can receive services from one or more CSPs 110 via interconnections. In this example, cloud exchange 100 is a multi-tenant, multi-homed, multi-protocol network that provides a platform for NSPs 106, customers 108, and CSPs 110 to exchange services and traffic. Figure 2 In this example, network infrastructure 206 represents the switching fabric of the interconnection facilities of cloud exchange 100 and includes a plurality of ports that can be dynamically interconnected with virtual circuits by, for example, invoking service interfaces 114 of programmable network platform 104. Each port is associated with an NSP 106, customer 108, and CSP 110. This enables NSP customers to have the option of creating many-to-many interconnections with only a single hook-up to the switching network and underlying network infrastructure 206 that presents the interconnection platform for cloud exchange 100. In other words, cloud exchange 100 allows customers to interconnect to multiple CSPs 110 using network infrastructure 206 within data center 200, rather than having to establish separate connections on a transit network to access different CSPs 110.

[0054] The interconnections described herein can refer to, for example, physical cross- connects, Ethernet connections such as Layer 2 VPNs or virtual private LANs (e.g., E-LINE, E-LAN, E-TREE, or E-access), Internet Exchange based interconnections in which respective network devices (e.g., routers and / or switches) of interconnection customers directly peer and exchange Layer 3 routes for service traffic exchanged via the network infrastructure 206, and cloud exchanges in which customer routers peer with network infrastructure 206 (or "provider") network devices rather than directly with other customers. The cloud exchange 100 can provide interconnection services to customers to network services provided by the CSP 110. That is, the interconnection services of the cloud exchange 100 provide access to network services (e.g., VNFs) provided by the CSP 110.

[0055] For Layer 3 or above interconnections, customers 108 can receive services either directly via Layer 3 peering and physical connections to one of the co-located facility exchange points or indirectly via one of the NSPs 106. The NSPs 106 provide "transit" by maintaining a physical presence within the data center 200 and aggregating Layer 3 access from one or more customers 108. The NSPs 106 can directly peer with the data center 200 at Layer 3 and, in doing so, provide indirect Layer 3 connectivity and peering to one or more customers 108 from which the customers 108 can obtain services from the cloud exchange 100.

[0056] In instances in which cloud exchange 100 provides an Internet exchange, network infrastructure 206 can be assigned a different autonomous system number (ASN). As a result, network infrastructure 206 is the next hop in a path vector routing protocol (e.g., BGP) path from CSPs 110 to customers 108 and / or NSPs 106. As a result, although cloud exchange 100 is not a transit network with one or more wide area network links and attendant Internet access and transit policies, cloud exchange 100 can peer with multiple different autonomous systems via external BGP (eBGP) or other external gateway routing protocols in order to exchange, aggregate, and route service traffic from one or more CSPs 110 to customers 108. In other words, cloud exchange 100 can internalize eBGP peering relationships that customers 108 and CSPs 110 would otherwise maintain on a pairwise basis. Conversely, customers 108 can configure a single eBGP peering relationship with cloud exchange 100 and receive multiple services from one or more CSPs 110 via the cloud exchange. While described herein primarily with respect to eBGP or other layer 3 routing protocol peering between co-located facility points and customer, NSP, or service provider networks, co-located facility points can learn routes from these networks in other ways, such as through static configuration, or via routing information protocol (RIP), open shortest path first (OSPF), intermediate system to intermediate system (IS-IS), or other route distribution protocols.

[0057] As an example of the cloud exchange deployment described above, Figure 2 Customer network 202B in FIG. 1 is shown as having contracted with the cloud exchange provider of cloud exchange 100 to access layer 3 services directly via cloud exchange 100, and also contracted with NSP 106B to access layer 3 services via the transit network of NSP 106B. Customer network 202A is shown as having contracted with NSP 106B to access layer 3 services via the transit network of NSP 106B. The contracts described above can be instantiated in network infrastructure 206 of cloud exchange 100 by L3 peering configurations within NSPs 106 and exchange equipment of cloud exchange 100, and L3 connections (e.g., layer 3 virtual circuits) established within cloud exchange 100 to interconnect CSPs 110 to NSPs 106 and customer networks 202, all with at least one port providing connectivity within cloud exchange 100.

[0058] In some examples, the network infrastructure 206 includes one or more virtual machines or containers of the NFVi 102 that are used to deploy virtualized network functions. In these examples, the network service orchestrator 112 can receive a request via the service interface 114 to deploy one or more virtualized network functions (e.g., virtual routers, load balancers, etc.) implemented in the NFVi 102 of the network infrastructure 206. The network service orchestrator 112 can request a VNF provisioner (e.g., the VNF provisioner 210) for a VNF distribution including one or more VNF images.

[0059] With respect to Figure 3 further details of the example network system and Figure 2 the example data center 200 can be found in U.S. Provisional Patent Application Serial No. 62 / 908,976, filed October 1, 2009, and entitled “VIRTUALIZED NETWORK FUNCTIONS VERIFICATION USING DECENTRALIZED IDENTIFIERS,” which is incorporated by reference herein.

[0060] As described above with respect to Figure 1 the orchestrator 112 can dynamically allocate computing resources of servers of the NFVI 102 to tenant VNFs of a tenant of the data center 200 / NFVI 102 provisioner based on an aggregate bandwidth of flows processed by the tenant VNFs.

[0061] Figure 3 is a block diagram illustrating an example architecture of a network function virtualization infrastructure with tenant-driven dynamic resource allocation in accordance with the techniques described herein. In Figure 3 the example, the system 300 refers to a point of exchange (e.g., a cloud exchange point) of a network function virtualization infrastructure (NFVi) 102 with customer devices connected via a gateway 320 and a cloud service provider running on a cloud network 322. In some examples, the NFVi 102, the gateway 320, and / or the cloud network 322 can be provided in a data center environment.

[0062] The NFVI 102 includes one or more servers 302A-302N (servers 302) for executing / hosting network services including virtual network devices that connect customer devices with cloud services. An example architecture of the NFVi 102 enables deployment of one or more services, such as virtualized network functions (VNFs), on the servers 302. The NFVi 102 includes computing hardware, storage hardware, and network hardware for executing virtual network functions (VNFs). The NFV management 304 handles deployment and organization of these network services, for example, by instantiating VNFs on the servers 302 to execute these network services. The NFV management 304 designates resources (e.g., resource capacity) in the servers 302 to support execution of the VNFs as instructed by an orchestration system.

[0063] A VNF can provide similar functionality to a hardware-based network device, such as a dedicated network device, but the VNF provides this functionality in software. The VNF is primarily a software construct and, thus, can be decoupled from the underlying hardware. For example, the VNF 306A can provide the same routing, switching, firewall, intrusion detection, or other services that are traditionally provided by dedicated hardware but are provided in software. The VNF 306A can provide forwarding and network address translation services for network traffic to and from the VNF 306A. In some examples, the VNF 306A, as a routing VNF or virtual router to the cloud network 322, performs routing and forwarding operations on packets from customer devices.

[0064] In examples, Figure 3 The NFVI 102 platform includes a server (e.g., server 302A) running virtualization software (e.g., a hypervisor) in a virtualization layer that implements a virtual execution environment on which VNF images (including network infrastructure software) are deployed. The server 302A can represent a server 302 in the NFVi 102 that is configured to execute a VNF 306A. The server 302A can include a processor 310A, a memory 312A, and a network interface 314A. The processor 310A can be a central processing unit (CPU), a graphics processing unit (GPU), or another processor known in the art. The memory 312A can include random access memory (RAM), read only memory (ROM), or another memory known in the art. The network interface 314A can be a network interface card (NIC), a network adapter, or another network interface known in the art. The server 302A can include other components not shown in FIG. 3, such as a bus, a power supply, or other components known in the art. Figure 1The server 302A can provide one or more virtual machines (VMs) via the virtualization layer 308A, where each VM emulates hardware for running software. In other words, an example VM (e.g., a LINUX kernel VM (KVM)) provides a virtualized operating system and application suite (e.g., to deploy a VNF) for customer access. Alternatively or additionally, the server 302A can provide containers (e.g., such as those provided by the open source Docker container application) or other virtual execution environments in which VNFs are implemented. In some examples, the NFVI 102 also includes a virtualization layer 308A on hardware to provide virtual compute, virtual storage, and virtual networks for executing VNFs. The NFVI 102 can be executed by one or more computing devices in a centralized or distributed manner.

[0065] In Figure 3 In examples, the server 302A can be part of a computer cluster or pod whose physical resources are virtualized into a network infrastructure, such as the NFVI 102. The computer cluster can be labeled as a network edge of a cloud service provider. Each cloud service provider can be a data center tenant with one or more VNFs running in the server 302 (e.g., server 302A) to provide access to cloud services from devices in the cloud network 322. As a network edge, the server 302A executes VNFs to perform network edge services, such as routing and forwarding operations for packets directed to and intended for the cloud service provider on the cloud network 322 or received from the cloud service provider on the cloud network 322. However, the VNFs can apply network functions to flows to or from any network that are associated with one or more tenants of the NFVI 102 provider.

[0066] A provisioning system (e.g., the network service provisioner 112) that controls the provisioning of resources in the NFVI 102 can use the NFV management 304 of the NFVI 102 to instruct the server 302A to allocate compute resources among VNFs according to the techniques described herein. These techniques direct resource allocation toward executing VNFs based on usage. An example technique optimizes resource allocation when balancing two tenant / data center conditions: 1) a quality of service (QoS) expectation (i.e., actual resource utilization) on behalf of a tenant and 2) a quality of service (QoS) obligation (i.e., resource requirements) on behalf of a data center provider.

[0067] The NFV management 304 distributes the physical resources of the resource pool 310 to VNFs running in the servers 302 of the NFVI 102. Examples of various physical resources include processing resources (e.g., processor cores), networking resources (e.g., physical network interface cards (NICs)), and memory resources. Multiple processor cores 314A-314N ("processor cores 314") can be an example of processing or computing resources. In one example, using the virtualization layer 308A to generate an abstraction of the various physical resources of the resource pool 310A, the NFV management 304 configures the abstraction as countable virtual resources for use by VNFs running in the servers 302 of the NFVI 102. The physical resources can be virtualized into virtual computing resources that each include one or more processor cores. Another computing resource can be a compute node such as a virtual machine. Other virtual resources include virtual storage resources and virtual network resources.

[0068] The NFV management 304 can couple virtual network interfaces in the VNI space, such as the VNIs 316A-316N (or VNIs 316), to the virtual switch 318A. The virtual switch 318A is configured with the VNIs 316, which are logical interfaces that encapsulate and decapsulate the virtual networking that occurs in the NFVI 102. Each VNI 316 can be associated with a virtual network of the NFVI 102 that is assigned to a tenant. That is, one or more virtual networks can be assigned to a tenant for packet flows. The virtual networks can have corresponding virtual network identifiers, such as VXLAN network identifiers. Packet flows transmitted using these virtual networks are associated with the tenant, and the packets of such packet flows can include the virtual network identifier of the virtual network on which the packet flow is transmitted.

[0069] In one example, the virtual switch 318A can be configured with VXLAN interfaces, each VXLAN interface configured with a different one of the VNIs 316 and a corresponding VNI identifier. When a physical network interface (e.g., NIC 317) in the server 302A receives network traffic in the form of one or more packet flows, the packets of that packet flow include information identifying those VNIs 316 (e.g., having the same VNI identifier). The virtual switch 318A switches each packet flow to its correct VNF, which can be the VNF to which the VNI is assigned. The sFlow agent 326A collects packet flow data including the VNI of each flow and sends the collected packet flow data to the collector.

[0070] Statistics and other data points associated with the transmitted packet flows can provide useful information, for example, for vector packet processing by the VPP 318A, and for tenant-driven dynamic resource allocation for VNFs by an orchestration system of the NFVI 102, which can run in the same data center environment as the server 302A. The sFlow agent 326A captures these statistics and other data points, and then provides them to the sflow collector 324, which combines the provided statistics and other data points with information provided by other sFlow agents 326 in other servers 302.

[0071] In some examples, the sFlow collector 324 utilizes SNMP to communicate with the sFlow agent 326A in the server 302A in order to configure sFlow monitoring on the VNF 306A. The sFlow agent 326A uses two forms of sampling mechanisms: packet-based statistics sampling of the switched or routed packet flows, and time-based sampling of counters. Generally, packet flow sampling and counter sampling are performed by sFlow instances associated with individual data sources within the sFlow agent 326A. To perform packet flow sampling, an sFlow instance is configured with a sampling rate. The packet flow sampling process results in the generation of packet flow records. To perform counter sampling, an sFlow instance is configured with a sampling interval. The counter sampling process results in the generation of counter records. The sFlow agent 326A collects the counter records and packet flow records, and sends them to the sFlow collector 324 in the form of sFlow datagrams.

[0072] Packet flow sampling, one example of packet flow monitoring, is implemented as follows: When a packet arrives at an interface (e.g., VNI 316A), VNF 306A makes a filtering decision to determine whether the packet should be dropped. If the packet is not filtered, a target interface is assigned by the switching / routing functionality of VNF 306A. At this point, sFlow agent 326A determines whether to sample the packet. sFlow agent 326A uses a counter that decrements with each packet. When the counter reaches zero, whether or not sampling occurs, a counter Total_Packets is incremented, and Total_Packets is a count of all packets that can have been sampled. Using counters such as Total_Packets, sFlow agent 326A generates various information including flow statistics. The agents / components of the orchestration system described herein can use these flow statistics to instruct NFVI 102 about whether to modify the allocated resource capacity, such as allocation of compute resources. In one example, the collected flow statistics can be used to add or subtract one or more compute resources (e.g., virtual and physical compute nodes such as processor cores in a multi-core environment).

[0073] To illustrate by way of example, the orchestration system can modify the (current) allocation of compute resources to a running VNF (e.g., VNF 306A) based on actual bandwidth usage, such that the running VNF can consume at most the modified allocation of compute resources. In one implementation, sFlow agent 326A can calculate bandwidth utilization based on counter Total_Packets, one or more flow statistics, and time information (e.g., time interval in seconds), sFlow agent 326A calculates bandwidth utilization (e.g., packet processing rate or throughput over a given time interval). In one example, sFlow agent 326A calculates flow statistics to assist in bandwidth utilization calculation, one example flow statistic includes number of packets per second (or simply packets per second (PPS)). In another example, sFlow agent 326A calculates a flow statistic referred to as average packet size per individual flow, which can be aggregated into average packet size per tenant flow. By multiplying average packet size with packets per second, sFlow agent 326A calculates bandwidth utilization. As an alternative, sFlow agent 326A can use a different counter to calculate bandwidth utilization, such as a counter that increments for each byte in the packet flow. This counter can determine total number of bytes in the packet flow. Using this counter and timestamp data, sFlow agent 326A can calculate bandwidth utilization by dividing total number of bytes by time interval in seconds.

[0074] Taking a sample includes copying the header of the packet or extracting features from the packet and storing the sample information in an sFlow datagram. Example flow properties of the sample information include: source address SRC, destination address DEST, virtual network identifier (e.g., one of the virtual network identifiers such as VXLAN network identifiers or VNIs 316), and packet size. Based on the average packet size (assuming the packet size is consistent across the packets) and the counter Total_Packets average (e.g., per flow), the sFlow agent 326A calculates the average bandwidth utilization.

[0075] An sFlow datagram contains a list of packet flow records and counter records. The format of each record is identified by the data_format value. The data_format namespace is extensible, allowing the addition of standard record types as well as vendor-specific extensions. Several standard record types have been defined. However, the sFlow agent does not need to support all the different record types, only those that are applicable to its processing of the particular packet it is reporting on. For example, if the VNF 306A implements layer 2 / 3 switching, the VNF 306A reports layer 2 information for packets it switches, and layer 2 and layer 3 information for packets it routes, to the sFlow agent 326A. The data_format uniquely identifies the format of an opaque structure in the sFlow specification. The data_format is constructed to uniquely identify the format of a structure (e.g., a standard structure). An example data_format when used to describe flow_data can identify a set of flow properties.

[0076] Each time a sample is taken, the counter Total_Samples is incremented. Total_Samples is a count of the number of samples generated. The sFlow instance sends the sample to the sFlow agent 326A for processing. The sample includes the packet information, as well as the values of the Total_Packets and Total_Samples counters. The sFlow agent 326A can then use the sample to obtain additional information about the trajectory of the packet through the NFVI 102. Such information depends on the forwarding functionality of the VNF 306A. Examples of trajectory information provided are source interface and destination interface, source address and destination address, source VLAN and destination VLAN, next hop subnet, full AS path. The details of the trajectory information are stored in the sFlow datagram format along with the average bandwidth rate. The virtual switch 318A assumes that the trajectory information applies to each packet.

[0077] The virtual switch 318A refers to a vector packet processing application (e.g., a software switch) built on a software platform (e.g., a virtual switch 318A running on a host 304A). proprietary and open source versions of vector packet processing technology). In general, virtual switch 318A provides data plane functionality (e.g., in a virtual router or virtual switch) that includes packet forwarding operations. Virtual switch 318A includes a set of forwarding nodes and a support framework arranged in a directed graph. The framework has all the basic data structures, timers, drivers (and interfaces to driver software development kits (e.g., data plane development kit (DPDK)), a scheduler that allocates CPU time between graph nodes, performance and debugging tools like counters and built-in packet tracing. The latter enables capturing path or trace information taken by packets in the graph with high timestamp granularity, giving full visibility into processing at per-packet level. Virtual switch 318A can process trace information, such as trace information determined by the framework and any trace information generated via flow monitoring. Virtual switch 318A can be coupled to a VNI interface and process packets that arrive through physical network hardware on server 302A. Using the trace information, virtual switch 318A assembles these packets into vectors, e.g., virtual switch 318A classifies packets by protocol or format, and when software nodes in virtual switch 318A are scheduled, virtual switch 318A takes its packet vectors and processes them in tight dual (or quad) loops, with prefetching to CPU caches for optimal performance.

[0078] Cloud network 322 can communicatively couple VNF 306A to one or more cloud services 327. Cloud network 322 is generally hidden or otherwise not available to devices on the public network. For example, cloud network 322 can receive, from VNF 306A, a stream of packets transmitted from a client device or another cloud service to a cloud service. Examples of cloud services include Google Cloud, Azure, Oracle Cloud, Amazon web services (AWS), IBM Cloud, Alibaba Cloud, and Salesforce. In some aspects, cloud network 322 can be an Equinix Cloud Exchange provided by Equinix, Inc. of Redwood City, CA. VNF 306A can be a provider-neutral VNF that combines two or more cloud services into a hybrid cloud service.

[0079] Gateway 320 can communicably couple VNF 306A to a public network and / or a private network of customer devices. The public network can be a publicly available network with little or no restrictions. For example, the public network can be a network that is part of the Internet. The private network can be a network that is part of an enterprise network and is only accessible by authorized users. The customer devices are customers of VNF 306A and, as an example, can be computing devices located in a branch office of a tenant or otherwise associated with the tenant or customer. The public gateway can receive traffic from the public network with a destination address of server 302A hosting VNF 306A within the data center. VNF 306A can receive network traffic from gateway 320.

[0080] As an example, when the orchestration system (e.g., orchestrator 112 of Figure 1 performs an initial resource allocation for VNF 306A, a VNF of a particular tenant, in response, NFV management 304 deploys VNF 306A on server 302A as a virtual point-of-presence VNF (e.g., a routing VNF that connects customer devices to a cloud service of the particular tenant in cloud network 322). NFV management 304 also assigns a portion of the VNI space to the particular tenant for receiving / transmitting packet flows to / from customer devices / cloud services. The VNI space can refer to a range of virtual network identifiers for corresponding virtual network interfaces 316A-316N (VNIs 316). NFV management 304 distributes virtual networks in the physical network among multiple tenants for NFVI 102. Each virtual network can represent an overlay network, such as a VXLAN, and is associated with a virtual network identifier (e.g., a VXLAN network identifier). The virtual network identifier can be included in a packet to identify the virtual network of virtual switch 318A for forwarding by virtual switch 318A to one of VNFs 306 and a next-hop device.

[0081] In one example, the NFV management 304 assigns the VNI 316A and a corresponding address to a VNF of a particular tenant in the server 302A, the VNF 306A. The corresponding address can be a Layer 2 or Layer 3 construct to uniquely identify the VNI 316A as a target of packet flows directed to a cloud service of the particular tenant. For at least this reason, the assigned network address of the particular tenant can be stored in packets of the packet flows (e.g., in a packet header) - possibly along with the assigned VNI 316A. When the packets arrive at the gateway 320 (or in another network device), the assigned address is extracted and converted to a VNI identifier (e.g., number) that matches the assigned VNI identifier of the VNI 316A. Thus, the corresponding VNI identifier of the VNI 316A uniquely identifies the assigned VNI of the particular tenant such that packets delivered to that VNI are forwarded (in part) by the virtual switch 318A to the VNF 306, i.e., the VNF of the particular tenant.

[0082] The VNF 306A manages network traffic to and from cloud services communicably coupled to the server 302A via the cloud network 322. When communicating on the gateway 320 or the cloud network 322, the VNF 306A can handle network traffic that includes a network address (e.g., an IP address of the server 302A) as a source address or a target address. In some examples, multiple VNFs on the server 302A can each have a different public IP address that share a single physical network interface. In some examples, the VNF 306A can be a virtual machine or a container. Figure 1 In the example, the server 302 and the VNF 306 are described as a current NFVi infrastructure. Note that the current NFVi infrastructure can include only one server and one VNF, or alternatively, more than one server with more than one VNF.

[0083] According to the techniques described in this disclosure, in some aspects, the VNF 306A can form part of a network service of a particular tenant. The tenant can have purchased resources from a data center provider for executing the network service such that the purchased resources are a requirement that the data center provider has an obligation to provide. As described herein, in several instances, the network service of the tenant does not actually utilize all of the purchased resources, leaving a significant amount of unused and wasted resource capacity.

[0084] In some examples, the amount of network traffic directed to a VNF of a particular tenant fluctuates in such a way that, in certain situations, the VNF of the particular tenant can not fully utilize the initial allocation of resources (e.g., computing resources). The particular tenant, cloud service provider, and / or data center provider can have overestimated the expected amount of network traffic and, as a result, purchased an unnecessary amount of resource capacity including extraneous bandwidth capacity (e.g., in a service level agreement (SLA)). Over time, the orchestration system of the NFVI 102 can determine that the VNF of the particular tenant consistently utilizes only a portion of the initial allocation of physical resources (bare metal servers, communication infrastructure, etc.) provided by the data center and, for at least some reason, can modify the (current) resource allocation of the VNF of the particular tenant, e.g., to match the actual resource usage (e.g., bandwidth utilization) of the VNF of the particular tenant. In some examples, the orchestration system of the NFVI 102 also modifies the SLA to update the resource requirements / obligations to the particular tenant executing the VNF of that tenant; in this way, the orchestration system of the NFVI 102 modifies the resource allocation without violating the SLA by modifying the resource allocation while maintaining the expected quality of service. In some examples, the orchestration system of the NFVI 102 implements an optimal resource allocation in which the maximum capacity of computing resources is available for executing new VNFs and / or concurrently executing the maximum number of VNFs.

[0085] As described herein, the flow information captured by the sFlow agent 326A can be mined for flow statistics associated with tenants and tenant VNFs including the particular tenant and the particular tenant VNF (VNF 306A). Over a period of time, each tenant of the NFVI 102 and the server 302A receives a plurality of packet flows. In some examples, the orchestration system of the NFVI 102 can use the NFV management 304 to map each of the plurality of packet flow VNIs to a VNI space corresponding to the VNI 316 by correlating the flow statistics with the tenant information. In some examples, the orchestration system uses the sflow agent 326A to identify the VNI(s) assigned to the particular tenant from the tenant information and uses the assigned VNI(s) to identify the corresponding flow statistics having the same VNI(s).

[0086] The orchestration system uses the corresponding flow statistics to compute various statistics and other information data points. The corresponding flow statistics can indicate values that describe various variables of the packet flow, including total number of packets, total amount in bytes, packets per second, average packet size or distribution of packet size(s), and so on. In some examples, the orchestration system uses these flow statistics to compute an aggregate bandwidth of a plurality of flows associated with a particular tenant of the data center provider and processed by the VNF of the tenant executing on the server 302A, and then modifies the allocation of computing resources of the server 302A executing the VNF based on the aggregate bandwidth. In some examples, the allocation of computing resources refers to the allocation of physical (hardware) processor resources (e.g., processing circuitry) for executing the VNF.

[0087] In some examples, the orchestration system is configured to compute an average bandwidth of the (packet) flow through one VNI based on the corresponding flow statistics. For example, by aggregating the packet sizes and dividing by the total number of packets, the orchestration system can determine the average packet size from the corresponding flow statistics. As an alternative, the orchestration system can determine the average packet size from the corresponding flow statistics, for example, by extracting this average from the stored sflow datagrams. Once the average packet size is known, the orchestration system can compute the average bandwidth by multiplying the packets per second (PPS) by the average packet size, and compute the aggregate bandwidth as the sum of each average bandwidth of the plurality of packet flows. In another example, the orchestration system can compute the average bandwidth by dividing the total number of bytes by the amount of time in seconds, and compute the aggregate bandwidth as the sum of each average bandwidth of the plurality of packet flows.

[0088] In some examples, the orchestration system compares the bandwidth allocation to the aggregate bandwidth to produce a comparison result and (possibly) stores the comparison result for trend analysis. In some examples, to comply with the SLA obligation resources of the server 302A for executing the VNF, the orchestration system of the NFVI 102 can provide an allocation of computing resources that satisfies at least the minimum or sufficient resource requirement for executing the VNF. The orchestration system of the NFVI 102 can analyze the comparison result to determine whether to (or not to) modify the allocation of computing resources (e.g., while maintaining compliance with the SLA). If the comparison result indicates that the resource utilization of the VNF is insufficient (e.g., of computing resources or another resource (e.g., bandwidth)), the orchestration system of the NFVI 102 can modify the allocation by reducing the allocation of computing resources to a capacity that is optimal for the aggregate bandwidth. In some instances, the orchestration system of the NFVI 102 can break the SLA and allocate less resources than the allocation of computing resources.

[0089] In some examples, the orchestration system of the NFVI 102 repeatedly compares (e.g., after a time interval) and stores each comparison result for trend analysis. Over time, the comparison results develop into a relationship between the performance of the VNF (e.g., in terms of bandwidth utilization) and the allocation of computing resources to the VNF. For example, the orchestration system of the NFVI 102 can combine the (series of) aggregated bandwidth measurements into an aggregated bandwidth trend. The orchestration system of the NFVI 102 can modify the allocation of computing resources to the execution of the VNF based on the aggregated bandwidth trend. Assuming stable network traffic, the aggregated bandwidth trend eventually converges to a well-defined mapping that can be used when deploying the tenant VNF in another virtual execution environment in another NFVI. When the aggregated bandwidth changes (e.g., due to a change in level of network traffic), the orchestration system of the NFVI 102 performs a comparison between the current bandwidth allocation and the updated aggregated bandwidth to produce an updated comparison result for trend analysis. In some instances, the updated comparison result can indicate that sufficient bandwidth cannot be met and indicate an opportunity to modify the resource allocation by adding one or more computing resources (e.g., processor cores).

[0090] The orchestration system can perform the above comparisons with different resource configurations to determine a maximum achievable (aggregated) bandwidth trend of the allocation of computing resources. The orchestration system can store a number of mappings between different allocations of computing resources and corresponding maximum achievable (aggregated) bandwidth trends in a resource allocation table. The orchestration system can modify the allocation of computing resources based on the resource allocation table. In other examples, the orchestration system can use the resource allocation table for an initial allocation or a modified allocation of computing resources to execute the same or similar VNF in another NFVI or another data center.

[0091] Figure 4 is a flow diagram illustrating example operations for tenant-driven resource allocation based on actual resource usage, in accordance with the techniques described herein.

[0092] The (following) example operations can be performed by an orchestration system, a computing system that controls NFVIs in a data center, such as when the orchestration system identifies that a tenant's VNF bandwidth consumption (rate) fails to meet the tenant's VNF bandwidth allocation. In some aspects, the tenant's VNF can be implemented on a physical or virtual machine in the data center. The data center can include a gateway (e.g., a public or private gateway) that receives network packets from a network (e.g., a customer network or a cloud service network) with a destination address (e.g., a public IP address) of the tenant's VNF. If the tenant's customer rarely, if ever, needs the total VNF bandwidth allocation, the orchestration system can perform the following example operations, e.g., instruct a server hosting the tenant's VNF to increase or decrease the allocated resource capacity of the VNF.

[0093] The following operations are described with respect to Figure 1 The orchestrator 112 is described as an example orchestration system. In some examples, the orchestrator 112 determines the resource usage context by mapping the flow VNI to the tenant VNI (402). The tenant information 452 stores information associated with each tenant of the data center provider in the data center that is assigned one or more VNFs. The tenant information 452 also stores, for each tenant, virtual network identifiers of virtual networks that are configured for use with the flows associated with the tenant. Some of these virtual network identifiers for a tenant can be configured on servers of the NFVI to cause virtual switches of the servers to switch packets having those virtual network identifiers to the tenant VNFs. This set of virtual network identifiers is referred to as the VNF VNI space. The flow information 454 stores information describing a plurality of flows associated with each tenant of the data center provider, each flow referring to packets that are forwarded through the NFVI between endpoints, such as between a tenant customer and a service provider. Each of the tenant information 402 and the flow information 404 stores VNIs that can be used as an index to map each tenant to its corresponding flows. The tenant information 402 and the flow information 404 are combined so that each tenant VNI in the tenant information is mapped to each flow VNI in the flow information 404. A dynamic resource mapping database 456 can be created to store the mapping between the flow VNIs and the tenant VNIs.

[0094] If the tenant information 452 is organized into a database, an example entry in the database can be structured as follows:

[0095] Tenant 1 {

[0096] VNF Type: Router

[0097] VNF Vendor: CSCO

[0098] VNF License: Security

[0099] VNF Bandwidth: 5G

[0100] VNF VNI Space: 10-20

[0101] Public IP Address: 182.1.1.2 / 32

[0102] If the flow information 454 is organized into a database, an example entry in the database can be structured as follows:

[0103] Flow 1 {

[0104] Source Address: 10.18.91.100

[0105] Destination Address: 20.16.17.11

[0106] VNI: 10

[0107] Average Bandwidth: 100 Mbps

[0108] Flow 2 {

[0109] Source Address: 11.12.172.11

[0110] Destination Address: 33.192.11.1190

[0111] VNI: 12

[0112] AVG BW: 1 Gbps

[0113] To identify flows associated with a tenant, the network service orchestrator 112 maps the VNI of a flow (e.g., VNI: 10 for flow 1) to the VNF VNI space for the tenant (e.g., VNF VNI space: 10-20 for tenant 1). If the VNF VNI space for the tenant includes the VNI for a flow, then the flow is associated with the tenant. Multiple flows associated with a tenant are then aggregated based on the flow records to compute, for example, average packet size and average bandwidth. When the flow information 454 and tenant information 452 are combined into dynamic resource mapping database information 456, an example entry in the database can be structured as follows:

[0114] Tenant 1 {

[0115] Actual Resources: {

[0116] Actual BW: 10G

[0117] Cores: 8

[0118] Memory: 8 GB

[0119] Bandwidth Usage: {

[0120] Flows: 1000

[0121] Average Packet Size: 256

[0122] Average BW: 4.5G

[0123] Tenant 2 {

[0124] Actual Resources: {

[0125] Actual BW: 5G

[0126] Cores: 4

[0127] Memory: 4 GB

[0128] Bandwidth Usage: {

[0129] Flows: 560

[0130] Average packet size: 1000

[0131] Average BW: 2.5G

[0132] The orchestrator 112 determines the average packet size for all tenant flows (404). The flow information 454 stores packet flow statistics corresponding to the amount of data (in number of bytes) received at the network edge of the data center over a period of time. The network edge can be a switch point of a server cluster or a pod that includes virtualized resources for the NFVI. In some examples, the dynamic resource mapping database 456 stores the rate (e.g., in packets or bytes) at which data for a particular tenant is received and forwarded through the NFVI. From this rate, the orchestrator 112 calculates the average packet size, which can also be stored in the dynamic resource mapping database 456. In some examples, the dynamic resource mapping database 456 stores the average packet size for data at each VNI assigned to a particular tenant.

[0133] The orchestrator 112 determines the aggregate bandwidth (i.e., aggregate average bandwidth) for all flows associated with a tenant (406). As mentioned above, the dynamic resource mapping database 456 stores the rate (e.g., in packets or bytes) at which the VNFs for a particular tenant consume bandwidth over time and for data received at each VNI, which can represent the average rate for each packet flow; by combining the corresponding rates for each VNI, the orchestrator 112 determines the aggregate rate for data received at all VNIs assigned to that particular tenant.

[0134] In some examples, the orchestrator 112 creates a tenant bandwidth usage trend (408). In some examples, the orchestrator 112 plots data points representing the aggregate average bandwidth against the allocation of computing resources and stores the data points for trend analysis. The data points can associate the actual rate at which packet flows for a particular tenant are received and forwarded through the NFVI 102 with the number of processor cores allocated to such packet processing. Over time, the orchestrator 112 plots multiple data points that represent a trend between the allocation of processor cores and the achievable aggregate bandwidth for the VNFs of a particular tenant. In this way, the orchestrator 112 can calculate the maximum bandwidth rate possible for any VNF being executed in the NFVI 102 based on the allocation of processor cores (or another computing resource) for the VNF. The orchestrator 112 can store the maximum bandwidth rate and the allocation of processor cores for the VNF in a database such as the dynamic resource mapping database 456. The orchestrator 112 can use the trend to determine whether to allocate additional processor cores to the VNFs of a particular tenant. For example, if the trend indicates that the aggregate bandwidth for a particular tenant is approaching the maximum bandwidth rate possible for the VNFs of that tenant, the orchestrator 112 can allocate additional processor cores to the VNFs of that tenant to increase the aggregate bandwidth for that tenant. Figure 2the resource allocation table 208. Based on trends in the aggregate bandwidth of the flows associated with a tenant of the data center provider and processed by the tenant VNF, the orchestrator 112 can determine that more computing resources should be assigned to the tenant VNF (for trending higher aggregate bandwidth) or fewer computing resources (for trending lower aggregate bandwidth).

[0135] The number of processor cores that the orchestrator 112 is to add / remove to modify the resource allocation for a particular tenant (410). In some examples, the orchestrator 112 modifies the allocation of computing resources for executing the VNFs of a particular tenant. The orchestrator 112 can access the resource allocation table 208 and determine the modified allocation of computing resources. If the bandwidth utilization of a tenant VNF changes, as indicated by the aggregate bandwidth of the tenant flows processed by the tenant VNF, the orchestrator 112 can access the resource allocation table 208 and, from that table, modify the allocation of computing resources to the servers executing the VNFs of the particular tenant. For example, the orchestrator 112 can use the resource allocation table 208 to determine the number of processor cores to add or remove in order to meet the changed bandwidth utilization. In some examples, the orchestrator 112 issues a function call through the NFVI interface 420 to instruct the NFVI management to add or remove the number of processor cores to the servers executing the VNFs of the particular tenant.

[0136] Figure 5 is a block diagram illustrating a conceptual view of bandwidth usage based processor core management in accordance with one or more techniques of the present disclosure. In Figure 5 multiple data center tenants (or multiple tenants) initiate VNFs in the NFVI 102 running in the compute node 500. Each tenant is illustrated in Figure 5 as an oval shape with a letter “T” and a number. Each tenant is further illustrated in Figure 5 where one or two black circular shapes represent one or two processor cores and overlap the oval shape of the tenant. The compute node 500 can represent a physical compute node, or in some cases a virtual compute resource, which is an abstraction of a physical processing resource in a computer cluster (e.g., a pod). The virtual switch 502 is coupled to the virtual compute resource and uses the virtualized physical processing resource to process packet flows received via a virtual network resource of the compute node 500. In some examples, the virtual network resource can correspond to one or more VNIs.

[0137] Orchestration system 504 represents a functional block of NFVI 102 (e.g., a network service orchestrator). Generally, orchestration system 504 orchestrates resources (e.g., virtual resources) to instantiate VNFs to run network services for multiple tenants. In some examples, orchestration system 504 allocates virtual compute resources, such as one or more processor cores or other processing resources, to each VNF. In some examples, orchestration system 504 maintains a core reservation 506 to include one or more processor cores that are not available for VNF allocation and utilization. In some examples, orchestration system 504 also maintains a core pool 508 to include one or more processor cores that are available for VNF allocation and utilization. When compared to conventional NFVI implementations, Figure 5 FIGURE 1 illustrates an improved NFVI with an increased maximum number of available cores for allocation to VNFs. In Figure 5 In the example of FIGURE 2, additional cores between 2 and 4 are allocated to VNFs associated with a tenant of compute node 500.

[0138] The following table illustrates some examples of benefits realized from transitioning from a conventional resource orchestration technique (i.e., the present operational mode (PMO)) to the technique described herein (i.e., the future operational mode (FMO)):

[0139]

[0140]

[0141] The above table demonstrates that a technician in the NFVi can allocate additional available cores to increase the total number of VNFs executing at a particular moment in time. In the above table, “POD” refers to a cluster of servers hosting the improved NFVI described herein (e.g., servers 302 of FIGURE 1). Figure 3

[0142] Figure 6 is a block diagram illustrating further details of one example of a computing device operating in accordance with one or more techniques of this disclosure. Figure 6 A particular example of a server or other computing device 600 that can illustrate one or more processors 602 for executing any one or more of any of the systems, applications, or modules described herein can be illustrated. For example, one or more processors 602 can execute instructions of an orchestration system 620 to instantiate and deploy VNFs to a NFVI and apply the techniques described herein to determine and configure compute resource allocations for VNFs executing on servers of the NFVI. As such, computing device 600 can represent an example instance of network service orchestrator 112 or other orchestration system or other system to configure resource allocations for VNFs executing on servers of a NFVI in accordance with techniques of this disclosure. Other examples of computing device 600 can be used in other instances. While the following description is made with respect to a server, other examples can be used in other instances.​Figure 6 The device is shown as a standalone computing device 600, but a computing device can be any component or system including one or more processors or other suitable computing environments for executing software instructions, and does not necessarily need to include, for example, a... Figure 6 One or more components are shown (e.g., communication unit 606; and in some examples, components such as (multiple) storage devices 608 may not coexist with other components or be in the same chassis).

[0143] like Figure 6 As shown in a specific example, computing device 600 includes one or more processors 602, one or more input devices 604, one or more communication units 606, one or more output devices 612, one or more storage devices 608, and a user interface (UI) device 610 and communication unit 606. In one example, computing device 600 also includes one or more applications 622, multiple programmable network platform applications 624, and an operating system 616 executable by computing device 600. Each of components 602, 604, 606, 608, 610, and 612 is (physically, communicatively, and / or operatively) coupled for inter-component communication. In some examples, communication channel 614 may include a system bus, network connection, inter-process communication data structure, or any other method for transmitting data. As an example, components 602, 604, 606, 608, 610, and 612 may be coupled via one or more communication channels 614.

[0144] In one example, processor 602 is configured to implement functional and / or processing instructions for execution within computing device 600. For example, processor 602 may be able to process instructions stored in storage device 608. Examples of processor 602 may include any one or more of a microprocessor, controller, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or equivalent discrete or integrated logic circuitry.

[0145] The one or more storage devices 608 can be configured to store information within computing device 600 during operation. In some examples, the storage device 608 is described as a computer-readable storage medium. In some examples, the storage device 608 is a temporary memory, meaning that a primary purpose of the storage device 608 is not long-term storage. In some examples, the storage device 608 is described as a volatile memory, meaning that the storage device 608 does not maintain stored contents when the computer is off. Examples of volatile memories include random access memories (RAM), dynamic random access memories (DRAM), static random access memories (SRAM), and other forms of volatile memory known in the art. In some examples, the storage device 608 is used to store program instructions for execution by the processor 602. In one example, the storage device 608 is used by software or applications running on the computing device 600 to temporarily store information during program execution.

[0146] In some examples, the storage device 608 also includes one or more computer- readable storage media. The storage device 608 can be configured to store larger amounts of information than a volatile memory. The storage device 608 can further be configured for long-term storage of information. In some examples, the storage device 608 includes non-volatile storage elements. Examples of such non-volatile storage elements include magnetic hard discs, optical discs, floppy discs, flash memories, or forms of electrically programmable memories (EPROM) or electrically erasable and programmable (EEPROM) memories.

[0147] In some examples, the computing device 600 also includes one or more communication units 606. In one example, the computing device 600 utilizes the communication unit 606 to communicate with external devices via one or more networks, such as one or more wired / wireless / mobile networks. The communication unit 606 can include a network interface card, a modem, or any other type of device capable of transmitting and receiving information. In some examples, the computing device 600 uses the communication unit 606 to communicate with external devices.

[0148] In one example, the computing device 600 also includes one or more user interface devices 610. In some examples, the user interface device 610 is configured to receive input from a user through tactile, audio, or video feedback. Examples of the user interface device(s) 610 include a presence-sensitive display, a mouse, a keyboard, a voice responsive system, video game controller, camera, microphone or any other type of device for detecting a command from a user. In some examples, the presence-sensitive display includes a touch-sensitive screen.

[0149] One or more output devices 612 can also be included in computing device 600. In some examples, output devices 612 are configured to provide output to a user using tactile, audio, or video stimuli. In one example, output devices 612 include a presence-sensitive display, a sound card, a video graphics adapter card, or any other type of device for converting a signal into an appropriate form understandable to humans or machines. Other examples of output devices 612 include speakers, a cathode ray tube (CRT) monitor, a liquid crystal display (LCD), or any other type of device that can generate an understandable output to a user.

[0150] Computing device 600 can include an operating system 616. In some examples, operating system 616 controls the operation of components of computing device 600. For example, in one example, operating system 616 facilitates communication of one or more applications 618 with processor 602, communication unit 606, storage device 608, input device 604, user interface device 610, and output device 612.

[0151] Applications 618 and orchestration system 620 can also include program instructions and / or data that are executable by computing device 600. Further, as an example, orchestration system 620 can include software that implements orchestrator 112 of Figure 1 As indicated by orchestration system 620 running in a data center in which computing device 600 is resident, a maximum number of VNFs are instantiated and concurrently executed. This can be achieved in part by implementing the techniques described herein. As one example, orchestration system 620 can apply the techniques described herein to determine an allocation of computing resources of servers for executing virtual network functions, and configure an NFVI having multiple processor cores of the servers in accordance with the allocation (via a coupled-to NFVI management system, e.g., NFVI management 304 of Figure 3 As one example, computing device 600 uses information stored in memory (e.g., resource allocation table 208 of Figure 2 to determine a number of processor cores to allocate.

[0152] The techniques described herein can be implemented in hardware, software, firmware, or any combination thereof. Various features described as modules, units or components can be implemented together in an integrated logic device, or separately as discrete but interoperable logic devices or other hardware devices. In some cases, various features of electronic circuitry can be implemented as one or more integrated circuit devices, such as integrated circuit chips or chip sets.

[0153] If implemented in hardware, the present disclosure can involve the use of one or more apparatus such as a processor or an integrated circuit device, such as an integrated circuit chip or chipset. Alternatively or additionally, if implemented in software or firmware, the techniques can at least partly be implemented by computer-readable data storage medium comprising instructions that, when executed by a processor, cause the processor to carry out one or more of the methods described above. For example, the computer-readable data storage medium can store the instructions that are executed by the processor.

[0154] The computer-readable medium can form part of a computer program product, which can include packaging material. The computer-readable medium can include a computer data storage medium such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), FLASH memory, magnetic or optical data storage media, and the like. In some examples, the article of manufacture can include one or more computer-readable storage media.

[0155] In some examples, the computer-readable storage media can include non-transitory media. The term "non-transitory" can indicate that the storage medium is not embodied in a carrier wave or a propagated signal. In certain examples, a non-transitory storage medium can store data that can, over time, change (e.g., in RAM or cache).

[0156] Code or instructions can be software and / or firmware executed by processing circuitry including one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Accordingly, the term "processor" as used herein can refer to any of the foregoing structure or any other structure suitable for implementation of the techniques described herein. In addition, in some aspects, the functionality described herein can be provided within software modules or hardware modules within the devices described herein.

Claims

1. An orchestration system operated by a data center provider of a data center, the orchestration system comprising: a processing circuit coupled to a memory; software instructions stored in the memory and configured for execution by the processing circuit to: map a packet flow virtual network interface (VNI) to a VNI space of a plurality of tenants of the data center, wherein a tenant of the data center is associated with a first VNI of the VNI space and a second VNI of the VNI space; based on the first and second VNIs in the mapping, identify flow statistics corresponding to the tenant; based on the flow statistics indicating a first packet flow having the first VNI associated with the tenant and a second packet flow having the second VNI associated with the tenant, compute an aggregate bandwidth of a plurality of packet flows associated with the tenant and processed by a virtual network function (VNF), the VNF assigned to the tenant and executing on a physical computing resource of a server; and modify an allocation of the physical computing resource of the server for executing the virtual network function based on the aggregate bandwidth.

2. The orchestration system of claim 1, wherein the software instructions are further configured for execution by the processing circuit to compute an average bandwidth for each of the first and second packet flows based on the flow statistics.

3. The orchestration system of claim 1, wherein the software instructions are further configured for execution by the processing circuit to compare an allocation of bandwidth to the aggregate bandwidth to produce a comparison result and store the comparison result for trend analysis.

4. The orchestration system of claim 1, wherein the software instructions are further configured for execution by the processing circuit to compute a bandwidth utilization based on an average packet size of one or more packet flows of the plurality of flows.

5. The orchestration system of claim 1, wherein the software instructions are further configured for execution by the processing circuit to modify the allocation of the physical computing resource based on an aggregate bandwidth trend.

6. The orchestration system of claim 1, wherein the software instructions are further configured for execution by the processing circuit to modify the allocation of the physical computing resource based on a resource allocation table.

7. The orchestration system of claim 1, wherein the software instructions are further configured for execution by the processing circuit to: compute an average bandwidth by multiplying a number of packets per second by an average packet size; and compute the aggregate bandwidth as a sum of each average bandwidth of the plurality of packet flows.

8. The orchestration system of claim 1, wherein the software instructions are further configured for execution by the processing circuit to: compute an average bandwidth by dividing a total number of bytes by an amount of time in seconds; and compute the aggregate bandwidth as a sum of each average bandwidth of a plurality of packet flows.

9. A method of an orchestration system operated by a data center provider of a data center, the method comprising: mapping packet flow virtual network interfaces (VNIs) to VNI spaces of a plurality of tenants of the data center, wherein a tenant of the data center is associated with a first VNI of the VNI spaces and a second VNI of the VNI spaces; identifying flow statistics corresponding to the tenant based on the first and second VNIs in the mapping; computing an aggregate bandwidth of a plurality of packet flows associated with the tenant and processed by a virtual network function (VNF) based on the flow statistics indicating a first packet flow having the first VNI associated with the tenant and a second packet flow having the second VNI associated with the tenant, the VNF assigned to the tenant and executing on physical computing resources of a server; and modifying an allocation of the physical computing resources of the server executing the virtual network function based on the aggregate bandwidth for executing the VNF.

10. The method of claim 9, wherein calculating the aggregated bandwidth further comprises: computing an average bandwidth for each of the first and second packet flows based on the flow statistics.

11. The method of claim 9, wherein modifying the allocation of physical computing resources further comprises: comparing the allocation of bandwidth to the aggregate bandwidth to produce a comparison result and storing the comparison result for trend analysis.

12. The method of claim 9, wherein calculating the aggregated bandwidth further comprises: determining an average packet size based on the flow statistics.

13. The method of claim 9, wherein modifying the allocation of physical computing resources further comprises: modifying the allocation of the physical computing resources based on an aggregate bandwidth trend.

14. The method of claim 9, wherein modifying the allocation of physical computing resources based on the aggregated bandwidth further comprises: modifying the allocation of the physical computing resources based on a resource allocation table.

15. The method of claim 9, wherein computing the aggregate bandwidth further comprises: computing an average bandwidth by multiplying a number of packets per second by an average packet size; and computing the aggregate bandwidth as a sum of each average bandwidth of the plurality of packet flows.

16. The method of claim 9, wherein computing the aggregate bandwidth further comprises: computing an average bandwidth by dividing a total number of bytes by an amount of time in seconds; and computing the aggregate bandwidth as a sum of each average bandwidth of the plurality of packet flows.

17. An interconnection system comprising: at least one interconnection facility and a programmable network platform, the at least one interconnection facility comprising: a cluster comprising one or more computing devices to host virtual network functions (VNFs) for tenants, the programmable network platform configured to: map packet flow virtual network interfaces (VNIs) to VNI spaces of a plurality of tenants of the at least one interconnection facility, wherein a tenant of the at least one interconnection facility is associated with a first VNI of the VNI spaces and a second VNI of the VNI spaces; identify flow statistics corresponding to the tenant based on the first and second VNIs in the mapping; compute an aggregate bandwidth of a plurality of packet flows associated with the tenant and processed by a virtual network function (VNF) based on the flow statistics indicating a first packet flow having the first VNI associated with the tenant and a second packet flow having the second VNI associated with the tenant, the VNF assigned to the tenant and executing on physical computing resources of a server; and modify an allocation of the physical computing resources of the server executing the virtual network function based on the aggregated bandwidth for execution of the VNF.

Images