Tenant-driven dynamic resource allocation for virtual network functions

Abstract

Techniques for tenant-driven dynamic resource allocation in a network function virtualization infrastructure (NFVI). In one example, an orchestration system is operated by a data center provider of a data center, and the orchestration system includes processing circuitry coupled to a memory; logic stored in the memory and configured to be executed by the processing circuitry, wherein the logic is operable to: compute an aggregate bandwidth for a plurality of streams associated with tenants of a datacenter provider and processed by virtual network functions assigned to the tenants executing on servers of the datacenter; and modifying the allocation of computing resources for the server executing the virtual network function based on the aggregated bandwidth.

Description

Related application citation

[0001] This application is a divisional application of the invention patent application with international application number PCT / US2020 / 067710, international application date of December 31, 2020, entry into the Chinese national phase date of November 28, 2022, Chinese national application number 202080101492.7, and invention title "Tenant-Driven Dynamic Resource Allocation for Virtual Network Functions".

[0002] This application claims the benefit of U.S. Application No. 16 / 888,280, filed May 29, 2020, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to computer networks, and more specifically to virtual network functions provided on computer networks. Summary of the Invention

[0004] Virtualized data centers are becoming a core foundation of modern information technology (IT) infrastructure. Specifically, modern data centers have widely utilized virtualized environments, where virtual machines, also referred to in this paper as virtual execution elements such as virtual machines or containers, are deployed and executed on the underlying computing platform of physical computing devices. Virtualization within the data center offers several advantages. One advantage is that virtualization can provide significant improvements in efficiency. With the advent of multi-core microprocessor architectures with a large number of cores per physical CPU, the underlying physical computing devices (i.e., servers) have become increasingly powerful, and the benefits of virtualization have increased significantly. Another advantage is that virtualized network functions can be performed by servers, switches, storage devices, and cloud computing infrastructure, rather than by custom hardware devices for each network function.

[0005] Network operators can provide a Network Functions Virtualization (NFV) architecture or Network Functions Virtualization Infrastructure (NFVI) capable of orchestrating and managing multiple Virtualized Network Functions (VNFs), or a physical device that applies network functions (or "services") to packet flows in an ordered sequence. Example VNF instances can provide firewalls, routing / switching, Carrier-Grade Network Address Translation (CG-NAT), video performance enhancement proxies, Transmission Control Protocol (TCP) optimization and header enrichment, caching, load balancing, or other network functions. To implement service chaining, one or more virtual networks are used to interconnect VNF ​​instances, forwarding packet flows along an ordered sequence for the applications constituting the service chain. In response to increased load, the service in the service chain applied by one or more VNF instances can be scaled up proportionally by spawning more VNF instances. Similarly, in response to decreased load, the service can be scaled down proportionally by removing one or more VNF instances spawned for that service. One or more VNF instances for a single service can be hosted by separate compute devices (e.g., compute nodes or servers), but a compute device can host multiple VNF instances for one or more services. Summary of the Invention

[0006] Generally, this disclosure describes a technique for a network service orchestrator for Network Functions Virtualization Infrastructure (NFVI) to dynamically allocate computing resources among virtual network functions assigned to tenants based on tenant flow statistics. For example, an NFVI provider may deploy an NFVI hosting VNFs, which are assigned to multiple different customer tenants (more simply referred to as tenants or customers) of the NFVI provider. Each VNF performs its corresponding network function to process packet flows associated with one of the tenants of the NFVI provider to which the VNF was assigned. As described herein, the network service orchestrator for NFVI dynamically distributes physical resources, such as multiple processing cores in the NFVI server, among the VNFs running on the server based on the actual utilization of the VNFs to process the flows associated with the tenants assigned to the VNFs.

[0007] The described techniques can provide one or more technical advantages that enable at least one practical application. For example, the techniques described herein can improve computing resource utilization by allowing server computing resources to execute more VNFs through on-demand resource allocation, while still satisfying the Tenant Service Level Agreement (SLA) for streaming processing. For example, when the actual bandwidth utilization of a tenant allocated to a VNF, as determined by tenant flow statistics, does not reach the bandwidth allocated to the VNF (and the corresponding processing resource allocation under a static allocation scheme), there is an envelope of unused processing resources dynamically reassigned to one or more other VNFs. Thus, implementing these techniques described herein can increase the maximum number of VNFs that can be executed on the server at one time.

[0008] As another advantage, these technologies allow tenant-driven dynamic resource allocation by an orchestration system in the data center, rather than from components within the same server as the tenant's VNF. The orchestration system provides numerous benefits for resource allocation, including access to tenant flow statistics for multiple VNFs scattered across NFVIs and assigned to multiple different tenants. At least for this reason, such dynamic resource allocation can be performed even when a tenant's client equipment is not in the same data center environment (e.g., interconnection facility) as the NFVI on which the tenant's VNF is running.

[0009] In one example, an orchestration system operated by a data center provider includes: processing circuitry coupled to a memory; and logic stored in the memory and configured for execution by the processing circuitry, wherein the logic is operable to: calculate aggregate bandwidth of multiple flows associated with tenants of the data center provider and processed by virtual network functions, the virtual network functions assigned to the tenants being executed on servers in the data center; and modify the allocation of computing resources of the servers used to execute the virtual network functions based on the aggregate bandwidth.

[0010] In one example, a method for an orchestration system operated by a data center provider includes: calculating aggregate bandwidth of multiple flows associated with tenants of the data center provider and processed by virtual network functions, the virtual network functions assigned to the tenants being executed on servers in the data center; and modifying the allocation of computing resources of the servers executing the virtual network functions based on the aggregate bandwidth.

[0011] In one example, an interconnection system includes: at least one interconnection facility and a programmable network platform, the at least one interconnection facility comprising: a cluster including one or more computing devices for hosting virtual network functions (VNFs) for tenants, wherein each tenant corresponds to one or more VNFs forming a virtual router, through which packet flows are exchanged between services and clients, the programmable network platform being configured to: calculate aggregate bandwidth of multiple flows associated with one tenant and processed by one VNF, the VNFs assigned to the tenants executing on the computing devices of the cluster; and modify the allocation of computing resources of the computing devices executing the virtual network functions based on the aggregate bandwidth.

[0012] Details of one or more examples are set forth in the accompanying drawings and the following description. Other features, objects, and advantages will be apparent from the specification and drawings, as well as from the claims. Attached Figure Description

[0013] Figure 1 This is a block diagram illustrating a conceptual view of a network system with metropolitan area-based cloud switching providing multiple cloud switching points and network function virtualization infrastructure, based on the technology described herein.

[0014] Figure 2 This is a block diagram illustrating an example data center providing an operating environment with tenant-driven resource allocation for a VNF according to one or more aspects of the technology described in this disclosure.

[0015] Figure 3 This is a block diagram illustrating an example network function virtualization infrastructure with tenant-driven resource allocation according to the technology described herein.

[0016] Figure 4 This is a flowchart illustrating an example operation of tenant-driven resource allocation based on actual resource usage, according to the technology described herein.

[0017] Figure 5 This is a block diagram illustrating further details of an example of a computing device operating according to one or more techniques of the present disclosure.

[0018] Figure 6 This is a block diagram illustrating a conceptual view of processor core management based on bandwidth usage according to one or more technologies disclosed herein.

[0019] In all the accompanying drawings and text, the same reference numerals denote the same elements. Detailed Implementation

[0020] Figure 1This is a block diagram illustrating a conceptual view of a network system with a metropolitan area-based cloud exchange providing multiple cloud exchange points and with network function virtualization infrastructure, according to the technology described herein. Each cloud-based service exchange point 120A-120C (hereinafter described as a "cloud exchange point" and collectively referred to as "cloud exchange point 120") of the cloud-based service exchange 100 ("cloud exchange 100") can represent different data centers (or interconnection facilities) geographically located within the same metropolitan area ("metropolitan area-based," e.g., New York City, New York; Silicon Valley, California; Seattle-Tacoma, Washington; Minneapolis-St. Paul, Minnesota; London, United Kingdom; etc.) to provide resilient and independent cloud-based service exchange, through which cloud-based service customers ("cloud customers") and cloud-based service providers ("cloud providers") connect to receive and provide cloud services, respectively. In various examples, cloud exchange 100 may include more or fewer cloud exchange points 120. In some instances, cloud exchange 100 includes only one cloud exchange point 120. As used herein, references to “cloud exchange” or “cloud-based service exchange” may refer to a cloud exchange point. A cloud exchange provider may deploy instances of cloud exchange 100 in multiple different metropolitan areas, each instance of cloud exchange 100 having one or more cloud exchange points 120.

[0021] Any one or more of the cloud exchange points 120 can be used to at least partially implement NFVI 102. Each cloud exchange point 120 includes network infrastructure and an operating environment through which cloud customers 108A-108C (collectively, "cloud customers 108") receive cloud services from multiple cloud service providers 110A-110N (collectively, "cloud service providers 110"). Cloud service providers 110 can host one of the multiple cloud services. As noted above, cloud service providers 110 can be public or private cloud service providers.

[0022] Cloud Exchange 100 provides secure, private, and virtual connections to multiple cloud service providers (CSPs) for its customers (such as enterprises, network operators, network service providers, and SaaS customers). Multiple CSPs participate in cloud exchange by having at least one accessible port within the exchange, through which customers can connect to one or more cloud services provided by each CSP. Cloud Exchange 100 allows any customer's private network to directly cross-connect to any other customer at a common point, thus enabling direct exchange of network traffic between customer networks.

[0023] Cloud customer 108 may receive cloud-based services directly via a Layer 3 peering and physical connection with one of the cloud exchange points 120, or indirectly via one of the network service providers 106A-106B (collectively, “NSP 106”, or alternatively, “Carrier 106”). Cloud customer 108 may include customers associated with VNF 306A as described above. For example, cloud customer 108 may include systems used by any or all customer devices used by cloud client 118 to access cloud services via a VNF implemented in NFVI 102 of cloud exchange point 120. NSP 106 provides “cloud transit” by maintaining a physical presence within one or more of the cloud exchange points 120 and aggregating Layer 3 access from one or more customers 108. NSP 106 may peer directly with one or more cloud exchange points 120 at Layer 3 and, in doing so, provide indirect Layer 3 connectivity and peering to one or more customers 108 through which customers 108 can obtain cloud services from cloud exchange 100. Figure 1 In the example, each cloud exchange point 120 is assigned a different Autonomous System Number (ASN). For example, cloud exchange point 120A is assigned ASN 1, cloud exchange point 120B is assigned ASN 2, and so on. Therefore, each cloud exchange point 120 is the next hop in the path vector routing protocol (e.g., BGP) path from cloud service provider 110 to customer 108. As a result, although each cloud exchange point 120 is not a transit network with one or more WAN links and accompanying Internet access and transit policies, each cloud exchange point 120 can peer with multiple different autonomous systems via external BGP (eBGP) or other external gateway routing protocols to exchange, aggregate, and route service traffic from one or more cloud service providers 110 to customers. In other words, cloud exchange point 120 can internalize the eBGP peering relationship that cloud service provider 110 and customer 108 will maintain on a pairwise basis. Conversely, customer 108 can configure a single eBGP peer with cloud exchange point 120 and receive multiple cloud services from one or more cloud service providers 110 via cloud exchange. While this document primarily describes eBGP or other Layer 3 routing protocol peering between cloud exchange points and customer, NSP, or cloud service provider networks, cloud exchange points can learn routes from these networks in other ways, such as through static configuration or via Routing Information Protocol (RIP), Open Shortest Path First (OSPF), Intermediate System to Intermediate System (IS-IS), or other routing distribution protocols.

[0024] As an example above, customer 108C is illustrated as having subscribed to a cloud exchange provider of cloud exchange 100 to directly access Layer 3 cloud services via cloud exchange point 120C. In this way, customer 108C, for example, receives redundant Layer 3 connectivity from cloud service provider 110A. In contrast, customer 108C is illustrated as having subscribed to cloud exchange provider 100 of cloud exchange to directly access Layer 3 cloud services via cloud exchange point 120C, and also subscribed to NSP 106B to access Layer 3 cloud services via the transit network of NSP 106B. Customer 108B is illustrated as having subscribed to multiple NSPs 106A and NSP 106B to have redundant cloud access to cloud exchange points 120A and 120B via the respective transit networks of NSPs 106A and NSP 106B. The aforementioned contract is instantiated in the network infrastructure of cloud exchange point 120 through L3 peering configuration within NSP 106 and the switching equipment of cloud exchange point 120, and L3 connections (e.g., layer 3 virtual circuits) established within cloud exchange point 120 to interconnect cloud service provider 110 network to NSP 106 network and customer 108 network, all of which have at least one port providing connectivity within one or more cloud exchange points 120.

[0025] In some examples, cloud exchange 100 allows a corresponding customer of any network service provider (NSP) or "operator" 106A-106B (collectively, "operator 106"), such as customer 108A, customer 108B, or other cloud customers including customer 108C, to directly connect to any other customer network and / or CSP 110 via a virtual Layer 2 (L2) or Layer 3 (L3) connection, thereby allowing direct exchange of network services between the customer network and CSP 110. The virtual L2 or L3 connection may be referred to as a "virtual circuit".

[0026] Operator 106 may each represent a network service provider associated with the switching network through which network subscribers of Operator 106 can access cloud services provided by CSP 110 via cloud exchange 100. Generally, customers of CSP 110 may include network operators, large enterprises, managed service providers (MSPs), and customers of Software as a Service (SaaS), Platform as a Service (aaS), Infrastructure as a Service (IaaS), Virtualization as a Service (VaaS), and Data Storage as a Service (DSaaS) services provided by CSP 110 via cloud exchange 100.

[0027] In this way, cloud exchange 100 streamlines and simplifies the collaboration process between CSP 110 and customers in a transparent and neutral manner (via operator 106 or directly). An example application of cloud exchange 100 is co-location and interconnection of data centers, where CSP 110 and operator 106 and / or customer 108 may already have a network presence, such as representing any cloud exchange point 120 by having one or more accessible ports within the data center available for interconnection. This allows participating operators, customers, and CSPs to have a wide range of interconnection options within the same facility. In this way, operators / customers can have the option to create many-to-many interconnections with only a single hookup to one or more cloud exchange points 120. In other words, cloud exchange 100 allows customers to interconnect to multiple CSPs and cloud services, rather than having to establish separate connections on a transit network to access different cloud service providers or different cloud services from one or more cloud service providers.

[0028] Cloud exchange 100 includes a programmable network platform 104 for dynamically programming the cloud exchange 100 to responsively and reliably fulfill service requests that encapsulate business requirements for services provided by the cloud exchange 100 and / or cloud service providers 110 coupled to the cloud exchange 100. The programmable network platform 104 may include a network service orchestrator 112 that processes tenant (e.g., cloud client) requests for VNF deployments. For example, the network service orchestrator 112 may organize, bootstrap, and integrate underlying services via virtual machines (or containers) and other software and network subsystems for managing various services (e.g., VNF deployments). As a result, the programmable network platform 104 can orchestrate business-grade services across heterogeneous cloud service providers 110, based on well-defined service policies, quality of service policies, service level agreements and costs, and further, based on the service topology used for business-grade services.

[0029] The hardware and / or software components of NFVI 102 implement a network management and resource orchestration system, at least one of which generally performs at least one of the techniques described herein. In one example, the network management and resource orchestration system forms an architecture with at least three functional blocks. One example functional block, the orchestration system, is responsible for onboarding new network service (NS) and virtual network function (VNF) encapsulation packages; NS lifecycle management; global and local resource management; and authentication and authorization of network function virtualization infrastructure (NFVI) resource requests. In some examples, the network service orchestrator 112 in the programmable network platform 104 is an example of the orchestration system, while in other examples, the orchestration system instructs (e.g., via function calls) the network service orchestrator 112 to dynamically allocate resources (e.g., compute resources). Other functional blocks (such as the management block) oversee the lifecycle management of VNF instances; fill in the coordination and adaptation roles of configuration and event reporting between the NFVI infrastructure and the component / network management system; and control and manage NFVI compute, storage, and network resources. Although shown separately as part of the programmable network platform 104, these management blocks can reside in NFVI 102 and work with the orchestration system when deploying VNFs.

[0030] NFVI 102 includes one or more servers 123A-123N (server 123) for executing / hosting virtual network functions (VNFs) 124A-124N that apply network services to packet flows. Network service orchestrator 112 handles the deployment and organization of these network services, for example, by instantiating VNFs on server 123 to execute these network services.

[0031] The programmable network platform 104 enables cloud service providers 110, which manage cloud exchange 100, to dynamically configure and manage cloud exchange 100, for example, to facilitate virtual connections for cloud-based service delivery from multiple cloud service providers 110 to one or more cloud customers 108. Cloud exchange 100 can enable cloud customers 108 to bypass the public internet to connect directly to cloud service providers 110, thereby improving performance, reducing costs, increasing connection security and privacy, and leveraging cloud computing for additional applications. In this way, for example, enterprises, network operators, and SaaS customers can integrate cloud services with their internal applications at least in some respects, as if such services were part of their own data center network or as if they were directly coupled to their own data center network.

[0032] In other examples, the programmable network platform 104 enables cloud service providers to configure cloud exchange 100 using L3 instances requested by cloud customer 108, as described herein. For example, customer 108 may request L3 instances to link multiple cloud service providers (e.g., for transferring customer data between two cloud service providers, or for obtaining a service mesh from multiple cloud service providers).

[0033] Programmable network platform 104 may represent an application running within one or more data centers of cloud exchange 100, or alternatively, an application running off-site in a cloud provider's (e.g., a back office or branch office). Programmable network platform 104 may be distributed wholly or partially within data centers, each associated with a different cloud exchange point 120 to form cloud exchange 100. Although shown as managing a single cloud exchange 100, programmable network platform 104 may control the provisioning of services for multiple different cloud exchanges. Alternatively or additionally, multiple individual instances of programmable network platform 104 may control the provisioning of services for corresponding multiple different cloud exchanges.

[0034] In the illustrated example, the programmable network platform 104 includes a service interface (or “service API”) 114, which defines how applications 111 (such as a customer portal) can invoke methods, fields, and / or other software primitives of the programmable network platform 104. The service interface 114 can allow operators 106, customers 108, cloud service providers 110, and / or cloud exchange providers to programmably access the capabilities and assets of the cloud exchange 100 according to the technologies described herein.

[0035] For example, service interface 114 can facilitate machine-to-machine communication to enable the dynamic provisioning of virtual circuits in cloud exchange for interconnecting customer and / or cloud service provider networks. In this way, programmable network platform 104 automates various aspects of cloud service provisioning. For example, service interface 114 can provide customers with an automated and seamless way to establish, offload, and manage interconnections between multiple different cloud providers participating in cloud exchange.

[0036] Further examples of cloud-based service exchange can be found in the following documents: U.S. Patent Application No. 15 / 099,407, entitled “CLOUD-BASED SERVICES EXCHANGE”, filed April 14, 2006; U.S. Patent Application No. 14 / 927,451, entitled “INTERCONNECTION PLATFORM FOR REAL-TIME CONFIGURATION AND MANAGEMENT OF A CLOUD-BASED SERVICES EXCHANGE”, filed October 29, 2005; and U.S. Patent Application No. 14 / 927,106, entitled “ORCHESTRATION ENGINE FOR REAL-TIME CONFIGURATION AND MANAGEMENT OF INTERCONNECTIONS WITHIN A CLOUD-BASED SERVICES EXCHANGE”, filed October 29, 2005; each of which is incorporated herein by reference in its entirety.

[0037] Customer 108B represents (or includes) the tenant network of cloud exchange 100. Customer 108B exchanges packetized data in packet streams with one or more other networks, for example, via virtual circuits or other connections through cloud exchange point 120A. In some cases, NFVI 102 applies one or more network services to packet streams on behalf of the tenant associated with customer 108.

[0038] In the illustrated example, customer 108B exchanges packets of packet stream 125A with cloud service provider network 110A and packets of packet stream 125B with cloud service provider network 110N. One or more VNFs 124A, executed by server 123A, apply network functions (in some cases as part of a network service) to packets of packet streams 125A-125B (collectively referred to as “packet stream 125”). This VNF may be referred to as a tenant VNF ​​because it is assigned to or associated with a tenant of NFVI 102 provider (which is also cloud exchange provider 100 in this case). In other words, the tenant VNF ​​processes packet stream 125 associated with the tenant. The tenant VNF ​​may be leased or sold to the tenant by NFVI 102 provider, or the tenant may deploy the tenant VNF ​​to server 123A or its virtual execution element. The virtual execution element may include a virtual machine or a container. Tenants may invoke service interface 114 or otherwise request a service chain in NFVI 102, which includes tenant VNFs for packet flows destined for and / or originating from client 108B. Server 123A may execute multiple VNFs 124A associated with multiple different tenants, each VNF in VNF 124A processing one or more packet flows associated with the tenant for that VNF.

[0039] According to the technology disclosed herein, a network service adapter 112 for NFVI 102 dynamically allocates physical resources, such as multiple processing cores in each server 123, among VNFs 124 based on the actual utilization of VNFs 124, to process flows associated with tenants associated with VNFs 124. For example, the network service adapter 112 generates flow statistics for server 123A indicating the bandwidth of each flow 125 forwarded by server 123A. That is, server 123A performs flow monitoring of flows passing through server 123A and generates flow records 127 for the flows. The network service adapter 112 collects flow records that can be used to calculate the estimated bandwidth of each flow, as well as other flow attributes, such as the virtual network interface of server 123A used by the flow, and the source and destination network addresses for the flow. Using the collected flow records, the network service adapter 112 calculates the estimated bandwidth for each flow 125. Furthermore, the network service adapter 112 determines, based on flow records, that flow 125 is associated with customer 108B, a tenant acting as an NFVI provider. The network service adapter 112 can then combine this calculated data into tenant flow statistics.

[0040] In the example where the packet stream is encapsulated in frames transmitted according to the User Datagram Protocol (UDP), the following steps are performed to calculate the estimated bandwidth: 1) Assuming the sFlow sampling rate is 0.25%, then 50,000 groups are samples, and the total number of groups / frames on the line is 20,000,000. 2) Assuming each packet carries 1518 bytes, the total number of bytes sent on the line is calculated as 20,000,000 × 1518 = 30,360,000,000. 3) The bits for converting bytes into the total number of bits are calculated as 30360,000,000 × 8. 4) Divide the total number of bits by the average delay measured on the link to obtain the total bandwidth utilization. For example, if the delay is sixty (60) seconds, then the total bandwidth utilization is calculated as (30360,000,000 × 8) / 60 or 4048000000 bits / second (i.e., 4,048 megabits per second (Mbps)).

[0041] In the example of transmitting packet streams according to the Transmission Control Protocol (TCP), the following steps are performed to calculate the estimated bandwidth: 1) Convert the TCP window size from bytes to bits. 2) Use the standard 64KB TCP window size, where 64KB = 65536 bytes. The TCP window size in bits will be calculated as 65536. 8 = 524288 bits. 3) After converting the TCP window to bits, divide the window by the link's round-trip time in seconds. If the round-trip time is thirty (30) milliseconds, the bandwidth utilization can be calculated as 524288 bits / 0.030 seconds = 17476266 bits per second throughput or 17.4 Mbps maximum possible throughput.

[0042] Tenant flow statistics typically include measured or calculated packet flow information (e.g., packet size, packets per second, estimated bandwidth, etc.) for each of one or more tenants of the NFVI 102 provider. As described above, the network service arranger 112 can map tenant information to flow records to determine which flows are associated with at least one specific tenant. As described herein, tenant flow statistics can indicate the actual aggregate bandwidth handled by a tenant VNF ​​for the tenant, and this information can be compared with resource allocation information to dynamically allocate resources. For example, in response to determining that the aggregate bandwidth handled by a tenant VNF ​​124A does not meet bandwidth allocation / requirements, the techniques described herein can modify the allocation of computer resources on server 123A, such as the number of processor cores or the amount of memory allocated to the tenant VNF.

[0043] In this way, by determining the rate at which server 123A forwards each packet flow 125 associated with a tenant, this rate correspondingly indicates the rate at which the tenant VNF ​​processes packet flows 125 (e.g., the aggregate bandwidth or throughput of the tenant VNF). These rates can be combined to obtain aggregate bandwidth, and in response to determining the aggregate bandwidth of packet flows 125 associated with and processed by the tenant VNF, network service arranger 112 can compare the aggregate bandwidth with the bandwidth requirements provided by resource allocation information for the VNF and produce a comparison result. The comparison result can indicate the expected aggregate bandwidth of the tenant VNF, which indicates more or less computing resources are required to process the expected aggregate bandwidth compared to the amount of computing resources currently allocated to the tenant VNF. As a result, and therefore based on the aggregate bandwidth, network service arranger 112 can output configuration data 129 to cause server 123 to modify the allocation of computing resources of server 123A, thereby allocating more or less computing resources to perform the tenant VNF. In some examples, the network service arranger 112 periodically repeats the comparison and maintains the comparison results used to determine trends, such that a correlation can be identified between the required number of processor cores and actual bandwidth usage over time. This correlation can be a function or lookup table used by the network service arranger 112 to determine the appropriate number of processor cores to meet the expected bandwidth for executing the tenant VNF ​​to continue processing flow 125 (and in some cases, other flows associated with customer 108B). Although described herein as being performed primarily by the network service arranger, the techniques described in this disclosure can be performed by a separate computing system, such as a controller or application, that has determined modifications to the computational resources of the tenant VNF, invoking the network service arranger 112 to request modifications to the computational resources of the tenant VNF ​​in NFVI 102.

[0044] Figure 2 This is a block diagram illustrating an example data center providing an operating environment with tenant-driven resource allocation for a VNF according to one or more aspects of the technology described in this disclosure.

[0045] In this example data center, cloud exchange 100 allows a corresponding one of any NSP 106A-106C or other customer's customer networks 202A, 202B and NSP networks 204A-204C (collectively, "NSP" or "carrier" network 204) to be directly cross-connected to any other customer network via a Layer 2 (L2) or Layer 3 (L3) connection, thereby allowing the exchange of service traffic between the customer network and CSP 110. Data center 200 may be entirely located within a centralized area, such as a warehouse or localized data center consortium, and provides power, cabling, security, and other services to NSPs, customers, and cloud service providers that locate their respective networks within data center 200 (e.g., for co-location) and / or connect to data center 200 via one or more external links.

[0046] Cloud exchange 100 includes network infrastructure 206 (e.g., for a virtual network) and an operating environment through which customer network 202 can receive services from one or more CSPs 110 via interconnection. Figure 2 In the example, network infrastructure 206 represents the switching structure of the interconnection facility of cloud switch 100 and includes multiple ports that can be dynamically interconnected with virtual circuits via, for example, by invoking service interface 114 of programmable network platform 104. Each port is associated with NSP 106, customer 108, and CSP 110. This allows NSP customers to have the option to create many-to-many interconnects by hooking only once to the switching network and underlying network infrastructure 206 that presents the interconnection platform for cloud switch 100. In other words, cloud switch 100 allows customers to interconnect to multiple CSPs 110 using network infrastructure 206 within data center 200, rather than having to establish separate connections on a switching network to access different CSPs 110.

[0047] The interconnect described herein can refer to, for example, physical cross-connections, Ethernet connections such as Layer 2 VPNs or Virtual Private LANs (e.g., E-LINE, E-LAN, E-TREE, or E-access), Internet-based switching interconnections where the corresponding network devices (e.g., routers and / or switches) of interconnected customers directly peer and exchange Layer 3 routes for service traffic exchanged via network infrastructure 206, and cloud switching where customer routers peer with network infrastructure 206 (or the “provider”) network devices, rather than directly with other customers. Cloud switch 100 can provide customers with interconnection services to network services provided by CSP 110. That is, the interconnection service of cloud switch 100 provides access to network services (e.g., VNFs) provided by CSP 110.

[0048] For Layer 3 or higher interconnections, customer 108 can receive services directly via Layer 3 peering and physical connection to one of the colocation facility exchange points or indirectly via one of the NSPs 106. The NSP 106 provides "transfer" by maintaining a physical presence within data center 200 and aggregating Layer 3 access from one or more customers 108. The NSP 106 can peer directly with data center 200 at Layer 3 and, in doing so, provide indirect Layer 3 connectivity and peering to one or more customers 108 through which customers 108 can obtain services from cloud exchange 100.

[0049] In an instance where cloud exchange 100 provides Internet switching, network infrastructure 206 can be assigned different Autonomous System Numbers (ASNs). Therefore, network infrastructure 206 is the next hop in the path vector routing protocol (e.g., BGP) path from CSP 110 to customer 108 and / or NSP 106. As a result, although cloud exchange 100 is not a transit network with one or more WAN links and accompanying Internet access and transit policies, cloud exchange 100 can peer with multiple different Autonomous Systems via external BGP (eBGP) or other external gateway routing protocols to exchange, aggregate, and route service traffic from one or more CSPs 110 to customer 108. In other words, cloud exchange 100 can internalize the eBGP peering relationship that CSP 110 and customer 108 will maintain on a pairwise basis. Conversely, customer 108 can configure a single eBGP peering relationship with cloud exchange 100 and receive multiple services from one or more CSPs 110 via cloud exchange. While this document primarily describes eBGP or other Layer 3 routing protocol peering between a colocation facility point and customer, NSP, or service provider networks, colocation facilities points may learn routes from these networks in other ways, such as through static configuration or via Routing Information Protocol (RIP), Open Shortest Path First (OSPF), Intermediate System to Intermediate System (IS-IS), or other routing distribution protocols.

[0050] As an example of the aforementioned cloud exchange deployment. Figure 2Customer network 202B is shown to have contracted with the cloud switching provider of cloud switch 100 to directly access Layer 3 services via cloud switch 100, and also with NSP 106B to access Layer 3 services via the transit network of NSP 106B. Customer network 202A is shown to have contracted with NSP 106B to access Layer 3 services via the transit network of NSP 106B. The aforementioned contracts can be instantiated in the network infrastructure 206 of cloud switch 100 through L3 peering configuration within the switching equipment of NSP 106 and cloud switch 100, and L3 connections (e.g., Layer 3 virtual circuits) established within cloud switch 100 to interconnect CSP 110 to NSP 106 and customer network 202, all of which have at least one port providing connectivity within cloud switch 100.

[0051] In some examples, network infrastructure 206 includes one or more virtual machines or containers used to deploy NFVi 102 virtualized network functions. In these examples, network service adapter 112 may receive requests via service interface 114 to deploy one or more virtualized network functions (e.g., virtual routers, load balancers, etc.) implemented in the NFVi 102 of network infrastructure 206. Network service adapter 112 may request VNF ​​distributions including one or more VNF images from a VNF provider (e.g., VNF provider 210).

[0052] about Figure 3 Example network systems and Figure 2 Further details of the example data center 200 can be found in U.S. Provisional Patent Application Serial No. 62 / 908,976 entitled “VIRTUALIZED NETWORK FUNCTIONS VERIFICATION USING DECENTRALIZED IDENTIFIERS”, filed October 1, 2009, which is incorporated herein by reference.

[0053] As mentioned above Figure 1 The arranger 112 can dynamically allocate computing resources for servers of NFVI 102 to tenant VNFs of tenants of the data center 200 / NFVI 102 provider based on the aggregated bandwidth of the streams processed by the tenant VNF.

[0054] Figure 3 This is a block diagram illustrating an example architecture of a network functions virtualization infrastructure with tenant-driven dynamic resource allocation according to the technology described herein. Figure 3In the example, system 300 refers to a switching point (e.g., a cloud switching point) having a network function virtualization infrastructure (NFVi) 102 that connects customer devices via gateway 320, and a cloud service provider running on cloud network 322. In some examples, NFVi 102, gateway 320, and / or cloud network 322 may be provided in a data center environment.

[0055] NFVi 102 includes one or more servers 302A-302N (server 302) for executing / hosting network services, including virtual network devices that connect client devices to cloud services. The example architecture of NFVi 102 enables the deployment of one or more services, such as Virtualized Network Functions (VNFs), on server 302. NFVi 102 includes compute hardware, storage hardware, and network hardware for executing the Virtual Network Functions (VNFs). NFV Management 304 handles the deployment and organization of these network services, for example, by instantiating VNFs on server 302 to execute these network services. As instructed by the orchestration system, NFV Management 304 specifies resources (e.g., resource capacity) in server 302 to support the execution of the VNFs.

[0056] VNFs can provide similar functionality to hardware-based network devices such as dedicated network equipment, but they deliver this functionality in software. VNFs are primarily software-built and therefore decoupled from the underlying hardware. For example, a VNF 306A can provide the same routing, switching firewall, intrusion detection, or other services traditionally provided by dedicated hardware but delivered in software. A VNF 306A can provide forwarding and network address translation services for network traffic going to and from it. In some examples, a VNF 306A—acting as a routing VNF or virtual router to cloud network 322—performs routing and forwarding operations on packets from client devices.

[0057] exist Figure 3 In the example, the NFVI 102 platform includes a server (e.g., server 302A) that runs virtualization software (e.g., a hypervisor) in a virtualization layer, on which a virtual execution environment (including network infrastructure software) is deployed. Server 302A can represent... Figure 1Example instances of any server 123. Virtualization layer 308A can operate a platform for virtualizing network infrastructure components (e.g., for data plane and control plane functionality), including networking protocols such as those used in routing / switching. Server 302A can provide one or more virtual machines (VMs) via virtualization layer 308A, where each VM emulates the hardware used to run software. In other words, example VMs (e.g., Linux kernel VM (KVM)) provide a virtualized operating system and application suite for client access (e.g., to deploy VNFs). Alternatively or additionally, server 302A can provide containers (e.g., those provided by open-source Docker container applications) or other virtual execution environments in which VNFs are implemented. In some examples, NFVI 102 also includes a hardware-based virtualization layer 308A to provide virtual computing, virtual storage, and virtual networking for executing VNFs. NFVI 102 can be executed by one or more computing devices in a centralized or distributed manner.

[0058] exist Figure 3 In the example, server 302A may be part of a computer cluster or pod whose physical resources are virtualized into a network infrastructure such as NFVI 102. The computer cluster can be labeled as the network edge of a cloud service provider. Each cloud service provider may be a data center tenant with one or more VNFs running in server 302 (e.g., server 302A) to provide access to cloud services from devices in cloud network 322. As the network edge, server 302A executes the VNF to perform network edge services, such as routing and forwarding operations for packets directed to or received from cloud service providers on or from cloud network 322. However, the VNF can apply network functions to flows to or originating from any network associated with one or more tenants of the NFVI 102 provider.

[0059] An orchestration system (e.g., network service orchestrator 112) that controls resource orchestration in NFVI 102 can use NFV management 304 of NFVI 102 to instruct server 302A to allocate compute resources among VNFs according to the techniques described herein. These techniques will direct resource allocation based on usage to the executing VNF. Taking into account the service level agreements and overall objectives of the data center and / or tenant, an example technique optimizes resource allocation when balancing the conditions of two tenants / data center: 1) representing the tenant's quality of service (QoS) expectations (i.e., actual resource utilization) and 2) representing the data center provider's quality of service (QoS) obligations (i.e., resource requirements).

[0060] NFV Management 304 distributes the physical resources of Resource Pool 310 to VNFs running in Server 302 of NFVI 102. Examples of various physical resources include processing resources (e.g., processor cores), networking resources (e.g., physical network interface cards (NICs)), and memory resources. Multiple processor cores 314A-314N (“Processor Cores 314”) can be examples of processing or computing resources. In one example, virtualization layer 308A is used to generate abstractions of the various physical resources of Resource Pool 310A, which NFV Management 304 configures as countable virtual resources to be used by VNFs running in Server 302 of NFVI 102. Physical resources can be virtualized into virtual computing resources where each resource (unit) includes one or more processor cores. Another computing resource can be a compute node such as a virtual machine. Other virtual resources include virtual storage resources and virtual network resources.

[0061] The NFV management 304 can couple virtual network interfaces in the VNI space, such as VNIs 316A-316N (or VNI 316), to the virtual switch 318A. The virtual switch 318A is configured with VNIs 316, which are logical interfaces for encapsulation and decapsulation of virtual networks occurring in NFVI 102. Each VNI 316 can be associated with a virtual network assigned to a tenant in NFVI 102. That is, one or more virtual networks can be assigned to a tenant for packet flows. Virtual networks can have corresponding virtual network identifiers, such as VXLAN network identifiers. Packet flows transmitted using these virtual networks are associated with tenants, and packets in such packet flows can include the virtual network identifier of the virtual network on which the packet flow is transmitted.

[0062] In one example, virtual switch 318A may be configured with VXLAN interfaces, each configured with a different VNI in VNI 316 and a corresponding VNI identifier. When a physical network interface in server 302A (e.g., NIC 317) receives network traffic as one or more packet flows, the packets in that packet flow include information identifying those VNIs 316 (e.g., having the same VNI identifier). Virtual switch 318A switches each packet flow to its correct VNF, which may be the VNF assigned the VNI. sFlow agent 326A collects the packet flow data, including the VNI for each flow, and sends the collected packet flow data to a collector.

[0063] Statistics and other data points associated with the transmitted packet stream can provide useful information, such as for vector packet processing by VPP 318A and for tenant-driven dynamic resource allocation for VNFs by the orchestration system of NFVI 102, which can operate in the same data center environment as server 302A. sFlow agent 326A captures these statistics and other data points and then provides them to sFlow collector 324, which combines the provided statistics and other data points with information provided by other sFlow agents 326 in other servers 302.

[0064] In some examples, the sFlow collector 324 utilizes SNMP to communicate with the sFlow agent 326A in server 302A to configure sFlow monitoring on VNF 306A. The sFlow agent 326A uses two forms of sampling mechanisms: packet-based statistical sampling of switched or routed packet flows, and time-based sampling of counters. Generally, packet flow sampling and counter sampling are performed by the sFlow instance associated with the individual data source within the sFlow agent 326A. To perform packet flow sampling, the sFlow instance is configured with a sampling rate. The packet flow sampling process results in the generation of packet flow records. To perform counter sampling, the sFlow instance is configured with a sampling interval. The counter sampling process results in the generation of counter records. The sFlow agent 326A collects the counter records and packet flow records and sends them to the sFlow collector 324 in the form of sFlow datagrams.

[0065] Packet flow sampling, an example of packet flow monitoring, is implemented as follows: When a packet arrives at an interface (e.g., VNI316A), the VNF 306A makes a filtering decision to determine whether the packet should be dropped. If the packet is not filtered, the VNF306A's switching / routing functions assign a target interface. At this point, the sFlow agent 326A determines whether to sample the packet. The sFlow agent 326A uses a counter that decrements with each packet. When the counter reaches zero, regardless of whether sampling has occurred, the counter Total_Packets is incremented, and Total_Packets is the count of all packets that may have been sampled. Using counters such as Total_Packets, the sFlow agent 326A generates various information, including flow statistics. Agents / components of the orchestration system described herein can use these flow statistics to instruct NFVI 102 whether to modify allocated resource capacity, such as the allocation of compute resources. In one example, the collected flow statistics can be used to add or subtract one or more compute resources (e.g., virtual and physical compute nodes such as processor cores in a multi-core environment).

[0066] To illustrate this example, the orchestration system can modify the (current) allocation of compute resources to a running VNF (e.g., VNF 306A) based on actual bandwidth usage, such that running the VNF consumes at most the modified allocation of compute resources. In one implementation, the sFlow agent 326A can calculate bandwidth utilization based on a counter `Total_Packets`, one or more flow statistics, and time information (e.g., time intervals in seconds). The sFlow agent 326A calculates bandwidth utilization (e.g., packet processing rate or throughput over a given time interval). In one example, the sFlow agent 326A calculates flow statistics to aid in bandwidth utilization calculations; an example flow statistic includes the number of packets per second (or simply packets per second (PPS)). In another example, the sFlow agent 326A calculates flow statistics referred to as the average packet size per individual flow, which can be aggregated into the average packet size of tenant flows. By multiplying the average packet size by the packets per second, the sFlow agent 326A calculates bandwidth utilization. Alternatively, the sFlow Proxy 326A can use different counters to calculate bandwidth utilization, such as a counter that increments for each byte in the packet stream. This counter determines the total number of bytes in the packet stream. Using this counter and timestamp data, the sFlow Proxy 326A can calculate bandwidth utilization by dividing the total number of bytes by the time interval (in seconds).

[0067] Sampling involves copying the packet header or extracting features from the packet and storing the sampled information in the sFlow datagram. Example flow attributes of the sampled information include: source address (SRC), destination address (DEST), virtual network identifier (e.g., a virtual network identifier such as a VXLAN network identifier or one of VNI 316), and packet size. The sFlow agent 326A calculates average bandwidth utilization based on the average packet size (assuming the packet size is consistent across packets) and the average of the counter Total_Packets (e.g., per flow).

[0068] An sFlow datagram contains a list of packet flow records and counter records. The format of each record is identified by the `data_format` value. The `data_format` namespace is extensible, allowing the addition of standard record types as well as vendor-specific extensions. Several standard record types have been defined. However, the sFlow agent does not need to support all the different record types, only those applicable to its processing of the specific packets being reported. For example, if a VNF 306A implements layer 2 / 3 switching, the VNF 306A reports to the sFlow agent 326A the layer 2 information of the packets it exchanges, as well as the layer 2 and layer 3 information of the packets it routes. `data_format` uniquely identifies the format of opaque structures in the sFlow specification. Construct a `data_format` to uniquely identify the format of a structure (e.g., a standard structure). When used to describe flow data, a sample `data_format` can identify a set of flow attributes.

[0069] Each time a sample is taken, the counter Total_Samples increments. Total_Samples is a count of the number of samples generated. The sFlow instance sends the samples to the sFlow agent 326A for processing. The samples include packet information, as well as the values ​​of the Total_Packets and Total_Samples counters. The sFlow agent 326A can then use the samples to obtain additional information about the trace of the packets through NFVI 102. Such information depends on the forwarding capabilities of the VNF 306A. Examples of trace information provided include source and destination interfaces, source and destination addresses, source and destination VLANs, next-hop subnet, and the entire AS path. Details of the trace information, along with the average bandwidth rate, are stored in the sFlow datagram format. The virtual switch 318A assumes that the trace information is applied to each packet.

[0070] The Virtual Switch 318A refers to a vector packet processing application built on a software platform (e.g., proprietary and open-source versions of Cisco® Vector Packet Processing technology). Generally, the Virtual Switch 318A provides data plane functionality (e.g., within a virtual router or virtual switch), including packet forwarding operations. The Virtual Switch 318A comprises a set of forwarding nodes and a supporting framework arranged in a directed graph. The framework includes all the basic data structures, timers, drivers (and interfaces to driver software development kits, such as the Data Plane Development Kit (DPDK)), a scheduler for allocating CPU time among graph nodes, performance and debugging tools such as counters, and built-in packet tracing. The latter enables the capture of path or trajectory information taken by packets in the graph with high timestamp granularity, providing a complete understanding of processing at each packet level. The virtual switch 318A can process trajectory information, such as trajectory information determined by the framework and any trajectory information generated via flow monitoring. The virtual switch 318A can couple to the VNI interface and process packets arriving via the physical network hardware on server 302A. Using the trajectory information, the virtual switch 318A assembles these packets into vectors; for example, the virtual switch 318A classifies packets by protocol or format, and when a software node in the virtual switch 318A is scheduled, the virtual switch 318A acquires its packet vectors and processes them in a tight two-loop (or four-loop) manner, while prefetching them to the CPU cache for optimal performance.

[0071] Cloud network 322 can communicatively couple VNF 306A to one or more cloud services 327. Cloud network 322 is typically hidden from or otherwise unavailable to devices on a public network. For example, cloud network 322 can receive packet streams from a client device or another cloud service delivered to the cloud service from VNF 306A. Examples of cloud services include Google Cloud, Azure, Oracle Cloud, Amazon Web Services (AWS), IBM Cloud, Alibaba Cloud, and Salesforce. In some respects, cloud network 322 may be the Equinix Cloud Exchange Fabric provided by Equinix Ltd., Redwood, California. VNF 306A may be a provider-neutral VNF that combines two or more cloud services into a hybrid cloud service.

[0072] Gateway 320 can communicatively couple VNF 306A to public and / or private networks of customer devices. Public networks can be networks with little or no restriction that are publicly available. For example, a public network can be a network that is part of the Internet. Private networks can be networks that are part of an enterprise network and are accessible only to authorized users. Customer devices are customers of the VNF 306A and, as an example, can be computing devices located in a tenant's branch office or otherwise associated with a tenant or customer. The public gateway can receive traffic from the public network with the destination address of server 302A hosting the VNF 306A within the data center. The VNF 306A can receive network traffic from gateway 320.

[0073] As an example, when the above-mentioned orchestration system (e.g., Figure 1 When the assigner 112 performs the initial resource allocation for VNF 306A, a specific tenant's VNF, in response, NFV management 304 deploys VNF 306A on server 302A as a Virtual Place of Presence VNF (e.g., a routing VNF that connects a client device to a specific tenant's cloud service in cloud network 322). NFV management 304 also assigns a portion of the VNI space to that specific tenant for receiving / transmitting packet streams to / from the client device / cloud service. The VNI space may refer to the range of virtual network identifiers for the corresponding virtual network interfaces 316A-316N (VNI 316). NFV management 304 distributes virtual networks in the physical network for NFVI 102 among multiple tenants. Each virtual network may represent an overlay network, such as a VXLAN, and is associated with a virtual network identifier (e.g., a VXLAN network identifier). The virtual network identifier may be included in packets to identify the virtual network of virtual switch 318A for forwarding by virtual switch 318A to one of VNF 306A and the next-hop device.

[0074] In one example, NFV management 304 assigns VNI 316A and its corresponding address to a specific tenant's VNF, VNF 306A, within server 302A. The corresponding address can be constructed at Layer 2 or Layer 3 to uniquely identify VNI 316A as the target of a packet flow directed to the specific tenant's cloud service. For at least this reason, the assigned network address of the specific tenant can be stored in the packets of the packet flow (e.g., in the packet header) – possibly along with the assigned VNI 316A. When the packet arrives at gateway 320 (or in another network device), the assigned address is extracted and translated into a VNI identifier (e.g., a number) that matches the assigned VNI identifier of VNI 316A. Thus, the corresponding VNI identifier of VNI 316A uniquely identifies the specific tenant's assigned VNI, causing packets delivered to that VNI to be (partially) forwarded by virtual switch 318A to VNF 306, the specific tenant's VNF.

[0075] VNF 306A manages network traffic to and from cloud services that is communicatively coupled to server 302A via cloud network 322. When communication occurs on gateway 320 or cloud network 322, VNF 306A can handle network traffic that includes network addresses (e.g., the IP address of server 302A) as source or destination addresses. In some examples, multiple VNFs on server 302A may each have different public IP addresses sharing a single physical network interface. Figure 1 In the example, server 302 and VNF 306 are described as the current NFVi infrastructure. Note that the current NFVi infrastructure may consist of only one server and one VNF, or alternatively, more than one server including more than one VNF.

[0076] According to the technology described in this disclosure, in some aspects, the VNF 306A can form part of a particular tenant's network service. The tenant may have purchased resources from a data center provider for performing the network service, such that the purchased resources are required by the data center provider. As described herein, in several instances, the tenant's network service does not actually utilize all the purchased resources, leaving a significant amount of unused and wasted resource capacity.

[0077] In some examples, the volume of network traffic directed to a particular tenant's VNF fluctuates in such a way that, in certain situations, the particular tenant's VNF may not fully utilize the initial allocation of resources (e.g., compute resources). The particular tenant, cloud service provider, and / or data center provider may have overestimated the expected volume of network traffic and therefore purchased unnecessary resource capacity, including irrelevant bandwidth capacity (e.g., in the Service Level Agreement (SLA)). Over time, the NFVI 102 orchestration system can determine that a particular tenant's VNF consistently utilizes only a portion of the initial allocation of physical resources (bare metal servers, communications infrastructure, etc.) provided by the data center, and can modify the (current) resource allocation of the particular tenant's VNF, for example, to match the actual resource usage (e.g., bandwidth utilization) of the particular tenant's VNF, at least for some reason. In some examples, the NFVI 102 orchestration system also modifies the SLA to update the resource requirements / obligations to the particular tenant executing that tenant's VNF; in this way, the NFVI 102 orchestration system modifies resource allocation without violating the SLA by modifying resource allocation while maintaining the expected quality of service. In some examples, the orchestration system of NFVI 102 achieves optimal resource allocation, where the maximum capacity of computing resources can be used to execute new VNFs and / or execute the maximum number of VNFs simultaneously.

[0078] As described herein, flow information captured by the sFlow agent 326A can be mined for flow statistics associated with tenants and tenant VNFs, including specific tenants and specific tenant VNFs (VNF 306A). Over a time period, each tenant of NFVI 102 and server 302A receives multiple packet flows. In some examples, the orchestration system of NFVI 102 can use NFV management 304 to map each of the multiple packet flow VNIs to a VNI space corresponding to VNI 316 by associating flow statistics with tenant information. In some examples, the orchestration system uses the sflow agent 326A to identify the VNI(s) assigned to a specific tenant from the tenant information and uses the assigned VNI to identify the corresponding flow statistics for those with the same VNI(s).

[0079] The orchestration system uses corresponding flow statistics to calculate various statistical and other information data points. These flow statistics can indicate values ​​for various variables describing packet flows, including the total number of packets, the total number of bytes, packets per second, the average packet size, or the distribution of packet sizes, etc. In some examples, the orchestration system uses these flow statistics to calculate the aggregate bandwidth of multiple flows associated with a specific tenant of the data center provider and processed by the tenant's VNF running on server 302A, and then modifies the allocation of computing resources on server 302A executing the VNF based on the aggregate bandwidth. In some examples, the allocation of computing resources refers to the allocation of physical (hardware) processor resources (e.g., processing circuitry) used to execute the VNF.

[0080] In some examples, the orchestration system is configured to calculate the average bandwidth of a (packet) flow through a VNI based on the corresponding flow statistics. For example, by aggregating the packet size and dividing by the total number of packets, the orchestration system can determine the average packet size from the corresponding flow statistics. Alternatively, the orchestration system can determine the average packet size from the corresponding flow statistics, for example, by extracting the average from stored sflow datagrams. Once the average packet size is known, the orchestration system can calculate the average bandwidth by multiplying the packets per second (PPS) by the average packet size and aggregate the bandwidth as the sum of the average bandwidths of each of the multiple packet flows. In another example, the orchestration system can calculate the average bandwidth by dividing the total number of bytes by the amount of time in seconds and aggregate the bandwidth as the sum of the average bandwidths of each of the multiple packet flows.

[0081] In some examples, the orchestration system compares bandwidth allocations with aggregated bandwidth to produce comparison results and (possibly) stores these results for trend analysis. In some examples, to comply with the SLA obligations of server 302A for executing the VNF, the orchestration system of NFVI 102 can provide allocations of compute resources that meet at least the minimum or sufficient resource requirements for executing the VNF. The orchestration system of NFVI 102 can analyze the comparison results to determine whether (or not) the allocation of compute resources should be modified (e.g., while maintaining consistency with the SLA). If the comparison results indicate underutilization of resources in the VNF (e.g., underutilization of compute resources or another resource (e.g., bandwidth)), the orchestration system of NFVI 102 can modify the allocation by reducing the compute resource allocation to a capacity optimal for the aggregated bandwidth. In some instances, the orchestration system of NFVI 102 can break the SLA and allocate fewer resources than the allocated compute resources.

[0082] In some examples, the NFVI 102 orchestration system periodically repeats comparisons (e.g., after time intervals) and stores each comparison result for trend analysis. Over time, the comparison results evolve into a relationship between the VNF's performance (e.g., in terms of bandwidth utilization) and the VNF's compute resource allocation. For example, the NFVI 102 orchestration system can combine (a series of) aggregated bandwidth measurements into an aggregated bandwidth trend. The NFVI 102 orchestration system can then modify the allocation of compute resources for VNF execution based on the aggregated bandwidth trend. Assuming stable network traffic, the aggregated bandwidth trend eventually converges to a well-defined mapping that can be used when a tenant VNF ​​is deployed in another virtual execution environment in another NFVI. When the aggregated bandwidth changes (e.g., due to a change in network traffic level), the NFVI 102 orchestration system performs a comparison between the current bandwidth allocation and the updated aggregated bandwidth to produce updated comparison results for trend analysis. In some instances, the updated comparison results can indicate insufficient bandwidth and the opportunity to modify resource allocation by adding one or more compute resources (e.g., processor cores).

[0083] The orchestration system can perform the comparisons described above with different resource configurations to determine the maximum achievable (aggregate) bandwidth trend of the computing resource allocation (progress). The orchestration system can store several mappings between different computing resource allocations and their corresponding maximum achievable (aggregate) bandwidth trends in a resource allocation table. The orchestration system can modify the allocation of computing resources based on the resource allocation table. In other examples, the orchestration system can use the resource allocation table for initial or modified allocations of computing resources to perform the same or similar VNFs in another NFVI or another data center.

[0084] Figure 4 This is a flowchart illustrating an example operation of tenant-driven resource allocation based on actual resource usage, according to the technology described herein.

[0085] When the orchestration system identifies that a tenant's VNF bandwidth consumption (rate) is insufficient to meet the tenant's VNF bandwidth allocation, the following example operations can be performed by the orchestration system—a computing system that controls the NFVIs in the data center. In some aspects, a tenant's VNF can be implemented on a physical or virtual machine within the data center. The data center may include gateways (e.g., public or private gateways) that receive network packets from a network (e.g., a customer network or cloud service network) that has the destination address (e.g., a public IP address) of the tenant's VNF. If the tenant's customers rarely (if any) require the total VNF bandwidth allocation, the orchestration system can perform example operations such as instructing the server hosting the tenant's VNF to increase or decrease the allocated resource capacity of the VNF.

[0086] The following operations are about Figure 1 The orchestrator 112 is described as an example orchestration system. In some examples, the orchestrator 112 determines the resource usage context (402) by mapping flow VNIs to tenant VNIs. Tenant information 452 stores information associated with each tenant of a data center provider assigned one or more VNFs in the data center. Tenant information 452 also stores for each tenant a virtual network identifier configured for the virtual network of the flow associated with the tenant. Some of these virtual network identifiers for tenants can be configured on the NFVI server so that the server's virtual switches switch packets with those virtual network identifiers to the tenant VNF. This set of virtual network identifiers is referred to as the VNF VNI space. Flow information 454 stores information describing multiple flows associated with each tenant of the data center provider, each flow referring to packets forwarded via the NFVI between endpoints (e.g., between tenant customers and service providers). Each of tenant information 402 and flow information 404 stores a VNI that can be used as an index to map each tenant to its corresponding flow. Tenant information 402 and flow information 404 are combined such that each tenant VNI in the tenant information is mapped to each flow VNI in the flow information 404. A dynamic resource mapping database 456 can be created to store the mapping between flow VNIs and tenant VNIs.

[0087] If tenant information 452 is organized into the database, then sample entries in that database can be constructed as follows: Tenant 1 { VNF type: Router VNF vendor: CSCO VNF License: Security VNF bandwidth: 5G VNF VNI space: 10-20 Public IP address: 182.1.1.2 / 32

[0088] If the stream information 454 is organized into a database, then example entries in that database can be constructed as follows: Stream 1 { Source address: 10.18.91.100 Target address: 20.16.17.11 VNI: 10 Average bandwidth: 100 Mbps Stream 2 { Source address: 11.12.172.11 Target address: 33.192.11.1190 VNI: 12 AVG BW: 1 Gbps

[0089] To identify flows associated with a tenant, the network service arranger 112 maps the flow's VNI (e.g., VNI for flow 1: 10) to the tenant's VNF VNI space (e.g., VNF VNI space for tenant 1: 10-20). If the VNF VNI space for a tenant includes the VNI for a flow, then the flow is associated with the tenant. Multiple flows associated with a tenant are then aggregated based on the flow records to calculate, for example, average packet size and average bandwidth. When flow information 454 and tenant information 452 are combined into dynamic resource mapping database information 456, sample entries in this database can be constructed as follows: Tenant 1 { Actual resources: { Actual BW utilization: 10G (Multiple) cores: 8 Storage: 8GB Bandwidth usage: { Flow: 1000 Average group size: 256 Average BW: 4.5 G Tenant 2 { Actual resources: { Actual BW: 5G Core: 4 Storage: 4GB Bandwidth usage: { Flow: 560 Average group size: 1000 Average BW: 2.5 G

[0090] The arranger 112 determines the average packet size (404) for all tenant flows. Flow information 454 stores packet flow statistics corresponding to the amount of data (in bytes) received at the network edge of the data center over a time period. The network edge can be a switching point of a cluster of servers or pods that includes resources virtualized for NFVI. In some examples, the dynamic resource mapping database 456 stores the rate (e.g., in packets or bytes) at which data for a particular tenant is received and forwarded via NFVI. From this rate, the arranger 112 calculates the average packet size, which can also be stored in the dynamic resource mapping database 456. In some examples, the dynamic resource mapping database 456 stores the average packet size of data assigned to each VNI for a particular tenant.

[0091] The arranger 112 determines the aggregate bandwidth (i.e., aggregate average bandwidth) of all flows associated with the tenant (406). As mentioned above, the dynamic resource mapping database 456 stores the VNFs of a particular tenant over time and the rate at which bandwidth is consumed for data received at each VNI (e.g., in packets or bytes), which can represent the average rate of each packet flow; by combining the corresponding rates of each VNI, the arranger 112 determines the aggregate rate of data received at all VNIs assigned to that particular tenant.

[0092] In some examples, the arranger 112 creates tenant bandwidth usage trends (408). In some examples, the arranger 112 plots data points representing aggregated average bandwidth relative to the allocation of computing resources and stores these data points for trend analysis. These data points can correlate the actual rate at which packet flows for a particular tenant are received and forwarded through NFVI 102 with the multiple processor cores allocated to such packet processing. Over time, the arranger 112 plots multiple data points representing the trend between the allocation of processor cores and the achievable aggregate bandwidth for a particular tenant's VNF. In this way, the arranger 112 can calculate the maximum possible bandwidth rate for any VNF executing in NFVI 102 based on the allocation of processor cores (or other computing resources) of the VNF. The arranger 112 can store the maximum bandwidth rate and the allocation of processor cores of the VNF in a database such as... Figure 2 In a resource allocation table such as resource allocation table 208, based on the trend of aggregate bandwidth of flows associated with tenants of the data center provider and processed by tenant VNFs, the arranger 112 can determine whether more computing resources (for tending towards higher aggregate bandwidth) or fewer computing resources (for tending towards lower aggregate bandwidth) should be assigned to the tenant VNF.

[0093] The allocator 112 adds / removes a number of processor cores to modify the resource allocation for a particular tenant (410). In some examples, the allocator 112 modifies the allocation of compute resources to run a particular tenant's VNF. The allocator 112 can access the resource allocation table 208 and determine the modified compute resource allocation. If the bandwidth utilization of a tenant's VNF changes, as indicated by the aggregate bandwidth of the tenant streams processed by the tenant's VNF, the allocator 112 can access the resource allocation table 208 and, based on that table, modify the allocation of compute resources to the server running the particular tenant's VNF. For example, the allocator 112 can use the resource allocation table 208 to determine the number of processor cores to add or remove to accommodate the changed bandwidth utilization. In some examples, the allocator 112 issues a function call via the NFVI interface 420 to instruct the NFVI management to add or remove a number of processor cores to the server running the particular tenant's VNF.

[0094] Figure 5 This is a block diagram illustrating a conceptual view of bandwidth-based processor core management according to one or more technologies of this disclosure. Figure 5 In this scenario, multiple data center tenants (or multiple tenants) initiate a VNF in NFVI 102 running on compute node 500. Each tenant in Figure 5 The image is depicted as an oval shape with the letter "T" and numbers. Figure 5 The diagram further illustrates each tenant, with one or two black circular shapes representing one or two processor cores and overlapping with an elliptical shape representing that tenant. Compute node 500 may represent a physical compute node, or in some cases, a virtual compute resource, which is an abstraction of the physical processing resources within a computer cluster (e.g., a pod). Virtual switch 502 is coupled to the virtual compute resource and uses the virtualized physical processing resources to process packet streams received via the virtual network resource of compute node 500. In some examples, the virtual network resource may correspond to one or more VNIs.

[0095] The orchestration system 504 represents a functional block (e.g., a network service orchestrator) of NFVI 102. Generally, the orchestration system 504 orchestrates resources (e.g., virtual resources) to instantiate VNFs to run network services for multiple tenants. In some examples, the orchestration system 504 allocates virtual computing resources, such as one or more processor cores or other processing resources, to each VNF. In some examples, the orchestration system 504 maintains a core reservation 506 to include one or more processor cores that are not available for VNF allocation and utilization. In some examples, the orchestration system 504 also maintains a core pool 508 to include one or more processor cores that are available for VNF allocation and utilization. When compared to a conventional NFVI implementation, Figure 5 The diagram illustrates an improved NFVI with an increased maximum number of available cores for allocation to a VNF. Figure 5 In this context, additional cores between 2 and 4 are assigned to the VNF associated with the tenant of compute node 500.

[0096] The table below illustrates some examples of the benefits achieved by transitioning from traditional resource orchestration techniques (i.e., current operating mode (PMO)) to the techniques described in this paper (i.e., future operating mode (FMO)):

[0097] The table above shows that technicians in NFVi can allocate additional available cores to increase the total number of VNFs running at a given time. In the table, "POD" refers to the server cluster hosting the improved NFVi described in this article (e.g., Figure 3 Server 302).

[0098] Figure 6 This is a block diagram illustrating further details of an example of a computing device operating according to one or more techniques of the present disclosure. Figure 6 A specific example of a server or other computing device 600 may be illustrated, including one or more processors 602 for executing any one or more of any systems, applications, or modules described herein. For example, one or more processors 602 may execute instructions of orchestration system 620 to instantiate and deploy a VNF to an NFVI, and apply the techniques described herein to determine and configure the computational resource allocation of the VNF executing on the server of the NFVI. Thus, computing device 600 may represent an example instance of network service orchestrator 112 or other orchestration system or other system for configuring the resource allocation of a VNF executing on the server of the NFVI according to the techniques of this disclosure. Other examples of computing device 600 may be used in other instances. Although for illustrative purposes... Figure 6 The device is shown as a standalone computing device 600, but a computing device can be any component or system including one or more processors or other suitable computing environments for executing software instructions, and does not necessarily need to include, for example, a... Figure 6 One or more components shown (e.g., communication unit 606; and in some examples, components such as (multiple) storage devices 608 may not coexist with other components or be in the same chassis).

[0099] like Figure 6 As shown in a specific example, computing device 600 includes one or more processors 602, one or more input devices 604, one or more communication units 606, one or more output devices 612, one or more storage devices 608, and a user interface (UI) device 610 and communication unit 606. In one example, computing device 600 also includes one or more applications 622, multiple programmable network platform applications 624, and an operating system 616 executable by computing device 600. Each of components 602, 604, 606, 608, 610, and 612 is physically, communicatively, and / or operatively coupled for inter-component communication. In some examples, communication channel 614 may include a system bus, network connection, inter-process communication data structure, or any other method for transmitting data. As an example, components 602, 604, 606, 608, 610, and 612 may be coupled via one or more communication channels 614.

[0100] In one example, processor 602 is configured to implement functional and / or processing instructions for execution within computing device 600. For example, processor 602 may be able to process instructions stored in storage device 608. Examples of processor 602 may include any one or more of a microprocessor, controller, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or equivalent discrete or integrated logic circuitry.

[0101] One or more storage devices 608 may be configured to store information within computing device 600 during operation. In some examples, storage device 608 is described as a computer-readable storage medium. In some examples, storage device 608 is temporary memory, meaning that the primary purpose of storage device 608 is not long-term storage. In some examples, storage device 608 is described as volatile memory, meaning that storage device 608 does not retain its stored contents when the computer is shut down. Examples of volatile memory include random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), and other forms of volatile memory known in the art. In some examples, storage device 608 is used to store program instructions for execution by processor 602. In one example, storage device 608 is used by software or an application running on computing device 600 to temporarily store information during program execution.

[0102] In some examples, storage device 608 also includes one or more computer-readable storage media. Storage device 608 can be configured to store a larger amount of information than volatile memory. Storage device 608 can also be configured for long-term storage of information. In some examples, storage device 608 includes non-volatile storage elements. Examples of such non-volatile storage elements include magnetic hard disks, optical disks, floppy disks, flash memory, or electrically programmable memory (EPROM) or electrically erasable programmable memory (EEPROM).

[0103] In some examples, computing device 600 also includes one or more communication units 606. In one example, computing device 600 uses communication unit 606 to communicate with external devices via one or more networks, such as one or more wired / wireless / mobile networks. Communication unit 606 may include a network interface card such as an Ethernet card, an optical transceiver, a radio frequency transceiver, or any other type of device capable of sending and receiving information. In some examples, computing device 600 uses communication unit 606 to communicate with external devices.

[0104] In one example, computing device 600 also includes one or more user interface devices 610. In some examples, user interface devices 610 are configured to receive input from a user via haptic, audio, or video feedback. Examples of the multiple user interface devices 610 include presence-sensitive displays, mice, keyboards, voice response systems, cameras, microphones, or any other type of device for detecting commands from the user. In some examples, presence-sensitive displays include touch-sensitive screens.

[0105] The computing device 600 may also include one or more output devices 612. In some examples, the output device 612 is configured to provide output to a user using tactile, audio, or video stimuli. In one example, the output device 612 includes a presence-sensitive display, a sound card, a video graphics adapter card, or any other type of device for converting signals into an appropriate form that is understandable to humans or machines. Other examples of the output device 612 include a speaker, a cathode ray tube (CRT) monitor, a liquid crystal display (LCD), or any other type of device that can generate understandable output to a user.

[0106] Computing device 600 may include operating system 616. In some examples, operating system 616 controls the operation of components of computing device 600. For example, in one example, operating system 616 facilitates communication between one or more applications 618 and processor 602, communication unit 606, storage device 608, input device 604, user interface device 610, and output device 612.

[0107] The (multiple) applications 618 and the assembly system 620 may also include program instructions and / or data executable by the computing device 600. Furthermore, as an example, the assembly system 620 may include implementations... Figure 1 The software of the orchestrator 112 can operate as illustrated and described herein. As indicated by the orchestration system 620 running in the data center where computing device 600 resides, a maximum number of VNFs are instantiated and executed simultaneously. This can be achieved in part by implementing the techniques described herein. As an example, the orchestration system 620 can apply the techniques described herein to determine the allocation of computing resources for servers performing virtual network functions and configure NFVIs with multiple processor cores of the servers (via a coupled NFVI management system (e.g., ...)) based on that allocation. Figure 3 (NFVI management 304). In one example technology, computing device 600 uses information stored in memory (e.g., Figure 2 The resource allocation table (208) is used to determine the number of processor cores to be allocated.

[0108] The techniques described herein can be implemented in hardware, software, firmware, or any combination thereof. Various features described as modules, units, or components can be implemented together in an integrated logic device, or individually as discrete but interoperable logic devices or other hardware devices. In some cases, various features of an electronic circuit can be implemented as one or more integrated circuit devices, such as integrated circuit chips or chipsets.

[0109] If implemented in hardware, this disclosure may relate to apparatus such as a processor or integrated circuit device (such as an integrated circuit chip or chipset). Alternatively or additionally, if implemented in software or firmware, the technology may be implemented at least in part by a computer-readable data storage medium including instructions that, when executed, cause the processor to perform one or more of the methods described above. For example, the computer-readable data storage medium may store these instructions executed by the processor.

[0110] Computer-readable media can form part of a computer program product, which may include packaging materials. Computer-readable media may include computer data storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, and so on. In some examples, the article of manufacture may include one or more computer-readable storage media.

[0111] In some examples, computer-readable storage media may include non-transitory media. The term "non-transitory" can indicate that the storage medium is not embodied in a carrier wave or propagating signal. In some examples, non-transitory storage media may store data that can change over time (e.g., in RAM or cache).

[0112] The code or instructions can be software and / or firmware executed by processing circuitry, which includes one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Therefore, the term "processor" as used herein can refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. Additionally, in some aspects, the functionality described in this invention can be provided within software or hardware modules.

Claims

1. A computing system, the computing system comprising: Processing circuitry, which accesses a storage device, is configured to: Determine that the first and second packet flows are associated with a specific tenant. The first packet flow and the second packet flow are processed by a Virtual Network Function (VNF) executed on the server's computing resources, and The VNF is assigned to the specific tenant; Based on packet flow statistics for the first packet flow and the second packet flow, the aggregate bandwidth of multiple packet flows processed by the VNF and associated with the specific tenant is calculated; as well as Based on the aggregate bandwidth of the plurality of packet flows processed by the VNF and associated with the particular tenant, the allocation of computing resources of the server used to execute the VNF is modified.

2. The computing system according to claim 1, The storage device stores the bandwidth allocated to the VNF, and In order to modify the allocation of computing resources for the server used to execute the VNF, the processing circuitry is configured to modify the allocation of computing resources based on determining that the aggregate bandwidth of the plurality of packet flows processed by the VNF and associated with the particular tenant is less than the bandwidth allocated to the VNF.

3. The computing system according to claim 1, wherein the computing system comprises: One or more of an orchestration system for virtual network functionality including the VNF or a controller for virtual network functionality including the VNF.

4. The computing system of claim 1, wherein, in order to modify the allocation of computing resources of the server for executing the VNF, the processing circuitry is configured to modify the number of processor cores of the server allocated to executing the VNF.

5. The computing system according to any one of claims 1-4, The processing circuitry is configured to calculate, based on the packet flow statistics for the first and second packet flows, the aggregate bandwidth trend of the plurality of packet flows processed by the VNF and associated with the specific tenant; and In order to modify the allocation of computing resources of the server used to execute the VNF, the processing circuitry is configured to modify the allocation of computing resources of the server used to execute the VNF based on the aggregated bandwidth trend of the plurality of packet flows processed by the VNF and associated with the particular tenant.

6. The computing system according to any one of claims 1-4, The storage device stores a mapping of one or more virtual network identifiers associated with the specific tenant, and In order to determine that the first packet flow and the second packet flow are associated with the specific tenant, the processing circuitry is configured to determine that the first packet flow and the second packet flow are each associated with at least one of the one or more virtual network identifiers associated with the specific tenant.

7. The computing system according to claim 6, The first packet flow is associated with a first virtual network identifier among the one or more virtual network identifiers associated with the specific tenant, and The second packet flow is associated with a different second virtual network identifier among the one or more virtual network identifiers associated with the specific tenant.

8. The computing system according to any one of claims 1-4, The VNF mentioned above includes a first VNF, and The processing circuitry is configured to initiate a second VNF to be executed on the server based on a modified allocation of the computing resources of the server for executing the VNF.

9. A computer networking method, comprising: The computing system determines that the first and second packet flows are associated with a specific tenant. The first packet flow and the second packet flow are processed by a Virtual Network Function (VNF) executed on the server's computing resources, and The VNF is assigned to the specific tenant; Based on packet flow statistics for the first packet flow and the second packet flow, the computing system calculates the aggregate bandwidth of multiple packet flows processed by the VNF and associated with the specific tenant; as well as Based on the aggregated bandwidth of the multiple packet flows processed by the VNF and associated with the specific tenant, the computing system modifies the allocation of computing resources of the server used to execute the VNF.

10. The computer networking method of claim 9, wherein modifying the allocation of computing resources of the server executing the VNF comprises: Based on the determination that the aggregate bandwidth of the plurality of packet flows processed by the VNF and associated with the particular tenant is less than the bandwidth allocated to the VNF, the allocation of the computing resources of the server used to execute the VNF is modified.

11. The computer networking method according to claim 9, wherein the computing system comprises: One or more of an orchestration system for virtual network functionality including the VNF or a controller for virtual network functionality including the VNF.

12. The computer networking method of claim 9, wherein modifying the allocation of computing resources of the server for executing the VNF further comprises: Modify the number of processor cores allocated to the server for executing the VNF.

13. The computer networking method according to any one of claims 9-12, further comprising: Based on the packet flow statistics for the first packet flow and the second packet flow, the computing system calculates the aggregate bandwidth trend of the multiple packet flows processed by the VNF and associated with the specific tenant; The modification of the allocation of computing resources of the server used to perform the VNF includes: modifying the allocation of computing resources of the server used to perform the VNF based on the aggregated bandwidth trend of the plurality of packet flows processed by the VNF and associated with the particular tenant.

14. The computer networking method according to any one of claims 9-12, further comprising: Obtain a mapping of one or more virtual network identifiers associated with the specific tenant. Determining that the first packet flow and the second packet flow are associated with the specific tenant includes: determining that the first packet flow and the second packet flow are each associated with at least one of the one or more virtual network identifiers associated with the specific tenant.

15. The computer networking method according to claim 14, The first packet flow is associated with a first virtual network identifier among the one or more virtual network identifiers associated with the specific tenant, and The second packet flow is associated with a different second virtual network identifier among the one or more virtual network identifiers associated with the specific tenant.

16. The computer networking method according to any one of claims 9-12, The VNF includes a first VNF, and the method further includes: A second VNF is initiated to be executed on the server based on the modified allocation of the computing resources of the server used to execute the VNF.

17. A computer-readable storage medium encoded with instructions for configuring one or more programmable processors as a computing system according to any one of claims 1-8, or for performing a computer networking method according to any one of claims 9-16.

Images