Virtual gateway in cloud exchange

Abstract

In an example, a system includes a first cloud exchange network for a first cloud exchange, the first cloud exchange network located within a first data center and configured with a first dedicated virtual gateway configured to interface with a first virtual connector to a customer network, a second virtual connector to a first cloud service provider (CSP) network, and a third virtual connector to a second CSP network. Network traffic between the customer network, the first CSP network, and the second CSP network is routed through the first dedicated virtual gateway. The first dedicated virtual gateway dynamically polices the network traffic based on an aggregate bandwidth subscription configured in the first cloud exchange network that limits a total bandwidth that can be used on the first cloud exchange network between the customer network, the first CSP network, and the second CSP network.

Description

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 025,352, filed May 15, 2020, the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0002] The present disclosure relates to computer networks, and more specifically to cloud exchanges for cloud services. BACKGROUND

[0003] Cloud computing refers to the use of dynamically scalable computing resources accessible via a network, such as the Internet. Computing resources, often referred to as “the cloud,” provide one or more services to users. These services can be categorized according to service type, which can include, for example, applications / software, platforms, infrastructure, virtualization, and servers and data storage. The names of service types are often suffixed with the phrase “as a service,” such that the delivery of applications / software and infrastructure can be referred to as software as a service (SaaS), platform as a service (PaaS), and infrastructure as a service (IaaS).

[0004] The term “cloud-based service” or more simply “cloud service” refers not only to a service provided by a cloud, but also to a form of service provision in which a cloud customer contracts with a cloud service provider for the online delivery of a service provided by a cloud. The cloud service provider manages a public cloud, a private cloud, or a hybrid cloud to facilitate the online delivery of cloud services to one or more cloud customers. SUMMARY

[0005] In general, the present disclosure describes establishing one or more dedicated virtual gateways for a customer in one or more cloud exchanges to facilitate a service delivery model having a uniform approach to managing bandwidth usage of network traffic for multiple virtual connections of the customer. A cloud exchange provider can provision and offer the service delivery model to customers. In some examples, the virtual gateways are configured as routing instances within one or more network devices of the cloud exchange. The cloud exchange provider manages the cloud exchange, for example, to facilitate virtual connections between multiple cloud service providers and customer networks, between customer networks of a customer located in different cloud exchanges, between cloud service provider networks for a customer’s service traffic, and other virtual connections attributed to network traffic of a customer. The cloud exchange enables a customer to request direct connections between multiple cloud service providers via dedicated routing instances operating in the cloud exchange to bypass the public Internet. For example, connections between multiple service providers can allow a customer to transfer large amounts of data between cloud service providers. Typically, these cloud service providers establish bandwidth limits for network traffic to and from the cloud service provider networks.

[0006] A dedicated virtual gateway to a cloud exchange facilitates managing and monitoring customer aggregate network traffic across the cloud exchange by establishing and managing single point connections for customer traffic within the cloud exchange. The dedicated virtual gateway polices traffic through the dedicated virtual gateway based on the destination of the traffic. Thus, rather than customers managing bandwidth usage for each service provider network separately, a dedicated routing instance can be provisioned to dynamically balance bandwidth usage based on demand, as network traffic between customer networks and cloud service provider networks varies within a local cloud exchange with a virtual gateway and between a local cloud exchange and a virtual gateway of a remote cloud exchange. These techniques can facilitate dynamically and directly routing network traffic between customer networks and service provider networks and simplify managing the network resources required to manage network traffic in view of changing traffic patterns and cloud service provider bandwidth limitations. These techniques can also enable a simplified Layer 3 service model for a cloud exchange provider in which customers can request a single aggregate bandwidth that is then dynamically allocated and rate-limited across multiple virtual connections by the cloud exchange, thereby improving customer experience.

[0007] In one example, a system includes a first cloud exchange network for a first cloud exchange, the first cloud exchange network located within a first data center and configured with a first dedicated virtual gateway configured to interface with a first virtual connector to a customer network, a second virtual connector to a first cloud service provider network, and a third virtual connector to a second cloud service provider network, wherein network traffic between the customer network, the first cloud service provider network, and the second cloud service provider network is routed through the first dedicated virtual gateway, and wherein the first dedicated virtual gateway dynamically polices the network traffic based on an aggregate bandwidth subscription configured in the first cloud exchange network that limits a total bandwidth that can be used on the first cloud exchange network between the customer network, the first cloud service provider network, and the second cloud service provider network.

[0008] In one example, a system includes a first cloud exchange network for a first cloud exchange, the first cloud exchange network located within a first data center and configured with a first dedicated virtual gateway storing first routes to a customer network; and a second cloud exchange network for a second cloud exchange, the second cloud exchange network located within a second data center geographically remote from the first data center and configured with a second dedicated virtual gateway storing second routes to a cloud service provider network, wherein the first dedicated virtual gateway polices network traffic on the first routes based on a first aggregate bandwidth subscription configured in the first cloud exchange network and network traffic between the first dedicated virtual gateway and the second dedicated virtual gateway, and wherein the second dedicated virtual gateway polices network traffic on the second routes based on a second aggregate bandwidth subscription established by the second cloud exchange network and network traffic between the first dedicated virtual gateway and the second dedicated virtual gateway.

[0009] In one example, a method includes: configuring, by a programmable network platform, a first dedicated virtual gateway located in a first cloud exchange network within a first data center to interface with a first virtual connector to a customer network, a second virtual connector to a first cloud service provider network, and a third virtual connector to a second cloud service provider network, wherein network traffic between the customer network, the first cloud service provider network, and the second cloud service provider network is routed through the first dedicated virtual gateway; and policing, by the first dedicated virtual gateway, network traffic dynamically based on an aggregate bandwidth subscription that limits a total bandwidth that can be used on the first cloud exchange network between the customer network, the first cloud service provider network, and the second cloud service provider network.

[0010] The details of one or more examples are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF DRAWINGS

[0011] Figure 1 is a block diagram illustrating a conceptual view of a system with metro-based cloud exchange with virtual gateways provided across multiple cloud exchange points in accordance with the techniques described herein.

[0012] Figure 2 is a block diagram illustrating a high-level view of a data center providing an operating environment for cloud-based service exchange in accordance with the techniques described in this disclosure.

[0013] Figures 3A-3B is a block diagram illustrating an example network infrastructure for a cloud exchange that aggregates bandwidth for virtual connections between multiple cloud service providers and customers of a cloud exchange provider and service provisioning provided by a programmable network platform in accordance with the techniques described in this disclosure.

[0014] Figure 4 is a block diagram illustrating an example of a data center-based cloud exchange point in accordance with the techniques described herein, where a router of the cloud exchange point is configured by a programmable network platform, where a virtual gateway is used to route and forward aggregated service traffic between multiple cloud service provider networks and a customer network.

[0015] Figure 5 is a block diagram illustrating an example of a data center-based cloud exchange point in accordance with the techniques described herein, where the cloud exchange point is configured to apply network address translation and route and forward aggregated service traffic between multiple cloud service provider networks and a customer network via a virtual gateway.

[0016] Figure 6 is a block diagram illustrating an example use case in accordance with one or more aspects of the techniques of this disclosure.

[0017] Figure 7 is a block diagram illustrating an example system in which a CSP and customers are spread across multiple metro data centers, where traffic at and between the data centers is managed in an aggregated manner, according to the techniques described herein.

[0018] Like reference characters denote like elements throughout the drawings and text. DETAILED DESCRIPTION

[0019] A cloud exchange located in a metropolitan area includes a dedicated virtual gateway that serves as a routing instance for customers at the cloud exchange. The dedicated virtual gateway routes customer-related network traffic. For example, the dedicated virtual gateway routes network traffic between a cloud service provider and a co-located customer router. Instead of maintaining a physical connection to the cloud service provider, the co-located customer router maintains a single connection to the dedicated virtual gateway, which is configured as a routing instance on the cloud exchange's network equipment. By subscription, the dedicated virtual gateway maintains connections to one or more cloud service providers. Cloud exchange providers' "customers" can include enterprises and cloud service providers.

[0020] When a customer subscribes to a cloud service provider and / or owns equipment (e.g., edge metal, co-located routers, etc.) at multiple cloud exchanges, a dedicated virtual gateway serves as the customer's routing instance at that cloud exchange. The collection of virtual gateways across different cloud exchanges forms the customer's network traffic backbone. Dedicated virtual gateways interconnect so that traffic between cloud exchanges is routed through dedicated virtual gateways. Dedicated virtual gateways automatically exchange routes. Furthermore, when a customer subscribes to a cloud service provider in a new metropolitan area, a dedicated virtual gateway is automatically provisioned for the customer at the corresponding cloud exchange and added to the customer's collection of dedicated virtual gateways.

[0021] Under a unified subscription model for cloud exchange providers, customers license network services for each dedicated virtual gateway. This licensed service defines the maximum bandwidth (i.e., rate) that can be routed through the dedicated virtual gateway, and therefore defines the total licensed bandwidth available to the customer for all of their services. This licensed bandwidth can include traffic between dedicated virtual gateways at different cloud exchanges in different metropolitan areas. The dedicated virtual gateway then polices traffic to ensure that traffic between the dedicated virtual gateway and the cloud service provider does not exceed the customer's subscribed bandwidth limit. This provides customers with a single interface to monitor and manage bandwidth usage across all customer subscriptions.

[0022] A dedicated virtual gateway can use destination-based rate limiting, where network traffic to a set of one or more destinations is rate limited, for example. In some examples, a dedicated virtual gateway uses a destination class usage (DCU) to perform destination-based rate limiting. A DCU counts packets from a customer by performing a lookup on the destination address. For example, a DCU enables traffic originating from a customer and destined for a particular cloud service provider to be tracked. This facilitates traffic shaping, for example, such that traffic to a cloud service provider does not exceed a customer’s subscription and is not at risk of being dropped by the cloud service provider. By policing aggregate traffic to a customer’s destinations, a dedicated virtual gateway can ensure that the bandwidth of the aggregate traffic does not exceed the customer’s licensed bandwidth. As used herein, aggregate / aggregated traffic can refer to traffic for a customer exchanged between a virtual gateway and multiple networks, such that traffic to / from multiple networks is exchanged over a common connector with another network.

[0023] Figure 1 is a block diagram illustrating a conceptual view of a system with metro-based cloud exchange providing virtual gateways across multiple cloud exchange points in accordance with the techniques described herein. Each cloud-based service exchange point 128A-128D (described below as “cloud exchange points,” and collectively as “cloud exchange points 128”) of a cloud-based service exchange 100 (“cloud exchange 100”) can represent different data centers geographically located in the same metro area (“metro-based,” e.g., New York City, New York; Silicon Valley, California; Seattle-Tacoma, Washington; Minneapolis-St. Paul, Minnesota; London, United Kingdom, etc.) to provide a resilient and independent cloud-based service exchange through which cloud-based service customers (“cloud customers”) and cloud-based service providers (“cloud providers”) connect to receive and provide cloud services, respectively. In various examples, the cloud exchange 100 can include more or fewer cloud exchange points 128. In some instances, the cloud exchange 100 includes only one cloud exchange point 128. As used herein, a reference to a “cloud exchange” or “cloud-based service” can refer to a cloud exchange point. A cloud exchange provider can deploy instances of the cloud exchange 100 in multiple different metro areas, each instance of the cloud exchange 100 having one or more cloud exchange points 128.

[0024] Each of the cloud exchange points 128 includes network infrastructure and an operating environment through which the cloud customers 108A-108D (collectively, "cloud customers 108") receive cloud services from a plurality of cloud service providers 110A-110N (collectively, "cloud service providers 110"). The cloud exchange 100 provides exchanged customers (e.g., enterprises, network operators, network service providers, and SaaS customers) with secure, private, virtual connections to a global plurality of cloud service providers (CSPs). The plurality of CSPs participate in the cloud exchange by virtue of having at least one accessible port in the cloud exchange through which the customers can respectively connect to one or more cloud services offered by the CSPs. The cloud exchange 100 allows a dedicated network of any customer to be directly cross-connected at a common point to any other customer, thereby allowing network traffic to be directly exchanged between the customers' networks.

[0025] When connected to the cloud exchange 100, the CSPs 110 establish a service profile for the customers 108 to subscribe to the services of the CSPs 100. The service profile is maintained by the cloud exchange 100. The service profile defines, for example, rules and parameters associated with connecting to the CPSs 110 through the cloud exchange 100. In addition, the service profile specifies one or more bandwidth limits for the customers 108 to select when subscribing to the CSPs 110. As described below, the cloud exchange 100 polices traffic between the subscribed CPSs 110 and the customers 108 based on the selected bandwidth limits in the service profile.

[0026] The cloud customers 108 can receive cloud-based services directly via Layer 3 peering and physical connectivity with one of the cloud exchange points 128, or indirectly via one of the network service providers 106A-106B (collectively, "NSPs 106," or alternatively, "carriers 106"). The NSPs 106 provide "cloud transit" by maintaining a physical presence in one or more of the cloud exchange points 128 and aggregating Layer 3 access from one or more customers 108. The NSPs 106 can operate directly at Layer 3 with one or more of the cloud exchange points 128 and, in doing so, provide indirect Layer 3 connectivity and peering operations to one or more customers 108 through which the customers 108 can obtain cloud services from the cloud exchange 100. In Figure 1In the example of FIG. 1, each of the cloud exchange points 128 is assigned a different autonomous system number (ASN). For example, cloud exchange point 128A is assigned ASN 1, cloud exchange point 128B is assigned ASN 2, and so on. As a result, each cloud exchange point 128 is the next hop in a path vector routing protocol (e.g., BGP) path from the cloud service providers 110 to the customers 108. As a result, although not a transit network with one or more wide area network links and attendant Internet access and transit policies, each cloud exchange point 128 can peer with multiple different autonomous systems via external BGP (eBGP) or other external gateway routing protocols in order to exchange, aggregate, and route service traffic 110 from one or more cloud service providers to customers. In other words, the cloud exchange points 128 can internalize eBGP peering relationships that the cloud service providers 110 and customers 108 would otherwise maintain on a pairwise basis. Conversely, the customers 108 can configure a single eBGP peering relationship with the cloud exchange points 128 and receive multiple cloud services from one or more cloud service providers 110 via the cloud exchange. While described herein primarily with respect to eBGP or other layer 3 routing protocol peering between cloud exchange points and customer, NSP, or cloud service provider networks, the cloud exchange points can learn routes from these networks in other ways, such as through static configuration, or via routing information protocol (RIP), open shortest path first (OSPF), intermediate system to intermediate system (IS-IS), or other route distribution protocols.

[0027] As an example of the above, customer 108C is illustrated as having contracted with the cloud exchange provider of the cloud exchange 100 to directly access layer 3 cloud services via cloud exchange point 128C. In this way, customer 108C receives, for example, redundant layer 3 connectivity to cloud service provider 110A. In contrast, customer 108B is illustrated as having contracted with the cloud exchange provider of the cloud exchange 100 to directly access layer 3 cloud services via cloud exchange point 128C and also contracted with NSP 106B to access layer 3 cloud services via the transit network of NSP 106B. Customer 108B is illustrated as having contracted with multiple NSPs 106A, 106B to redundantly cloud access cloud exchange points 128A, 128B via the corresponding transit networks of NSPs 106A, 106B. The above contracts are instantiated in the network infrastructure of the cloud exchange points 128 through L3 peering configuration within the exchange equipment of the NSPs 106 and cloud exchange points 128 and L3 connectivity, which is established within the cloud exchange points 128 to interconnect the cloud service provider 110 networks to the NSP 106 networks and customer 108 networks, all with at least one port providing connectivity within one or more cloud exchange points 128.

[0028] The cloud exchange point 128 is configured with dedicated virtual gateways 130A-130C (collectively, "dedicated virtual gateways 130"). The dedicated virtual gateways 130 are virtual routing instances dedicated to network traffic for a particular customer 108. For example, the dedicated virtual gateway 130A can be a destination for network traffic to the customer 108A. Network traffic for the customer network of the customer 108A can be routed through the dedicated virtual gateway 130A to the CSP 110. The cloud exchange 100 allows any network service provider (NSP) or "carrier" 106A-106B (collectively, "carriers 106") or other cloud customers including the customer 108C to connect to any other customer network and / or to any CSP 110 via their dedicated virtual gateway. This allows network traffic to be exchanged between customer networks and CSPs 110 via the dedicated virtual gateways 130 such that all network traffic flows through the dedicated virtual gateways 130. In some examples, the connections can be IP-VPN or other types of attachment circuits and can be referred to herein as virtual connectors or simply connectors. For example, a virtual connector can connect a customer network to a virtual gateway, connect a cloud service provider network to a virtual gateway of a customer network, connect a virtual gateway of a customer network to a virtual gateway of a cloud service provider network, connect a virtual gateway of a customer in a first metro-based cloud exchange to a virtual gateway of a customer in a different, second metro-based cloud exchange. Likewise, a customer of the cloud exchange provider can include a cloud service provider. The dedicated virtual gateways 130 can eliminate the need for the customer 108 to establish separate network profiles for L2 and L3 access. In some examples, the dedicated virtual gateways 130 can provide L2 and L3 connectivity such that the type of virtual connector between the dedicated virtual gateways 130 and the CSPs 110 is independent of the type of virtual connector between the dedicated virtual gateways 130 and the customers 108.

[0029] Generally, a customer 108 subscribes to the cloud exchange 100 and to each CSP 110 to which the customer connects via the cloud exchange 100. These subscriptions specify the licensed traffic that the customer 108 can use. For example, the customer 108 licenses the traffic of each private virtual gateway 130 in the cloud exchange 100. This licensed traffic limits the maximum bandwidth (e.g., rate, etc.) that can be routed through the private virtual gateway 130, and thus limits the total amount of licensed bandwidth available to the customer 108 for all network traffic to the CSPs 100 via the cloud exchange 100. Similarly, the subscription to the CSPs 110 can limit the maximum ingress and / or egress network traffic that the customer 108 can use. The private virtual gateway 130 provides a unified subscription model for the customer 108. The private virtual gateway 130 monitors and manages (e.g., polices, shapes, etc.) the network traffic of the customer 108 at the cloud exchange 100 to account for both the licensed traffic subscription to the cloud exchange 100 and the licensed traffic subscription to the various CSPs 110. The private virtual gateway 130 polices the network traffic to ensure that the traffic between the private virtual gateway 130 and the CSPs 110 does not exceed the bandwidth limits to which the customer subscribes the CSPs 110. The private virtual gateway 130 provides a single interface for the customer 108 to monitor and manage the bandwidth usage of all customer subscriptions. In some examples, the private virtual gateway 130 facilitates CSP-to-CSP network traffic such that data from one CSP (e.g., CSP 110A) can be provided to another CSP (e.g., CSP 110N) without transiting through the customer network. The private virtual gateway 130 can dynamically provision network traffic in response to dynamically changing network usage while managing the network traffic according to the subscriptions of the customer 108. The private virtual gateway 130 can be configured for a particular customer 108 across multiple cloud exchange points 128 within the cloud exchange 100 such that network traffic is uniformly monitored and managed according to the techniques of the present disclosure regardless of which cloud exchange point 128 the network traffic is routed through.

[0030] The carriers 106 can each represent a network service provider associated with a transit network through which network subscribers of the carriers 106 can access cloud services provided by the CSPs 110 via the cloud exchange 100. Generally, customers of the CSPs 110 can include network carriers, large enterprises, managed service providers (MSPs), and software as a service (SaaS), platform as a service (PaaS), infrastructure as a service (IaaS), virtualization as a service (VaaS), and data storage as a service (dSaaS) customers for such cloud-based services provided by the CSPs 110 via the cloud exchange 100.

[0031] In this manner, the cloud exchange 100 streamlines and simplifies the process of CSPs 110 and customers (via carriers 106 or directly) working together in a transparent and neutral manner. One example application of the cloud exchange 100 is in co-location and interconnection data centers, where CSPs 110 and carriers 106 and / or customers 108 can already have a network presence, such as by having one or more accessible ports available for interconnection within a data center, which can represent any cloud exchange point 128. This allows participating carriers, customers, and CSPs to have extensive interconnectivity options within the same facility. In this manner, a carrier / customer can have the option to create multiple-to-multiple interconnections with only one hook-up to one or more cloud exchange points 128. In other words, the cloud exchange 100 allows customers to interconnect to multiple CSPs and cloud services, rather than having to establish separate connections over a transit network to access different cloud service providers or different cloud services of one or more cloud service providers.

[0032] The cloud exchange 100 includes a programmable network platform 120 for dynamically programming the cloud exchange 100 to responsively and assuredly meet service requests encapsulating business requirements for services provided by the cloud exchange 100 and / or cloud service providers 110 coupled to the cloud exchange 100. As a result, the programmable network platform 120 can coordinate business-class services across heterogeneous cloud service providers 110 in accordance with explicitly defined service policies, quality-of-service policies, service-level agreements, and costs, and further in accordance with service topologies for business-class services.

[0033] The programmable network platform 120 enables cloud service providers managing the cloud exchange 100 to dynamically configure and manage the cloud exchange 100 to, for example, facilitate virtual connections for cloud-based service delivery from multiple cloud service providers 110 to one or more cloud customers 108 through dedicated virtual gateways 130. The cloud exchange 100 can enable cloud customers 108 to bypass the public Internet to connect to cloud service providers 110 through dedicated virtual gateways 130, thereby improving performance, reducing costs, increasing security and privacy of the connection, and leveraging cloud computing for additional applications. In this manner, for example, enterprises, network carriers, and SaaS customers can at least in some respects integrate cloud services with their internal applications as if such services were part of their own data center network or as if otherwise directly coupled to their own data center network. Further, the dedicated virtual gateways 130 can simplify management of business subscriptions of the cloud exchange 100 and cloud service providers 110 by providing a single entity within the cloud exchange 100 to monitor and manage network traffic.

[0034] The programmable network platform 120 can represent an application executing within one or more data centers of the cloud exchange 100, or alternatively can represent an application executing off-premise, e.g., at a cloud provider’s back office or branch office. The programmable network platform 120 can be distributed in whole or in part among data centers each associated with a different cloud exchange point 128 to make up the cloud exchange 100. While shown as managing a single cloud exchange 100, the programmable network platform 120 can control service provisioning for multiple different cloud exchanges. Alternatively or additionally, multiple separate instances of the programmable network platform 120 can control service provisioning for respective multiple different cloud exchanges.

[0035] In the illustrated example, the programmable network platform 120 includes a service interface (or “service API”) 114 that defines methods, fields, and / or other software primitives that applications 130 such as a customer portal can invoke to program the programmable network platform 120. The service interface 114 can allow the operators 106, customers 108, cloud service providers 110, and / or cloud exchange providers to programmably access the capabilities and assets of the cloud exchange 100 according to the techniques described herein.

[0036] For example, the service interface 114 can facilitate machine-to-machine communication to enable dynamic provisioning of virtual connectors in the cloud exchange for interconnecting customer and / or cloud service provider networks. In this way, the programmable network platform 120 enables automation of various aspects of cloud service provisioning. For example, the service interface 114 can provide customers with an automated and seamless way to establish, offload, and manage interconnections between multiple different cloud providers participating in the cloud exchange.

[0037] Further example details of cloud-based services exchanges can be found in U.S. Patent Application No. 15 / 099,407, filed April 14, 2006, entitled "CLOUD-BASED SERVICES EXCHANGE"; U.S. Patent Application No. 14 / 927,451, filed October 29, 2005, entitled "INTERCONNECTION PLATFORM FOR REAL-TIME CONFIGURATION AND MANAGEMENT OF A CLOUD-BASED SERVICES EXCHANGE"; U.S. Patent Application No. 14 / 927,306, filed October 29, 2015, entitled "ORCHESTRATION ENGINE FOR REAL-TIME CONFIGURATION AND MANAGEMENT OF INTERCONNECTIONS WITHIN A CLOUD-BASED SERVICES EXCHANGE"; and U.S. Patent Application No. 15 / 396,349, filed December 30, 2016, entitled "LAYER THREE INSTANCES FOR A CLOUD-BASED SERVICES EXCHANGE"; each of which is incorporated by reference herein in its respective entirety.

[0038] Figure 2 is a block diagram illustrating a high-level view of a data center 201 providing an operating environment for a cloud-based services exchange 200 in accordance with the techniques described herein. The cloud-based services exchange 200 ("cloud exchange 200") allows any NSP 106A-106C or other cloud customer (including customers 108A, 108B) to directly connect a corresponding one of the customer networks 204D, 204E and NSP networks 204A-204C (collectively, the "private" or "carrier" networks 204) to any other customer network and / or any cloud service provider 110A-110N via a dedicated virtual gateway 130, thereby allowing for the exchange of services traffic between customer networks and / or CSPs 110. The data center 201 can be located entirely within a centralized region (such as a warehouse or localized data center consortium) and provide power, wiring, security, and other services to NSPs, customers, and cloud service providers that position their respective networks within the location data center 201 (e.g., for co-location) and / or connect to the data center 201 through one or more external links.

[0039] The network service providers 106 can each represent a network service provider associated with a transit network through which network subscribers of the NSPs 106 can access cloud services provided by the CSPs 110 via the cloud exchange 200. Generally, customers of the CSPs 110 can include network operators, large enterprises, managed service providers (MSPs), and software as a service (SaaS), platform as a service (PaaS), infrastructure as a service (IaaS), virtualization as a service (VaaS), and data storage as a service (dSaaS) customers for such cloud-based services provided by the CSPs 110 via the cloud exchange 200.

[0040] In this manner, the cloud exchange 200 streamlines and simplifies the process for CSPs 110 and customers 108 to cooperate in a transparent and neutral manner, either indirectly via NSPs 106 or directly. Further, the cloud exchange 200 facilitates a unified subscription model based on the customers 108 to manage network traffic between the CSPs 110 and the customers 108 in a unified manner. One example application of the cloud exchange 200 is in co-located and interconnected data centers where the CSPs 110, NSPs 106, and / or customers 108 can already have a network presence, such as by having one or more accessible ports available for interconnection within a data center. This allows participating operators, customers, and CSPs to have extensive interconnectivity options in the same facility.

[0041] The cloud exchange 200 of the data center 201 includes a network infrastructure 222 that provides an L2 / L3 switching fabric through which the CSPs 110 and customers / NSPs are interconnected by dedicated virtual gateways 130 that route L2 / L3 traffic. This enables NSPs / customers to have the option to create many-to-many interconnections with only one hook into the switching network and the underlying network infrastructure 222 that presents an interconnection platform for the cloud exchange 200. In other words, instead of having to establish separate connections over a transit network to access different cloud service providers or different cloud services of one or more cloud service providers, the cloud exchange 200 allows customers to use the network infrastructure 222 within the data center 201 to interconnect to multiple CSPs and cloud services via dedicated virtual gateways 130, which can at least partially represent any edge network described in this disclosure.

[0042] By using the cloud exchange 200, customers can purchase services and reach a plurality of end users in a plurality of different geographic regions without incurring the same costs typically associated with installing and maintaining a plurality of virtual connections with a plurality of CSPs 110. For example, NSP 106A can use the network 204B of NSP 106B to extend its services. By connecting to the cloud exchange 200, NSP 106 can be able to generate additional revenue by offering its network services to other carriers. For example, NSP 106C can offer other NSPs the opportunity to use the NSP network 204C.

[0043] The cloud exchange 200 includes a programmable network platform 120 that exposes at least one service interface, which in some examples can include and is alternatively referred to herein as an application programming interface (API) in that the API defines the methods, fields, and / or other software primitives through which applications can call the programmable network platform 120. The software interface allows NSPs 206 and customers 108 programmable access to the capabilities and assets of the cloud exchange 200. The programmable network platform 120 can alternatively be referred to as a controller, provisioning platform, provisioning system, service coordination system, etc., for establishing end-to-end services in accordance with the techniques described herein, including connectivity between customers and cloud service providers, for example.

[0044] On the buy-side, the software interface presented by the underlying interconnection platform provides an extensible framework that allows software developers associated with customers of the cloud exchange 200 (e.g., customers 108 and NSPs 206) to create software applications that allow and leverage access to the programmable network platform 120 through which the cloud exchange 200 can be requested to establish connectivity between customers and cloud services offered by any CSP 110. For example, these buy-side software interfaces can allow customer applications for NSPs and enterprise customers, for example, to obtain authorization to access the cloud exchange, obtain information about available cloud services, obtain customer’s active ports and metro details, create virtual connectors of different bandwidths to access cloud services, including dynamically selecting bandwidth to create on-demand and demand-based virtual connectors to or between cloud service providers based on purchased cloud services, delete virtual connectors, obtain active virtual connector information, obtain details about CSPs that partner with the cloud exchange provider, obtain custom analytics data, validate partner access to interconnection assets, and ensure service delivery.

[0045] On the cloud service provider seller side, the software interfaces can allow software developers associated with the cloud providers to manage their cloud services and enable customers to connect to their cloud services. For example, these seller-side software interfaces can allow cloud service provider applications to get authorization to access the cloud exchange, get information about available cloud services, get provider active ports and metro details, get active port details for the provider in a given data center, approve or reject virtual connectors of different bandwidths created by customers for access to cloud services, get virtual connectors to be added and confirm adding virtual connectors, get virtual connectors to be deleted and confirm deleting virtual connectors, get custom analytics data, verify partner access to interconnection assets, and ensure service delivery.

[0046] The service interfaces 114 facilitate machine-to-machine communication to enable dynamic service provisioning and service delivery assurance. In this way, the programmable network platform 120 enables automation of various aspects of cloud service provisioning. For example, the software interfaces can provide customers with an automated and seamless way to establish, offload, and manage interconnections with multiple different cloud providers participating in the cloud exchange, and interconnections between them. In various examples, the programmable network platform 120 can be executed on one or more virtual machines and / or real servers of the data center 201, or off-premise.

[0047] In Figure 2 In examples, the network infrastructure 222 represents the cloud exchange fabric and includes a plurality of ports that can be dynamically interconnected with virtual connectors by, for example, invoking the service interfaces 114 of the programmable network platform 120. Each port is associated with one of the carriers 106, customers 108, and CSPs 110.

[0048] In some examples, a cloud exchange seller (e.g., a CSP nested within a CSP or an enterprise) can request and obtain an L3 instance, can then create a seller profile associated with the L3 instance, and subsequently operate as a seller on the cloud exchange. The techniques of the present disclosure enable multiple CSPs to participate in an enterprise’s L3 instance (e.g., an L3 “routing instance” or an L2 “bridging instance”) without needing to anchor each CSP flow with enterprise equipment.

[0049] In some aspects, the programmable network platform 120 can provide a cloud exchange to deliver a service composed of multiple component services provided by multiple different cloud service providers, where this is provided as a service described herein via an L3 instance. Each of these component services is referred to herein as a "microservice" because it is part of an overall service applied to the service business. That is, multiple microservices can be applied to the service business in a specific "arrangement," "sequencing," or "topology" to form the overall service of the service business. The microservices themselves can be applied or provided by the cloud service provider 110.

[0050] When customer 108 subscribes to cloud exchange 200 (e.g., establishes a direct or indirect connection, licenses bandwidth for traffic passing through cloud exchange 200, etc.), programmable network platform 120 can provision a virtual gateway 130 dedicated to customer 108 such that network traffic flowing to and originating from ports associated with customer 108 is routed through the corresponding dedicated virtual gateway 130. In some examples, virtual gateway 130 facilitates connectivity between the network of CSP 100 and customer 108 when customer 108 does not have connectivity to a physical port of the network infrastructure. Virtual gateway 130 can facilitate customer 108 operating from a virtual port that is based on a physical port of another entity.

[0051] Figures 3A-3B is an illustration of an example network infrastructure for a cloud exchange (e.g., Figure 2 110 ). FIGURE 1 illustrates a block diagram of a network infrastructure 122 (often referred to as a network infrastructure 122) and a programmable network platform that shapes aggregate bandwidth to multiple cloud service providers that are offered to customers of the cloud exchange provider. In this example, customer networks 308A-308C (collectively, "customer networks 308"), each associated with a different customer, access a cloud exchange point within data center 300 to receive aggregated cloud services from one or more cloud service provider networks 320, each associated with a different cloud service provider 110. Furthermore, in some examples, customer networks 308 can access a cloud exchange point within data center 300 to receive aggregated cloud services from one or more remote cloud service provider networks 324, which are directly connected to a remote cloud exchange 326 (e.g., a cloud exchange operating in a different data center). In some examples, each of customer networks 308 includes endpoint devices that consume cloud services provided by cloud service provider networks 320 and 324. Example endpoint devices include servers, smartphones, television set-top boxes, workstations, laptops / tablets, video game systems, teleconferencing systems, media players, and the like.

[0052] Customer networks 308A-B include respective provider edge / autonomous system border routers (PE / ASBR) 310A-B. Each of PE / ASBRs 310A, 310B can execute an external gateway routing protocol to peer with one of PE routers 302A-B over one of access links 316A-B (collectively, "access links 316") to execute an external gateway routing protocol. In the illustrated example, each access link 316 represents a transit link between an edge router of customer network 308 and an edge router (or autonomous system border router) of cloud exchange point 303. For example, PE 310A and PE 302A can directly peer via an external gateway protocol (e.g., external BGP) to exchange L3 routes over access link 316A and exchange L3 data traffic between customer network 308A and cloud service provider network 320. Access links 316 can in some cases represent and alternatively be referred to as attachment circuits of IP-VPNs configured in IP / MPLS fabric 301, as described in more detail below. IP / MPLS fabric 301 is an example of a cloud exchange network. Access links 316 can in some cases each include a direct physical connection between at least one port of customer network 308 and at least one port of cloud exchange point 303, without an intermediate transit network. Access links 316 can operate over a VLAN or stacked VLAN (e.g., QinQ), VxLAN, LSP, GRE tunnel, or other type of tunnel.

[0053] While illustrated and described with respect to L3 connectivity, PE routers 302 can in addition provide L2 connectivity between customer networks 308 and cloud service provider networks 320 via virtual gateways 130 via access links 316. For example, a port of PE router 302A can be configured with an L2 interface that provides L2 connectivity to cloud service provider 320A to customer network 308A via access link 316A, with cloud service provider 320A router 312A coupled to a port of PE router 304A that is also configured with an L2 interface. The port of PE router 302A can additionally be configured with an L3 interface that provides L3 connectivity to cloud service provider 320B to customer network 308A via access link 316A via virtual gateway 130. PE 302A can be configured with multiple L2 and / or L3 sub-interfaces such that one-to-many connectivity to multiple cloud service providers 320 can be provided to customer 308A by the cloud exchange provider.

[0054] To create an L2 interconnection between customer network 308 and cloud service provider network 320, in some examples, IP / MPLS fabric 301 is configured with an L2 bridging domain (e.g., an L2 virtual private network (L2VPN), such as a virtual private LAN service (VPLS), E-LINE, or E-LAN) to bridge L2 traffic between customer-facing ports of PE 302 and CSP-facing ports of cloud service provider 320. In some cases, cloud service provider 320 and customer 308 can have access links to the same PE router 302, 304, which use the bridging domain to bridge L2 traffic.

[0055] To create an L3 interconnection between customer network 308 and cloud service provider network 320, in some examples, IP / MPLS fabric 301 is configured with an L3 virtual routing and forwarding instance (VRF), as described in further detail below with respect to Figure 4 In some cases, IP / MPLS fabric 301 can be configured with one L3 instance that includes one or more VRFs, and the L3 instance can link multiple cloud service provider networks 320. In such cases, customer network 308 can not need to be interconnected or have any physical presence in a cloud exchange or data center.

[0056] Each of access links 316 and aggregation links 322 can include a network interface device (NID) that connects customer network 308 or cloud service provider 328 to a network link between one of PE routers 302, 304 and the NID. Each access link 316 and aggregation link 322 can represent or include any of a variety of different types of links that provide L2 and / or L3 connectivity.

[0057] In this example, customer network 308C represents an autonomous system with an autonomous system number. Customer network 308C can represent an enterprise, a network service provider, or other customer network within a routing overlay region of a cloud exchange point. Customer network includes customer edge (CE) device 311, which can execute an exterior gateway routing protocol to peer with PE router 302B over access link 316C. In various examples, any of PE 310A-310B can alternatively be or otherwise represent a CE device.

[0058] Access links 316 include physical links. PE / ASBR 310A-310B, CE device 311, and PE routers 302A-302B exchange L2 / L3 packets via access links 316. In this regard, access links 316 constitute transport links for cloud access via cloud exchange point 303. Cloud exchange point 303 can represent an example of any cloud exchange point 128. Data center 300 can represent an example of data center 201.

[0059] In some examples, the cloud exchange point 303 aggregates customer 308 access to the cloud exchange point 303, and thus to any one or more cloud service providers 320. For example, Figures 3A-3B Access links 316A-316B connecting respective customer networks 308A-308B to PE routers 302A of the cloud exchange point 303 and access link 316C connecting customer network 308C to PE router 302B are illustrated. Any one or more of the PE routers 302, 304 can comprise an ASBR. The PE routers 302, 304 and IP / MPLS fabric 301 can be configured to interconnect any access link 316 to any cloud aggregation link 322 according to the techniques described herein. Thus, a cloud service provider network 320A, for example, need only configure a single cloud aggregation link (here, access link 322A) in order to provide services to multiple customer networks 308. That is, a cloud service provider operating the cloud service provider network 320A need not provision and configure a separate service link from the cloud service provider network 320A to each PE router 310, 311 in order to provide services to each customer network 308, for example. The cloud exchange point 303 can instead connect the cloud aggregation link 322A of the cloud service provider network 320A and PE 312A to multiple cloud access links 316 in order to provide Layer 3 peering and network reachability for cloud service delivery. In some examples, in order to provide connectivity to remove interconnectivity between service provider networks 324 (and in some examples, customer networks) of the cloud exchange 326, the PE routers 302, 304, 328 and IP / MPLS fabric 301 can be configured to interconnect any access link 316 to exchange interconnect links 330 according to the techniques described herein.

[0060] Further, a single customer network, such as customer network 308A, need only configure a single cloud access link (here, access link 316A) to the cloud exchange point 303 within the data center 300 in order to obtain services from multiple cloud service provider networks 320 and 324 that provide cloud services via the cloud exchange point 303. That is, a customer or network service provider operating the customer network 308A need not provision and configure separate service links connecting the customer network 308A to different PE routers 312 in order to obtain services from multiple cloud service provider networks 320 and 324, for example. The cloud exchange point 303 can instead connect the cloud access link 316A (again, as one example) to multiple cloud aggregation links 322 and / or exchange interconnect links 330 to provide Layer 3 peering operations and network reachability for cloud service delivery to the customer network 308A via the dedicated virtual gateway 130. Traffic to and from the cloud access link 316A (again, as one example) is routed via the dedicated virtual gateway 130A.

[0061] The cloud service provider networks 320 each include servers configured to provide one or more cloud services to users. These services can be categorized according to service type, which can include, for example, applications / software, platforms, infrastructure, virtualization, and servers and data storage. Example cloud services can include content / media delivery, cloud-based storage, cloud computing, online gaming, IT services, etc.

[0062] The cloud service provider networks 320 include PE routers 312A-312C that each perform an external gateway routing protocol (e.g., eBGP) to exchange routes with the PE routers 304A-304B (collectively, "PE routers 304") of the cloud exchange point 303. Each of the cloud service provider networks 320 can represent a public cloud, a private cloud, or a hybrid cloud. Each cloud service provider network 320 can have an assigned autonomous system number or be part of the autonomous system coverage area of the cloud exchange point 303.

[0063] In the illustrated example, an Internet Protocol / Multi-Protocol Label Switching (IP / MPLS) fabric 301 interconnects the PEs 302 and 304. The IP / MPLS fabric 301 includes one or more switching and routing devices, including the PEs 302, 304, that provide IP / MPLS switching and routing of IP packets to form an IP backbone network. In some examples, the IP / MPLS fabric 301 can implement one or more different tunneling protocols (i.e., other than MPLS) to route traffic between the PE routers and / or associate traffic with different IP-VPNs. In accordance with the techniques described herein, the IP / MPLS fabric 301 implements an IP Virtual Private Network (IP-VPN) to connect any customer 308 with multiple cloud service provider networks 320 to provide data center based "transport" and Layer 3 connectivity.

[0064] While service provider based IP backbone networks require a wide area network (WAN) connection with limited bandwidth to transport service traffic from the Layer 3 service provider to the customer, the cloud exchange point 303 'transports' service traffic and connects the cloud service provider 320 to the customer 308 in the high bandwidth local environment of the data center 300 provided by the data center based IP / MPLS fabric 301 as described herein. In some examples, the IP / MPLS fabric 301 implements IP=VPN using the count described in Rosen & Rekhter, "BGP / MPLS IP Virtual Private Networks (VPNs)," Request for Comments 4364, February 2006, Internet Engineering Task Force (IETF) Network Working Group, the entire contents of which are incorporated herein by reference. In some example configurations, the customer network 308 and the cloud service provider network 320 can be connected to the same PE router of the IP / MPLS fabric 301 via respective links.

[0065] The access links 316 and the aggregation links 322 can include attachment circuits that exchange traffic with the connected customer network 308 or cloud service provider network 320 with a virtual routing and forwarding instance (VRF) configured in the PE 302, 304 and corresponding to an IP-VPN operating over the IP / MPLS fabric 301. For example, the PE 302A can exchange IP packets with the PE 310A over a bidirectional label switched path (LSP) operating over the access link 316A, the LSP being an attachment circuit for a VRF configured in the PE 302A. As another example, the PE 304A can exchange IP packets with the PE 312A over a bidirectional label switched path (LSP) operating over the access link 322A, the LSP being an attachment circuit for a VRF configured in the PE 304A. Each VRF can include or represent different routes and forwarding tables with different routes.

[0066] PE routers 302, 304 of IP / MPLS fabric 301 can be configured in a corresponding hub-and-spoke arrangement for cloud services, where PE 304 implements the cloud service hub and PE 302 is configured as a spoke of the hub (for various hub-and-spoke instances / arrangements). The hub-and-spoke arrangement ensures that service traffic can flow between the hub PE and any spoke PE, but not directly between different spoke PEs. As further described below, in a hub-and-spoke arrangement of IP / MPLS fabric 301 based on a data center and for southbound service traffic (i.e., from CSP to customer), PE 302 advertises routes received from PE 310 to PE 304, which in turn advertises the routes to PE 312. For northbound service traffic (i.e., from customer to CSP), PE 304 advertises routes received from PE 312 to PE 302, which in turn advertises the routes to PE 310.

[0067] For some customers of cloud exchange point 303, the cloud exchange point 303 provider may configure a full mesh arrangement, whereby a set of PEs 302, 304 are each coupled to a different customer site network of the customer. In this case, IP / MPLS fabric 301 implements a Layer 3 VPN (L3VPN) for box-to-box or redundant traffic (also known as east-west or horizontal traffic). L3VPNs can implement closed user groups, whereby each customer site network can send traffic to each other, but cannot send or receive traffic outside the L3VPN.

[0068] PE routers may be coupled to each other according to a peer-to-peer model without using an overlay network. That is, PE 310 and PEs 312 and 328 may not operate directly as peers with each other to exchange routes, but may exchange routes indirectly via IP / MPLS architecture 301. Figure 3BIn the example of FIG, cloud exchange point 303 is configured to implement multiple Layer 3 virtual connectors 332A-332C (collectively, “virtual connectors 332”) to interconnect customer networks 308 to virtual gateways 130 dedicated to customer and cloud service provider networks 322 using end-to-end IP paths. Dedicated virtual connectors 334A and 334B (collectively, “dedicated virtual connectors 334”) are defined between cloud exchange point 303 and customer networks 308 via dedicated virtual gateways 130. These virtual connectors 332 and 334 are configured such that all network traffic between customer networks 308, cloud service provider networks 320, and remote cloud exchange 328 is routed through dedicated virtual gateways 130. Each of cloud service provider 320 and virtual gateway 130 can be an endpoint for multiple virtual connectors 330, which traverse one or more attachment circuits between PE / PE or PE / CE pairs for IP / MPLS fabric 301 and CSP / customer. Virtual connector 330 represents a Layer 3 path between virtual gateway 130 and the attachment circuit connecting the cloud service provider network to fabric 301, through IP / MPLS fabric 301. Dedicated virtual connector 334 represents a path between virtual gateway 130 and access link 316 connecting the customer network to fabric 301. Each virtual connector 330 may include at least one tunnel (e.g., an LSP and / or Generic Routing Encapsulation (GRE) tunnel) having endpoints at PEs 302, 304. PEs 302, 304 may establish a complete mesh of tunnels interconnecting one another.

[0069] Each virtual connector 330 may comprise a different hub-and-spoke network configured within an IP / MPLS network 301, having PE routers 302, 304 that exchange routes using a full or partial mesh of Border Gateway Protocol (BGP) peering sessions, in this example a full mesh of Multi-Protocol Interior Border Gateway Protocol (MP-iBGP) peering sessions. MP-iBGP, or MP-BGP for short, is an example of a protocol by which routers exchange labeled routes to implement MPLS-based VPNs. However, PEs 302, 304 may use other technologies and / or protocols to exchange routes to implement IP-VPNs.

[0070] In the example of virtual connector 332A, PE router 312A of cloud service provider network 320A can send a route for cloud service provider network 320A to PE 304A via a routing protocol (e.g., eBGP) peering connection with PE 304A. PE 304A associates the route with a hub-and-spoke network, which can have an associated VRF that includes spoke PE routers 302A. PE 304A then exports the route to virtual gateway 130A operating on PE router 302A. Virtual gateway 130A can export the route with virtual gateway 130A designated as the next-hop router and a label identifying the hub-and-spoke network. Virtual gateway 130A sends the route to PE router 310B via a routing protocol connection with PE 310B. Virtual gateway 130A can send the route after adding an autonomous system number of cloud exchange point 303 (e.g., to a BGP autonomous system path (AS_PATH) attribute) and designating PE router 302A as the next-hop router. Thus, cloud exchange point 303 is an autonomous system “hop” in the autonomous system path from customer 308 to cloud service provider 320 (and vice versa), even though cloud exchange point 303 can be located within a data center. PE router 310B installs the route to a routing database, such as a BGP routing information base (RIB), to provide Layer 3 reachability to cloud service provider network 320A. In this way, cloud exchange point 303 “leaks out” routes from cloud service provider network 320 to customer network 308 without requiring a direct layer peering connection from cloud service provider network 320 to customer network 308.

[0071] PE routers 310B, 302A, 304A, and 312A can perform similar operations in the opposite direction to forward routes initiated by customer network 308B to PE 312A and thus provide connectivity from cloud service provider network 320A to customer network 308B. In the example of virtual connectors 332B and 334A, PE routers 312B, 304A, and 310B and virtual gateway 130A exchange routes for customer network 308B and cloud service provider 320B in a similar manner to that described above for establishing virtual connectors 332B and 334A. As a result, cloud exchange point 303 within data center 300 internalizes the peering connections that would otherwise be established between PE 310B and each of PEs 312A, 312B in order to perform cloud aggregation for multiple Layer 3 cloud services provided by different cloud service provider networks 320A, 320B and deliver the multiple aggregated Layer 3 cloud services to customer network 308B having a single access link 316B to cloud exchange point 303.

[0072] In the example where the IP / MPLS fabric 301 implements BGP / MPLS IP VPNs or other IP- VPNs that use route targets to control the distribution of routes within the IP backbone network, the PEs 304 can be configured to import routes from the virtual gateway 130 of the PE 302 using different asymmetric route targets and to export routes received from the PE 312 using asymmetric route targets. Likewise, the virtual gateway 130 of the PE 302 can be configured to import routes from the PEs 304 using asymmetric route targets and to export routes received from the PE 310 using asymmetric route targets. Thus, the PEs 302, 304 can be configured to implement advanced L3VPNs that each include a basic backbone L3VPN of the IP / MPLS fabric 301 and an extranet that attaches to any of the cloud service provider networks 320 and any of the customer networks 308 of the basic backbone L3VPN.

[0073] Each advanced L3VPN constitutes a cloud service delivery network from the cloud service provider networks 320 to one or more of the customer networks 308, and vice versa. In this way, the cloud exchange point 303 enables any of the cloud service provider networks 320 to exchange cloud service traffic with any of the customer networks 308 while internalizing what would otherwise be a Layer 3 routing protocol peering connection between the customer networks 308 and the cloud service provider networks 320 pair established for any cloud service connection between the given pair. In other words, the cloud exchange point 303 allows each of the customer networks 308 and the cloud service provider networks 320 to establish a single (or multiple for redundancy or other reasons) Layer 3 routing protocol peering connection to a data center based Layer 3 connection. By filtering routes from the cloud service provider networks 320 to the customer networks 308, and vice versa, the PEs 302, 304 thereby control the establishment of the virtual connectors 330 and the associated flow of cloud service traffic within the data center 300 between the customer networks 308 and the cloud service provider networks 320. The routes distributed into the MP-iBGP mesh 318 can be VPN-IPv4 routes and associated with route distinguishers to distinguish routes from different sites that have overlapping address spaces.

[0074] The programmable network platform 120 can receive a service request to create, read, update, and / or delete an end-to-end service of the cloud exchange point 303. In response, the programmable network platform 120 can configure the virtual gateway 130 of the PE 302 and can configure the PE 304 and / or other network infrastructure of the IP / MPLS fabric 301 to provision or obtain performance or other operational information about the service. Operations to provision the service and performed by the programmable network platform 120 can include configuring or updating a VRF, installing SDN forwarding information, configuring LSPs or other tunnels, configuring BGP, configuring access links 316 and aggregation links 322, or otherwise modifying a configuration of the IP / MPLS fabric 301. Other operations can include issuing service requests to a coordination system of the cloud service provider network 320, as described in further detail below.

[0075] Figure 4 is a block diagram illustrating an example of a data center based cloud exchange point, in which routers of the cloud exchange point are configured by the programmable network platform 120, in which VPN routing and forwarding instances are used to route and forward aggregated service traffic from multiple cloud service provider networks to a customer network, in accordance with the techniques described herein. In this example, to establish the virtual connectors 332A-332B, the PE router 302A is configured with the virtual gateway 103, and the PE router 304A is configured with VRFs 402 and 404. The VRFs 402 and 404 are configured to import routes exported by the virtual gateway 130. The configuration can include asymmetric route targets for import / export between the VRFs 402 and 404 and the virtual gateway 130. This configuration thereby enables the customer to access multiple layer 3 services from different CSPs, each CSP associated with a separate VRF to access the layer 3 service, providing isolation of the exchanged respective traffic from the CSPs. As mentioned above with respect to Figures 3A-3B The PE routers 302, 304 can also be configured to bridge layer 2 traffic between the customer 308B and the cloud service providers 320.

[0076] In this example, the PE router 304A operates BGP or other route distribution protocol peering connections 406B, 408B with the respective PE routers 312A, 312B to exchange routes with the respective cloud service provider networks 320A, 320B. The PE router 302A operates BGP or other route distribution protocol peering connection 410 with the PE router 310B to exchange routes with the customer network 308B. In some examples, the PE routers 302A, 304A can be statically configured with routes for the site network.

[0077] The administrator or programmable network platform described herein for the cloud exchange point 303 can configure the PE routers 302A with virtual gateways 130 and the PE routers 304A with VRFs 402 and 404 to leak out routes between the PE router 312 and the PE router 310B and to facilitate Layer 3 connectivity for end-to-end IP paths illustrated here by the virtual connectors 332 and 334, while possibly optimizing the end-to-end IP paths by facilitating data center or at least metro-based connectivity. Thus, the cloud exchange point 303 can provide dedicated cloud service provider access to the customer network 308B through private and / or public routing for the cloud service provider network 320. In the northbound direction, the cloud exchange point 303 can provide dedicated cloud service provider distribution to multiple customer networks 308 through private and / or public routing for the customer networks 308. The PE router 310B and any of the PE routers 302A, 304A need not access the full Internet BGP routing table to reach the cloud service provider network 320 or the customer networks 308. Further, the PE routers 302A, 304A can be configured to aggregate customer / CSP routing and / or service traffic based on any one or more of physical, IP, service, and VRF.

[0078] Figure 5 is a block diagram illustrating an example of a data center-based cloud exchange point configured to apply network address translation and route and forward aggregated service traffic from multiple cloud service provider networks to customer networks in accordance with the techniques described herein.

[0079] For ease of illustration, Figure 5 The cloud service provider networks 320, the remote cloud exchange network 326, and the customer networks 308 are not shown in FIG. 7. In these examples, the data center-based cloud exchange point 303 applies a network address translation (NAT) service 719 to partially enforce network address separation between a cloud service layer accessible via the cloud aggregation links 322 and a cloud access layer accessible via the cloud access links 316.

[0080] The (multiple) cloud exchange point 303 NAT devices that apply the NAT service 719 perform NAT (or NAPT), which may also or alternatively include carrier-grade NAT ("CG-NAT" or "CGN") to translate cloud exchange point 303 addresses and CSP routes and / or translate cloud exchange point 303 addresses and customer routes. The (multiple) cloud exchange point 303 NAT devices that apply the NAT service 719 (also referred to herein as "NAT service 719 devices") may include one or more dedicated NAT devices, one or more virtual machines that execute on (multiple) real servers and are configured to apply NAT using network function virtualization (NFV), one or more service cards that are configured to apply the NAT service 719 and are inserted into the inbox or outbox of one or more of the PEs 302, 304 or (multiple) other devices.

[0081] Figure 5 The NAT service 719 can be implemented in one or more NAT service devices. Figure 5 708A-708B. NAT service 719 is associated with address pool 720, which is configured with routes for the cloud exchange point 303 autonomous system, and NAT service 719 can draw from address pool 720 to automatically provision and map routes received via peering sessions 700 and 708A-708B, respectively, to customers and / or cloud service providers for NAT purposes. The network addresses used to configure routes in address pool 720 (or "NAT pool 720") can be public, private, or a combination thereof, and can represent IPv4 and / or IPv6 routes. In some examples, the network addresses are public to provide global uniqueness to the network addresses.

[0082] The address mapping 722 may specify one or more NAT mappings and / or network address and port translations (NAPTs) that associate routes from the address pool 720 of the cloud exchange point 303 with routes received by the cloud exchange point 303 router from any of the PEs 310, 312. Routes received from any of the PEs 310, 312 for translation and use in end-to-end service delivery may include any IP address / prefix from an enterprise / NSP customer of the cloud exchange provider, such addresses including private and / or public IPv4 and / or IPv6 addresses, and received at any one or more cloud exchange points managed by the cloud exchange provider.

[0083] As noted above, the NAT service 719 can perform NAT to translate traffic destined for the customer network 308B ( Figure 5 ) and advertises the routes to the cloud exchange point 303 of PE 312A, 328 for aggregated cloud access. As a result, the CSP network 320 ( Figure 5The cloud exchange point 303 therefore is able to filter customer network information from the CSP, and the CSP receives cloud exchange point 303 routes associated with a single autonomous system (i.e., one ASN per cloud exchange point 303), rather than customer routes (which can be in the millions) associated with multiple different autonomous systems (and corresponding ASNs, which can be in the hundreds) for different customers (enterprises and / or NSPs).

[0084] Further, since the cloud exchange point 303 does not advertise its routes other than to customers and CSPs, the cloud exchange point 303 does not advertise its routes to the Internet, which can improve security and reduce the likelihood of denial of service (DoS) or other malicious activity against the cloud exchange point 303 and customers / CSPs that have peering relationships with the cloud exchange point 303. Further, the above-described techniques can simplify end-to-end cloud service delivery processing and improve performance by ensuring that local traffic is handled locally (within the cloud exchange point 303).

[0085] In the illustrated example, the NAT service 719 is associated with an ingress service VRF 712 ("ingress 712") and an egress service VRF 714 ("egress 714") for attracting service traffic associated with the customer network 308B, that is, traffic that is NATed (translated by network address translation). The ingress 712 and egress 714 constitute part of a customer service chain for cloud service traffic between the customer network 308B and the CSP networks 320A, 320B. The private virtual gateway 710 associated with the customer network 308B receives routes from the customer PE 310B via the peering session 700. The private virtual gateway 710 can be configured in a VPN full mesh relationship with the ingress service VRF distributed in the cloud exchange point 303 (however, only one peering session 702 is illustrated).

[0086] In some examples, PE 302A distributes the customer routes received via peering session 700 to NAT service 719 for dedicated virtual gateway 710, which dynamically maps the customer route prefixes to cloud exchange point route prefixes drawn from address pool 720. The customer routes are installed to ingress service VRF 712. NAT service 719 installs the mappings to address mappings 722 and to egress service VRF 714, which cloud exchange routes specify cloud exchange point route prefixes and NAT service 719 as next hop. In this way, NAT service 719, and more particularly egress service VRF 714, attracts downstream traffic from CSP network 320 that is destined for customer network 308B but destined for cloud exchange routes installed to egress service VRF 714. Ingress service VRF 712 and egress service VRF 714 can establish peering session 704 and be configured with route targets to leak routes to each other, e.g., via iBGP.

[0087] Egress service VRF 714 can operate as a spoke VRF for corresponding hub VRFs 730A, 730B in a manner similar to dedicated virtual gateway 710 operating PE 302A in the example of Figure 4 That is, egress service VRF 714 and VRFs 730A, 730B are configured with each other’s route targets such that egress service VRF 714 advertises routes for egress service VRF 714 to install to VRFs 730A, 730B, and VRFs 730A, 730B advertise routes to egress service VRF 714 with routes for corresponding CSP networks 320A and remote cloud exchanges 326. NATed (network address translated) upstream service traffic destined for any of CSP networks 320A and remote cloud exchanges 326 pass through corresponding hub VRFs 730A, 730B. Each of peering sessions 706A, 706B can be used in this way to create hub-and-spoke VPNs for respective CSP networks 320A, 320B.

[0088] PEs 302, 304 can establish tunnels with NAT service 719 device. Routes exchanged via peering sessions 702 and 706A, 706B can include labeled routes for implementing MPLS / BGP IP-VPN according to RFC 4364, incorporated above.

[0089] As follows, cloud exchange point 303 can forward and apply NAT service 719 to downstream service traffic from PE 312A that is destined for customer network 308A. PE 304A receives the service packet on aggregation link 322A. The destination address of the packet is the cloud exchange point 303 address drawn from address pool 720. VRF 730A associated with aggregation link 322A stores a route for the destination address that specifies the address of the NAT service 719 device, and PE 304A uses VRF 730A to tunnel the packet to the NAT service 719 device for application of the NAT service.

[0090] NAT service 719 performs NAT using address mapping 722 that was dynamically provisioned for customer network 308A and received from PE 302A, and replaces the service packet destination address with the destination address in customer network 308A. NAT service 719 device can determine a labeled route to PE 302A in ingress service VRF 712 (labeled to identify private virtual gateway 710), and tunnel the modified service packet to PE 302A, which can identify private virtual gateway 710 from the label attached to the modified service packet. PE 302A forwards the modified service packet to PE 310B via access link 316B. In this way, cloud exchange point 303 provides NAT service for customers to separate customers from the cloud service layer. In a similar way, cloud exchange point 303 can apply NAT to upstream traffic to separate cloud service providers from the customer network access cloud exchange point cloud or network access layer.

[0091] Figure 6 is a block diagram illustrating an example use case according to one or more aspects of the techniques of this disclosure. The techniques of this disclosure can allow SaaS and / or PaaS providers that are currently hosted within other public IaaS providers to become sellers on a cloud exchange 504. In this example, a service provider purchases an L3 instance 512 within cloud exchange 504 and becomes a seller in cloud exchange 504. A customer 510 can purchase services (e.g., data storage and / or data analytics services) from services offered via cloud exchange 504. At the back end, the service provider can use a direct connect service offered by CSP 320A to transmit data of customer 510 back to its primary compute services running in CSP 320A via private virtual gateway 130, in some examples. Thus, customer 510 does not need to establish a relationship with CSP 320 in order to establish a relationship with the service provider. This facilitates SaaS providers operating on third-party IaaS (here, CSP 320A).

[0092] Underlying CSP connectivity to CSP 320A can be "linked" to an L3 instance 512 owned by the service provider. L3 instance 512 allows cloud exchange 504 to be configured to support the service provider. In some examples, if the primary owner of a service port has been designated as a CSP host, cloud exchange 504 can be configured to terminate service profiles to physical ports belonging to entities other than the owner of the service profile. The hosting service provider provides information about its environment, and the system of cloud exchange 504 can validate against that information. In this way, use of L3 instance 512 can allow for additional physical ports and letters of agreement (LOAs) from providers other than the port owner. For example, when a service provider residing on CSP 320A wants a port, cloud exchange 504 delivers the port to the facilities of CSP 320A and delivers the LOA to the service provider.

[0093] In some examples, the service provider's utilization and reporting is limited to those services that are terminated at the service provider, or their virtual port channel (VPC) in the case of provisioning on the same port. In some examples, the owner of the primary port can set limits on the primary port, such as limits on how much bandwidth that has been sold out can be used by the service provider. CSP 320A can be considered a "reseller" of cloud exchange assets (e.g., physical ports or virtual port channels), and CSP 320A can use the techniques described in U.S. Patent Application No. 15 / 414,697, filed January 25, 2017, entitled "ASSET-BASED PERMISSIONS MANAGEMENT FOR RESELLERS OF CLOUD EXCHANGE ASSETS," the entirety of which is incorporated by reference herein, to control asset-based permissions management.

[0094] L3 instance 512 can represent a VRF configured by a PE (e.g., a PE associated with CSP 320A) to establish a virtual connector between the service provider and the virtual gateway 130 of customer 510. The service provider connects to customer 510 through an end-to-end L3 connection 518 of L3 instance 512 and a dedicated virtual gateway 130. In some examples, connection 518 can be a virtual connector as described above.

[0095] Figure 7 is a block diagram illustrating an example system 800 in which cloud service providers (e.g., CSPs 802A-802C) exist in one or more metro data centers 816A-816C. Metro data centers 816A-816C can be geographically separate from one another. For example, metro data center 816A can be located in Chicago, while metro data center 816B can be located in Dallas. Metro data centers 816A-816C are interconnected.

[0096] In Figure 7 In the illustrated example, CSP A 802A has CSP A devices 810A that are physically located in metro data center 816A (e.g., co-located in a rack in a data center operated by a cloud exchange provider). CSP B 802B has CSP B devices 810B that are physically located in metro data center 816B (e.g., co-located in a rack in a data center operated by a cloud exchange provider). Further, CSP C 802C has CSP C devices 810C that are physically located in metro data center 816C (e.g., co-located in a rack in a data center operated by a cloud exchange provider). Additionally, in the illustrated example, a customer has customer devices 812A in metro data center B 816B (e.g., co-located in a rack in a data center operated by a cloud exchange provider) and customer devices 812B in metro data center B 816B.

[0097] A customer can subscribe to connect CSPs 802A-802C at different metro data centers 816A-816C for the customer network to access via customer devices 812A and 812B connected at different metro data centers 816A-816C. When a customer subscribes to CSPs 802A-802C in metro data centers 816A-816C, the corresponding metro cloud exchanges 820A-820C configure a dedicated virtual gateway 804A-804C in the respective PE routers 814A-814C through which traffic related to the customer flows through metro cloud exchanges 820A-820C. For example, when a customer subscribes to CSP B 802B, metro cloud exchange 820B configures a dedicated virtual gateway 804B. When a customer subscribes to CSPs 802A-802C located at another metro data center 816A-816C, the dedicated virtual gateways 804-804C automatically exchange routes. Intra-cloud exchange traffic (e.g., network traffic between CSP B devices 810B and customer devices 812A, etc.) and inter-exchange traffic (e.g., network traffic between metro cloud exchange 820A and metro cloud exchange 820B, etc.) to the customer are routed through the dedicated virtual gateway 804A. For example, when traffic from customer devices 812A is destined for CSP C devices 810C, the network traffic is routed through dedicated virtual gateway 814C and dedicated virtual gateway 814C. That is, by establishing a virtual connector with the dedicated virtual gateway 804C in metro cloud exchange 820C, CSP C 802C can establish an end-to-end connection 822 between CSP C devices 810C in metro data center 816C and customer devices 812A in metro B data center 816B.

[0098] The dedicated virtual gateways 814A-C monitor and manage network traffic for the metro cloud exchanges 820A-C. The customer is licensed for bidirectional traffic at each metro data center 816A-C. This licensed traffic defines a maximum bandwidth, i.e., rate, that can be routed through the dedicated virtual gateway(s) 804A-C at the metro data center 816A-C, and thus defines a total amount of licensed bandwidth that the customer can use for all of its services. This bidirectional traffic subscription includes inter-exchange network traffic and intra-exchange network traffic. For example, the license for metro data center 816A can define a maximum bandwidth of 2 Gbps. Typically, when a customer subscribes with a cloud service provider, the subscription includes a cloud service traffic license. For example, the license for CSP A 802A can define a maximum bandwidth of 500 Mbps.

[0099] The dedicated virtual gateways 814A-B police network traffic for the customer to ensure that traffic between the dedicated virtual gateways 814A-B (e.g., inter-exchange network traffic) and traffic to and from the cloud service providers 802A-C does not exceed the bandwidth limits of the customer’s subscription. This provides the customer with a single interface to monitor and manage all of the customer’s subscribed bandwidth usage. For example, network traffic through the dedicated virtual gateway 804C can include network traffic between CSP C device 810C and customer device 812B, network traffic between dedicated virtual gateway 804C and dedicated virtual gateway 804A, and network traffic between dedicated virtual gateway 804C and dedicated virtual gateway 804B. For example, if the customer’s subscription for metro C data center 816C is 1 Gbps, and the network traffic between dedicated virtual gateway 804C and dedicated virtual gateway 804A is 200 Mbps, then the local traffic (e.g., intra-exchange traffic) cap at metro C data center 816C can be 800 Mbps. As the traffic demand changes with inter-exchange traffic and intra-exchange traffic, the dedicated virtual gateways 804A-C dynamically shape the traffic to comply with the customer’s subscription.

[0100] In some examples, the dedicated virtual gateways 804A-C can use destination class usage (DCU) to use destination-based rate limiting. The DCU counts packets from the customer by performing a lookup on the destination address. The DCU enables tracking of traffic that originates from the customer and is destined for a particular cloud service provider. For example, this facilitates traffic shaping so that traffic destined for a cloud service provider (e.g., CSP 802A-C, etc.) does not exceed the customer’s subscription and does not risk being dropped by the cloud service provider. By policing aggregate traffic to customer destinations, the dedicated virtual gateways can ensure that the bandwidth of the aggregate traffic does not exceed the customer’s licensed bandwidth.

[0101] For example, the programmable network platform 820 for a cloud exchange can map the respective border gateway protocol communities of the CSP networks 802C announced by the VG 804C to a single policer for the VG 804A, and then configure the remote bandwidth limits for the customer's aggregate bandwidth subscription in the single policer in the PE 814A 804A that limits bandwidth between the VG 804A and the VG 804C.

[0102] In contrast to the management of virtual link subscriptions and network traffic, this management of data center subscriptions and network traffic using the dedicated virtual gateways 804A-804C simplifies subscription management. For example, instead of managing subscriptions and network traffic for (i) to (ix) below: (i) virtual connectors between the customer device 812A and the CSP B device 810B, (ii) virtual connectors between the customer device 812A and the CSP A device 810A, (iii) virtual connectors between the customer device 812A and the CSP B device 810C, (iv) virtual connectors between the customer device 812B and the CSP B device 810B, (v) virtual connectors between the customer device 812B and the CSP A device 810A, (vi) virtual connectors between the customer device 812B and the CSP B device 810C, (vii) virtual connectors between the CSP A device 810A and the CSP B device 810B, (viii) virtual connectors between the CSP A device 810A and the CSP C device 810C, and (ix) virtual connectors between the CSP B device 810B and the CSP C device 810C, the dedicated virtual gateways 804A-804C facilitate management of subscriptions and network traffic for (i) to (vi) below: (i) metro A subscriptions, (ii) metro B subscriptions, (iii) metro C subscriptions, (iv) CSP A subscriptions, (v) CSP B subscriptions, and (vi) CSP C subscriptions. Further, for example, if a customer wants to subscribe to an additional CSP at the metro A data center 816, only an additional subscription needs to be configured at the dedicated virtual gateway 804A.

[0103] The techniques described herein can be implemented in hardware, software, firmware, or any combination thereof. Various features described as modules, units or components can be implemented together in an integrated logic device, or separately as discrete but interoperable logic devices or other hardware devices. In some cases, various features of electronic circuitry can be implemented as one or more integrated circuit devices, such as integrated circuit chips or chip sets.

[0104] If implemented in hardware, the present disclosure can involve the use of one or more apparatus such as a processor or an integrated circuit device, such as an integrated circuit chip or chipset. Alternatively or additionally, if implemented in software or firmware, the techniques can at least partially be implemented by computer-readable data storage medium comprising instructions that, when executed by a processor, perform one or more of the methods described above. For example, the computer-readable data storage medium can store the instructions that are executed by the processor.

[0105] The computer-readable medium can form part of a computer program product, which can include packaging materials. The computer-readable medium can include a computer data storage medium such as a random access memory (RAM), a read-only memory (ROM), a non-volatile random access memory (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a FLASH memory, a magnetic or optical data storage medium, and / or the like. In some examples, the article of manufacture can include one or more computer-readable storage media.

[0106] In some examples, the computer-readable storage media can include a non-transitory medium. The term “non-transitory” can indicate that the storage medium is not embodied in a carrier wave or a propagated signal. In certain examples, a non-transitory storage medium can store data that can change over time (e.g., in RAM or a cache).

[0107] Code or instructions can be software and / or firmware executed by processing circuitry, which includes one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Accordingly, the term “processor” as used herein can refer to any of the foregoing structure or any other structure suitable for implementation of the techniques described herein. In addition, in some aspects, the functionality described herein can be provided within software modules or hardware modules within the devices described herein.

Claims

1. A networking system comprising: a first cloud exchange network of a first cloud exchange, the first cloud exchange network being located in a first data center and being configured with a first dedicated virtual gateway, the first dedicated virtual gateway being configured to interface with a first virtual connector to a customer network, a second virtual connector to a first cloud service provider network, and a third virtual connector to a second cloud service provider network, wherein network traffic between the customer network, the first cloud service provider network, and the second cloud service provider network is routed through the first dedicated virtual gateway; as well as A programmable network platform comprising processing circuitry and configured to: obtaining an aggregate bandwidth for an aggregate bandwidth subscription, the aggregate bandwidth subscription defining a total bandwidth for network traffic associated with the first cloud exchange network, the network traffic associated with the first cloud exchange network including network traffic between the customer network and the first cloud service provider network and network traffic between the customer network and the second cloud service provider network; Obtaining local bandwidth for forwarding by the first dedicated virtual gateway to a network co-located in the first data center; setting a remote bandwidth limit for network traffic forwarded by the first dedicated virtual gateway to a second dedicated virtual gateway based on a difference between the aggregate bandwidth and the local bandwidth; as well as The policer of the first dedicated virtual gateway is configured with the remote bandwidth limit so that the first dedicated virtual gateway limits the remote bandwidth for network traffic forwarded to the second dedicated virtual gateway.

2. The networked system according to claim 1, further comprising: For a second cloud exchange network of a second cloud exchange, the second cloud exchange network is located in a second data center that is geographically remote from the first data center, wherein the first dedicated virtual gateway includes a fourth virtual connector to a second dedicated virtual gateway configured on the second cloud exchange network.

3. The networking system of claim 2 , wherein the first dedicated virtual gateway dynamically regulates the network traffic associated with the first cloud exchange network by limiting the total bandwidth for the network traffic associated with the first cloud service provider exchange network based on the aggregate bandwidth subscription, the network traffic associated with the first cloud service provider exchange network also including network traffic between the first dedicated virtual gateway and the second dedicated virtual gateway. 4 . The networking system of claim 2 , wherein the first dedicated virtual gateway exchanges routes with the second dedicated virtual gateway.

5. The networking system according to claim 1, wherein the first dedicated virtual gateway performs destination-based rate limiting such that network traffic to one or more destinations is rate limited, and The first dedicated virtual gateway uses destination class usage and performs the destination-based rate limiting by monitoring network traffic from the customer network by looking up a destination address of network traffic sent to at least one of the first cloud service provider network or the second cloud service provider network.

6. The networking system of claim 5 , wherein based on the destination-based rate limiting, the first dedicated virtual gateway is configured with a first policer and a second policer, the first policer being configured to shape network traffic between the first dedicated virtual gateway and the first cloud service provider network, and the second policer being configured to shape network traffic between the first dedicated virtual gateway and the customer network.

7. The networking system of claim 6, wherein the first dedicated virtual gateway does not directly shape network traffic between the first cloud service provider network and the customer network.

8. The networked system of claim 1 , further comprising: a second cloud exchange network of a second cloud exchange, the second cloud exchange network being located in a second data center that is geographically remote from the first data center, wherein the first dedicated virtual gateway is configured to dynamically shape network traffic on the first cloud exchange network based on inter-exchange traffic between the first dedicated virtual gateway and a second dedicated virtual gateway operating in the second cloud exchange network.

9. The networked system of claim 1 , further comprising: With respect to a second cloud exchange network of a second cloud exchange, the second cloud exchange network is located in a second data center that is geographically remote from the first data center, wherein when the customer network is subscribed to a third cloud service provider connected to the second cloud exchange network, the first dedicated virtual gateway automatically establishes a fourth virtual connector with a second dedicated virtual gateway configured on the second cloud exchange network.

10. A networked system comprising: a first cloud exchange network of a first cloud exchange, the first cloud exchange network being located in a first data center and configured with a first dedicated virtual gateway; a second cloud exchange network of a second cloud exchange, the second cloud exchange network being located in a second data center that is geographically remote from the first data center and configured with a second dedicated virtual gateway; and A programmable network platform comprising processing circuitry and configured to: obtaining an aggregate bandwidth for an aggregate bandwidth subscription, the aggregate bandwidth subscription defining a total bandwidth for network services associated with the first cloud exchange network; Obtaining local bandwidth for forwarding by the first dedicated virtual gateway to a network co-located in the first data center; setting a remote bandwidth limit for network services forwarded by the first dedicated virtual gateway to the second dedicated virtual gateway based on a difference between the aggregate bandwidth and the local bandwidth; as well as The policer of the first dedicated virtual gateway is configured with the remote bandwidth limit so that the first dedicated virtual gateway limits the remote bandwidth for network traffic forwarded to the second dedicated virtual gateway.

11. The networking system of claim 10, wherein the first dedicated virtual gateway configured with the remote bandwidth limit performs destination-based rate limiting to police the network traffic to limit the network traffic forwarded to the second dedicated virtual gateway.

12. The networking system of claim 10, wherein the programmable network platform is configured to: mapping corresponding Border Gateway Protocol communities for one or more networks advertised by the second dedicated virtual gateway to a single supervisor for the first dedicated virtual gateway; and In the first dedicated virtual gateway, the single policer for the first dedicated virtual gateway is configured with a remote bandwidth limit based on a first aggregate bandwidth subscription.

13. A networking method, comprising: configuring, by the programmable network platform, a first dedicated virtual gateway in a first cloud exchange network to interface with a first virtual connector to a customer network, with a second virtual connector to a first cloud service provider network, and with a third virtual connector to a second cloud service provider network, wherein network traffic between the customer network, the first cloud service provider network, and the second cloud service provider network is routed through the first dedicated virtual gateway; implementing, by the programmable network platform, aggregate bandwidth for an aggregate bandwidth subscription, the aggregate bandwidth subscription defining a total bandwidth for network traffic associated with the first cloud exchange network, the network traffic associated with the first cloud exchange network including network traffic between the customer network and the first cloud service provider network and network traffic between the customer network and the second cloud service provider network; Obtaining, by the programmable network platform, local bandwidth for forwarding by the first dedicated virtual gateway to a network co-located in a data center having the first cloud exchange network; Setting, by the programmable network platform, a remote bandwidth limit for network services forwarded by the first dedicated virtual gateway to the second dedicated virtual gateway based on a difference between the aggregate bandwidth and the local bandwidth; as well as The programmable network platform configures a supervisor of the first dedicated virtual gateway using the remote bandwidth limit, so that the first dedicated virtual gateway limits the remote bandwidth for network traffic forwarded to the second dedicated virtual gateway.

14. The networking method of claim 13, wherein the first dedicated virtual gateway comprises a fourth virtual connector to the second dedicated virtual gateway configured on a second cloud exchange network of a second cloud exchange, the second cloud exchange network being located in a second data center that is geographically remote from the first data center.

15. The networking method according to claim 14, further comprising: Dynamically policing the network traffic associated with the first cloud exchange network by limiting total bandwidth for the network traffic associated with the first cloud exchange network based on the aggregate bandwidth subscription, the network traffic associated with the first cloud exchange network also including network traffic between the first dedicated virtual gateway and the second dedicated virtual gateway.

16. The networking method according to claim 14, further comprising: A route is received by the first dedicated virtual gateway from the second dedicated virtual gateway, the route indicating a network reachable via the second dedicated virtual gateway.

17. The method according to claim 16, further comprising: Destination-based rate limiting is performed based on the route from the second dedicated virtual gateway.

18. The method according to claim 17, further comprising: The programmable network platform configures the first dedicated virtual gateway with a first policer and a second policer, wherein the first policer is configured to perform destination-based rate limiting to shape network traffic between the first dedicated virtual gateway and the first cloud service provider network, and the second policer is configured to perform destination-based rate limiting to shape network traffic between the first dedicated virtual gateway and the customer network.

19. The method according to claim 13, further comprising: The first dedicated virtual gateway is configured, by the programmable network platform, to dynamically shape network traffic on the first cloud switch network based on inter-switch traffic between the first dedicated virtual gateway and a second dedicated virtual gateway operating in a second cloud switch network.

Images