Architecture for multi-cloud network links

Abstract

Techniques for creating a network link between a first guest virtual network in a first cloud environment and a second guest virtual network in a second cloud environment are described. A first guest virtual network in the first cloud environment is created to enable a user associated with a guest lease in the second cloud environment to access one or more services provided in the first cloud environment. The network links are created based on one or more links deployed in the first cloud environment and the second cloud environment enabling a virtual network.

Description

[0001] Cross Reference to Related Applications

[0002] This application is a non-provisional of and claims the benefit of each of the following provisional applications. The entire contents of each of the following provisional applications are incorporated herein by reference for all purposes.

[0003] (1) U.S. Provisional Application No. 63 / 465,394, filed May 10, 2023; and

[0004] (2) U.S. Provisional Application No. 63 / 594,759, filed October 31, 2023. TECHNICAL FIELD

[0005] The present disclosure relates to cloud architectures, and more specifically to techniques for linking two cloud environments so that users of one cloud environment can use services provided by the other cloud environment. BACKGROUND

[0006] Over the past few years, the adoption of cloud services has risen dramatically and this trend is only expected to grow. Different cloud service providers (CSPs) provide various different cloud environments, each of which provides a set of one or more cloud services. The set of cloud services provisioned in a cloud environment can include one or more different types of services, including but not limited to software-as-a-service (SaaS) services, infrastructure-as-a-service (IaaS) services, platform-as-a-service (PaaS) services, and the like.

[0007] While there are currently a variety of different cloud environments available, each cloud environment provides a closed ecosystem for its subscribing customers. As a result, customers of a cloud environment are limited to using services provisioned only in that cloud environment. For customers who subscribe to a particular cloud environment provided by a particular CSP, there is no easy way to use service(s) provisioned in a different cloud environment provided by a different CSP via that particular cloud environment.

[0008] The embodiments discussed herein address these and other issues. SUMMARY

[0009] The present disclosure relates to improved cloud architectures, and more specifically to techniques for linking two clouds so that users of one cloud environment can use services provided by the other different cloud environment. Various embodiments are described herein, including methods, systems, non-transitory computer-readable storage media storing programs, code, or instructions executable by one or more processors, and the like. Some embodiments can be implemented by using a computer program product including computer programs / instructions that, when executed by a processor, cause the processor to perform any of the methods described in this disclosure.

[0010] Embodiments of the present disclosure provide a multi-cloud control plane (MCCP) framework that provides the ability to deliver services of a particular cloud network (e.g., Oracle Cloud Infrastructure (OCI)) to users on other clouds (e.g., in Google Cloud (GCP)). The MCCP framework allows users (of the other cloud environment(s)) to access services (e.g., PaaS services) of the cloud environment while providing a user experience as close to the native cloud environment(s) of that user as possible. The key value proposition of the MCCP is that customers will be able to experience the full data plane capabilities of the services in the external cloud.

[0011] One aspect of the present disclosure provides a method comprising: receiving, by a multi-cloud infrastructure included in a first cloud environment, a request to create a network link between a first customer virtual network in the first cloud environment and a second customer virtual network in a second cloud environment; deploying a first link-enabling virtual network in the second cloud environment, the first link-enabling virtual network communicatively coupled to a first data plane hub virtual network, a first control plane hub virtual network, and the second customer virtual network, wherein the first data plane hub virtual network and the first control plane hub virtual network are deployed in the second cloud environment; and enabling traffic to be communicated from the second customer virtual network in the second cloud environment to the first customer virtual network in the first cloud environment via the first link-enabling virtual network, wherein the first link-enabling virtual network receives network configuration information from the first control plane hub virtual network and transmits the traffic to the first cloud environment via the first data plane hub virtual network based on the network configuration information.

[0012] With some embodiments, the first link-enabling virtual network is associated with a first virtual network interface card (VNIC) that communicatively couples the first link-enabling virtual network to the first data plane hub virtual network, a second VNIC that communicatively couples the first link-enabling virtual network to the first control plane hub virtual network, and a third VNIC that communicatively couples the first link-enabling virtual network to the second customer virtual network.

[0013] In certain aspects, the first link-enabled virtual network comprises a first pair of virtual network adapters, each of the first pair of virtual network adapters configured to encapsulate traffic received from the second customer virtual network to generate encapsulated traffic. Additionally, the method according to aspects of the present disclosure further comprises the steps of: encapsulating, by the first link-enabled virtual network in the second cloud environment, traffic received from the second customer virtual network based on network configuration information to generate encapsulated traffic; and forwarding, by the first link-enabled virtual network in the second cloud environment, the encapsulated traffic to the first data plane hub virtual network; deploying a second control plane hub virtual network in the first cloud environment, the second control plane hub virtual network comprising one or more distribution service nodes, wherein the second control plane hub virtual network is communicatively coupled to the first control plane hub virtual network in the second cloud environment via a public interconnection link; and deploying a second link-enabled virtual network in the first cloud environment, the second link-enabled virtual network communicatively coupled to the first customer virtual network, the second link-enabled virtual network comprising a second pair of virtual network adapters, each of the second pair of virtual network adapters configured to decapsulate encapsulated traffic received from the first link-enabled virtual network included in the second cloud environment.

[0014] By one embodiment, at least one of the one or more distribution service nodes is configured to transmit network configuration information to the first control plane hub virtual network in the second cloud environment, wherein the network configuration information is forwarded to the first link-enabled virtual network via the first control plane hub virtual network, and wherein the network configuration information comprises at least (i) tunnel encapsulation-decapsulation parameters, and health information of the second link-enabled virtual network deployed in the first cloud environment.

[0015] By one embodiment, the second cloud environment comprises a plurality of customer tenancies, and wherein for each customer tenancy included in the plurality of customer tenancies: (i) a first link-enabled virtual network is created in the second cloud environment, and (ii) a second link-enabled virtual network is created in the first cloud environment.

[0016] By one embodiment, the method further comprises the steps of: decapsulating, by the second link-enabled virtual network in the first cloud environment, encapsulated traffic received from the first data plane hub virtual network included in the second cloud environment to generate decapsulated traffic; and transmitting, by the second link-enabled virtual network in the first cloud environment, the decapsulated traffic to the first customer virtual network in the first cloud environment.

[0017] By an embodiment, the second link-enabled virtual network in the first cloud environment receives encapsulated traffic from a second data plane hub virtual network included in the first cloud environment, the second data plane hub virtual network communicatively coupled to the first data plane hub virtual network via a private high-bandwidth interconnect coupling the second cloud environment to the first cloud environment. Each of the first link-enabled virtual network and the second link-enabled virtual network is assigned a unique classless inter-domain routing IP address. The multi-cloud infrastructure includes a first portion deployed in the first cloud environment and a second portion deployed in the second cloud environment, each of the first portion and the second portion of the multi-cloud infrastructure controlled by a first cloud service provider of the first cloud environment, the first cloud service provider being different from a second cloud service provider of the second cloud environment.

[0018] One aspect of the present disclosure provides a computing device comprising one or more data processors and a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the computing device to perform part or all of one or more methods disclosed herein.

[0019] Another aspect of the present disclosure provides a computer program product tangibly embodied in a non-transitory machine-readable storage medium including instructions configured to cause one or more data processors to perform part or all of one or more methods disclosed herein.

[0020] The foregoing and other features and embodiments can become more apparent from the following description, claims, and accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0021] The features, embodiments, and advantages of the present disclosure can be better understood with reference to the following specification, claims, and accompanying drawings.

[0022] Figure 1 is a high-level diagram illustrating a distributed environment of virtual or overlay cloud networks hosted by cloud service provider infrastructures, in accordance with certain embodiments.

[0023] Figure 2 depicts a simplified architecture diagram of physical components in a physical network within a CSPI, in accordance with certain embodiments.

[0024] Figure 3 illustrates an example arrangement within a CSPI, in accordance with certain embodiments, in which host machines are connected to multiple network virtualization devices (NVDs).

[0025] Figure 4 depicts connectivity between host machines and NVDs for providing I / O virtualization to support multitenancy, in accordance with certain embodiments.

[0026] Figure 5 A simplified block diagram depicting a physical network provided by a CSPI, in accordance with certain embodiments.

[0027] Figure 6 A simplified high-level diagram depicting a distributed environment comprising multiple cloud environments provided by different cloud service providers (CSPs), in accordance with certain embodiments, wherein a cloud environment comprises a particular cloud environment providing specialized infrastructure that enables one or more cloud services provided by that particular cloud environment to be used by customers of other cloud environments.

[0028] Figure 7 An exemplary high-level architecture of a multi-cloud control plane (MCCP), in accordance with some embodiments, is depicted.

[0029] Figure 8 An exemplary system diagram illustrating components of a multi-cloud control plane (MCCP), in accordance with some embodiments, is depicted.

[0030] Figure 9 A high-level block diagram of a network link component, in accordance with certain embodiments, is depicted.

[0031] Figure 10 A detailed architecture of a network link, in accordance with certain embodiments, is depicted.

[0032] Figure 11 An exemplary flow diagram illustrating a process of establishing a network link, in accordance with certain embodiments, is depicted.

[0033] Figure 12 A schematic diagram of a compute virtual machine comprising multiple VNICs, in accordance with certain embodiments, is illustrated.

[0034] Figure 13 A schematic diagram of a compute virtual machine comprising multiple network namespaces, in accordance with certain embodiments, is illustrated.

[0035] Figure 14 A spoke and hub network architecture, in accordance with certain embodiments, is depicted.

[0036] Figure 15 An exemplary flow diagram illustrating a process performed in a spoke and hub architecture of Figure 14 to enable communication, in accordance with certain embodiments, is depicted.

[0037] Figure 16 is a block diagram illustrating one mode for implementing a cloud infrastructure as a service system, in accordance with at least one embodiment.

[0038] Figure 17 is a block diagram illustrating another mode for implementing a cloud infrastructure as a service system, in accordance with at least one embodiment.

[0039] Figure 18 is a block diagram illustrating another mode for implementing a cloud infrastructure as a service system, in accordance with at least one embodiment.

[0040] Figure 19 is a block diagram illustrating another mode for implementing a cloud infrastructure as a service system, in accordance with at least one embodiment.

[0041] Figure 20 is a block diagram illustrating an example computer system, in accordance with at least one embodiment. DETAILED DESCRIPTION

[0042] In the following description, for the purposes of explanation, specific details are set forth in order to provide a thorough understanding of certain embodiments. However, it will be apparent that various embodiments can be practiced without these specific details. The drawings and description are not intended to be restrictive. The word “exemplary” is used herein to mean “serving as an example, instance, or illustration.” Any implementation described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other implementations.

[0043] The present disclosure relates generally to improved cloud architectures, and more particularly to techniques for linking two clouds so that users of one cloud environment can use services provided by another, different cloud environment. Various embodiments are described herein, including methods, systems, non-transitory computer-readable storage media storing programs, code, or instructions executable by one or more processors, and the like. Some embodiments can be implemented by use of a computer program product comprising a computer program / instructions that, when executed by a processor, cause the processor to perform any of the methods described in the present disclosure.

[0044] Embodiments of the present disclosure provide a multi-cloud control plane (MCCP) framework that provides the ability to deliver services of a particular cloud network (e.g., Oracle Cloud Infrastructure (OCI)) to users on other clouds (e.g., Google Cloud). The MCCP framework allows users (of other cloud environment(s)) to access services (e.g., PaaS services) of a cloud environment while providing a user experience as close to the user’s native cloud environment(s) as possible. The key value proposition of MCCP is that customers will be able to experience the full data plane capabilities of services in an external cloud.

[0045] MCCP enables users of a second cloud infrastructure (e.g., Azure users) to use resources (e.g., database resources) provided by a first cloud infrastructure (e.g., OCI) in a manner that is transparent to the users. Specifically, services provided by the first cloud infrastructure appear as “native” services in the second cloud infrastructure. This allows customers of the second cloud infrastructure to access services provided by the first cloud infrastructure natively. As will be described below, MCCP is a collection of microservices executing in the first cloud infrastructure that expose resources of the first cloud infrastructure for use by external cloud users (e.g., users of the second cloud infrastructure). Each microservice acts as a proxy that provides communication with resources provided by the first cloud infrastructure.

[0046] Examples of cloud networks

[0047] The term cloud service is generally used to refer to a service made available to users or customers on demand (e.g., via a subscription model) by a cloud service provider (CSP) using systems and infrastructure (cloud infrastructure) provided by the CSP. Typically, the servers and systems that make up the infrastructure of the CSP are separate from the customer’s own on-premise servers and systems. Thus, customers can take advantage of cloud services provided by the CSP without having to purchase separate hardware and software resources for the services. Cloud services are designed to provide simple, scalable access to applications and computing resources for subscribers without requiring the subscribers to invest in the infrastructure needed to provide the services.

[0048] There are several cloud service providers that provide various types of cloud services. There are various different types or models of cloud services, including software as a service (SaaS), platform as a service (PaaS), infrastructure as a service (IaaS), etc.

[0049] A customer can subscribe to one or more cloud services provided by a CSP. A customer can be any entity, such as an individual, an organization, a business, etc. When a customer subscribes or registers for a service provided by a CSP, a tenancy or account is created for that customer. The customer can then access one or more cloud resources of the subscription associated with the account via this account.

[0050] As noted above, infrastructure as a service (IaaS) is a particular type of cloud computing service. In an IaaS model, a CSP provides infrastructure that customers can use to build their own customizable networks and deploy customer resources (referred to as cloud service provider infrastructure or CSPI). Thus, the customer’s resources and networks are hosted in a distributed environment by the infrastructure provided by the CSP. This is different from traditional computing, in which a customer’s resources and networks are hosted by infrastructure provided by the customer.

[0051] CSPI can include interconnected high-performance computing resources, including various host machines, memory resources, and network resources that form a physical network, also referred to as substrate network or underlay network. Resources in CSPI can be spread across one or more data centers, which can be geographically spread across one or more geographic regions. Virtualization software can be executed by these physical resources to provide a virtualized distributed environment. Virtualization creates an overlay network (also referred to as software-based network, software-defined network, or virtual network) on top of the physical network. The CSPI physical network provides the underlay foundation for creating one or more overlay or virtual networks on top of the physical network. The physical network (or substrate network or underlay network) includes physical network devices such as physical switches, routers, computers, and host machines, etc. The overlay network is a logical (or virtual) network that runs on top of the physical substrate network. A given physical network can support one or more overlay networks. Overlay networks typically use encapsulation techniques to differentiate traffic belonging to different overlay networks. Virtual or overlay networks are also referred to as virtual cloud networks (VCNs). Virtual networks are implemented using software virtualization techniques (e.g., hypervisors, virtualization functions implemented by network virtualization devices (NVDs) such as smartNICs, top-of-rack (TOR) switches, smart TORs that implement one or more functions performed by NVDs, and other mechanisms) to create a network abstraction layer that can run on top of the physical network. Virtual networks can take many forms, including peer-to-peer networks, IP networks, etc. Virtual networks are typically layer 3 IP networks or layer 2 VLANs. This approach to virtual or overlay networking is often referred to as virtual or overlay layer 3 networking. Examples of protocols developed for virtual networks include IP-in-IP (or generic routing encapsulation (GRE)), virtual extensible LAN (VXLAN — IETF RFC 7348), virtual private networks (VPNs) (e.g., MPLS layer 3 virtual private networks (RFC 4364)), VMware’s NSX, GENEVE (generic network virtualization encapsulation), etc.

[0052] For IaaS, the infrastructure provided by the CSP (CSPI) can be configured to provide virtualized computing resources over a public network (e.g., the Internet). In an IaaS model, a cloud computing service provider can host infrastructure components (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer), etc.). In some cases, an IaaS provider can also provision various services to accompany those infrastructure components (e.g., billing, monitoring, logging, security, load balancing and clustering, etc.). Thus, since these services can be policy driven, an IaaS user can be able to implement policies to drive load balancing to maintain application availability and performance. The CSPI provides infrastructure and a complementary set of cloud services that enable customers to build and run a wide range of applications and services in a highly available hosted, distributed environment. The CSPI provides high-performance computing resources and capabilities, as well as storage capacity, in flexible virtual networks that are securely accessible from a variety of networked locations, such as from a customer's on-premise network. When a customer subscribes or registers for IaaS services provided by the CSP, a tenancy created for that customer is a secure and isolated partition within the CSPI in which the customer can create, organize, and manage their cloud resources.

[0053] A customer can build their own virtual networks using the computing, memory, and networking resources provided by the CSPI. One or more customer resources or workloads, such as compute instances, can be deployed on these virtual networks. For example, a customer can use the resources provided by the CSPI to build one or more customizable and private virtual networks, referred to as virtual cloud networks (VCNs). The customer can deploy one or more customer resources, such as compute instances, on the customer VCNs. The compute instances can take the form of virtual machines, bare-metal instances, etc. Thus, the CSPI provides infrastructure and a complementary set of cloud services that enable customers to build and run a wide range of applications and services in a highly available virtual hosted environment. The customer does not manage or control the underlying physical resources provided by the CSPI, but has control over operating systems, storage, and deployed applications; and possibly limited control of selected networking components (e.g., firewalls).

[0054] A CSP can provide a console that enables customers and network administrators to use CSPI resources to configure, access, and manage resources deployed in the cloud. In certain embodiments, the console provides a web-based user interface that can be used to access and manage the CSPI. In certain implementations, the console is a web-based application provided by the CSP.

[0055] CSPI can support a single-tenancy or multi-tenancy architecture. In a single-tenancy architecture, software (e.g., applications, databases) or hardware components (e.g., host machines or servers) serve a single customer or tenant. In a multi-tenancy architecture, software or hardware components serve multiple customers or tenants. Thus, in a multi-tenancy architecture, CSPI resources are shared among multiple customers or tenants. In a multi-tenancy scenario, precautions are taken and safeguards are implemented in CSPI to ensure that each tenant’s data is isolated and remains invisible to other tenants.

[0056] In a physical network, a network endpoint (“endpoint”) refers to a computing device or system that is connected to the physical network and communicates back and forth with the network to which it is connected. Network endpoints in a physical network can be connected to a local area network (LAN), a wide area network (WAN), or other types of physical networks. Examples of traditional endpoints in a physical network include modems, hubs, bridges, switches, routers, and other network devices, physical computers (or host machines), and the like. Each physical device in a physical network has a fixed network address that can be used to communicate with the device. This fixed network address can be a layer 2 address (e.g., a MAC address), a fixed layer 3 address (e.g., an IP address), and the like. In a virtualized environment or virtual network, endpoints can include various virtual endpoints, such as virtual machines hosted by components of the physical network (e.g., hosted by physical host machines). These endpoints in a virtual network are addressed through overlay addresses, such as overlay layer 2 addresses (e.g., overlay MAC addresses) and overlay layer 3 addresses (e.g., overlay IP addresses). Network overlays enable flexibility by allowing network managers to move around overlay addresses associated with network endpoints using software management (e.g., via software implementing a control plane for the virtual network). Thus, unlike a physical network, in a virtual network, an overlay address (e.g., an overlay IP address) can be moved from one endpoint to another using network management software. Since a virtual network is built on top of a physical network, communication between components in a virtual network involves both the virtual network and the underlying physical network. To facilitate such communication, components of CSPI are configured to learn and store mappings that map overlay addresses in the virtual network to actual physical addresses in the substrate network, and vice versa. These mappings are then used to facilitate communication. Customer traffic is encapsulated to facilitate routing in the virtual network.

[0057] Thus, physical addresses (e.g., physical IP addresses) are associated with components in the physical network, and overlay addresses (e.g., overlay IP addresses) are associated with entities in the virtual or overlay network. A physical IP address is an IP address associated with a physical device (e.g., a network device) in the substrate or physical network. For example, each NVD has an associated physical IP address. An overlay IP address is an overlay address associated with an entity in the overlay network, such as an overlay address associated with a compute instance in a customer's virtual cloud network (VCN). Two different customers or tenants, each with their own private VCN, can potentially use the same overlay IP address in their VCNs without needing to know about each other. Both physical IP addresses and overlay IP addresses are types of real IP addresses. They are separate from virtual IP addresses. A virtual IP address is typically a single IP address that represents or maps to multiple real IP addresses. A virtual IP address provides a one-to-many mapping between the virtual IP address and multiple real IP addresses. For example, a load balancer can use a VIP to map or represent multiple servers, each with its own real IP address.

[0058] The cloud infrastructure or CSPI is physically hosted in one or more data centers in one or more regions around the world. The CSPI can include components in the physical or substrate network and virtualized components (e.g., virtual networks, compute instances, virtual machines, etc.) located in virtual networks built on top of the physical network components. In certain embodiments, the CSPI is organized and hosted in domains, regions, and availability domains. A region is typically a localized geographic area containing one or more data centers. Regions are generally independent of each other and can be far apart, e.g., across countries or even continents. For example, a first region can be in Australia, another in Japan, another in India, etc. The CSPI resources are divided among the regions so that each region has its own independent subset of CSPI resources. Each region can provide a set of core infrastructure services and resources, such as compute resources (e.g., bare metal servers, virtual machines, containers, and related infrastructure, etc.); storage resources (e.g., block volume storage, file storage, object storage, archival storage); networking resources (e.g., virtual cloud networks (VCNs), load balancing resources, connectivity to pre-built networks), database resources; edge networking resources (e.g., DNS); and access management and monitoring resources, etc. Each region generally has multiple paths connecting it to other regions in the domain.

[0059] Generally, applications are deployed in the region where they are used most frequently (i.e., on the infrastructure associated with that region) because it is faster to use resources nearby than to use resources far away. Applications can also be deployed in different regions for various reasons, such as redundancy to mitigate the risk of region-wide events (such as large weather systems or earthquakes), to meet different requirements of legal jurisdictions, tax jurisdictions, and other business or social standards, etc.

[0060] Data centers within a region can be further organized and subdivided into availability domains (ADs). An availability domain can correspond to one or more data centers located within a region. A region can be composed of one or more availability domains. In such a distributed environment, CSPI resources are either region-specific, such as virtual cloud networks (VCNs), or availability domain-specific, such as compute instances.

[0061] ADs within a region are isolated from each other, fault-tolerant, and configured so that they are extremely unlikely to fail at the same time. This is achieved by ADs not sharing critical infrastructure resources (such as networking, physical cables, cable paths, cable entry points, etc.) so that a failure at one AD within a region is unlikely to impact the availability of other ADs within the same region. ADs within the same region can be connected to each other through a low-latency, high-bandwidth network, which enables high-availability connectivity to other networks (e.g., the Internet, customers’ on-premise networks, etc.) and building replication systems across multiple ADs to simultaneously achieve high availability and disaster recovery. Cloud services use multiple ADs to ensure high availability and prevent resource failures. As the infrastructure offered by an IaaS provider grows, more regions and ADs and additional capacity can be added. Traffic between availability domains is typically encrypted.

[0062] In certain embodiments, regions are grouped into realms. A realm is a logical collection of regions. Realms are isolated from each other and do not share any data. Regions in the same realm can communicate with each other, but regions in different realms cannot. A customer’s tenancy or account with a CSP exists in a single realm and can be spread across one or more regions belonging to that realm. Typically, when a customer subscribes to an IaaS service, a tenancy or account for the customer is created in the customer-specified region (called the “home” region) within the realm. The customer can extend the customer’s tenancy to one or more other regions within the realm. The customer cannot access regions that are not in the realm of the customer’s tenancy.

[0063] IaaS providers can offer multiple tenancies, each satisfying the needs of a particular set of customers or users. For example, a commercial tenancy can be offered for business customers. As another example, a tenancy can be offered for customers within a particular country. As yet another example, a government tenancy can be offered for governments, and so on. For example, a government tenancy can satisfy the needs of a particular government and can have a higher level of security than a commercial tenancy. For example, Oracle Cloud Infrastructure (OCI) currently offers a tenancy for commercial regions and two tenancies for government cloud regions (e.g., FedRAMP authorized and IL5 authorized).

[0064] In certain embodiments, an AD can be subdivided into one or more fault domains. A fault domain is a grouping of infrastructure resources within an AD to provide anti-affinity. Fault domains allow distribution of compute instances such that these instances are not located on the same physical hardware within a single AD. This is referred to as anti-affinity. A fault domain refers to a group of hardware components (computers, switches, etc.) that share a single point of failure. A compute pool is logically divided into fault domains. Thus, a hardware failure or compute hardware maintenance event affecting one fault domain does not affect instances in other fault domains. The number of fault domains per AD can vary depending on the embodiment. For example, in certain embodiments, each AD contains three fault domains. Fault domains act as logical data centers within an AD.

[0065] When a customer subscribes to an IaaS service, resources from CSPI are provisioned to the customer and associated with the customer’s tenancy. The customer can use these provisioned resources to build private networks and deploy resources on these networks. A customer network hosted by CSPI in the cloud is referred to as a virtual cloud network (VCN). A customer can set up one or more virtual cloud networks (VCNs) using CSPI resources allocated to the customer. A VCN is a virtual or software-defined private network. Customer resources deployed in a customer’s VCN can include compute instances (e.g., virtual machines, bare-metal instances) and other resources. These compute instances can represent various customer workloads, such as applications, load balancers, databases, and the like. Compute instances deployed on a VCN can communicate with publicly accessible endpoints (“public endpoints”) over a public network, such as the Internet, with other instances in the same VCN or other VCNs (e.g., other VCNs of the customer or VCNs not belonging to the customer), with the customer’s on-premise data center or network, and with service endpoints and other types of endpoints.

[0066] CSPs can use CSPI to offer various services. In some cases, customers of CSPI can themselves behave like service providers and offer services using CSPI resources. Service providers can expose service endpoints characterized by identifying information (e.g., IP addresses, DNS names, and ports). Resources (e.g., compute instances) of customers can use a particular service by accessing a service endpoint exposed by the service for that particular service. These service endpoints are generally endpoints that users can publicly access via a public communications network, such as the Internet, using a public IP address associated with the endpoint. Publicly accessible network endpoints are sometimes also referred to as public endpoints.

[0067] In certain embodiments, a service provider can expose a service via an endpoint for the service, sometimes referred to as a service endpoint. Customers of the service can then use this service endpoint to access the service. In certain implementations, a service endpoint provided for a service can be accessible by multiple customers that intend to consume the service. In other implementations, a dedicated service endpoint can be provided for a customer such that only that customer can use the dedicated service endpoint to access the service.

[0068] In certain embodiments, when a VCN is created, it is associated with a private overlay Classless Inter-Domain Routing (CIDR) address space, which is a range of private overlay IP addresses (e.g., 10.0 / 16) assigned to the VCN. A VCN includes associated subnets, route tables, and gateways. A VCN resides within a single region, but can span one or more or all of the availability domains in that region. Gateways are virtual interfaces configured for the VCN and enable the transfer of traffic to and from the VCN to one or more endpoints outside the VCN. One or more different types of gateways can be configured for a VCN to enable communication to and from different types of endpoints.

[0069] A VCN can be subdivided into one or more subnetworks, such as one or more subnets. Thus, a subnet is a unit or subdivision of configuration that can be created within a VCN. A VCN can have one or more subnets. Each subnet within a VCN is associated with a contiguous range of overlay IP addresses (e.g., 10.0.0.0 / 24 and 10.0.1.0 / 24) that do not overlap with other subnets in that VCN and represent a subset of the address space within the address space of the VCN.

[0070] Each compute instance is associated with a virtual network interface card (VNIC), which enables the compute instance to participate in a subnet of a VCN. A VNIC is a logical representation of a physical network interface card (NIC). Generally, a VNIC is an interface between an entity (e.g., compute instance, service) and a virtual network. A VNIC exists in a subnet, has one or more associated IP addresses, and associated security rules or policies. A VNIC is equivalent to a layer 2 port on a switch. A VNIC is attached to a compute instance and a subnet within a VCN. The VNIC associated with a compute instance enables the compute instance to be part of a subnet of a VCN and enables the compute instance to communicate (e.g., send and receive packets) with endpoints on the same subnet as the compute instance, with endpoints in different subnets in the VCN, or with endpoints outside the VCN. Thus, the VNIC associated with a compute instance determines how the compute instance connects with endpoints inside and outside the VCN. When a compute instance is created and added to a subnet within a VCN, a VNIC for the compute instance is created and associated with the compute instance. For a subnet that includes a set of compute instances, the subnet contains VNICs corresponding to the set of compute instances, each VNIC attached to a compute instance within the set of compute instances.

[0071] Each compute instance is assigned a private overlay IP address via the VNIC associated with the compute instance. This private overlay network IP address is assigned to the VNIC associated with the compute instance when the compute instance is created and is used to route traffic to and from the compute instance. All VNICs in a given subnet use the same routing table, security list, and DHCP options. As described above, each subnet within a VCN is associated with a contiguous range of overlay IP addresses (e.g., 10.0.0.0 / 24 and 10.0.1.0 / 24) that do not overlap with other subnets in the VCN and represent a subset of the address space within the address space of the VCN. For a VNIC on a particular subnet of a VCN, the private overlay IP address assigned to the VNIC is an address from the contiguous range of overlay IP addresses allocated for the subnet.

[0072] In certain embodiments, in addition to a private overlay IP address, a compute instance can optionally be assigned additional overlay IP addresses, such as one or more public IP addresses if in a public subnet, for example. The multiple addresses are assigned either on the same VNIC or on multiple VNICs associated with the compute instance. However, each instance has a primary VNIC, which is created during instance launch and is associated with the overlay private IP address assigned to the instance - this primary VNIC cannot be deleted. Additional VNICs, referred to as secondary VNICs, can be added to existing instances in the same availability domain as the primary VNIC. All VNICs are in the same availability domain as the instance. The secondary VNICs can be in the same subnet in the same VCN as the primary VNIC, or in different subnets in the same or different VCNs.

[0073] If a compute instance is in a public subnet, it can optionally be assigned a public IP address. At the time a subnet is created, the subnet can be designated as a public subnet or a private subnet. A private subnet means that resources (e.g., compute instances) and associated VNICs in the subnet cannot have public overlay IP addresses. A public subnet means that resources and associated VNICs in the subnet can have public IP addresses. A customer can specify that a subnet exists in a single availability domain or across multiple availability domains in a region or realm.

[0074] As noted above, a VCN can be subdivided into one or more subnets. In certain embodiments, a virtual router (VR) configured for a VCN (referred to as a VCN VR or simply a VR) enables communication between subnets of the VCN. For a subnet within a VCN, the VR represents a logical gateway for that subnet that enables the subnet (i.e., compute instances on the subnet) to communicate with endpoints on other subnets within the VCN as well as with other endpoints outside the VCN. The VCN VR is a logical entity that is configured to route traffic between VNICs in the VCN and virtual gateways ("gateways") associated with the VCN. More detail is provided below regarding Figure 1A gateway is further described. A VCN VR is a layer 3 / IP layer concept. In one embodiment, there is one VCN VR for a VCN, where the VCN VR potentially has an unlimited number of ports addressed by IP addresses, one port for each subnet in the VCN. In this way, the VCN VR has a different IP address for each subnet in the VCN to which the VCN VR is attached. The VR is also connected to various gateways configured for the VCN. In certain embodiments, a particular overlay IP address in the overlay IP address range for a subnet is reserved for a port of the VCN VR for that subnet. For example, consider a VCN has two subnets with associated address ranges of 10.0 / 16 and 10.1 / 16, respectively. For the first subnet in the VCN with address range 10.0 / 16, addresses in this range are reserved for ports of the VCN VR for that subnet. In some cases, the first IP address in the range can be reserved for the VCN VR. For example, for a subnet with an overlay IP address range of 10.0 / 16, the IP address 10.0.0.1 can be reserved for a port of the VCN VR for that subnet. For the second subnet in the same VCN with address range 10.1 / 16, the VCN VR can have a port for the second subnet with IP address 10.1.0.1. The VCN VR has a different IP address for each subnet in the VCN.

[0075] In some other embodiments, each subnet within a VCN can have its own associated VR that is addressable by the subnet using a reserved or default IP address associated with the VR. For example, the reserved or default IP address can be the first IP address in the IP address range associated with the subnet. VNICs in the subnet can communicate (e.g., send and receive packets) with the VR associated with the subnet using this default or reserved IP address. In such embodiments, the VR is the ingress / egress point for the subnet. The VR associated with a subnet within a VCN can communicate with other VRs associated with other subnets within the VCN. The VR can also communicate with gateways associated with the VCN. The VR functionality for a subnet runs on or is performed by one or more NVDs that perform VNIC functionality for VNICs in the subnet.

[0076] A route table, security rules, and DHCP options can be configured for a VCN. A route table is a virtual route table for a VCN and includes rules for routing traffic from subnets within the VCN to destinations outside the VCN through gateways or specially configured instances. The route table for a VCN can be customized to control how packets are forwarded / routed to and from the VCN. DHCP options refer to configuration information that is automatically provided to instances at instance launch time.

[0077] Security rules configured for a VCN represent overlay firewall rules for the VCN. Security rules can include ingress and egress rules and specify the types of traffic (e.g., based on protocol and port) that are allowed to enter or exit instances within the VCN. A customer can choose whether a given rule is stateful or stateless. For example, a customer can allow incoming SSH traffic from anywhere to a set of instances by setting up a stateful ingress rule with source CIDR 0.0.0.0 / 0 and destination TCP port 22. Security rules can be implemented using network security groups or security lists. A network security group consists of a set of security rules that only apply to resources in that group. On the other hand, a security list includes rules that apply to all resources in any subnet that uses that security list. A default security list with default security rules can be provided for a VCN. DHCP options configured for a VCN provide configuration information that is automatically provided to instances in the VCN at instance launch time.

[0078] In certain embodiments, configuration information for a VCN is determined and stored by a VCN control plane. For example, configuration information for a VCN can include information about the address range associated with the VCN, subnets and associated information within the VCN, one or more VRs associated with the VCN, compute instances and associated VNICs in the VCN, NVDs (e.g., VNICs, VRs, gateways) that perform various virtualized network functions associated with the VCN, state information for the VCN, and other VCN-related information. In certain embodiments, a VCN distribution service publishes to NVDs configuration information or portions thereof stored by the VCN control plane. The distributed information can be used to update information (e.g., forwarding tables, routing tables, etc.) stored by the NVDs and used to forward packets to and from compute instances in the VCN.

[0079] In certain embodiments, creation of VCNs and subnets is handled by a VCN control plane (CP) and launch of compute instances is handled by a compute control plane. The compute control plane is responsible for allocating physical resources for compute instances and then invoking the VCN control plane to create VNICs and attach them to the compute instances. The VCN CP also sends VCN data mappings to a VCN data plane that is configured to perform packet forwarding and routing functions. In certain embodiments, the VCN CP provides a distribution service that is responsible for providing updates to the VCN data plane. Examples of a VCN control plane are also depicted in Figure 6 、 Figure 7 、 Figure 8 and Figure 9 (see reference numbers 616, 716, 816, and 916) and described below.

[0080] A customer can create one or more VCNs using resources hosted by CSPI. Compute instances deployed on a customer VCN can communicate with different endpoints. These endpoints can include endpoints hosted by CSPI and endpoints outside of CSPI.

[0081] Various different architectures for implementing cloud-based services using CSPI are depicted in Figure 1 、 Figure 2 、 Figure 3 、 Figure 4 、 Figure 5 and Figures 18-2 2, and are described below. Figure 1 is a high-level diagram of a distributed environment 100 showing an overlay or customer VCN hosted by CSPI, in accordance with certain embodiments. Figure 1 The distributed environment depicted in Figure 1 The distributed environment 100 depicted in Figure 1 may have more or fewer systems or components than those shown in Figure 1 may combine two or more systems, or can have a different configuration or arrangement of systems.

[0082] As shown in the example depicted in Figure 1 The distributed environment 100 includes a CSPI 101 that provides services and resources that customers can subscribe to and use to build their virtual cloud networks (VCNs), as shown in the example depicted in Figure 1 In certain embodiments, CSPI 101 provides IaaS services to subscribing customers. Data centers within CSPI 101 can be organized into one or more regions.

[0083] In the example depicted in Figure 1 The customer VCN 104 includes two subnets, “Subnet-1” and “Subnet-2,” each with its own CIDR IP address range, in the embodiment depicted in Figure 1In this example, the overlay IP address range for subnet-1 is 10.0 / 16, and the address range for subnet-2 is 10.1 / 16. The VCN virtual router 105 represents a logical gateway for the VCN that enables communication between the subnets of the VCN 104 and with other endpoints outside the VCN. The VCN VR 105 is configured to route traffic between VNICs in the VCN 104 and gateways associated with the VCN 104. The VCN VR 105 provides a port for each subnet of the VCN 104. For example, the VR 105 can provide a port for subnet-1 with IP address 10.0.0.1 and a port for subnet-2 with IP address 10.1.0.1.

[0084] Multiple compute instances can be deployed on each subnet, where the compute instances can be virtual machine instances and / or bare metal instances. The compute instances in a subnet can be hosted by one or more host machines within the CSPI 101. The compute instances participate in the subnet via VNICs associated with the compute instances. For example, as shown in Figure 1 In this example, the compute instance CI is part of subnet-1 via a VNIC associated with the compute instance. Likewise, the compute instance C2 is part of subnet-1 via a VNIC associated with C2. In a similar manner, multiple compute instances (which can be virtual machine instances or bare metal instances) can be part of subnet-1. Via their associated VNICs, each compute instance is assigned a private overlay IP address and a MAC address. For example, in Figure 1 In this example, the compute instance CI is part of subnet-1 via a VNIC associated with the compute instance. Likewise, the compute instance C2 is part of subnet-1 via a VNIC associated with C2. In a similar manner, multiple compute instances (which can be virtual machine instances or bare metal instances) can be part of subnet-1. Via their associated VNICs, each compute instance is assigned a private overlay IP address and a MAC address. For example, in

[0085] Multiple compute instances can be deployed on each subnet, where the compute instances can be virtual machine instances and / or bare metal instances. The compute instances in a subnet can be hosted by one or more host machines within the CSPI 101. The compute instances participate in the subnet via VNICs associated with the compute instances. For example, as shown in Figure 1 In this example, the compute instance CI is part of subnet-1 via a VNIC associated with the compute instance. Likewise, the compute instance C2 is part of subnet-1 via a VNIC associated with C2. In a similar manner, multiple compute instances (which can be virtual machine instances or bare metal instances) can be part of subnet-1. Via their associated VNICs, each compute instance is assigned a private overlay IP address and a MAC address. For example, in Figure 1 In the example shown in Figure 1 In the example shown in Figure 1 In the example shown in

[0086] VCN A 104 can also include one or more load balancers. For example, a load balancer can be provisioned for a subnet and can be configured to load balance traffic across multiple compute instances on the subnet. Load balancers can also be provisioned to load balance traffic across subnets in a VCN.

[0087] A particular compute instance deployed on VCN 104 can communicate with a variety of different endpoints. These endpoints can include endpoints hosted by CSPI 200 and endpoints outside of CSPI 200. Endpoints hosted by CSPI 101 can include: endpoints on the same subnet as the particular compute instance (e.g., communication between two compute instances in Subnet-1); endpoints on a different subnet but within the same VCN (e.g., communication between a compute instance in Subnet-1 and a compute instance in Subnet-2); endpoints in a different VCN in the same region (e.g., communication between a compute instance in Subnet-1 and an endpoint in a VCN in the same region 106 or 110, communication between a compute instance in Subnet-1 and an endpoint in Service Gateway 110 in the same region); or endpoints in a VCN in a different region (e.g., communication between a compute instance in Subnet-1 and an endpoint in a VCN in a different region 108). Compute instances in subnets hosted by CSPI 101 can also communicate with endpoints that are not hosted by CSPI 101 (i.e., outside of CSPI 101). These external endpoints include endpoints in the customer’s on-premise network 116, endpoints in other remote cloud-hosted networks 118, public endpoints 114 accessible via a public network such as the Internet, and other endpoints.

[0088] Communication between compute instances on the same subnet is facilitated using the VNICs associated with the source and destination compute instances. For example, compute instance Cl in subnet-1 can want to send a packet to compute instance C2 in subnet-1. For packets originating from a source compute instance and destined for another compute instance in the same subnet, the packet is first processed by the VNIC associated with the source compute instance. The processing performed by the VNIC associated with the source compute instance can include determining the destination information for the packet from the packet header, identifying any policies (e.g., security lists) configured for the VNIC associated with the source compute instance, determining the next hop for the packet, performing any packet encapsulation / decapsulation functions as needed, and then forwarding / routing the packet to the next hop with the goal of facilitating communication of the packet to its intended destination. When the destination compute instance is in the same subnet as the source compute instance, the VNIC associated with the source compute instance is configured to identify the VNIC associated with the destination compute instance and forward the packet to that VNIC for processing. The VNIC associated with the destination compute instance is then executed and forwards the packet to the destination compute instance.

[0089] For packets that are to be communicated from a compute instance in a subnet to an endpoint in a different subnet in the same VCN, communication is facilitated by the VNICs associated with the source and destination compute instances and the VCN VR. For example, if Figure 1 Compute instance Cl in subnet-1 wants to send a packet to compute instance Dl in subnet-2, then the packet is first processed by the VNIC associated with compute instance Cl. The VNIC associated with compute instance Cl is configured to route the packet to the VCN VR 105 using the default route or port 10.0.0.1 of the VCN VR. The VCN VR 105 is configured to route the packet to subnet-2 using port 10.1.0.1. The VNIC associated with Dl then receives and processes the packet and the VNIC forwards the packet to compute instance Dl.

[0090] For packets that are to be communicated from a compute instance in VCN 104 to an endpoint outside of VCN 104, communication is facilitated by the VNIC associated with the source compute instance, the VCN VR 105, and a gateway associated with VCN 104. One or more types of gateways can be associated with a VCN. A gateway is an interface between a VCN and another endpoint where the other endpoint is outside of the VCN. Gateways are Layer 3 / IP layer concepts and enable a VCN to communicate with endpoints outside of the VCN. Thus, gateways facilitate traffic flow between a VCN and other VCNs or networks. Various different types of gateways can be configured for a VCN to facilitate different types of communication with different types of endpoints. Depending on the gateway, the communication can be over a public network (e.g., the Internet) or over a private network. Various communication protocols can be used for these communications.

[0091] For example, compute instance CI can want to communicate with an endpoint outside of VCN 104. The packet can first be processed by the VNIC associated with source compute instance CI. The VNIC processing determines that the destination of the packet is outside of subnet-1 for CI. The VNIC associated with CI can forward the packet to VCN VR 105 for VCN 104. VCN VR 105 then processes the packet and, as part of the processing, determines, based on the destination of the packet, a particular gateway associated with VCN 104 as the next hop for this packet. VCN VR 105 can then forward the packet to the identified particular gateway. For example, if the destination is an endpoint within a customer's on-premise network, then the packet can be forwarded by VCN VR 105 to a dynamic routing gateway (DRG) gateway 122 configured for VCN 104. The packet can then be forwarded from the gateway to the next hop to facilitate the transfer of the packet to its ultimate intended destination.

[0092] Various different types of gateways can be configured for a VCN. Examples of gateways that can be configured for a VCN are depicted in Figure 1 and described below. Examples of gateways associated with a VCN are also depicted in Figure 18 , Figure 19 , Figure 20 and FIG. 21 (e.g., the gateways referenced by reference numbers 1834, 1836, 1838, 1934, 1936, 1938, 2034, 2036, 2038, 2134, 2136, and 2138) and described below. As Figure 1As shown in the depicted embodiment, a dynamic routing gateway (DRG) 122 can be added to or associated with a customer VCN 104 and provide a path for private network traffic communication between the customer VCN 104 and another endpoint, which can be a customer on-premise network 116, a VCN 108 in a different region of CSPI 101, or other remote cloud network 118 not hosted by CSPI 101. A customer on-premise network 116 can be a customer network built using customer’s resources or a customer data center. Access to a customer on-premise network 116 is generally very restricted. For a customer who has both a customer on-premise network 116 and one or more VCNs 104 deployed or hosted in the cloud by CSPI 101, the customer can want their on-premise network 116 and their cloud-based VCN 104 to be able to communicate with each other. This enables the customer to build an extended hybrid environment that covers both the customer’s VCN 104 hosted by CSPI 101 and their on-premise network 116. The DRG 122 enables such communication. To enable such communication, a communication channel 124 is set up, with one endpoint of the channel in the customer on-premise network 116 and the other endpoint in CSPI 101 and connected to the customer VCN 104. The communication channel 124 can be over a public communication network, such as the Internet, or a private communication network. Various different communication protocols can be used, such as IPsec VPN technology over a public communication network such as the Internet, Oracle’s FastConnect technology that uses a private network instead of a public network, etc. The device or equipment in the customer on-premise network 116 that forms one endpoint of the communication channel 124 is referred to as customer premises equipment (CPE), such as Figure 1 CPE 126 depicted in the figure. On the CSPI 101 side, the endpoint can be a host machine that executes the DRG 122.

[0093] In certain embodiments, a remote peering connection (RPC) can be added to the DRG, which allows a customer to peer one VCN with another VCN in a different region. Using such RPC, a customer VCN 104 can connect with a VCN 108 in another region using the DRG 122. The DRG 122 can also be used to communicate with other remote cloud networks 118 not hosted by CSPI 101, such as Microsoft Azure cloud, Amazon AWS cloud, etc.

[0094] As Figure 1As shown in the middle, an Internet Gateway (IGW) 120 can be configured for the customer VCN 104, which enables compute instances on the VCN 104 to communicate with public endpoints 114 that are accessible through a public network such as the Internet. An IGW 120 is a gateway that connects a VCN to a public network such as the Internet. The IGW 120 enables public subnets (where resources in the public subnets have public overlay IP addresses) within a VCN (such as VCN 104) to directly access public endpoints 112 on the public network 114 (such as the Internet). Using the IGW 120, connections can be initiated from subnets within the VCN 104 or from the Internet.

[0095] A Network Address Translation (NAT) gateway 128 can be configured for the customer's VCN 104 and enables cloud resources in the customer's VCN that do not have a dedicated public overlay IP address to access the Internet, and it does so without exposing those resources to direct incoming Internet connections (e.g., L4-L7 connections). This enables private subnets within a VCN (such as Private Subnet-1 in VCN 104) to privately access public endpoints on the Internet. In a NAT gateway, connections to the public Internet can only be initiated from the private subnets, not from the Internet.

[0096] In certain embodiments, a Service Gateway (SGW) 126 can be configured for the customer VCN 104 and provides a path for private network traffic between the VCN 104 and service endpoints supported in the service network 110. In certain embodiments, the service network 110 can be provided by the CSP and can provide various services. An example of such a service network is Oracle's service network, which provides various services available for use by customers. For example, a compute instance (e.g., a database system) in a private subnet of the customer VCN 104 can back up data to a service endpoint (e.g., object storage) without needing a public IP address or access to the Internet. In certain embodiments, a VCN can have only one SGW, and connections can only be initiated from subnets within the VCN, not from the service network 110. If the VCN is peered with another, resources in the other VCN typically cannot access the SGW. Resources in a pre-existing network connected to the VCN with FastConnect or VPN Connect can also use the service gateway configured for that VCN.

[0097] In certain embodiments, the SGW 126 uses the concept of a service Classless Inter Domain Routing (CIDR) tag, which is a string that represents all the regional public IP address ranges for a service or group of services of interest. Customers use the service CIDR tag when they configure the SGW and related routing rules to control traffic to the service. Customers can optionally use it when configuring security rules without needing to adjust the security rules if the public IP addresses for the service change in the future.

[0098] A local peering gateway (LPG) 132 is a gateway that can be added to a customer VCN 104 and enables the VCN 104 to peer with another VCN in the same region. Peering means that the VCNs communicate using private IP addresses without the traffic traversing a public network (such as the Internet) or without the traffic being routed through the customer's on-premise network 116. In the preferred embodiment, a VCN has a separate LPG for each peering it establishes. Local peering or VCN peering is a common practice for establishing network connectivity between different applications or infrastructure management functions.

[0099] Service providers, such as the provider of a service in the service network 110, can provide access to the service using different access models. According to a public access model, a service can be exposed as a public endpoint that is publicly accessible by compute instances in a customer VCN via a public network, such as the Internet, and / or can be accessed privately via the SGW 126. According to a specific private access model, a service can be accessed as a private IP endpoint in a private subnet in a customer's VCN. This is referred to as private endpoint (PE) access and enables a service provider to expose its service as an instance in a customer's private network. A private endpoint resource represents a service within a customer's VCN. Each PE appears as a VNIC (referred to as a PE-VNIC, with one or more private IPs) in a subnet in the customer's VCN, selected by the customer. Thus, a PE provides a way to present a service in a private customer VCN subnet using a VNIC. Since the endpoint is exposed as a VNIC, all the features associated with a VNIC (such as routing rules, security lists, etc.) are now available to the PE VNIC.

[0100] Service providers can register their services to enable access through a PE. A provider can associate policies with a service that limit the visibility of the service to customer tenancies. A provider can register multiple services under a single virtual IP address (VIP), especially for multi-tenancy services. There can be multiple such private endpoints (in multiple VCNs) representing the same service.

[0101] Compute instances in the private subnet can then access the service using the private IP address or service DNS name of the PE VNIC. Compute instances in the customer VCN can access the service by sending traffic to the private IP address of the PE in the customer VCN. The Private Access Gateway (PAGW) 130 is a gateway resource that can be attached to a service provider VCN (e.g., a VCN in the service network 110) that acts as an ingress / egress point for all traffic from / to the customer subnet private endpoints. The PAGW 130 enables a provider to scale the number of PE connections without utilizing its internal IP address resources. The provider only needs to configure one PAGW for any number of services registered in a single VCN. The provider can represent a service as a private endpoint in multiple VCNs of one or more customers. From the customer’s perspective, the PE VNIC is not attached to the customer’s instance, but appears to be attached to the service that the customer wants to interact with. Traffic to the private endpoint is routed to the service via the PAGW 130. These are referred to as customer-to-service private connections (C2S connections).

[0102] By allowing traffic to flow through the FastConnect / IPsec link and the private endpoint in the customer VCN, the PE concept can also be used to extend private access to a service to a customer’s on-premise network and data center. By allowing traffic to flow between the LPG 132 and the PE in the customer’s VCN, private access to a service can also be extended to a customer’s peered VCN.

[0103] Customers can control routing in a VCN at the subnet level, so customers can specify which subnets in a customer’s VCN (such as the VCN 104) use each gateway. The routing table of a VCN is used to decide whether to allow traffic to leave the VCN through a particular gateway. For example, in a particular instance, the routing table for public subnets within the customer VCN 104 can send non-local traffic through the IGW 120. The routing table for private subnets within the same customer VCN 104 can send traffic destined for CSP services through the SGW 126. All remaining traffic can be sent via the NAT gateway 128. The routing table only controls traffic leaving the VCN.

[0104] Security lists associated with a VCN are used to control traffic entering the VCN via gateways via inbound connections. All resources in a subnet use the same route table and security list. Security lists can be used to control which specific types of traffic are allowed to and from instances in a subnet of a VCN. Security list rules can include ingress (inbound) and egress (outbound) rules. For example, ingress rules can specify allowed source address ranges, while egress rules can specify allowed destination address ranges. Security rules can specify specific protocols (e.g., TCP, ICMP), specific ports (e.g., 22 for SSH, 3389 for Windows RDP), etc. In certain embodiments, an instance's operating system can enforce its own firewall rules that comply with the security list rules. Rules can be stateful (e.g., track connections and automatically allow responses without explicit security list rules for response traffic) or stateless.

[0105] Access from a customer VCN (i.e., by resources or compute instances deployed on the VCN 104) can be categorized as public access, private access, or dedicated access. Public access refers to an access model that uses public IP addresses or NAT to access public endpoints. Private access enables customer workloads in the VCN 104 (e.g., resources in a private subnet) with private IP addresses to access services without traversing a public network such as the Internet. In certain embodiments, CSPI 101 enables customer VCN workloads with private IP addresses to access services (of public service endpoints) using a service gateway. Thus, the service gateway provides a private access model by establishing a virtual link between the customer's VCN and the public endpoints of services that reside outside the customer's private network.

[0106] Further, CSPI can provide dedicated public access using techniques such as FastConnect public peering, where customer on-premise instances can access one or more services in the customer's VCN using FastConnect connections without traversing a public network such as the Internet. CSPI can also provide dedicated private access using FastConnect private peering, where customer on-premise instances with private IP addresses can access the customer's VCN workloads using FastConnect connections. FastConnect is a network connectivity alternative to using the public Internet to connect a customer's on-premise network to CSPI and its services. FastConnect provides a simple, flexible, and economical way to create dedicated and private connections with higher bandwidth options and a more reliable and consistent network experience compared to Internet-based connections.

[0107] Figure 1The accompanying description above describes various virtualized components in an example virtual network. As described above, the virtual network is built on an underlying physical or substrate network. Figure 2 A simplified architectural diagram of physical components in a physical network within CSPI 200 providing an underlay for virtual networks is depicted in accordance with certain embodiments. As shown, CSPI 200 provides a distributed environment including components and resources (e.g., compute, memory, and network resources) provided by a cloud service provider (CSP). These components and resources are used to provide cloud services (e.g., IaaS services) to subscribing customers (i.e., customers that have subscribed to one or more services provided by the CSP). Based on the services to which a customer subscribes, a subset of the resources of CSPI 200 (e.g., compute, memory, and network resources) are provisioned for the customer. The customer can then build its own customizable and private virtual network based on the cloud (i.e., CSPI-hosted) using the physical compute, memory, and networking resources provided by CSPI 200. As indicated earlier, these customer networks are referred to as virtual cloud networks (VCNs). The customer can deploy one or more customer resources, such as compute instances, on these customer VCNs. The compute instances can be in the form of virtual machines, bare-metal instances, and the like. CSPI 200 provides the infrastructure and a complementary set of cloud services that enable customers to build and run a wide range of applications and services in a highly available hosted environment.

[0108] In Figure 2 example embodiments depicted in FIG. 1, the physical components of CSPI 200 include one or more physical host machines or physical servers (e.g., 202, 206, 208), network virtualization devices (NVDs) (e.g., 210, 212), top-of-rack (TOR) switches (e.g., 214, 216), and a physical network (e.g., 218), as well as switches in the physical network 218. The physical host machines or servers can host and execute various compute instances that participate in one or more subnets of a VCN. The compute instances can include virtual machine instances and bare-metal instances. For example, Figure 1 The various compute instances depicted in FIG. 1 can be hosted by Figure 2 physical host machines depicted in FIG. 1. Virtual machine compute instances in a VCN can be executed by one host machine or multiple different host machines. The physical host machines can also host virtual host machines, container-based hosts, or functions, and the like. Figure 1 The VNICs and VCN VRs depicted in FIG. 1 can be executed by Figure 2 NVDs depicted in FIG. 1. Figure 1 The gateways depicted in FIG. 1 can be executed by host machines and / or by Figure 2 NVDs described in FIG. 1.

[0109] A host machine or server can execute a hypervisor (also referred to as a virtual machine monitor or VMM) that creates and enables virtualized environments on the host machine. Virtualization or virtualized environments facilitate cloud-based computing. One or more compute instances can be created, executed, and managed by the hypervisor on the host machine. The hypervisor on the host machine enables the physical computing resources (e.g., compute, memory, and network resources) of the host machine to be shared among various compute instances executed by the host machine.

[0110] For example, as depicted in Figure 2 , host machines 202 and 208 execute hypervisors 260 and 266, respectively. These hypervisors can be implemented using software, firmware, or hardware, or a combination thereof. Generally, a hypervisor is a process or software layer that sits on top of an operating system (OS) of a host machine, which in turn executes on a hardware processor of the host machine. The hypervisor provides a virtualized environment by enabling the physical computing resources (e.g., processing resources such as processors / cores, memory resources, network resources) of the host machine to be shared among various virtual machine compute instances executed by the host machine. For example, in Figure 2 , hypervisor 260 can sit on top of the OS of host machine 202 and enable the compute resources (e.g., processing, memory, and network resources) of host machine 202 to be shared among compute instances (e.g., virtual machines) executed by host machine 202. A virtual machine can have its own operating system (referred to as a guest operating system), which can be the same as or different from the OS of the host machine. The operating system of a virtual machine executed by a host machine can be the same as or different from the operating system of another virtual machine executed by the same host machine. Thus, the hypervisor enables multiple operating systems to be executed simultaneously while sharing the same compute resources of the host machine. Figure 2 The host machines depicted in

[0111] A compute instance can be a virtual machine instance or a bare metal instance. In Figure 2 , compute instance 268 on host machine 202 and compute instance 274 on host machine 208 are examples of virtual machine instances. Host machine 206 is an example of a bare metal instance provided to a customer.

[0112] In some cases, an entire host machine can be provisioned to a single customer, and one or more compute instances (either virtual machines or bare metal instances) hosted by the host machine all belong to the same customer. In other cases, a host machine can be shared among multiple customers (i.e., multiple tenants). In such multi-tenancy scenarios, a host machine can host virtual machine compute instances belonging to different customers. These compute instances can be members of different VCNs of different customers. In certain embodiments, a bare metal compute instance is hosted by a bare metal server without a hypervisor. When a bare metal compute instance is provisioned, a single customer or tenant maintains control over the physical CPU, memory, and network interfaces of the host machine that hosts the bare metal instance, and the host machine is not shared with other customers or tenants.

[0113] As previously described, each compute instance that is part of a VCN is associated with a VNIC that enables the compute instance to be a member of a subnet of the VCN. The VNIC associated with a compute instance facilitates the communication of packets or frames to and from the compute instance. A VNIC is associated with a compute instance when the compute instance is created. In certain embodiments, for a compute instance executed by a host machine, the VNIC associated with the compute instance is executed by an NVD connected to the host machine. For example, in Figure 2 embodiment depicted in FIG. 2, host machine 202 executes virtual machine compute instance 268 associated with VNIC 276, and VNIC 276 is executed by NVD 210 connected to host machine 202. As another example, bare metal instance 272 hosted by host machine 206 is associated with VNIC 280 executed by NVD 212 connected to host machine 206. As yet another example, VNIC 284 is associated with compute instance 274 executed by host machine 208, and VNIC 284 is executed by NVD 212 connected to host machine 208.

[0114] For a compute instance hosted by a host machine, the NVD connected to the host machine also executes a VCN VR corresponding to the VCN of which the compute instance is a member. For example, in the embodiment depicted in Figure 2 FIG. 2, NVD 210 executes VCN VR 277 corresponding to the VCN of which compute instance 268 is a member. NVD 212 can also execute one or more VCN VRs 283 corresponding to the VCNs corresponding to the compute instances hosted by host machines 206 and 208.

[0115] The host machines can include one or more network interface cards (NICs) that enable the host machines to connect to other devices. The NICs on the host machines can provide one or more ports (or interfaces) that enable the host machines to communicatively connect to another device. For example, the host machines can connect to the NVDs using one or more ports (or interfaces) provided on the host machines and on the NVDs. The host machines can also connect to other devices, such as another host machine.

[0116] For example, in Figure 2 , the host machine 202 connects to the NVD 210 using a link 220 that extends between a port 234 provided by a NIC 232 of the host machine 202 and a port 236 of the NVD 210. The host machine 206 connects to the NVD 212 using a link 224 that extends between a port 246 provided by a NIC 244 of the host machine 206 and a port 248 of the NVD 212. The host machine 208 connects to the NVD 212 using a link 226 that extends between a port 252 provided by a NIC 250 of the host machine 208 and a port 254 of the NVD 212.

[0117] The NVDs, in turn, connect to top-of-rack (TOR) switches via communication links, which connect to a physical network 218 (also referred to as a switching fabric). In certain embodiments, the links between the host machines and the NVDs and between the NVDs and the TOR switches are Ethernet links. For example, in Figure 2 , the NVDs 210 and 212 connect to the TOR switches 214 and 216 using links 228 and 230, respectively. In certain embodiments, the links 220, 224, 226, 228, and 230 are Ethernet links. The collection of host machines and NVDs that connect to the TORs is sometimes referred to as a rack.

[0118] The physical network 218 provides a communication fabric that enables the TOR switches to communicate with each other. The physical network 218 can be a multi-tier network. In certain implementations, the physical network 218 is a multi-tier Clos network of switches, where the TOR switches 214 and 216 represent leaf-level nodes of the multi-tier and multi-node physical switching network 218. Different Clos network configurations are possible, including but not limited to 2-tier networks, 3-tier networks, 4-tier networks, 5-tier networks, and generally “n”-tier networks. An example of a Clos network is depicted in Figure 5 and described below.

[0119] Various different connection configurations are possible between the host machines and the NVDs, such as one-to-one configurations, many-to-one configurations, one-to-many configurations, and the like. In one-to-one configuration implementations, each host machine connects to its own separate NVD. For example, in Figure 2 In a one-to-one configuration, one host machine is connected to one NVD. For example, in Figure 2 In a one-to-many configuration, one host machine is connected to multiple NVDs.

[0120] In a one-to-many configuration, one host machine is connected to multiple NVDs. Figure 3 An example is shown within CSPI 300 in which a host machine is connected to multiple NVDs. As shown in Figure 3 As shown in the example of FIG. 3, host machine 302 includes a network interface card (NIC) 304, which includes multiple ports 306 and 308. Host machine 300 is connected to a first NVD 310 via port 306 and link 320, and to a second NVD 312 via port 308 and link 322. Ports 306 and 308 can be Ethernet ports, and links 320 and 322 between host machine 302 and NVDs 310 and 312 can be Ethernet links. NVD 310 is in turn connected to a first TOR switch 314 and NVD 312 is connected to a second TOR switch 316. Links between NVDs 310, 312 and TOR switches 314, 316 can be Ethernet links. TOR switches 314 and 316 represent layer 0 switching devices in a multi-layer physical network 318.

[0121] Figure 3 The arrangement depicted in FIG. 3 provides two separate physical network paths from physical switch network 318 to host machine 302: a first path through TOR switch 314 to NVD 310 and to host machine 302, and a second path through TOR switch 316 to NVD 312 and to host machine 302. The separate paths provide enhanced availability (referred to as high availability) of host machine 302. If one of the paths (e.g., a link in one of the paths breaks) or devices (e.g., a particular NVD is not functioning) has a problem, then the other path can be used for communications to / from host machine 302.

[0122] In the configuration depicted in FIG. 3, the host machine is connected to two different NVDs using two different ports provided by the NIC of the host machine. In other embodiments, a host machine can include multiple NICs that enable the host machine to connect to multiple NVDs. Figure 3

[0123] Returning to the one-to-one configuration of FIG. 2, host machine 202 is connected to NVD 210 via NIC 232 of host machine 202. In the one-to-many configuration of FIG. 3, host machines 206 and 208 are connected to NVD 212 via NICs 244 and 250, respectively. Figure 2 ​NVDs are physical devices or components that perform one or more network and / or storage virtualization functions. An NVD can be any device with one or more processing units (e.g., CPUs, network processing units (NPUs), FPGAs, packet processing pipelines, etc.), memory including caches, and ports. Various virtualization functions can be performed by software / firmware executed by one or more processing units of the NVD.

[0124] NVDs can be implemented in a variety of different forms. For example, in certain embodiments, NVDs are implemented as interface cards or smartNICs with on-board embedded processors, which are referred to as smartNICs. A smartNIC is a device that is separate from the NICs on the host machines. In Figure 2 In particular embodiments, NVDs 210 and 212 can be implemented as smartNICs that are connected to host machines 202 and 206 and 208, respectively.

[0125] However, smartNICs are just one example of an NVD implementation. Various other implementations are possible. For example, in some other implementations, NVDs or one or more functions performed by NVDs can be incorporated into or performed by one or more host machines, one or more TOR switches, and other components of CSPI 200. For example, an NVD can be implemented in a host machine, where the functions performed by the NVD are performed by the host machine. As another example, an NVD can be part of a TOR switch, or a TOR switch can be configured to perform the functions performed by an NVD, which enables the TOR switch to perform various complex packet transformations for the public cloud. A TOR that performs the functions of an NVD is sometimes referred to as a smartTOR. In other implementations where virtual machine (VM) instances are provided to customers instead of bare metal (BM) instances, the functions performed by an NVD can be implemented inside the hypervisor of a host machine. In some other implementations, some of the functions of an NVD can be offloaded to a centralized service running on a set of host machines.

[0126] In certain embodiments, such as when implemented as a smartNIC as shown in Figure 2 In particular embodiments, an NVD can include a plurality of physical ports that enable it to be connected to one or more host machines and one or more TOR switches. The ports on an NVD can be classified as host-facing ports (also referred to as “southbound ports”) or network-facing or TOR-facing ports (also referred to as “northbound ports”). The host-facing ports of an NVD are the ports used to connect the NVD to host machines. Figure 2 Examples of host-facing ports include ports 236 on NVD 210 and ports 248 and 254 on NVD 212. The network-facing ports of an NVD are the ports used to connect the NVD to TOR switches. Figure 2 Examples of network-facing ports include port 256 on NVD 210 and port 258 on NVD 212. As shown in FIG. 2, NVD 210 is connected to TOR switch 214 using link 228 that extends from port 256 of NVD 210 to TOR switch 214. Similarly, NVD 212 is connected to TOR switch 216 using link 230 that extends from port 258 of NVD 212 to TOR switch 216. Figure 2

[0127] NVDs receive packets and frames from host machines via host-facing ports (e.g., packets and frames generated by compute instances hosted by the host machines) and, after performing necessary packet processing, can forward the packets and frames to TOR switches via the network-facing ports of the NVDs. NVDs can receive packets and frames from TOR switches via the network-facing ports of the NVDs and, after performing necessary packet processing, can forward the packets and frames to host machines via the host-facing ports of the NVDs.

[0128] In certain embodiments, there can be multiple ports and associated links between a NVD and a TOR switch. These ports and links can be aggregated to form a link aggregation group (referred to as a LAG) of multiple ports or links. Link aggregation allows multiple physical links between two endpoints (e.g., between a NVD and a TOR switch) to be treated as a single logical link. All physical links in a given LAG can operate in full-duplex mode at the same speed. LAGs help increase the bandwidth and reliability of the connection between two endpoints. If one of the physical links in a LAG fails, then traffic is dynamically and transparently reassigned to one of the other physical links in the LAG. Aggregated physical links deliver higher bandwidth than each individual link. Multiple ports associated with a LAG are treated as a single logical port. Traffic can be load balanced across the multiple physical links of a LAG. One or more LAGs can be configured between two endpoints. The two endpoints can be between a NVD and a TOR switch, between a host machine and a NVD, and so on.

[0129] ​NVDs implement or perform network virtualization functions. These functions are performed by software / firmware executed by the NVDs. Examples of network virtualization functions include, but are not limited to: packet encapsulation and decapsulation functions; functions to create VCN networks; functions to implement network policies, such as VCN security list (firewall) functionality; functions to facilitate routing and forwarding of packets to and from compute instances in a VCN; and so on. In certain embodiments, upon receiving a packet, the NVD is configured to execute a packet processing pipeline for processing the packet and determining how to forward or route the packet. As part of this packet processing pipeline, the NVD can perform one or more virtual functions associated with an overlay network, such as executing a VNIC associated with a compute instance in a VCN, executing a virtual router (VR) associated with a VCN, encapsulation and decapsulation of packets to facilitate forwarding or routing in a virtual network, execution of certain gateways (e.g., local peering gateway), implementation of security lists, network security groups, network address translation (NAT) functionality (e.g., translating public IPs to private IPs on a per-host basis), throttling functionality, and other functions.

[0130] In certain embodiments, the packet processing datapath in an NVD can include multiple packet pipelines, each composed of a series of packet transformation stages. In certain implementations, upon receiving a packet, the packet is parsed and classified into a single pipeline. The packet is then processed in a linear fashion, one stage after another, until the packet is either dropped or sent out an interface of the NVD. These stages provide basic functional packet processing building blocks (e.g., validate headers, enforce throttling, insert new layer 2 headers, enforce L4 firewall, VCN encapsulation / decapsulation, etc.) so that new pipelines can be constructed by combining existing stages, and new functionality can be added by creating new stages and inserting them into existing pipelines.

[0131] An NVD can perform both control plane and data plane functions corresponding to the control plane and data plane of a VCN. Examples of a VCN control plane are also depicted in Figure 18 、 Figure 19 、 Figure 20 and FIG. 21 (see reference numerals 1816, 1916, 2016, and 2116) and described below. Examples of a VCN data plane are depicted in Figure 18 、 Figure 19 、 Figure 20and described below. Control plane functions include functions to configure the network on how control data is forwarded (e.g., set up routes and routing tables, configure VNICs, etc.). In certain embodiments, a VCN control plane is provided that computes all overlay-to-substrate mappings and publishes them to the NVDs and virtual network edge devices (such as various gateways, such as DRGs, SGWs, IGWs, etc.). Firewall rules can also be published using the same mechanism. In certain embodiments, the NVDs only get the mappings that are relevant to that NVD. Data plane functions include functions to actually route / forward packets based on the configuration set up using the control plane. The VCN data plane is implemented by encapsulating the customer's network packets before they traverse the substrate network. The encapsulation / decapsulation functionality is implemented on the NVDs. In certain embodiments, the NVDs are configured to intercept all network packets going to and from the host machines and perform the network virtualization functions.

[0132] As indicated above, the NVDs perform various virtualization functions, including VNICs and VCN VRs. The NVDs can perform VNICs associated with compute instances hosted by one or more host machines connected to the VNICs. For example, as depicted in Figure 2 FIG. 21, NVD 210 performs the functionality of VNIC 276 associated with compute instance 268 hosted by host machine 202 connected to NVD 210. As another example, NVD 212 performs VNIC 280 associated with bare-metal compute instance 272 hosted by host machine 206 and performs VNIC 284 associated with compute instance 274 hosted by host machine 208. The host machines can host compute instances belonging to different VCNs (that belong to different customers), and the NVDs connected to the host machines can perform the VNICs corresponding to the compute instances (i.e., perform the VNIC-related functionality).

[0133] The NVDs also perform VCN virtual routers corresponding to the VCNs of the compute instances. For example, in the embodiment depicted in Figure 2 FIG. 21, NVD 210 performs VCN VR 277 corresponding to the VCN to which compute instance 268 belongs. NVD 212 performs one or more VCN VRs 283 corresponding to one or more VCNs to which the compute instances hosted by host machines 206 and 208 belong. In certain embodiments, the VCN VR corresponding to a VCN is performed by all NVDs connected to the host machines that host at least one compute instance belonging to that VCN. If the host machines host compute instances belonging to different VCNs, then the NVDs connected to that host machine can perform VCN VRs corresponding to those different VCNs.

[0134] In addition to VNICs and VCN VRs, NVDs can also execute various software (e.g., daemons) and include one or more hardware components that facilitate various network virtualization functions performed by the NVDs. For simplicity, these various components are grouped together as a “packet processing component” as shown in Figure 2 For example, NVD 210 includes packet processing component 286 and NVD 212 includes packet processing component 288. For example, a packet processing component for an NVD can include a packet processor configured to interact with the ports and hardware interfaces of the NVD to monitor all packets received by and transmitted using the NVD and store network information. Network information may, for example, include network flow information and per-flow information (e.g., per-flow statistics) that identifies different network flows handled by the NVD. In certain embodiments, network flow information can be stored on a per-VNIC basis. The packet processor can perform per-packet manipulations and implement stateful NAT and L4 firewall (FW). As another example, a packet processing component can include a replication agent configured to replicate information stored by the NVD to one or more different replication target repositories. As yet another example, a packet processing component can include a logging agent configured to perform logging functions for the NVD. A packet processing component can also include software for monitoring the performance and health of the NVD and possibly also the status and health of other components connected to the NVD.

[0135] Figure 1 Components of an example virtual or overlay network are shown, including a VCN, a subnet within the VCN, a compute instance deployed on the subnet, a VNIC associated with the compute instance, a VR for the VCN, and a set of gateways configured for the VCN. Figure 1 The overlay components depicted in Figure 2 may be executed or hosted by one or more of the physical components depicted in Figure 2 For example, a compute instance in a VCN can be executed or hosted by one or more host machines depicted in For compute instances hosted by a host machine, a VNIC associated with that compute instance is typically executed by an NVD connected to that host machine (i.e., VNIC functionality is provided by an NVD connected to that host machine). VCN VR functionality for a VCN is executed by all NVDs connected to the host machines that host or execute compute instances that are part of that VCN. Gateways associated with a VCN can be executed by one or more different types of NVDs. For example, certain gateways can be executed by a smartNIC, while other gateways can be executed by one or more host machines or other implementations of NVDs.

[0136] As described above, a compute instance in a customer VCN can communicate with various different endpoints, where the endpoints can be within the same subnet as the source compute instance, in a different subnet but within the same VCN as the source compute instance, or the endpoints are outside the VCN of the source compute instance. These communications are facilitated using the VNICs associated with the compute instances, the VCN VRs, and gateways associated with the VCNs.

[0137] For communication between two compute instances on the same subnet in a VCN, the communication is facilitated using the VNICs associated with the source and destination compute instances. The source and destination compute instances can be hosted by the same host machine or different host machines. A packet originating from the source compute instance can be forwarded from the host machine hosting the source compute instance to the NVD connected to that host machine. On the NVD, the packet is processed using a packet processing pipeline, which can include execution of the VNIC associated with the source compute instance. Since the destination endpoint of the packet is within the same subnet, execution of the VNIC associated with the source compute instance results in the packet being forwarded to the NVD executing the VNIC associated with the destination compute instance, which then processes the packet and forwards it to the destination compute instance. The VNICs associated with the source and destination compute instances can be executed on the same NVD (e.g., when the source and destination compute instances are hosted by the same host machine) or on different NVDs (e.g., when the source and destination compute instances are hosted by different host machines connected to different NVDs). The VNICs can use the routing / forwarding table stored by the NVD to determine the next hop for the packet.

[0138] For a packet to be communicated from a compute instance in a subnet to an endpoint in a different subnet in the same VCN, the packet originating from the source compute instance is communicated from the host machine hosting the source compute instance to the NVD connected to that host machine. On the NVD, the packet is processed using a packet processing pipeline, which can include execution of one or more VNICs and a VR associated with the VCN. For example, as part of the packet processing pipeline, the NVD executes or invokes functionality corresponding to the VNIC associated with the source compute instance (also referred to as executing the VNIC). The functionality executed by the VNIC can include looking at the VLAN tag on the packet. Since the destination of the packet is outside the subnet, the VCN VR functionality is next invoked and executed by the NVD. The VCN VR then routes the packet to the NVD executing the VNIC associated with the destination compute instance. The VNIC associated with the destination compute instance then processes the packet and forwards it to the destination compute instance. The VNICs associated with the source and destination compute instances can be executed on the same NVD (e.g., when the source and destination compute instances are hosted by the same host machine) or on different NVDs (e.g., when the source and destination compute instances are hosted by different host machines connected to different NVDs).

[0139] If the destination of the packet is outside the VCN of the source compute instance, the packet originating from the source compute instance is transmitted from the host machine hosting the source compute instance to an NVD connected to the host machine. The NVD executes the VNIC associated with the source compute instance. Since the destination endpoint of the packet is outside the VCN, the packet is then processed by the VCN VR for the VCN. The NVD invokes the VCN VR functionality, which can result in the packet being forwarded to an NVD executing the appropriate gateway associated with the VCN. For example, if the destination is an endpoint within a customer's on-premise network, the packet can be forwarded by the VCN VR to an NVD executing the DRG gateway configured for the VCN. The VCN VR can be executed on the same NVD as the NVD executing the VNIC associated with the source compute instance, or by a different NVD. The gateway can be executed by an NVD, which can be a smartNIC, a host machine, or other NVD implementation. The packet is then processed by the gateway and forwarded to the next hop, which facilitates transmitting the packet to its intended destination endpoint. For example, in the embodiment depicted in Figure 2 , a packet originating from compute instance 268 can be transmitted from host machine 202 to NVD 210 over link 220 (using NIC 232). On NVD 210, VNIC 276 is invoked, as it is the VNIC associated with source compute instance 268. VNIC 276 is configured to inspect the information encapsulated in the packet and determine the next hop for forwarding the packet, with the goal of facilitating transmitting the packet to its intended destination endpoint, and then forward the packet to the determined next hop.

[0140] Compute instances deployed on a VCN can communicate with a variety of different endpoints. These endpoints can include endpoints hosted by CSPI 200 and endpoints outside of CSPI 200. Endpoints hosted by CSPI 200 can include instances in the same VCN or other VCNs, which can be a customer's VCN, or a VCN not belonging to a customer. Communication between endpoints hosted by CSPI 200 can be performed over physical network 218. Compute instances can also communicate with endpoints not hosted by CSPI 200 or outside of CSPI 200. Examples of these endpoints include endpoints within a customer's on-premise network or data center, or public endpoints accessible over a public network, such as the Internet. Communication with endpoints outside of CSPI 200 can be performed over a public network (e.g., the Internet) using various communication protocols (not shown in Figure 2 ) or a private network (not shown in Figure 2 ).

[0141] Figure 2The architecture of the CSPI 200 depicted herein is merely an example and is not intended to be limiting. Variations, alternatives, and modifications are possible in alternative embodiments. For example, in some implementations, the CSPI 200 may have a more advanced architecture than... Figure 2 The systems or components shown may include more or fewer systems or components, and may combine two or more systems, or may have different system configurations or arrangements. Figure 2 The systems, subsystems, and other components described herein may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective system, using hardware, or a combination thereof. The software may be stored on a non-transitory storage medium (e.g., a memory device).

[0142] Figure 4 The connectivity between the host machine and the NVD, according to certain embodiments, is described for providing I / O virtualization to support multi-tenancy. For example... Figure 4 As depicted, host machine 402 executes a hypervisor 404 that provides a virtualized environment. Host machine 402 executes two virtual machine instances, VM1 406 belonging to customer / tenant #1 and VM2 408 belonging to customer / tenant #2. Host machine 402 includes a physical NIC 410 connected to NVD 412 via link 414. Each compute instance is attached to a VNIC executed by NVD 412. Figure 4 In one embodiment, VM1 406 is attached to VNIC-VM1 420 and VM2 408 is attached to VNIC-VM2 422.

[0143] like Figure 4 As shown, NIC 410 includes two logical NICs, logical NIC A 416 and logical NIC B 418. Each virtual machine is attached to its own logical NIC and configured to work with its own logical NIC. For example, VM1 406 is attached to logical NIC A 416 and VM2 408 is attached to logical NIC B 418. Although host machine 402 includes only one physical NIC 410 shared by multiple tenants, each tenant's virtual machines believe they have their own host machine and NIC due to the logical NICs.

[0144] In certain embodiments, each logical NIC is assigned its own VLAN ID. Thus, a particular VLAN ID is assigned to logical NIC A 416 for tenant #1, and a different VLAN ID is assigned to logical NIC B 418 for tenant #2. When a packet is transmitted from VM1 406, a tag assigned to tenant #1 is appended to the packet by the hypervisor, and the packet is then transmitted from host machine 402 to NVD 412 over link 414. In a similar manner, when a packet is transmitted from VM2 408, a tag assigned to tenant #2 is appended to the packet by the hypervisor, and the packet is then transmitted from host machine 402 to NVD 412 over link 414. Thus, packets 424 transmitted from host machine 402 to NVD 412 have associated tags 426 that identify the particular tenant and associated VM. On the NVD, for packets 424 received from host machine 402, the tag 426 associated with the packet is used to determine whether the packet is to be handled by VNIC-VM1 420 or VNIC-VM2 422. The packet is then handled by the corresponding VNIC. Figure 4 The configuration described in the foregoing enables each tenant's compute instance to believe that they own their own host machine and NIC. Figure 4 The setup described in the foregoing provides I / O virtualization to support multi tenancy.

[0145] Figure 5 A simplified block diagram of a physical network 500 is depicted in accordance with certain embodiments. Figure 5 The embodiments depicted in the foregoing are structured as a Clos network. A Clos network is a particular type of network topology that is designed to provide connectivity redundancy while maintaining high bisection bandwidth and maximum resource utilization. A Clos network is a non-blocking, multi-stage or multi-tiered switching network, where the number of stages or tiers can be two, three, four, five, etc. Figure 5 The embodiments depicted in the foregoing are 3-tier networks, including tiers 1, 2, and 3. TOR switches 504 represent tier 0 switches in the Clos network. One or more NVDs are connected to the TOR switches. Tier 0 switches are also referred to as edge devices of the physical network. Tier 0 switches are connected to tier 1 switches, which are also referred to as leaf switches. In the embodiments depicted in the foregoing, each tier 1 switch is connected to two tier 0 switches. Tier 1 switches are also connected to tier 2 switches, which are also referred to as spine switches. In the embodiments depicted in the foregoing, each tier 2 switch is connected to two tier 1 switches. Tier 2 switches are also connected to tier 3 switches, which are also referred to as host machines. In the embodiments depicted in the foregoing, each tier 3 switch is connected to two tier 2 switches. Figure 5In the depicted embodiment, a set of "n" tier-0 TOR switches are connected to a set of "n" tier-1 switches and form a pod. Each tier-0 switch in a pod is interconnected to all tier-1 switches in that pod, but there is no switch connectivity between pods. In certain implementations, two pods are referred to as a block. Each block is served by or connected to a set of "n" tier-2 switches (sometimes referred to as backbone switches). There can be several blocks in the physical network topology. The tier-2 switches, in turn, are connected to "n" tier-3 switches (sometimes referred to as super backbone switches). Communication of packets on the physical network 500 is typically performed using one or more tier-3 communication protocols. Typically, all tiers of the physical network (except the TOR tier) are n-way redundant, thus allowing for high availability. Policies can be specified for pods and blocks to control the visibility of switches to each other in the physical network, thus enabling scaling of the physical network.

[0146] A feature of Clos networks is that the maximum number of hops from one tier-0 switch to another tier-0 switch (or from an NVD connected to a tier-0 switch to another NVD connected to a tier-0 switch) is fixed. For example, in a 3-tier Clos network, a packet takes at most seven hops to go from one NVD to another NVD, where the source and target NVDs are connected to the leaf tier of the Clos network. Likewise, in a 4-tier Clos network, a packet takes at most nine hops to go from one NVD to another NVD, where the source and target NVDs are connected to the leaf tier of the Clos network. Thus, Clos network architectures maintain consistent latency across the network, which is important for intra-data center and inter-data center communications. Clos topologies scale horizontally and are cost effective. The bandwidth / throughput capacity of the network can be easily increased by adding more switches at various tiers (e.g., more leaf switches and backbone switches) and by increasing the number of links between switches of adjacent tiers.

[0147] In certain embodiments, each resource within CSPI is assigned a unique identifier, referred to as a cloud identifier (CID). This identifier is included as part of the information for the resource and can be used to manage the resource, e.g., via a console or through an API. An example syntax for a CID is:

[0148] ocid1.<resource type>.<realm>.[region][.future use].<unique ID>

[0149] wherein,

[0150] ocid1: a literal string indicating the version of the CID;

[0151] resource type: the type of the resource (e.g., instance, volume, VCN, subnet, user, group, etc.);

[0152] Domain: The domain in which the resource resides. Example values are "cl" for the commercial domain, "c2" for the government cloud domain, or "c3" for the federal government cloud domain, etc. Each domain can have its own domain name;

[0153] Region: The region in which the resource resides. This section can be empty if the region does not apply to the resource;

[0154] Future Use: Reserved for future use.

[0155] Unique ID: The unique portion of the ID. The format can vary by the type of resource or service.

[0156] Multi-Cloud Introduction

[0157] Figure 6 A simplified high-level diagram depicting a distributed environment 600 comprising multiple cloud environments provided by different cloud service providers (CSPs) in accordance with certain embodiments, in which the cloud environments include a particular cloud environment that provides specialized infrastructure that enables one or more cloud services provided by that particular cloud environment to be used by customers of other cloud environments. As depicted in Figure 6 As depicted in FIG. 1, various different cloud environments (also referred to as “clouds”) can be provided by different cloud service providers (CSPs), each cloud environment or cloud offering one or more cloud services that can be subscribed to by one or more customers of that cloud environment. The set of cloud services provided by a cloud environment provided by a CSP can include one or more different types of cloud services, including but not limited to software-as-a-service (SaaS) services, infrastructure-as-a-service (IaaS) services, platform-as-a-service (PaaS) services, database-as-a-service (DBaaS) services, and the like. Examples of cloud environments provided by various CSPs include Oracle® Cloud Infrastructure (OCI) provided by Oracle Corporation, Microsoft® Azure provided by Microsoft Corporation, Google Cloud™ provided by Google LLC, Amazon Web Services (AWS®) provided by Amazon Corporation, and the like. The cloud services provided by a particular cloud environment can differ from the set of cloud services provided by another cloud environment.

[0158] In a typical cloud environment, a CSP provides a cloud service provider infrastructure (CSPI) that is used to provide one or more cloud services offered by the cloud environment to its customers. The CSPI provided by a CSP can include various types of hardware and software resources, including computing resources, memory resources, networking resources, consoles for accessing cloud services, and the like. Customers of a cloud environment provided by a CSP can subscribe to one or more of the cloud services offered by the cloud environment. A CSP can provide various subscription models to its customers. After a customer subscribes to a cloud service offered by a cloud environment, one or more users can be associated with the subscribing customer and these users can use the cloud service to which the customer subscribes. In certain implementations, when a customer subscribes to a cloud service offered by a particular cloud environment, a customer account or customer tenancy is created for the customer. One or more users can then be associated with the customer tenancy and these users can then use the service to which the customer subscribes under the customer tenancy. Information about the service to which a customer subscribes, the users associated with the customer tenancy, and the like are typically stored within the cloud environment and associated with the customer tenancy.

[0159] For example, Figure 6 Three different cloud environments provided by three different CSPs are depicted in FIG. 6. These include cloud environment A (Cloud A) 610 provided by CSP A, cloud environment B (Cloud B) 640 provided by CSP B, and cloud environment C (Cloud C) 660 provided by CSP C. Cloud A 610 includes infrastructure CSPI_A 612 provided by CSP A, and this infrastructure can be used to provide a set of services “Services A” 614 offered by Cloud A 610. One or more customers (e.g., Customer Al 616-1, Customer A2 616-2) can subscribe to one or more of the services of Services A 614 offered by Cloud A 610. One or more users 618-1 can be associated with Customer Al 616-1 and can use the services to which Customer Al 616-1 subscribes in Cloud A 610. In a similar manner, one or more users 618-2 can be associated with Customer A2 616-2 and can use the services to which Customer A2 616-2 subscribes in Cloud A 610. In various use cases, the services to which Customer Al 616-1 subscribes can be different from the services to which Customer A2 616-2 subscribes.

[0160] As Figure 6As depicted in the middle, cloud B 640 includes infrastructure CSPI B 642 provided by CSP B, and this infrastructure can be used to provide a set of services "Services B" 644 offered by cloud B 640. One or more customers (e.g., customer Bl 646-1) can subscribe to one or more services in Services B 644. One or more users 648-1 can be associated with customer Bl 646-1 and can use the services subscribed to by customer Bl 646-1 in cloud B 640.

[0161] As Figure 6 As depicted in the middle, cloud B 640 includes infrastructure CSPI B 642 provided by CSP B, and this infrastructure can be used to provide a set of services "Services B" 644 offered by cloud B 640. One or more customers (e.g., customer Bl 646-1) can subscribe to one or more services in Services B 644. One or more users 648-1 can be associated with customer Bl 646-1 and can use the services subscribed to by customer Bl 646-1 in cloud B 640.

[0162] In existing cloud implementations, each cloud provides a closed ecosystem for its subscribing customers and associated users. As a result, customers of a cloud environment and their associated users are limited to using the services offered by the cloud to which the customer subscribes. For example, customer Bl 646-1 and its users 648-1 are limited to using Services B 644 offered by cloud B 640, and cannot access services from a different cloud environment, such as services in Services A 614 offered by cloud A 610 or services in Services C 664 offered by cloud C 660, using their account in cloud B 640. The teachings described herein overcome this limitation. As described in this disclosure, various techniques are described that enable the creation of a link between two cloud environments that enables services offered by a first cloud environment provided by a first CSP to be used by a customer (and associated users) of a different second cloud environment provided by a second different CSP using the customer's account in the second cloud environment.

[0163] For example, in Figure 6In the depicted embodiment, in addition to other infrastructure 620, the infrastructure CSPI A 612 provided by CSP A includes special infrastructure 622 (referred to as multi-cloud enablement infrastructure 622 or MEI 622 or multi-cloud infrastructure 622) that enables one or more services 614 provided by cloud A to be used by customers and related users of other clouds, such as cloud B 640 and C 660, using customer accounts in those other clouds. In certain implementations, customers of cloud B and C do not have to open a separate account with cloud A to use one or more of the services 614 provided by cloud A 610. Customer Bl 646-1 and related user 648-1 of cloud B 640 can use their customer account or tenancy in cloud B 640 to use one or more services 614 provided by cloud A 610. As another example, customer Cl 666-1 and related user 668-1 of cloud C 660 can use their customer account or tenancy in cloud C 660 to use one or more services 614 provided by cloud A 610.

[0164] In certain implementations, the MEI 622 enables links to be created between cloud A 610 and other clouds, where these links can be used by customers of the other clouds and their associated users to access and use services provided by cloud A 610. This is illustrated in Figure 6 symbolic form as links 670 created between cloud A 610 and cloud B 640, and links 672 created between cloud A 610 and cloud C 660. Via links 670, customers of cloud B 640 can access or use one or more services 614 provided by cloud A 610. Likewise, via links 672, customers of cloud C 660 can access or use one or more services 614 provided by cloud A 610.

[0165] There are different ways in which the MEI 612 can be implemented. In certain embodiments, the MEI 612 can include components that enable links to be established with different clouds. For example, in Figure 6 In the depicted embodiment, the MEI 622 includes infrastructure components 624 responsible for enabling links 670 with cloud B 640, and infrastructure components 626 for enabling links 672 with cloud C 660. In a similar manner, the MEI 622 can include other components that enable and facilitate links with other clouds. In some embodiments, the components of the MEI 622 can also facilitate links with multiple different clouds.

[0166] There are several reasons why a customer of one cloud can want or desire to use cloud services provided by a different cloud. For example, Figure 6For example, customer Bl 646-1 of cloud B 640 can want to use cloud services 614 provided by cloud A 610 for a number of reasons. In one use case scenario, this can occur because cloud A 610 provides cloud services that have functionality that is not provided by cloud B 640. As another use case scenario, cloud A and B can provide similar services, but the services provided by cloud A 610 can be better (e.g., more features / functionality, faster, etc.) than the corresponding services provided by cloud B 640. As another use case scenario, customer Bl 646-1 of cloud B 640 can want to use cloud services provided by cloud A 610 because the price of the services is cheaper than what is provided by cloud B 640. In some cases, there can be a geographic restriction or other reason why customer Bl 646-1 of cloud B 640 can want to use cloud services provided by cloud A 610. For example, cloud A 610 can provide the desired services in a geographic area where cloud B 640 does not provide services, or cloud B 640 does not provide a particular service in a geographic area where the customer requires the service. There can be several other use case scenarios why a customer of one cloud can want to use services provided by a different cloud.

[0167] In certain embodiments, MEI 622 provides capabilities and performs functions to create a link between cloud A 610 and another cloud, and via the link, enables users associated with customers of the other cloud to access and use services provided by cloud A 610 from the other cloud itself in a seamless manner. For example, MEI 622 enables user 648-1 associated with customer Bl 646-1 of cloud 640 to access services in services A 614 provided by cloud A 610 in a seamless manner. In certain implementations, a user interface (e.g., a console) that user 648-1 can access from within cloud B 640 can be provided that enables the user to see a list of services 614 provided by cloud A 610 and select a particular service that user 648-1 wishes to access. In response to the user selection, MEI 622 is responsible for performing the processing to establish a link 670 between cloud A and B to enable access to the requested service. The processing to set up link 670 is performed substantially automatically by MEI 622. Customer Bl 646-1 or the associated user 648-1 does not have to worry about performing any system, networking, or other configuration changes needed to facilitate creating, maintaining, and using link 670 between cloud A 610 and B 640. There is no burden on the user or customer in creating the link between the clouds. The link is created in a quick and efficient manner using the techniques described in this disclosure.

[0168] MEI 622 can use various techniques to make the creation and use of the link seamless to users and customers, thereby providing an enhanced user experience. In certain implementations, MEI 622 makes the user interfaces (e.g., graphical user interfaces, GUIs, etc.) and processing flows (such as for requesting services from cloud A 610 and for accessing the requested services from cloud A 610) that customer Bl and associated user 648-1 interact with substantially similar to the interfaces and processing flows that the customer / user would experience in cloud B 640. In this way, a customer or user who can be accustomed to the interfaces and processing flows of cloud B 640 does not have to learn new interfaces and processing flows to access services 614 from cloud A 610. MEI 622 can present different interfaces and processing flows for users of different cloud environments. For example, a first set of user interfaces and flows that are substantially similar to the user interfaces and flows of cloud B can be presented to users from cloud B 640, while another set of user interfaces and flows that are substantially similar to the user interfaces and flows of cloud C can be presented to users accessing cloud A 610 from cloud C 660. This is done to simplify the access to services 614 of cloud A 610 from other clouds and thus enhance the user’s experience.

[0169] As another example, each cloud environment typically includes an identity management system configured to provide security for the cloud environment. The identity management system is configured to protect resources in the cloud environment, including resources provided by the CSP and resources of the subscribing cloud customers deployed in the cloud environment. Functions performed by the identity management system include, for example, managing identity credentials (e.g., usernames, passwords, etc.) associated with subscribing customers of the cloud and associated users, regulating access of users to cloud resources and services based on permissions / access policies configured for the cloud environment using the identity credentials, and other functions. Different clouds can use different identity management systems and associated techniques. For example, the identity management system and associated processes in cloud A 610 can be completely different from the identity management system and associated processes in cloud B 640, which in turn can be completely different from the identity management system and associated processes in cloud C 660. In certain implementations, while there are differences in the identity management systems and associated processes between different cloud environments, the techniques described herein enable users associated with a customer of a first cloud to access cloud services provided by a different cloud using the same identity credentials associated with the customer and users in the first cloud.

[0170] For example, in Figure 6In the depicted embodiment, cloud B 640, provided by CSP B, can include an identity management system that assigns or allocates identity credentials to its subscribing customers and associated users, such as customer B1 646-1 and associated user 648-1. These identity credentials are associated with a tenancy created in cloud B 640 for customer B1 646-1. In certain implementations, MEI 622 provided by cloud A 610 enables user 648-1, associated with cloud B customer B1 646-1, to access services from service A 614 in cloud A 610 using identity credentials associated with user 648-1 and customer B1 646-1 in cloud B 640. This greatly enhances the user experience of user 648-1, as they do not have to create new identity credentials specific to cloud A 610 just to access services 614 in cloud A 610. MEI 622 facilitates this access.

[0171] As an example, customer B1 of cloud B 640 can select to use a service from the set of services 614 provided by cloud A 610, such as a database as a service (DBaaS). In response to this selection, MEI 622 causes a link 670 to be automatically created between cloud A 610 and cloud B 640 to enable user 648-1, associated with customer B1 646-1, to use the DBaaS service provided by cloud A 610. The automatic setup of link 670 is facilitated by MEI 622. After link 670 has been setup, user 648-1 can use the DBaaS service in cloud A 610 via cloud B 640. As part of using this service, user 648-1 can send a request to cloud A 610 via cloud B 640 to create a database resource. In response, CSPI_A 612 can create the requested database in cloud A 610. In certain implementations, the created database can be provisioned in a virtual network (e.g., a virtual cloud network or VCN) created for customer B1 in cloud A 610 and accessible by user 648-1 via cloud B 640. User 648-1 can then send requests to cloud A 610 from cloud B 640 to use the provisioned database. These requests can include, for example, requests to write data to the database, update data stored in the database, delete data in the database, delete the database, create additional databases, etc. In some use cases, these requests can originate from user 648-1 or from services 644 provided by cloud B 640 via cloud B 640. In this way, MEI 622 provided by cloud A 610 enables users associated with customers of different clouds provided by different CSPs to seamlessly access services provided by cloud A 610.

[0172] Figure 6The depicted distributed environment 600 is merely an example and is not intended to unduly limit the scope of claimed embodiments. Numerous variations, alternatives, and modifications are possible. For example, in alternative embodiments, the distributed environment 600 can have more or less cloud environments. The cloud environments can also have more or less systems and components, or can have different configurations or arrangements of systems and components. Figure 6 The depicted systems and components can be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective systems, in hardware, or combination thereof. The software can be stored on a non-transitory storage medium (e.g., on a memory device).

[0173] Multi-Cloud Control Plane (MCCP)

[0174] Figure 7 A high-level architecture of a multi-cloud control plane (MCCP) is depicted in accordance with some embodiments. As Figure 7 As shown, the high-level architecture 700 includes a first cloud environment provided by a first cloud service provider (e.g., OCI) 720 and a second cloud environment provided by a second cloud service provider 710 (i.e., Azure). The first cloud environment 720 includes a first cloud infrastructure 720A that provides a variety of services to users of the first cloud environment 720. In addition, the first cloud environment 720 includes a multi-cloud infrastructure 720B that provisions the ability to deliver services of the first cloud infrastructure 720A (e.g., Oracle Cloud Infrastructure (OCI)) to users of other cloud environments (e.g., second cloud infrastructure 710A). The multi-cloud infrastructure 720B allows users to access services on other clouds (e.g., Oracle PaaS services) with a user experience as close to the user’s native cloud environment as possible, while providing for easy integration between cloud environments.

[0175] The second cloud infrastructure 710A includes a second cloud portal 711, an active directory 712, a resource manager 713, a customer subscription 715, a subscription of a first cloud provider 717 (i.e., a subscription of a multi-cloud infrastructure 720B in the second cloud environment). The second cloud portal 711 is a centralized access point where customers of the second environment 710 can log in and manage their cloud deployments and instances. It is to be noted that the second cloud portal can provide options for monitoring and operating the services provided by the second cloud infrastructure. The active directory 712 is a service provided by the second cloud infrastructure 710A that provides administrators the ability to manage end-user identities and access privileges. Its services can include core directory, access management, and identity protection. The resource manager 713 is a deployment and management service of the second cloud infrastructure, i.e., provides a management layer to users to perform operations (e.g., create, update, delete, etc.) on resources deployed in the customer subscription 715. It is to be noted that the customer subscription can also be referred to as a virtual network (VNET) where customers applications are deployed and executed. The subscription of the first cloud provider 717 in the second cloud infrastructure 710A includes express route and spoke and hub VNETs that provision network connectivity to be established with the second cloud infrastructure 710A (e.g., from a pre-located location, from an external cloud environment). It is to be noted that such connectivity can not be routed through the public internet, thereby providing users with higher reliability, faster speed, consistent latency, and higher security.

[0176] The first cloud infrastructure 720A includes a control plane 724, a customer tenancy 726, and a multi-cloud infrastructure 720B. The control plane 724 of the first cloud infrastructure 720A is the native control plane of the first cloud environment that provides management and orchestration across the cloud environment. Here, configuration baselines are set, user and role access is provisioned, and applications reside so that they can be executed with the relevant services. The multi-cloud infrastructure 720B includes a multi-cloud platform data plane 722 and a multi-cloud platform data plane 728. As previously described, the multi-cloud infrastructure 720B provisions functionality that enables users of other cloud environments (e.g., the second cloud environment 710) to access services provided by the first cloud environment in a user experience that is as close to possible to that of the users’ native cloud environment (e.g., the second cloud environment 710), while providing for easy integration between the cloud environments.

[0177] Figure 7 The MCCP architecture of FIG. 7 also includes a multi-cloud console 721 (different from the second cloud portal 711) that allows users authenticated in the second cloud infrastructure 710 to perform control plane operations on resources of the first cloud infrastructure 720 exposed via the multi-cloud infrastructure 720B. In some embodiments, as Figure 7As shown in the middle, all user 705 requests transmitted to the multi-cloud console 721 are directed to the multi-cloud infrastructure included in the first cloud environment. It should be appreciated that the user 705 can directly transmit requests (e.g., CRUD requests) for resources provided by the first cloud infrastructure to the multi-cloud console 721. The multi-cloud console 721 is configured to process requests having a syntax or format similar to that used natively by the second cloud infrastructure. In other words, the multi-cloud console 721 has a similar look and feel, and uses similar terminology as the second cloud portal 711 included in the second cloud infrastructure 710A. In some embodiments, a link (e.g., weblink) can be provided in the second cloud console 711 that directs the user to the multi-cloud console 721.

[0178] The multi-cloud infrastructure 720B included in the first cloud infrastructure 720A includes a plurality of microservices, such as a permissions module 722A, a proxy module 722B, a platform services module 722C, a cloud-link adapter 722D, an adapter pool 722E (including Adapter 1, Adapter 2, Adapter 3, and Adapter 4), and a network link adapter 722F. The adapter pool 722E can include adapters such as an Exa-data cloud services adapter, an autonomous database shared adapter, an autonomous database dedicated adapter, and a virtual machine database adapter.

[0179] Each adapter included in the adapter pool 722E is responsible for exposing a unique set of underlying resources (provided by the first cloud infrastructure 720A) to users of other cloud environments (e.g., the second cloud environment). Specifically, each adapter in the adapter pool 722E maps to a particular product or resource provided by the first cloud infrastructure 720A. It is noted that the actual resources are created by a native control plane 724 of the first cloud infrastructure. For example, with respect to a database as a service (DBaaS), a DBaaS control plane included in the control plane 724 is configured to instantiate Exa database resources in a customer tenancy 726 of the first cloud environment.

[0180] The incoming requests received by the multi-cloud infrastructure 720B are processed by the permissions module 722A to perform authentication and access control. Each request includes a token associated with the user's account in the second cloud infrastructure. The permissions module extracts the token and validates the token in conjunction with the active directory 712 (i.e., the identity provider system of the second cloud infrastructure 710A). Upon successful validation, the permissions module 722A can check the roles (i.e., a set of privileges) associated with the user. It is to be noted that a role can be associated with one or more tasks / operations permitted for that role. According to an embodiment, the permissions module 722A is responsible for authenticating the incoming requests to the MCCP and authorizing whether the user is allowed to perform the requested operation based on the roles associated with the token. In some implementations, the permissions module 722A can perform the above authentication process by leveraging the custom authentication features of the service platform (i.e., SPLAT associated with the first cloud infrastructure). The SPLAT accepts the incoming request and forwards it to the permissions module 722A, which further parses the incoming request to determine the permissions decision and returns a success or failure message to the SPLAT. If successful, the SPLAT passes the request to the routing agent 722B, whereas if failed, the SPLAT returns an error response directly to the caller.

[0181] The proxy module 722B (also referred to as the routing agent module) included in the multi-cloud infrastructure 720B is a component that receives incoming requests from the multi-cloud console 721 and routes the request to a particular adapter included in the adapter pool 722E. According to an embodiment, the proxy module 722B accepts the pre-authenticated requests from the service platform (i.e., SPLAT) of the first cloud infrastructure and routes the request to the appropriate adapter based on the path information included in the incoming request. In some implementations, the proxy module 722B extracts the identifier corresponding to the service provider (from the incoming request) and routes the request to the appropriate adapter in the adapter pool 722E.

[0182] The cloud-link adapter 722D included in the multi-cloud infrastructure 720B is responsible for handling the lifecycle operations of the resources provided by the first cloud infrastructure. The cloud-link adapter 722D is configured to create a mapping (or relationship created in the registration process) between the active directory tenant (and its associated subscription) of the second cloud infrastructure and the corresponding lease / account of the user in the first cloud infrastructure. In other words, the cloud-link adapter 722D generates a mapping of the first identifier associated with the user's lease in the first cloud infrastructure to the second identifier associated with the user's account in the second cloud infrastructure.

[0183] In some embodiments, the cloud-link adapter 722D performs a translation between an external cloud identifier (e.g., a second identifier associated with a user’s account in the second cloud infrastructure) and a first identifier (associated with the user’s tenancy in the first cloud infrastructure) to enable operations through the multi-cloud control plane 722 to map to the appropriate underlying resources in the first cloud infrastructure. In some embodiments, the cloud-link adapter generates a data object to store the above mapping information. Additionally, the cloud-link adapter 722D also generates a resource principal associated with the data object. Based on the token (and its associated roles) included in the request, one or more permissions are assigned to the resource principal. The user accesses downstream services provided by the first cloud infrastructure from the second cloud infrastructure based on the resource principal. The cloud-link adapter 722D can store the data object and the associated resource principal in a root compartment of the user’s tenancy in the first cloud infrastructure. Alternatively or additionally, the cloud-link adapter 722D can also save the data object and the resource principal locally on the platform services module 722C of the multi-cloud infrastructure for seamless access by other adapters included in the multi-cloud infrastructure.

[0184] The network link module (also referred to as a network adapter) 722F is responsible for creating a network link between the customer subscription 715 (in the second cloud infrastructure) and the corresponding customer tenancy / account (in the first cloud infrastructure) 726. According to some embodiments, the network link module 722F obtains the token (from the platform services module 722C) and creates (1) a first peering relationship between the multi-cloud platform data plane 728 and the customer tenancy 726 (in the first cloud environment), and (2) a second peering relationship between the customer subscription 715 and a subscription of the first cloud service provider 717 included in the second cloud infrastructure (in the second cloud environment).

[0185] The network link module 722F is also configured to establish network connectivity between the first cloud environment and the second cloud environment, i.e., the network link module 722F can configure the interconnection 719 to communicatively couple the two cloud environments. It is to be appreciated that after the network link is formed between the two cloud environments, an application executing in the customer’s subscription (e.g., in a VNET of the second cloud infrastructure) is able to access resources, e.g., an Exa database resource deployed in the customer tenancy 726 of the first cloud infrastructure. Additionally, as Figure 7As shown in the middle, within the tenancy 726, functionality to provide telemetry is provided. This functionality involves maintaining logs, metrics, and other performance parameters related to the resources deployed in the customer tenancy and their respective usage. According to some embodiments, the MCCP produces a mirror of the logs and metrics associated with different resources in the first cloud environment and publishes the metrics, logs, events, etc. for further processing, e.g., in a dashboard (e.g., in an application insight module included in the customer subscription 715 of the second cloud environment).

[0186] In some embodiments, the platform services module 722C included in the multi-cloud infrastructure is configured to store credentials associated with the services of the first cloud infrastructure provided to the second cloud infrastructure. The platform services module 722C provides, for example, tokens / resource principals for different adapters included in the adapter pool 722E so that the adapters can communicate with the native control plane 724 of the first cloud infrastructure. Through some embodiments, the platform services module 722C exposes APIs called by different adapters to perform tasks such as:

[0187] • selling a minimum scope of access tokens (issued by the second cloud infrastructure) to the adapters. For example, the network adapter 722F requires an access token to perform the network peering operations described above.

[0188] • providing resource principals that the adapters will use to call downstream services to create resources in the customer tenancy of the first cloud infrastructure.

[0189] • triggering replication of observability data (logs, metrics, events) from the first cloud infrastructure to the second cloud infrastructure.

[0190] As mentioned before, the adapter pool includes multiple adapters, each responsible for exposing a unique set of underlying resources of the first cloud infrastructure to the users of the second cloud infrastructure, i.e., each adapter maps to a specific product or resource provided by the first cloud environment. For example, an Exa Database adapter acts as a proxy for the second cloud infrastructure users to create and utilize Exa Database resources. Exa Database is a pre-configured combination of hardware and software that provides an infrastructure for executing databases. According to some embodiments, Exa Database includes a set of resources: (a) Exadata infrastructure (i.e., hardware), (b) VM cloud cluster, (c) container database, and (d) pluggable database. According to some embodiments, the multi-cloud infrastructure provides the ability to analyze each level of the stack for the users of the second cloud infrastructure. Moreover, the MCCP provides flexibility to the users who simply issue a create command for a workflow (via the multi-cloud console 721) after which the MCCP automatically creates individual resources at each level of the stack. It should be recognized that while the Exa Database is used as an example, other products provided by the first cloud environment can be used as well. Figure 7The adapter pool 722F depicted in the middle includes four different adapters, but this is in no way limiting the scope of the MCCP architecture 700. The MCCP architecture can include other adapters, for example, specialized adapters based on the requirements of a cloud service provider for exclusive use by the particular cloud service provider.

[0191] Figure 8 An exemplary system diagram illustrating components of a multi-cloud control plane (MCCP) in accordance with some embodiments is depicted. The MCCP 800 includes a service platform (SPLAT) 810, a routing agent 815, a cloud-link adapter 820, a database adapter 825, a network adapter 830, and an MCCP platform 835. After a user successfully completes a registration process, the user can utilize a multi-cloud console 805 to issue a command to access, create, or update a resource in the user's tenancy in a first cloud infrastructure. For ease of illustration, in the following, a scenario is described in which the user utilizes the multi-cloud console to issue a request to create an Exa database resource.

[0192] In some implementations, the user accesses the multi-cloud console 805 and provides login information, for example, the user's credentials in a second cloud infrastructure. The multi-cloud console 805 provides a plurality of options, for example, create resource, access resource, update resource, etc. Such options can be provided to the user in the form of selectable icons (e.g., buttons) in the multi-cloud console 805. When the user performs a selection (e.g., create resource), an API call to the service platform 810 is triggered. It should be appreciated that the request issued to the service platform 810 is not a native call to the first cloud infrastructure. Rather, the call is a REST type call that includes an authorization header that includes a token associated with the user in the second cloud infrastructure.

[0193] The REST call that includes the token is further forwarded to the routing agent module 815 that performs authentication and access control operations. According to some embodiments, the routing agent module 815 performs an authentication operation by extracting the token included in the REST call. The routing agent module 815 validates the token by comparing a signature (used to sign the request) with a publicly available signature of the second cloud infrastructure to ensure that the request originated from a valid customer associated with the second cloud infrastructure. Additionally, the routing agent module 815 can also check the role (i.e., privileges) associated with the token, for example, whether the role corresponds to an Exadata DB administrator or the like. Based on the role, the routing agent module 815 can route the request to the appropriate adapter included in the MCCP framework 800.

[0194] According to one embodiment, the routing agent module 815 compares the role (associated with the token) to a preconfigured list of roles published and assigned (as part of the API specification) for each adapter. For example, if the role associated with the token corresponds to "Exadata DB Admin," then the request can be understood as a request to create an Exa database, and thus the request is forwarded to the database adapter 825. Further, according to some embodiments, the routing agent module 815 can analyze information included in the REST call, such as the provider ID, the type of resource requested, etc., and based on the analyzed information, the routing agent module 815 can forward the request to the appropriate adapter.

[0195] In some implementations, the request obtained by the routing agent module 815 can not include information identifying the user tenancy in the first cloud infrastructure in which the resource is to be deployed. Accordingly, the routing agent module 815 communicates with the cloud-link adapter 820 to obtain mapping information of the user account in the second cloud infrastructure to the user tenancy in the first cloud infrastructure. If the mapping information exists, the routing agent module 815 obtains information related to the user tenancy in the first cloud infrastructure and passes the information to the database adapter 825. In this way, the database adapter 825 is aware of the user tenancy in the first cloud infrastructure in which the resource is to be created / deployed. However, if the cloud-link adapter 820 determines that the mapping information does not exist, then the routing agent module 815 can simply issue an "unauthorized access" message as a response to the request to create the database resource, which is transmitted back to the user.

[0196] It is noted that in some implementations, the cloud-link adapter 820 creates a data object (referred to herein as a cloud-link resource object) for storing metadata information identifying the two linked accounts. For example, the data object stores metadata information including a mapping of a first identifier associated with a tenancy (i.e., account) in the first cloud infrastructure to a second identifier associated with an account of a user of the second cloud service provider. Further, the cloud-link adapter 820 also creates a resource principal (referred to herein as a cloud-link resource principal) for a resource (e.g., a database) that the user of the second cloud infrastructure desires to create / manage. The cloud-link adapter 820 can maintain the data object as well as the resource principal within a root compartment of the user tenancy in the first cloud infrastructure. In some embodiments, the cloud-link adapter 820 can also save the data object and / or the resource principal locally in the MCCP platform 835.

[0197] In some embodiments, the database adapter 825 can (indicate or) communicate with the network adapter 830 to create a network link between the user account in the second cloud infrastructure and the user tenancy in the first cloud infrastructure. For example, the network adapter 830 can obtain a token associated with the user from the native service 845 in the second cloud infrastructure module and create: (1) a first peer relationship between the data plane of the MCCP and the tenancy of the user in the first cloud infrastructure (in the first cloud environment), and (2) a second peer relationship between the account of the user and a subscription of the first cloud provider included in the second cloud infrastructure (in the second cloud environment).

[0198] The network adapter 830 is also configured to establish network connectivity between the first cloud infrastructure and the second cloud infrastructure, i.e., the network adapter 830 can configure an interconnect (e.g., the interconnect 719 in FIG. 7) that communicatively couples the two cloud environments. Figure 7 It is to be appreciated that after the network link is formed between the user tenancy in the first cloud infrastructure and the user account in the second cloud infrastructure, applications executing in the subscription / account of the user are able to access resources, e.g., an Exa database deployed in the tenancy of the first cloud infrastructure. Moreover, the creation of the peer relationship provisions metrics, e.g., database usage metrics, that are accessible in the second cloud infrastructure, e.g., in a dashboard application executing in the subscription of the user in the second cloud infrastructure.

[0199] In some implementations, the database adapter 825 can obtain a resource principal that is locally saved in the MCCP platform 835. The database adapter 825 can transmit a request (including the resource principal) to one or more downstream services 840 included in the first cloud infrastructure to create a resource in the tenancy of the user in the first cloud infrastructure. In other words, the downstream services 840 included in the first cloud infrastructure utilize the identity (i.e., the resource principal) obtained from the MCCP platform 835 to create / deploy the required resource, e.g., an Exa database, in the tenancy of the user in the first cloud infrastructure. After the user issues a request to create the Exa database, the user can intermittently poll the MCCP 800 to obtain the status of the request. After the downstream services 840 of the first cloud infrastructure create the resource in the tenancy of the user in the first cloud infrastructure and the network adapter 830 establishes the peer relationship, the MCCP 800 can notify the user of the successful completion of the request.

[0200] Multi-Cloud Network Service - Network Adapter

[0201] Referring to Figure 7, the architecture of the multi-cloud control plane 700 enables customers of an external cloud environment (e.g., the second cloud environment 710) (provided by an external cloud service provider) to deploy resources (e.g., database resources) offered in the first cloud environment 720, execute services, etc. by leveraging the multi-cloud console 721 and the multi-cloud infrastructure 720B (included in the first cloud environment). In some embodiments, network connectivity between the first cloud environment and the second cloud environment is configured and maintained in order to expose offerings (e.g., PaaS offerings) of the first cloud environment to customers of the second cloud environment. As described in detail below, a network adapter (e.g., the network link component 722F of the multi-cloud control plane 700) is responsible for handling such network-related resources. Figure 7

[0202] In certain embodiments, a multi-cloud network (MCN) service is provided, which is responsible for configuring and maintaining network connectivity between the first cloud environment and other cloud environments (e.g., the second cloud environment). The MCN exposes services such as PaaS offerings to customers of the other cloud environments in a seamless manner. It can be appreciated that the MCN service is suitable for Figure 7 the architecture of the multi-cloud control plane 700 of the multi-cloud platform as the network adapter component 722F. According to some embodiments, the MCN service is responsible for:

[0203] 1. Integrating with the multi-cloud platform and exposing customer-facing API objects (referred to as network-links) that will be used as an abstraction for cross-cloud virtual network interconnection for customers.

[0204] 2. Configuring and maintaining direct interconnect links (e.g., ExpressRoute, FastConnect, etc.) between the first cloud environment and external cloud environments to be shared across N customers.

[0205] 3. Launching and monitoring packet processor instances for each customer to enable end-to-end packet flow between the external cloud environments and the first cloud environment.

[0206] 4. Configuring any domain name system (DNS) related resources in the external cloud environments and the first cloud environment to allow seamless name resolution.

[0207] ​A network-link, as the name implies, is an abstraction that interconnects two cloud environments. For example, a network-link can interconnect a customer's virtual network in the second cloud environment to the customer's virtual cloud network (VCN) in the first cloud environment. It can be appreciated that if the customer has to manually set up the network-link, then the customer is responsible for a large number of tasks such as configuring an interconnect (e.g., Express Route) for the second cloud environment, configuring another interconnect (e.g., Fast Connect) for the first cloud environment, configuring a dynamic routing gateway, configuring gateway attachments and routing tables, and so on. Such tasks are by no means trivial in nature and result in a poor customer experience. As described by embodiments of the present disclosure, a multi-cloud network (MCN) service is provided that automatically configures the network-link (and related networking resources) with little to no customer interaction(s). Thus, by using the MCN service, the customer need not worry about the internal structure of the network-link components provided by the multi-cloud infrastructure of the first cloud environment and can simply use the network-link as a black box that allows traffic to and from the customer's virtual network in different cloud environments. Further, it is noted that the network-link established between the different cloud environments can need to have very low transmission latency. This is due to the fact that the resources deployed in the first cloud environment can be resources such as Exa database resources that handle online transaction processing events. Since such events require minimal latency, the multi-cloud network (MCN) service of the present disclosure relies on establishing a high-performance and highly available end-to-end network-link between the different cloud environments with low transmission latency.

[0208] Figure 9 A high-level block diagram of a network-link component is depicted in accordance with certain embodiments. In particular, Figure 9 A network-link component 907 (also referred to herein as a multi-cloud network infrastructure (MCNI)) is depicted that is established between a first cloud environment 905 and a second cloud environment 910. As Figure 9 As shown in the middle, the network-link component 907 is established between a customer's virtual private cloud (VPC) (e.g., VPC 922 included in the second cloud environment 910) and the customer's virtual cloud network (VCN) 912 (i.e., an account in the first cloud environment 905). Thus, resources 913 (e.g., Exa database) hosted in the customer's VCN 912 (in the first cloud environment 905) can be accessed / utilized by virtual machines 923 hosted in the customer's VPC 922 in the second cloud environment 910.

[0209] As Figure 9As shown in FIG. 9, on the second cloud environment side, the network link component 907 includes a VPC (labeled spoke VPC) 919 that is peered with a customer's VPC 922 included in the second cloud environment 910. Any traffic to and from the customer's VCN 912 is put on this VPC peering. Note that the spoke VPC 919 appears to the customer as a generic "network link VPC." On the first cloud environment side, the network link component 907 has a dynamic routing gateway (DRG) attachment, named multi-cloud attachment 917. In some embodiments, a customer can use a VCN attachment to attach one's VCN (e.g., customer VCN 912) to a DRG 915 owned by the customer, and the MCNI 907 links into this DRG 915 via the multi-cloud attachment 917. Note that any traffic to and from the customer's VPC 922 (included in the second cloud environment 910) is put on the VCN attachment and then to the multi-cloud attachment 917 within the DRG.

[0210] As will be described below with reference to Figure 11 As will be described, some portions of the network link component 907 will be included in the first cloud environment 905, while some other portions of the network link component 907 will be included in the second cloud environment 910. Further, it can be appreciated that one of the goals of the network link component 907 is to construct a multi-tenant tunneling processing infrastructure to share the interconnection(s) between the first cloud environment and the second cloud environment. In order to have multi-tenant traffic share a single intercloud interconnection, note that the network link component 907 provisions tunneling processing to encapsulate / decapsulate customer traffic, e.g., by using GENEVE tunnels.

[0211] Turning to Figure 10 depicted is a detailed architecture 1000 of a network link configuration, in accordance with certain embodiments. As Figure 10 As shown in FIG. 10, the network link communicatively couples regions of the second cloud environment 1005 to regions of the first cloud environment 1035. The regions of the second cloud environment 1005 can include one or more customer virtual private cloud (VPC) networks (also referred to as "customer virtual networks"). For example, the regions of the second cloud environment 1005 include two customer VPCs, namely, customer 1 VPC 1001 and customer 2 VPC 1002. Accordingly, the portion of the second cloud environment 1005 hosting the customer VPCs is denoted as customer domain 1050A. Similarly, the regions of the first cloud environment 1035 can include one or more customer VCNs (i.e., customer virtual networks). For example, the regions of the first cloud environment 1035 include two customer VCNs, namely, customer 1 VCN 1031 and customer 2 VCN 1032. Accordingly, the portion of the first cloud environment 1035 hosting the customer VCNs is denoted as customer domain 1050C.

[0212] According to some embodiments, a multi-cloud network infrastructure (MCNI) domain 1050B (i.e., network-link) communicatively couples a pair of customer virtual networks. For example, as shown in FIG. 10B, customer 1 VPC 1001 (included in a region of second cloud environment 1005) is communicatively coupled with customer 1 VCN 1031 (included in a region of first cloud environment 1035). In a similar manner, customer 2 VPC 1002 (included in a region of second cloud environment 1005) is communicatively coupled with customer 2 VCN 1032 (included in a region of first cloud environment 1035). It is noted that MCNI domain 1050 corresponds to a pair of network links and is deployed between dotted lines 1060 and 1070. More specifically, MCNI domain 1050B includes a first portion deployed in a region of second cloud environment 1005 and a second portion deployed in a region of first cloud environment 1035. It is appreciated that MCNI domain 1050B is controlled by a cloud service provider of first cloud environment 1035. Figure 10

[0213] In some implementations, each pair of customer virtual networks (e.g., customer 1 VPC 1001 in a region of second cloud environment 1005 and customer 1 VCN 1031 in a region of first cloud environment 1035) is communicatively coupled using a plurality of virtual networks (referred to herein as link-enabled virtual networks) deployed by a multi-cloud network infrastructure (MCNI) service (in the first and second cloud environments). For example, a first link-enabled virtual network 1007 (labeled spoke 1 VPC) and a second link-enabled virtual network 1023 (labeled spoke 1 VCN) are deployed in second cloud environment 1005 and first cloud environment 1035, respectively. It is appreciated that each pair of customer virtual networks (i.e., 1001 and 1031) is associated with a pair of link-enabled virtual networks (i.e., first link-enabled virtual network 1007 and second link-enabled virtual network 1023). In a similar manner, customer 2 VPC 1002 is associated with a link-enabled virtual network 1008 (labeled spoke 2 VPC) in the second cloud environment and another link-enabled virtual network 1024 (labeled spoke 2 VCN) deployed in first cloud environment 1035. Thus, in the architecture of FIG. 10A, each pair of customer virtual networks is associated with a unique pair of link-enabled virtual networks. Figure 10

[0214] ​​According to some embodiments, in the region of the second cloud environment 1005, a data plane hub virtual network (VN) 1013 and a control plane hub VN 1012 are deployed. The data plane hub virtual network (VN) 1013 is shared among different customer virtual networks included in the second cloud environment. In other words, the data plane hub virtual network (VN) 1013 handles traffic for multiple customer tenancies (i.e., 1001, 1002) included in the region of the second cloud environment 1005. Similarly, the region of the first cloud environment 1035 includes another data plane hub VN 1022 and a control plane hub VN 1025. The data plane hub VN 1022 located in the first cloud environment 1035 is shared among different customer virtual cloud networks (e.g., 1031 and 1032) included in the first cloud environment, i.e., the data plane hub VN 1022 handles traffic for multiple customer VCNs included in the region of the first cloud environment 1035. As Figure 10 As shown in the middle, the data plane hub VN 1013 included in the region of the second cloud environment 1005 is communicatively connected to the data plane hub VN 1022 included in the region of the first cloud environment 1035 through a high-bandwidth network interconnect 1015A (i.e., a private communication channel). The control plane hub VN 1025 included in the first cloud environment 1035 is communicatively coupled to the control plane hub VN 1012 included in the second cloud environment 1035 via a communication link 1015B (e.g., a public communication channel such as the Internet).

[0215] According to some embodiments, the first link-enabled virtual network 1007 is associated with a first virtual network interface card (VNIC) that communicatively couples the first link-enabled virtual network 1007 to the data plane hub virtual network 1013, a second VNIC that communicatively couples the first link-enabled virtual network 1007 to the control plane hub virtual network 1012, and a third VNIC that communicatively couples the first link-enabled virtual network 1007 to the customer virtual network 1001 in the second cloud environment 1005. The control plane hub VN 1025 included in the first cloud environment 1035 includes one or more distribution service nodes 1025A, each of which is configured to transmit network configuration information to the control plane hub virtual network 1012 included in the second cloud environment 1005. The control plane hub virtual network 1012, in turn, can forward the network configuration information to the first link-enabled virtual network 1007, as the control plane hub virtual network 1012 is coupled to the first link-enabled virtual network 1007 via a VNIC attachment (e.g., VNIC 1016). It is noted that the control plane hub virtual network 1012 can serve as a gateway to distribute the network configuration information (obtained from the one or more distribution service nodes 1025A) to other link-enabled virtual networks (e.g., spoke 2 VPC 1008) associated with other customers of the second cloud environment 1005. Additionally, the control plane hub VN 1025 included in the first cloud environment 1035 includes a service gateway (SGW) that provides, for example, endpoints for customers of the second cloud environment 1005 to access other services 1027 provided by the first cloud environment 1035, such as telemetry, identity services, etc.

[0216] According to some embodiments, the network configuration information includes at least: (i) tunnel encapsulation-decapsulation parameters, and (ii) health information of a second link-enabled virtual network deployed in the first cloud environment. Such information is conveyed to the link-enabled virtual network in the second cloud environment 1005 in order to enable the link-enabled virtual network to configure an end-to-end network path between a customer virtual network (e.g., customer VN 1001) in the second cloud environment 1005 and another customer virtual network (e.g., customer VN 1031) in the first cloud environment 1035.

[0217] As Figure 10As shown in the middle, the first link-enabled virtual network 1007 (i.e., Spoke 1 VPC) is peered (i.e., attached via VNIC) with the customer 1 VPC 1001 in order to receive traffic from a virtual machine (e.g., operated by user 1001A) executing in the customer 1 VPC 1001. Since the data plane hub VN 1013 is shared across multiple customer VPCs in the second cloud environment 1005, in some embodiments of the present disclosure, a framework of encapsulation and tunneling is utilized to distinguish between different customer traffic originating from different customer virtual networks in the second cloud environment 1005. The first link-enabled virtual network 1007 includes a pair of network adapters 1007A and 1007B (referred to herein as remote virtual network adapters (RVNAs)). Each network adapter included in the first link-enabled virtual network 1007 is configured to encapsulate traffic received from the customer 1 VPC 1001 to generate encapsulated traffic. In a similar manner, the link-enabled virtual network 1008 (corresponding to the customer virtual network 1002) includes a pair of RVNAs 1008A and 1008B configured to encapsulate traffic received from the customer 2 VPC 1002 to generate encapsulated traffic that is transmitted through the shared data plane hub virtual network 1013.

[0218] In some embodiments, the pair of network adapters (e.g., 1007A, 1007B) are utilized for high availability purposes and are maintained in an active-active state. In other words, each adapter included in the pair of network adapters has functionality and encapsulates traffic received from the customer 1 VPC 1001. In some embodiments, the pair of network adapters can utilize services provided by the second cloud environment 1005 for the purpose of performing BGP peering. Through the use of mechanisms such as equal cost multi-path routing (ECMP), the pair of network adapters can instruct such services to spread traffic (originating from the customer VPC) in a uniform manner.

[0219] Traffic encapsulated by the network adapters included in the first link-enabled virtual network 1007 is forwarded to the data plane hub VN 1013 (e.g., to the VNIC 1017 included in the data plane hub VN 1013). The data plane hub VN 1013 thereby receives encapsulated traffic from different link-enabled virtual networks (associated with different customer virtual networks of the second cloud environment 1005) and forwards the traffic to the data plane hub VN 1022 included in the first cloud environment 1035 via the high-speed dedicated interconnect channel 1015A. The data plane hub VN 1022 in turn forwards the encapsulated traffic to the second link-enabled virtual network 1023 deployed in the first cloud environment 1035 (e.g., via the VNIC 1022A included in the data plane hub VN 1022).

[0220] The second link-enabled virtual network 1023 includes a pair of virtual network adapters 1023A, 1023B (labeled local virtual network adapters (LVNAs)), each of which is configured to decapsulate encapsulated traffic received from the VNIC 1022A included in the data plane hub VN 1022. Additionally, as shown in Figure 10 The pair of virtual network adapters 1023A / 1023B included in the second link-enabled virtual network 1023 of the first cloud environment 1035 transmit the decapsulated traffic to the customer 1 VN 1031 (e.g., to the resource 1031A deployed in the customer 1 VCN 1031) via a dynamic routing gateway (DRG) attachment.

[0221] According to some embodiments of the present disclosure, in setting up the network link between the customer virtual network in the second cloud environment 1005 and the customer virtual network 1031 in the first cloud environment 1035, routing tables (RTs) are configured for each of the virtual networks included in the customer domains 1050A and 1050C and the virtual network deployed in the multi-cloud network infrastructure domain 1050B. For example, as shown in Figure 10 the second cloud environment 1005, RT 1003 is configured for the customer 1 VPC 1001, RT 1004 is configured for the customer 2 VPC 1002, RT 1009 is configured for the first link-enabled virtual network 1007, RT 1014A is configured for the control plane hub VN 1012, and RT 1014B is configured for the data plane hub VN 1013.

[0222] Similarly, with reference to the first cloud environment 1035 of Figure 10 it is noted that RT 1026 is configured for the control plane hub VN 1025, RT 1022B is configured for the data plane hub VN 1022, RT 1023B is configured for the second link-enabled virtual network 1023, and RT 1031B is configured for the customer 1 VN 1031. It can be appreciated that the entries in the RTs correspond to the address of the next hop destination in transmitting data (i.e., packets) from the second cloud environment 1005 to the first cloud environment 1035 or vice versa. Additionally, Figure 10Routing information (e.g., RT 1010 and RT 1034) for tunnels formed between virtual network adapters included in the first link-enabled virtual network 1007 and virtual network adapters included in the second link-enabled virtual network 1023 is also depicted. As an example, RT 1010 depicts that a tunnel formed from virtual network adapter 1007A (included in the first link-enabled virtual network 1007) to virtual network adapter 1023A (included in the second link-enabled virtual network 1023) is designated as a first peering tunnel between address 100.70.128.2 (i.e., the address of RVNA 1 1007A) to address 100.65.128.97 (i.e., the address of LVNA 1 1023A).

[0223] Thus, in the manner described above, an end-to-end network link is established between Customer 1 VPC 1001 in a region of the second cloud environment 1005 and Customer 1 VN 1031 in a region of the first cloud environment 1035 via the first link-enabled virtual network 1003, the second link-enabled virtual network 1023, the data plane hub VN 1013 (included in the second cloud environment 1005), and the data plane hub VN 1022 (in the first cloud environment). It can be appreciated that a network link can be established between Customer 2 VPC 1002 (located in a region of the second cloud environment 1005) and Customer 2 VN 1032 (located in a region of the first cloud environment 1035) in a manner similar to that described above with respect to the network link established between the Customer 1 VPC and the Customer 1 VN. Further, it is noted that while the customer virtual networks in the first and second cloud environments can share IP address space, each of the first and second link-enabled virtual networks, as well as the hub virtual networks included in the first and second cloud environments, are assigned unique classless inter-domain routing IP address space to avoid traffic conflicts.

[0224] As described above with respect to the architecture of the network link created between the first cloud environment and the second cloud environment Figure 10 The architecture of the network link created between the first cloud environment and the second cloud environment as described above with respect to

[0225] Figure 11 An exemplary flowchart illustrating a process of establishing a network link in accordance with certain embodiments is depicted. Figure 11The processes depicted in the figures can be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective system, hardware, or combinations thereof. The software can be stored on a non-transitory storage medium (e.g., a memory device). Figure 11 The methods presented and described below are intended to be illustrative and non-limiting. While Figure 11 Various process steps are depicted as occurring in a particular order or sequence, but this is not intended to be limiting. In certain alternative embodiments, steps can be performed in a different order, or some steps can be performed in parallel.

[0226] The process 1100 begins at step 1105 where a multi-cloud infrastructure included in a first cloud environment receives a request to create a network link between a first customer virtual network in the first cloud environment and a second customer virtual network in a second cloud environment. It should be appreciated that the first cloud environment (e.g., OCI) is provided by a first cloud service provider that is different from a second cloud service provider that provides the second cloud environment (e.g., GCP).

[0227] The process then moves to step 1110 where a first link-enabled virtual network is deployed in the second cloud environment. The first link-enabled virtual network is communicatively coupled to a first data plane hub virtual network, a first control plane hub virtual network, and the second customer virtual network. It is noted that the first data plane hub virtual network and the first control plane hub virtual network are also deployed in the second cloud environment. To establish such a connection, it is noted that the first link-enabled virtual network is provisioned with three VNICs, namely: a first VNIC that communicatively couples the first link-enabled virtual network to the first data plane hub virtual network, a second VNIC that communicatively couples the first link-enabled virtual network to the first control plane hub virtual network, and a third VNIC that communicatively couples the first link-enabled virtual network to the second customer virtual network.

[0228] The process then moves to step 1115 where the first link-enabled virtual network in the second cloud environment receives network configuration information from the first control plane hub virtual network in the second cloud environment. As previously described, such network configuration information can be generated (and distributed) by one or more distribution service nodes (e.g., nodes 1025A) included in the control plane hub virtual network in the first cloud environment and transmitted to the first control plane hub virtual network in the second cloud environment. Figure 10

[0229] ​Additionally, in step 1120, traffic is enabled to be communicated from a second customer virtual network in the second cloud environment to a first customer virtual network in the first cloud environment via a first link-enabled virtual network, where the first link-enabled virtual network transports traffic to the first cloud environment via a first data plane hub virtual network based on the network configuration information.

[0230] Hub & Spoke Network Topology

[0231] Different cloud service providers can provide a hub and spoke network architecture that includes a central hub network communicatively coupled to several spoke networks. Such hub and spoke architectures are generally used for transit routing, where the central hub network provides routing to many spoke networks for network traffic transit. The hub and spoke topology is a powerful architecture when a solution is needed to establish many isolated spoke networks that need to transit their network traffic through the hub network. The hub serves as a central gateway network that provides routing for all attached spoke networks, typically dynamic routing through BGP. In a virtual networking scenario (i.e., a cloud environment), such a topology allows all spoke networks to communicate with some networks outside the cloud. In such a scenario, all spoke networks always have the same routing (i.e., via the hub), and routing changes in the hub are applied to all spoke networks simultaneously.

[0232] However, a limitation of implementing such a hub and spoke network in a cloud environment is a scalability limitation. For example, some cloud service providers provision a solution known as “peering,” where a spoke network can be configured to peer with the central hub network. Such a peering solution limits the number of spoke networks that can be communicatively coupled with the hub network to at most a few dozen spoke networks (e.g., 50 spoke networks). Such a limitation is typically due to inherent cloud service provider offerings. In the following, a unique solution is provided to address the scalability problem when deploying a hub and spoke network architecture in a cloud environment.

[0233] It can be appreciated that in a cloud environment setting, the hub virtual network acts as a central transit network to other destinations. The destinations can be another network in the cloud, or a pre-provisioned location. The hub network learns routes (e.g., via the BGP protocol) to such networks and / or endpoints dynamically, and can instantiate security rules at the appropriate locations to allow such connectivity. On the other hand, the spoke virtual network contains the networking needed to perform some local routing within the network.

[0234] Turning to Figure 12 FIG. 1 illustrates a schematic diagram of a computing virtual machine (VM) including multiple VNICs, according to certain embodiments. For example, Figure 12A compute VM 1200 is depicted that includes two VNICs 1205 and 1210. The compute VM 1200 is created with a first VNIC (e.g., VNIC 1205) associated with a customer virtual cloud network (VCN) 1215 and a second VNIC (e.g., VNIC 1210) associated with a hub network 1220 (also referred to herein simply as a hub). In doing so, the compute VM 1200 allows a workload to have connectivity to the customer VCN 1215 as well as the hub 1220. Note that the customer VCN and the hub network are completely unconnected in the virtual networking layer and connectivity between the two only exists within the compute VM. Further, note that while the compute VM 1200 includes multiple VNICs (e.g., VNICs 1205 and 1210), the multiple VNICs share the same network namespace. Further, note that even though the compute VM 1200 includes multiple VNICs, the multiple VNICs are not allowed in some cloud environments (e.g., Azure) due to restrictions imposed by the cloud service provider. Figure 12 The basic architecture of the compute virtual machine 1200 depicted in the middle can also not be allowed in some cloud environments (e.g., Azure) due to restrictions imposed by the cloud service provider. For example, certain cloud service providers prohibit a virtual machine from having multiple VNICs where each VNIC is associated with a different network.

[0235] Figure 13 A diagram illustrating a compute virtual machine that includes multiple network namespaces is shown. A network namespace can be defined as a logical copy of a network stack of a host system. Network namespaces are useful for setting up containers or virtual environments. Network namespaces provide isolation of resources of a system associated with networking. In other words, independent networking stacks can be created where each network namespace has its own interfaces, IPv4 and IPv6 protocol stacks, routing tables, address resolution protocol (ARP) tables, firewall rules, and the like.

[0236] Reference is made to Figure 13The compute virtual machine 1300 includes an application layer 1301 and is configured with multiple network namespaces, e.g., a first network namespace (depicted as a customer VCN namespace 1305) and a second network namespace (depicted as a hub network namespace 1310). Note that the customer VCN network and the hub network are separate from each other and have their own interfaces, routing tables, ARP entries, next-hop destinations, etc. In addition, a first VNIC (e.g., VNIC 1315) is associated with the first network namespace (i.e., the customer VCN namespace 1305) and enables communication between the customer VCN and the compute virtual machine 1300. The compute virtual machine 1300 also includes a second VNIC (e.g., VNIC 1320) that is associated with the second network namespace (i.e., the hub network namespace 1310) and enables communication between the compute virtual machine 1300 and the hub. Note that the first network namespace is different from the second network namespace and each network namespace can be configured with unique configuration information (i.e., IPv4 and IPv6 protocol stacks, routing tables, address resolution protocol (ARP) tables, firewall rules, DNS resolvers, etc.). In other words, the first network namespace is programmed with first configuration information while the second network namespace is programmed with second configuration information that is different from the first configuration information. Moreover, it should be appreciated that the compute virtual machine 1300 is by no means limited to including only two network namespaces and two corresponding VNICs. Rather, the compute virtual machine 1300 can include a higher number of network namespaces. For example, the compute virtual machine 1300 can include a third VNIC associated with a third network namespace that enables communication between the compute virtual machine 1300 and another virtual network (e.g., a control plane hub network).

[0237] Figure 14 A spoke and hub network architecture deployed in a cloud environment is depicted in accordance with certain embodiments. Figure 14 A first cloud environment 1405 including a hub and spoke topology architecture deployed therein, and a second cloud environment 1410 including one or more resources desired to be used by a customer of the first cloud environment 1405 is depicted in accordance with certain embodiments. Figure 14 As shown in FIG. 14, the first cloud environment 1405 includes a plurality of spoke networks 1430, 1432, 1434, and a hub network (also referred to herein as a hub) 1450. Each spoke network includes a computer virtual machine, e.g., spoke network 1 1430 includes compute VM 1420, spoke network 2 1432 includes compute VM 1422, and spoke network N 1434 includes compute VM 1424. Each of the compute VMs 1420, 1422, and 1424 are configured in a similar manner as the compute VMs described previously with reference to FIG. 13. Figure 13 ​

[0238] In particular, consider a compute VM 1 1420 (also referred to as a first compute VM), noting that such a compute VM is provided in the first cloud environment 1405. The first compute VM 1420 is connected to a first customer virtual cloud network (VCN) 1442 in the first cloud environment and is connected to the hub 1450. The first compute VM 1420 is associated with a first plurality of virtual network interface cards (VNICs). In particular, the plurality of VNICs includes: (i) a first VNIC (e.g., VNIC 1421) that is associated with a first network namespace (i.e., the network namespace of the customer 1 VCN 1442) and enables communication between the first customer VCN 1442 and the first compute virtual machine, and (ii) a second VNIC (e.g., VNIC 1423) that is associated with a second network namespace (i.e., the network namespace of the hub 1450) and enables communication between the first compute virtual machine and the hub. As noted previously, the first network namespace is different from the second network namespace.

[0239] Similarly, as shown in FIG. 14B, a second compute virtual machine 1422 is provided in the first cloud environment 1405 that is connected to a second customer VCN 1444 in the first cloud environment and is connected to the hub 1450. The second compute virtual machine 1422 is associated with a second plurality of VNICs. In particular, the second plurality of VNICs includes: (i) a first VNIC (e.g., VNIC 1425) that is associated with a third network namespace (i.e., the network namespace of the customer 2 VCN 1444) and enables communication between the second customer VCN 1444 and the second compute virtual machine 1422, and (ii) a second VNIC (e.g., VNIC 1427) that is associated with the second network namespace (i.e., the network namespace of the hub 1450) and enables communication between the second compute virtual machine 1422 and the hub 1450. Note that the third network namespace is different from the second network namespace. Also, it should be appreciated that the first network namespace (of the compute VM 1 1420 and associated with the customer 1 VCN 1442) is different from the third network namespace (of the compute VM 2 1422 and associated with the customer 2 VCN 1444). The second network namespace (i.e., the network namespace associated with the hub) of each of the compute VMs 1 and 2 can be the same. Figure 14

[0240] ​Furthermore, compute VM N 1424 is configured in a manner similar to that described above with reference to compute VMs 1420 and 1422. Specifically, compute VM N 1424 includes a third plurality of VNICs: (i) a first VNIC (e.g., VNIC 1429) associated with a fourth network namespace (i.e., the network namespace of client N VCN 1446) and enabling communication between client VCN 1444 and compute VM N 1424; and (ii) a second VNIC (e.g., VNIC 1431) associated with a second network namespace (i.e., the network namespace of hub 1450) and enabling communication between compute VM N 1424 and hub 1450. It should be noted that the fourth network namespace is different from the first, second, and third network namespaces.

[0241] In this way, all customer VCN networks (i.e., 1442, 1444, and 1446) are isolated from each other while sharing the same route (i.e., via hub 1450) to the provisioned or cloud network 1460 (located in the second cloud environment 1410). Therefore, customers of the first cloud environment 1405 associated with different customer VCNs (i.e., 1442, 1444, and 1446) can seamlessly access resources, for example, provided by the second cloud environment 1410 via hub 1450. It should be recognized that... Figure 14 Each compute VM depicted includes two distinct network namespaces. However, it should be understood that this in no way limits the scope of this disclosure. Rather, each VM may include a higher number of network namespaces, and each compute VM may have a different number of network namespaces compared to the number of network namespaces included in another compute VM. Therefore, by referring to the above... Figure 13 and Figure 14 The described hub-and-spoke topology architecture should be recognized that the number of spoke nodes can be extended to tens of thousands.

[0242] Turn Figure 15 The illustration depicts, according to certain embodiments, for the purpose of... Figure 14 An exemplary flowchart of a process performed by enabling communication in a spoke and hub architecture. Figure 15 The processing described herein can be implemented using software (e.g., code, instructions, programs), hardware, or a combination thereof, executed by one or more processing units (e.g., processors, cores) of a corresponding system. The software can be stored on a non-transitory storage medium (e.g., on a memory device). Figure 15 The methods presented and described below are intended to be illustrative and non-limiting. Although Figure 15Various processing steps are depicted as occurring in a particular order or sequence, but this is not intended to be limiting. In certain alternative embodiments, steps can be performed in a different order or some steps can be performed in parallel.

[0243] The process begins at step 1505, where a first compute virtual machine is provisioned in a first cloud environment. The first compute virtual machine is connected to a first customer virtual cloud network (VCN) in the first cloud environment and is connected to a hub. The first compute virtual machine is associated with a first plurality of virtual network interface cards (VNICs) including: (i) a first VNIC associated with a first network namespace and enabling communication between the first customer VCN and the first compute virtual machine, and (ii) a second VNIC associated with a second network namespace and enabling communication between the first compute virtual machine and the hub. Note that the first network namespace is different from the second network namespace.

[0244] The process then moves to step 1510, where a second compute virtual machine is provisioned in the first cloud environment. The second compute virtual machine is connected to a second customer VCN in the first cloud environment and is connected to the hub. The second compute virtual machine is associated with a second plurality of VNICs including: (i) a first VNIC associated with a third network namespace and enabling communication between the second customer VCN and the second compute virtual machine, and (ii) a second VNIC associated with the second network namespace and enabling communication between the second compute virtual machine and the hub.

[0245] Additionally, in step 1515, the process uses the first compute virtual machine and the hub to enable communication between the first customer VCN in the first cloud environment and a second cloud environment, and in step 1520, the process uses the second compute virtual machine and the hub to enable communication between the second customer VCN in the first cloud environment and the second cloud environment.

[0246] Examples of cloud infrastructure

[0247] As noted above, infrastructure as a service (IaaS) is a particular type of cloud computing. IaaS can be configured to provide virtualized computing resources over a public network (e.g., the Internet). In an IaaS model, a cloud computing provider can host infrastructure components (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer), etc.). In some cases, an IaaS provider can also provide various services to accompany these infrastructure components (e.g., billing, monitoring, logging, security, load balancing and clustering, etc.). Thus, because these services can be policy driven, an IaaS user can be able to implement policies to drive load balancing to maintain availability and performance of applications.

[0248] In some cases, an IaaS customer can access resources and services over a wide area network (WAN), such as the Internet, and can use the cloud provider's services to install the remaining elements of an application stack. For example, a user can log into an IaaS platform to create virtual machines (VMs), install an operating system (OS) on each VM, deploy middleware such as a database, create storage buckets for workloads and backups, and even install enterprise software into that VM. The customer can then use the provider's services to perform various functions, including balancing network traffic, troubleshooting applications, monitoring performance, managing disaster recovery, and so on.

[0249] In most cases, the cloud computing model will require the involvement of a cloud provider. A cloud provider can be, but is not necessarily, a third-party service that specializes in providing (e.g., provisioning, renting, selling) IaaS. An entity can also choose to deploy a private cloud, thereby becoming its own infrastructure service provider.

[0250] In some examples, IaaS deployment is the process of placing a new application or a new version of an application onto a prepared application server, and so on. It can also include the process of preparing a server (e.g., installing libraries, daemons, and so on). This is typically managed by the cloud provider, below the hypervisor layer (e.g., servers, storage, network hardware, and virtualization). Thus, the customer can be responsible for the process of (OS), middleware, and / or application deployment (e.g., onto a self-service virtual machine, which can be launched on-demand, and so on).

[0251] In some examples, IaaS provisioning can refer to the acquisition of computers or virtual hosts for use, even the installation of required libraries or services on them. In most cases, deployment does not include provisioning, and provisioning can need to be performed first.

[0252] In some cases, there are two different challenges to IaaS provisioning. First, there is the initial challenge of provisioning an initial set of infrastructure before anything is running. Second, there is the challenge of evolving an existing infrastructure (e.g., adding new services, changing services, removing services, and so on) once everything has been provisioned. In some cases, both challenges can be addressed by enabling a configuration that defines the infrastructure in a declarative manner. In other words, the infrastructure (e.g., what components are needed and how they interact) can be defined by one or more configuration files. Thus, the overall topology of the infrastructure (e.g., which resources depend on which resources and how they work together) can be described in a declarative manner. In some cases, once the topology is defined, a workflow can be generated that creates and / or manages the different components described in the configuration file.

[0253] In some examples, the infrastructure can have many interconnected elements. For example, there can be one or more virtual private clouds (VPCs) (e.g., potential on-demand pools of configurable and / or shareable computing resources), also referred to as core networks. In some examples, there can also be provisioned one or more security group rules to define how to set the security of the network and one or more virtual machines (VMs). Other infrastructure elements can also be provisioned, such as load balancers, databases, etc. As more and more infrastructure elements are desired and / or added, the infrastructure can evolve over time.

[0254] In some cases, continuous deployment techniques can be employed to enable deployment of infrastructure code across various virtual computing environments. Moreover, the described techniques can enable infrastructure management within these environments. In some examples, a service team can write code that is desired to be deployed to one or more, but typically many, different production environments (e.g., across various different geographic locations, sometimes spanning the entire world). However, in some examples, the infrastructure on which the code is to be deployed must first be set up. In some cases, provisioning can be done manually, resources can be provisioned with a provisioning tool, and / or once the infrastructure is provisioned, the code can be deployed with a deployment tool.

[0255] Figure 16is a block diagram 1600 illustrating an example pattern of an IaaS architecture, in accordance with at least one embodiment. A service operator 1602 can be communicatively coupled to a secure host tenancy 1604 that can include a virtual cloud network (VCN) 1606 and a secure host subnet 1608. In some examples, the service operator 1602 can use one or more client computing devices, which can be portable handheld devices (e.g., iPhone®, cellular phones, iPad®, computing tablets, personal digital assistants (PDAs) or wearable devices (e.g., Google Glass® head mounted display), running software such as Microsoft Windows Mobile® and / or various mobile operating systems (such as iOS, Windows Phone, Android, BlackBerry 8, Palm OS, etc.), and being Internet, e-mail, short message service (SMS), Blackberry® or other communication protocol enabled. Alternatively, a client computing device can be a general purpose personal computer including, for example, a personal computer and / or a laptop computer running various versions of Microsoft Windows®, Apple Macintosh® and / or Linux operating systems. A client computing device can be a workstation computer running any of a variety of commercially-available UNIX® or UNIX-like operating systems, including, without limitation, the variety of GNU / Linux operating systems, such as, for example, Google Chrome OS. Alternatively, or additionally, a client computing device can be any other electronic device, such as a thin-client computer, an Internet-enabled gaming system (e.g., a Microsoft Xbox game console with or without a Kinect® gesture input device), and / or a personal messaging device, capable of communicating over the network 1604, which can access the VCN 1606 and / or the Internet.

[0256] The VCN 1606 can include a local peering gateway (LPG) 1610 that can be communicatively coupled to a secure shell (SSH) VCN 1612 via the LPG 1610 contained in the SSH VCN 1612. The SSH VCN 1612 can include an SSH subnet 1614, and the SSH VCN 1612 can be communicatively coupled to a control plane VCN 1616 via the LPG 1610 contained in the control plane VCN 1616. Also, the SSH VCN 1612 can be communicatively coupled to a data plane VCN 1618 via the LPG 1610. The control plane VCN 1616 and the data plane VCN 1618 can be contained in a service tenancy 1619 that can be owned and / or operated by an IaaS provider.

[0257] The control plane VCN 1616 can include a control plane demilitarized zone (DMZ) tier 1620 that acts as a perimeter network, e.g., a portion of a company network between a company intranet and external networks. DMZ-based servers can assume limited responsibilities and help control security breaches. Further, the DMZ tier 1620 can include one or more load balancer (LB) subnets 1622, a control plane application tier 1624 that can include application subnet(s) 1626, a control plane data tier 1628 that can include database (DB) subnet(s) 1630, e.g., front-end DB subnet(s) and / or back-end DB subnet(s). The LB subnet(s) 1622 contained in the control plane DMZ tier 1620 can be communicatively coupled to the application subnet(s) 1626 contained in the control plane application tier 1624 and to an internet gateway 1634 that can be contained in the control plane VCN 1616, and the application subnet(s) 1626 can be communicatively coupled to the DB subnet(s) 1630 contained in the control plane data tier 1628 as well as to a service gateway 1636 and a network address translation (NAT) gateway 1638. The control plane VCN 1616 can include the service gateway 1636 and the NAT gateway 1638.

[0258] The control plane VCN 1616 can include a data plane mirror application tier 1640 that can include the application subnet(s) 1626. The application subnet(s) 1626 contained in the data plane mirror application tier 1640 can include a virtual network interface controller (VNIC) 1642 that can execute a compute instance 1644. The compute instance 1644 can communicatively couple the application subnet(s) 1626 of the data plane mirror application tier 1640 to the application subnet(s) 1626 that can be contained in a data plane application tier 1646.

[0259] The data plane VCN 1618 can include the data plane application tier 1646, a data plane DMZ tier 1648, and a data plane data tier 1650. The data plane DMZ tier 1648 can include the LB subnet(s) 1622 that can be communicatively coupled to the application subnet(s) 1626 of the data plane application tier 1646 and to the internet gateway 1634 of the data plane VCN 1618. The application subnet(s) 1626 can be communicatively coupled to the service gateway 1636 of the data plane VCN 1618 and to the NAT gateway 1638 of the data plane VCN 1618. The data plane data tier 1650 can also include the DB subnet(s) 1630 that can be communicatively coupled to the application subnet(s) 1626 of the data plane application tier 1646.

[0260] The internet gateway 1634 of the control plane VCN 1616 and the data plane VCN 1618 can be communicatively coupled to a metadata management service 1652, which can be communicatively coupled to the public internet 1654. The public internet 1654 can be communicatively coupled to the NAT gateway 1638 of the control plane VCN 1616 and the data plane VCN 1618. The service gateway 1636 of the control plane VCN 1616 and the data plane VCN 1618 can be communicatively coupled to cloud services 1656.

[0261] In some examples, the service gateway 1636 of the control plane VCN 1616 or the data plane VCN 1618 can make application programming interface (API) calls to the cloud services 1656 without going through the public internet 1654. The API calls from the service gateway 1636 to the cloud services 1656 can be one-way: the service gateway 1636 can make API calls to the cloud services 1656, and the cloud services 1656 can send requested data to the service gateway 1636. However, the cloud services 1656 can not initiate API calls to the service gateway 1636.

[0262] In some examples, the secure host tenancy 1604 can be directly connected to the service tenancy 1619, which can otherwise be isolated. The secure host subnet 1608 can communicate with the SSH subnet 1614 through the LPG 1610, which can enable two-way communication on otherwise isolated systems. Connecting the secure host subnet 1608 to the SSH subnet 1614 can enable the secure host subnet 1608 to access other entities within the service tenancy 1619.

[0263] The control plane VCN 1616 can allow users of the service tenancy 1619 to set or otherwise provision desired resources. The desired resources provisioned in the control plane VCN 1616 can be deployed or otherwise used in the data plane VCN 1618. In some examples, the control plane VCN 1616 can be isolated from the data plane VCN 1618, and a data plane mirror application tier 1640 of the control plane VCN 1616 can communicate with a data plane application tier 1646 of the data plane VCN 1618 via a VNIC 1642, which can be contained in the data plane mirror application tier 1640 and the data plane application tier 1646.

[0264] In some examples, a user or customer of the system can make a request, such as a create, read, update, or delete (CRUD) operation, through the public Internet 1654 that can communicate the request to the metadata management service 1652. The metadata management service 1652 can communicate the request to the control plane VCN 1616 through the Internet gateway 1634. The request can be received by the LB subnet(s) 1622 contained in the control plane DMZ tier 1620. The LB subnet(s) 1622 can determine that the request is valid, and in response to the determination, the LB subnet(s) 1622 can transmit the request to the application subnet(s) 1626 contained in the control plane application tier 1624. If the request is validated and requires a call to the public Internet 1654, the call to the public Internet 1654 can be transmitted to the NAT gateway 1638 that can make the call to the public Internet 1654. The memory that the request can desire to store can be stored in the DB subnet(s) 1630.

[0265] In some examples, the data plane mirror application tier 1640 can facilitate direct communication between the control plane VCN 1616 and the data plane VCN 1618. For example, it can be desirable to apply a change, update, or other appropriate modification to a configuration to a resource contained in the data plane VCN 1618. Via the VNIC 1642, the control plane VCN 1616 can directly communicate with the resource contained in the data plane VCN 1618 and, as a result, can perform the change, update, or other appropriate modification to the configuration.

[0266] In some embodiments, the control plane VCN 1616 and the data plane VCN 1618 can be contained in the service tenancy 1619. In this case, a user or customer of the system can not own or operate the control plane VCN 1616 or the data plane VCN 1618. Instead, an laaS provider can own or operate the control plane VCN 1616 and the data plane VCN 1618, both of which can be contained in the service tenancy 1619. This embodiment can enable isolation of networks that can prevent a user or customer from interacting with resources of other users or other customers. Also, this embodiment can allow a user or customer of the system to privately store databases without having to rely on the public Internet 1654 for storage that can not have a desired level of threat prevention.

[0267] In other embodiments, one or more LB subnets 1622 included in the control plane VCN 1616 may be configured to receive signals from the service gateway 1636. In this embodiment, the control plane VCN 1616 and the data plane VCN 1618 may be configured to be invoked by the IaaS provider's customers without invoking the public internet 1654. The IaaS provider's customers may expect this embodiment because the database(s) used by the customer can be controlled by the IaaS provider and can be stored on a service lease 1619, which may be isolated from the public internet 1654.

[0268] Figure 17 This is a block diagram 1700 illustrating another example pattern of an IaaS architecture according to at least one embodiment. Service operator 1702 (e.g., Figure 16 The service provider 1602) can communicatively couple to the secure host lease 1704 (e.g., Figure 16 Secure hosting lease 1604), the secure hosting lease 1704 may include a virtual cloud network (VCN) 1706 (e.g., Figure 16 VCN 1606) and Secure Host Subnet 1708 (e.g., Figure 16 The secure host subnet 1608). VCN 1706 may include a local peering gateway (LPG) 1710 (e.g., Figure 16 The LPG 1610), which can be communicatively coupled to the Secure Shell (SSH) VCN 1712 (e.g., via the LPG 1710 contained in the SSH VCN 1712) Figure 16 SSH VCN 1612). SSH VCN 1712 can include SSH subnet 1714 (e.g., Figure 16 SSH subnet 1614), and SSH VCN 1712 can be communicatively coupled to control plane VCN 1716 via LPG 1710 contained in control plane VCN 1716 (e.g., Figure 16 Control plane VCN 1616). Control plane VCN 1716 may be included in service lease 1719 (e.g., Figure 16 In the service lease 1619), and the data plane VCN 1718 (e.g., Figure 16 The data plane VCN 1618 may be included in a customer lease 1721 that may be owned or operated by the system’s users or customers.

[0269] Control plane VCN 1716 may include control plane DMZ layer 1720 (e.g., Figure 16 The control plane DMZ layer 1620), which may include one or more LB subnets 1722 (e.g., Figure 16 one or more LB subnets 1622), can include one or more application subnets 1726 (e.g., Figure 16 one or more application subnets 1626) of the control plane application tier 1724 (e.g., Figure 16 the control plane application tier 1624), can include one or more database (DB) subnets 1730 (e.g., similar to Figure 16 one or more DB subnets 1630) of the control plane data tier 1728 (e.g., Figure 16 the control plane data tier 1628). The one or more LB subnets 1722 contained in the control plane DMZ tier 1720 can be communicatively coupled to the one or more application subnets 1726 contained in the control plane application tier 1724 and to an internet gateway 1734 (e.g., Figure 16 the internet gateway 1634) that can be contained in the control plane VCN 1716, and the one or more application subnets 1726 can be communicatively coupled to the one or more DB subnets 1730 contained in the control plane data tier 1728 and to a service gateway 1736 (e.g., Figure 16 the service gateway) and a network address translation (NAT) gateway 1738 (e.g., Figure 16 the NAT gateway 1638). The control plane VCN 1716 can include the service gateway 1736 and the NAT gateway 1738.

[0270] The control plane VCN 1716 can include a data plane mirror application tier 1740 (e.g., Figure 16 the data plane mirror application tier 1640) that can include the one or more application subnets 1726. The one or more application subnets 1726 contained in the data plane mirror application tier 1740 can include virtual network interface controllers (VNICs) 1742 (e.g., VNICs of 1642) that can execute compute instances 1744 (e.g., similar to Figure 16 the compute instances 1644). The compute instances 1744 can facilitate communication between the one or more application subnets 1726 of the data plane mirror application tier 1740 and the one or more application subnets 1726 that can be contained in a data plane application tier 1746 (e.g., Figure 16 the data plane application tier 1646) via the VNICs 1742 contained in the data plane mirror application tier 1740 and the VNICs 1742 contained in the data plane application tier 1746.

[0271] The internet gateway 1734 contained in the control plane VCN 1716 can be communicatively coupled to a metadata management service 1752 (e.g., Figure 16 Metadata Management Service 1652), which can be communicatively coupled to a public Internet 1654 (e.g., the Internet 104 of FIG. 1). The public Internet 1654 can be communicatively coupled to a NAT gateway 1638 contained in the control plane VCN 1616. A service gateway 1636 contained in the control plane VCN 1616 can be communicatively coupled to cloud services 1656 (e.g., the cloud services 106 of FIG. 1). Figure 16 Figure 16

[0272] In some examples, the data plane VCN 1718 can be contained in a customer tenancy 1721. In this case, the IaaS provider can provide a control plane VCN 1716 for each customer, and the IaaS provider can set up a unique compute instance 1744 contained in the service tenancy 1719 for each customer. Each compute instance 1744 can allow for communication between the control plane VCN 1716 contained in the service tenancy 1719 and the data plane VCN 1718 contained in the customer tenancy 1721. The compute instance 1744 can allow resources provisioned in the control plane VCN 1716 contained in the service tenancy 1719 to be deployed or otherwise used in the data plane VCN 1718 contained in the customer tenancy 1721.

[0273] In other examples, a customer of the IaaS provider can have a database that exists in the customer tenancy 1721. In this example, the control plane VCN 1716 can include a data plane mirror application tier 1740, which can include application subnet(s) 1726. The data plane mirror application tier 1740 can reside in the data plane VCN 1718, but the data plane mirror application tier 1740 can not be in the data plane VCN 1718. That is, the data plane mirror application tier 1740 can have access to the customer tenancy 1721, but the data plane mirror application tier 1740 can not exist in the data plane VCN 1718 or be owned or operated by the customer of the IaaS provider. The data plane mirror application tier 1740 can be configured to make calls to the data plane VCN 1718, but can not be configured to make calls to any entity contained in the control plane VCN 1716. The customer can desire to deploy or otherwise use resources provisioned in the control plane VCN 1716 in the data plane VCN 1718, and the data plane mirror application tier 1740 can facilitate the customer’s desired deployment or other use of the resources.

[0274] ​​In some embodiments, a customer of the IaaS provider can apply filters to the data plane VCN 1718. In this embodiment, the customer can determine what the data plane VCN 1718 can access, and the customer can limit access to the public Internet 1754 from the data plane VCN 1718. The IaaS provider can not be able to apply filters or otherwise control access to any external network or database by the data plane VCN 1718. The customer applying filters and controls to the data plane VCN 1718 contained in the customer tenancy 1721 can help isolate the data plane VCN 1718 from other customers and the public Internet 1754.

[0275] In some embodiments, cloud services 1756 can be invoked by the service gateway 1736 to access services that can not be present on the public Internet 1754, the control plane VCN 1716, or the data plane VCN 1718. Connections between the cloud services 1756 and the control plane VCN 1716 or the data plane VCN 1718 can not be real-time or continuous. The cloud services 1756 can exist on a different network owned or operated by the IaaS provider. The cloud services 1756 can be configured to receive invocations from the service gateway 1736 and can be configured to not receive invocations from the public Internet 1754. Some cloud services 1756 can be isolated from other cloud services 1756, and the control plane VCN 1716 can be isolated from cloud services 1756 that can not be in the same region as the control plane VCN 1716. For example, the control plane VCN 1716 can be located in “Region 1,” and the cloud service “Deployment 16” can be located in Region 1 and “Region 2.” If the service gateway 1736 contained in the control plane VCN 1716 located in Region 1 makes an invocation to Deployment 16, the invocation can be transmitted to Deployment 16 in Region 1. In this example, the control plane VCN 1716 or Deployment 16 in Region 1 can not be communicatively coupled or otherwise in communication with Deployment 16 in Region 2.

[0276] Figure 18 is a block diagram 1800 illustrating another example pattern of an IaaS architecture, in accordance with at least one embodiment. A service operator 1802 (e.g., the service operator 1602 of Figure 16 can be communicatively coupled to a secure host tenancy 1804 (e.g., the secure host tenancy 1604 of Figure 16 can include a virtual cloud network (VCN) 1806 (e.g., the VCN 1606 of Figure 16 can include a secure host subnet 1808 (e.g., the secure host subnet 1608 of Figure 16 The VCN 1806 can include an LPG 1810 (e.g., the LPG 1610 of Figure 16 The LPG 1610), which can be communicatively coupled to the SSH VCN 1812 via the LPG 1810 included in the SSH VCN 1812 (e.g., Figure 16 SSH VCN 1612). SSH VCN 1812 can include SSH subnet 1814 (e.g., Figure 16 SSH subnet 1614), and SSH VCN 1812 can be communicatively coupled to control plane VCN 1816 via LPG 1810 contained in control plane VCN 1816 (e.g., Figure 16 The control plane VCN 1616) and coupled to the data plane VCN 1818 via the LPG 1810 contained in the data plane VCN 1818 (e.g., Figure 16 Data plane 1618). Control plane VCN 1816 and data plane VCN 1818 may be included in service lease 1819 (e.g., Figure 16 In the service rental (1619).

[0277] The control plane VCN 1816 may include one or more load balancer (LB) subnets 1822 (e.g., Figure 16 The control plane DMZ layer 1820 of (one or more) LB subnets 1622) (e.g., Figure 16 The control plane DMZ layer 1620 may include one or more application subnets 1826 (e.g., similar to...). Figure 16 The control plane application layer 1824 of (one or more) application subnets 1626 (e.g., Figure 16 The control plane application layer 1624), and may include (one or more) DB subnets 1830, and the control plane data layer 1828 (e.g., Figure 16 The control plane data layer 1628). One or more LB subnets 1822 contained in the control plane DMZ layer 1820 can be communicatively coupled to one or more application subnets 1826 contained in the control plane application layer 1824 and an Internet gateway 1834 that can be contained in the control plane VCN 1816 (e.g., Figure 16 Internet gateway 1634), and application subnet(s) 1826 can communicatively couple to DB subnet(s) 1830 contained in control plane data layer 1828 and service gateway 1836 (e.g., Figure 16 The service gateway) and Network Address Translation (NAT) gateway 1838 (e.g., Figure 16 (NAT gateway 1638). The control plane VCN 1816 may include the service gateway 1836 and the NAT gateway 1838.

[0278] Data plane VCN 1818 can include a data plane app tier 1846 (e.g., data plane app tier 1646 of FIG. 16), a data plane DMZ tier 1848 (e.g., data plane DMZ tier 1648 of FIG. 16), and a data plane data tier 1850 (e.g., data plane data tier 1650 of FIG. 16). Data plane DMZ tier 1848 can include trusted app subnet(s) 1860 and untrusted app subnet(s) 1862 that can be communicatively coupled to data plane app tier 1846 and LB subnet(s) 1822 that include internet gateway 1834 in data plane VCN 1818. Trusted app subnet(s) 1860 can be communicatively coupled to service gateway 1836 included in data plane VCN 1818, NAT gateway 1838 included in data plane VCN 1818, and DB subnet(s) 1830 included in data plane data tier 1850. Untrusted app subnet(s) 1862 can be communicatively coupled to service gateway 1836 included in data plane VCN 1818 and DB subnet(s) 1830 included in data plane data tier 1850. Data plane data tier 1850 can include DB subnet(s) 1830 that can be communicatively coupled to service gateway 1836 included in data plane VCN 1818. Figure 16 Figure 16 Figure 16 Data plane DMZ tier 1848 can include LB subnet(s) 1822 that can be communicatively coupled to internet gateway 1834 included in data plane VCN 1818. Data plane app tier 1846 can include trusted app subnet(s) 1860 that can be communicatively coupled to service gateway 1836 included in data plane VCN 1818, NAT gateway 1838 included in data plane VCN 1818, and DB subnet(s) 1830 included in data plane data tier 1850. Untrusted app subnet(s) 1862 can be communicatively coupled to service gateway 1836 included in data plane VCN 1818 and DB subnet(s) 1830 included in data plane data tier 1850. Data plane data tier 1850 can include DB subnet(s) 1830 that can be communicatively coupled to service gateway 1836 included in data plane VCN 1818.

[0279] Untrusted app subnet(s) 1862 can include one or more primary VNICs 1864(1)-(N) that can be communicatively coupled to tenant virtual machines (VMs) 1866(1)-(N). Each tenant VM 1866(1)-(N) can be communicatively coupled to a respective app subnet 1867(1)-(N) that can be included in a respective container egress VCN 1868(1)-(N) that can be included in a respective customer tenancy 1870(1)-(N). Respective secondary VNICs 1872(1)-(N) can facilitate communication between untrusted app subnet(s) 1862 included in data plane VCN 1818 and app subnets included in container egress VCNs 1868(1)-(N). Each container egress VCN 1868(1)-(N) can include NAT gateway 1838 that can be communicatively coupled to public internet 1854 (e.g., public internet 1654 of FIG. 16). Figure 16

[0280] ​​​An Internet gateway 1834 included in the control plane VCN 1816 and included in the data plane VCN 1818 can be communicatively coupled to a metadata management service 1852 (e.g., a metadata management system 1652 of Figure 16 The metadata management service 1852 can be communicatively coupled to a public Internet 1854. The public Internet 1854 can be communicatively coupled to a NAT gateway 1838 included in the control plane VCN 1816 and included in the data plane VCN 1818. A service gateway 1836 included in the control plane VCN 1816 and included in the data plane VCN 1818 can be communicatively coupled to a cloud service 1856.

[0281] In some embodiments, the data plane VCN 1818 can be integrated with customer tenancies 1870. In some cases, such as in cases where support for executing code can be desired, such integration can be useful or desirable for customers of the IaaS provider. Customers can provide code that can be disruptive, can communicate with other customer resources, or can otherwise cause undesirable effects to run. In response to this, the IaaS provider can determine whether to run the code provided by the customer to the IaaS provider.

[0282] In some examples, a customer of the IaaS provider can grant temporary network access to the IaaS provider and request functionality to be attached to the data plane tier application 1846. Code to run the functionality can be executed in VMs 1866(1)-(N), and the code can not be configured to run anywhere else on the data plane VCN 1818. Each VM 1866(1)-(N) can be connected to one customer tenancy 1870. Respective containers 1871(1)-(N) included in the VMs 1866(1)-(N) can be configured to run the code. In this case, there can be double isolation (e.g., the containers 1871(1)-(N) run the code, where the containers 1871(1)-(N) can be included in the VMs 1866(1)-(N) that can be included in the untrusted app subnet(s) 1862), which can help prevent incorrect or otherwise undesirable code from damaging the network of the IaaS provider or damaging the network of a different customer. The containers 1871(1)-(N) can be communicatively coupled to the customer tenancy 1870 and can be configured to transmit or receive data from the customer tenancy 1870. The containers 1871(1)-(N) can not be configured to transmit or receive data from any other entity in the data plane VCN 1818. Upon completion of running the code, the IaaS provider can terminate or otherwise dispose of the containers 1871(1)-(N).

[0283] In some embodiments, the trusted application subnet(s) 1860 can run code that can be owned or operated by the IaaS provider. In this embodiment, the trusted application subnet(s) 1860 can be communicatively coupled to the DB subnet(s) 1830 and configured to perform CRUD operations in the DB subnet(s) 1830. The untrusted application subnet(s) 1862 can be communicatively coupled to the DB subnet(s) 1830, but in this embodiment, the untrusted application subnet(s) can be configured to perform read operations in the DB subnet(s) 1830. Containers 1871(1)- (N) that can be contained in each customer's VM 1866(1)- (N) and that can run code from a customer can not be communicatively coupled to the DB subnet(s) 1830.

[0284] In other embodiments, the control plane VCN 1816 and the data plane VCN 1818 can not be directly communicatively coupled. In this embodiment, there can be no direct communication between the control plane VCN 1816 and the data plane VCN 1818. However, communication can occur indirectly by at least one method. The LPG 1810 can be established by the IaaS provider, which can facilitate communication between the control plane VCN 1816 and the data plane VCN 1818. In another example, the control plane VCN 1816 or the data plane VCN 1818 can invoke a cloud service 1856 via a service gateway 1836. For example, an invocation of a cloud service 1856 from the control plane VCN 1816 can include a request for a service that can be in communication with the data plane VCN 1818.

[0285] Figure 19 is a block diagram 1900 illustrating another example pattern of an IaaS architecture, in accordance with at least one embodiment. A service operator 1902 (e.g., the service operator 1602 of Figure 16 ) can be communicatively coupled to a secure host tenancy 1904 (e.g., the secure host tenancy 1604 of Figure 16 ), which can include a virtual cloud network (VCN) 1906 (e.g., the VCN 1606 of Figure 16 ) and a secure host subnet 1908 (e.g., the secure host subnet 1608 of Figure 16 ). The VCN 1906 can include an LPG 1910 (e.g., the LPG 1610 of Figure 16 ), which can be in communication with a SSH VCN 1912 (e.g., the SSH VCN 1612 of Figure 16The LPG 1910 in SSH VCN 1612 is communicatively coupled to SSH VCN 1912. SSH VCN 1912 may include SSH subnet 1914 (e.g., Figure 16 SSH subnet 1614), and SSH VCN 1912 can be communicatively coupled to control plane VCN 1916 via LPG 1910 contained in control plane VCN 1916 (e.g., Figure 16 The control plane VCN 1616) and coupled to the data plane VCN 1918 via the LPG 1910 contained in the data plane VCN 1918 (e.g., Figure 16 Data plane 1916. Control plane VCN 1916 and data plane VCN 1918 may be contained in service lease 1919 (e.g., Figure 16 In the service rental (1619).

[0286] The control plane VCN 1916 may include (one or more) LB subnets 1922 (e.g., Figure 16 The control plane DMZ layer 1920 of (one or more) LB subnets 1622 (e.g., Figure 16 The control plane DMZ layer 1620 may include one or more application subnets 1926 (e.g., Figure 16 The control plane application layer 1924 of (one or more) application subnets 1626 (e.g., Figure 16 The control plane application layer 1624) may include (one or more) DB subnets 1930 (e.g., Figure 18 The control plane data layer 1928 of (one or more) DB subnets 1830 (e.g., Figure 16 The control plane data layer 1628). One or more LB subnets 1922 contained in the control plane DMZ layer 1920 can be communicatively coupled to one or more application subnets 1926 contained in the control plane application layer 1924 and an Internet gateway 1934 that can be contained in the control plane VCN 1916 (e.g., Figure 16 Internet gateway 1634), and application subnet(s) 1926 can communicatively couple to DB subnet(s) 1930 contained in control plane data layer 1928 and service gateway 1936 (e.g., Figure 16 The service gateway) and Network Address Translation (NAT) gateway 1938 (e.g., Figure 16 (NAT gateway 1638). The control plane VCN 1916 may include the service gateway 1936 and the NAT gateway 1938.

[0287] Data plane VCN 1918 may include data plane application layer 1946 (e.g., Figure 16 Data plane application layer 1646), data plane DMZ layer 1948 (e.g., Figure 16 Data plane DMZ layer 1648), and data plane data layer 1950 (e.g., Figure 16 The data plane data layer 1650). The data plane DMZ layer 1948 may include one or more trusted application subnets 1960 that can be communicatively coupled to the data plane application layer 1946 (e.g., Figure 18 (one or more) trusted application subnets 1860) and (one or more) untrusted application subnets 1962 (e.g., Figure 18 The data plane VCN 1918 may include one or more untrusted application subnets 1862 and one or more LB subnets 1922 of Internet gateway 1934. One or more trusted application subnets 1960 may communicatively couple to service gateway 1936, NAT gateway 1938, and DB subnets 1930 in data plane VCN 1950. One or more untrusted application subnets 1962 may communicatively couple to service gateway 1936 and DB subnets 1930 in data plane VCN 1918. Data plane VCN 1950 may include one or more DB subnets 1930 that may communicatively couple to service gateway 1936 in data plane VCN 1918.

[0288] One or more untrusted application subnets 1962 may include a primary VNIC 1964(1)-(N) communicatively coupled to tenant virtual machines (VMs) 1966(1)-(N) residing within one or more untrusted application subnets 1962. Each tenant VM 1966(1)-(N) may run code in a corresponding container 1967(1)-(N) and is communicatively coupled to an application subnet 1926 that may be contained in a data plane application layer 1946 contained in a container egress VCN 1968. A corresponding secondary VNIC 1972(1)-(N) may facilitate communication between one or more untrusted application subnets 1962 contained in a data plane VCN 1918 and the application subnet contained in a container egress VCN 1968. The container egress VCN may include a primary VNIC 1964(1)-(N) communicatively coupled to the public Internet 1954 (e.g., Figure 16 The public internet (1654) uses a NAT gateway (1938).

[0289] Internet gateway 1934, contained in control plane VCN 1916 and data plane VCN 1918, can communicatively couple to metadata management service 1952 (e.g., Figure 16 A metadata management system 1652, which is communicatively coupled to the public internet 1954, is also communicatively coupled to a NAT gateway 1938 contained in a control plane VCN 1916 and a data plane VCN 1918. A service gateway 1936 contained in both the control plane VCN 1916 and the data plane VCN 1918 is communicatively coupled to a cloud service 1956.

[0290] In some examples, Figure 19 The architecture shown in the block diagram 1900 can be considered as... Figure 18 This is an exception to the pattern shown in the architecture diagram 1800, and this pattern may be what the IaaS provider's customers would expect if the IaaS provider cannot communicate directly with the customer (e.g., in a disconnected region). The customer can access in real time the corresponding container 1967(1)-(N) contained in each customer's VM 1966(1)-(N). Container 1967(1)-(N) can be configured to invoke the corresponding auxiliary VNIC 1972(1)-(N) contained in one or more application subnets 1926 of the data plane application layer 1946, which may be contained in the container egress VCN 1968. The auxiliary VNIC 1972(1)-(N) can transmit the call to the NAT gateway 1938, which can then transmit the call to the public internet 1954. In this example, containers 1967(1)-(N), which can be accessed by clients in real time, can be isolated from the control plane VCN 1916 and from other entities contained in the data plane VCN 1918. Containers 1967(1)-(N) can also be isolated from resources from other clients.

[0291] In other examples, a customer can use containers 1967(1)-(N) to invoke cloud service 1956. In this example, the customer can run code in containers 1967(1)-(N) that requests a service from cloud service 1956. Containers 1967(1)-(N) can transmit the request to secondary VNICs 1972(1)-(N), which can transmit the request to a NAT gateway, which can transmit the request to public internet 1954. Public internet 1954 can transmit the request to LB subnet(s) 1922 contained in control plane VCN 1916 via internet gateway 1934. In response to determining that the request is valid, the LB subnet(s) can transmit the request to application subnet(s) 1926, which can transmit the request to cloud service 1956 via service gateway 1936.

[0292] It should be appreciated that the IaaS architectures 1600, 1700, 1800, 1900 depicted in the figures can have other components than those depicted. In addition, the embodiments shown in the figures are merely some examples of cloud infrastructure systems with which embodiments of the present disclosure can be practiced. In some other embodiments, an IaaS system can have more or fewer components than shown in the figures, can combine two or more components, or can have a different configuration or arrangement of components.

[0293] In certain embodiments, the IaaS systems described herein can include an application suite, middleware, and database service products delivered to customers in a self-service, subscription-based, elastically scalable, reliable, highly available, and secure manner. An example of such an IaaS system is Oracle Cloud Infrastructure (OCI) offered by the present assignee.

[0294] Figure 20 An example computer system 2000 in which various embodiments can be implemented is illustrated in FIG. 20. System 2000 can be used to implement any of the computer systems described above. As shown in FIG. 20, computer system 2000 includes a processing unit 2004 that communicates with a number of peripheral subsystems via a bus subsystem 2002. These peripheral subsystems can include a processing acceleration unit 2006, an I / O subsystem 2008, a storage subsystem 2018 and a communications subsystem 2024. Storage subsystem 2018 includes tangible computer-readable storage media 2022 and a system memory 2010.

[0295] Bus subsystem 2002 provides a mechanism for letting the various components and subsystems of computer system 2000 communicate with each other as intended. Although bus subsystem 2002 is shown schematically as a single bus, alternative embodiments of the bus subsystem can utilize multiple buses. Bus subsystem 2002 can be any of several types of bus structures including a memory bus or memory controller, a peripheral bus or external bus, and a local bus using any of a variety of bus architectures. For example, such architectures can include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus, which can be implemented as Mezzanine bus manufactured to the IEEE P1386.1 standard.

[0296] Processing unit 2004, which can be implemented as one or more integrated circuits (e.g., a conventional microprocessor or microcontroller), controls the operation of computer system 2000. One or more processors can be included in processing unit 2004. These processors can include single or multiple threaded processors. In certain embodiments, processing unit 2004 can be implemented as one or more independent processing units 2032 and / or 2034, with single or multiple threaded processors included in each processing unit. In other embodiments, processing unit 2004 can also be implemented as a quad-core processing unit formed by integrating two dual-core processors into a single chip.

[0297] In various embodiments, processing unit 2004 can execute a variety of programs in response to program code and can maintain multiple concurrently executing programs or processes. At any given time, some or all of the program code to be executed can be resident in processor(s) 2004 and / or in storage subsystem 2018. Through suitable programming, processor(s) 2004 can provide various functionalities described above. Computer system 2000 can additionally include a processing acceleration unit 2006, which can include a digital signal processor (DSP), a special-purpose processor, and / or the like.

[0298] I / O subsystem 2008 can include user interface input devices and user interface output devices. User interface input devices can include, for example, a keyboard, pointing devices such as a mouse or trackball, touchpads, touch screens such as Apple® iPad® or Microsoft® Surface® touch screens, wheels, click wheels, dials, buttons, switches, key pads, keyboards, microphones for voice inputs, and other types of input devices. User interface input devices can include, for example, motion sensing and / or gesture recognition devices such as the Microsoft Kinect® motion sensor that enables users to control and interact with an input device such as the Microsoft Xbox® 360 game controller using a natural user interface. User interface input devices can also include eye gesture recognition devices such as Google Glass® blink detector that detects eye activity (for example, “blinking” of the eye) from users in a manner that mimics other input devices (for example, pointing devices or mouse), and

[0299] User interface input devices can also include, without limitation, three dimensional (3D) mice, joysticks or pointing sticks, gamepads and graphic tablets, and audio / visual devices such as speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, barcode reader 3D scanners, 3D printers, laser rangefinders, and eye gaze tracking devices. Additionally, user interface input devices can include, for example, medical imaging input devices such as computerized tomography, magnetic resonance imaging, position emission tomography devices, medical ultrasonic devices. User interface input devices can also include, for example, audio input devices such as MIDI keyboards, digital musical instruments and the like.

[0300] User interface output devices can include, for example, a display subsystem, indicator lights, or non-visual displays such as audio output devices, etc. The display subsystem can be a cathode ray tube (CRT), a flat-panel device such as that using a liquid crystal display (LCD) or plasma display, a projection device, a touch screen, or any other display device suitable for use in a computing environment. In general, use of the term “output device” is intended to include all possible types of devices and mechanisms for outputting information from the computer system 2000 to a user or other computer. For example, user interface output devices can include, without limitation, a variety of display devices that visually convey text, graphics, and audio / video information such as monitors, printers, speakers, headphones, automotive navigation systems, plotters, voice output devices, and modems.

[0301] Computer system 2000 can include a storage subsystem 2018 that includes software elements, shown as being currently located within a system memory 2010. System memory 2010 can store program instructions that can be loaded onto processing unit 2004 and executed by processing unit 2004. System memory 2010 can also store data that is used in the execution of these programs.

[0302] Depending on the configuration and type of computer system 2000, system memory 2010 can be volatile (such as random access memory (RAM)) and / or non-volatile (such as read-only memory (ROM), flash memory, etc.). RAM typically contains data and / or program modules that are immediately accessible to and / or presently being operated and executed by processing unit 2004. In some implementations, system memory 2010 can include multiple different types of memory, such as static random access memory (SRAM) or dynamic random access memory (DRAM). In some implementations, a basic input / output system (BIOS), containing the basic routines that help to transfer information between elements within computer system 2000, such as during start-up, can typically be stored in ROM. By way of example, and not limitation, system memory 2010 also illustrates application programs 2012, which can include client applications, Web browsers, mid-tier applications, relational database management systems (RDBMS), etc., program data 2014, and an operating system 2016. As examples, operating system 2016 can include various versions of Microsoft Windows®, Apple Macintosh® and / or Linux operating systems, UNIX® or UNIX-like operating systems, including but not limited to the GNU / Linux® operating system, the Google Chrome® OS, and / or a

[0303] Storage subsystem 2018 can also provide a repository for storing data used in accordance with the present disclosure. Software (programs, code modules, instructions) that when executed by a processor provide the functionality described above can be stored in storage subsystem 2018. These software modules or instructions can be executed by processing unit 2004. Storage subsystem 2018 can also provide a repository for storing data used in accordance with the present disclosure.

[0304] Storage subsystem 2000 can also include a computer-readable storage media reader 2020 that can further be connected to computer-readable storage media 2022. Together and, optionally, in combination with the system memory 2010, the computer-readable storage media 2022 can comprehensively represent remote, local, fixed, and / or removable storage devices plus storage media for temporarily and / or more permanently storing computer-readable information

[0305] The computer-readable storage media 2022, including the code or portions of the code, can also include any appropriate media known or used in the art, including storage media and communication media, such as but not limited to, volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage and / or transmission of information such as computer readable instructions, data structures, program modules, and other data including any appropriate media known or used in the art. This can include tangible computer-readable storage media such as RAM, ROM, electronically erasable programmable ROM (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disks (DVD), or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or other tangible computer-readable media. This can also include non-tangible computer-readable media, such as data signals, data transmissions, or any other medium which can be used to transfer the desired information and which can be accessed by computing system 2000.

[0306] By way of example, computer-readable storage media 2022 can include a hard disk drive that reads from or writes to non-removable, non- volatile magnetic media; a magnetic disk drive that reads from or writes to a removable, non-volatile magnetic disk; and an optical disk drive that reads from or writes to a removable, non- volatile optical disk such as a CD ROM, DVD, or other optical media. Computer-readable storage media 2022 can include, but is not limited to, Zip® drives, flash memory cards, universal serial bus (USB) flash drives, secure digital (SD) cards, DVD disks, digital video tape, and the like. Computer-readable storage media 2022 can also include, solid-state drives (SSDs), such as flash memory based SSDs, enterprise flash drives, solid state ROM, and the like, SSDs using a combination of DRAM and flash memory based storage, and hybrid SSDs using a combination of DRAM, flash memory based storage, and one or more other types of storage depending on

[0307] The communications subsystem 2024 provides an interface to other computer systems and networks. The communications subsystem 2024 serves as an interface for receiving data from and transmitting data to other systems from the computer system 2000. For example, the communications subsystem 2024 can enable the computer system 2000 to connect to one or more devices via the Internet. In some embodiments, the communications subsystem 2024 can include a radio frequency (RF) transceiver component (e.g., using cellular telephone technology, advanced data network technology, such as 3G, 4G or EDGE (enhanced data rates for global evolution), Wi-Fi (IEEE 802.11 family standards), or other mobile communication technologies, or any combination thereof), a Global Positioning System (GPS) receiver component, and / or other components, for accessing a wireless voice and / or data network. In some embodiments, the communications subsystem 2024 can provide wired network connectivity (e.g., Ethernet) in addition to or instead of a wireless interface.

[0308] In some embodiments, the communications subsystem 2024 can also receive input, including structured and / or unstructured data feeds 2026, event streams 2028, event updates 2030, and the like from one or more users of the computer system 2000.

[0309] For example, the communications subsystem 2024 can be configured to receive data feeds 2026 in real-time from users of social networks and / or other communication services, such as Twitter® feeds, Facebook® updates, web feeds such as Rich Site Summary (RSS) feeds, and / or real-time updates from one or more third party information sources.

[0310] Further, the communications subsystem 2024 can also be configured to receive data in the form of continuous data streams, which can include event streams 2028 of real-time events and / or event updates 2030, that can be continuous or unbounded in nature with no explicit end. Examples of applications that generate continuous data can include, for example, sensor data applications, financial tickers, network performance measuring tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, and so on.

[0311] The communications subsystem 2024 can also be configured to output structured and / or unstructured data feeds 2026, event streams 2028, event updates 2030, and the like to one or more databases that can be in communication with one or more stream computing devices coupled to computer system 2000.

[0312] Computer systems 2000 can be of various types, including handheld portable devices (e.g., iPhone® cellular phones, iPad® computing tablets, PDAs), wearable devices (e.g., Google® Glass head-mounted displays), PCs, workstations, mainframes, information stations, server racks, or any other data processing systems.

[0313] Due to the ever-evolving nature of computers and networks, the description of the computer system 2000 depicted in the figures is merely a concrete example. Many other configurations with more or fewer components than the system depicted in the figures are possible. For example, custom hardware may be used and / or specific elements may be implemented using hardware, firmware, software (including applets), or a combination thereof. Additionally, connections to other computing devices, such as network input / output devices, may also be employed. Based on the disclosure and teachings provided herein, those skilled in the art will recognize other ways and / or methods for implementing the various embodiments.

[0314] While specific embodiments have been described, various modifications, alterations, alternative constructions, and equivalents are also included within the scope of this disclosure. The embodiments are not limited to operation within certain specific data processing environments, but can be freely operated within multiple data processing environments. Furthermore, although the embodiments have been described using a specific series of transactions and steps, those skilled in the art will understand that the scope of this disclosure is not limited to the described series of transactions and steps. Various features and aspects of the above embodiments can be used individually or in combination.

[0315] Furthermore, while embodiments have been described using specific combinations of hardware and software, it should be recognized that other combinations of hardware and software are also within the scope of this disclosure. Embodiments may be implemented using only hardware, or only software, or a combination thereof. The various processes described herein can be implemented in any combination on the same processor or on different processors. Accordingly, where a component or module is described as being configured to perform certain operations, such configuration can be accomplished, for example, by designing electronic circuits to perform operations, by programming programmable electronic circuits (such as microprocessors), or any combination thereof. Processes may communicate using a variety of technologies, including but not limited to conventional technologies for inter-process communication, and different pairs of processes may use different technologies, or the same pair of processes may use different technologies at different times.

[0316] Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense. It is certainly apparent, however, that additions, subtractions, deletions, and other modifications and changes can be made to the described embodiments without departing from the broader spirit and scope of the claims which are to be afforded a full scope of equivalents. Thus, although specific embodiments have been described, these are not intended to be limiting. Various modifications and equivalent forms are within the scope of claims.

[0317] The use of the terms "a" and "one" and "the" and "at least one" and "one or more" and "and / or" in the context of describing the disclosed embodiments (especially in the context of the following claims) are to be construed to cover both the singular and the plural, unless otherwise indicated herein or clearly contradicted by context. The use of the term "including," "having," "containing," or "comprising" is meant to be inclusive, and to mean "including but not limited to," unless otherwise indicated or unless clearly contradicted by context. The use of the term "connected" is meant to include "directly connected," "indirectly connected" through one or more intervening components, or "connected together." Unless otherwise indicated herein, the

[0318] Disjunctive language such as the phrase "at least one of X, Y, or Z," unless specifically stated otherwise, is otherwise understood within the context as used in general to convey that an item, term, etc. can be either X, Y, or Z, or any combination thereof (e.g., X, Y, and / or Z). Thus, such disjunctive language is not generally intended to be limiting, unless specifically stated otherwise. It is further understood that the use of the conjunctive "and" in the context of describing items, terms, etc. is not intended to be limiting, unless specifically stated otherwise.

[0319] Preferred embodiments of the present disclosure are described herein, including the best mode known to the inventors for practicing the present disclosure. Variations of those preferred embodiments can become apparent to those of ordinary skill in the art upon reading the foregoing description. The inventors expect skilled artisans to employ such variations as appropriate, and the present disclosure is intended to cover such

[0320] All references, including publications, patent applications, and patents, cited

[0321] In the foregoing specification, aspects of the disclosure are described with reference to specific embodiments thereof, but those skilled in the art will recognize that the disclosure is not limited thereto. Various features and aspects of the above-described disclosure can be used individually or jointly. In addition, embodiments can be used in any number of environments and applications beyond the type described herein, without departing from the broader spirit and scope of the specification. Accordingly, the specification and drawings are to be regarded as illustrative in nature and not as restrictive.

Claims

1. A method comprising: receiving, by a multi-cloud infrastructure included in a first cloud environment, a request to create a network link between a first customer virtual network in the first cloud environment and a second customer virtual network in a second cloud environment; deploying, in the second cloud environment, a first link-enabled virtual network, the first link-enabled virtual network communicatively coupled to a first data plane hub virtual network, a first control plane hub virtual network, and the second customer virtual network, wherein the first data plane hub virtual network and the first control plane hub virtual network are deployed in the second cloud environment; and enabling traffic to be communicated from the second customer virtual network in the second cloud environment to the first customer virtual network in the first cloud environment via the first link-enabled virtual network, wherein the first link-enabled virtual network receives network configuration information from the first control plane hub virtual network and transmits the traffic to the first cloud environment via the first data plane hub virtual network based on the network configuration information.

2. The method of claim 1, wherein the first link-enabled virtual network is associated with a first virtual network interface card (VNIC) that communicatively couples the first link-enabled virtual network to the first data plane hub virtual network, a second VNIC that communicatively couples the first link-enabled virtual network to the first control plane hub virtual network, and a third VNIC that communicatively couples the first link-enabled virtual network to the second customer virtual network.

3. The method of claim 1, wherein the first link-enabled virtual network includes a first pair of virtual network adapters, each of the first pair of virtual network adapters configured to encapsulate traffic received from the second customer virtual network to generate encapsulated traffic.

4. The method of claim 1, further comprising: encapsulating, by the first link-enabled virtual network in the second cloud environment, traffic received from the second customer virtual network based on the network configuration information to generate encapsulated traffic; and forwarding, by the first link-enabled virtual network in the second cloud environment, the encapsulated traffic to the first data plane hub virtual network.

5. The method of claim 1, further comprising: deploying, in the first cloud environment, a second control plane hub virtual network, the second control plane hub virtual network including one or more distribution service nodes, wherein the second control plane hub virtual network is communicatively coupled to the first control plane hub virtual network in the second cloud environment via a public interconnection link.

6. The method of claim 5, wherein at least one of the one or more distribution service nodes is configured to transmit network configuration information to the first control plane hub virtual network in the second cloud environment, wherein the network configuration information is forwarded to the first link-enabled virtual network via the first control plane hub virtual network, and wherein the network configuration information includes at least (i) tunnel encapsulation-decapsulation parameters, and health information of a second link-enabled virtual network deployed in the first cloud environment. ​ 7. The method of claim 1, wherein the second cloud environment comprises a plurality of customer tenancies, and wherein for each customer tenancy included in the plurality of customer tenancies: (i) a first link-enabled virtual network is created in the second cloud environment, and (ii) a second link-enabled virtual network is created in the first cloud environment.

8. The method of claim 1, further comprising: deploying a second link-enabled virtual network in the first cloud environment, the second link-enabled virtual network communicatively coupled to the first customer virtual network, the second link-enabled virtual network comprising a second pair of virtual network adapters, each virtual network adapter of the second pair of virtual network adapters configured to de-encapsulate encapsulated traffic received from the first link-enabled virtual network included in the second cloud environment.

9. The method of claim 8, further comprising: de-encapsulating, by the second link-enabled virtual network in the first cloud environment, encapsulated traffic received from the first data plane hub virtual network included in the second cloud environment to generate de-encapsulated traffic, and transmitting, by the second link-enabled virtual network in the first cloud environment, the de-encapsulated traffic to the first customer virtual network in the first cloud environment.

10. The method of claim 9, wherein the second link-enabled virtual network in the first cloud environment receives encapsulated traffic from a second data plane hub virtual network included in the first cloud environment, the second data plane hub virtual network communicatively coupled to the first data plane hub virtual network via a private high-bandwidth interconnect coupling the second cloud environment to the first cloud environment.

11. The method of claim 8, wherein each of the first link-enabled virtual network and the second link-enabled virtual network is assigned a unique classless inter-domain routing IP address.

12. The method of claim 1, wherein the multi-cloud infrastructure comprises a first portion deployed in the first cloud environment and a second portion deployed in the second cloud environment, each of the first portion and the second portion of the multi-cloud infrastructure controlled by a first cloud service provider of the first cloud environment, the first cloud service provider different from a second cloud service provider of the second cloud environment.

13. One or more computer-readable non-transitory media storing computer-executable instructions that, when executed by one or more processors, cause: receiving, by a multi-cloud infrastructure included in a first cloud environment, a request to create a network link between a first customer virtual network in the first cloud environment and a second customer virtual network in a second cloud environment; deploying a first link-enabled virtual network in the second cloud environment, the first link-enabled virtual network communicatively coupled to a first data plane hub virtual network, a first control plane hub virtual network, and the second customer virtual network, wherein the first data plane hub virtual network and the first control plane hub virtual network are deployed in the second cloud environment; and and to transmit the traffic to the first cloud environment via a first data plane hub virtual network based on the network configuration information.

14. The one or more computer-readable non-transitory media storing computer-executable instructions of claim 13, wherein the first link-enabled virtual network is associated with a first virtual network interface card (VNIC) that communicatively couples the first link-enabled virtual network to the first data plane hub virtual network, a second VNIC that communicatively couples the first link-enabled virtual network to the first control plane hub virtual network, and a third VNIC that communicatively couples the first link-enabled virtual network to the second customer virtual network.

15. The one or more computer-readable non-transitory media storing computer-executable instructions of claim 13, wherein the first link-enabled virtual network includes a first pair of virtual network adapters, each of the first pair of virtual network adapters configured to encapsulate traffic received from the second customer virtual network to generate encapsulated traffic.

16. The one or more computer-readable non-transitory media storing computer-executable instructions of claim 13, further comprising instructions that, when executed by one or more processors, cause performance of: encapsulating, by the first link-enabled virtual network in the second cloud environment, traffic received from the second customer virtual network based on the network configuration information to generate encapsulated traffic; and forwarding, by the first link-enabled virtual network in the second cloud environment, the encapsulated traffic to the first data plane hub virtual network.

17. The one or more computer-readable non-transitory media storing computer-executable instructions of claim 13, further comprising instructions that, when executed by one or more processors, cause performance of: deploying a second control plane hub virtual network in the first cloud environment, the second control plane hub virtual network including one or more distribution service nodes, wherein the second control plane hub virtual network is communicatively coupled to the first control plane hub virtual network in the second cloud environment via a public interconnection link.

18. The one or more computer-readable non-transitory media storing computer-executable instructions of claim 17, wherein at least one of the one or more distribution service nodes is configured to transmit network configuration information to the first control plane hub virtual network in the second cloud environment, wherein the network configuration information is forwarded to the first link-enabled virtual network via the first control plane hub virtual network, and wherein the network configuration information includes at least (i) tunnel encapsulation-decapsulation parameters, and health information of a second link-enabled virtual network deployed in the first cloud environment.

19. The one or more computer-readable non-transitory media storing computer- executable instructions of claim 13, wherein the second cloud environment comprises a plurality of customer tenancies, and wherein for each customer tenancy included in the plurality of customer tenancies: (i) a first link-enabled virtual network is created in the second cloud environment, and (ii) a second link-enabled virtual network is created in the first cloud environment.

20. A computing device comprising: one or more processors; and memory comprising instructions that, when executed with the one or more processors, cause the computing device to at least: receive, by a multi-cloud infrastructure included in a first cloud environment, a request to create a network link between a first customer virtual network in the first cloud environment and a second customer virtual network in a second cloud environment; deploy, in the second cloud environment, a first link-enabled virtual network communicatively coupled to a first data plane hub virtual network, a first control plane hub virtual network, and the second customer virtual network, wherein the first data plane hub virtual network and the first control plane hub virtual network are deployed in the second cloud environment; and enable traffic to be communicated from the second customer virtual network in the second cloud environment to the first customer virtual network in the first cloud environment via the first link-enabled virtual network, wherein the first link-enabled virtual network receives network configuration information from the first control plane hub virtual network and transmits the traffic to the first cloud environment via the first data plane hub virtual network based on the network configuration information. ​

Images