Configuring network links to establish communication between different cloud environments

Abstract

Techniques are described for creating a network link between a first virtual network in a first cloud environment and a second virtual network in a second cloud environment. The first virtual network in the first cloud environment is created to enable a user associated with a customer's tenancy in the second cloud environment to access one or more services provided in the first cloud environment. The network link is created based on one or more link-enabled virtual networks deployed in the first cloud environment and the second cloud environment.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a nonprovisional application of, and claims the benefit of, each of the following provisional applications, the entire contents of each of which are incorporated herein by reference for all purposes: (1) U.S. Provisional Patent Application No. 63 / 306,007, filed February 2, 2022; (2) U.S. Provisional Patent Application No. 63 / 306,918, filed February 4, 2022; (3) U.S. Provisional Patent Application No. 63 / 321,614, filed March 18, 2022; (4) U.S. Provisional Patent Application No. 63 / 333,965, filed April 22, 2022; (5) U.S. Provisional Patent Application No. 63 / 336,811, filed April 29, 2022; (6) U.S. Provisional Patent Application No. 63 / 339,297, filed May 6, 2022; (7) U.S. Provisional Patent Application No. 63 / 346,004, filed May 26, 2022; (8) U.S. Provisional Patent Application No. 63 / 389,305, filed July 14, 2022; (9) U.S. Provisional Patent Application No. 63 / 389,145, filed on July 14, 2022; and (10) U.S. Provisional Patent Application No. 63 / 380,326, filed October 20, 2022.

[0002] Field The present disclosure relates to cloud architectures, and more particularly to techniques for linking two cloud environments so that users of one cloud environment can use services offered by the other cloud environment. [Background technology]

[0003] background The past few years have seen a dramatic increase in the adoption of cloud services, and this trend is only increasing. Various cloud environments are offered by different cloud service providers (CSPs), with each cloud environment offering a set of one or more cloud services. The set of cloud services offered by a cloud environment may include one or more different types of services, including, but not limited to, Software-as-a-Service (SaaS) services, Infrastructure-as-a-Service (IaaS) services, Platform-as-a-Service (PaaS) services, etc.

[0004] Although a variety of cloud environments are currently available, each cloud environment provides a closed ecosystem to subscribing customers. As a result, customers of a cloud environment are limited to using the services offered by that cloud environment. There is no easy way for a customer subscribing to a cloud environment offered by a CSP to use, via that cloud environment, services offered in a different cloud environment offered by a different CSP. The embodiments described herein address these and other issues. The embodiments described herein address these and other issues. Summary of the Invention

[0005] overview The present disclosure relates generally to improved cloud architectures, and more particularly to techniques for linking two clouds such that users of one cloud environment can use services provided by another, different cloud environment. Various embodiments are described herein, including methods, systems, non-transitory computer-readable storage media that store programs, code, or instructions executable by one or more processors, and the like. Some embodiments may be implemented by using a computer program product that includes computer programs / instructions that, when executed by a processor, cause the processor to perform any of the methods described in the present disclosure.

[0006] Embodiments of the present disclosure provide a multi-cloud control plane (MCCP) framework that provisions functions for delivering services of a particular cloud network (e.g., Oracle Cloud Infrastructure (OCI)) to users on other clouds (e.g., in Microsoft Azure). The MCCP framework allows users (of other cloud environments) to access services of the cloud environment (e.g., PaaS services) while providing a user experience as close as possible to the user's native cloud environment user experience. A key challenge of MCCP is to enable customers to experience the full data plane capabilities of services in external clouds.

[0007] One embodiment of the present disclosure is directed to a method including receiving, by a multi-cloud infrastructure included in a first cloud environment, a request to create a network link between a first virtual network in the first cloud environment and a second virtual network in a second cloud environment, where the first virtual network in the first cloud environment has been pre-created to enable a user associated with a customer's tenancy in the second cloud environment to access one or more services provided in the first cloud environment; and creating a network link between the first virtual network and the second virtual network, where the creating includes receiving a request from the second virtual network by a first link-enabled virtual network in the second cloud environment. and forwarding, by a second link-enabled virtual network in the second cloud environment, the encapsulated traffic received from the first link-enabled virtual network to generate encapsulated traffic; forwarding, by a second link-enabled virtual network in the second cloud environment, the encapsulated traffic received from the first link-enabled virtual network to a hub virtual network included in the second cloud environment; decapsulating, by a third link-enabled virtual network in the first cloud environment, the encapsulated traffic received from the hub virtual network included in the second cloud environment to generate decapsulated traffic; and transmitting, by the third link-enabled virtual network in the first cloud environment, the decapsulated traffic to the first virtual network in the first cloud environment.

[0008] Aspects of the present disclosure provide a computing device comprising one or more data processors and a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the computing device to perform some or all of one or more of the methods disclosed herein.

[0009] Another aspect of the disclosure provides a computer program product, tangibly embodied in a non-transitory machine-readable storage medium, that includes instructions configured to cause one or more data processors to perform some or all of one or more of the methods disclosed herein.

[0010] The foregoing, together with other features and embodiments, will become more apparent when considered in conjunction with the following specification, claims, and accompanying drawings.

[0011] The features, embodiments, and advantages of the present disclosure will be better understood when the following detailed description is read in conjunction with the accompanying drawings. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a high-level diagram of a distributed environment illustrating a virtual cloud network or overlay cloud network hosted by a cloud service provider infrastructure, according to an embodiment. [Diagram 2] FIG. 2 illustrates a simplified architectural diagram of physical components in a physical network within CSPI, according to one embodiment. [Diagram 3] FIG. 2 illustrates an exemplary arrangement within CSPI in which a host machine is connected to multiple network virtualization devices (NVDs), according to one embodiment. [Figure 4] A diagram illustrating connections between host machines and NVDs to achieve I / O virtualization to support multi-tenancy functionality in one embodiment. [Diagram 5] FIG. 2 illustrates a simplified block diagram of a physical network provided by CSPI, according to one embodiment. [Figure 6]1 is a simplified high-level diagram of a distributed environment including multiple cloud environments offered by different cloud service providers (CSPs), according to one embodiment, where the cloud environments include a particular cloud environment that provides specialized infrastructure that enables one or more cloud services offered by that particular cloud environment to be used by customers of the other cloud environments. [Figure 7] FIG. 1 illustrates an example high-level architecture of a multi-cloud control plane (MCCP), according to some embodiments. [Figure 8A] FIG. 1 illustrates an example process for linking two user accounts in different cloud environments according to some embodiments. [Figure 8B] FIG. 1 illustrates an example process for linking two user accounts in different cloud environments according to some embodiments. [Figure 9] FIG. 1 illustrates an example system diagram illustrating components of a Multi-Cloud Control Plane (MCCP), in accordance with some embodiments. [Figure 10] FIG. 2 illustrates a high level block diagram of network link components according to one embodiment. [Figure 11] FIG. 2 illustrates a detailed architecture of a network link according to an embodiment. [Figure 12A] 12 illustrates exemplary latency values ​​observed for the network link architecture of FIG. 11, in accordance with an embodiment. [Figure 12B] 12 illustrates exemplary latency values ​​observed for the network link architecture of FIG. 11, in accordance with an embodiment. [Figure 12C] FIG. 2 illustrates an exemplary flowchart illustrating a process of establishing a network link, according to an embodiment. [Figure 13] FIG. 2 illustrates another detailed architecture of a network link according to an embodiment. [Figure 14A]FIG. 14 illustrates exemplary latency values ​​observed for the network link architecture of FIG. 13, according to an embodiment. [Figure 14B] FIG. 1 illustrates another exemplary flowchart illustrating a process of establishing a network link, according to an embodiment. [Figure 15] FIG. 1 illustrates a system diagram of a multi-cloud service control plane, according to an embodiment. [Figure 16A] FIG. 2 illustrates a swim diagram showing multi-cloud service control plane interactions with different cloud environments according to an embodiment. [Figure 16B] FIG. 2 illustrates an exemplary flowchart illustrating a process performed by a multi-cloud service control plane, according to an embodiment. [Figure 17A] FIG. 1 illustrates an exemplary placement strategy for compute instances within availability domains of different cloud environments, according to an embodiment. [Figure 17B] FIG. 2 illustrates an exemplary flowchart illustrating a process performed by a multi-cloud service control plane in determining the availability domains of different cloud environments in which to place compute instances, according to an embodiment. [Figure 18] FIG. 1 is a block diagram illustrating one pattern for implementing a cloud infrastructure as a service system, according to at least one embodiment. [Figure 19] FIG. 1 is a block diagram illustrating another pattern for implementing a cloud infrastructure as a service system, according to at least one embodiment. [Figure 20] FIG. 1 is a block diagram illustrating another pattern for implementing a cloud infrastructure as a service system, according to at least one embodiment. [Figure 21] FIG. 1 is a block diagram illustrating another pattern for implementing a cloud infrastructure as a service system, according to at least one embodiment. [Figure 22]FIG. 1 is a block diagram illustrating an exemplary computer system in accordance with at least one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Detailed Description In the following description, for purposes of explanation, specific details are set forth in order to provide a thorough understanding of certain embodiments. It will be apparent, however, that various embodiments may be practiced without those specific details. The figures and descriptions are not intended to be limiting. The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment or design described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments or designs.

[0014] The present disclosure relates generally to improved cloud architectures, and more particularly to techniques for linking two clouds such that users of one cloud environment can use services provided by another, different cloud environment. Various embodiments are described herein, including methods, systems, non-transitory computer-readable storage media that store programs, code, or instructions executable by one or more processors, and the like. Some embodiments may be implemented by using a computer program product that includes computer programs / instructions that, when executed by a processor, cause the processor to perform any of the methods described in the present disclosure.

[0015] Embodiments of the present disclosure provide a Multi-Cloud Control Plane (MCCP) framework that provisions capabilities for delivering services of a particular cloud network (e.g., Oracle Cloud Infrastructure (OCI)) to users on other clouds (e.g., in Microsoft Azure). The MCCP framework allows users (of other cloud environments) to access services of the cloud environment (e.g., PaaS services) while providing a user experience as close as possible to the user's native cloud environment user experience. A key challenge of MCCP is to enable customers to experience the full data plane capabilities of services in external clouds.

[0016] The MCCP enables users of the second cloud infrastructure (e.g., Azure users) to utilize resources (e.g., database resources) provided by the first cloud infrastructure (e.g., OCI) in a manner that is transparent to the users. In particular, the services provided by the first cloud infrastructure appear as "native" services in the second cloud infrastructure. This allows customers of the second cloud infrastructure to natively access the services provided by the first cloud infrastructure. As described below with reference to Figures 6-17B, the MCCP is a collection of microservices running in the first cloud infrastructure that expose the resources of the first cloud infrastructure for utilization by external cloud users (e.g., users of the second cloud infrastructure). Each of these microservices acts as a proxy that provides communication with the resources provided by the first cloud infrastructure.

[0017] Cloud Network Example The term cloud services is typically used to refer to services made available on demand (e.g., by a subscription model) by a cloud service provider (CSP) to users or customers using systems and infrastructure (cloud infrastructure) provided by the CSP. Typically, the servers and systems that make up the CSP's infrastructure are separate from the customers' own on-premise servers and systems. Thus, customers can utilize cloud services provided by the CSP without needing to purchase separate hardware and software resources for the services. Cloud services are designed to provide subscribing customers with easy and scalable access to applications and computing resources that do not require the customers to invest in procuring the infrastructure used to provide the services.

[0018] There are multiple cloud service providers offering different types of cloud services. There are different types or models of cloud services, including Software-as-a-Service (SaaS), Platform-as-a-Service (PaaS), Infrastructure-as-a-Service (IaaS), etc.

[0019] A customer may subscribe to one or more cloud services offered by a CSP. A customer may be any entity, such as an individual, an organization, a business, etc. When a customer subscribes or registers for a service offered by a CSP, a tenancy or account is created for that customer. The account then enables the customer to access one or more subscribed cloud resources associated with the account.

[0020] As mentioned earlier, infrastructure as a service (IaaS) is one particular type of cloud computing service. In the IaaS model, the CSP provides infrastructure (called cloud services provider infrastructure or CSPI) that can be used by the customer to build their own customizable network and deploy their resources. Thus, the customer's resources and network are hosted in a distributed environment by the infrastructure provided by the CSP. This differs from traditional computing, where the customer's resources and network are hosted by the infrastructure provided by the customer.

[0021] CSPI may comprise interconnected high performance computing resources including various host machines, memory resources, and network resources that form a physical network, also called an infrastructure or underlay network. The resources in CSPI may be distributed across one or more data centers that may be geographically distributed across one or more geographic regions. Virtualization software may be executed by these physical resources to provide a virtualized distributed environment. This virtualization creates an overlay network (also known as a software-based network, software-defined network, or virtual network) on top of the physical network. The physical network of CSPI provides the underlying foundation on which one or more overlay or virtual networks can be created on top of the physical network. The physical network (or infrastructure or underlay network) includes physical network devices such as physical switches, routers, computers, and host machines. An overlay network is a logical (or virtual) network that operates on top of the physical infrastructure network. A particular physical network may support one or more overlay networks. An overlay network typically uses encapsulation techniques to differentiate traffic belonging to different overlay networks. A virtual network or overlay network is also called a virtual cloud network (VCN). A virtual network is implemented using software virtualization technologies (e.g., hypervisors, network virtualization devices (NVDs) (e.g., smart NICs), top-of-rack (TOR) switches, virtualization functions performed by smart TORs that perform one or more functions performed by NVDs, and other mechanisms) to create a layer of network abstraction that can run on top of a physical network. Virtual networks can take many forms, including peer-to-peer networks, IP networks, and the like.A virtual network is typically either a Layer 3 IP network or a Layer 2 VLAN. This method of virtual or overlay networking is often called a virtual Layer 3 network or an overlay Layer 3 network. Examples of protocols developed for virtual networks include IP-in-IP (or Generic Routing Encapsulation (GRE)), Virtual Extensible LAN (VXLAN - IETF RFC 7348), Virtual Private Networks (VPN) (e.g. MPLS Layer 3 Virtual Private Networks (RFC 4364)), VMware's NSX, Generic Network Virtualization Encapsulation (GENEVE), etc.

[0022] In the case of IaaS, the infrastructure provided by the CSP (CSPI) may be configured to provide virtualized computing resources over a public network (e.g., the Internet). In the IaaS model, the cloud computing service provider may host the infrastructure components (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer), etc.). In some cases, the IaaS provider may provide various services (e.g., billing, monitoring, logging, security, load balancing, and clustering, etc.) to accompany those infrastructure components. Thus, these services may be policy-driven, so that IaaS users may implement policies to drive load balancing to maintain application availability and performance. CSPI provides infrastructure and a set of complementary cloud services that enable customers to build and run a wide range of applications and services within a highly available, hosted, distributed environment. CSPI provides high-performance computing resources and computing power as well as storage capacity within a flexible virtual network that is securely accessible from various networked locations, such as from the customer's on-premise network. When a customer subscribes or registers for an IaaS service offered by a CSP, the tenancy created for that customer is a secure, isolated partition within the CSP where the customer can create, organize, and manage their cloud resources.

[0023] Customers can build their own virtual networks using compute, memory, and network resources provided by CSPI. One or more customer resources or workloads, such as compute instances, can be deployed into these virtual networks. For example, customers can build one or more customizable private virtual networks, called Virtual Cloud Networks (VCNs), using resources provided by CSPI. Customers can deploy one or more customer resources, such as compute instances, into the customer's VCN. Compute instances can take the form of virtual machines, bare metal instances, etc. Thus, CSPI provides infrastructure and a set of complementary cloud services that enable customers to build and run a wide range of applications and services within a highly available hosted virtual environment. Customers do not manage or control the underlying physical resources provided by CSPI, but they do have control over the operating system, storage, deployed applications, and in some cases, limited control over selected network components (e.g., firewalls).

[0024] The CSP may provide a console that allows customers and network administrators to configure, access, and manage resources deployed in the cloud using CSPI resources. In one embodiment, the console provides a web-based user interface that can be used to access and manage the CSPI. In some implementations, the console is a web-based application provided by the CSP.

[0025] CSPI may support single-tenancy or multi-tenancy architectures. In a single-tenancy architecture, a software component (e.g., application, database) or hardware component (e.g., host machine or server) serves a single customer or tenant. In a multi-tenancy architecture, a software component or hardware component serves multiple customers or tenants. Thus, in a multi-tenancy architecture, CSPI resources are shared among multiple customers or tenants. In a multi-tenancy situation, precautions are taken and safeguards are implemented within CSPI to ensure that each tenant's data remains isolated and invisible to other tenants.

[0026] In a physical network, a network endpoint ("endpoint") refers to a computing device or system that is connected to the physical network and communicates with the connected network. A network endpoint in a physical network may be connected to a local area network (LAN), a wide area network (WAN), or other types of physical networks. Examples of traditional endpoints in a physical network include modems, hubs, bridges, switches, routers, and other network devices, physical computers (or host machines), and the like. Each physical device in a physical network has a fixed network address that can be used to communicate with the device. This fixed network address can be a layer 2 address (e.g., a MAC address), a fixed layer 3 address (e.g., an IP address), and the like. In a virtual environment or network, the endpoints can include various virtual endpoints, such as virtual machines hosted by components of the physical network (e.g., hosted by a physical host machine). These endpoints in the virtual network are addressed by overlay addresses, such as overlay layer 2 addresses (e.g., an overlay MAC address) and overlay layer 3 addresses (e.g., an overlay IP address). Network overlays enable flexibility by allowing network administrators to use software management (e.g., by software implementing the virtual network's control plane) to move between overlay addresses associated with network endpoints. Thus, unlike physical networks, in a virtual network, overlay addresses (e.g., overlay IP addresses) may be moved from one endpoint to another using network management software. Because virtual networks are built on top of physical networks, communication between components in a virtual network involves both the virtual network and the underlying physical network.To facilitate such communications, the CSPI components are configured to learn and store mappings that map overlay addresses in the virtual network to actual physical addresses in the underlying network, and vice versa. These mappings are then used to facilitate communications. Customer traffic is encapsulated to facilitate routing within the virtual network.

[0027] Thus, a physical address (e.g., a physical IP address) is associated with a component in a physical network, and an overlay address (e.g., an overlay IP address) is associated with an entity in a virtual or overlay network. A physical IP address is an IP address associated with a physical device (e.g., a network device) in an underlying or physical network. For example, each NVD has an associated physical IP address. An overlay IP address is an overlay address associated with an entity in an overlay network, such as associated with a compute instance in a customer's virtual cloud network (VCN). Two different customers or tenants, each with their own private VCN, could potentially use the same overlay IP address in their VCNs without having any knowledge of each other. Both physical and overlay IP addresses are types of real IP addresses. These IP addresses exist separately from virtual IP addresses. A virtual IP address is typically a single IP address that represents or maps to multiple real IP addresses. A virtual IP address provides a one-to-many mapping between a virtual IP address and multiple real IP addresses. For example, a load balancer may use a VIP to map to or represent multiple servers, each with its own real IP address.

[0028] A cloud infrastructure or CSPI is physically hosted in one or more data centers in one or more regions around the world. The CSPI may include components in a physical or foundational network and virtual components (e.g., virtual networks, compute instances, virtual machines, etc.) in a virtual network built on top of the components of the physical network. In an embodiment, the CSPI is organized and hosted in realms, regions, and availability domains. A region is a localized geographic area that typically includes one or more data centers. Regions are generally independent of each other and may be separated by vast distances, for example, across multiple countries or continents. For example, a first region may be in Australia, another region may be in Japan, yet another region may be in India, etc. CSPI resources are divided between regions such that each region includes its own independent subset of CSPI resources. Each region may provide a set of core infrastructure services and resources, such as compute resources (e.g., bare metal servers, virtual machines, containers, and related infrastructure), storage resources (e.g., block volume storage, file storage, object storage, archival storage), network resources (e.g., virtual cloud networks (VCNs), load balancing resources, connections to on-premises networks), database resources, edge network resources (e.g., DNS), and access management and monitoring resources. Each region typically has multiple paths that connect it to other regions within the realm.

[0029] Because using nearby resources is faster than using resources that are farther away, applications are typically deployed in the region (i.e., deployed to the infrastructure associated with that region) where the application is most frequently used. Applications may also be deployed in different regions for a variety of reasons, such as redundancy to reduce the risk of region-wide events such as major weather systems or earthquakes, to meet changing requirements for legal jurisdictions, tax areas, and other business or social criteria.

[0030] Data centers within a region may be further organized and subdivided into availability domains (AD). An availability domain may correspond to one or more data centers located within a region. A region may be composed of one or more availability domains. In such a distributed environment, CSPI resources are specific to a region, such as a virtual cloud network (VCN), or specific to an availability domain, such as a compute instance.

[0031] The ADs in a region are configured to be isolated from each other, fault tolerant, and highly unlikely to fail simultaneously. This is achieved by the ADs not sharing critical infrastructure resources such as networks, physical cables, cable paths, cable entry points, etc., such that a failure in one AD in a region is unlikely to affect the availability of other ADs in the same region. ADs in the same region may be connected to each other by low-latency, high-bandwidth networks that provide high availability connectivity to other networks (e.g., the Internet, customer on-premises networks, etc.), as well as allowing for the creation of replicated systems in multiple ADs for both high availability and disaster recovery. Cloud services use multiple ADs to ensure high availability and protect against resource failures. As the infrastructure provided by the IaaS provider grows, more regions and ADs with additional capacity may be added. Traffic between availability domains is typically encrypted.

[0032] In an embodiment, regions are grouped into realms. A realm is a logical collection of regions. Realms are isolated from each other and do not share any data. Regions in the same realm may communicate with each other, but regions in different realms cannot communicate. A customer's tenancy or account, along with a CSP, exists in a single realm and can be distributed across one or more regions that belong to that realm. Typically, when a customer subscribes to an IaaS service, a tenancy or account is created for the customer in a region designated by the customer in a realm (called the "home" region). The customer can extend the customer's tenancy across one or more other regions in the realm. The customer cannot access regions that are not in the realm in which the customer's tenancy resides.

[0033] An IaaS provider may offer multiple realms, each catering to a particular set of customer or user requirements. For example, a commercial realm may be offered to commercial customers. As another example, a realm may be offered to a particular country for customers in that country. As yet another example, a government realm may be offered to a government, etc. For example, the government realm may cater to a particular government requirement and may have a higher level of security than the commercial realm. For example, Oracle Cloud Infrastructure (OCI) currently offers a realm for a commercial region and two realms for a government cloud region (e.g., FedRAMP certified and IL5 certified).

[0034] In an embodiment, an AD may be subdivided into one or more failure domains. A failure domain is a group of infrastructure resources within an AD to provide anti-affinity. Fault domains allow for distribution of compute instances so that multiple compute instances are not on the same physical hardware within a single AD. This distribution is known as anti-affinity. A failure domain refers to a set of hardware components (computers, switches, etc.) that share a single point of failure. A compute pool is logically divided into failure domains. Thus, a hardware failure or compute hardware maintenance event that affects one failure domain does not affect instances in other failure domains. Depending on the embodiment, the number of failure domains per AD may vary. For example, in an embodiment, each AD includes three failure domains. Fault domains act as logical data centers within an AD.

[0035] When a customer subscribes to an IaaS service, resources from CSPI are provisioned for the customer and associated with the customer's tenancy. The customer can use these provisioned resources to build private networks and deploy resources into these networks. A customer's network hosted in the cloud by CSPI is called a Virtual Cloud Network (VCN). A customer can set up one or more Virtual Cloud Networks (VCNs) using the CSPI resources allocated to the customer. A VCN is a Virtual Private Network or a Software-Defined Private Network. The customer's resources deployed in the customer's VCN can include compute instances (e.g., virtual machines, bare metal instances) and other resources. These compute instances may represent various customer workloads, such as applications, load balancers, databases, etc. The compute instances deployed in a VCN can communicate with endpoints that are publicly accessible over a public network such as the Internet ("public endpoints"), with other instances in the same VCN or other VCNs (e.g., other VCNs of the customer or VCNs that do not belong to the customer), with the customer's on-premises data center or network, and with service endpoints and other types of endpoints.

[0036] CSPs may offer various services using CSPI. In some cases, customers of a CSPI may act as service providers themselves and offer services using CSPI resources. Service providers may expose service endpoints characterized by identifying information (e.g., IP addresses, DNS names, and DNS ports). Customer resources (e.g., compute instances) can consume a particular service by accessing the service endpoints exposed by the service for that particular service. These service endpoints are generally endpoints that are publicly accessible by users over a public communications network, such as the Internet, using a public IP address associated with the endpoint. Publicly accessible network endpoints are sometimes referred to as public endpoints.

[0037] In one embodiment, a service provider may expose a service via an endpoint for the service (sometimes referred to as a service endpoint). Customers of the service can then access the service using this service endpoint. In some implementations, a service endpoint provided for a service may be accessed by multiple customers wishing to consume the service. In other implementations, a dedicated service endpoint may be provided to a customer, allowing only that customer to access the service using that dedicated service endpoint.

[0038] In an embodiment, when a VCN is created, it is associated with a private overlay Classless Inter-Domain Routing (CIDR) address space, which is a range of private overlay IP addresses (e.g., 10.0 / 16) that are assigned to the VCN. A VCN includes associated subnets, route tables, and gateways. A VCN exists within a single region but can span one or more or all of the region's availability domains. A gateway is a virtual interface configured for a VCN that enables traffic to and from the VCN to one or more endpoints outside the VCN. One or more different types of gateways may be configured for a VCN to enable communication to and from different types of endpoints.

[0039] A VCN may be subdivided into one or more subnetworks, such as one or more subnets. A subnet is thus a unit of configuration or subdivision that may be created within a VCN. A VCN may contain one or more subnets. Each subnet in a VCN is associated with a contiguous range of overlay IP addresses (e.g., 10.0.0.0 / 24 and 10.0.1.0 / 24) that represents a subset of address space within the VCN's address space that does not overlap with other subnets in that VCN.

[0040] Each compute instance is associated with a virtual network interface card (VNIC) that allows the compute instance to participate in a subnet of a VCN. A VNIC is a logical representation of a physical network interface card (NIC). In general, a VNIC is an interface between an entity (e.g., compute instance, service) and a virtual network. A VNIC resides in a subnet and has one or more associated IP addresses and associated security rules or policies. A VNIC is equivalent to a layer 2 port on a switch. A VNIC is connected to a compute instance and is connected to a subnet in a VCN. A VNIC associated with a compute instance allows the compute instance to be part of a subnet of a VCN and allows the compute instance to communicate (e.g., send and receive packets) with endpoints on the same subnet as the compute instance, with endpoints in a different subnet in the VCN, or with endpoints outside the VCN. Thus, a VNIC associated with a compute instance determines how the compute instance connects with endpoints inside and outside the VCN. A VNIC for a compute instance is created and associated with the compute instance when the compute instance is created and added to a subnet in a VCN. For a subnet that contains a set of compute instances, the subnet contains VNICs that correspond to the set of compute instances, and each VNIC connects to one compute instance in the set of compute instances.

[0041] Each compute instance is assigned a private overlay IP address through the VNIC associated with the compute instance. This private overlay IP address is assigned to the VNIC associated with the compute instance when the compute instance is created and is used to route traffic to and from the compute instance. All VNICs within a particular subnet use the same route table, security lists, and DHCP options. As previously mentioned, each subnet in a VCN is associated with a contiguous range of overlay IP addresses (e.g., 10.0.0.0 / 24 and 10.0.1.0 / 24) that represents a subset of address space within the VCN's address space that does not overlap with other subnets in that VCN. For a VNIC on a particular subnet of a VCN, the private overlay IP address assigned to the VNIC is an address from the contiguous range of overlay IP addresses assigned to this subnet.

[0042] In an embodiment, a compute instance may be optionally assigned additional overlay IP addresses, such as one or more public IP addresses if in a public subnet, in addition to the private overlay IP address. These multiple addresses may be assigned to the same VNIC or across multiple VNICs associated with the compute instance. However, each instance has a primary VNIC that is created during instance launch and associated with the private overlay IP address assigned to the instance, and this primary VNIC cannot be removed. Additional VNICs, called secondary VNICs, may be added to an existing instance in the same availability domain as the primary VNIC. All VNICs are in the same availability domain as the instance. The secondary VNICs can be in a subnet in the same VCN as the primary VNIC, or in a different subnet, either in the same VCN or in a different VCN.

[0043] If a compute instance is in a public subnet, the compute instance may optionally be assigned a public IP address. When a subnet is created, it can be specified as either a public subnet or a private subnet. A private subnet means that resources in the subnet (e.g., compute instances) and associated VNICs cannot have public overlay IP addresses. A public subnet means that resources in the subnet and associated VNICs can have public IP addresses. Customers can specify a subnet to exist in a single availability domain or across multiple availability domains within a region or realm.

[0044] As mentioned above, a VCN may be subdivided into one or more subnets. In one embodiment, a Virtual Router (VR) configured for a VCN (referred to as a VCN VR or simply VR) enables communication between subnets of the VCN. For a subnet in a VCN, the VR represents a logical gateway for that subnet, allowing the subnet (i.e., the compute instances on that subnet) to communicate with endpoints on other subnets in the VCN and with other endpoints outside the VCN. A VCN VR is a logical entity configured to route traffic between VNICs in a VCN and a virtual gateway ("gateway") associated with the VCN. Gateways are further described below with respect to FIG. 1. A VCN VR is a Layer 3 / IP layer concept. In one embodiment, there is one VCN VR for a VCN, which may have an unlimited number of ports addressed by IP addresses, one port for each subnet of the VCN. In this way, a VCN VR has a different IP address for each subnet in the VCN to which the VCN VR is connected. The VRs are also connected to various gateways configured for the VCN. In one embodiment, a particular overlay IP address from a subnet's overlay IP address range is reserved for the ports of the VCN VR of that subnet. For example, consider a VCN that includes two subnets with associated address ranges 10.0 / 16 and 10.1 / 16, respectively. For the first subnet in the VCN with address range 10.0 / 16, an address from this range is reserved for the ports of the VCN VR of that subnet. In some cases, the first IP address from this range may be reserved for a VCN VR. For example, for a subnet with overlay IP address range 10.0 / 16, IP address 10.0.0.1 may be reserved for the ports of the VCN VR of that subnet.For a second subnet in the same VCN with address range 10.1 / 16, the VCN VR may have a port in that second subnet with IP address 10.1.0.1. The VCN VR has a different IP address for each of the subnets in the VCN.

[0045] In some other embodiments, each subnet in a VCN may include a VR associated with it that is addressable by the subnet using a reserved or default IP address associated with the VR. The reserved or default IP address may be, for example, the first IP address from a range of IP addresses associated with the subnet. VNICs in a subnet can use this default or reserved IP address to communicate (e.g., send and receive packets) with the VR associated with the subnet. In such embodiments, a VR is an ingress / egress point for that subnet. VRs associated with a subnet in a VCN can communicate with other VRs associated with other subnets in the VCN. VRs can also communicate with gateways associated with the VCN. The VR functions of a subnet are running on or performed by one or more NVDs that are running the VNIC functions of the VNICs in the subnet.

[0046] Route tables, security rules, and DHCP options may be configured for a VCN. A route table is a virtual route table for a VCN and contains rules for routing traffic from subnets in the VCN to destinations outside the VCN via gateways or specially configured instances. A VCN's route table can be customized to control how packets are forwarded / routed to and from the VCN. DHCP options refer to configuration information that is automatically provided to an instance when the instance launches.

[0047] Security rules configured for a VCN represent the overlay firewall rules for the VCN. Security rules include ingress and egress rules and can specify the type of traffic (e.g., based on protocol and port) that is allowed in and out of instances in the VCN. Customers can choose whether a particular rule is stateful or stateless. For example, a customer can allow inbound SSH traffic from any location to a set of instances by configuring a stateful ingress rule with source CIDR 0.0.0.0 / 0 and destination TCP port 22. Security rules can be implemented using network security groups or security lists. A network security group consists of a set of security rules that apply only to resources in that group. A security list, on the other hand, contains rules that apply to all resources in any subnet that uses the security list. A VCN may include a default security list that contains the default security rules. DHCP options configured for a VCN provide configuration information that is automatically provided to instances in the VCN when the instances launch.

[0048] In one embodiment, configuration information for a VCN is determined and stored by a VCN control plane. The configuration information for a VCN may include, for example, information about address ranges associated with the VCN, subnets and associated information within the VCN, one or more VRs associated with the VCN, compute instances and associated VNICs within the VCN, NVDs performing various virtualized network functions associated with the VCN (e.g., VNICs, VRs, gateways), state information for the VCN, and other VCN-related information. In one embodiment, a VCN distribution service publishes the configuration information stored by the VCN control plane or a portion thereof to the NVD. The distributed information may be used to update information (e.g., forwarding tables, routing tables, etc.) stored and used by the NVD to forward packets to and from compute instances in the VCN.

[0049] In one embodiment, the creation of VCNs and subnets is handled by a VCN Control Plane (CP), and the launch of compute instances is handled by the compute control plane. The compute control plane is responsible for allocating physical resources to compute instances and then calling the VCN control plane to create and connect VNICs to compute instances. The VCN CP also sends VCN data mappings to the VCN data plane, which is configured to perform packet forwarding and routing functions. In one embodiment, the VCN CP provides a distribution service that is responsible for providing updates to the VCN data plane. Examples of the VCN control plane are also shown in Figures 6, 7, 8, and 9 (see reference numbers 616, 716, 816, and 916) and are described below.

[0050] A customer may create one or more VCNs using resources hosted by CSPI. Compute instances deployed in a customer's VCN may communicate with various endpoints. These endpoints may include endpoints hosted by CSPI and endpoints outside of CSPI.

[0051] A variety of different architectures for implementing cloud-based services using CSPI are shown in Figures 1, 2, 3, 4, 5, and 18-22 and described below. Figure 1 is a high-level diagram of a distributed environment 100 illustrating an overlay VCN or customer VCN hosted by CSPI according to an embodiment. The distributed environment illustrated in Figure 1 includes multiple components within an overlay network. The distributed environment 100 illustrated in Figure 1 is merely an example and is not intended to unduly limit the scope of the claimed embodiments. Many variations, alternatives, and modifications are possible. For example, in some implementations, the distributed environment illustrated in Figure 1 may include more or fewer systems or components than those illustrated in Figure 1, may combine two or more subsystems, or may include a different configuration or arrangement of systems.

[0052] As shown in the example depicted in FIG. 1, distributed environment 100 includes CSPI 101, which provides services and resources that customers can subscribe to and use to build their own virtual cloud networks (VCNs). In one embodiment, CSPI 101 provides IaaS services to subscribing customers. Data centers in CSPI 101 may be organized into one or more regions. One exemplary region, “Region US” 102, is shown in FIG. 1. A customer configures a VCN for a customer of Oracle International Corporation with region 102. A customer may deploy various compute instances into VCN 104, which may include virtual machines or bare metal instances. Example instances include applications, databases, load balancers, etc.

[0053] In the embodiment shown in FIG. 1, customer's VCN 104 includes two subnets, "Subnet 1" and "Subnet 2," each with its own CIDR IP address range. In FIG. 1, the overlay IP address range of Subnet 1 is 10.0 / 16, and the address range of Subnet 2 is 10.1 / 16. VCN virtual router 105 represents the logical gateway of the VCN, enabling communication between the subnets of VCN 104 and with other endpoints outside the VCN. VCN VR 105 is configured to route traffic between VNICs in VCN 104 and the gateway associated with VCN 104. VCN VR 105 provides a port for each subnet of VCN 104. For example, VR 105 may provide a port with IP address 10.0.0.1 for Subnet 1 and a port with IP address 10.1.0.1 for Subnet 2.

[0054] Multiple compute instances may be deployed in each subnet, and the compute instances can be virtual machine instances and / or bare metal instances. The compute instances in a subnet may be hosted by one or more host machines in CSPI101. The compute instances join the subnet through the VNIC associated with the compute instance. For example, as shown in FIG. 1, compute instance C1 becomes part of subnet 1 through the VNIC associated with the compute instance. Similarly, compute instance C2 becomes part of subnet 1 through the VNIC associated with C2. In a similar manner, multiple compute instances, which may be virtual machine instances or bare metal instances, may become part of subnet 1. Each compute instance is assigned a private overlay IP address and a MAC address through its associated VNIC. For example, in FIG. 1, compute instance C1 has an overlay IP address of 10.0.0.2 and a MAC address of M1, while compute instance C2 has a private overlay IP address of 10.0.0.3 and a MAC address of M2. Each compute instance in Subnet1, including compute instances C1 and C2, has a default route to VCN VR105 using IP address 10.0.0.1, which is the IP address of a port in VCN VR105 in Subnet1.

[0055] Multiple compute instances, including virtual machine instances and / or bare metal instances, may be deployed in Subnet 2. For example, as shown in FIG. 1, compute instances D1 and D2 become part of Subnet 2 via VNICs associated with the respective compute instances. In the embodiment shown in FIG. 1, compute instance D1 has an overlay IP address of 10.1.0.2 and a MAC address of MM1, while compute instance D2 has a private overlay IP address of 10.1.0.3 and a MAC address of MM2. Each compute instance in Subnet 2, including compute instances D1 and D2, has a default route to VCN VR105 using IP address 10.1.0.1, which is the IP address of a port in VCN VR105 in Subnet 2.

[0056] VCN A 104 may include one or more load balancers. For example, a load balancer may be provided to a subnet and configured to load balance traffic across multiple compute instances on the subnet. A load balancer may be provided to load balance traffic across multiple subnets in a VCN.

[0057] A particular compute instance deployed in VCN 104 may communicate with various endpoints. These endpoints may include endpoints hosted by CSPI 200 and endpoints outside CSPI 200. Endpoints hosted by CSPI 101 may include endpoints on the same subnet as the particular compute instance (e.g., communication between two compute instances in subnet 1), endpoints on a different subnet but within the same VCN (e.g., communication between a compute instance in subnet 1 and a compute instance in subnet 2), endpoints in a different VCN in the same region (e.g., communication between a compute instance in subnet 1 and an endpoint in a VCN in the same region 106 or 110, communication between a compute instance in subnet 1 and an endpoint in a service network 110 in the same region), or endpoints in a VCN in a different region (e.g., communication between a compute instance in subnet 1 and an endpoint in a VCN in a different region 108). Compute instances in a subnet hosted by CSPI 101 may communicate with endpoints not hosted by CSPI 101 (i.e., outside CSPI 101). These external endpoints include endpoints within the customer's on-premise network 116, endpoints within other remote cloud-hosted networks 118, public endpoints 114 accessible via public networks such as the Internet, and other endpoints.

[0058] Communication between compute instances on the same subnet is facilitated using VNICs associated with the source and destination compute instances. For example, compute instance C1 in subnet 1 may want to send a packet to compute instance C2 in subnet 1. For a packet originating from a source compute instance and destined for another compute instance in the same subnet, the packet is first processed by the VNIC associated with the source compute instance. The processing performed by the VNIC associated with the source compute instance may include determining the packet's destination information from the packet header, identifying any policies (e.g., security lists) configured for the VNIC associated with the source compute instance, determining the packet's next hop, performing any packet encapsulation / decapsulation functions as necessary, and then forwarding / routing the packet to the next hop for the purpose of facilitating communication of the packet with its intended destination. If the destination compute instance is in the same subnet as the source compute instance, the VNIC associated with the source compute instance is configured to identify the VNIC associated with the destination compute instance and forward the packet to that VNIC for processing. The VNIC associated with the destination compute instance is then executed to forward the packet to the destination compute instance.

[0059] For packets traveling from a compute instance in a subnet to an endpoint in a different subnet in the same VCN, this communication is facilitated by the VNICs associated with the source and destination compute instances as well as the VCN VRs. For example, if compute instance C1 in subnet 1 of FIG. 1 wants to send a packet to compute instance D1 in subnet 2, the packet is first processed by the VNIC associated with compute instance C1. The VNIC associated with compute instance C1 is configured to route the packet to VCN VR105 using the VCN VR's default route or port 10.0.0.1. VCN VR105 is configured to route the packet to subnet 2 using port 10.1.0.1. The packet is then received and processed by the VNIC associated with D1, which forwards the packet to compute instance D1.

[0060] For packets traveling from a compute instance in VCN 104 to an endpoint outside VCN 104, the communication is facilitated by a VNIC associated with the source compute instance, VCN VR 105, and a gateway associated with VCN 104. One or more types of gateways may be associated with VCN 104. A gateway is an interface between a VCN and another endpoint, the other endpoint being outside the VCN. A gateway is a Layer 3 / IP layer concept that allows a VCN to communicate with endpoints outside the VCN. Thus, a gateway facilitates traffic flow between a VCN and other VCNs or networks. Different types of gateways may be configured for a VCN to facilitate different types of communication with different types of endpoints. Depending on the gateway, the communication may go over a public network (e.g., the Internet) or over a private network. Different communication protocols may be used for these communications.

[0061] For example, compute instance C1 may wish to communicate with an endpoint outside VCN 104. The packet may be first processed by a VNIC associated with the source compute instance C1. This VNIC processing determines that the packet's destination is outside of Subnet 1 of C1. The VNIC associated with C1 may forward the packet to VCN VR 105 of VCN 104. VCN VR 105 then processes the packet and, as part of this processing, determines a particular gateway associated with VCN 104 as the packet's next hop based on the packet's destination. VCN VR 105 may then forward the packet to the particular identified gateway. For example, if the destination is an endpoint within a customer's on-premise network, VCN VR 105 may forward the packet to a Dynamic Routing Gateway (DRG) gateway 122 configured for VCN 104. The packet may then be forwarded from the gateway to the next hop to facilitate propagation of the packet to its final intended destination.

[0062] Various types of gateways may be configured for a VCN. Examples of gateways that may be configured for a VCN are shown in FIG. 1 and described below. Examples of gateways associated with a VCN are also shown in FIG. 18, FIG. 19, FIG. 20, and FIG. 21 (e.g., gateways referenced by reference numbers 1834, 1836, 1838, 1934, 1936, 1938, 2034, 2036, 2038, 2134, 2136, and 2138) and described below. As shown in the embodiment shown in FIG. 1, a dynamic routing gateway (DRG) 122 may be added to or associated with a customer's VCN 104 to provide a path for private network traffic communication between the customer's VCN 104 and another endpoint, which may be the customer's on-premise network 116, a VCN 108 in a different region of the CSPI 101, or another remote cloud network 118 not hosted by the CSPI 101. The customer on-premise network 116 may be a customer network or a customer data center built using the customer's resources. Access to the customer on-premise network 116 is usually highly restricted. For a customer that has both a customer on-premise network 116 and one or more VCNs 104 deployed or hosted in the cloud by CSPI 101, the customer may want the customer on-premise network 116 and the customer cloud-based VCN 104 to be able to communicate with each other. This allows the customer to build an extended hybrid environment that encompasses the customer's VCNs 104 hosted by CSPI 101 and the customer on-premise network 116. The DRG 122 enables this communication. To enable such communication, a communication channel 124 is set up, one endpoint of which is in the customer on-premise network 116 and the other endpoint is in CSPI 101 and connected to the customer's VCN 104. The communication channel 124 may traverse a public or private communication network, such as the Internet.Various communication protocols may be used, such as IPsec VPN technology over a public communication network such as the Internet, Oracle's FastConnect technology using a private network instead of a public network, etc. A device or equipment in the customer's on-premise network 116 that forms one endpoint of the communication channel 124 is called customer premise equipment (CPE), such as CPE 126 shown in Figure 1. On the CSPI 101 side, the endpoint may be a host machine running DRG 122.

[0063] In an embodiment, a Remote Peering Connection (RPC) may be added to the DRG, allowing a customer to peer one VCN with another VCN in a different region. Using such an RPC, the customer's VCN 104 can connect with a VCN 108 in another region using the DRG 122. The DRG 122 may be used to communicate with other remote cloud networks 118 not hosted by the CSPI 101, such as the Microsoft Azure cloud, the Amazon AWS cloud, etc.

[0064] As shown in Figure 1, an Internet Gateway (IGW) 120 may be configured for a customer's VCN 104 that allows compute instances on VCN 104 to communicate with public endpoints 114 accessible over a public network, such as the Internet. The IGW 120 is a gateway that connects a VCN to a public network, such as the Internet. The IGW 120 enables direct access for public subnets in a VCN, such as VCN 104 (resources in the public subnet have public overlay IP addresses) to public endpoints 112 on the public network 114, such as the Internet. Using the IGW 120, connections may be initiated from subnets in VCN 104 or from the Internet.

[0065] A Network Address Translation (NAT) gateway 128 is configured for customer VCN 104 to enable access to the Internet for cloud resources in the customer's VCN that do not have dedicated public overlay IP addresses, without exposing those resources to direct incoming Internet connections (e.g., L4-L7 connections). This allows private subnets in a VCN, such as private subnet 1 in VCN 104, to private endpoints on the Internet. The NAT gateway only allows connections to be initiated from the private subnet to the public Internet, and connections cannot be initiated from the Internet to the private subnet.

[0066] In one embodiment, a Service Gateway (SGW) 126 can be configured for a customer's VCN 104 and provides a path for private network traffic between the VCN 104 and supported service endpoints in the service network 110. In one embodiment, the service network 110 can be provided by a CSP and can provide a variety of services. An example of such a service network is Oracle's Services Network, which provides a variety of services that can be used by a customer. For example, a compute instance (e.g., a database system) in a private subnet of a customer's VCN 104 can back up data to a service endpoint (e.g., object storage) without needing a public IP address or access to the Internet. In one embodiment, a VCN can include only one SGW, and connections can be initiated only from subnets in the VCN, not from the service network 110. When a VCN is peered with another VCN, resources in the other VCN typically do not have access to the SGW. Resources in an on-premises network connected to a VCN using a FastConnect or VPN connection can also use a service gateway configured for that VCN.

[0067] In one implementation, SGW 126 uses the concept of a service Classless Inter-Domain Routing (CIDR) label, which is a string that represents the public IP address range of all regions for a service or group of services of interest. A customer uses the service CIDR label when configuring the SGW and associated route rules to control traffic to the service. A customer can optionally utilize the service CIDR label when configuring security rules, without having to adjust those security rules if the service's public IP address changes in the future.

[0068] A Local Peering Gateway (LPG) 132 is a gateway that can be added to a customer's VCN 104, allowing the VCN 104 to peer with another VCN in the same region. Peering means that the VCNs communicate using private IP addresses without the traffic traversing a public network such as the Internet or routing the traffic through the customer's on-premises network 116. In a preferred embodiment, a VCN includes a separate LPG for each peering it establishes. Local peering or VCN peering is a common method used to establish network connectivity between different applications or infrastructure management functions.

[0069] A service provider, such as a provider of services in service network 110, may provide access to the services using various access models. According to a public access model, the service may be exposed as a public endpoint, which may be publicly accessible by compute instances in the customer's VCN over a public network such as the Internet, and / or privately accessible through SGW 126. According to a particular private access model, the service is made accessible as a private IP endpoint in a private subnet in the customer's VCN. This access is called Private Endpoint (PE) access and allows the service provider to expose the service as an instance in the customer's private network. A private endpoint resource represents a service in the customer's VCN. Each PE appears as a VNIC (called a PE-VNIC with one or more private IPs) in a customer-selected subnet in the customer's VCN. Thus, the PE provides a way to present a service in a subnet of the private customer's VCN using a VNIC. Because the endpoint is exposed as a VNIC, all the features associated with the VNIC, such as routing rules, security lists, etc., are available to the PE VNIC.

[0070] A service provider can register a service to make it accessible through the PE. The provider can associate a policy with the service that limits the visibility of the service to the customer's tenancy. The provider can register multiple services under a single virtual IP address (VIP), especially in the case of multi-tenant services. There can be multiple such private endpoints (in multiple VCNs) representing the same service.

[0071] Compute instances in the private subnet can then access the service using the private IP address of the PE VNIC or the DNS name of the service. Compute instances in the customer's VCN can access the service by sending traffic to the private IP address of the PE in the customer's VCN. A Private Access Gateway (PAGW) 130 is a gateway resource that can be connected to a service provider's VCN (e.g., a VCN in service network 110) and serves as an ingress / egress point for all traffic to and from the private endpoints of the customer's subnet. The PAGW 130 allows the provider to scale the number of PE connections without utilizing internal IP address resources. The provider needs to configure only one PAGW for any number of services registered in a single VCN. The provider can represent the services as private endpoints in multiple VCNs for one or more customers. From the customer's perspective, the PE VNIC appears to be connected to the service the customer wants to interact with instead of being connected to the customer's instance. Traffic going to the private endpoint is routed to the service through the PAGW 130. These are called customer-to-service private connections (C2S (customer-to-service) connections).

[0072] The PE concept can also be used to extend the private access of a service to a customer's on-premises network and data center by allowing traffic to flow through a FastConnect / IPsec link and a private endpoint in the customer's VCN. The private access of a service can also be extended to a customer's peered VCN by allowing traffic to flow between LPG 132 and a PE in the customer's VCN.

[0073] A customer can control routing within a VCN at the subnet level, so that the customer can specify which subnets within a customer's VCN, such as VCN 104, use each gateway. A VCN's route table is used to determine whether traffic is allowed to exit the VCN through a particular gateway. For example, in a particular example, a route table for a public subnet in a customer's VCN 104 may send non-local traffic through IGW 120. A route table for a private subnet in the same customer's VCN 104 may send traffic going to a CSP service through SGW 126. All remaining traffic may be sent through NAT gateway 128. The route table controls only traffic that exits the VCN.

[0074] Security lists associated with a VCN are used to control traffic entering the VCN through a gateway via an inbound connection. All resources within a subnet use the same route table and security lists. Security lists may be used to control the specific types of traffic allowed into and out of instances within a subnet of a VCN. Security list rules may include ingress (inbound) rules and egress (outbound) rules. For example, an ingress rule may specify an allowed source address range, while an egress rule may specify an allowed destination address range. Security rules may specify a specific protocol (e.g., TCP, ICMP), a specific port (e.g., 22 for SSH, 3389 for Windows RDP), etc. In some implementations, the instance's operating system may enforce its own firewall rules that match the security list rules. Rules may be stateful (e.g., connections are tracked and responses are automatically allowed without the use of explicit security list rules for the response traffic) or stateless.

[0075] Access from a customer's VCN (i.e., by resources or compute instances deployed in VCN 104) may be categorized as public access, private access, or dedicated access. Public access refers to an access model in which public IP addresses or NATs are used to access public endpoints. Private access allows customer workloads (e.g., resources in private subnets) in VCN 104 with private IP addresses to access services without traversing a public network such as the Internet. In one embodiment, CSPI 101 allows workloads in a customer's VCN with private IP addresses to access (public service endpoints of) services using a service gateway. Thus, the service gateway provides a private access model by establishing a virtual link between the customer's VCN and the public endpoints of the services that reside outside the customer's private network.

[0076] Additionally, CSPI may provide dedicated public access using technologies such as FastConnect public peering, where a customer's on-premises instances can use a FastConnect connection to access one or more services in the customer's VCN without traversing a public network such as the Internet. CSPI may provide dedicated private access using FastConnect private peering, where a customer's on-premises instances with private IP addresses can use a FastConnect connection to access workloads in the customer's VCN. FastConnect is a network connection alternative to using the public Internet to connect a customer's on-premises network to CSPI and its services. FastConnect provides an easy, resilient, and economical way to create dedicated private connections with higher bandwidth options and a more reliable and consistent network experience when compared to Internet-based connections.

[0077] FIG. 1 and the accompanying description above describe various virtual components in an exemplary virtual network. As previously mentioned, a virtual network is built on an underlying physical network or infrastructure network. FIG. 2 illustrates a simplified architecture diagram of physical components in an underlying physical network in CSPI 200 for a virtual network, according to an embodiment. As shown in the figure, CSPI 200 provides a distributed environment including components and resources (e.g., compute resources, memory resources, and network resources) provided by a cloud service provider (CSP). These components and resources are used to provide cloud services (e.g., IaaS services) to subscribing customers, i.e., customers who subscribe to one or more services provided by the CSP. Based on the services to which the customer subscribes, a subset of the resources (e.g., compute resources, memory resources, and network resources) of CSPI 200 is provisioned for the customer. The customer can then build their own cloud-based (i.e., CSPI-hosted) customizable private virtual network using the physical compute resources, memory resources, and network resources provided by CSPI 200. As previously indicated, these customer networks are referred to as Virtual Cloud Networks (VCNs). Customers can deploy one or more customer resources, such as compute instances, into these customer VCNs. The compute instances can be in the form of virtual machines, bare metal instances, etc. CSPI 200 provides infrastructure and a set of complementary cloud services that enable customers to build and run a wide range of applications and services within a highly available hosted environment.

[0078] In the example embodiment shown in FIG. 2, the physical components of CSPI 200 include one or more physical host machines or servers (e.g., 202, 206, 208), network virtualization devices (NVDs) (e.g., 210, 212), top-of-rack (TOR) switches (e.g., 214, 216), and physical networks (e.g., 218), and switches within physical network 218. The physical host machines or servers may host and execute various compute instances that participate in one or more subnets of the VCN. The compute instances may include virtual machine instances and bare metal instances. For example, the various compute instances shown in FIG. 1 may be hosted by the physical host machines shown in FIG. 2. The virtual machine compute instances in the VCN may be executed by one host machine or by multiple different host machines. The physical host machines may host virtual host machines, container-based hosts or functions, etc. The VNICs and VCN VRs shown in FIG. 1 may be executed by the NVDs shown in FIG. 2. The gateway shown in FIG. 1 may be executed by a host machine and / or by the NVD shown in FIG.

[0079] A host machine or server may run a hypervisor (also called a virtual machine monitor or VMM) that creates and enables a virtual environment on the host machine. The virtualized environment or virtual environment facilitates cloud-based computing. On a host machine, one or more computing instances may be created, executed, and managed by the hypervisor on the host machine. The hypervisor on the host machine enables the physical computing resources of the host machine (e.g., computing resources, memory resources, and network resources) to be shared among the various computing instances executed by the host machine.

[0080] For example, as shown in FIG. 2, host machines 202 and 208 run hypervisors 260 and 266, respectively. These hypervisors may be implemented using software, firmware, or hardware, or a combination thereof. Typically, a hypervisor is a process or software layer that resides on top of a host machine's operating system (OS), which runs on the host machine's hardware processor. A hypervisor provides a virtual environment by allowing the host machine's physical computing resources (e.g., processing resources such as processors / cores, memory resources, network resources) to be shared among various virtual machine computing instances executed by the host machine. For example, in FIG. 2, hypervisor 260 may reside on top of the host machine's OS and allows the host machine's 202 computing resources (e.g., processing resources, memory resources, and network resources) to be shared among computing instances (e.g., virtual machines) executed by the host machine 202. A virtual machine can have its own operating system (called a guest operating system), which may be the same as or different from the host machine's OS. The operating system of a virtual machine executed by a host machine may be the same or different from the operating system of another virtual machine executed by the same host machine. Thus, the hypervisor allows multiple operating systems to run in parallel with each other while sharing the same computing resources of the host machine. The host machines shown in FIG. 2 may have the same or different types of hypervisors.

[0081] A compute instance can be a virtual machine instance or a bare metal instance. In Figure 2, compute instance 268 on host machine 202 and compute instance 274 on host machine 208 are examples of virtual machine instances. Host machine 206 is an example of a bare metal instance that is provided to a customer.

[0082] In one example, an entire host machine may be provisioned to a single customer, and all of the one or more compute instances (either virtual machines or bare metal instances) hosted by that host machine belong to that same customer. In other examples, a host machine may be shared among multiple customers (i.e., multiple tenants). In such a multi-tenancy situation, a host machine may host virtual machine compute instances that belong to different customers. These compute instances may be members of different VCNs for different customers. In one embodiment, bare metal compute instances are hosted by bare metal servers that do not have a hypervisor. When bare metal compute instances are provisioned, a single customer or tenant maintains control of the physical CPU, memory, and network interfaces of the host machine hosting the bare metal instance, and the host machine is not shared with other customers or tenants.

[0083] As previously mentioned, each compute instance that is part of a VCN is associated with a VNIC that enables the compute instance to be a member of a subnet of the VCN. The VNIC associated with a compute instance facilitates communication of packets or frames to and from the compute instance. The VNIC is associated with the compute instance when the compute instance is created. In an embodiment, for a compute instance executed by a host machine, the VNIC associated with the compute instance is executed by the NVD connected to the host machine. For example, in FIG. 2, host machine 202 executes virtual machine compute instance 268 that is associated with VNIC 276, which is executed by NVD 210 connected to host machine 202. As another example, bare metal instance 272 hosted by host machine 206 is associated with VNIC 280 that is executed by NVD 212 connected to host machine 206. As yet another example, VNIC 284 is associated with compute instance 274 executed by host machine 208, which is executed by NVD 212 connected to host machine 208.

[0084] For compute instances hosted by a host machine, the NVD connected to that host machine also runs VCN VRs corresponding to the VCNs of which those compute instances are members. For example, in the embodiment shown in Figure 2, NVD 210 runs VCN VR 277 corresponding to the VCN of which compute instance 268 is a member. NVD 212 may run one or more VCN VRs 283 corresponding to the VCNs corresponding to the compute instances hosted by host machines 206 and 208.

[0085] A host machine may include one or more network interface cards (NICs) that allow the host machine to be connected to other devices. The NICs on a host machine may provide one or more ports (or interfaces) that allow the host machine to be communicatively connected to another device. For example, a host machine may be connected to an NVD using one or more ports (or interfaces) provided on the host machine and the NVD. A host machine may be connected to other devices, such as another host machine.

[0086] 2, host machine 202 is connected to NVD 210 using link 220 extending between port 234 provided by NIC 232 of host machine 202 and port 236 of NVD 210. Host machine 206 is connected to NVD 212 using link 224 extending between port 246 provided by NIC 244 of host machine 206 and port 248 of NVD 212. Host machine 208 is connected to NVD 212 using link 226 extending between port 252 provided by NIC 250 of host machine 208 and port 254 of NVD 212.

[0087] The NVDs are then connected via communication links to top-of-rack (TOR) switches (also called switch fabrics) that are connected to a physical network 218. In one embodiment, the links between the host machines and the NVDs and between the NVDs and the TOR switches are Ethernet links. For example, in FIG. 2, links 228 and 230 are used to connect NVDs 210 and 212 to TOR switches 214 and 216, respectively. In one embodiment, links 220, 224, 226, 228, and 230 are Ethernet links. The collection of host machines and NVDs connected to a TOR may be referred to as a rack.

[0088] The physical network 218 provides a communications fabric that allows the TOR switches to communicate with each other. The physical network 218 can be a multi-tier network. In one implementation, the physical network 218 is a multi-tier Clos network of switches, with the TOR switches 214 and 216 representing leaf-level nodes of the multi-tier, multi-node physical switching network 218. A variety of Clos network configurations are possible, including, but not limited to, two-tier networks, three-tier networks, four-tier networks, five-tier networks, and generally, "n"-tier networks. An example Clos network is shown in FIG. 5 and described below.

[0089] Various connection configurations are possible between host machines and NVDs, such as one-to-one, many-to-one, and one-to-many configurations. In a one-to-one implementation, each host machine is connected to its own separate NVD. For example, in FIG. 2, host machine 202 is connected to NVD 210 via NIC 232 of host machine 202. In a many-to-one configuration, multiple host machines are connected to one NVD. For example, in FIG. 2, host machines 206 and 208 are connected to the same NVD 212 via NICs 244 and 250, respectively.

[0090] In a one-to-many configuration, one host machine is connected to multiple NVDs. Figure 3 shows an example in CSPI 300 where a host machine is connected to multiple NVDs. As shown in Figure 3, a host machine 302 includes a network interface card (NIC) 304 that includes multiple ports 306 and 308. The host machine 300 is connected to a first NVD 310 via port 306 and link 320, and to a second NVD 312 via port 308 and link 322. The ports 306 and 308 may be Ethernet ports, and the links 320 and 322 between the host machine 302 and the NVDs 310 and 312 may be Ethernet links. The NVD 310 is then connected to a first TOR switch 314, and the NVD 312 is connected to a second TOR switch 316. The links between the NVDs 310 and 312 and the TOR switches 314 and 316 may be Ethernet links. TOR switches 314 and 316 represent layer 0 switching devices in a multi-tier physical network 318 .

[0091] 3 provides two separate physical network paths between the physical switch network 318 and the host machine 302: a first path traversing the TOR switch 314 through the NVD 310 to the host machine 302, and a second path traversing the TOR switch 316 through the NVD 312 to the host machine 302. The separate paths result in improved availability (called high availability) of the host machine 302. If there is a problem with one of the paths (e.g., a link in one of the paths fails) or devices (e.g., a particular NVD is not functioning), the other path may be used for communication to and from the host machine 302.

[0092] In the configuration shown in Figure 3, the host machine is connected to two different NVDs using two different ports provided by the host machine's NIC. In other embodiments, the host machine may include multiple NICs allowing the host machine to be connected to multiple NVDs.

[0093] Referring again to Figure 2, an NVD is a physical device or component that performs one or more network and / or storage virtualization functions. An NVD may be any device that has one or more processing units (e.g., CPUs, Network Processing Units (NPUs), FPGAs, packet processing pipelines, etc.), memory including caches, and ports. Various virtualization functions may be performed by software / firmware executed by the one or more processing units of the NVD.

[0094] The NVD may be implemented in various forms. For example, in one embodiment, the NVD is implemented as an interface card called a smart NIC or intelligent NIC that contains an embedded processor. A smart NIC is a separate device from the NIC on the host machine. In FIG. 2, the NVDs 210 and 212 may be implemented as smart NICs connected to the host machine 202 and the host machines 206 and 208, respectively.

[0095] However, smart NICs are just one example of an implementation of the NVD. Various other implementations are possible. For example, in some other implementations, the NVD or one or more functions performed by the NVD may be incorporated in or performed by one or more host machines, one or more TOR switches, and other components of CSPI200. For example, the NVD may be embodied in a host machine, and the functions performed by the NVD are performed by the host machine. As another example, the NVD may be part of a TOR switch, or the TOR switch may be configured to perform the functions performed by the NVD, which allows the TOR switch to perform various complex packet transformations used in public clouds. A TOR that performs the functions of the NVD may be referred to as a smart TOR. In still other implementations where virtual machine (VM) instances rather than bare metal (BM) instances are provided to customers, the functions performed by the NVD may be implemented inside the hypervisor of the host machine. In some other implementations, some of the functions of the NVD may be offloaded to a centralized service running on a set of host machines.

[0096] In one embodiment, such as when implemented as a smart NIC as shown in FIG. 2, the NVD may comprise multiple physical ports that allow the NVD to be connected to one or more host machines and one or more TOR switches. Ports on the NVD may be classified as host-facing ports (also referred to as "south ports") or network-facing or TOR-facing ports (also referred to as "north ports"). The host-facing ports of the NVD are the ports used to connect the NVD to the host machines. Examples of host-facing ports in FIG. 2 include port 236 on the NVD 210, and ports 248 and 254 on the NVD 212. The network-facing ports of the NVD are the ports used to connect the NVD to the TOR switches. Examples of network-facing ports in FIG. 2 include port 256 on the NVD 210 and port 258 on the NVD 212. As shown in FIG. 2, the NVD 210 is connected to the TOR switch 214 using a link 228 that extends from port 256 of the NVD 210 to the TOR switch 214. Similarly, the NVD 212 is connected to the TOR switch 216 using a link 230 that extends from a port 258 of the NVD 212 to the TOR switch 216 .

[0097] The NVD may receive packets and frames (e.g., packets and frames generated by compute instances hosted by the host machine) from the host machine via a host-facing port, and after performing any necessary packet processing, may forward those packets and frames to the TOR switch via the NVD's network-facing port. The NVD may receive packets and frames from the TOR switch via the NVD's network-facing port, and after performing any necessary packet processing, may forward those packets and frames to the host machine via the NVD's host-facing port.

[0098] In an embodiment, there may be multiple ports and associated links between the NVD and the TOR switch. These ports and links may be aggregated to form a link aggregator group (called a link aggregator group (LAG)) of multiple ports or links. Link aggregation allows multiple physical links between two endpoints (e.g., between the NVD and the TOR switch) to be treated as a single logical link. All physical links within a particular LAG may operate in full-duplex mode at the same speed. LAGs help increase bandwidth and improve the reliability of the connection between two endpoints. If one of the physical links in the LAG fails, traffic is dynamically and transparently reassigned to one of the other physical links in the LAG. The aggregated physical link provides higher bandwidth than an individual link. Multiple ports associated with a LAG are treated as a single logical port. Traffic may be load-balanced across the multiple physical links of the LAG. One or more LAGs may be configured between two endpoints. Two endpoints may exist between the NVD and the TOR switch, between a host machine and the NVD, etc.

[0099] The NVD implements or performs network virtualization functions. These functions are performed by software / firmware executed by the NVD. Examples of network virtualization functions include, but are not limited to, packet encapsulation and decapsulation functions, functions for creating VCN networks, functions for enforcing network policies such as VCN security list (firewall) functions, functions for facilitating routing and forwarding of packets to and from compute instances in the VCN, etc. In an embodiment, upon receipt of a packet, the NVD is configured to execute a packet processing pipeline to process the packet and determine how the packet is to be forwarded or routed. As part of this packet processing pipeline, the NVD may perform one or more virtual functions associated with the overlay network, such as running VNICs associated with compute instances in the VCN, running virtual routers (VRs) associated with the VCN, encapsulating and decapsulating packets to facilitate forwarding or routing within the virtual network, running certain gateways (e.g., local peering gateways), enforcing security lists, network security groups, network address translation (NAT) functions (e.g., per-host public IP to private IP translation), bandwidth throttling functions, and other functions.

[0100] In one embodiment, the packet processing data path within the NVD may comprise multiple packet pipelines, each consisting of a series of packet transformation stages. In one implementation, upon receipt of a packet, the packet is parsed and sorted into a single pipeline. The packet is then processed in a linear fashion, one stage at a time, until the packet is either dropped or sent out via an interface of the NVD. These stages provide the basic functional packet processing building blocks (e.g., authenticate headers, perform bandwidth throttling, insert new layer 2 headers, perform L4 firewalling, VCN encapsulation / decapsulation, etc.), such that new pipelines can be constructed by assembling existing stages, and new functionality can be added by creating and inserting new stages into existing pipelines.

[0101] The NVD may perform both control plane and data plane functions corresponding to the control and data planes of the VCN. Examples of the VCN control plane are also shown in Figures 18, 19, 20, and 21 (see reference numbers 1816, 1916, 2016, and 2116) and described below. Examples of the VCN data plane are shown in Figures 18, 19, 20, and 21 (see reference numbers 1818, 1918, 2018, and 2118) and described below. The control plane functions include functions used to configure the network (e.g., set routes and route tables, configure VNICs, etc.) that control how data is forwarded. In one embodiment, a VCN control plane is provided that centrally computes mappings between all overlays and infrastructure and publishes those mappings to the NVD and to virtual network edge devices such as various gateways such as DRGs, SGWs, IGWs, etc. The same mechanism may be used to publish firewall rules. In one embodiment, the NVD retrieves only mappings that are relevant to the NVD. The data plane functions include functionality for the actual routing / forwarding of packets based on configuration settings using the control plane. The VCN data plane is implemented by encapsulating customer network packets before they traverse the underlying network. The encapsulation / decapsulation functions are implemented in the NVD. In one embodiment, the NVD is configured to intercept all network packets entering and leaving the host machine and perform network virtualization functions.

[0102] As indicated above, the NVD performs various virtualization functions including VNICs and VCN VRs. The NVD may execute VNICs associated with compute instances hosted by one or more host machines connected to the VNICs. For example, as shown in FIG. 2, the NVD 210 executes functions of the VNIC 276 associated with the compute instance 268 hosted by the host machine 202 connected to the NVD 210. As another example, the NVD 212 executes VNIC 280 associated with the bare metal compute instance 272 hosted by the host machine 206 and executes VNIC 284 associated with the compute instance 274 hosted by the host machine 208. The host machines may host compute instances belonging to different VCNs belonging to different customers, and the NVDs connected to the host machines may execute VNICs (i.e., execute functions related to the VNICs) corresponding to the compute instances.

[0103] NVDs also run VCN virtual routers corresponding to the VCNs of compute instances. For example, in the embodiment shown in FIG. 2, NVD 210 runs VCN VR 277 corresponding to the VCN to which compute instance 268 belongs. NVD 212 runs one or more VCN VRs 283 corresponding to one or more VCNs to which compute instances hosted by host machines 206 and 208 belong. In an embodiment, a VCN VR corresponding to that VCN is run by all NVDs connected to a host machine that hosts at least one compute instance that belongs to that VCN. If a host machine hosts compute instances that belong to different VCNs, the NVDs connected to that host machine may run VCN VRs corresponding to those different VCNs.

[0104] In addition to the VNICs and VCN VRs, the NVD may run various software (e.g., daemons) and may include one or more hardware components that facilitate various network virtualization functions performed by the NVD. For simplicity, these various components are grouped together as a "packet processing component" shown in FIG. 2. For example, the NVD 210 includes a packet processing component 286, and the NVD 212 includes a packet processing component 288. For example, the packet processing component of the NVD may include a packet processor configured to interact with the ports and hardware interfaces of the NVD to monitor all packets received by and communicated using the NVD and to store network information. The network information may include, for example, network flow information and per-flow information (e.g., per-flow statistics) that identify various network flows processed by the NVD. In an embodiment, the network flow information may be stored per VNIC. In addition to performing per-packet operations, the packet processor may implement a stateful NAT and an L4 firewall (FW). As another example, the packet processing component may include a replication agent configured to replicate information stored by the NVD to one or more distinct replication target stores. As yet another example, the packet processing component may include a logging agent configured to perform logging functions for the NVD. The packet processing component may also include software for monitoring the performance and health of the NVD, and possibly software for monitoring the status and health of other components connected to the NVD.

[0105] FIG. 1 illustrates components of an exemplary virtual or overlay network, including a VCN, a subnet in the VCN, compute instances deployed in the subnet, VNICs associated with the compute instances, VRs of the VCN, and a set of gateways configured for the VCN. The overlay components illustrated in FIG. 1 may be executed or hosted by one or more of the physical components illustrated in FIG. 2. For example, the compute instances in the VCN may be executed or hosted by one or more host machines illustrated in FIG. 2. For a compute instance hosted by a host machine, the VNICs associated with the compute instance are typically executed by the NVD connected to the host machine (i.e., the VNIC functionality is provided by the NVD connected to the host machine). The VCN VR functionality of the VCN is executed by all NVDs connected to the host machines that host or are running compute instances that are part of the VCN. The gateways associated with the VCN may be executed by one or more different types of NVDs. For example, some gateways may be executed by smart NICs, while other gateways may be executed by one or more host machines or other implementations of NVDs.

[0106] As previously mentioned, compute instances in a customer's VCN may communicate with various endpoints, which may be in the same subnet as the source compute instance, or in a different subnet within the same VCN as the source compute instance, or the endpoints may be outside the VCN of the source compute instance. These communications are facilitated using VNICs associated with the compute instances, VCN VRs, and gateways associated with the VCN.

[0107] For communication between two compute instances on the same subnet within a VCN, the communication is facilitated using a VNIC associated with the source compute instance and the destination compute instance. The source compute instance and the destination compute instance may be hosted by the same host machine or by different host machines. A packet originating from the source compute instance may be forwarded from the host machine hosting the source compute instance to an NVD connected to that host machine. In the NVD, the packet is processed using a packet processing pipeline, which may include execution of the VNIC associated with the source compute instance. Because the destination endpoint of the packet is in the same subnet, execution of the VNIC associated with the source compute instance causes the packet to be forwarded to an NVD running the VNIC associated with the destination compute instance, which then processes and forwards the packet to the destination compute instance. The VNICs associated with the source and destination compute instances may run on the same NVD (e.g., when the source and destination compute instances are both hosted by the same host machine) or on different NVDs (e.g., when the source and destination compute instances are hosted by different host machines connected to different NVDs). The VNICs may use routing / forwarding tables stored by the NVD to determine the next hop of a packet.

[0108] For packets traveling from a compute instance in a subnet to an endpoint in a different subnet in the same VCN, the packet originating from the source compute instance is traveled from the host machine hosting the source compute instance to the NVD connected to that host machine. In the NVD, the packet is processed using a packet processing pipeline, which may include the execution of one or more VNICs and VRs associated with the VCN. For example, as part of the packet processing pipeline, the NVD executes or invokes a function corresponding to the VNIC associated with the source compute instance (also referred to as executing the VNIC). The function executed by the VNIC may include examining the VLAN tag on the packet. Because the destination of the packet is outside the subnet, a VCN VR function is then invoked and executed by the NVD. The VCN VR then routes the packet to the NVD that is executing the VNIC associated with the destination compute instance. The VNIC associated with the destination compute instance then processes the packet and forwards the packet to the destination compute instance. The VNICs associated with the source compute instance and the destination compute instance may run on the same NVD (e.g., when the source compute instance and the destination compute instance are both hosted by the same host machine) or on different NVDs (e.g., when the source compute instance and the destination compute instance are hosted by different host machines connected to different NVDs).

[0109] If the packet's destination is outside the VCN of the source compute instance, the packet originating from the source compute instance is conveyed from the host machine hosting the source compute instance to the NVD connected to that host machine. The NVD runs the VNIC associated with the source compute instance. Because the packet's destination endpoint is outside the VCN, the packet is then processed by the VCN VR for that VCN. The NVD may invoke a VCN VR function to cause the packet to be forwarded to an NVD running an appropriate gateway associated with the VCN. For example, if the destination is an endpoint in the customer's on-premises network, the packet may be forwarded by the VCN VR to an NVD running a DRG gateway configured for the VCN. The VCN VR may be executed on the same NVD as the NVD running the VNIC associated with the source compute instance or by a different NVD. The gateway may be executed by the NVD, which can be a smart NIC, a host machine, or another NVD implementation. The packet is then processed by the gateway and forwarded to a next hop that facilitates the packet's conveyance to the intended destination endpoint. 2, a packet originating from compute instance 268 may be communicated from host machine 202 (using NIC 232) to NVD 210 via link 220. At NVD 210, VNIC 276 is invoked because it is the VNIC associated with source compute instance 268. VNIC 276 is configured to examine information encapsulated within the packet, determine a next hop for forwarding the packet in order to facilitate communication of the packet to an intended destination endpoint, and then forward the packet to the determined next hop.

[0110] Compute instances deployed in a VCN may communicate with various endpoints. These endpoints may include endpoints hosted by CSPI200 and endpoints outside of CSPI200. Endpoints hosted by CSPI200 may include instances in the same VCN or other VCNs, which may be the customer's VCN or VCNs not belonging to the customer. Communications between endpoints hosted by CSPI200 may be performed over physical network 218. Compute instances may communicate with endpoints not hosted by CSPI200 or outside of CSPI200. Examples of these endpoints include endpoints in a customer's on-premise network or data center, or public endpoints accessible over a public network such as the Internet. Communications with endpoints outside of CSPI200 may be performed over a public network (e.g., the Internet) (not shown in FIG. 2) or a private network (not shown in FIG. 2) using various communication protocols.

[0111] The architecture of CSPI 200 shown in FIG. 2 is merely an example and is not intended to be limiting. In alternative embodiments, variations, alternatives, and modifications are possible. For example, in some implementations, CSPI 200 may include more or fewer systems or components than those shown in FIG. 2, may combine two or more systems, or may include a different configuration or arrangement of systems. The systems, subsystems, and other components shown in FIG. 2 may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective systems, using hardware, or a combination thereof. The software may be stored in a non-transitory storage medium (e.g., in a memory device).

[0112] FIG. 4 illustrates a connection between a host machine and an NVD to provide I / O virtualization to support multitenancy functionality, according to an embodiment. As illustrated in FIG. 4, a host machine 402 runs a hypervisor 404 that provides a virtual environment. The host machine 402 runs two virtual machine instances, VM1 406 belonging to customer / tenant #1 and VM2 408 belonging to customer / tenant #2. The host machine 402 includes a physical NIC 410 that is connected to an NVD 412 via link 414. Each of the compute instances is connected to a VNIC that is executed by the NVD 412. In the embodiment of FIG. 4, VM1 406 is connected to VNIC-VM1 420, and VM2 408 is connected to VNIC-VM2 422.

[0113] 4, NIC 410 includes two logical NICs, logical NIC A 416 and logical NIC B 418. Each virtual machine is connected to and configured to function with its own logical NIC. For example, VM1 406 is connected to logical NIC A 416, and VM2 408 is connected to logical NIC B 418. Because of the logical NICs, each tenant's virtual machine has the assurance that it has its own host machine and NIC, even though host machine 402 includes only one physical NIC 410 that is shared by multiple tenants.

[0114] In one embodiment, each logical NIC is assigned its own VLAN ID. Thus, a particular VLAN ID is assigned to logical NIC A 416 for tenant #1, and another VLAN ID is assigned to logical NIC B 418 for tenant #2. When a packet is communicated from VM1 406, a tag assigned to tenant #1 is attached to the packet by the hypervisor, and the packet is then communicated from host machine 402 to NVD 412 via link 414. In a similar manner, when a packet is communicated from VM2 408, a tag assigned to tenant #2 is attached to the packet by the hypervisor, and the packet is then communicated from host machine 402 to NVD 412 via link 414. Thus, a packet 424 communicated from host machine 402 to NVD 412 has an associated tag 426 that identifies the particular tenant and associated VM. At the NVD, for a packet 424 received from a host machine 402, a tag 426 associated with the packet is used to determine whether the packet should be processed by VNIC-VM1 420 or VNIC-VM2 422. The packet is then processed by the corresponding VNIC. The configuration shown in Figure 4 allows each tenant's compute instance to be confident that it owns its own host machine and NIC. The setup shown in Figure 4 enables I / O virtualization to support multi-tenancy capabilities.

[0115] FIG. 5 illustrates a simplified block diagram of a physical network 500, according to an embodiment. The embodiment illustrated in FIG. 5 is structured as a Clos network. A Clos network is a particular type of network topology designed to provide connection redundancy while maintaining high bisection bandwidth and maximum resource utilization. A Clos network is a type of non-blocking multi-stage or multi-layer switching network, the number of stages or layers of which can be two, three, four, five, etc. The embodiment illustrated in FIG. 5 is a three-layer network, including layers 1, 2, and 3. A TOR switch 504 represents a layer 0 switch in a Clos network. One or more NVDs are connected to a TOR switch. A layer 0 switch is also referred to as an edge device of the physical network. A layer 0 switch is connected to a layer 1 switch, also referred to as a leaf switch. In the embodiment illustrated in FIG. 5, a set of "n" layer 0 TOR switches are connected to a set of "n" layer 1 switches, together forming a pod. Each layer 0 switch in a pod is interconnected to all layer 1 switches within the pod, but there are no switch connections between pods. In one implementation, the two pods are referred to as blocks. Each block is served by or connected to a set of "n" layer 2 switches (sometimes referred to as spine switches). There can be multiple blocks in a physical network topology. The layer 2 switches are then connected to "n" layer 3 switches (sometimes referred to as super spine switches). Communication of packets through the physical network 500 is typically performed using one or more layer 3 communication protocols. Typically, all layers of the physical network except the TOR layer have n-way redundancy, thus enabling high availability. Policies may be specified for pods and blocks to control the visibility of switches to each other within the physical network to enable scaling of the physical network.

[0116] A feature of Clos networks is that the maximum number of hops to reach from one tier 0 switch to another tier 0 switch (or from an NVD connected to a tier 0 switch to another NVD connected to a tier 0 switch) is fixed. For example, in a three-tier Clos network, a maximum of seven hops are required for a packet to reach from one NVD to another, with the source NVD and the target NVD connected to the leaf layer of the Clos network. Similarly, in a four-tier Clos network, a maximum of nine hops are required for a packet to reach from one NVD to another, with the source NVD and the target NVD connected to the leaf layer of the Clos network. Thus, the Clos network architecture maintains consistent latency throughout the network, which is important for intra- and inter-datacenter communications. Clos topologies scale horizontally and are cost-effective. The bandwidth / throughput capacity of the network can be easily increased by adding more switches (e.g., more leaf switches and spine switches) to various tiers and by increasing the number of links between switches at adjacent tiers.

[0117] In one embodiment, each resource in the CSPI is assigned a unique identifier called a Cloud Identifier (CID). This identifier is included as part of the resource's information and can be used to manage the resource, for example, via a console or via an API. An exemplary syntax for a CID is as follows: ocid1.<resource type>.<realm>.[region][.future use].<unique ID> Where: ocid1: A literal column that indicates the version of the CID. Resource Type: The type of resource (for example, instance, volume, VCN, subnet, user, group, etc.). Realm: The realm in which the resource resides. Example values ​​are "c1" for a commercial realm, "c2" for a government cloud realm, or "c3" for a federal cloud realm, etc. Each realm may have its own domain name. Region: The region the resource is in. This part may be blank if a region is not applicable to the resource. Future Use: Reserved for future use. Unique ID: The unique part of the ID. The format may vary depending on the type of resource or service.

[0118] Multi-Cloud Adoption FIG. 6 illustrates a simplified high-level diagram of a distributed environment 600 including multiple cloud environments offered by different cloud service providers (CSPs), according to an embodiment, including a particular cloud environment that provides specialized infrastructure that allows one or more cloud services offered by that particular cloud environment to be used by customers of the other cloud environments. As illustrated in FIG. 6, various cloud environments (also referred to as "clouds") may be offered by different cloud service providers (CSPs), with each cloud environment or cloud offering one or more cloud services to which one or more customers of that cloud environment may subscribe. The set of cloud services offered by the cloud environments offered by the CSPs may include one or more different types of cloud services, including, but not limited to, Software-as-a-Service (SaaS) services, Infrastructure-as-a-Service (IaaS) services, Platform-as-a-Service (PaaS) services, Database-as-a-Service (DBaaS), etc. Examples of cloud environments offered by various CSPs include Oracle® Cloud Infrastructure (OCI) offered by Oracle Corporation, Microsoft® Azure offered by Microsoft Corporation, Google Cloud™ offered by Google® LLC, Amazon Web Services (AWS®) offered by Amazon Corporation, etc. The set of cloud services offered by a particular cloud environment may differ from the set of cloud services offered by another cloud environment.

[0119] In a typical cloud environment, a CSP provides a cloud service provider infrastructure (CSPI) that is used to provide one or more cloud services offered by the cloud environment to customers. The CSPI provided by the CSP may include various types of hardware and software resources, including computational resources, memory resources, network resources, consoles for accessing cloud services, and the like. A customer of a cloud environment provided by a CSP may subscribe to one or more of the cloud services offered by the cloud environment. Various subscription models may be offered to customers by the CSP. After a customer subscribes to a cloud service offered by a cloud environment, one or more users may be associated with the subscribing customer, and these users may use the cloud services to which the customer subscribes. In one implementation, when a customer subscribes to a cloud service offered by a particular cloud environment, a customer account or customer tenancy is created for the customer. One or more users may then be associated with the customer tenancy, and these users may then use the services to which the customer subscribes under the customer tenancy. Information about the services a customer subscribes to, users associated with the customer's tenancy, etc. is typically stored within the cloud environment and associated with the customer's tenancy.

[0120] For example, three different cloud environments offered by three different CSPs are shown in FIG. 6. These cloud environments include cloud environment A (Cloud A) 610 offered by CSP A, cloud environment B (Cloud B) 640 offered by CSP B, and cloud environment C (Cloud C) 660 offered by CSP C. Cloud A 610 includes an infrastructure CSPI_A 612 offered by CSP A, which may be used to provide a set of services “Service A” 614 offered by Cloud A 610. One or more customers (e.g., Customer A1 616-1, Customer A2 616-2) may subscribe to one or more services from Service A 614 offered by Cloud A 610. One or more users 618-1 may be associated with Customer A1 616-1 and may use the services subscribed to by Customer A1 616-1 in Cloud A 610. In a similar manner, one or more users 618-2 may be associated with customer A2 616-2 and may use services subscribed to by customer A2 616-2 within cloud A 610. In various use cases, the services subscribed to by customer A1 616-1 may be different than the services subscribed to by customer A2 616-2.

[0121] 6, Cloud B 640 includes an infrastructure CSPI_B 642 provided by CSP B, which may be used to provide a set of services “Service B” 644 offered by Cloud B 640. One or more customers (e.g., Customer B1 646-1) may subscribe to one or more services from Service B 644. One or more users 648-1 may be associated with Customer B1 646-1 and can use the services to which Customer B1 646-1 has subscribed in Cloud B 640.

[0122] As shown in FIG. 6, Cloud C 660 includes an infrastructure CSPI_C 662 provided by CSP C, which may be used to provide a set of services “Services C” 664 offered by Cloud C 660. One or more customers (e.g., Customer C1 666-1) may subscribe to one or more services from Service C 664. One or more Users 668-1 may be associated with Customer C1 666-1 and can use the services subscribed to by Customer C1 666-1 in Cloud C 660. It should be noted that Service A 614, Service B 644, and Service C 664 can be different from each other.

[0123] In existing cloud implementations, each cloud provides a closed ecosystem to subscribing customers and associated users. As a result, customers and associated users of a cloud environment are restricted to using services offered by the cloud to which the customer subscribes. For example, customer B1 646-1 and its user 648-1 are restricted to using service B 644 offered by cloud B 640 and cannot use their account in cloud B 640 to access services from a different cloud environment, such as services from service A 614 offered by cloud A 610 or services from service C 664 offered by cloud C 660. The teachings described herein overcome this limitation. As described in this disclosure, various techniques are described that allow a link to be created between two cloud environments, thereby allowing services offered by a first cloud environment offered by a first CSP to be used by a customer (and associated users) of a second, different cloud environment offered by a second, different CSP, using the customer's account in the second cloud environment.

[0124] For example, in the embodiment shown in FIG. 6, infrastructure CSPI_A 612 provided by CSP A includes, in addition to other infrastructure 620, a specialized infrastructure 622 (referred to as multi-cloud enabling infrastructure 622 or MEI (multi-cloud enabling infrastructure) 622 or multi-cloud infrastructure 622) that enables one or more services 614 provided by cloud A to be used by customers and associated users of other clouds, such as clouds B 640 and C 660, using the customer's accounts in those other clouds. In one implementation, customers of clouds B and C do not need to open separate accounts with cloud A to use one or more of the services 614 provided by cloud A 610. Customer B1 646-1 and associated user 648-1 of cloud B 640 can use one or more services 614 provided by cloud A 610 using the customer's account or tenancy in cloud B 640. As another example, a customer C1 666-1 and associated user 668-1 of Cloud C 660 may use one or more services 614 provided by Cloud A 610 using the customer's account or tenancy in Cloud C 660.

[0125] In one implementation, the MEI 622 allows links to be created between Cloud A 610 and other clouds, which may be used by customers and associated users of the other clouds to access and utilize services offered by Cloud A 610. This is symbolically illustrated in FIG. 6 as a link 670 created between Cloud A 610 and Cloud B 640, and a link 672 created between Cloud A 610 and Cloud C 660. Through the link 670, customers of Cloud B 640 can access or use one or more services 614 offered by Cloud A 610. Similarly, through the link 672, customers of Cloud C 660 can access or use one or more services 614 offered by Cloud A 610.

[0126] There are various ways in which the MEI 612 may be implemented. In an embodiment, the MEI 612 may include components that allow links to different clouds to be established. For example, in FIG. 6, the MEI 622 includes an infrastructure component 624 responsible for enabling a link 670 with cloud B 640 and an infrastructure component 626 responsible for enabling a link 672 with cloud C 660. In a similar manner, the MEI 622 may include other components that enable and facilitate links with other clouds. In some implementations, the components of the MEI 622 may facilitate links with multiple different clouds.

[0127] There are multiple reasons why a customer of one cloud may want or desire to use a cloud service provided by a different cloud. Using FIG. 6 as an example, there are multiple reasons why a customer B1 646-1 of cloud B 640 may want to use a cloud service 614 provided by cloud A 610. In one use case situation, this may occur because cloud A 610 provides a cloud service with features not provided by cloud B 640. In another use case situation, clouds A and B may provide similar services, but the service provided by cloud A 610 may be better (e.g., more features / functionality, faster speed, etc.) than the corresponding service provided by cloud B 640. In yet another use case situation, a customer B1 646-1 of cloud B 640 may want to use a cloud service provided by cloud A 610 because the service is provided at a lower price than the cloud service provided by cloud B 640. In some cases, there may be geographic constraints or other reasons why a customer B1 646-1 of Cloud B 640 would like to use a cloud service offered by Cloud A 610. For example, Cloud A 610 may offer a desired service in a geographic region that is not serviced by Cloud B 640, or a particular service is not offered by Cloud B 640 in the geographic region where the customer wants the service. Multiple other use case situations are also possible as to why a customer of one cloud would like to use a service offered by a different cloud.

[0128] In an embodiment, the MEI 622 provides the capability and performs the functions to create a link between Cloud A 610 and another cloud, through which a user associated with a customer of the other cloud can access and use the services provided by Cloud A 610 from the other cloud itself in a seamless manner. For example, the MEI 622 enables a user 648-1 associated with a customer B1 646-1 of cloud 640 to access services from Service A 614 provided by Cloud A 610 in a seamless manner. In an implementation, a user interface (e.g., a console) may be provided that is accessible to the user 648-1 from within Cloud B 640, which allows the user to view a list of services 614 provided by Cloud A 610 and to select a particular service that the user 648-1 wishes to access. In response to the user's selection, the MEI 622 is responsible for performing a process to establish a link 670 between Clouds A and B to enable access to the requested service. The process to set up the link 670 is performed substantially automatically by the MEI 622. Customer B1 646-1 or associated user 648-1 does not have to worry about performing all the system, network, or other configuration changes required to facilitate the creation, maintenance, and use of the link 670 between clouds A 610 and B 640. There is no burden on users or customers in creating links between clouds. Using the techniques described in this disclosure, links are created in a fast and efficient manner.

[0129] The MEI 622 can use various techniques to create and use links that are seamless for users and customers, thus providing an improved user experience. In one implementation, the MEI 622 makes the user interface (e.g., graphical user interfaces (GUIs) and the like) and process flows that Customer B1 and associated users 648-1 interact with, such as to request services from Cloud A 610 and access requested services from Cloud A 610, substantially similar to the interfaces and process flows that customers / users experience in Cloud B 640. In this way, a customer or user who may be familiar with the interfaces and process flows of Cloud B 640 does not have to learn new interfaces and process flows to access services 614 from Cloud A 610. The MEI 622 may present different interfaces and process flows to users of different cloud environments. For example, a first set of user interfaces and process flows that are substantially similar to the user interfaces and flows of cloud B may be presented to a user from cloud B 640, while a different set of user interfaces and process flows that are substantially similar to the user interfaces and flows of cloud C may be presented to a user accessing cloud A 610 from cloud C 660. This is done to simplify, and therefore improve, the user's experience for accessing the services 614 of cloud A 610 from other clouds.

[0130] As another example, each cloud environment typically includes an identity management system configured to provide security to the cloud environment. The identity management system is configured to protect resources in the cloud environment, including resources provided by the CSP and resources of subscribing cloud customers that are deployed in the cloud environment. Functions performed by the identity management system include, for example, managing identity credentials (e.g., usernames, passwords, etc.) associated with the cloud's subscribing customers and associated users, using the identity credentials to regulate users' access to cloud resources and services based on authorization / access policies configured for the cloud environment, and other functions. Different clouds may use different identity management systems and associated technologies. For example, the identity management system and associated procedures in cloud A 610 may be completely different from the identity management system and associated procedures in cloud B 640, which may further be completely different from the identity management system and associated procedures in cloud C 660. In one implementation, despite these differences in identity management systems and associated procedures between different cloud environments, the techniques described herein enable a user associated with a customer of a first cloud to access cloud services provided by a different cloud using the same identity credentials associated with the customer and user in the first cloud.

[0131] For example, in the embodiment shown in FIG. 6, Cloud B 640 provided by CSP B may include an identity management system that assigns or allocates identity credentials to subscribing customers and associated users, such as Customer B1 646-1 and associated user 648-1. These identity credentials are associated with a tenancy created for Customer B1 646-1 in Cloud B 640. In one implementation, MEI 622 provided by Cloud A 610 allows User 648-1 associated with Customer B1 646-1 of Cloud B to access services from Service A 614 in Cloud A 610 using identity credentials associated with User 648-1 and Customer B1 646-1 in Cloud B 640. This significantly improves the user experience for User 648-1, as User 648-1 does not need to create new identity credentials specific to Cloud A 610 simply to access services 614 in Cloud A 610. The MEI 622 facilitates such access.

[0132] As an example, customer B1 of cloud B 640 may select to use a service such as database-as-a service (DBaaS) from the set of services 614 offered by cloud A 610. In response to such a selection, MEI 622 causes a link 670 to be automatically created between cloud A 610 and cloud B 640 to enable user 648-1 associated with customer B1 646-1 to use the DBaaS service offered by cloud A 610. The automatic establishment of link 670 is facilitated by MEI 622. After link 670 is established, user 648-1 may use the DBaaS service in cloud A 610 via cloud B 640. As part of using this service, user 648-1 may send a request to cloud A 610 via cloud B 640 to create a database resource. In response, CSPI_A 612 may create the requested database in cloud A 610. In an implementation, the created database may be provisioned in a virtual network (e.g., virtual cloud network or VCN) created for customer B1 in cloud A 610 and is accessible by user 648-1 via cloud B 640. User 648-1 may then send a request to cloud A 610 from cloud B 640 to use the provisioned database. These requests may include, for example, requests to write data to the database, update data stored in the database, delete data in the database, delete the database, create additional databases, etc. In some use cases, these requests may originate from user 648-1 via cloud B 640 or from services 644 provided by cloud B 640. In this manner, the MEI 622 provided by cloud A 610 allows users associated with customers of different clouds provided by different CSPs to seamlessly access services provided by cloud A 610.

[0133] The distributed environment 600 illustrated in FIG. 6 is merely an example and is not intended to unduly limit the scope of the claimed embodiments. Many variations, alternatives, and modifications are possible. For example, in alternative embodiments, the distributed environment 600 may include more or fewer cloud environments. The cloud environments may include more or fewer systems and components, or may have different configurations or arrangements of systems and components. The systems and components illustrated in FIG. 6 may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective systems, using hardware, or a combination thereof. The software may be stored in a non-transitory storage medium (e.g., in a memory device).

[0134] Multi-Cloud Control Plane (MCCP) FIG. 7 illustrates a high-level architecture of a multi-cloud control plane (MCCP) according to some embodiments. As illustrated in FIG. 7, the high-level architecture 700 includes a first cloud environment provided by a first cloud service provider (e.g., OCI) 720 and a second cloud environment provided by a second cloud service provider 710 (i.e., Azure). The first cloud environment 720 includes a first cloud infrastructure 720A that provides a number of services to users of the first cloud environment 720. Furthermore, the first cloud environment 720 includes a multi-cloud infrastructure 720B that provisions the ability to deliver services of the first cloud infrastructure 720A (e.g., Oracle Cloud Infrastructure (OCI)) to users of other cloud environments (e.g., second cloud infrastructure 710A). The multi-cloud infrastructure 720B allows users to access services on other clouds (e.g., Oracle PaaS services) with a user experience as close as possible to the user experience of the user's native cloud environment, while providing simple integration between the cloud environments.

[0135] The second cloud infrastructure 710A includes a second cloud portal 711, an active directory 712, a resource manager 713, a customer subscription 715, and a first cloud provider subscription 717. The second cloud portal 711 is a centralized access point where customers of the second environment 710 can log in and manage their cloud deployments and instances. Note that the second cloud portal may provide a choice of both monitoring and operational services provided by the second cloud infrastructure. The active directory 712 is a service provided by the second cloud infrastructure 710A that provides administrators with the ability to manage end-user identities and access privileges. The services may include core directory, access management, and identity protection. The resource manager 713 is a management layer that provides the deployment and management services of the second cloud infrastructure, i.e., users to perform operations (e.g., create, update, delete, etc.) on resources deployed in the customer subscription 715. Note that the customer subscription may also be referred to as a virtual network (VNET) in which the customer's applications are deployed and run. The subscription of the first cloud provider 717 in the second cloud infrastructure 710A includes Express Routes and Hub and Spoke VNETs that provision network connections (e.g., from on-premise locations, from external cloud environments) established with the second cloud infrastructure 710A. Note that such connections may not have to be routed over the public Internet, thereby providing users with more reliability, faster speeds, consistent latency, and higher security.

[0136] The first cloud infrastructure 720A includes a control plane 724, a customer tenancy 726, and a multi-cloud infrastructure 720B. The control plane 724 of the first cloud infrastructure 720A is the native control plane of the first cloud environment that provides management and orchestration across the cloud environments. In the control plane 724, configuration baselines are set, user and role access is provisioned, and applications exist so that they can run along with their associated services. The multi-cloud infrastructure 720B includes a multi-cloud platform data plane 722, and a multi-cloud platform data plane 728. As previously mentioned, the multi-cloud infrastructure 720B provisions for users of other cloud environments (e.g., the second cloud environment 710) to access services provided by the first cloud environment with a user experience as close as possible to that of the user's native cloud environment (e.g., the second cloud environment 710), while providing simple integration between the cloud environments.

[0137] The MCCP architecture of FIG. 7 further includes a multi-cloud console 721 (different from the second cloud portal 711) that allows users authenticated in the second cloud infrastructure 710 to perform control plane operations on resources of the first cloud infrastructure 720 exposed through the multi-cloud infrastructure 720B. In some implementations, as shown in FIG. 7, all user 705 requests sent to the multi-cloud console 721 are directed to the multi-cloud infrastructure included in the first cloud environment. It is understood that the user 705 can directly send requests (e.g., CRUD requests) related to resources provided by the first cloud infrastructure to the multi-cloud console 721. The multi-cloud console 721 is configured to process requests having a syntax or format similar to the syntax or format used locally in the second cloud infrastructure. In other words, the multi-cloud console 721 has a similar look and feel as well as uses similar terminology to the second cloud portal 711 included in the second cloud infrastructure 710A. In some implementations, a link (e.g., a web link) may be provided within the second cloud console 711 that directs the user to the multi-cloud console 721.

[0138] The multi-cloud infrastructure 720B included in the first cloud infrastructure 720A includes multiple microservices, such as an authority module 722A, a proxy module 722B, a platform services module 722C, a cloud link adapter 722D, a pool of adapters 722E including adapter 1, adapter 2, adapter 3, and adapter 4, and a network link adapter 722F. The pool of adapters 722E can include adapters such as an Exadata cloud services adapter, an autonomous database sharing adapter, an autonomous database dedicated adapter, and a virtual machine database adapter.

[0139] Each of the adapters included in the pool of adapters 722E is responsible for exposing a unique set of underlying resources (provided by the first cloud infrastructure 720A) to users of other cloud environments (e.g., the second cloud environment). In particular, each of the adapters in the pool of adapters 722E maps to a specific product or resource provided by the first cloud infrastructure 720A. Note that the actual resources are created by the native control plane 724 of the first cloud infrastructure. For example, with respect to database as a service (DBaaS), the DBaaS control plane included in the control plane 724 is configured to instantiate an exadatabase resource in a customer's tenancy 726 of the first cloud environment.

[0140] Incoming requests received by the multi-cloud infrastructure 720B are processed by the authority module 722A to perform authentication and access control. Each request includes a token associated with the user's account in the second cloud infrastructure. The authority module extracts this token and performs token authentication with the active directory 712 (i.e., the identity provider system of the second cloud infrastructure 710A). Upon successful authentication, the authority module 722A may check the role (i.e., set of permissions) associated with the user. Note that a role may be associated with one or more tasks / operations that are permitted for the role. According to one embodiment, the authority module 722A is responsible for authenticating incoming requests to the MCCP and for authorizing if the user is allowed to perform the requested operation based on the role associated with the token. In some implementations, the authority module 722A may perform the aforementioned authentication process by utilizing custom authentication capabilities of the service platform (i.e., the SPLAT associated with the first cloud infrastructure). SPLAT receives incoming requests and forwards them to the authority module 722A, which further parses the incoming request to determine an authorization decision and returns a success or failure message to SPLAT. On success, the request is passed by SPLAT to the routing proxy 722B, while on failure, SPLAT returns an error response directly to the caller.

[0141] The proxy module 722B (also referred to as a routing proxy module) included in the multi-cloud infrastructure 720B is a component that receives incoming requests from the multi-cloud console 721 and routes the requests to a particular adapter included in the pool of adapters 722E. According to one embodiment, the proxy module 722B receives a pre-authenticated request from a service platform (i.e., SPLAT) of the first cloud infrastructure and routes the request to an appropriate adapter based on route information included in the incoming request. In some implementations, the proxy module 722B extracts an identifier corresponding to a provider of the service (from the incoming request) and routes the request to an appropriate adapter in the pool of adapters 722E.

[0142] The cloud link adapter 722D included in the multi-cloud infrastructure 720B is responsible for handling lifecycle operations of resources provided by the first cloud infrastructure. The cloud link adapter 722D is configured to create a mapping (or a relationship created in the sign-up process) between the active directory tenants (and associated subscriptions) of the second cloud infrastructure and the user's corresponding tenancy / account in the first cloud infrastructure. In other words, the cloud link adapter 722D generates a mapping of a first identifier associated with the user's tenancy in the first cloud infrastructure to a second identifier associated with the user's account in the second cloud infrastructure.

[0143] In some implementations, the cloud link adapter 722D performs a translation between an external cloud identifier (e.g., a second identifier associated with the user's account in the second cloud infrastructure) and a first identifier (associated with the user's tenancy in the first cloud infrastructure) to enable operations passing through the multi-cloud control plane 722 to be mapped to the appropriate underlying resource in the first cloud infrastructure. In some embodiments, the cloud link adapter generates a data object to store such mapping information. In addition, the cloud link adapter 722D also generates a resource principal associated with the data object. The resource principal is assigned one or more permissions based on the token (and its associated role) included in the request. Access to downstream services provided by the first cloud infrastructure is achieved by the user from the second cloud infrastructure based on the resource principal. The cloud link adapter 722D may store the data object and the associated resource principal in a root partition of the user's tenancy in the first cloud infrastructure. Alternatively or additionally, the cloud link adapter 722D may maintain data objects and resource principals locally in the multi-cloud infrastructure's platform services module 722C for seamless access by other adapters included in the multi-cloud infrastructure.

[0144] The network link module (also referred to as a network adapter) 722F is responsible for creating a network link between a customer subscription 715 (in the second cloud infrastructure) and a corresponding customer tenancy / account 726 (in the first cloud infrastructure). In some embodiments, the network link module 722F obtains a token (from the platform services module 722C) and creates (1) a first peering relationship (in the first cloud environment) between the multi-cloud platform data plane 728 and the customer tenancy 726, and (2) a second peering relationship (in the second cloud environment) between the customer subscription 715 and the first cloud service provider subscription 717 contained in the second cloud infrastructure.

[0145] The network link module 722F may also be configured to establish a network connection between the first cloud environment and the second cloud environment, i.e., the network link module 722F may be configured to configure the interconnect 719 to communicatively couple the two cloud environments. It is understood that forming a network link between the two cloud environments allows applications running in the customer's subscription (e.g., within the VNET of the second cloud infrastructure) to access resources (e.g., exadatabase resources deployed in the customer's tenancy 726 of the first cloud infrastructure). Additionally, as shown in FIG. 7, within the tenancy 726, telemetry functionality is provided. Such functionality relates to maintaining logs, metrics, and other performance parameters associated with resources deployed in the customer's tenancy and their respective usage. According to some embodiments, the MCCP mirrors logs and metrics associated with different resources in the first cloud environment and exposes the metrics, logs, events, etc., for further processing, e.g., to a dashboard (e.g., to an application insights module included in the customer's subscription 715 of the second cloud environment).

[0146] In some embodiments, a platform services module 722C included in the multi-cloud infrastructure is configured to store authentication information associated with services of a first cloud infrastructure provided to a second cloud infrastructure. The platform services module 722C provides tokens / resource principals for various adapters, e.g., included in the pool of adapters 722E, so that the adapters can communicate with the native control plane 724 of the first cloud infrastructure. In some embodiments, the platform services module 722C exposes APIs that are called by the various adapters to perform tasks such as: · Selling minimally scoped access tokens (issued by the second cloud infrastructure) to adapters. For example, the network adapter 722F needs an access token to perform the network peering operations mentioned above. Provides a resource principal that the adapter uses to invoke the downstream service to create a resource in the customer's tenancy in the first cloud infrastructure. · Causes replication of observability data (logs, metrics, events) from the first cloud infrastructure to the second cloud infrastructure.

[0147] As mentioned above, the pool of adapters includes multiple adapters, each of which is responsible for exposing a unique set of underlying resources of the first cloud infrastructure to users of the second cloud infrastructure, i.e., each adapter maps to a specific product or resource provided by the first cloud environment. For example, the Exadatabase adapter serves as a proxy for users of the second cloud infrastructure to create and utilize Exadatabase resources. An Exadatabase is a pre-configured combination of hardware and software that provides the infrastructure for running a database. According to some embodiments, an Exadatabase comprises a stack of resources: (a) Exadata infrastructure (i.e., hardware), (b) VM cloud clusters, (c) container databases, and (d) pluggable databases. According to some embodiments, the multicloud infrastructure provides (to users of the second cloud infrastructure) the ability to analyze each of the stacked infrastructure levels. Additionally, the MCCP provides the flexibility for users to simply issue workflow creation commands (via the multicloud console 721), and the MCCP then executes the automated creation of individual resources at each level of the stack. 7, the pool of adapters 722F includes four different adapters, although it is understood that this in no way limits the scope of the MCCP architecture 700. The MCCP architecture may include other adapters, for example, specialized adapters intended for use by specific cloud service providers based on the cloud service provider's requirements.

[0148] 8A and 8B show an example flow diagram for linking two user accounts in different cloud environments according to some embodiments. The two user accounts may correspond to a first user account (e.g., tenancy) in a first cloud infrastructure and a second user account in a second cloud infrastructure. FIG. 8A shows a process of linking two user accounts, where first, the user's tenancy is created in the first cloud infrastructure and then linked to the user's account in the second cloud infrastructure. FIG. 8B shows a process of linking two user accounts, where the user's tenancy in the first cloud infrastructure has already been created, i.e., the tenancy already exists.

[0149] FIG. 8A illustrates a data flow that is executed when a user representing a customer, such as a system administrator, makes a call from a second cloud infrastructure to a multi-cloud infrastructure (included in the first cloud infrastructure) to sign up for a service offered in the first cloud infrastructure. A special console (referred to herein as a multi-cloud console) may be provided for a user of the second cloud infrastructure to open an account in the first cloud infrastructure and to link the user's account in the first cloud infrastructure to the user's account in the second cloud infrastructure. Note that linking accounts in the first and second cloud infrastructures allows the user (from the second cloud infrastructure) to use one or more services offered by the first cloud infrastructure. In one implementation, the user signs up for the service using the multi-cloud console. In response to the sign-up, a URL may be sent to the user that allows the user to log in to the multi-cloud console. The multi-cloud console exposes a UI and API similar to the UI and API of the second cloud infrastructure.

[0150] The multi-cloud console may provide various UIs (e.g., GUIs) that provide user-selectable options that enable a user of the second cloud infrastructure to open an account in the first cloud infrastructure and further link the user accounts in the first and second cloud infrastructures. For example, the multi-cloud console may provide a sign-up UI that enables a user to create an account / tenancy in the first cloud infrastructure and link the user's account (in the first cloud infrastructure) to the user's account in the second cloud infrastructure. After a process triggered in response to the sign-up / linking request is completed, the user's accounts in the respective cloud infrastructures are linked together. As part of this process, the multi-cloud console enables a user (e.g., a system administrator) to log into the second cloud infrastructure and (a) request the creation of the user's account (in the first cloud infrastructure) and link the user's account, or (b) in the case of an account the user already has in the first cloud infrastructure, request that the user's first cloud infrastructure account be linked to the user's second cloud infrastructure account.

[0151] Thus, in a use case, there are two possibilities: (a) the user already has an existing account or tenancy in the first cloud infrastructure, and the user is now requesting, via the multi-cloud console sign-up UI, that a link be created between the user's accounts (details regarding this process will now be described with reference to FIG. 8B), or (b) the user requests both the creation of a new account in the first cloud infrastructure and the linking of the newly created account with the user's account in the second cloud infrastructure. Details regarding this process will now be described with reference to FIG. 8A. It is understood that the linking of accounts allows the user to use resources in the first cloud via the second cloud. For example, via the second cloud, the user can request that resources be created or provisioned in the first cloud, utilize and manage resources in the first cloud, delete resources in the first cloud, etc.

[0152] As shown in FIG. 8A, in step 1, in response to a user's input provided to the multicloud console (e.g., the multicloud console's sign-up UI) requesting that a new account for the user be created in the first cloud infrastructure and linked to the user's account in the second cloud infrastructure, a call is made by the multicloud console to an account service (in the first cloud infrastructure) to create the new account. Note that this call includes a token generated by the second cloud infrastructure. The token generated by the second cloud infrastructure and included in the call to the account service enables the account or tenancy created in the first cloud infrastructure to be linked to the user's account in the second cloud infrastructure.

[0153] In step 2, as part of the process to set up a new account for the user, the account service may make a call to an authority / proxy module included in the multi-cloud control plane (e.g., authority module 722A included in the multi-cloud infrastructure of FIG. 7) to perform token authentication. The authority / proxy module may perform token authentication (as previously described with reference to FIG. 7), and processing may proceed further only upon successful token authentication. Upon successful token authentication, the account service creates a new account for the user in the first cloud infrastructure. As part of creating the new account, a native user may be created and associated with the newly created account. However, it is noted that the credentials of this native user are not used. Rather, the user's identity in the second cloud infrastructure is used to manage the account (from the second cloud infrastructure) and its resources in the first cloud infrastructure.

[0154] After the new account is successfully created by the account service in the first cloud infrastructure, in step 3, the account service makes a call to a cloud link adapter (included in the multi-cloud infrastructure) to create a link (also referred to as a cloud link) between the user's account in the second cloud infrastructure and the newly created account in the first cloud infrastructure. In one implementation, the cloud link adapter exposes APIs to the account service and to the multi-cloud console. These APIs may be called by the account service (or by the multi-cloud console) to request the linking of the accounts. Note that in some implementations, the call in step 3 does not include a token because the call to request the link is not made by the user, but rather is initiated by the account service. In this case, the cloud link adapter does not perform another authentication requiring another token because it trusts that the account service has already performed the necessary authentication for the user as part of the process to set up the account using the token received in step 1.

[0155] In some implementations, the processing performed by the cloud link adapter in step 3 to link the two accounts includes: (a) The cloud link adapter creates a data object (also referred to herein as a cloud link resource object) for storing metadata information identifying the two accounts being linked. For example, the data object stores metadata information including a mapping of a first identifier associated with a tenancy (i.e., an account) created in a first cloud infrastructure and a second identifier associated with a user's account with a second cloud service provider. (b) The cloud link adapter creates a new partition on the first cloud infrastructure side to store and contain the resource, and the user can / manages this partition using the multi-cloud console. In some implementations, the cloud link resource object is created at the root of the customer's account in the first cloud infrastructure. (c) The cloud link adapter creates a new resource principal (referred to herein as a cloud link resource principal) for the resource desired to be used by the user of the second cloud infrastructure. (d) The cloud link adapter may perform one or more federation configurations that facilitate linking a domain in the first cloud infrastructure to an active directory (e.g., active directory 712 of the second cloud infrastructure as shown in FIG. 7). The process of federating the accounts of the first cloud infrastructure and the second cloud infrastructure allows a user / group of users in the second cloud infrastructure to be authenticated in the first cloud infrastructure, and a user / group from the first cloud infrastructure to be authenticated into the second cloud infrastructure.

[0156] In one implementation, the authorization is associated with the cloud link resource principal based on the user's credentials / token in the account of the second cloud infrastructure. The consent to set the resource principal is provided by the user when the user requests a new account using the multi-cloud console or requests that the user's account in the first cloud infrastructure be linked to the user's account in the second cloud infrastructure. Furthermore, as shown in step 4, the cloud link resource principal is sent to the downstream service to enable the user to utilize the downstream service (e.g., one or more services provided by the first cloud infrastructure) from the second cloud infrastructure.

[0157] Referring to FIG. 8B, a process associated with linking an existing account or tenancy in a first cloud infrastructure to a user's account in a second cloud infrastructure is shown. As shown in FIG. 8B, a user requests that the user's accounts in the first cloud infrastructure and the second cloud infrastructure be linked via a sign-up UI provided by the multi-cloud console. In response, in step 1, the multi-cloud console makes a call to a cloud link adapter to link the accounts. Such a call may be made using one of the APIs exposed by the cloud link adapter in the sign-up UI. In one implementation, the information passed to the cloud link adapter in step 1 includes the user's account in the second cloud infrastructure and further information identifying the user's account in the first cloud infrastructure, a token issued by the second cloud infrastructure, etc.

[0158] In step 2, the cloud link adapter makes a call to an authority / proxy module included in the multi-cloud infrastructure to authenticate the token received in step 1. As part of this authentication, the authority / proxy module determines whether the user making the request has sufficient authority to make a link request on the second cloud infrastructure side. As part of this authentication, the role and permission settings on the second cloud infrastructure side may be checked. In response to successful authentication of the user, the cloud link adapter retrieves a resource principal associated with a resource desired to be utilized by the user from a data object previously created for the user. Further, as shown in step 3, the cloud link resource principal is sent to the downstream service to enable the user to utilize the downstream service (e.g., one or more services provided by the first cloud infrastructure) from the second cloud infrastructure.

[0159] FIG. 9 illustrates an exemplary system diagram illustrating components of a multi-cloud control plane (MCCP), according to some embodiments. The MCCP 900 includes a service platform (SPLAT) 910, a routing proxy 915, a cloud link adapter 920, a database adapter 925, a network adapter 930, and an MCCP platform 935. As described above with reference to FIG. 8A and FIG. 8B, once a user has successfully completed the sign-up process, the user may utilize the multi-cloud console 905 to issue commands to access, create, or update resources in the user's tenancy in the first cloud infrastructure. For purposes of explanation, the following describes a situation in which a user utilizes the multi-cloud console to issue a request to create an exadatabase resource.

[0160] In some implementations, a user accesses the multicloud console 905 and provides login information, e.g., authentication information of a user in a second cloud infrastructure. The multicloud console 905 provides a number of options, e.g., create a resource, access a resource, update a resource, etc. Such options may be provided to the user in the form of selectable icons (e.g., buttons) in the multicloud console 905. When the user performs a selection (e.g., to create a resource), an API call is triggered to the service platform 910. It is understood that the request made to the service platform 910 is not a native call with respect to the first cloud infrastructure. Rather, the call is a REST-style call that includes an authorization header that includes a token associated with the user in the second cloud infrastructure.

[0161] The REST call containing the token is further forwarded to a routing proxy module 915, which performs authentication and access control operations. According to some embodiments, the routing proxy module 915 performs authentication operations by extracting the token included in the REST call. The routing proxy module 915 performs authentication of the token by comparing the signature (used to sign the request) with the published signature of the second cloud infrastructure to ensure that the request originates from a valid customer associated with the second cloud infrastructure. Furthermore, the routing proxy module 915 may check the role, i.e., the privileges associated with the token, such as whether the role corresponds to an Exadata DB Administrator. Based on the role, the routing proxy module 915 may route the request to an appropriate adapter included in the MCCP framework 900.

[0162] According to one embodiment, the routing proxy module 915 compares the role (associated with the token) with a pre-configured list of roles published and assigned (as part of the API specification) to each of the adapters. For example, if the role associated with the token corresponds to "Exadata DB Administrator", the request may be understood as a request to create an Exadatabase, and the request is forwarded to the database adapter 925 accordingly. Additionally, according to some embodiments, the routing proxy module 915 may analyze information contained in the REST call, such as the provider ID, the type of resource requested, etc., and based on the analyzed information, the routing proxy module 915 may forward the request to the appropriate adapter.

[0163] In some implementations, the request obtained by the routing proxy module 915 may not include information identifying the tenancy of the user in the first cloud infrastructure where the resource should be deployed. Thus, the routing proxy module 915 communicates with the cloud link adapter 920 to obtain mapping information of the user's account in the second cloud infrastructure to the user's tenancy in the first cloud infrastructure. If the mapping information exists, the routing proxy module 915 obtains information about the user's tenancy in the first cloud infrastructure and passes this information to the database adapter 925. In this way, the database adapter 925 knows the tenancy of the user in the first cloud infrastructure where the resource should be created / deployed. However, if the cloud link adapter 920 determines that the mapping information does not exist, the routing proxy module 915 may simply issue a "not authorized access" message that is returned to the user as a response to the request to create the database resource.

[0164] Note that in some implementations, the cloud link adapter 920 creates a data object (referred to herein as a cloud link resource object) to store metadata information identifying the two accounts being linked. For example, the data object stores metadata information including a mapping of a first identifier associated with a tenancy (i.e., an account) in the first cloud infrastructure and a second identifier associated with the user's account with the second cloud service provider. Additionally, the cloud link adapter 920 also creates a resource principal (referred to herein as a cloud link resource principal) for a resource (e.g., a database) that is desired to be created / managed by the user in the second cloud infrastructure. The cloud link adapter 920 may maintain the data object and resource principal in the root partition of the user's tenancy in the first cloud infrastructure. In some embodiments, the cloud link adapter 920 may maintain the data object and / or resource principal locally in the MCCP platform 935.

[0165] In some embodiments, the database adapter 925 may communicate with (or direct the network adapter 930 to) create a network link between the user's account in the second cloud infrastructure and the user's tenancy in the first cloud infrastructure. For example, the network adapter 930 may obtain a token associated with the user from a native service in the second cloud infrastructure module 945 and create (1) a first peering relationship (in the first cloud environment) between the MCCP's data plane and the user's tenancy in the first cloud infrastructure, and (2) a second peering relationship (in the second cloud environment) between the user's account included in the second cloud infrastructure and the first cloud provider's subscription.

[0166] The network adapter 930 is also configured to establish a network connection between the first cloud infrastructure and the second cloud infrastructure, i.e., the network adapter 930 can configure an interconnect (e.g., interconnect 719 of FIG. 7) that communicatively couples the two cloud environments. It is understood that forming a network link between a user's tenancy in the first cloud infrastructure and a user's account in the second cloud infrastructure allows an application running in the user's subscription / account to access resources, e.g., an exadatabase deployed in the tenancy of the first cloud infrastructure. Additionally, the creation of the peering relationship provisions metrics, e.g., database usage metrics, to be accessible in the second cloud infrastructure, e.g., in a dashboard application running in the user's subscription in the second cloud infrastructure.

[0167] In some implementations, the database adapter 925 may obtain resource principals maintained locally within the MCCP platform 935. The database adapter 925 may send a request (including the resource principals) to one or more downstream services included in the first cloud infrastructure 940 to create resources in the user's tenancy in the first cloud infrastructure. In other words, the downstream services included in the first cloud infrastructure 940 utilize the identity, i.e., the resource principals obtained from the MCCP platform 935 to create / deploy the requested resource (e.g., an exadatabase in the user's tenancy in the first cloud infrastructure). Once the user issues a request to create an exadatabase, the user may intermittently poll the MCCP 900 to obtain the status of the request. Once the downstream services of the first cloud infrastructure 940 create the resources in the user's tenancy in the first cloud infrastructure and the network adapter 930 establishes the peering relationship, the MCCP 900 may notify the user regarding successful completion of the request.

[0168] Multi-Cloud Network Services - Network Adapters 7, the multi-cloud control plane 700 architecture enables a customer of an external cloud environment (provided by an external cloud service provider), e.g., a second cloud environment 710, to deploy resources (e.g., database resources), run services, etc., offered within the first cloud environment 720 by utilizing a multi-cloud console 721 and a multi-cloud infrastructure 720B (included in the first cloud environment). In some implementations, a network connection between the first cloud environment and the second cloud environment should be configured and maintained to expose the service offerings (e.g., PaaS offerings) of the first cloud environment to customers of the second cloud environment. As described in more detail below, a network adapter (e.g., network link component 722 of FIG. 7) is responsible for handling resources related to such a network.

[0169] In an embodiment, a multi-cloud network (MCN) service is provided that is responsible for configuring and maintaining network connectivity between a first cloud environment and another cloud environment, e.g., a second cloud environment. The MCN exposes services, such as PaaS offerings, in a seamless manner to customers of the other cloud environments. It is understood that the MCN service fits into the architecture of the multi-cloud control plane 700 of FIG. 7, such as the network adapter component 722F. According to some embodiments, the MCN service is responsible for: 1. Expose customer-facing API objects called Network Links that integrate with the multi-cloud platform and act as an abstraction for cross-cloud virtual network interconnections. 2. Configure and maintain a direct interconnect link (e.g. ExpressRoute, FastConnect, etc.) between the external cloud environment and the first cloud environment, shared among the N customers. 3. Launch and monitor per-customer packet processor instances to enable end-to-end packet flows between the external cloud environment and the first cloud environment. 4. Configure any domain name system (DNS) related resources within the external cloud environment and the first cloud environment to enable seamless name resolution.

[0170] As the name suggests, a network link is an abstraction that interconnects two cloud environments. For example, a network link may interconnect a customer's virtual network in a second cloud environment to a customer's virtual cloud network (VCN) in a first cloud environment. It is understood that if a customer needs to manually configure a network link, the customer is responsible for a number of tasks, such as configuring an interconnect (e.g., Express Route) for the second cloud environment, configuring another interconnect (e.g., Fast Connect) for the first cloud environment, configuring a dynamic routing gateway, configuring gateway connections and route tables, etc. Such tasks are in fact far from trivial and further cause poor customer satisfaction experiences. As described by the embodiments of the present disclosure, a multi-cloud network (MCN) service is provided that automatically configures a network link (and related network resources) with minimal or no customer interaction. Thus, by utilizing the MCN service, the customer does not need to worry about the internals of the network link components provided by the multi-cloud infrastructure of the first cloud environment, but simply uses the network link as a black box that enables traffic between the customer's virtual networks in the different cloud environments. Furthermore, it is noted that the network link established between the different cloud environments may need to have extremely low transmission latency. This is due to the fact that the resources deployed in the first cloud environment may be resources such as exadatabase resources that process online transaction processing events. Because such events require minimal latency, the multi-cloud network (MCN) service of the present disclosure is trusted to establish high-performance, highly available end-to-end network links with low transmission latency between the different cloud environments.

[0171] FIG. 10 illustrates a high-level block diagram of a network link component, according to an embodiment. In particular, FIG. 10 illustrates a network link component 1007 (also referred to herein as multi-cloud network infrastructure (MCNI)) established between a first cloud environment 1005 and a second cloud environment 1010. As illustrated in FIG. 10, the network link component 1007 is established between a customer virtual network (VNET), e.g., a VNET 1022 included in the second cloud environment 1010, and a customer virtual cloud network (VCN) 1012, i.e., a VNET in the first cloud environment 1005. Thus, a resource 1013 (e.g., an exadatabase) hosted in the customer's VCN 1012 (in the first cloud environment 1005) may be accessed / utilized by a virtual machine 1023 hosted in the customer's VNET 1022 in the second cloud environment 1010.

[0172] As shown in FIG. 10, on the second cloud environment side, the network link component 1007 includes a VNET 1019 (labeled as a leaf VNET) that peers with a customer's VNET 1022 included in the second cloud environment 1010. Any traffic to or from the customer's VCN 1012 is routed to this VNET peering. Note that the leaf VNET 1019 appears as a generic "network link VNET" to the customer. On the first cloud environment side, the network link component 1007 includes a dynamic routing gateway (DRG) connection named a multi-cloud connection 1017. In some implementations, a customer may connect a customer's VCN (e.g., the customer's VCN 1012) to a customer-owned DRG 1015 using a VCN connection, and the MCNI 1007 is linked to this DRG 1015 via the multi-cloud connection 1017. Note that any traffic to or from the customer's VNET 1022 (contained in the second cloud environment 1010) is routed to a VCN connection and then to a multi-cloud connection 1017 in the DRG.

[0173] As described below with reference to Figures 11-13, a portion of the network link component 1007 is included in the first cloud environment 1005, while another portion of the network link component 1007 is included in the second cloud environment 1010. It is further understood that one of the purposes of the network link component 1007 is to build a multi-tenant tunnel infrastructure to share the interconnection between the first cloud environment and the second cloud environment. It is noted that in order for the traffic of the multiple tenants to share a single inter-cloud interconnection, the network link component 1007 provisions tunnels to encapsulate / decapsulate customer traffic, for example by using GENEVE tunnels.

[0174] Referring to FIG. 11, a detailed architecture 1100 of a network link configuration is shown, according to an embodiment. As shown in FIG. 11, a network link communicatively couples a region of the second cloud environment 1105 to a region of the first cloud environment 1135. The region of the second cloud environment 1105 may include one or more customer VNETs. For example, the region of the second cloud environment 1105 includes two customer VNETs, ​​namely, customer 1 VNET 1101 and customer 2 VNET 1102. Thus, the portion of the second cloud environment 1105 hosting the customer VNETs is represented as customer domain 1150A. Similarly, the region of the first cloud environment 1135 may include one or more customer VCNs (i.e., customer virtual networks). For example, the region of the first cloud environment 1135 includes two customer VCNs, namely, customer 1 VCN 1131 and customer 2 VCN 1132. Thus, the portion of the first cloud environment 1135 that hosts the customer's VCN is represented as customer domain 1150C.

[0175] According to some embodiments, a multi-cloud network infrastructure (MCNI) domain 1150B, i.e., a network link, communicatively couples a pair of customer virtual networks. For example, as shown in FIG. 11, a VNET 1101 of customer 1 (contained in a region of the second cloud environment 1105) is communicatively coupled to a VCN 1131 of customer 1 (contained in a region of the first cloud environment 1135). In a similar manner, a VNET 1102 of customer 2 (contained in a region of the second cloud environment 1105) is communicatively coupled to a VCN 1132 of customer 2 (contained in a region of the first cloud environment 1135). Note that the MCNI domain 1150 corresponds to a network link domain and is located between the dotted lines 1160 and 1170. More specifically, the MCNI domain includes a first portion located in a region of the second cloud environment 1105 and a second portion located in a region of the first cloud environment 1135.

[0176] In some implementations, each pair of customer virtual networks (e.g., CUSTOMER 1's VNET 1101 in the second cloud environment 1105's region and CUSTOMER 1's VCN 1131 in the first cloud environment 1135's region) are communicatively coupled using multiple virtual networks (referred to herein as link-enabled virtual networks) deployed (in the first and second cloud environments) by the multi-cloud network (MCN) service. For example, a first link-enabled virtual network 1103 (labeled as Leaf 1 VNET) and a second link-enabled virtual network 1107 (labeled as Spoke 1 VNET) are deployed in the second cloud environment 1105's region and associated with CUSTOMER 1's VNET 1101. Additionally, a third link-enabled virtual network 1123 (labeled as Spoke 1 VCN) is deployed in the first cloud environment 1135's region and associated with CUSTOMER 1's VCN 1131. In a similar manner, Customer 2's VNET 1102 is associated with a link-enabled virtual network 1104 (labeled as Leaf 2 VNET) and another link-enabled virtual network 1106 (labeled as Spoke 2 VNET) deployed in a region of a second cloud environment 1105, while Customer 2's VCN 1132 is associated with a different link-enabled virtual network 1124 (labeled as Spoke 2 VCN). Thus, in the architecture of Figure 11, each pair of customer virtual networks is associated with three link-enabled virtual networks.

[0177] Further, the region of the second cloud environment 1105 includes a hub VNET 1110 shared between virtual networks of different customers included in the second cloud environment. In other words, the hub VNET 1110 handles traffic of tenancies of multiple customers included in the region of the second cloud environment 1105. Similarly, the region of the first cloud environment 1135 includes a hub VNET 1122 shared between virtual cloud networks of different customers included in the first cloud environment, i.e., the hub VNET 1122 handles traffic of VCNs of multiple customers included in the region of the first cloud environment 1135. As shown in FIG. 11, the region of the second cloud environment 1105 is communicatively connected to the region of the first cloud environment 1135 by a high-bandwidth network interconnect 1115. The following provides a detailed description of configuring an end-to-end network path between Customer 1's VNET 1101 (contained in the region of the second cloud environment 1105) and Customer 1's VCN 1131 (contained in the region of the first cloud environment 1135).

[0178] As shown in FIG. 11, a first link-enabled virtual network 1103 (i.e., Leaf 1 VNET) is peered to Customer 1's VNET 1101 to receive traffic from virtual machines (e.g., operated by User 1101A) running in Customer 1's VNET 1101. Because the hub VNET 1110 is shared among multiple customer's VNETs in the region of the second cloud environment 1105, in some implementations of the present disclosure, an encapsulation and tunnel framework is utilized to differentiate different customer's traffic originating from the customer's VNETs in the region of the second cloud environment 1105. The first link-enabled virtual network 1103 includes a pair of network adapters 1103A (referred to herein as remote virtual network adaptors (RVNAs)). Each of the network adapters 1103A included in the first link-enabled virtual network 1103 is configured to encapsulate traffic received from Customer 1's VNET 1101 to generate encapsulated traffic.

[0179] In some implementations, the pair of network adapters 1103A is used for high availability purposes and is maintained in an active-active state. In other words, each adapter included in the pair of network adapters 1103A is functional and encapsulates traffic received from VNET 1101 of customer 1. In some embodiments, the pair of network adapters 1103A may utilize services provided by the second cloud environment 1105 for performing BGP peering. The pair of network adapters 1103A may instruct such services to distribute traffic (originating from the customer's VNET) in a uniform manner by using mechanisms such as equal cost multi-path routing (ECMP). For example, considering that the second cloud environment 1105 corresponds to the Azure cloud (i.e., a cloud operated by Microsoft), the pair of network adapters 1103A may utilize the Azure route server (ARS) for performing BGP peering.

[0180] However, the use of such a service (e.g., ARS) of the second cloud environment imposes some limitations. For example, a service such as ARS can only learn and propagate routes within a directly peered VNET. This means that in order for the service to inject RVNA routes into a customer's VNET, the service (provided by the second cloud environment) must either reside in the first link-enabled virtual network (i.e., leaf VNET 1103) or in a separate VNET that is directly peered with the customer's VNET 1101. To prevent customer confusion from including peers of two VNETs in the customer's network, according to one embodiment of the present disclosure, the service (e.g., ARS) is preferably placed in the first link-enabled virtual network 1103.

[0181] According to some embodiments, certain limitations of the second cloud environment are that it prohibits route propagation through VNETs when both VNETs in the peering contain either a virtual network gateway (VNG) or a service (e.g., ARS) deployed within them. Because the hub VNET 1110 contains a VNG 1112 and the first link-enabled virtual network 1103 hosts a service (e.g., ARS), directly peering these two VNETs means that the route for the hub VCN 1122's CIDR that is next hopped to the VNG is not automatically in place, nor is it configurable by user defined routes (UDR). Therefore, an additional layer of network indirection is introduced between the first link-enabled virtual network 1103 and the hub VNET 1110. In particular, as shown in FIG. 11, to forward traffic from the first link-enabled virtual network 1103 to the hub VNET 1110, a second link-enabled virtual network 1107 is introduced as the next hop.

[0182] The second link-enabled virtual network 1107 includes a pair of virtual network forwarders (VNFs) 1107A. Each VNF included in the pair of VNFs 1107A is configured to forward encapsulated traffic received from the first link-enabled virtual network 1103 to a hub virtual network 1110 included in a region of the second cloud environment 1105. In some implementations, the pair of virtual network forwarders (VNFs) 1107A performs a network address translation (NAT) on the encapsulated traffic received from the first link-enabled virtual network 1103. In particular, the pair of virtual network forwarders (VNFs) 1107A performs a NAT operation on each data packet received from the first link-enabled virtual network 1103 such that the data packet is sent to a virtual network interface card (VNIC), e.g., VNIC 1122A included in the hub VCN 1122 of the region of the first cloud environment 1135. It is understood that a pair of virtual network forwarders (VNFs) 1107A forwards the traffic to a VNG 1112 contained in a hub VNET 1110, and then the traffic is transmitted to a hub VCN 1122 (contained in a region of the first cloud environment 1135) via a high-bandwidth interconnect 1115.

[0183] According to some embodiments, the encapsulated traffic received by a VNIC (e.g., VNIC 1122A) included in the hub VCN 1122 is forwarded to a third link-enabled virtual network 1123 (labeled as a spoke VCN) included in a region of the first cloud environment 1135. The third link-enabled virtual network 1123 includes a pair of virtual network adapters 1123A (labeled as local virtual network adaptors (LVNAs)), each of which is configured to decapsulate the encapsulated traffic received from the VNIC 1122A included in the hub VCN 1122. Further, as shown in FIG. 11 , the pair of virtual network adapters 1123A included in the third link-enabled virtual network 1123 of the first cloud environment 1135 sends the decapsulated traffic to Customer 1's VCN 1131 (e.g., to resources 1131A deployed in Customer 1's VCN 1131) via a dynamic routing gateway (DRG) connection.

[0184] In this manner, an end-to-end network link is established between Customer 1's VNET 1101 in the region of the second cloud environment 1105 and Customer 1's VCN 1131 in the region of the first cloud environment 1135 via the first link-enabled virtual network 1103, the second link-enabled virtual network 1107, the hub VNET 1110 (in the first cloud environment), the hub VCN 1122 (in the second cloud environment), and the third link-enabled virtual network 1123. It is understood that a network link may be established between Customer 2's VNET 1102 (in the region of the second cloud environment 1105) and Customer 2's VCN 1132 (in the region of the first cloud environment 1135) in a manner similar to that described above with respect to the network link established between Customer 1's VNET and Customer 1's VCN. Further, it is noted that although customer virtual networks in the first cloud environment and the second cloud environment may share IP address space, to prevent traffic collisions, each of the first link-enabled virtual network, the second link-enabled virtual network, the third link-enabled virtual network, and the hub virtual network included in the first cloud environment and the second cloud environment is assigned a unique classless inter-domain routing IP address space.

[0185] 12A and 12B show exemplary latency values ​​observed for the network link architecture of FIG. 11, according to an embodiment. In particular, the network link architecture described above with reference to FIG. 11 results in an observed latency (e.g., corresponding to a communication path between CUSTOMER 1's VNET and CUSTOMER 1's VCN as shown by the dotted line) to have a latency of 3 milliseconds. A breakdown of the observed latency between individual components on the aforementioned path is shown in FIG. 12A. Note that in the example shown in FIG. 12A, the RVNA and VNF are randomly placed within the second cloud environment. According to some embodiments, using a placement strategy in which the RVNA and VNF are placed according to certain conditions (as opposed to being randomly placed) improves the observed latency. For example, placing the RVNA and VNF (i.e., the RVNA and VNF associated with the customer's virtual network) in the same zone (e.g., availability zone of the second cloud environment) improves the observed latency. For example, as shown in Figure 12B, which provides a breakdown of the latency between individual components, it is observed that by placing the RVNA and the VNF in the same availability domain of the second cloud environment, the end-to-end latency is reduced from 3 ms to 1.6 ms.

[0186] FIG. 12C shows an exemplary flow chart illustrating a process of establishing a network link, according to an embodiment. The process shown in FIG. 12C may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective systems, hardware, or a combination thereof. The software may be stored in a non-transitory storage medium (e.g., in a memory device). The method presented in FIG. 12C and described below is intended to be exemplary and non-limiting. Although FIG. 12C shows various processing steps occurring in a particular sequence or order, this is not intended to be limiting. In an alternative embodiment, these steps may be performed in some different order, or some steps may be performed in parallel.

[0187] The process begins at step 1250, where a multi-cloud infrastructure included in a first cloud environment (e.g., multi-cloud infrastructure 720B of FIG. 7) receives a request to create a network link between a first virtual network in the first cloud environment (e.g., CUSTOMER 1 VCN 1131 of FIG. 11) and a second virtual network in a second cloud environment (e.g., CUSTOMER 1 VNET 1101 of FIG. 11). Note that the first virtual network in the first cloud environment has already been created to enable users associated with the customer's tenancy in the second cloud environment 1105 to access one or more services provided in the first cloud environment 1135. The process then proceeds to step 1260 to create a network link between the first virtual network and the second virtual network.

[0188] In step 1260, the process of creating the network link includes encapsulating, by the first link-enabled virtual network in the second cloud environment, traffic being received from the second virtual network to generate encapsulated traffic. For example, referring to Figure 11, the first link-enabled virtual network 1103 including the RVNA encapsulates traffic received from VNET 1101 of Customer 1 (step 1263).

[0189] Further, in step 1265, a second link-enabled virtual network in the second cloud environment (e.g., link-enabled virtual network 1107 in FIG. 11 ) forwards (e.g., by a VNF) the encapsulated traffic received from the first link-enabled virtual network to a hub virtual network (e.g., Hub VNET 1110) included in the second cloud environment. The process then moves to step 1267, where a third link-enabled virtual network in the first cloud environment (e.g., link-enabled virtual network 1123 in FIG. 11 ) decapsulates (e.g., by an LVNA) the encapsulated traffic received from a hub virtual network, e.g., Hub VCN 1122. Further, in step 1269, the third link-enabled virtual network in the first cloud environment transmits the decapsulated traffic to the first virtual network in the first cloud environment (e.g., Customer 1's VCN 1131 in FIG. 11 ) via a DRG connection. In this way, the multi-cloud infrastructure of the present disclosure creates high-performance, highly available, low-latency network links between virtual networks of different customers.

[0190] Referring to FIG. 13, a detailed architecture 1300 of a network link configuration is shown, according to an embodiment. As shown in FIG. 13, a network link communicatively couples a region of the second cloud environment 1305 to a region of the first cloud environment 1335. The region of the second cloud environment 1305 may include one or more customer VNETs. For example, the region of the second cloud environment 1305 includes two customer VNETs, ​​namely, customer 1 VNET 1301 and customer 2 VNET 1302. Thus, the portion of the second cloud environment 1305 hosting the customer VNETs is represented as customer domain 1350A. Similarly, the region of the first cloud environment 1335 may include one or more customer VCNs (i.e., customer virtual networks). For example, the region of the first cloud environment 1335 includes two customer VCNs, namely, customer 1 VCN 1331 and customer 2 VCN 1332. Thus, the portion of the first cloud environment 1335 that hosts the customer's VCN is represented as customer domain 1350C.

[0191] According to some embodiments, a multi-cloud network infrastructure (MCNI) domain 1350B, i.e., a network link, communicatively couples a pair of customer virtual networks. For example, as shown in FIG. 13, a VNET 1301 of customer 1 (contained in a region of the second cloud environment 1305) is communicatively coupled to a VCN 1331 of customer 1 (contained in a region of the first cloud environment 1335). In a similar manner, a VNET 1302 of customer 2 (contained in a region of the second cloud environment 1305) is communicatively coupled to a VCN 1332 of customer 2 (contained in a region of the first cloud environment 1335). Note that the MCNI domain 1350 corresponds to a network link domain and is located between the dotted lines 1360 and 1370. More specifically, the MCNI domain includes a first portion located in a region of the second cloud environment 1305 and a second portion located in a region of the first cloud environment 1335.

[0192] In some implementations, each pair of customer virtual networks (e.g., CUSTOMER 1's VNET 1301 in the second cloud environment 1305's region and CUSTOMER 1's VCN 1331 in the first cloud environment 1335's region) are communicatively coupled using multiple virtual networks (referred to herein as link-enabled virtual networks) that are deployed (in the first and second cloud environments) by a multi-cloud network (MCN) service. For example, a first link-enabled virtual network 1303 (labeled as Leaf 1 VNET) is deployed in the second cloud environment 1305's region. A second link-enabled virtual network 1323 (labeled as Spoke 1 VCN) is deployed in the first cloud environment 1335's region. In a similar manner, Customer 2's VNET 1302 is associated with a link-enabled virtual network 1304 (labeled as Leaf 2 VNET) and another link-enabled virtual network 1324 (labeled as Spoke 2 VCN) that are deployed in a region of the second cloud environment 1305 and a region of the first cloud environment 1335, respectively. Thus, in the architecture of Figure 13, each pair of customer virtual networks is associated with two link-enabled virtual networks.

[0193] Further, the region of the second cloud environment 1305 includes a hub VNET 1310 shared between virtual networks of different customers included in the second cloud environment. In other words, the hub VNET 1310 handles traffic of tenancies of multiple customers included in the region of the second cloud environment 1305. Similarly, the region of the first cloud environment 1335 includes a hub VCN 1322 shared between virtual cloud networks of different customers included in the first cloud environment, i.e., the hub VCN 1322 handles traffic of VCNs of multiple customers included in the region of the first cloud environment 1335. As shown in FIG. 13, the region of the second cloud environment 1305 is communicatively connected to the region of the first cloud environment 1335 by a high-bandwidth network interconnect 1315. The following provides a detailed description of configuring an end-to-end network path between Customer 1's VNET 1301 (contained in the region of the second cloud environment 1305) and Customer 1's VCN 1331 (contained in the region of the first cloud environment 1335).

[0194] As shown in FIG. 13, a first link-enabled virtual network 1303 (i.e., Leaf 1 VNET) is peered to Customer 1's VNET 1301 to receive traffic from virtual machines operated in Customer 1's VNET 1301 (e.g., operated by User 1301A). Because the hub VNET 1310 is shared among multiple customer's VNETs in the region of the second cloud environment 1305, in some implementations of the present disclosure, an encapsulation and tunnel framework is utilized to differentiate different customer's traffic originating from the customer's VNETs in the region of the second cloud environment 1305. The first link-enabled virtual network 1303 includes a pair of network adapters 1303A (referred to herein as remote virtual network adapters (RVNAs)). Each of the network adapters 1303A included in the first link-enabled virtual network 1303 is configured to encapsulate traffic received from Customer 1's VNET 1301 to generate encapsulated traffic.

[0195] In some implementations, the pair of network adapters 1303A is used for high availability purposes and is maintained in an active-active state. In other words, each adapter included in the pair of network adapters 1303A is functional and encapsulates traffic received from customer 1's VNET 1301. In some embodiments, the pair of network adapters 1303A may utilize services provided by the second cloud environment 1305 for the purpose of performing BGP peering. The pair of network adapters 1303A may instruct such services to distribute traffic (originating from the customer's VNET) in a uniform manner by using mechanisms such as equal-cost multipath routing (ECMP).

[0196] As shown in FIG. 13 , the hub VNET 1310 includes a pair of virtual network forwarders (VNFs) 1306 and 1307. A pair of VNFs is associated with each customer's VNET included in the region of the second cloud environment 1305. For example, VNF pair 1307 is associated with customer 1's VNET 1301, and VNF pair 1306 is associated with customer 2's VNET 1301. Each VNF included in the VNF pair 1307 is configured to forward encapsulated traffic received from the first link-enabled virtual network 1303 (i.e., from a corresponding RVNA) to a hub virtual network 1322 included in the region of the first cloud environment 1335.

[0197] In some implementations, the pair of virtual network forwarders (VNFs) 1307 performs network address translation (NAT) on encapsulated traffic being received from the first link-enabled virtual network 1303. In particular, the pair of virtual network forwarders (VNFs) 1307 performs a NAT operation on each data packet being received from the first link-enabled virtual network 1303 such that the data packet is sent to a virtual network interface card (VNIC), e.g., VNIC 1322A, included in the hub VCN 1322 of the region of the first cloud environment 1335. It is understood that the pair of virtual network forwarders (VNFs) 1307 forwards the traffic to a virtual network gateway (VNG) 1312 included in the hub VNET 1310, after which the traffic is conveyed to the hub VCN 1322 (included in the region of the first cloud environment 1335) via the high bandwidth interconnect 1315.

[0198] According to some embodiments, the encapsulated traffic received by a VNIC (e.g., VNIC 1322A) included in the hub VCN 1322 is forwarded to a second link-enabled virtual network 1323 (labeled as a spoke VCN) included in a region of the first cloud environment 1335. The second link-enabled virtual network 1323 includes a pair of virtual network adapters 1323A (labeled as a local virtual network adapter (LVNA)), each of which is configured to decapsulate the encapsulated traffic received from the VNIC 1322A included in the hub VCN 1322. Further, as shown in FIG. 13 , the pair of virtual network adapters 1323A included in the second link-enabled virtual network 1323 of the first cloud environment 1335 sends the decapsulated traffic to Customer 1's VCN 1331 (e.g., to resources 1331A deployed in Customer 1's VCN 1331) via a dynamic routing gateway (DRG) connection.

[0199] In this manner, an end-to-end network link is established between CUSTOMER 1's VNET 1301 in the region of the second cloud environment 1305 and CUSTOMER 1's VCN 1331 in the region of the first cloud environment 1335 via the first link-enabled virtual network 1303, the second link-enabled virtual network 1323, the hub VNET 1310 (in the first cloud environment), and the hub VCN 1322 (in the second cloud environment). It is understood that a network link may be established between CUSTOMER 2's VNET 1302 (in the region of the second cloud environment 1305) and CUSTOMER 2's VCN 1332 (in the region of the first cloud environment 1335) in a manner similar to that described above with respect to the network link established between CUSTOMER 1's VNET and CUSTOMER 1's VCN. Further, it is noted that although customer virtual networks in the first cloud environment and the second cloud environment may share IP address space (e.g., have overlapping IP address space), to prevent traffic collisions, each of the first link-enabled virtual network, the second link-enabled virtual network, and the hub virtual network included in the first cloud environment and the second cloud environment is assigned a unique classless inter-domain routing IP address space.

[0200] FIG. 14A illustrates an exemplary observed latency breakdown of the interconnect sharing structure of FIG. 13 according to some embodiments. In this configuration, end-to-end latency (for exemplary Route 1 from Customer 1's VNET to Customer 1's VCN) is observed to be 950 microseconds. Thus, the present disclosure provides strategic placement of packet processors, e.g., VNFs placed in the hub VNET to achieve low latency advantageous for supporting high throughput latency sensitive applications, such as exadatabase applications. Note that in the above embodiment, the latency measurements correspond to a particular configuration of customer's VNET and customer's VCN, i.e., within availability zones and regions.

[0201] FIG. 14B shows another exemplary flow chart illustrating a process of establishing a network link, according to an embodiment. The process shown in FIG. 14B may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective systems, hardware, or a combination thereof. The software may be stored in a non-transitory storage medium (e.g., in a memory device). The method presented in FIG. 14B and described below is intended to be exemplary and non-limiting. Although FIG. 14B shows various processing steps occurring in a particular sequence or order, this is not intended to be limiting. In an alternative embodiment, these steps may be performed in some different order, or some steps may be performed in parallel.

[0202] The process begins at step 1450, where a multi-cloud infrastructure included in a first cloud environment (e.g., multi-cloud infrastructure 720B of FIG. 7) receives a request to create a network link between a first virtual network in the first cloud environment (e.g., CUSTOMER 1 VCN 1331 of FIG. 13) and a second virtual network in a second cloud environment (e.g., CUSTOMER 1 VNET 1301 of FIG. 13). Note that the first virtual network in the first cloud environment has already been created to enable users associated with the customer's tenancy in the second cloud environment 1305 to access one or more services provided in the first cloud environment 1335. The process then proceeds to step 1455 to create a network link between the first virtual network and the second virtual network using multiple link-enabled virtual networks.

[0203] The process then moves to step 1460, where a first link-enabled virtual network from the plurality of link-enabled virtual networks is located in the second cloud environment. For example, referring to FIG. 13, the first link-enabled virtual network 1303 is deployed in a region of the second cloud environment to peer with the customer's VNET. The process then located / deployed in step 1465 a second link-enabled virtual network from the plurality of link-enabled virtual networks in the first cloud environment. For example, referring to FIG. 13, the second link-enabled virtual network 1323 is deployed in a region of the first cloud environment to peer with the customer's VCN via a DRG connection. In this manner, the multi-cloud infrastructure of the present disclosure configures high-performance, highly available, low-latency network links between virtual networks of different customers.

[0204] FIG. 15 illustrates an architecture of a service infrastructure 1500 including a multi-cloud network service control plane (MCNS-CP) according to an embodiment. The MCNS-CP is responsible for configuring network links such as those described above with reference to FIG. 11 and FIG. 13. As illustrated in FIG. 15, the service architecture 1500 includes a region of a second cloud environment 1501 and a region of a first cloud environment 1551 that may be communicatively coupled to each other via a high-bandwidth interconnect (e.g., Express Route or Fast Connect). The region of the second cloud environment 1501 includes a customer VNET 1503, a spoke VNET 1505, a leaf VNET 1507, a hub VNET 1509, and a multi-cloud service VNET 1510. The customer VNET 1503 corresponds to a customer domain, while the spoke VNET 1505 and the leaf VNET 1507 correspond to service domains instantiated per customer. It is understood that the hub VNET 1509 and the multi-cloud service VNET 1510 correspond to parts of a multi-tenanted service plane.

[0205] Leaf VNET 1507 includes one or more virtual network adapters (e.g., RVNA 1507A) configured to peer with customer VNET 1503 and encapsulate / decapsulate traffic received from customer VNET 1503. Spoke VNET 1505 includes one or more virtual network forwarders (e.g., VNF 1505A) configured to forward traffic received from leaf VNET 1507 to hub VNET 1509. Furthermore, each of leaf VNET 1507 and spoke VNET 1505 includes private endpoints, through which network configuration information may be provided to RVNA and VNF, respectively. For example, leaf VNET 1507 includes private endpoint 1507B, spoke VNET 1505 includes private endpoint 1505B, through which network configuration information (i.e., mapping information) may be obtained. Details regarding distribution of network configuration information are further described below.

[0206] The hub VNET 1509 includes a virtual network gateway (VNG) communicatively coupled to a high-bandwidth interconnect to couple the second cloud environment to the first cloud environment. In some implementations, the hub VNET 1509 may include a Bastion 1509A, a fully platform-managed PaaS service, provisioned within the virtual network. In particular, the Bastion 1509 is a service that allows connecting to virtual machines, for example, by using a browser, via native SSH, etc. The multi-cloud service VNET 1510 includes an internal load balancer (ILB) 1510B and an outpost 1510A. The combination of the outpost 1510A and the ILB 1510B is used to distribute network configuration information (received from the first cloud environment) to the private endpoints 1505B and 1507B. In this manner, VNF 1505A and RVNA 1507A can receive network configuration information from their respective private endpoints, i.e., private endpoint 1505B and private endpoint 1507B.

[0207] A region of a first cloud environment 1551 includes a customer VCN 1557 that hosts resources 1558 (e.g., an exadatabase), a spoke VCN 1555, a hub VCN 1553, and a multicloud service VCN 1560. The customer VCN 1557 corresponds to a customer domain, while the spoke VCN 1555 corresponds to a service domain instantiated per customer. It is understood that the hub VCN 1553 and the multicloud service VCN 1560 correspond to parts of a multi-tenanted service plane.

[0208] The spoke VCN 1555 includes one or more virtual network adapters (e.g., LVNA 1555A) configured to peer with the customer's VCN 1557 (e.g., via a DRG connection) and encapsulate / decapsulate traffic received from the customer's VCN 1557. The spoke VCN 1555 includes a VNIC connection and communicates with the multicloud service VCN 1560 via the VNIC connection. For example, the LVNA 1555A included in the spoke VCN 1555 connects to a VNIC included in the multicloud service VCN 1560 to receive network configuration information. The spoke VCN 1555 is further communicatively coupled to the hub VCN 1553 via another VNIC connection 1553B included in the hub VCN 1553. The hub VCN 1553 includes a DRG and via the DRG the hub VCN 1553 is coupled to a high bandwidth interconnect that couples the first cloud environment to the second cloud environment.

[0209] According to some embodiments, the multi-cloud service VCN 1560 corresponds to a stack of control planes built in the first cloud environment 1551. In particular, the multi-cloud service is composed of multiple network resources, such as VCNs, route tables, DRGs, connections, VNETs, ​​instances, VNICs, peerings, etc. The multi-cloud service VCN 1560 includes a load balancer 1560A, one or more API servers 1560B, a key-value store 1560 (labeled as a KV store), a distribution service 1560D, a workflow-as-a-service (WFaaS) worker 1560F, and a gateway 1560G. In some implementations, the one or more API servers 1560B receive a request to set up a network link between the first cloud environment and the second cloud environment. In response, the one or more API servers 1560B may start a worker (included in the WFaaS worker 1560F). In some implementations, the WFaaS worker 1560 initiates communication with the control planes of the first and second cloud environments, deploys the required network resources, and provisions the network links, e.g., launches link-enabled virtual networks in the first and second cloud environments, and peers the network link connections. One such exemplary process performed by the WFaaS worker 1560F is described below with reference to FIG. 16A. The one or more API servers 1560B are configured to manage CIDR address assignments to various components of the multi-cloud service.

[0210] The distribution service 1560D is a mapping service that receives requests (e.g., mapping requests) from the RVNAs, VNFs, and LVNAs. The distribution service 1560D responds to such mapping requests by providing network configuration information requested by the RVNAs, VNFs, and LVNAs to perform encapsulation / decapsulation of network packets (i.e., traffic). For example, in one implementation, the RVNAs, VNFs, and LVNAs may periodically poll the distribution service 1560D to obtain their respective network configuration information. The distribution service 1560D provides the network configuration information to the LVNAs 1555A (located in the spoke VCNs 1555) via a VNIC connection.

[0211] Further, the multi-cloud service VCN 1560 may set up a communication channel, e.g., a tunnel (e.g., an IPsec tunnel), with the multi-cloud service VNET 1510 to transmit the network configuration information to the Outpost 1510A (included in the multi-cloud service VNET 1510) so that the network configuration information can eventually be retrieved (via the respective private endpoints) by the VNFs 1505A and RVNAs 1507A, each of which is located in the second cloud environment. In particular, polling requests of the VNFs and RVNAs are received (via the respective endpoints 1505B and 1507B) by the Outpost 1510A. The Outpost 1510A then redirects such requests to the distribution service 1560D. In response to receiving the polling requests, the distribution service 1560D provides the corresponding network configuration information to the VNFs and RVNAs. In some implementations, the distribution service 1560D retrieves the network configuration information from the KV store 1560 and further distributes the retrieved information to various packet processors (i.e., VNFs, RVNAs, and LVNAs). It is understood that the distribution service 1560D may receive information indicative of a health status corresponding to each of the RVNAs, LVNAs, and VNFs in polling requests issued by the respective packet processors. Further, the multi-cloud services VCN 1560 includes a gateway 1560G, which is used to provision / enable calls / requests made by the multi-cloud services VCN 1560 to external parties, e.g., calls made to a second cloud environment, calls made to public IP addresses, etc.

[0212] 16A, a swim diagram is shown illustrating the interactions of a multi-cloud service control plane with different cloud environments, according to an embodiment. The swim diagram in FIG. 16A illustrates interactions between entities: a client 1601, a multi-cloud platform control plane 1602, an IaaS API for a second cloud environment 1603, and an IaaS API for a first cloud environment 1604.

[0213] 16A, in step S1, a request issued by a client 1601 to create a network link is received by the multi-cloud platform control plane 1602. In some implementations, the multi-cloud platform control plane 1602 receives the request and returns an acknowledgment to the client 1601 (step S2). The client 1601 may then poll the multi-cloud platform control plane 1602 to obtain the status of the request.

[0214] In step S3, upon receiving the request to create a network link, the multicloud platform control plane 1602 initiates a workflow, for example using a WFaaS worker (1560F in FIG. 15 ). In step S4, the multicloud platform control plane 1602 sends a first request to an IaaS API corresponding to the first cloud environment 1604. Note that such a request may be sent to a first endpoint of the first cloud environment (e.g., the first endpoint may correspond to a resource manager of the first cloud environment). The first request corresponds to a request being issued by the multicloud platform control plane 1602 and requests provisioning of a first set of resources in the first cloud environment. The first set of resources may correspond to the creation of an LVNA, a hub VCN, a spoke VCN, a VNIC, etc. It is understood that in some implementations, the first request may include a template that requests the first endpoint to collectively instantiate all required resources in the first cloud environment.

[0215] In step S5, the multicloud platform control plane 1602 sends a second request to an IaaS API corresponding to the second cloud environment 1603. Note that such request may be sent to a second endpoint of the second cloud environment (e.g., the second endpoint may correspond to a resource manager of the second cloud environment). The second request corresponds to a request being issued by the multicloud platform control plane 1602 and requests provisioning of a second set of resources in the second cloud environment. The second set of resources may correspond to the creation of RVNAs, VNFs, hub VNETs, ​​spoke VNETs, ​​service endpoints, network peering, etc. It is understood that in some implementations, the second request may include a template requesting a collective instantiation of all required resources in the second cloud environment to the second endpoint.

[0216] The multi-cloud platform control plane 1602 may receive acknowledgements from the first and second endpoints, respectively, in steps 6 and 7. Thereafter, upon receiving a polling request from the client 1601 regarding the status of the requested network link, the multi-cloud platform control plane 1602 may send a message to the client indicating successful creation of the network link (i.e., a message indicating that the network link is available) in step 9.

[0217] FIG. 16B illustrates an exemplary flow chart showing a process performed by a multi-cloud service control plane, according to an embodiment. The process illustrated in FIG. 16B may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective systems, hardware, or a combination thereof. The software may be stored in a non-transitory storage medium (e.g., in a memory device). The method presented in FIG. 16B and described below is intended to be exemplary and non-limiting. Although FIG. 16B illustrates various processing steps occurring in a particular sequence or order, this is not intended to be limiting. In an alternative embodiment, these steps may be performed in some different order, or some steps may be performed in parallel.

[0218] The process begins at step 1650, where a control plane of a multi-cloud infrastructure included in a first cloud environment receives a request to create a network link between a first virtual network in the first cloud environment and a second virtual network in a second cloud environment. Note that the first virtual network in the first cloud environment has already been created to enable users associated with a customer's tenancy in the second cloud environment to access one or more services offered in the first cloud environment. Upon receiving the request to create the network link, the control plane executes a workflow to initiate creation of the network link (step 1660).

[0219] In some implementations, the workflow may include multiple sub-steps that are performed to create a network link. For example, the workflow may include step 1663, in which a first request is sent to a first endpoint of a first cloud environment (e.g., the first endpoint may correspond to a resource manager of the first cloud environment). The first request corresponds to a request being issued by the multi-cloud platform control plane 1602, requesting provisioning of a first set of resources in the first cloud environment. The first set of resources may correspond to the creation of an LVNA, a hub VCN, a spoke VCN, a VNIC, etc.

[0220] The process then moves to step 1665 where a second request is sent to a second endpoint of the second cloud environment (e.g., the second endpoint may correspond to a resource manager of the second cloud environment). The second request corresponds to a request being issued by the multi-cloud platform control plane 1602, requesting provisioning of a second set of resources in the second cloud environment. The second set of resources may correspond to the creation of an RVNA, a VNF, a hub VNET, a spoke VNET, a service endpoint, a network peering, etc.

[0221] Further, in step 1670, in response to the first set of resources and the second set of resources being provisioned, the control plane may distribute network configuration information to one or more resources of the first set of resources and the second set of resources. The network configuration information may include, for example, information corresponding to an IP address of the resource. Thus, the packet processor is provided with information regarding the address of the next hop destination to which the traffic is forwarded (by the distribution service of the control plane). In other words, the network configuration information enables traffic to be communicated between a first virtual network in the first cloud environment and a second virtual network in the second cloud environment.

[0222] As described above with respect to Figures 11 and 13, the architecture of the network link created between the first and second cloud environments is related to the location and placement of packet processors in the link-enabled virtual network being created to provision the network link. It is understood that the network link created by the embodiments of the present disclosure supports high-throughput latency-sensitive applications, such as exadatabase applications. According to some embodiments, the transmission latency (e.g., round trip latency) of the communication between the first and second cloud environments may also be affected by the selection of the domains in the first and second cloud environments where the compute instances are provisioned. In other words, to achieve low latency, it is desirable to select an optimal pair of domains (i.e., one domain in each of the first and second cloud environments) to host the compute instances.

[0223] It is noted that cloud infrastructure is typically hosted within regions and domains (also called availability domains). A region is defined as a localized geographical area, and an availability domain may correspond to one or more data centers located within a region, i.e., a region is composed of one or more availability domains. Furthermore, cloud infrastructure resources may be specific to a region, such as a virtual cloud network, or specific to an availability domain, such as a compute instance. Availability domains are isolated from each other, fault-tolerant, and highly unlikely to fail simultaneously. Because availability domains do not share infrastructure such as power or cooling or an internal availability domain network, a failure in one availability domain within a region is unlikely to affect the availability of other availability domains within the same region. In the following, with reference to FIG. 17A, a framework is described for selecting availability domains within a first cloud environment and a second cloud environment for placement of compute instances such that low latency is achieved in the transmission of information from the first cloud environment to the second cloud environment and from the second cloud environment to the first cloud environment.

[0224] FIG. 17A illustrates an exemplary placement strategy of compute instances in availability domains of different cloud environments, according to an embodiment. For simplicity and illustrative purposes, FIG. 17A illustrates a first cloud environment 1701 including three availability domains (AD), namely, AD-1 1703A, AD-2 1703B, and AD-3 1703C. Similarly, a second cloud environment 1721 is illustrated as including three availability domains, namely, AD-1′ 1723A, AD-2′ 1723B, and AD-3′ 1723C. A customer in the second cloud environment may wish to utilize a service (e.g., database service) offered by the first cloud environment. Certain services (e.g., exadatabase service) offered by the first cloud environment may be high throughput in addition to being latency sensitive. Thus, as illustrated in FIG. 17A, according to some embodiments, provisioning a service may require deployment of one or more compute instances 1720 in a particular availability domain in each cloud environment. It is understood that ADs are assigned to customers in a random manner. In particular, AD1, AD2, and AD3 assigned to customer 1 in a particular cloud environment may not correspond (in terms of order) to the first, second, and third availability domains assigned to another customer, e.g., customer 2.

[0225] In the aforementioned framework, the selection of one AD in each cloud environment should be determined so that one or more computing instances can be deployed in the selected AD to enable the desired service for a customer in the second cloud environment to utilize the service provided by the first cloud environment. Note that the selection of a pair of ADs (i.e., one AD in the first cloud environment and another AD in the second cloud environment) should be performed in such a way that the latency incurred in providing the service by the selected AD is minimized. Such a selection strategy of ADs makes it possible to provide latency-sensitive services.

[0226] According to some embodiments, the selection of an AD in each cloud environment is empirically derived to obtain the lowest latency. In one implementation, the steps involved in selecting an AD in each cloud environment may be described as follows: Step 1 - Allocate a single compute instance (herein referred to as a test compute instance) in each availability domain within each cloud environment. Step 2-Get the transmission latency for each pair of ADs (k,j), where k = 1, 2, 3 and j = 1', 2', 3'. Note that the latency corresponds to the amount of time incurred in transmitting data from a compute instance deployed in one AD to another compute instance deployed in the other AD. Step 3--Select the AD pair with the shortest latency. Step 4--Shut down instances in the unselected ADs (in each cloud environment) and deploy a desired number of compute instances to each AD of the selected pair of ADs. In some applications, two compute instances (i.e., a primary compute instance and a secondary compute instance) may be deployed to the selected AD of each cloud environment. For example, as shown in FIG. 17A, AD-1 1703A in the first cloud environment and AD-2' 1723B in the second cloud environment are determined as the pair of ADs that results in the shortest transmission latency. Therefore, to provision the service, two compute instances are deployed to each of AD-1 1703A and AD-2' 1723B.

[0227] Note that according to some embodiments, for high availability purposes, the two compute instances may be deployed in different failure domains within an AD. Some other embodiments may choose to deploy one compute instance in the pair of ADs with the lowest latency and another compute instance in another pair of ADs with the second lowest latency. It is understood that other possible selections of ADs based on the above selection strategies are well within the scope of this disclosure.

[0228] FIG. 17B illustrates an exemplary flowchart illustrating a process performed by a multi-cloud service control plane in determining the availability domains of different cloud environments in which a compute instance is deployed, according to an embodiment. The process illustrated in FIG. 17B may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective systems, hardware, or a combination thereof. The software may be stored in a non-transitory storage medium (e.g., in a memory device). The method presented in FIG. 17B and described below is intended to be exemplary and non-limiting. Although FIG. 17B illustrates various processing steps occurring in a particular sequence or order, this is not intended to be limiting. In an alternative embodiment, these steps may be performed in some different order, or some steps may be performed in parallel.

[0229] The process begins at step 1755, where a multi-cloud infrastructure included in a first cloud environment receives a request to (i) deploy a first plurality of compute instances in a first availability domain of a first plurality of availability domains in the first cloud environment and (ii) deploy a second plurality of compute instances in a second availability domain of a second plurality of availability domains in a second cloud environment. The first plurality of compute instances in the first availability domain of the first cloud environment and the second plurality of compute instances in the second cloud environment enable a user in the second cloud environment to access one or more services offered in the first cloud environment.

[0230] The process then moves to step 1760, where it determines a first availability domain of the first plurality of availability domains in the first cloud environment and a second availability domain of the second plurality of availability domains in the second cloud environment. To determine the first and second availability domains, it moves to step 1765, where it iteratively performs a set of operations for each availability domain of the first plurality of availability domains in the first cloud environment and for each availability domain of the second plurality of availability domains in the second cloud environment. This iterative operation includes steps 1765A-1765D.

[0231] At step 1765A, a first test compute instance is placed in an availability domain of a first plurality of availability domains in a first cloud environment. At step 1765B, a second test compute instance is placed in an availability domain of a second plurality of availability domains in a second cloud environment. The iterative process then moves to step 1765C, where a test packet is sent from the first test compute instance to the second test compute instance. At step 1765D, a transmission latency corresponding to the pair of availability domains hosting the first and second test compute instances is calculated. Upon iteratively processing steps 1765A-1765D, the process moves to step 1770, where a first availability domain in the first cloud environment and a second availability domain in the second cloud environment are selected based on a condition associated with the calculated transmission latency for the pair of availability domains.

[0232] This condition may correspond to selecting the pair of ADs that results in the shortest transmission latency observed in the iterative process of steps 1765A-1765D. Upon identifying a first availability domain in the first cloud environment and a second availability domain in the second cloud environment, the process may instantiate a desired number of compute instances in the selected domains to enable users in the second cloud environment to utilize one or more services provided by the first cloud environment.

[0233] Cloud Infrastructure Examples As mentioned above, infrastructure as a service (IaaS) is one particular type of cloud computing. IaaS may be configured to provide virtualized computing resources over a public network (e.g., the Internet). In an IaaS model, a cloud computing provider may host infrastructure components (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer), etc.). In some cases, an IaaS provider may offer various services (e.g., billing, monitoring, logging, security, load balancing, clustering, etc.) to accompany those infrastructure components. Thus, these services may be policy-driven, so that an IaaS user may implement policies to drive load balancing to maintain application availability and performance.

[0234] In some cases, IaaS customers may access resources and services over a wide area network (WAN), such as the Internet, and can use the cloud provider's services to install the remaining elements of their application stack. For example, a user can log into an IaaS platform to create virtual machines (VMs), install an operating system (OS) on each VM, deploy middleware such as databases, create storage buckets for workloads and backups, and even install enterprise software on the VMs. The customer can then use the provider's services to perform a variety of functions, including balancing network traffic, troubleshooting application issues, monitoring performance, managing disaster recovery, etc.

[0235] In most cases, the cloud computing model requires the participation of a cloud provider, which may be, but need not be, a third-party service that specializes in providing (e.g., offering, renting, selling) IaaS. An entity may choose to deploy a private cloud and become its own provider of infrastructure services.

[0236] In some examples, an IaaS deployment is the process of hooking up a new application or a new version of an application to a prepared application server or the like. This process may include the process of preparing the server (e.g., installing libraries, daemons, etc.). This process is often managed by the cloud provider below the hypervisor layer (e.g., server, storage, network hardware, and virtualization). Thus, the customer may be responsible for handling the deployment of the OS, middleware, and / or application (e.g., on top of self-service virtual machines (e.g., that may be spun up on demand)).

[0237] In some examples, IaaS provisioning may also refer to obtaining computers or virtual hosts for use and installing needed libraries or services on those computers or virtual hosts. In most cases, deployment does not include provisioning, which may need to be performed first.

[0238] In some cases, there are two distinct problems in IaaS provisioning. First, there is the initial challenge of provisioning an initial set of infrastructure before anything can be run. Second, there is the challenge of evolving the existing infrastructure (e.g., adding new services, modifying services, removing services, etc.) after everything has been provisioned. In some cases, these two challenges may be addressed by allowing the configuration of the infrastructure to be defined declaratively. In other words, the infrastructure (e.g., which components are required and how those components interact) may be defined by one or more configuration files. In this way, the entire topology of the infrastructure (e.g., which resources depend on which resources and how each of those resources work together) may be described declaratively. In some cases, after the topology is defined, workflows may be generated that create and / or manage the various components described in the configuration files.

[0239] In some examples, the infrastructure may include many interconnected elements. For example, there may be one or more virtual private clouds (VPCs) (e.g., a configurable and / or shared, possibly on-demand pool of computing resources), also known as a core network. In some examples, there may be one or more security group rules provisioned to define how the security of the network is configured, and one or more virtual machines (VMs). Other infrastructure elements, such as load balancers, databases, etc., may be provisioned. The infrastructure may evolve over time as more infrastructure elements are desired and / or added.

[0240] In some cases, continuous deployment techniques may be employed to enable deployment of infrastructure code across various virtual computing environments. Additionally, the described techniques may enable infrastructure management within these environments. In some cases, a service team may write code that is desired to be deployed to one or more, but often many, different production environments (e.g., across various geographic locations, sometimes across the world). However, in some cases, the infrastructure to which the code is deployed must first be set up. In some cases, provisioning may be done manually, and provisioning tools may be utilized to provision resources and / or deployment tools may be utilized to deploy the code after the infrastructure is provisioned.

[0241] FIG. 18 is a block diagram 1800 illustrating an example pattern of an IaaS architecture according to at least one embodiment. A service operator 1802 may be communicatively coupled to a secure host tenancy 1804, which may include a virtual cloud network (VCN) 1806 and a secure host subnet 1808. In some examples, the service operator 1802 may employ one or more client computing devices, which may be portable handheld devices (e.g., iPhone®, mobile phones, iPad®, computing tablets, personal digital assistants (PDAs)) or wearable devices (e.g., Google Glass® head-mounted displays) that are Internet, email, short message service (SMS), Blackberry®, or other communication protocol enabled running software such as Microsoft Windows Mobile®, and / or various mobile operating systems such as iOS, Windows Phone, Android, BlackBerry 8, Palm OS, etc. Alternatively, the client computing devices can be general purpose personal computers, including, by way of example, personal and / or laptop computers running various versions of Microsoft Windows, Apple Macintosh, and / or Linux operating systems. The client computing devices can be workstation computers running any of a variety of commercially available UNIX or UNIX-like operating systems, including, but not limited to, various GNU / Linux operating systems, such as Google Chrome OS.Alternatively or additionally, the client computing devices may be any other electronic devices, such as thin-client computers, Internet-enabled gaming systems (e.g., Microsoft Xbox gaming consoles with or without a Kinect® gesture input device), and / or personal messaging devices, capable of communicating over a network accessible to VCN 1806 and / or the Internet.

[0242] The VCN 1806 may include a local peering gateway (LPG) 1810, which may be communicatively coupled to a secure shell (SSH) VCN 1812 via an LPG 1810 included in the SSH VCN 1812. The SSH VCN 1812 may include an SSH subnet 1814, which may be communicatively coupled to a control plane VCN 1816 via an LPG 1810 included in the control plane VCN 1816. The SSH VCN 1812 may also be communicatively coupled to a data plane VCN 1818 via the LPG 1810. The control plane VCN 1816 and the data plane VCN 1818 may be included in a service tenancy 1819, which may be owned and / or operated by the IaaS provider.

[0243] The control plane VCN 1816 may include a control plane demilitarized zone (DMZ) tier 1820 that serves as a perimeter network (e.g., a portion of an enterprise network between an enterprise intranet and an external network). Servers based in the DMZ may have limited responsibility and help keep security breaches contained. Additionally, the DMZ tier 1820 may include one or more load balancer (LB) subnets 1822, a control plane app tier 1824 that may include an app subnet 1826, a control plane data tier 1828 that may include a database (DB) subnet 1830 (e.g., a front-end DB subnet and / or a back-end DB subnet). The LB subnet 1822 included in the control plane DMZ tier 1820 can be communicatively coupled to an app subnet 1826 and an Internet gateway 1834 included in a control plane app tier 1824 that may be included in the control plane VCN 1816, and the app subnet 1826 can be communicatively coupled to a DB subnet 1830 included in the control plane data tier 1828, as well as a service gateway 1836 and a network address translation (NAT) gateway 1838. The control plane VCN 1816 can include the service gateway 1836 and the NAT gateway 1838.

[0244] The control plane VCN 1816 can include a data plane mirrored app layer 1840 that can include an app subnet 1826. The app subnet 1826 included in the data plane mirrored app layer 1840 can include a virtual network interface controller (VNIC) 1842 that can run a compute instance 1844. The compute instance 1844 can communicatively couple the app subnet 1826 of the data plane mirrored app layer 1840 to the app subnet 1826 that can be included in the data plane app layer 1846.

[0245] The data plane VCN 1818 may include a data plane app layer 1846, a data plane DMZ layer 1848, and a data plane data layer 1850. The data plane DMZ layer 1848 may include a LB subnet 1822, which may be communicatively coupled to an app subnet 1826 of the data plane app layer 1846 and an Internet gateway 1834 of the data plane VCN 1818. The app subnet 1826 may be communicatively coupled to a service gateway 1836 of the data plane VCN 1818 and a NAT gateway 1838 of the data plane VCN 1818. The data plane data layer 1850 may also include a DB subnet 1830, which may be communicatively coupled to the app subnet 1826 of the data plane app layer 1846.

[0246] The Internet gateways 1834 of the control plane VCNs 1816 and of the data plane VCNs 1818 may be communicatively coupled to a metadata management service 1852, which may be communicatively coupled to the public Internet 1854. The public Internet 1854 may be communicatively coupled to NAT gateways 1838 of the control plane VCNs 1816 and of the data plane VCNs 1818. The service gateways 1836 of the control plane VCNs 1816 and of the data plane VCNs 1818 may be communicatively coupled to cloud services 1856.

[0247] In some examples, a service gateway 1836 in the control plane VCN 1816 or in the data plane VCN 1818 can make application programming interface (API) calls to cloud services 1856 without traversing the public Internet 1854. The API calls from the service gateway 1836 to the cloud services 1856 can be one-way: the service gateway 1836 can make the API call to the cloud services 1856, and the cloud services 1856 can send the requested data to the service gateway 1836. However, the cloud services 1856 may not initiate the API calls to the service gateway 1836.

[0248] In some examples, secure host tenancy 1804 can be directly connected to service tenancy 1819 or may be otherwise separate. Secure host subnet 1808 can communicate with SSH subnet 1814 through LPG 1810, which may enable bidirectional communication on otherwise separate systems. Connecting secure host subnet 1808 to SSH subnet 1814 may give secure host subnet 1808 access to other entities in service tenancy 1819.

[0249] The control plane VCN 1816 may enable users of the service tenancy 1819 to configure or otherwise provision desired resources. The desired resources provisioned in the control plane VCN 1816 may be deployed or otherwise used in the data plane VCN 1818. In some examples, the control plane VCN 1816 may be separate from the data plane VCN 1818, and the data plane mirror app layer 1840 of the control plane VCN 1816 may communicate with the data plane app layer 1846 of the data plane VCN 1818 via a VNIC 1842, which may be included in the data plane mirror app layer 1840 and the data plane app layer 1846.

[0250] In some examples, a user or customer of the system may make a request, e.g., a create, read, update, or delete (CRUD) operation, via the public internet 1854, which may communicate the request to a metadata management service 1852. The metadata management service 1852 may communicate the request to the control plane VCN 1816 via an internet gateway 1834. The request may be received by a LB subnet 1822 included in the control plane DMZ tier 1820. The LB subnet 1822 may determine that the request is valid, and in response to this determination, the LB subnet 1822 may send the request to an app subnet 1826 included in the control plane app tier 1824. If the request is authenticated and the request requires a call to the public internet 1854, the call to the public internet 1854 may be sent to a NAT gateway 1838, which may make the call to the public internet 1854. Memory that may be desirable to be stored by the request may be stored within the DB subnet 1830.

[0251] In some examples, the data plane mirror app layer 1840 can facilitate direct communication between the control plane VCN 1816 and the data plane VCN 1818. For example, it may be desirable for changes, updates, or other suitable modifications to a configuration to be applied to resources included in the data plane VCN 1818. Through the VNIC 1842, the control plane VCN 1816 can communicate directly with the resources included in the data plane VCN 1818, thereby performing changes, updates, or other suitable modifications to the configuration of the resources.

[0252] In some embodiments, the control plane VCN 1816 and the data plane VCN 1818 may be included in the service tenancy 1819. In this case, a user or customer of the system may not own or operate either the control plane VCN 1816 or the data plane VCN 1818. Instead, an IaaS provider may own or operate the control plane VCN 1816 and the data plane VCN 1818, both of which may be included in the service tenancy 1819. This embodiment may allow for network isolation that may prevent a user or customer from interacting with other users' resources or other customers' resources. This embodiment may also allow a user or customer of the system to store databases privately without having to rely on the public internet 1854 for storage, which may not have a desirable level of security.

[0253] In another embodiment, the LB subnet 1822 included in the control plane VCN 1816 may be configured to receive signals from the service gateway 1836. In this embodiment, the control plane VCN 1816 and the data plane VCN 1818 may be configured to be called by the IaaS provider's customers without calling the public Internet 1854. The IaaS provider's customers may desire this embodiment because databases used by the customers may be stored in a service tenancy 1819 that may be controlled by the IaaS provider and may be isolated from the public Internet 1854.

[0254] FIG. 19 is a block diagram 1900 illustrating another example pattern of an IaaS architecture, according to at least one embodiment. A service operator 1902 (e.g., service operator 1802 of FIG. 18 ) may be communicatively coupled to a secure host tenancy 1904 (e.g., secure host tenancy 1804 of FIG. 18 ), which may include a virtual cloud network (VCN) 1906 (e.g., VCN 1806 of FIG. 18 ) and a secure host subnet 1908 (e.g., secure host subnet 1808 of FIG. 18 ). The VCN 1906 may include a local peering gateway (LPG) 1910 (e.g., LPG 1810 of FIG. 18 ), which may be communicatively coupled to an SSH VCN 1912 (e.g., SSH VCN 1812 of FIG. 18 ) via an LPG 1810 included in a secure shell (SSH) VCN 1912. SSH VCN 1912 can include an SSH subnet 1914 (e.g., SSH subnet 1814 in FIG. 18), which can be communicatively coupled to a control plane VCN 1916 (e.g., control plane VCN 1816 in FIG. 18) via an LPG 1910 included in the control plane VCN 1916. The control plane VCN 1916 can be included in a service tenancy 1919 (e.g., service tenancy 1819 in FIG. 18), and the data plane VCN 1918 (e.g., data plane VCN 1818 in FIG. 18) can be included in a customer tenancy 1921, which can be owned or operated by a user or customer of the system.

[0255] The control plane VCN 1916 may include a control plane DMZ tier 1920 (e.g., control plane DMZ tier 1820 of FIG. 18 ) that may include a LB subnet 1922 (e.g., LB subnet 1822 of FIG. 18 ), a control plane app tier 1924 (e.g., control plane app tier 1824 of FIG. 18 ) that may include an app subnet 1926 (e.g., app subnet 1826 of FIG. 18 ), and a control plane data tier 1928 (e.g., control plane data tier 1828 of FIG. 18 ) that may include a database (DB) subnet 1930 (e.g., similar to the database (DB) subnet 1830 of FIG. 18 ). The LB subnet 1922 included in the control plane DMZ tier 1920 can be communicatively coupled to an app subnet 1926 and an Internet gateway 1934 (e.g., Internet gateway 1834 of FIG. 18 ) included in the control plane app tier 1924, which may be included in the control plane VCN 1916, and the app subnet 1926 can be communicatively coupled to a DB subnet 1930 and a service gateway 1936 (e.g., service gateway of FIG. 18 ) and a network address translation (NAT) gateway 1938 (e.g., NAT gateway 1838 of FIG. 18 ) included in the control plane data tier 1928. The control plane VCN 1916 can include the service gateway 1936 and the NAT gateway 1938.

[0256] The control plane VCN 1916 can include a data plane mirror app layer 1940 (e.g., data plane mirror app layer 1840 of FIG. 18 ), which can include an app subnet 1926. The app subnet 1926 included in the data plane mirror app layer 1940 can include a virtual network interface controller (VNIC) 1942 (e.g., VNIC of 1842) that can run a compute instance 1944 (e.g., similar to compute instance 1844 of FIG. 18 ). The compute instance 1944 can facilitate communication between the app subnet 1926 of the data plane mirror app layer 1940 and the app subnet 1926 that can be included in the data plane app layer 1946 (e.g., data plane app layer 1846 of FIG. 18 ) via the VNIC 1942 included in the data plane mirror app layer 1940 and the VNIC 1942 included in the data plane app layer 1946.

[0257] The Internet gateway 1934 included in the control plane VCN 1916 may be communicatively coupled to a metadata management service 1952 (e.g., metadata management service 1852 of FIG. 18 ), which may be communicatively coupled to a public Internet 1954 (e.g., public Internet 1854 of FIG. 18 ). The public Internet 1954 may be communicatively coupled to a NAT gateway 1938 included in the control plane VCN 1916. The service gateway 1936 included in the control plane VCN 1416 may be communicatively coupled to cloud services 1956 (e.g., cloud services 1856 of FIG. 18 ).

[0258] In some examples, the data plane VCN 1918 may be included in the customer's tenancy 1921. In this case, the IaaS provider may provide a control plane VCN 1916 for each customer, and the IaaS provider may configure a unique compute instance 1944 for each customer that is included in the service tenancy 1919. Each compute instance 1944 may enable communication between the control plane VCN 1916 included in the service tenancy 1919 and the data plane VCN 1918 included in the customer's tenancy 1921. The compute instance 1944 may enable resources provisioned in the control plane VCN 1916 included in the service tenancy 1919 to be deployed or otherwise used in the data plane VCN 1918 included in the customer's tenancy 1921.

[0259] In another example, the IaaS provider's customer may have a database that persists in the customer's tenancy 1921. In this example, the control plane VCN 1916 may include a data plane mirror app tier 1940, which may include an app subnet 1926. The data plane mirror app tier 1940 may reside in the data plane VCN 1918, but the data plane mirror app tier 1940 may not persist in the data plane VCN 1918. That is, the data plane mirror app tier 1940 may have access to the customer's tenancy 1921, but the data plane mirror app tier 1940 may not reside in the data plane VCN 1918 and may not be owned or operated by the IaaS provider's customer. The data plane mirror app tier 1940 may be configured to make calls to the data plane VCN 1918, but may not be configured to make calls to any entities included in the control plane VCN 1916. A customer may desire to deploy or otherwise use resources in the data plane VCN 1918 that have been provisioned in the control plane VCN 1916, and the data plane mirror app layer 1940 can facilitate the desired deployment or other use of the customer's resources.

[0260] In some embodiments, the IaaS provider's customer can apply filters to the data plane VCN 1918. In this embodiment, the customer can determine which data plane VCNs 1918 are accessible, and the customer may limit access from the data plane VCN 1918 to the public Internet 1954. The IaaS provider may not be able to apply filters or otherwise control the access of the data plane VCN 1918 to any external networks or databases. Applying filters and controls by the customer to the data plane VCN 1918 contained in the customer's tenancy 1921 can help isolate the data plane VCN 1918 from other customers and from the public Internet 1954.

[0261] In some embodiments, cloud services 1956 may be called by service gateway 1936 to access services that may not be in the public internet 1954, the control plane VCN 1916, or the data plane VCN 1918. The connection between cloud services 1956 and the control plane VCN 1916 or the data plane VCN 1918 may not be up and running or continuous. Cloud services 1956 may be in different networks owned or operated by an IaaS provider. Cloud services 1956 may be configured to receive calls from service gateway 1936 and may not be configured to receive calls from the public internet 1954. Some cloud services 1956 may be isolated from other cloud services 1956, and control plane VCN 1916 may be isolated from cloud services 1956 that may not be in the same region as the control plane VCN 1916. For example, control plane VCN 1916 may be located in "region 1," and a cloud service "deployment 13" may be located in region 1 and "region 2." When a call is made to deployment 13 by a service gateway 1936 included in control plane VCN 1916 located in region 1, the call may be sent to deployment 13 in region 1. In this example, control plane VCN 1916, or deployment 13 in region 1, may not be communicatively coupled to or otherwise in communication with deployment 13 in region 2.

[0262] 20 is a block diagram 2000 illustrating another example pattern of an IaaS architecture, according to at least one embodiment. A service operator 2002 (e.g., service operator 1802 of FIG. 18) may be communicatively coupled to a secure host tenancy 2004 (e.g., secure host tenancy 1804 of FIG. 18), which may include a virtual cloud network (VCN) 2006 (e.g., VCN 1806 of FIG. 18) and a secure host subnet 2008 (e.g., secure host subnet 1808 of FIG. 18). The VCN 2006 may include an LPG 2010 (e.g., LPG 1810 of FIG. 18), which may be communicatively coupled to an SSH VCN 2012 (e.g., SSH VCN 1812 of FIG. 18) via an LPG 2010 included in the SSH VCN 2012. SSH VCN 2012 can include an SSH subnet 2014 (e.g., SSH subnet 1814 in FIG. 18 ), which can be communicatively coupled to a control plane VCN 2016 (e.g., control plane VCN 1816 in FIG. 18 ) via an LPG 2010 included in control plane VCN 2016, and to a data plane VCN 2018 (e.g., data plane 1818 in FIG. 18 ) via an LPG 2010 included in data plane VCN 2018. The control plane VCN 2016 and the data plane VCN 2018 can be included in a service tenancy 2019 (e.g., service tenancy 1819 in FIG. 18 ).

[0263] The control plane VCN 1816 may include a control plane DMZ tier 1820 (e.g., the control plane DMZ tier 1820 of FIG. 18 ) that may include a load balancer (LB) subnet 1822 (e.g., the LB subnet 1822 of FIG. 18 ), a control plane app tier 2024 (e.g., the control plane app tier 1824 of FIG. 18 ) that may include an app subnet 2026 (e.g., similar to the app subnet 1826 of FIG. 18 ), and a control plane data tier 2028 (e.g., the control plane data tier 1828 of FIG. 18 ) that may include a DB subnet 2030. The LB subnet 2022 included in the control plane DMZ tier 2020 can be communicatively coupled to an app subnet 2026 included in a control plane app tier 2024 that may be included in the control plane VCN 2016, and to an Internet gateway 1834 (e.g., Internet gateway 1834 of FIG. 18), which can be communicatively coupled to a DB subnet 1830 included in the control plane data tier 1828, as well as to a service gateway 1836 (e.g., service gateway of FIG. 18) and a network address translation (NAT) gateway 1838 (e.g., NAT gateway 1838 of FIG. 18). The control plane VCN 2016 can include the service gateway 2036 and the NAT gateway 2038.

[0264] The data plane VCN 2018 may include a data plane app layer 2046 (e.g., data plane app layer 1846 of FIG. 18 ), a data plane DMZ layer 2048 (e.g., data plane DMZ layer 1848 of FIG. 18 ), and a data plane data layer 2050 (e.g., data plane data layer 1850 of FIG. 18 ). The data plane DMZ layer 2048 may include a trusted app subnet 2060 and an untrusted app subnet 2062 of the data plane app layer 2046 and a LB subnet 2022 that may be communicatively coupled to an Internet gateway 2034 included in the data plane VCN 2018. The trusted app subnet 2060 may be communicatively coupled to a service gateway 2036 included in the data plane VCN 2018, a NAT gateway 2038 included in the data plane VCN 2018, and a DB subnet 2030 included in the data plane data layer 2050. The untrusted app subnet 2062 may be communicatively coupled to a service gateway 2036 included in the data plane VCN 2018 and to a DB subnet 2030 included in the data plane data layer 2050. The data plane data layer 2050 may include a DB subnet 2030 that may be communicatively coupled to a service gateway 2036 included in the data plane VCN 2018.

[0265] The untrusted app subnet 2062 may include one or more primary VNICs 2064(1)-(N), which may be communicatively coupled to tenant virtual machines (VMs) 2066(1)-(N). Each tenant VM 2066(1)-(N) may be communicatively coupled to a respective app subnet 2067(1)-(N), which may be included in a respective container egress VCN 2068(1)-(N), which may be included in a respective customer tenancy 2070(1)-(N). Each secondary VNIC 2072(1)-(N) may facilitate communication between the untrusted app subnet 2062 included in the data plane VCN 2018 and the app subnet included in the container egress VCN 2068(1)-(N). Each container egress VCN 2068(1)-(N) may include a NAT gateway 2038, which may be communicatively coupled to the public Internet 2054 (e.g., public Internet 1854 of FIG. 18).

[0266] The internet gateway 2034 included in the control plane VCN 2016 and included in the data plane VCN 2018 may be communicatively coupled to a metadata management service 2052 (e.g., metadata management system 1852 of FIG. 18 ), which may be communicatively coupled to the public internet 2054. The public internet 2054 may be communicatively coupled to a NAT gateway 2038 included in the control plane VCN 2016 and included in the data plane VCN 2018. The service gateway 2036 included in the control plane VCN 2016 and included in the data plane VCN 2018 may be communicatively coupled to cloud services 2056.

[0267] In some embodiments, the data plane VCN 2018 may be integrated with a customer tenancy 2070. This integration may be useful or desirable for an IaaS provider's customer in some cases, such as when they may want support when executing code. A customer may provide code for execution that may be disruptive, may communicate with other customers' resources, or may otherwise cause undesirable effects. In response, the IaaS provider may determine whether to execute the code provided to the IaaS provider by the customer.

[0268] In some examples, an IaaS provider's customer may grant temporary network access to the IaaS provider and request functionality connected to the data plane app layer 2046. Code to perform this functionality may be executed in VMs 2066(1)-(N), which may not be configured to run elsewhere on the data plane VCN 2018. Each VM 2066(1)-(N) may be connected to one customer's tenancy 2070. Each container 2071(1)-(N) contained in VM 2066(1)-(N) may be configured to execute code. In this case, there may be a double isolation (e.g., containers 2071(1)-(N) running code may be contained in at least VMs 2066(1)-(N) that are contained in untrusted app subnet 2062), which may help prevent incorrect or otherwise undesirable code from damaging the IaaS provider's network or damaging a different customer's network. Containers 2071(1)-(N) may be communicatively coupled to customer tenancy 2070 and may be configured to send or receive data to or from customer tenancy 2070. Containers 2071(1)-(N) may not be configured to send or receive data to or from any other entities in data plane VCN 2018. Upon completion of code execution, the IaaS provider may kill or otherwise destroy containers 2071(1)-(N).

[0269] In some embodiments, trusted app subnet 2060 may execute code that may be owned or operated by the IaaS provider. In this embodiment, trusted app subnet 2060 may be communicatively coupled to DB subnet 2030 and may be configured to perform CRUD operations within DB subnet 2030. Untrusted app subnet 2062 may be communicatively coupled to DB subnet 2030, but in this embodiment, untrusted app subnet 2062 may be configured to perform read operations within DB subnet 2030. Containers 2071(1)-(N) capable of executing code from the customer, which may be included in each customer's VMs 2066(1)-(N), may not be communicatively coupled to DB subnet 2030.

[0270] In other embodiments, the control plane VCN 2016 and the data plane VCN 2018 may not be directly communicatively coupled. In this embodiment, there may not be direct communication between the control plane VCN 2016 and the data plane VCN 2018. However, communication may occur indirectly by at least one method. The LPG 2010 may be established by an IaaS provider and may facilitate communication between the control plane VCN 2016 and the data plane VCN 2018. In another example, the control plane VCN 2016 or the data plane VCN 2018 may make a call to a cloud service 2056 via the service gateway 2036. For example, a call from the control plane VCN 2016 to the cloud service 2056 may include a request for a service that may communicate with the data plane VCN 2018.

[0271] FIG. 21 is a block diagram 2100 illustrating another example pattern of an IaaS architecture, according to at least one embodiment. A service operator 2102 (e.g., service operator 1802 of FIG. 18) may be communicatively coupled to a secure host tenancy 2104 (e.g., secure host tenancy 1804 of FIG. 18), which may include a virtual cloud network (VCN) 2106 (e.g., VCN 1806 of FIG. 18) and a secure host subnet 2108 (e.g., secure host subnet 1808 of FIG. 18). The VCN 2106 may include an LPG 2110 (e.g., LPG 1810 of FIG. 18), which may be communicatively coupled to an SSH VCN 2112 (e.g., SSH VCN 1812 of FIG. 18) via an LPG 2110 included in the SSH VCN 2112. SSH VCN 2112 can include an SSH subnet 2114 (e.g., SSH subnet 1814 in FIG. 18), which can be communicatively coupled to a control plane VCN 2116 (e.g., control plane VCN 1816 in FIG. 18) via an LPG 2110 included in control plane VCN 2116, and to a data plane VCN 2118 (e.g., data plane 1818 in FIG. 18) via an LPG 2110 included in data plane VCN 2118. The control plane VCN 2116 and the data plane VCN 2118 can be included in a service tenancy 2119 (e.g., service tenancy 1819 in FIG. 18).

[0272] The control plane VCN 2116 may include a control plane DMZ layer 2120 (e.g., control plane DMZ layer 1820 of FIG. 18 ) which may include a LB subnet 2122 (e.g., LB subnet 1822 of FIG. 18 ), a control plane app layer 2124 (e.g., control plane app layer 1824 of FIG. 18 ) which may include an app subnet 2126 (e.g., app subnet 1826 of FIG. 18 ), and a control plane data layer 2128 (e.g., control plane data layer 1828 of FIG. 18 ) which may include a DB subnet 2130 (e.g., DB subnet 2030 of FIG. 20 ). The LB subnet 2122 included in the control plane DMZ tier 2120 can be communicatively coupled to an app subnet 2126 included in a control plane app tier 2124 that may be included in the control plane VCN 2116, and to an Internet gateway 2134 (e.g., Internet gateway 1834 of FIG. 18), which can be communicatively coupled to a DB subnet 2130 included in the control plane data tier 2128, as well as to a service gateway 2136 (e.g., service gateway of FIG. 18) and a network address translation (NAT) gateway 2138 (e.g., NAT gateway 1838 of FIG. 18). The control plane VCN 2116 can include the service gateway 2136 and the NAT gateway 2138.

[0273] The data plane VCN 2118 can include a data plane app layer 2146 (e.g., data plane app layer 1846 of FIG. 18 ), a data plane DMZ layer 2148 (e.g., data plane DMZ layer 2148 of FIG. 18 ), and a data plane data layer 2150 (e.g., data plane data layer 1850 of FIG. 18 ). The data plane DMZ layer 2148 can include a trusted app subnet 2160 (e.g., trusted app subnet 2060 of FIG. 20 ) and an untrusted app subnet 2162 (e.g., untrusted app subnet 2062 of FIG. 20 ) of the data plane app layer 2146, as well as a LB subnet 2122 that can be communicatively coupled to an Internet gateway 2134 included in the data plane VCN 2118. The trusted app subnet 2160 may be communicatively coupled to a service gateway 2136 included in the data plane VCN 2118, a NAT gateway 2138 included in the data plane VCN 2118, and a DB subnet 2130 included in the data plane data layer 2150. The untrusted app subnet 2162 may be communicatively coupled to a service gateway 2136 included in the data plane VCN 2118, and a DB subnet 2130 included in the data plane data layer 2150. The data plane data layer 2150 may include a DB subnet 2130 that may be communicatively coupled to a service gateway 2136 included in the data plane VCN 2118.

[0274] The untrusted app subnet 2162 may include primary VNICs 2164(1)-(N) that may be communicatively coupled to tenant virtual machines (VMs) 2166(1)-(N) that reside within the untrusted app subnet 2162. Each tenant VM 2166(1)-(N) may execute code within a respective container 2167(1)-(N) and may be communicatively coupled to an app subnet 2126 that may be included in a data plane app layer 2146 that may be included in a container egress VCN 2168. Each secondary VNIC 2172(1)-(N) may facilitate communication between the untrusted app subnet 2162 included in the data plane VCN 2118 and the app subnet included in the container egress VCN 2168. The container egress VCN may include a NAT gateway 2138 that may be communicatively coupled to the public Internet 2154 (e.g., public Internet 1854 of FIG. 18 ).

[0275] The Internet gateway 2134 included in the control plane VCN 2116 and included in the data plane VCN 2118 may be communicatively coupled to a metadata management service 2152 (e.g., metadata management system 1852 of FIG. 18 ), which may be communicatively coupled to the public Internet 2154. The public Internet 2154 may be communicatively coupled to a NAT gateway 2138 included in the control plane VCN 2116 and included in the data plane VCN 2118. The service gateway 2136 included in the control plane VCN 2116 and included in the data plane VCN 2118 may be communicatively coupled to cloud services 2156.

[0276] In some examples, the pattern illustrated by the architecture of block diagram 2100 of FIG. 21 may be considered an exception to the pattern illustrated by the architecture of block diagram 2000 of FIG. 20 and may be desirable for a customer of an IaaS provider when the IaaS provider cannot directly communicate with the customer (e.g., disconnected region). Each container 2167(1)-(N) contained in a VM 2166(1)-(N) per customer may be accessed in real time by the customer. The containers 2167(1)-(N) may be configured to make calls to each secondary VNIC 2172(1)-(N) contained in the app subnet 2126 of the data plane app tier 2146 that may be contained in the container egress VCN 2168. The secondary VNIC 2172(1)-(N) may send the call to the NAT gateway 2138, which may send the call to the public Internet 2154. In this example, containers 2167(1)-(N) that may be accessed in real time by a customer may be isolated from control plane VCN 2116 and may be isolated from other entities included in data plane VCN 2118. Containers 2167(1)-(N) may be isolated from other customer resources.

[0277] In another example, a customer may use container 2167(1)-(N) to invoke cloud service 2156. In this example, the customer may execute code in container 2167(1)-(N) that requests a service from cloud service 2156. Container 2167(1)-(N) may send the request to secondary VNIC 2172(1)-(N), which may send the request to a NAT gateway, which may send the request to public Internet 2154. Public Internet 2154 may send the request via Internet gateway 2134 to LB subnet 2122, which is included in control plane VCN 2116. In response to determining that the request is valid, LB subnet 2126 may send the request to app subnet 2126, which may send the request via service gateway 2136 to cloud service 2156.

[0278] It should be understood that the IaaS architectures 1800, 1900, 2000, 2100 depicted in the figures may include components other than those depicted. Additionally, the embodiments depicted in the figures are merely some examples of cloud infrastructure systems that may incorporate embodiments of the present disclosure. In some other embodiments, the IaaS systems may include more or fewer components than those depicted in the figures, may combine two or more components, or may have a different configuration or arrangement of components.

[0279] In one embodiment, the IaaS system described herein may include an offering of a suite of application, middleware, and database services that are delivered to customers in a self-service, subscription-based, elastically scalable, reliable, highly available, and secure manner. An example of such an IaaS system is Oracle Cloud Infrastructure (OCI) offered by the present assignee.

[0280] 22 illustrates an exemplary computer system 2200 on which various embodiments of the disclosure may be implemented. The system 2200 may be used to implement any of the computer systems described above. As shown in the figure, the computer system 2200 includes a processing unit 2204 that communicates with a number of peripheral subsystems via a bus subsystem 2202. These peripheral subsystems may include a processing acceleration unit 2206, an I / O subsystem 2208, a storage subsystem 2218, and a communication subsystem 2224. The storage subsystem 2218 includes a tangible computer-readable storage medium 2222 and a system memory 2210.

[0281] Bus subsystem 2202 provides a mechanism for allowing the various components and subsystems of computer system 2200 to communicate with each other as intended. Although bus subsystem 2202 is shown diagrammatically as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses. Bus subsystem 2202 may be any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. For example, such architectures may include a Peripheral Component Interconnect (PCI) bus, which may be implemented as an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a mezzanine bus manufactured to the IEEE P1386.1 standard.

[0282] The processing unit 2204, which may be implemented as one or more integrated circuits (e.g., conventional microprocessors or microcontrollers), controls the operation of the computer system 2200. One or more processors may be included in the processing unit 2204. These processors may include single-core processors or multi-core processors. In an embodiment, the processing unit 2204 may be implemented as one or more independent processing units 2232 and / or 2234, with a single-core processor or a multi-core processor included in each processing unit. In other embodiments, the processing unit 2204 may be implemented as a quad-core processing unit formed by integrating two dual-core processors into a single chip.

[0283] In various embodiments, the processing unit 2204 may execute various programs according to program code and may maintain multiple simultaneously executing programs or processes. At any particular time, some or all of the program code being executed may reside in the processor 2204 and / or in the storage subsystem 2218. Through appropriate programming, the processor 2204 may provide various functions as previously discussed. The computer system 2200 may further include a processing acceleration unit 2206, which may include a digital signal processor (DSP), a special purpose processor, and / or the like.

[0284] The I / O subsystem 2208 may include user interface input devices and user interface output devices. User interface input devices may include a keyboard, a pointing device such as a mouse or trackball, a touchpad or touch screen integrated into a display, a scroll wheel, a click wheel, a dial, a button, a switch, a keypad, a voice input device with a voice command recognition system, a microphone, and other types of input devices. User interface input devices may include motion detection devices and / or gesture recognition devices, such as, for example, a Microsoft Kinect® motion sensor that allows a user to control and interact with an input device, such as a Microsoft Xbox® 360 game controller, via a natural user interface using gestures and spoken commands. User interface input devices may include eye gesture recognition devices, such as the Google Glass® blink detector that detects a user's eye activity (e.g., "blinking" when taking a picture and / or selecting a menu) and translates the eye gestures as input to an input device (e.g., Google Glass®). Additionally, the user interface input devices may include a voice recognition detection device that allows a user to interact with a voice recognition system (eg, the Siri® navigator) via voice commands.

[0285] User interface input devices may include, but are not limited to, three dimensional (3D) mice, joysticks or pointing sticks, game pads, and graphic tablets, as well as audio / visual devices such as speakers, digital cameras, digital video cameras, portable media players, webcams, image scanners, fingerprint scanners, barcode readers 3D scanners, 3D printers, laser range finders, and eye tracking devices. Additionally, user interface input devices may include medical imaging input devices, such as, for example, computed tomography, magnetic resonance imaging, position emission tomography, and medical ultrasound devices. User interface input devices may include audio input devices, such as, for example, MIDI keyboards, digital musical instruments, and the like.

[0286] User interface output devices may include non-visual displays such as a display subsystem, indicator lights, or audio output devices. The display subsystem may be a flat panel device such as a flat panel device using a cathode ray tube (CRT), a liquid crystal display (LCD) or a plasma display, a projection device, a touch screen, etc. In general, use of the term "output device" is intended to include all possible types of devices and mechanisms for outputting information from computer system 2200 to a user or to another computer. For example, user interface output devices may include, but are not limited to, various display devices that visually convey textual, graphical, and audio / video information, such as monitors, printers, speakers, headphones, navigation systems, plotters, audio output devices, and modems.

[0287] Computer system 2200 may include a storage subsystem 2218 that includes the software elements shown as presently residing in system memory 2210. System memory 2210 may store program instructions readable and executable by processing unit 2204, as well as data generated during the execution of these programs.

[0288] Depending on the configuration and type of computer system 2200, the system memory 2210 may be volatile (such as random-access memory (RAM)) and / or non-volatile (such as read-only memory (ROM), flash memory, etc.). RAM typically contains data and / or program modules that are immediately accessible to and / or currently being operated on and executed by the processing unit 2204. In some implementations, the system memory 2210 may include a number of different types of memory, such as static random-access memory (SRAM) or dynamic random-access memory (DRAM). In some implementations, a basic input / output system (BIOS), containing the basic routines that help to transfer information between elements within the computer system 2200, such as during start-up, may typically be stored in ROM. By way of example and not limitation, system memory 2210 also illustrates application programs 2212, program data 2214, and operating system 2216, which may include client applications, web browsers, mid-tier applications, relational database management systems (RDBMS), and the like.Examples of operating systems 2216 may include various versions of Microsoft Windows®, Apple Macintosh®, and / or Linux operating systems, various commercially available UNIX® or UNIX-like operating systems (including, but not limited to, various GNU / Linux operating systems, Google Chrome® OS, etc.), and / or mobile operating systems such as iOS, Windows® Phone, Android® OS, BlackBerry® 17 OS, and Palm® OS operating systems.

[0289] The storage subsystem 2218 may provide a tangible, computer-readable storage medium for storing the basic programming and data configurations that provide the functionality of some embodiments. Software (programs, code modules, instructions) that, when executed by a processor, provide the aforementioned functionality may be stored in the storage subsystem 2218. These software modules or instructions may be executed by the processing unit 2204. The storage subsystem 2218 may provide a repository for storing data used in accordance with the present disclosure.

[0290] Storage subsystem 2200 may include computer readable storage medium reader 2220, which may be further connected to computer readable storage medium 2222. In combination with system memory 2210, optionally together, computer readable storage medium 2222 may comprehensively represent remote, local, fixed, and / or removable storage media as well as storage media for containing, storing, transmitting, and retrieving computer readable information on a temporary and / or more permanent basis.

[0291] The computer readable storage medium 2222 containing the code or portions of code may include any suitable medium known or used in the art, including storage media and communication media, such as, but not limited to, volatile and non-volatile, removable and non-removable media, implemented in any manner or technology for storing and / or transmitting information. The computer readable storage medium 2222 may include tangible computer readable storage media, such as RAM, ROM, electronically erasable programmable ROM (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disk (DVD), or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage, or other magnetic storage devices, or other tangible computer readable media. The computer readable storage medium 2222 may also include non-tangible computer readable media, such as data signals, data transmissions, or any other medium that may be used to transmit the desired information and that may be accessed by the computing system 2200.

[0292] By way of example, computer readable storage media 2222 may include hard disk drives that read from or write to non-removable, non-volatile magnetic media, magnetic disk drives that read from or write to removable, non-volatile magnetic disks, and optical disk drives that read from or write to removable, non-volatile optical disks, such as CD ROMs, DVDs, and Blu-ray disks or other optical media. Computer readable storage media 2222 may include, but are not limited to, Zip drives, flash memory cards, universal serial bus (USB) flash drives, secure digital (SD) cards, DVD disks, digital video tapes, and the like. The computer-readable storage media 2222 may include solid-state drives (SSDs) based on non-volatile memory such as flash memory-based SSDs, enterprise flash drives, solid-state ROMs, SSDs based on volatile memory such as solid-state RAM, dynamic RAM, static RAM, DRAM-based SSDs, magneto resistive RAM (MRAM) SSDs, and hybrid SSDs using a combination of DRAM and flash memory-based SSDs. Disk drives and associated computer-readable media may provide non-volatile storage of computer readable instructions, data structures, program modules, and other data for the computer system 2200.

[0293] The communications subsystem 2224 provides an interface to other computer systems and networks. The communications subsystem 2224 serves as an interface for receiving data from and transmitting data to other systems of the computer system 2200. For example, the communications subsystem 2224 may enable the computer system 2200 to connect to one or more devices via the Internet. In some embodiments, the communications subsystem 2224 may include radio frequency (RF) transceiver components for accessing wireless voice and / or data networks (e.g., using cellular technology, advanced data network technologies such as 3G, 4G, or EDGE (enhanced data rates for global evolution), Wi-Fi (using the IEEE 802.11 family of standards, or other mobile communications technologies, or any combination thereof), global positioning system (GPS) receiver components, and / or other components. In some embodiments, the communications subsystem 2224 may provide a wired network connection (e.g., Ethernet) in addition to or instead of a wireless interface.

[0294] In some embodiments, the communications subsystem 2224 may receive incoming communications in the form of structured and / or unstructured data feeds 2226, event streams 2228, event updates 2230, etc. on behalf of one or more users who may use the computer system 2200.

[0295] By way of example, the communications subsystem 2224 may be configured to receive data feeds 2226 in real time from users of social networks and / or other communications services, such as Twitter® feeds, Facebook® updates, web feeds, such as Rich Site Summary (RSS) feeds, and / or real-time updates from one or more third party sources.

[0296] Additionally, the communications subsystem 2224 may be configured to receive data in the form of a continuous data stream, which may include an event stream 2228 of real-time events and / or event updates 2230 that may have no explicit end and may be continuous in nature or without boundaries. Examples of applications that generate continuous data may include, for example, sensor data applications, financial tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, and the like.

[0297] The communications subsystem 2224 may be configured to output structured and / or unstructured data feeds 2226, event streams 2228, event updates 2230, etc. to one or more databases that can communicate with one or more streaming data source computers coupled to the computer system 2200.

[0298] The computer system 2200 can be one of a variety of types, including a handheld portable device (e.g., an iPhone® mobile phone, an iPad® computing tablet, a PDA), a wearable device (e.g., a Google Glass® head mounted display), a PC, a workstation, a mainframe, a ticket machine, a server rack, or any other data processing system.

[0299] Due to the ever-changing nature of computers and networks, the description of the computer system 2200 shown in the figure is intended to be merely a specific example. Many other configurations are possible, including more or fewer components than the system shown in the figure. For example, customized hardware may be used and / or particular elements may be implemented in hardware, firmware, software (including applets), or a combination thereof. Furthermore, connections to other computing devices, such as network input / output devices, may be employed. Based on the disclosure and teachings provided herein, one of ordinary skill in the art will appreciate other ways and / or manners for implementing various embodiments.

[0300] Although specific embodiments of the present disclosure have been described, various modifications, variations, alternative constructions, and equivalents are also encompassed within the scope of the present disclosure. The embodiments of the present disclosure are not limited to operation in one particular data processing environment, but can freely operate in multiple data processing environments. Furthermore, while the embodiments of the present disclosure have been described using a particular sequence of transactions and steps, it should be apparent to those skilled in the art that the scope of the present disclosure is not limited to the sequence of transactions and steps described. Various features and aspects of the foregoing embodiments may be used individually or together.

[0301] Furthermore, while embodiments of the present disclosure have been described using a particular combination of hardware and software, it should be recognized that other combinations of hardware and software are within the scope of the present disclosure. Embodiments of the present disclosure may be implemented in hardware alone, or in software alone, or using a combination thereof. Various processes described herein may be performed on the same processor or different processors in any combination. Thus, when a component or module is described as being configured to perform an operation, such configuration may be realized, for example, by designing an electronic circuit to perform the operation, by programming a programmable electronic circuit (such as a microprocessor) to perform the operation, or by any combination thereof. Processes may communicate using a variety of techniques, including, but not limited to, conventional techniques for inter-process communication, and different pairs of processes may use different techniques, or the same pair of processes may use different techniques at different times.

[0302] Accordingly, the specification and drawings should be regarded as illustrative rather than limiting in any sense. However, it is apparent that additions, subtractions, deletions, and other modifications and changes may be made to the specification and drawings without departing from the broader spirit and scope as set forth in the claims. Thus, although certain disclosed embodiments have been described, they are not intended to be limiting. Various modifications and equivalents are within the scope of the appended claims.

[0303] The use of the terms "a" and "an" and "the" and similar referents in the context of describing the disclosed embodiments (particularly in the context of the appended claims) should be construed to cover both the singular and the plural, unless otherwise indicated herein or clearly contradicted by context. The terms "comprising," "having," "including," and "containing" should be construed as open-ended (i.e., meaning "including, but not limited to") terms, unless otherwise noted. The term "connected" should be construed as partially or completely contained within, connected to, or joined together, even if there is something intervening. The recitation of ranges of values ​​herein is intended merely to serve as a shorthand method of individually referring to each separate value included in the range, unless otherwise indicated herein, and each separate value is incorporated herein as if it were individually recited herein. All methods described herein can be performed in any suitable order, unless otherwise indicated herein or otherwise clearly contradicted by context. The use of any and all examples or exemplary language (e.g., "etc.") provided herein is intended merely to better illuminate embodiments of the disclosure and does not pose limitations on the scope of the disclosure unless specifically claimed. No language in the specification should be construed as indicating any non-claimed element as essential to the practice of the disclosure.

[0304] Disjunctive language, such as the phrase "at least one of X, Y, or Z," is generally intended to be understood within the context as being used to state that an item, condition, etc. may be either X, Y, or Z, or any combination thereof (e.g., X, Y, and / or Z), unless expressly stated otherwise. Thus, such disjunctive language is generally not intended to, and should not, imply that an embodiment requires that at least one of X, at least one of Y, or at least one of Z, respectively, be present.

[0305] Preferred embodiments of the present disclosure are described herein, including the best mode known to the inventors for carrying out the disclosure. Variations of such preferred embodiments may become apparent to those skilled in the art upon reading the foregoing description. The inventors expect that those skilled in the art will adopt such variations as necessary, and the inventors intend for the present disclosure to be practiced other than as specifically described herein. Accordingly, the present disclosure includes all modifications and equivalents of the subject matter recited in the claims appended hereto as permitted by applicable law. Moreover, any combination of the foregoing elements in all possible variations of the embodiments is encompassed by the present disclosure unless otherwise indicated herein or otherwise clearly contradicted by context.

[0306] All references cited in this specification, including publications, patent applications, and patents, are hereby incorporated by reference to the same extent as if each reference was individually indicated to be incorporated by reference and was set forth in its entirety herein.

[0307] Although aspects of the disclosure have been described in the foregoing specification with reference to specific embodiments thereof, those skilled in the art will recognize that the disclosure is not limited thereto. Various features and aspects of the foregoing disclosure may be used individually or together. Moreover, the embodiments may be utilized in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of the specification. Thus, the specification and drawings should be regarded as illustrative rather than restrictive.

Claims

1. 1. A computer-implemented method comprising: receiving, by a multi-cloud infrastructure included in a first cloud environment, a request to create a network link between a first virtual network in the first cloud environment and a second virtual network in a second cloud environment, wherein the first virtual network in the first cloud environment has been pre-created to enable users associated with a customer's tenancy in the second cloud environment to access one or more services offered in the first cloud environment; The method further includes creating the network link between the first virtual network and the second virtual network, the creating comprising: encapsulating, by a first link-enabled virtual network in the second cloud environment, traffic received from the second virtual network to generate encapsulated traffic; forwarding, by a second link-enabled virtual network in the second cloud environment, the encapsulated traffic received from the first link-enabled virtual network to a hub virtual network included in the second cloud environment; decapsulating, by a third link-enabled virtual network in the first cloud environment, the encapsulated traffic received from the hub virtual network included in the second cloud environment to generate decapsulated traffic; transmitting, by the third link-enabled virtual network in the first cloud environment, the decapsulated traffic to the first virtual network in the first cloud environment.

2. 2. The method of claim 1, wherein each of the first link-enabled virtual network, the second link-enabled virtual network, and the third link-enabled virtual network is assigned a unique classless inter-domain routing IP address space.

3. 10. The method of claim 1, wherein the second cloud environment includes multiple customer tenancies, and for each customer tenancy included in the multiple customer tenancies, (i) the first link-enabled virtual network and the second link-enabled virtual network are created in the second cloud environment, and (ii) the third link-enabled virtual network is created in the first cloud environment.

4. The method of claim 3 , wherein the hub virtual network included in the second cloud environment handles traffic for tenancies of the multiple customers included in the second cloud environment.

5. 10. The method of claim 1, wherein the hub virtual network included in the second cloud environment transmits the encapsulated traffic to a hub virtual network included in the first cloud environment via an interconnect coupling the second cloud environment to the first cloud environment.

6. 6. The method of claim 5, wherein the hub virtual network included in the second cloud environment sends the encapsulated traffic to a VNIC included in the hub virtual network included in the first cloud environment, and the VNIC is configured to forward the encapsulated traffic to the third link-enabled virtual network included in the first cloud environment.

7. 10. The method of claim 1, wherein the third link-enabled virtual network in the first cloud environment sends the decapsulated traffic to the first virtual network in the first cloud environment via a dynamic routing gateway connection.

8. 2. The method of claim 1, wherein the first link-enabled virtual network includes a first pair of virtual network adapters, each of the virtual network adapters configured to encapsulate traffic received from the second virtual network to generate the encapsulated traffic.

9. 2. The method of claim 1, wherein the second link-enabled virtual network includes a pair of virtual network forwarders, each of the virtual network forwarders configured to forward the encapsulated traffic received from the first link-enabled virtual network to the hub virtual network included in the second cloud environment.

10. 10. The method of claim 1, wherein the third link-enabled virtual network includes a second pair of virtual network adapters, each of the virtual network adapters configured to decapsulate the encapsulated traffic received from the hub virtual network included in the second cloud environment.

11. A program for causing one or more processors to execute the method according to any one of claims 1 to 10.

12. one or more processors; A computing device comprising: a storage medium storing the program of claim 11.

Images