Extensible hub and spoke topology for routing using compute instances

Abstract

A technique for creating a network link between a first customer's virtual network in a first cloud environment and a second customer's virtual network in a second cloud environment is described. A first customer's virtual network is created in the first cloud environment to allow users associated with the customer's tenancy in the second cloud environment to access one or more services provided in the first cloud environment. This network link is created based on one or more link-enabled virtual networks deployed in the first and second cloud environments.

Description

Technical Field

[0001] Cross - reference to Related Applications This application is a non - provisional application of each of the following provisional applications and claims the benefit of each of the following provisional applications. The entire content of each of the following provisional applications is incorporated herein by reference for all purposes. (1) U.S. Patent Provisional Application No. 63 / 465,394, filed on May 10, 2023, and (2) U.S. Patent Provisional Application No. 63 / 594,759, filed on October 31, 2023.

[0002] Field The present disclosure relates to cloud architectures, and more particularly, to techniques for linking two cloud environments so that a user of one cloud environment can use services provided by another cloud environment.

Background Art

[0003] Background In recent years, there has been a dramatic increase in the adoption of cloud services, and this trend is continuing to grow. Various cloud environments are provided by different cloud service providers (CSPs), and each cloud environment provides a set of one or more cloud services. The set of cloud services provided in a cloud environment may include, but is not limited to, one or more different types of services such as Software - as - a - Service (SaaS) services, Infrastructure - as - a - Service (IaaS) services, Platform - as - a - Service (PaaS) services, etc.

[0004] While various cloud environments are currently available, each cloud environment provides a closed ecosystem to its subscribers. As a result, customers of a cloud environment are limited to using only the services provided within that environment. There is no easy way for a customer subscribed to a specific cloud environment provided by a particular CSP to use services provided in different cloud environments provided by different CSPs through that specific cloud environment.

[0005] The embodiments described herein address these and other issues. [Overview of the Initiative]

[0006] overview This disclosure relates to an improved cloud architecture, and more specifically, to a technology for linking two clouds so that a user of one cloud environment can use services provided by another different cloud environment. Various embodiments are described herein, including methods, systems, and non-temporary computer-readable storage media for storing programs, code, or instructions executable by one or more processors. Some embodiments may be implemented by using a computer program product that contains computer programs / instructions, which, when executed by a processor, cause the processor to perform one of the methods described herein.

[0007] Embodiments of this disclosure provide a multi-cloud control plane (MCCP) framework that provisions the ability to deliver services from a specific cloud network (e.g., Oracle Cloud Infrastructure (OCI)) to users on other clouds (e.g., within Google Cloud (GCP)). The MCCP framework enables users (in other cloud environments) to access services in the cloud environment (e.g., PaaS services) while providing a user experience as close as possible to the user experience in the user's native cloud environment. The main value proposition of MCCP is that customers will be able to experience the full data plane functionality of services in external clouds.

[0008] One aspect of this disclosure is to provide a first compute virtual machine connected to a first customer's virtual cloud network (VCN) and a hub in a first cloud environment, wherein the first compute virtual machine is associated with a first plurality of virtual network interface cards (VNICs), and the first plurality of VNICs include (i) a first VNIC associated with a first network namespace that enables communication between the first customer's VCN and the first compute virtual machine, and (ii) a second VNIC associated with a second network namespace that enables communication between the first compute virtual machine and a hub, wherein the first network namespace is different from the second network namespace; and to provide a second compute virtual machine connected to a second customer's VCN and a hub in a first cloud environment, wherein the second compute virtual machine is associated with a second plurality of VNICs. The present invention provides a method comprising: providing a second set of VNICs including (i) a first VNIC associated with a third network namespace and enabling communication between a second customer's VCN and a second compute virtual machine; and (ii) a second VNIC associated with a second network namespace and enabling communication between a second compute virtual machine and a hub; enabling communication between a first customer's VCN in a first cloud environment and a second cloud environment using the first compute virtual machine and hub; and enabling communication between a second customer's VCN in a first cloud environment and a second cloud environment using the second compute virtual machine and hub.

[0009] Aspects of this disclosure provide a computing device comprising one or more data processors and a non-temporary computer-readable storage medium containing instructions, which, when executed on one or more data processors, cause the computing device to perform some or all of the methods disclosed herein.

[0010] Another aspect of the present disclosure provides a computer program product, tangibly embodied in a non-temporary machine-readable storage medium, which includes instructions configured to cause one or more data processors to perform some or all of the methods disclosed herein.

[0011] The foregoing, along with other features and embodiments, will become even clearer when referring to the following specification, claims, and accompanying drawings.

[0012] The features, embodiments, and advantages of this disclosure will be better understood when the following detailed description is read with reference to the accompanying drawings. [Brief explanation of the drawing]

[0013] [Figure 1] This is a high-level diagram of a distributed environment showing a virtual cloud network or overlay cloud network hosted by a cloud service provider infrastructure, according to one embodiment. [Figure 2] This figure shows a simplified architectural diagram of the physical components in the physical network within the CSPI according to one embodiment. [Figure 3] This figure shows an exemplary configuration within a CSPI in which a host machine is connected to multiple network virtualization devices (NVDs), according to one embodiment. [Figure 4] This figure shows the connection between a host machine and an NVD to implement I / O virtualization to support multi-tenancy functionality, according to one embodiment. [Figure 5] This figure shows a simplified block diagram of a physical network provided by CSPI according to one embodiment. [Figure 6]A simplified high-level diagram of a distributed environment comprising multiple cloud environments provided by different cloud service providers (CSPs) according to one embodiment, wherein a particular cloud environment includes a specific cloud environment that provides specialized infrastructure enabling one or more cloud services provided by that particular cloud environment to be used by customers of other cloud environments. [Figure 7] This figure shows an exemplary high-level architecture of a multi-cloud control plane (MCCP) according to some embodiments. [Figure 8] This figure shows an exemplary system diagram illustrating the components of a multi-cloud control plane (MCCP) according to some embodiments. [Figure 9] This figure shows a high-level block diagram of a network link component according to one embodiment. [Figure 10-1] This figure shows a detailed architecture of a network link according to one embodiment. [Figure 10-2] This figure shows a detailed architecture of a network link according to one embodiment. [Figure 11] This figure shows an exemplary flowchart illustrating the process of establishing a network link according to one embodiment. [Figure 12] This figure shows a schematic diagram of a computing virtual machine including multiple VNICs according to one embodiment. [Figure 13] This figure shows a schematic diagram of a computing virtual machine including multiple network namespaces according to one embodiment. [Figure 14] This figure shows a spoke and hub network architecture according to one embodiment. [Figure 15] This figure shows an exemplary flowchart illustrating the process performed to enable communication in the spoke and hub architecture of Figure 14, according to one embodiment. [Figure 16]A block diagram showing one pattern for implementing a cloud infrastructure as a service system according to at least one embodiment. [Figure 17] A block diagram showing another pattern for implementing a cloud infrastructure as a service system according to at least one embodiment. [Figure 18] A block diagram showing another pattern for implementing a cloud infrastructure as a service system according to at least one embodiment. [Figure 19] A block diagram showing another pattern for implementing a cloud infrastructure as a service system according to at least one embodiment. [Figure 20] A block diagram showing an exemplary computer system according to at least one embodiment.

Mode for Carrying Out the Invention

[0014] Detailed Description In the following description, for the purpose of explanation, specific details are set forth in order to provide a thorough understanding of one embodiment. However, it is apparent that various embodiments may be practiced without these specific details. Each figure and description are not intended to be limiting. The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." All embodiments or designs described herein as "exemplary" should not necessarily be construed as preferred or advantageous over other embodiments or designs.

[0015] This disclosure generally relates to improved cloud architectures, and more specifically, to techniques for linking two clouds so that a user of one cloud environment can use services provided by another different cloud environment. Various embodiments are described herein, including methods, systems, and non-temporary computer-readable storage media for storing programs, code, or instructions executable by one or more processors. Some embodiments may be implemented by using computer program products that, when executed by a processor, cause the processor to perform one of the methods described herein.

[0016] Embodiments of this disclosure provide a multi-cloud control plane (MCCP) framework that provisions the ability to deliver services from a specific cloud network (e.g., Oracle Cloud Infrastructure (OCI)) to users on other clouds (e.g., Google Cloud). The MCCP framework enables users (in other cloud environments) to access services (e.g., PaaS services) in the cloud environment while providing a user experience as close as possible to the user experience in the user's native cloud environment. The main value proposition of MCCP is that customers will be able to experience the full data plane functionality of services in external clouds.

[0017] MCCP enables users of a second cloud infrastructure (e.g., Azure users) to access resources (e.g., database resources) provided by a first cloud infrastructure (e.g., OCI) in a transparent manner. In particular, services provided by the first cloud infrastructure appear as "native" services in the second cloud infrastructure. This allows customers of the second cloud infrastructure to natively access services provided by the first cloud infrastructure. As described below, MCCP is a collection of microservices running in the first cloud infrastructure that expose resources of the first cloud infrastructure to be used by external cloud users (e.g., users of the second cloud infrastructure). Each of these microservices acts as a proxy, providing communication with resources provided by the first cloud infrastructure.

[0018] Examples of cloud networks The term "cloud service" is typically used to refer to services made available on demand (e.g., through a subscription model) by users or customers using systems and infrastructure provided by a Cloud Service Provider (CSP) (cloud infrastructure). Typically, the servers and systems that make up the CSP's infrastructure are separate from the customer's own on-premises servers and systems. Therefore, customers can utilize cloud services provided by the CSP without needing to purchase separate hardware and software resources for the service. Cloud services are designed to provide subscribers with easy and scalable access to application and computing resources without requiring the customer to invest in procuring the infrastructure used to deliver the service.

[0019] There are multiple cloud service providers offering various types of cloud services. These include various types or models of cloud services, such as SaaS (Software-as-a-Service), PaaS (Platform-as-a-Service), and IaaS (Infrastructure-as-a-Service).

[0020] Customers can subscribe to one or more cloud services provided by the CSP. Customers can be any entity, such as an individual, organization, or company. When a customer subscribes to or registers for a service provided by the CSP, a tenancy or account is created for that customer. The customer can then use that account to access one or more subscribed cloud resources associated with the account.

[0021] As mentioned earlier, IaaS (Infrastructure as a Service) is a specific type of cloud computing service. In the IaaS model, the CSP provides the infrastructure (called Cloud Service Provider Infrastructure or CSPI) that can be used by the customer to build their own customizable network and deploy their resources. Thus, the customer's resources and network are hosted in a distributed environment by the infrastructure provided by the CSP. This differs from traditional computing, where the customer's resources and network are hosted by infrastructure provided by the customer.

[0022] A CSPI can comprise interconnected high-performance computing resources, including various host machines, memory resources, and network resources that form a physical network, also known as an underlay network. Resources in a CSPI may be distributed across one or more data centers, which may be geographically dispersed across one or more geographical regions. To provide a virtualized distributed environment, virtualization software may run on these physical resources. This virtualization creates an overlay network (also known as a software-based network, software-defined network, or virtual network) on top of the physical network. The physical network of a CSPI provides the foundation for creating one or more overlay or virtual networks on top of the physical network. The physical network (or underlay network) includes physical network devices such as physical switches, routers, computers, and host machines. An overlay network is a logical (or virtual) network that operates on top of the physical underlay network. A particular physical network can support one or more overlay networks. Overlay networks typically use encapsulation techniques to distinguish traffic belonging to different overlay networks. A virtual network, or overlay network, is also called a virtual cloud network (VCN). A virtual network is implemented using software virtualization technologies (e.g., hypervisors, network virtualization devices (NVDs) (e.g., smart NICs), top-of-rack (TOR) switches, virtualization functions implemented by smart TORs that perform one or more functions performed by NVDs, and other mechanisms) to create a layer of network abstraction that can run on top of a physical network. Virtual networks can take many forms, including peer-to-peer networks and IP networks.A virtual network is typically either a Layer 3 IP network or a Layer 2 VLAN. This method of virtual networking or overlay networking is often referred to as a virtual Layer 3 network or overlay Layer 3 network. Examples of protocols developed for virtual networks include IP-in-IP (or Generic Routing Encapsulation (GRE)), Virtual Extensible LAN (VXLAN - IETF RFC 7348), Virtual Private Networks (VPN) (e.g., MPLS Layer 3 Virtual Private Network (RFC 4364)), VMware's NSX, and Geneve (Generic Network Virtualization Encapsulation).

[0023] In the case of IaaS, the infrastructure provided by the CSP (CSPI) can be configured to deliver virtualized computing resources over a public network (e.g., the internet). In the IaaS model, the cloud computing service provider can host infrastructure components (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer)). In some cases, the IaaS provider may also provide various services (e.g., billing, monitoring, logging, security, load balancing, and clustering) as they arise in conjunction with those infrastructure components. Since these services can be policy-driven, IaaS users may implement policies to drive load balancing to maintain application availability and performance. The CSPI provides infrastructure and a set of complementary cloud services that enable customers to build and run a wide range of applications and services in a highly available hosted distributed environment. The CSPI provides high-performance computing resources and computing power, as well as storage capacity, within a flexible virtual network that is securely accessible from a variety of networked locations, including the customer's on-premises network. When a customer subscribes to or registers for an IaaS service provided by a CSP, the tenancy created for that customer is a secure and isolated partition within the CSP where the customer can create, organize, and manage their own cloud resources.

[0024] Customers can build their own virtual networks using the computing, memory, and network resources provided by CSPI. One or more customer resources or workloads, such as compute instances, can be deployed to these virtual networks. For example, a customer can build one or more customizable private virtual networks called Virtual Cloud Networks (VCNs) using the resources provided by CSPI. A customer can deploy one or more customer resources, such as compute instances, to their VCNs. Compute instances can take the form of virtual machines, bare metal instances, etc. Thus, CSPI provides the infrastructure and a set of complementary cloud services that enable customers to build and run a wide range of applications and services in a highly available hosted virtual environment. Customers do not manage or control the underlying physical resources provided by CSPI, but they can control the operating system, storage, and deployed applications, and, in some cases, have limited control over selected network components (e.g., firewalls).

[0025] The CSP may provide a console that enables customers and network administrators to configure, access, and manage resources deployed in the cloud using CSPI resources. In one embodiment, this console provides a web-based user interface that can be used to access and manage the CSPI. In some implementations, the console is a web-based application provided by the CSP.

[0026] CSPI may support single-tenancy or multi-tenancy architectures. In a single-tenancy architecture, software components (e.g., applications, databases) or hardware components (e.g., host machines or servers) work for a single customer or tenant. In a multi-tenancy architecture, software components or hardware components work for multiple customers or tenants. Therefore, in a multi-tenancy architecture, CSPI resources are shared among multiple customers or tenants. In a multi-tenancy situation, precautions are taken and preventative measures are implemented within CSPI to ensure that each tenant's data is isolated and remains invisible to other tenants.

[0027] In a physical network, a network endpoint ("endpoint") refers to a computing device or system that is connected to a physical network and communicates with the connected network. Network endpoints within a physical network may be connected to a local area network (LAN), a wide area network (WAN), or other types of physical networks. Examples of traditional endpoints within a physical network include modems, hubs, bridges, switches, routers, and other network devices, as well as physical computers (or host machines). Each physical device within a physical network has a fixed network address that can be used to communicate with the device. This fixed network address can be a Layer 2 address (e.g., a MAC address), a fixed Layer 3 address (e.g., an IP address), etc. In a virtual environment or virtual network, endpoints can include various virtual endpoints, such as virtual machines hosted by components of a physical network (e.g., hosted by a physical host machine). These endpoints within a virtual network are addressed by overlay addresses, such as overlay Layer 2 addresses (e.g., overlay MAC addresses) and overlay Layer 3 addresses (e.g., overlay IP addresses). Network overlays enable flexibility by allowing network administrators to move overlay addresses between network endpoints using software management (e.g., by software implementing the control plane of the virtual network). Therefore, unlike physical networks, in virtual networks, overlay addresses (e.g., overlay IP addresses) can be moved from one endpoint to another using network management software. Because virtual networks are built on top of physical networks, communication between components within a virtual network involves both the virtual network and the underlying physical network.To facilitate such communication, CSPI components are configured to learn and store mappings that map overlay addresses in the virtual network to actual physical addresses in the underlying network, and vice versa. These mappings are then used to facilitate communication. Customer traffic is encapsulated to facilitate routing within the virtual network.

[0028] Therefore, physical addresses (e.g., physical IP addresses) are associated with components within a physical network, while overlay addresses (e.g., overlay IP addresses) are associated with entities within a virtual network or overlay network. A physical IP address is an IP address associated with a physical device (e.g., a network device) within an underlying network or physical network. For example, each NVD has an associated physical IP address. An overlay IP address is an overlay address associated with an entity within an overlay network, such as being associated with a compute instance within a customer's virtual cloud network (VCN). Two different customers or tenants, each having their own private VCN, may be able to use the same overlay IP address within their VCNs without knowing anything about each other. Both physical IP addresses and overlay IP addresses are types of real IP addresses. These IP addresses exist separately from virtual IP addresses. A virtual IP address is typically a single IP address that represents or maps to multiple real IP addresses. A virtual IP address provides a one-to-many mapping between a virtual IP address and multiple real IP addresses. For example, a load balancer may use a VIP to map to or represent multiple servers, each server having its own actual IP address.

[0029] The cloud infrastructure, or CSPI, is physically hosted in one or more data centers in one or more regions worldwide. The CSPI may include components within a physical network or underpinning network, and virtual components (e.g., virtual networks, compute instances, virtual machines, etc.) within a virtual network built on top of the physical network components. In some embodiments, the CSPI is organized and hosted within realms, regions, and availability domains. A region is typically a localized geographical area containing one or more data centers. Regions are generally independent of each other and may be separated by vast distances, for example, across multiple countries or continents. For example, one region may be in Australia, another in Japan, and yet another in India. CSPI resources are partitioned across regions such that each region contains an independent subset of CSPI resources within that region itself. Each region may provide a set of core infrastructure services and resources, including compute resources (e.g., bare metal servers, virtual machines, containers, and associated infrastructure), storage resources (e.g., block volume storage, file storage, object storage, archive storage), network resources (e.g., virtual cloud networks (VCNs), load balancing resources, connectivity to on-premises networks), database resources, edge network resources (e.g., DNS), and access management and monitoring resources. Each region generally has multiple paths to connect to other regions within the realm.

[0030] Because using nearby resources is faster than using distant resources, applications are generally deployed within the region where they are most frequently used (i.e., on the infrastructure associated with that region). Applications may also be deployed across different regions for various reasons, such as redundancy to mitigate the risk of region-wide events like major weather systems or earthquakes, to meet changing requirements regarding legal jurisdictions, tax areas, and other business or social standards.

[0031] Data centers within a region can be further organized and subdivided into availability domains (ADs). An availability domain may correspond to one or more data centers located within a region. A region may consist of one or more availability domains. In such a distributed environment, CSPI resources are either region-specific, such as virtual cloud networks (VCNs), or availability domain-specific, such as compute instances.

[0032] Active Directory (AD) instances within a region are configured to be isolated from each other, fault-tolerant, and extremely unlikely to fail simultaneously. This is achieved by ADs that do not share critical infrastructure resources such as networks, physical cables, cable routes, and cable entry points, so that a failure in one AD within a region is unlikely to affect the availability of other ADs within the same region. ADs within the same region may be connected to each other by low-latency, high-bandwidth networks, enabling the provision of high-availability connectivity to other networks (e.g., the internet, customer on-premises networks) and the construction of replicated systems across multiple ADs for both high availability and disaster recovery. Cloud services use multiple ADs to ensure high availability and protect against resource failures. As the infrastructure provided by the IaaS provider grows, more regions and ADs with additional capacity may be added. Traffic between availability domains is typically encrypted.

[0033] In one embodiment, regions are grouped into realms. A realm is a logical collection of regions. Realms are isolated from each other and do not share any data. Regions within the same realm may communicate with each other, but regions in different realms cannot. A customer's tenancy or account, along with the CSP, resides within a single realm and may be distributed across one or more regions belonging to that realm. Typically, when a customer subscribes to an IaaS service, a tenancy or account for that customer is created within a region specified by the customer within a realm (referred to as the "home" region). A customer can extend their tenancy across one or more other regions within a realm. A customer cannot access regions that are not in the realm where their tenancy resides.

[0034] An IaaS provider can offer multiple realms, each catering to the specific needs of a particular set of customers or users. For example, a commercial realm may be offered to commercial customers. Another example is a realm offered to a specific country for customers within that country. Yet another example is a government realm offered to a government, for example. A government realm may cater to the specific needs of a particular government and may have a higher level of security than a commercial realm. For example, Oracle Cloud Infrastructure (OCI) currently offers two realms: one for commercial regions and another for government cloud regions (e.g., FedRAMP certified and IL5 certified).

[0035] In some embodiments, an Active Directory (AD) may be subdivided into one or more fault domains. A fault domain is a group of infrastructure resources within the AD to provide anti-affinity. Fault domains enable the distribution of compute instances so that multiple compute instances do not reside on the same physical hardware within a single AD. This distribution is known as anti-affinity. A fault domain refers to a set of hardware components (computers, switches, etc.) that share a single point of failure. The compute pool is logically divided into fault domains. Therefore, a hardware failure or maintenance event on compute hardware affecting one fault domain does not affect instances in other fault domains. Depending on the embodiment, the number of fault domains per AD may vary. For example, in some embodiments, each AD contains three fault domains. Fault domains function as logical data centers within the AD.

[0036] When a customer subscribes to an IaaS service, resources from CSPI are provisioned for the customer and associated with the customer's tenancy. The customer can use these provisioned resources to build private networks and deploy resources to these networks. A customer's network hosted in the cloud by CSPI is called a Virtual Cloud Network (VCN). A customer can configure one or more Virtual Cloud Networks (VCNs) using the CSPI resources allocated to them. A VCN is a Virtual Private Network or Software-Defined Private Network. Customer resources deployed within a customer's VCN can include compute instances (e.g., virtual machines, bare metal instances) and other resources. These compute instances may represent various customer workloads, such as applications, load balancers, and databases. Compute instances deployed in a VCN can communicate with publicly accessible endpoints ("public endpoints") via public networks such as the internet, with other instances in the same VCN or other VCNs (e.g., other VCNs of the customer, or VCNs not belonging to the customer), with the customer's on-premises data centers or networks, and with service endpoints and other types of endpoints.

[0037] A CSP may use CSPI to provide various services. In some cases, a CSPI customer may act as a service provider themselves and provide services using CSPI resources. A service provider may expose service endpoints characterized by identification information (e.g., IP address, DNS name, and DNS port). A customer's resources (e.g., compute instances) can consume a particular service by accessing the service endpoint exposed by the service for that particular service. These service endpoints are generally publicly accessible by users using the public IP address associated with the endpoint over a public communication network such as the internet. Publicly accessible network endpoints are sometimes called public endpoints.

[0038] In one embodiment, a service provider may expose a service through an endpoint for the service (sometimes called a service endpoint). Customers of the service can then access the service using this service endpoint. In one implementation, the service endpoint provided for a service may be accessible to multiple customers who wish to consume that service. In another implementation, a dedicated service endpoint may be provided to a customer, allowing only that customer to access the service using that dedicated service endpoint.

[0039] In one embodiment, when a VCN is created, it is associated with a private overlay classless inter-domain routing (CIDR) address space, which is a range of private overlay IP addresses assigned to the VCN (e.g., 10.0 / 16). The VCN includes associated subnets, route tables, and gateways. The VCN resides within a single region but can span one, more, or all of the region's availability domains. A gateway is a virtual interface configured for the VCN that enables communication of traffic to and from the VCN and one or more endpoints outside the VCN. One or more different types of gateways may be configured for the VCN to enable communication with different types of endpoints.

[0040] A VCN can be subdivided into one or more subnets, such as one or more subnets. Therefore, a subnet is a unit of configuration or subdivision that can be created within a VCN. A VCN can contain one or more subnets. Each subnet within a VCN is associated with a contiguous range of overlay IP addresses (e.g., 10.0.0.0 / 24 and 10.0.1.0 / 24) that represents a subset of the address space within the VCN's address space and does not overlap with other subnets within that VCN.

[0041] Each compute instance is associated with a virtual network interface card (VNIC) that allows the compute instance to join a subnet in the VCN. A VNIC is a logical representation of a physical network interface card (NIC). Generally, a VNIC is the interface between an entity (e.g., compute instance, service) and a virtual network. A VNIC resides in a subnet and has one or more associated IP addresses and associated security rules or security policies. A VNIC is equivalent to a Layer 2 port on a switch. A VNIC is connected to a compute instance and connects to a subnet within the VCN. The VNIC associated with a compute instance enables the compute instance to become part of a subnet in the VCN and allows the compute instance to communicate (e.g., send and receive packets) with endpoints on the same subnet as the compute instance, endpoints in different subnets within the VCN, or endpoints outside the VCN. Thus, the VNIC associated with a compute instance determines how the compute instance connects to endpoints inside and outside the VCN. A VNIC for a compute instance is created and associated with that compute instance when the compute instance is created and added to a subnet within the VCN. For a subnet containing a set of compute instances, the subnet contains a VNIC corresponding to the set of compute instances, and each VNIC connects to one of the compute instances in the set.

[0042] Each compute instance is assigned a private overlay IP address via the VNIC associated with it. This private overlay IP address is assigned to the VNIC associated with the compute instance when the compute instance is created and is used to route traffic to and from the compute instance. All VNICs within a given subnet use the same route table, security lists, and DHCP options. As mentioned earlier, each subnet within a VCN is associated with a contiguous range of overlay IP addresses (e.g., 10.0.0.0 / 24 and 10.0.1.0 / 24) that represents a subset of the address space within the VCN's address space and does not overlap with other subnets within that VCN. For a VNIC on a particular subnet of a VCN, the private overlay IP address assigned to the VNIC is an address from the contiguous range of overlay IP addresses assigned to that subnet.

[0043] In one embodiment, a compute instance may optionally be assigned additional overlay IP addresses in addition to its private overlay IP address, such as one or more public IP addresses if it is in a public subnet. These multiple addresses may be assigned to the same VNIC or across multiple VNICs associated with the compute instance. However, each instance has a primary VNIC, which is created during instance startup and associated with the overlay private IP address assigned to the instance, and this primary VNIC cannot be removed. Additional VNICs, called secondary VNICs, may be added to an existing instance in the same availability domain as the primary VNIC. All VNICs are in the same availability domain as the instance. Secondary VNICs may be in a subnet within the same VCN as the primary VNIC, or in a different subnet that is either in the same VCN or a different VCN.

[0044] If a compute instance is located in a public subnet, it may optionally be assigned a public IP address. When a subnet is created, it can be designated as either a public or private subnet. A private subnet means that resources within the subnet (e.g., compute instances) and associated VNICs cannot have public overlay IP addresses. A public subnet means that resources within the subnet and associated VNICs can have public IP addresses. Customers can specify subnets to reside within a single availability domain or across multiple availability domains within a region or realm.

[0045] As mentioned earlier, a VCN may be subdivided into one or more subnets. In one embodiment, a virtual router (VR) configured for the VCN (referred to as a VCN VR or simply a VR) enables communication between subnets of the VCN. For subnets within a VCN, the VR represents the logical gateway for that subnet, allowing the subnet (i.e., compute instances on that subnet) to communicate with endpoints on other subnets within the VCN and with other endpoints outside the VCN. A VCN VR is a logical entity configured to route traffic between VNICs within the VCN and virtual gateways ("gateways") associated with the VCN. Gateways are described further below with respect to Figure 1. A VCN VR is a Layer 3 / IP layer concept. In one embodiment, there is one VCN VR for one VCN, and the VCN VR has an unlimited number of ports, possibly addressed by IP addresses, with one port for each subnet of the VCN. In this way, the VCN VR has a different IP address for each subnet within the VCN to which the VCN VR is connected. The VR is also connected to various gateways configured for the VCN. In one embodiment, a specific overlay IP address from a subnet's overlay IP address range is reserved for the VCN VR ports of that subnet. For example, consider a VCN that includes two subnets, each having the associated address ranges 10.0 / 16 and 10.1 / 16, respectively. For the first subnet in the VCN having the address range 10.0 / 16, addresses from this range are reserved for the VCN VR ports of that subnet. In some cases, the first IP address from this range may be reserved for the VCN VR. For example, for a subnet having the overlay IP address range 10.0 / 16, the IP address 10.0.0.1 may be reserved for the VCN VR ports of that subnet.For a second subnet within the same VCN having the address range 10.1 / 16, the VCN VR may have a port on that second subnet having the IP address 10.1.0.1. The VCN VR has a different IP address for each subnet within the VCN.

[0046] In some other embodiments, each subnet within a VCN may include a VR associated with itself, which is addressable by the subnet using a reserved IP address or a default IP address associated with the VR. The reserved IP address or default IP address may be, for example, the first IP address from the range of IP addresses associated with that subnet. A VNIC within a subnet can communicate with the VR associated with the subnet using this default IP address or reserved IP address (e.g., send and receive packets). In such embodiments, the VR is the entry / exit point for that subnet. A VR associated with a subnet within a VCN can communicate with other VRs associated with other subnets within the VCN. A VR can also communicate with a gateway associated with the VCN. The VR functionality of a subnet operates on or is performed by one or more NVDs that are running the VNIC functionality of the VNICs within the subnet.

[0047] Route tables, security rules, and DHCP options may be configured for the VCN. The route table is the VCN's virtual route table and contains rules for routing traffic from subnets within the VCN to destinations outside the VCN via a gateway or specially configured instance. The VCN's route table can be customized to control how packets are forwarded / routed to and from the VCN. DHCP options refer to configuration information that is automatically provided to an instance when it starts up.

[0048] Security rules configured for a VCN represent the VCN's overlay firewall rules. Security rules include ingress and egress rules and can specify the types of traffic (e.g., based on protocol and port) allowed to enter and exit instances within the VCN. Customers can choose whether specific rules are stateful or stateless. For example, a customer can allow incoming SSH traffic to a set of instances from any location by configuring a stateful ingress rule with source CIDR 0.0.0.0 / 0 and destination TCP port 22. Security rules can be implemented using network security groups or security lists. A network security group consists of a set of security rules that apply only to resources within that group. A security list, on the other hand, contains rules that apply to all resources within any subnet using the security list. A VCN may have a default security list containing default security rules. DHCP options configured for a VCN provide configuration information that is automatically provided to instances within the VCN when they start up.

[0049] In one embodiment, VCN configuration information is determined and stored by the VCN control plane. VCN configuration information may include, for example, information about address ranges associated with the VCN, subnets within the VCN and associated information, one or more VRs associated with the VCN, compute instances within the VCN and associated VNICs, NVDs (e.g., VNICs, VRs, gateways) that perform various virtualized network functions associated with the VCN, VCN status information, and other VCN-related information. In one embodiment, a VCN distribution service exposes the configuration information stored by the VCN control plane or a portion thereof to the NVD. The distributed information may be used to update information stored and used by the NVD to forward packets to and from compute instances within the VCN (e.g., forwarding tables, routing tables, etc.).

[0050] In one embodiment, the creation of VCNs and subnets is handled by a VCN control plane (CP), and the startup of compute instances is handled by the compute control plane. The compute control plane is responsible for allocating physical resources to compute instances and then calling the VCN control plane to create VNICs and connect them to the compute instances. The VCN CP also sends VCN data mappings to the VCN data plane, which is configured to perform packet forwarding and routing functions. In one embodiment, the VCN CP provides a distribution service, which is responsible for providing updates to the VCN data plane. Examples of VCN control planes are also shown in Figures 6, 7, 8, and 9 (see references 616, 716, 816, and 916) and are described below.

[0051] A customer may create one or more VCNs using resources hosted by CSPI. Compute instances deployed in a customer's VCN may communicate with various endpoints. These endpoints may include endpoints hosted by CSPI and endpoints outside of CSPI.

[0052] Various different architectures for implementing cloud-based services using CSPI are shown in Figures 1, 2, 3, 4, 5, and 18-22, and are described below. Figure 1 is a high-level diagram of a distributed environment 100 showing an overlay VCN or customer VCN hosted by CSPI according to one embodiment. The distributed environment shown in Figure 1 includes multiple components within the overlay network. The distributed environment 100 shown in Figure 1 is merely an example and is not intended to unduly limit the scope of the claimed embodiments. Many variations, alternatives, and modifications are possible. For example, in some implementations, the distributed environment shown in Figure 1 may include more or fewer systems or components than those shown in Figure 1, may combine two or more subsystems, or may include different configurations or arrangements of systems.

[0053] As shown in the example in Figure 1, the distributed environment 100 includes a CSPI 101 that provides services and resources that customers can subscribe to and use to build their own virtual cloud network (VCN). In one embodiment, the CSPI 101 provides IaaS services to customers who are subscribing. The data centers within the CSPI 101 may be organized into one or more regions. One exemplary region, "Region US" 102, is shown in Figure 1. The customer has configured an Oracle International Corporation customer VCN with respect to Region 102. The customer may deploy various compute instances to the VCN 104, which may include virtual machines or bare metal instances. Examples of instances include applications, databases, load balancers, etc.

[0054] In the embodiment shown in Figure 1, the customer's VCN104 includes two subnets, namely "Subnet 1" and "Subnet 2," each subnet having its own CIDR IP address range. In Figure 1, the overlay IP address range for Subnet 1 is 10.0 / 16, and the address range for Subnet 2 is 10.1 / 16. The VCN virtual router 105 represents the logical gateway of the VCN, enabling communication between subnets of VCN104 and with other endpoints outside the VCN. The VCN VR105 is configured to route traffic between VNICs within VCN104 and gateways associated with VCN104. The VCN VR105 provides a port for each subnet of VCN104. For example, the VR105 may provide a port with the IP address 10.0.0.1 for Subnet 1 and a port with the IP address 10.1.0.1 for Subnet 2.

[0055] Multiple compute instances may be deployed in each subnet, and compute instances can be virtual machine instances and / or bare metal instances. Compute instances within a subnet may be hosted by one or more host machines within CSPI101. Compute instances join the subnet via the VNIC associated with the compute instance. For example, as shown in Figure 1, compute instance C1 becomes part of subnet 1 via the VNIC associated with this compute instance. Similarly, compute instance C2 becomes part of subnet 1 via the VNIC associated with C2. In a similar manner, multiple compute instances, which may be virtual machine instances or bare metal instances, may become part of subnet 1. Each compute instance is assigned a private overlay IP address and MAC address via the VNIC associated with it. For example, in Figure 1, compute instance C1 has the overlay IP address 10.0.0.2 and the MAC address M1, while compute instance C2 has the private overlay IP address 10.0.0.3 and the MAC address M2. Each compute instance in subnet 1, including compute instances C1 and C2, has a default route to VCN VR105 using IP address 10.0.0.1, which is the IP address of the port of VCN VR105 in subnet 1.

[0056] Multiple compute instances, including virtual machine instances and / or bare metal instances, may be deployed in subnet 2. For example, as shown in Figure 1, compute instances D1 and D2 become part of subnet 2 via the VNICs associated with each compute instance. In the embodiment shown in Figure 1, compute instance D1 has the overlay IP address 10.1.0.2 and the MAC address of MM1, while compute instance D2 has the private overlay IP address 10.1.0.3 and the MAC address of MM2. Each compute instance in subnet 2, including compute instances D1 and D2, has a default route to VCN VR105 using the IP address 10.1.0.1, which is the IP address of the port of VCN VR105 in subnet 2.

[0057] VCN A104 may include one or more load balancers. For example, a load balancer may be provided to a subnet and configured to load balance traffic across multiple compute instances on the subnet. A load balancer may be provided to load balance traffic across multiple subnets within the VCN.

[0058] A specific compute instance deployed in VCN104 can communicate with various endpoints. These endpoints may include endpoints hosted by CSPI200 and endpoints outside of CSPI200. Endpoints hosted by CSPI101 may include endpoints on the same subnet as a particular compute instance (e.g., communication between two compute instances in subnet 1), endpoints on different subnets but within the same VCN (e.g., communication between a compute instance in subnet 1 and a compute instance in subnet 2), endpoints in different VCNs within the same region (e.g., communication between a compute instance in subnet 1 and an endpoint in the same VCN in region 106 or 110, or between a compute instance in subnet 1 and an endpoint in service network 110 within the same region), or endpoints in VCNs in different regions (e.g., communication between a compute instance in subnet 1 and an endpoint in the same VCN in region 108). A compute instance in a subnet hosted by CSPI101 may communicate with endpoints not hosted by CSPI101 (i.e., outside of CSPI101). These external endpoints include endpoints within the customer's on-premises network 116, endpoints within networks 118 hosted by other remote clouds, public endpoints 114 accessible over public networks such as the internet, and other endpoints.

[0059] Communication between compute instances on the same subnet is facilitated using VNICs associated with the source and destination compute instances. For example, compute instance C1 in subnet 1 may want to send a packet to compute instance C2, also in subnet 1. For a packet originating from the source compute instance and destined for another compute instance in the same subnet, the packet is first processed by the VNIC associated with the source compute instance. The processing performed by the VNIC associated with the source compute instance may include determining the packet's destination information from the packet header, identifying any policies (e.g., security lists) configured for the VNIC associated with the source compute instance, determining the packet's next hop, performing any packet encapsulation / decapsulation functions as needed, and then forwarding / routing the packet to the next hop to facilitate communication of the packet with its intended destination. If the destination compute instance is in the same subnet as the source compute instance, the VNIC associated with the source compute instance is configured to identify the VNIC associated with the destination compute instance and forward the packet to that VNIC for processing. The VNIC associated with the destination compute instance then executes and forwards the packet to the destination compute instance.

[0060] For packets propagating from a compute instance within a subnet to an endpoint in a different subnet within the same VCN, this communication is facilitated by the VNICs and VCN VRs associated with the source and destination compute instances. For example, if compute instance C1 in subnet 1 in Figure 1 wants to send a packet to compute instance D1 in subnet 2, this packet is first processed by the VNIC associated with compute instance C1. The VNIC associated with compute instance C1 is configured to route the packet to VCN VR105 using the VCN VR's default route or port 10.0.0.1. VCN VR105 is configured to route the packet to subnet 2 using port 10.1.0.1. Next, the packet is received and processed by the VNIC associated with D1, and the VNIC forwards the packet to compute instance D1.

[0061] For packets propagated from compute instances within VCN104 to endpoints outside VCN104, communication is facilitated by the VNIC associated with the source compute instance, VCN VR105, and gateways associated with VCN104. One or more types of gateways may be associated with VCN104. A gateway is an interface between a VCN and another endpoint, which is outside the VCN. A gateway is a Layer 3 / IP layer concept that enables a VCN to communicate with endpoints outside the VCN. Thus, gateways facilitate traffic flow between a VCN and other VCNs or networks. Different types of gateways may be configured for a VCN to facilitate different types of communication with different types of endpoints. Depending on the gateway, communication may traverse a public network (e.g., the Internet) or a private network. Different communication protocols may be used for these communications.

[0062] For example, compute instance C1 may want to communicate with an endpoint outside of VCN104. The packet may first be processed by the VNIC associated with source compute instance C1. This VNIC processing determines that the packet's destination is outside of subnet 1 of C1. The VNIC associated with C1 may forward the packet to VCN VR105 of VCN104. VCN VR105 then processes the packet and, as part of this processing, determines a specific gateway associated with VCN104 as the packet's next hop based on the packet's destination. VCN VR105 then may forward the packet to a specific identified gateway. For example, if the destination is an endpoint within a customer's on-premises network, VCN VR105 may forward the packet to a Dynamic Routing Gateway (DRG) gateway 122 configured for VCN104. The packet may then be forwarded from the gateway to the next hop to facilitate the packet's delivery to its final intended destination.

[0063] Various types of gateways may be configured for a VCN. An example of a gateway that may be configured for a VCN is shown in Figure 1 and described below. Examples of gateways associated with a VCN are also shown in Figures 18, 19, 20, and 21 (for example, gateways referenced by reference numbers 1834, 1836, 1838, 1934, 1936, 1938, 2034, 2036, 2038, 2134, 2136, and 2138) and described below. As shown in the embodiment shown in Figure 1, a dynamic routing gateway (DRG) 122 may be added to or associated with a customer's VCN 104 to provide a route for private network traffic communication between the customer's VCN 104 and another endpoint, which may be the customer's on-premises network 116, a VCN 108 in a different region of CSPI 101, or another remote cloud network 118 not hosted by CSPI 101. The customer's on-premises network 116 may be the customer's network or data center built using the customer's resources. Access to the customer's on-premises network 116 is typically extremely restricted. For a customer who has both the customer's on-premises network 116 and one or more VCNs 104 deployed or hosted in the cloud by CSPI 101, the customer may want the customer's on-premises network 116 and the customer's cloud-based VCNs 104 to be able to communicate with each other. This would allow the customer to build an extended hybrid environment encompassing the customer's VCNs 104 hosted by CSPI 101 and the customer's on-premises network 116. DRG 122 enables this communication. To enable such communication, a communication channel 124 is configured, with one endpoint of this channel located within the customer's on-premises network 116 and the other endpoint located within CSPI 101 and connected to the customer's VCNs 104. The communication channel 124 can be via a public communication network such as the internet or a private communication network.Various communication protocols may be used, such as IPsec VPN technology which uses a public communication network like the Internet, or Oracle's FastConnect technology which uses a private network instead of a public network. A device or equipment within the customer's on-premises network 116 that forms one endpoint of communication channel 124 is called customer premise equipment (CPE), such as CPE126 shown in Figure 1. On the CSPI101 side, the endpoint may be a host machine running DRG122.

[0064] In one embodiment, a Remote Peering Connection (RPC) may be added to the DRG, enabling a customer to peer one VCN with another VCN in a different region. Using such an RPC, a customer's VCN 104 can connect to a VCN 108 in a different region using the DRG 122. The DRG 122 may be used to communicate with other remote cloud networks 118 not hosted by the CSPI 101, such as the Microsoft Azure cloud or the Amazon AWS cloud.

[0065] As shown in Figure 1, an Internet Gateway (IGW) 120 may be configured for a customer's VCN104, enabling compute instances on VCN104 to communicate with a public endpoint 114 accessible via a public network such as the Internet. The IGW 120 is a gateway that connects the VCN to a public network such as the Internet. The IGW 120 enables direct access from public subnets within a VCN, such as VCN104 (resources within public subnets have public overlay IP addresses), to public endpoints 112 on the public network 114, such as the Internet. Connections can be initiated using the IGW 120 from subnets within VCN104 or from the Internet.

[0066] A Network Address Translation (NAT) gateway 128 is configured for the customer's VCN 104 to enable internet access for cloud resources within the customer's VCN that do not have dedicated public overlay IP addresses. The NAT gateway 128 enables such access without directly exposing those resources to incoming internet connections (e.g., L4-L7 connections). This allows private access from private subnets within the VCN, such as private subnet 1 within VCN 104, to public endpoints on the internet. The NAT gateway can only initiate connections from private subnets to the public internet; it cannot initiate connections from the internet to private subnets.

[0067] In one embodiment, a Service Gateway (SGW) 126 may be configured for a customer's VCN 104, providing a route for private network traffic between the VCN 104 and supported service endpoints within a service network 110. In one embodiment, the service network 110 may be provided by a CSP and may provide a variety of services. An example of such a service network is Oracle's Services Network, which provides a variety of services that can be used by a customer. For example, compute instances (e.g., database systems) within a private subnet of a customer's VCN 104 can back up data to a service endpoint (e.g., object storage) without requiring a public IP address or internet access. In one embodiment, a VCN may contain only one SGW, and connections may be initiated only from subnets within the VCN and not from the service network 110. If a VCN is peered with another VCN, resources in the other VCN typically cannot access the SGW. Resources in an on-premises network connected to a VCN using FastConnect or VPN connections may also use the service gateway configured for that VCN.

[0068] In one implementation, SGW126 uses the concept of service-classless inter-domain routing (CIDR) labels, which are strings representing the public IP address range for all regions of the target service or group of services. Customers use service CIDR labels when configuring SGW and associated route rules to control traffic to their services. Optionally, customers can also utilize service CIDR labels when configuring security rules, eliminating the need to adjust those security rules if the public IP addresses of their services change in the future.

[0069] A Local Peering Gateway (LPG) 132 is a gateway that can be added to a customer's VCN 104, enabling VCN 104 to peer with another VCN within the same region. Peering means that VCNs communicate using private IP addresses without traffic traversing a public network such as the internet or routing traffic through the customer's on-premises network 116. In a preferred embodiment, a VCN includes a separate LPG for each peering it establishes. Local peering, or VCN peering, is a common method used to establish network connectivity between different application or infrastructure management functions.

[0070] Service providers, such as service providers within service network 110, may provide access to their services using various access models. According to the public access model, a service may be exposed as a public endpoint, which may be publicly accessible by compute instances in the customer's VCN via a public network such as the internet, and / or privately accessible via SGW126. According to a particular private access model, a service may be made accessible as a private IP endpoint in a private subnet within the customer's VCN. This access is called private endpoint (PE) access and allows service providers to expose their services as instances in the customer's private network. A private endpoint resource represents a service within the customer's VCN. Each PE appears as a VNIC (called a PE-VNIC, which has one or more private IPs) in a subnet selected by the customer within the customer's VCN. Thus, a PE provides a way to present a service in a private subnet of the customer's VCN using a VNIC. Because the endpoint is exposed as a VNIC, all features associated with the VNIC, such as routing rules and security lists, are available to the PE VNIC.

[0071] A service provider can register a service to enable access via a Public Access Point (PE). The provider can associate policies with the service that restrict its visibility to the customer's tenancy. A provider can register multiple services under a single virtual IP address (VIP), especially in the case of multi-tenant services. Multiple such private endpoints (in multiple VCNs) representing the same service may exist.

[0072] Next, compute instances within a private subnet can access the service using the private IP address of the PE VNIC or the DNS name of the service. Compute instances within a customer's VCN can access the service by sending traffic to the private IP address of the PE within the customer's VCN. The Private Access Gateway (PAGW) 130 is a gateway resource that can be connected to the service provider's VCN (e.g., a VCN within service network 110) and acts as the entry / exit point for all traffic to and from the private endpoint in the customer's subnet. The PAGW 130 allows the provider to scale the number of PE connections without utilizing internal IP address resources. The provider only needs to configure one PAGW for any number of services registered within a single VCN. The provider can represent a service as a private endpoint in multiple VCNs for one or more customers. From the customer's perspective, the PE VNIC appears to be connected to the service the customer wishes to interact with, instead of being connected to the customer's instance. Traffic destined for the private endpoint is routed to the service via the PAGW 130. These are called customer-to-service (C2S) connections.

[0073] The PE concept can also be used to extend private access to a service to the customer's on-premises network and data center by allowing traffic to flow through FastConnect / IPsec links and private endpoints within the customer's VCN. Private access to the service can also be extended to the customer's peered VCN by allowing traffic to flow between LPG132 and the PE within the customer's VCN.

[0074] Customers can control routing within their VCN at the subnet level, allowing them to specify which subnets within their VCN, such as VCN104, use which gateways. The VCN's route table is used to determine whether traffic is allowed to exit the VCN through a particular gateway. For example, in a specific example, the route table for a public subnet within a customer's VCN104 may allow non-local traffic to be sent via IGW120. The route table for a private subnet within the same customer's VCN104 may allow traffic destined for CSP services to be sent via SGW126. All remaining traffic may be sent via NAT gateway 128. The route table controls only traffic leaving the VCN.

[0075] The security list associated with a VCN is used to control traffic entering the VCN via a gateway through inbound connections. All resources within a subnet use the same route table and security list. The security list may be used to control specific types of traffic that are allowed to enter and leave instances within the VCN subnet. Security list rules may include inbound and outbound rules. For example, inbound rules may specify allowed source address ranges, while outbound rules may specify allowed destination address ranges. Security rules may specify specific protocols (e.g., TCP, ICMP), specific ports (e.g., 22 for SSH, 3389 for Windows RDP), etc. In some implementations, the instance's operating system may enforce its own firewall rules that match the security list rules. Rules may be stateful (e.g., connections are tracked and responses are automatically allowed without using explicit security list rules for response traffic) or stateless.

[0076] Access from a customer's VCN (i.e., by resources or compute instances deployed on VCN104) can be classified as public access, private access, or dedicated access. Public access refers to an access model in which a public IP address or NAT is used to access a public endpoint. Private access allows a customer's workload within VCN104 with a private IP address (e.g., resources in a private subnet) to access a service without traversing a public network such as the internet. In one embodiment, CSPI101 allows a customer's VCN workload with a private IP address to access a service (or its public service endpoint) using a service gateway. Thus, the service gateway provides a private access model by establishing a virtual link between the customer's VCN and the public endpoint of the service that resides outside the customer's private network.

[0077] Furthermore, CSPI may provide dedicated public access technology using technologies such as FastConnect public peering, which allows a customer's on-premises instances to access one or more services within the customer's VCN using FastConnect connectivity without traversing public networks such as the internet. CSPI may also provide dedicated private access using FastConnect private peering, which allows a customer's on-premises instances with private IP addresses to access workloads in the customer's VCN using FastConnect connectivity. FastConnect is a network connectivity alternative to using the public internet for connecting a customer's on-premises network to CSPI and its services. FastConnect provides an easy, resilient, and economical way to create dedicated private connectivity with higher bandwidth options and a more reliable and consistent network experience compared to internet-based connectivity.

[0078] Figure 1 and the accompanying explanation above illustrate the various virtual components within an exemplary virtual network. As previously mentioned, the virtual network is built on top of an underlying physical network or infrastructure network. Figure 2 shows a simplified architectural diagram of the physical components in the physical network within the underlying CSPI200 for the virtual network, according to one embodiment. As shown in the figure, the CSPI200 provides a distributed environment that includes components and resources (e.g., compute resources, memory resources, and network resources) provided by a Cloud Service Provider (CSP). These components and resources are used to deliver cloud services (e.g., IaaS services) to subscribers, i.e., customers who subscribe to one or more services provided by the CSP. Based on the services subscribed by the customer, a subset of the CSPI200's resources (e.g., compute resources, memory resources, and network resources) is provisioned for the customer. The customer can then use the physical compute resources, memory resources, and network resources provided by the CSPI200 to build their own cloud-based (i.e., CSPI-hosted) customizable private virtual network. As already indicated, these customer networks are called Virtual Cloud Networks (VCNs). Customers can deploy one or more customer resources, such as compute instances, to these customer VCNs. Compute instances can take the form of virtual machines, bare metal instances, etc. CSPI200 provides infrastructure and a suite of complementary cloud services that enable customers to build and run a wide range of applications and services in a highly available hosted environment.

[0079] In the embodiment shown in Figure 2, the physical components of the CSPI200 include one or more physical host machines or physical servers (e.g., 202, 206, 208), network virtualization devices (NVDs) (e.g., 210, 212), top-of-rack (TOR) switches (e.g., 214, 216), and a physical network (e.g., 218), and switches within the physical network 218. The physical host machines or physical servers may host and run various compute instances participating in one or more subnets of the VCN. Compute instances may include virtual machine instances and bare metal instances. For example, the various compute instances shown in Figure 1 may be hosted by the physical host machines shown in Figure 2. Virtual machine compute instances in the VCN may run by one host machine or by several different host machines. The physical host machines may host virtual host machines, container-based hosts or functions, etc. The VNIC and VCN VR shown in Figure 1 may run by the NVDs shown in Figure 2. The gateway shown in Figure 1 may be run by the host machine and / or by the NVD shown in Figure 2.

[0080] A host machine or server may run a hypervisor (also known as a virtual machine monitor or VMM) that creates and activates virtual environments on the host machine. Virtualized environments facilitate cloud-based computing. On a host machine, one or more compute instances may be created, run, and managed by the hypervisor on that host machine. The hypervisor on the host machine enables the sharing of the host machine's physical computing resources (e.g., compute resources, memory resources, and network resources) among the various compute instances running on the host machine.

[0081] For example, as shown in Figure 2, host machines 202 and 208 run hypervisors 260 and 266, respectively. These hypervisors may be implemented using software, firmware, hardware, or a combination thereof. Typically, a hypervisor is a process or software layer that resides on top of the host machine's operating system (OS), which runs on the host machine's hardware processors. A hypervisor provides a virtual environment by enabling the sharing of the host machine's physical computing resources (e.g., processing resources such as processors / cores, memory resources, and network resources) among the various virtual machine computing instances run by the host machine. For example, in Figure 2, hypervisor 260 may reside on top of the host machine 202's OS, enabling the sharing of the host machine 202's computing resources (e.g., processing resources, memory resources, and network resources) among the computing instances (e.g., virtual machines) run by the host machine 202. A virtual machine may have its own operating system (called a guest operating system), which may be the same as or different from the host machine's OS. The operating system of a virtual machine running on a host machine may be the same as or different from the operating system of another virtual machine running on the same host machine. Therefore, a hypervisor allows multiple operating systems to run in parallel with each other while sharing the same computing resources on the host machine. The host machines shown in Figure 2 may have the same or different types of hypervisors.

[0082] Compute instances can be virtual machine instances or bare metal instances. In Figure 2, compute instance 268 on host machine 202 and compute instance 274 on host machine 208 are examples of virtual machine instances. Host machine 206 is an example of a bare metal instance provided to a customer.

[0083] In one example, an entire host machine may be provisioned to a single customer, and all one or more compute instances (either virtual machines or bare metal instances) hosted by that host machine belong to that same customer. In another example, a host machine may be shared among multiple customers (i.e., multiple tenants). In such a multi-tenancy scenario, the host machine may host virtual machine compute instances belonging to different customers. These compute instances may be members of different VCNs of different customers. In one embodiment, bare metal compute instances are hosted by bare metal servers that do not have a hypervisor. When bare metal compute instances are provisioned, a single customer or tenant maintains control of the physical CPU, memory, and network interfaces of the host machine hosting the bare metal instances, and the host machine is not shared with other customers or tenants.

[0084] As mentioned above, each compute instance that is part of a VCN is associated with a VNIC that enables the compute instance to be a member of the VCN's subnet. The VNIC associated with a compute instance facilitates the communication of packets or frames to and from the compute instance. The VNIC is associated with a compute instance when the compute instance is created. In one embodiment, for a compute instance run by a host machine, the VNIC associated with that compute instance is run by an NVD connected to the host machine. For example, in Figure 2, host machine 202 runs a virtual machine compute instance 268 associated with VNIC 276, and VNIC 276 is run by an NVD 210 connected to host machine 202. In another example, a bare metal instance 272 hosted by host machine 206 is associated with VNIC 280, which is run by an NVD 212 connected to host machine 206. In yet another example, VNIC 284 is associated with compute instance 274 run by host machine 208, and VNIC 284 is run by an NVD 212 connected to host machine 208.

[0085] For compute instances hosted by a host machine, the NVDs connected to that host machine also run the VCN VRs corresponding to the VCNs to which those compute instances are members. For example, in the embodiment shown in Figure 2, NVD210 runs VCN VR277 corresponding to the VCN to which compute instance 268 is a member. NVD212 may run one or more VCN VR283 corresponding to the VCNs to which compute instances hosted by host machines 206 and 208 are associated.

[0086] The host machine may include one or more network interface cards (NICs) that enable the host machine to connect to other devices. The NICs on the host machine may provide one or more ports (or interfaces) that enable the host machine to connect communicatively to another device. For example, the host machine may be connected to the NVD using one or more ports (or interfaces) provided on the host machine and the NVD. The host machine may also be connected to other devices, such as another host machine.

[0087] For example, in Figure 2, host machine 202 is connected to NVD210 using a link 220 extending between port 234 provided by NIC 232 of host machine 202 and port 236 of NVD210. Host machine 206 is connected to NVD212 using a link 224 extending between port 246 provided by NIC 244 of host machine 206 and port 248 of NVD212. Host machine 208 is connected to NVD212 using a link 226 extending between port 252 provided by NIC 250 of host machine 208 and port 254 of NVD212.

[0088] Next, the NVDs are connected via communication links to top-of-rack (TOR) switches (also called switch fabrics) connected to the physical network 218. In some embodiments, the links between the host machines and the NVDs, and between the NVDs and the TOR switches, are Ethernet® links. For example, in Figure 2, links 228 and 230 are used to connect NVDs 210 and 212 to TOR switches 214 and 216, respectively. In some embodiments, links 220, 224, 226, 228, and 230 are Ethernet links. The collection of host machines and NVDs connected to the TOR is sometimes referred to as a rack.

[0089] The physical network 218 provides a communication fabric that allows TOR switches to communicate with each other. The physical network 218 can be a multi-layer network. In one implementation, the physical network 218 is a multi-layer Clos network of switches, and TOR switches 214 and 216 represent leaf-level nodes in a multi-layer multi-node physical switching network 218. Various Clos network configurations are possible, including but not limited to 2-layer, 3-layer, 4-layer, 5-layer networks, and generally "n"-layer networks. An example of a Clos network is shown in Figure 5 and described below.

[0090] Various connection configurations are possible between the host machine and the NVD, including one-to-one, many-to-one, and one-to-many configurations. In a one-to-one configuration, each host machine is connected to its own separate NVD. For example, in Figure 2, host machine 202 is connected to NVD210 via host machine 202's NIC232. In a many-to-one configuration, multiple host machines are connected to a single NVD. For example, in Figure 2, host machines 206 and 208 are connected to the same NVD212, respectively, via NIC244 and 250.

[0091] In a one-to-many configuration, one host machine is connected to multiple NVDs. Figure 3 shows an example within CSPI300 where a host machine is connected to multiple NVDs. As shown in Figure 3, the host machine 302 has a network interface card (NIC) 304 with multiple ports 306 and 308. The host machine 300 is connected to a first NVD 310 via port 306 and link 320, and to a second NVD 312 via port 308 and link 322. Ports 306 and 308 may be Ethernet ports, and links 320 and 322 between the host machine 302 and the NVDs 310 and 312 may be Ethernet links. Next, the NVD 310 is connected to a first TOR switch 314, and the NVD 312 is connected to a second TOR switch 316. The links between the NVDs 310 and 312 and the TOR switches 314 and 316 may be Ethernet links. TOR switches 314 and 316 represent Layer 0 switching devices in a multilayer physical network 318.

[0092] The configuration shown in Figure 3 provides two separate physical network paths to the host machine 302 across the physical switch network 318: a first path that crosses TOR switch 314 and passes through NVD 310 to the host machine 302, and a second path that crosses TOR switch 316 and passes through NVD 312 to the host machine 302. These separate paths result in improved availability (referred to as high availability) of the host machine 302. If there is a problem with one of the paths (e.g., one link in the path fails) or a device (e.g., a particular NVD is not functioning), the other path may be used for communication with the host machine 302.

[0093] In the configuration shown in Figure 3, the host machine connects to two different NVDs using two different ports provided by the host machine's NIC. In other embodiments, the host machine may include multiple NICs that enable the host machine to connect to multiple NVDs.

[0094] Referring again to Figure 2, the NVD is a physical device or component that performs one or more network and / or storage virtualization functions. The NVD may be any device having one or more processing units (e.g., a CPU, Network Processing Units (NPUs), FPGA, packet processing pipeline, etc.), memory including a cache, and ports. Various virtualization functions may be performed by software / firmware running on one or more processing units of the NVD.

[0095] NVDs can be implemented in various forms. For example, in one embodiment, an NVD is implemented as an interface card called a smart NIC or intelligent NIC that incorporates an embedded processor. A smart NIC is a separate device from the NIC on the host machine. In Figure 2, NVDs 210 and 212 may be implemented as smart NICs connected to host machine 202, and to host machines 206 and 208, respectively.

[0096] However, a smart NIC is merely one example of an NVD implementation. Various other implementations are possible. For example, in some other implementations, the NVD or one or more functions performed by the NVD may be incorporated into or performed by one or more host machines, one or more TOR switches, and other components of the CSPI200. For example, the NVD may be embodied in a host machine, and the functions performed by the NVD are performed by the host machine. Another example is that the NVD may be part of a TOR switch, or the TOR switch may be configured to perform the functions performed by the NVD, thereby enabling the TOR switch to perform various complex packet translations used in public clouds. A TOR that performs the functions of the NVD is sometimes called a smart TOR. In yet another implementation where virtual machine (VM) instances rather than bare metal (BM) instances are provided to customers, the functions performed by the NVD may be implemented within the hypervisor of the host machine. In some other implementations, some of the functions of the NVD may be offloaded to a centralized service running on a set of host machines.

[0097] In some embodiments, such as when implemented as a smart NIC as shown in Figure 2, the NVD may have multiple physical ports that enable the NVD to connect to one or more host machines and one or more TOR switches. The ports on the NVD may be classified as host-facing ports (also called “south ports”) or network-facing or TOR-facing ports (also called “north ports”). Host-facing ports on the NVD are the ports used to connect the NVD to a host machine. An example of host-facing ports in Figure 2 includes port 236 on the NVD210, and ports 248 and 254 on the NVD212. Network-facing ports on the NVD are the ports used to connect the NVD to a TOR switch. An example of network-facing ports in Figure 2 includes port 256 on the NVD210 and port 258 on the NVD212. As shown in Figure 2, the NVD210 connects to the TOR switch 214 using a link 228 extending from port 256 on the NVD210 to the TOR switch 214. Similarly, the NVD212 is connected to the TOR switch 216 using a link 230 that extends from port 258 of the NVD212 to the TOR switch 216.

[0098] The NVD may receive packets and frames from the host machine via its host-facing port (for example, packets and frames generated by compute instances hosted by the host machine), perform the necessary packet processing, and then forward those packets and frames to the TOR switch via the NVD's network-facing port. The NVD may also receive packets and frames from the TOR switch via its network-facing port, perform the necessary packet processing, and then forward those packets and frames to the host machine via the NVD's host-facing port.

[0099] In one embodiment, there may be multiple ports and associated links between the NVD and the TOR switch. These ports and links may be aggregated to form a link aggregator group (LAG) of multiple ports or links. Link aggregation allows multiple physical links between two endpoints (e.g., between the NVD and the TOR switch) to be treated as a single logical link. All physical links within a given LAG may operate in full-duplex mode at the same speed. LAGs help increase bandwidth and improve the reliability of the connection between the two endpoints. If one of the physical links in the LAG fails, traffic is dynamically and transparently reassigned to one of the other physical links in the LAG. Aggregated physical links provide higher bandwidth than individual links. Multiple ports associated with a LAG are treated as a single logical port. Traffic can be load-balanced across the multiple physical links of the LAG. One or more LAGs may be configured between two endpoints. The two endpoints may be between the NVD and the TOR switch, between a host machine and the NVD, etc.

[0100] NVD implements or performs network virtualization functions. These functions are performed by software / firmware run by NVD. Examples of network virtualization functions include, but are not limited to, packet encapsulation and decapsulation functions, functions for creating VCN networks, functions for enforcing network policies such as VCN security list (firewall) functions, and functions for facilitating the routing and forwarding of packets to and from compute instances within the VCN. In one embodiment, upon receiving a packet, NVD is configured to run a packet processing pipeline to process the packet and determine how the packet should be forwarded or routed. As part of this packet processing pipeline, NVD may run one or more virtual functions associated with the overlay network, such as running a VNIC associated with compute instances within the VCN, running a virtual router (VR) associated with the VCN, encapsulating and decapsulating packets to facilitate forwarding or routing within the virtual network, running a gateway (e.g., a local peering gateway), enforcing security lists and network security groups, network address translation (NAT) functions (e.g., translation from public IP to private IP on a per-host basis), throttling functions, and other functions.

[0101] In one embodiment, the packet processing data path within the NVD may comprise multiple packet pipelines, each consisting of a series of packet translation stages. In one implementation, upon receipt of a packet, it is parsed and classified into a single pipeline. The packet is then processed linearly, one stage at a time, until it is either discarded or transmitted through the NVD's interface. These stages provide basic functional packet processing components (e.g., header validation, bandwidth adjustment, insertion of new Layer 2 headers, L4 firewall implementation, VCN encapsulation / decapsulation, etc.) such that new pipelines can be constructed by assembling existing stages, and new functionality can be added by creating new stages and inserting them into existing pipelines.

[0102] The NVD may perform both control plane and data plane functions corresponding to the VCN's control plane and data plane. Examples of the VCN control plane are also shown in Figures 18, 19, 20, and 21 (see references 1816, 1916, 2016, and 2116) and are described below. Examples of the VCN data plane are also shown in Figures 18, 19, 20, and 21 (see references 1818, 1918, 2018, and 2118) and are described below. Control plane functions include functions used to configure the network that controls how data is forwarded (e.g., setting routes and route tables, configuring VNICs, etc.). In one embodiment, a VCN control plane is provided that centrally calculates the mappings between all overlays and the substrate and exposes those mappings to the NVD and to various virtual network edge devices such as gateways such as DRGs, SGWs, and IGWs. Firewall rules may be exposed using the same mechanism. In one embodiment, the NVD retrieves only the mappings relevant to that NVD. The data plane functionality includes the ability to actually route / forward packets based on configuration settings that use the control plane. The VCN data plane is implemented by encapsulating customer network packets before they traverse the underlying network. The encapsulation / decapsulation function is implemented by the NVD. In one embodiment, the NVD is configured to intercept all network packets entering and leaving the host machine and to perform network virtualization functions.

[0103] As shown above, NVD performs various virtualization functions, including VNICs and VCN VRs. NVD may run VNICs associated with compute instances hosted by one or more host machines connected to a VNIC. For example, as shown in Figure 2, NVD210 runs the functions of VNIC276 associated with compute instance 268 hosted by host machine 202 connected to NVD210. As another example, NVD212 runs VNIC280 associated with bare-metal compute instance 272 hosted by host machine 206 and VNIC284 associated with compute instance 274 hosted by host machine 208. Host machines may host compute instances belonging to different VCNs belonging to different customers, and NVDs connected to host machines may run VNICs corresponding to compute instances (i.e., perform functions associated with VNICs).

[0104] An NVD also runs a VCN virtual router corresponding to the VCN of a compute instance. For example, in the embodiment shown in Figure 2, NVD210 runs a VCN VR277 corresponding to the VCN to which compute instance 268 belongs. NVD212 runs one or more VCN VRs283 corresponding to one or more VCNs to which compute instances hosted by host machines 206 and 208 belong. In one embodiment, the VCN VR corresponding to that VCN is run by all NVDs connected to the host machine hosting at least one compute instance belonging to that VCN. If a host machine hosts compute instances belonging to different VCNs, the NVDs connected to that host machine may run VCN VRs corresponding to those different VCNs.

[0105] In addition to VNICs and VCN VRs, the NVD may run various software (e.g., daemons) and may include one or more hardware components that facilitate various network virtualization functions performed by the NVD. For simplicity, these various components are grouped together as a “packet processing component” as shown in Figure 2. For example, NVD210 has a packet processing component 286, and NVD212 has a packet processing component 288. For example, the NVD’s packet processing component may include a packet processor configured to interact with the NVD’s ports and hardware interfaces to monitor all packets received by the NVD and packets propagated using the NVD, and to store network information. Network information may include, for example, network flow information that identifies various network flows processed by the NVD and per-flow information (e.g., per-flow statistics). In one embodiment, network flow information may be stored per VNIC. In addition to performing per-packet operations, the packet processor may implement stateful NAT and L4 firewall (FW). As another example, the packet processing component may include a replication agent configured to replicate information stored by the NVD to one or more different replication target stores. As yet another example, the packet processing component may include a logging agent configured to perform the logging function of the NVD. The packet processing component may also include software for monitoring the performance and health of the NVD, and optionally software for monitoring the status and health of other components connected to the NVD.

[0106] Figure 1 shows components of an exemplary virtual network or overlay network, including a VCN, subnets within the VCN, compute instances deployed on the subnets, VNICs associated with the compute instances, a VCN VR, and a set of gateways configured for the VCN. The overlay components shown in Figure 1 may be run or hosted by one or more of the physical components shown in Figure 2. For example, compute instances within a VCN may be run or hosted by one or more host machines shown in Figure 2. In the case of compute instances hosted by a host machine, the VNICs associated with those compute instances are typically run by NVDs connected to that host machine (i.e., the VNIC functionality is provided by the NVDs connected to that host machine). The VCN VR functionality of the VCN is run by all NVDs connected to the host machines that host or run compute instances that are part of that VCN. Gateways associated with a VCN may be run by one or more different types of NVDs. For example, one gateway may be run by a smart NIC, while others may be run by one or more host machines or other implementations of NVDs.

[0107] As mentioned above, compute instances within a customer's VCN may communicate with various endpoints, which may be on the same subnet as the source compute instance, on a different subnet within the same VCN as the source compute instance, or outside the source compute instance's VCN. These communications are facilitated using the VNIC associated with the compute instance, the VCN VR, and the gateway associated with the VCN.

[0108] For communication between two compute instances on the same subnet within a VCN, communication is facilitated using VNICs associated with the source and destination compute instances. The source and destination compute instances may be hosted by the same host machine or by different host machines. Packets originating from the source compute instance may be forwarded from the host machine hosting the source compute instance to an NVD connected to that host machine. In the NVD, packets are processed using a packet processing pipeline, which may include running the VNIC associated with the source compute instance. Because the destination endpoint of the packet is within the same subnet, running the VNIC associated with the source compute instance causes the packet to be forwarded to an NVD running the VNIC associated with the destination compute instance, which then processes the packet and forwards it to the destination compute instance. The VNICs associated with source and destination compute instances may run on the same NVD (for example, if both source and destination compute instances are hosted by the same host machine) or on different NVDs (for example, if source and destination compute instances are hosted by different host machines connected to different NVDs). The VNICs may use routing / forwarding tables stored by the NVD to determine the next hop of a packet.

[0109] For packets propagated from a compute instance within a subnet to an endpoint in a different subnet within the same VCN, the packet originating from the source compute instance is propagated from the host machine hosting the source compute instance to the NVD connected to that host machine. The NVD processes the packet using a packet processing pipeline, which can include the execution of one or more VNICs and VRs associated with the VCN. For example, as part of the packet processing pipeline, the NVD executes or invokes (also called executing the VNIC) a function corresponding to the VNIC associated with the source compute instance. The function executed by the VNIC may include examining the VLAN tag on the packet. Because the packet's destination is outside the subnet, the VCN VR function is then invoked and executed by the NVD. The VCN VR then routes the packet to the NVD running the VNIC associated with the destination compute instance. The VNIC associated with the destination compute instance then processes the packet and forwards it to the destination compute instance. The VNICs associated with the source compute instance and the destination compute instance may run on the same NVD (for example, if both the source and destination compute instances are hosted by the same host machine) or on different NVDs (for example, if the source and destination compute instances are hosted by different host machines connected to different NVDs).

[0110] If the destination of a packet is outside the VCN of the source compute instance, the packet originating from the source compute instance is propagated from the host machine hosting the source compute instance to an NVD connected to that host machine. The NVD runs the VNIC associated with the source compute instance. Because the destination endpoint of the packet is outside the VCN, the packet is then processed by the VCN VR of that VCN. The NVD may invoke the VCN VR function, causing the packet to be forwarded to an NVD running the appropriate gateway associated with the VCN. For example, if the destination is an endpoint within the customer's on-premises network, the packet may be forwarded by the VCN VR to an NVD running the DRG gateway configured for the VCN. The VCN VR may run on the same NVD as the NVD running the VNIC associated with the source compute instance, or by a different NVD. The gateway may run by an NVD, which can be a smart NIC, a host machine, or another NVD implementation. The packet is then processed by the gateway and forwarded to the next hop, which facilitates the propagation of the packet to the intended destination endpoint. For example, in the embodiment shown in Figure 2, packets originating from compute instance 268 may be propagated from host machine 202 to NVD210 via link 220 (using NIC 232). On NVD210, VNIC 276 is invoked because it is the VNIC associated with source compute instance 268. VNIC 276 is configured to examine the encapsulated information in the packet, determine the next hop for forwarding the packet to facilitate its delivery to the intended destination endpoint, and then forward the packet to the determined next hop.

[0111] Compute instances deployed on a VCN can communicate with various endpoints. These endpoints may include endpoints hosted by CSPI200 and endpoints outside of CSPI200. Endpoints hosted by CSPI200 may include instances within the same VCN or other VCNs, which may be customer VCNs or VCNs not belonging to a customer. Communication between endpoints hosted by CSPI200 may be performed over the physical network 218. Compute instances may also communicate with endpoints that are not hosted by CSPI200 or are outside of CSPI200. Examples of these endpoints include endpoints within the customer's on-premises network or data center, or public endpoints accessible over a public network such as the Internet. Communication with endpoints outside of CSPI200 may be performed over a public network (e.g., the Internet) (not shown in Figure 2) or a private network (not shown in Figure 2) using various communication protocols.

[0112] The architecture of the CSPI200 shown in Figure 2 is merely an example and is not intended to be limiting. Alternative embodiments are possible, with variations, alternatives, and modifications. For example, in some implementations, the CSPI200 may include more or fewer systems or components than those shown in Figure 2, may combine two or more systems, or may include different configurations or arrangements of systems. The systems, subsystems, and other components shown in Figure 2 may be implemented using hardware, or a combination thereof, in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of each system. The software may be stored on non-temporary storage media (e.g., memory devices).

[0113] Figure 4 shows the connection between a host machine and an NVD to implement I / O virtualization to support multi-tenancy functionality in one embodiment. As shown in Figure 4, host machine 402 runs a hypervisor 404 that provides a virtual environment. Host machine 402 runs two virtual machine instances: VM1 406 belonging to customer / tenant 1 and VM2 408 belonging to customer / tenant 2. Host machine 402 has a physical NIC 410 connected to the NVD 412 via link 414. Each compute instance is connected to a VNIC run by the NVD 412. In the embodiment of Figure 4, VM1 406 is connected to VNIC-VM1 420 and VM2 408 is connected to VNIC-VM2 422.

[0114] As shown in Figure 4, NIC410 has two logical NICs: logical NIC A 416 and logical NIC B 418. Each virtual machine is connected to its own logical NIC and configured to work with that logical NIC. For example, VM1 406 is connected to logical NIC A 416, and VM2 408 is connected to logical NIC B 418. Even though the host machine 402 has only one physical NIC 410 shared by multiple tenants, each tenant's virtual machine is confident that it has its own host machine and NIC due to the logical NICs.

[0115] In one embodiment, each logical NIC is assigned its own VLAN ID. Thus, a specific VLAN ID is assigned to logical NIC A 416 for tenant 1, and another VLAN ID is assigned to logical NIC B 418 for tenant 2. When a packet is transmitted from VM1 406, the tag assigned to tenant 1 is attached to the packet by the hypervisor, and then the packet is transmitted from host machine 402 to NVD412 via link 414. In a similar manner, when a packet is transmitted from VM2 408, the tag assigned to tenant 2 is attached to the packet by the hypervisor, and then the packet is transmitted from host machine 402 to NVD412 via link 414. Thus, the packet 424 transmitted from host machine 402 to NVD412 has an associated tag 426 that identifies a specific tenant and associated VM. In NVD, for packet 424 received from host machine 402, the tag 426 associated with the packet is used to determine whether the packet should be processed by VNIC-VM1 420 or VNIC-VM2 422. The packet is then processed by the corresponding VNIC. The configuration shown in Figure 4 ensures that each tenant's compute instance owns its own host machine and NIC. The configuration shown in Figure 4 enables I / O virtualization to support multi-tenancy functionality.

[0116] Figure 5 shows a simplified block diagram of a physical network 500 according to one embodiment. The embodiment shown in Figure 5 is structured as a Clos network. A Clos network is a specific type of network topology designed to provide connectivity redundancy while maintaining high bimodal bandwidth and maximum resource utilization. A Clos network is a type of non-blocking multi-stage or multi-layer switching network, the number of stages or layers can be two, three, four, five, etc. The embodiment shown in Figure 5 is a three-layer network including layers 1, 2, and 3. A TOR switch 504 represents a layer 0 switch in the Clos network. One or more NVDs are connected to the TOR switch. A layer 0 switch is also called an edge device of the physical network. A layer 0 switch is connected to a layer 1 switch, also called a leaf switch. In the embodiment shown in Figure 5, a set of "n" layer 0 TOR switches is connected to a set of "n" layer 1 switches, together forming a pod. Each layer 0 switch in a pod is interconnected to all layer 1 switches in the pod, but there are no switch connections between pods. In one implementation, two pods are called a block. Each block is serviced by or connected to a set of n Layer 2 switches (sometimes called spine switches). Multiple blocks can exist in a physical network topology. The Layer 2 switches are then connected to n Layer 3 switches (sometimes called superspine switches). Packet communication over the physical network is typically performed using one or more Layer 3 communication protocols. Typically, all layers of the physical network except the TOR layer have n-way redundancy, thus enabling high availability. Policies may be specified for pods and blocks to control the visibility of switches to each other within the physical network, enabling scaling of the physical network.

[0117] A key feature of Clos networks is that the maximum number of hops required to reach a packet from one Layer 0 switch to another (or from one NVD connected to a Layer 0 switch to another NVD connected to a Layer 0 switch) is fixed. For example, in a Layer 3 Clos network, a packet may require a maximum of 7 hops to reach one NVD to another, with the source and target NVDs connected to the leaf layers of the Clos network. Similarly, in a Layer 4 Clos network, a packet may require a maximum of 9 hops to reach one NVD to another, with the source and target NVDs connected to the leaf layers of the Clos network. Therefore, the Clos network architecture maintains consistent latency throughout the network, which is crucial for communication within and between data centers. Clos topologies scale horizontally and are cost-effective. The network's bandwidth / throughput capacity can be easily increased by adding more switches (e.g., more leaf and spine switches) to different layers and by increasing the number of links between switches in adjacent layers.

[0118] In one embodiment, each resource within the CSPI is assigned a unique identifier called a Cloud Identifier (CID). This identifier is included as part of the resource information and can be used to manage the resource, for example, via the console or via an API. An example syntax for a CID is as follows: ocid1.<Resource Type>.<Realm>.[Region][.Future Use].<Unique ID> Here, ocid1: A literal string indicating the CID version. Resource type: The type of resource (e.g., instance, volume, VCN, subnet, user, group, etc.). Realm: The realm in which the resource resides. Examples of realm values ​​include "c1" for a commercial realm, "c2" for a government cloud realm, or "c3" for a federal government cloud realm. Each realm may have its own domain name. Region: The region where the resource resides. This section can be left blank if the region cannot be applied to the resource. Future Use: Reserved for future use. Unique ID: The unique part of the ID. This format may vary depending on the type of resource or service.

[0119] Multi-cloud implementation Figure 6 shows a simplified high-level diagram of a distributed environment 600, according to one embodiment, which includes multiple cloud environments provided by different cloud service providers (CSPs), where each cloud environment includes a specific cloud environment that provides specialized infrastructure enabling one or more cloud services provided by that particular cloud environment to be used by customers of other cloud environments. As shown in Figure 6, various cloud environments (also called “clouds”) may be provided by different cloud service providers (CSPs), and each cloud environment or cloud provides one or more cloud services that one or more customers of that cloud environment can subscribe to. The set of cloud services provided by the cloud environments provided by the CSPs may include, but are not limited to, one or more different types of cloud services, such as SaaS (Software-as-a-Service) services, IaaS (Infrastructure-as-a-Service) services, PaaS (Platform-as-a-Service) services, DBaaS (Database-as-a-Service) services, etc. Examples of cloud environments provided by various CSPs include Oracle Cloud Infrastructure (OCI) provided by Oracle Corporation, Microsoft Azure provided by Microsoft Corporation, Google Cloud (trademark) provided by Google LLC, and Amazon Web Services (AWS (registered trademark)) provided by Amazon Corporation. The cloud services provided by a particular cloud environment may differ from the set of cloud services provided by another cloud environment.

[0120] In a standard cloud environment, a CSP provides Cloud Service Provider Infrastructure (CSPI) used to provide one or more cloud services offered to customers by that cloud environment. The CSPI provided by the CSP may include various types of hardware and software resources, including computing resources, memory resources, network resources, and consoles for accessing cloud services. Customers of a cloud environment provided by a CSP may subscribe to one or more of the cloud services offered by that cloud environment. The CSP may offer customers various subscription models. After a customer subscribes to a cloud service provided by the cloud environment, one or more users may be associated with the subscribed customer, and these users may use the cloud services that the customer has subscribed to. In one implementation, when a customer subscribes to a cloud service provided by a particular cloud environment, a customer account or customer tenancy is created for that customer. Then, one or more users may be associated with the customer tenancy, and these users may then use the services that the customer has subscribed to under the customer tenancy. Information about the services a customer has subscribed to, and the users associated with the customer's tenancy, is typically stored in a cloud environment and associated with the customer's tenancy.

[0121] For example, Figure 6 shows three different cloud environments provided by three different CSPs. These cloud environments include Cloud Environment A (Cloud A) 610 provided by CSP A, Cloud Environment B (Cloud B) 640 provided by CSP B, and Cloud Environment C (Cloud C) 660 provided by CSP C. Cloud A 610 includes infrastructure CSPI_A 612 provided by CSP A, which may be used to provide a set of services "Service A" 614 provided by Cloud A 610. One or more customers (e.g., Customer A1 616-1, Customer A2 616-2) may subscribe to one or more services from Service A 614 provided by Cloud A 610. One or more users 618-1 may be associated with Customer A1 616-1 and can use the services that Customer A1 616-1 has subscribed to within Cloud A 610. Similarly, one or more users 618-2 may be associated with customer A2 616-2 and can use the services that customer A2 616-2 subscribes to within cloud A 610. In various use cases, the services that customer A1 616-1 subscribes to may differ from the services that customer A2 616-2 subscribes to.

[0122] As shown in Figure 6, Cloud B 640 includes infrastructure CSPI_B 642 provided by CSP B, which may be used to provide a set of services "Service B" 644 provided by Cloud B 640. One or more customers (e.g., Customer B1 646-1) may subscribe to one or more services from Service B 644. One or more users 648-1 may be associated with Customer B1 646-1 and can use the services that Customer B1 646-1 has subscribed to within Cloud B 640.

[0123] As shown in Figure 6, Cloud C 660 includes infrastructure CSPI_C 662 provided by CSP C, which may be used to provide a set of services “Service C” 664 provided by Cloud C 660. One or more customers (e.g., Customer C1 666-1) may subscribe to one or more services from Service C 664. One or more users 668-1 may be associated with Customer C1 666-1 and can use the services subscribed to by Customer C1 666-1 within Cloud C 660. It should be noted that Services A 614, Services B 644, and Services C 664 may be different from each other.

[0124] In existing cloud implementations, each cloud provides a closed ecosystem to its subscribers and associated users. As a result, customers and associated users of a cloud environment are limited to using services provided by the cloud to which they subscribe. For example, customer B1 646-1 and its user 648-1 are limited to using service B 644 provided by cloud B 640 and cannot use their account in cloud B 640 to access services from different cloud environments, such as services from service A 614 provided by cloud A 610 or service C 664 provided by cloud C 660. The teachings described herein overcome this limitation. Various techniques are described that enable the creation of links between two cloud environments, as described in this disclosure, thereby enabling services provided by a first cloud environment provided by a first CSP to be used by customers (and associated users) of a second different cloud environment provided by a second different CSP, using the customer's account in the second cloud environment.

[0125] For example, in the embodiment shown in Figure 6, the infrastructure CSPI_A 612 provided by CSP A includes, in addition to other infrastructure 620, a special infrastructure 622 (referred to as multi-cloud enabling infrastructure 622 or MEI (multi-cloud enabling infrastructure) 622 or multi-cloud infrastructure 622) that enables one or more services 614 provided by Cloud A to be used by customers and associated users of other clouds such as Clouds B 640 and C 660, using the customer accounts in those other clouds. In one implementation, customers of Clouds B and C do not need to open separate accounts using Cloud A in order to use one or more of the services 614 provided by Cloud A 610. Customer B1 646-1 and associated user 648-1 of Cloud B 640 can use one or more services 614 provided by Cloud A 610 using the customer account or tenancy in Cloud B 640. As another example, customer C1 666-1 and associated user 668-1 of cloud C 660 can use one or more services 614 provided by cloud A 610 using the customer's account or tenancy within cloud C 660.

[0126] In one implementation, MEI622 enables the creation of links between Cloud A 610 and other clouds, which can be used by customers and associated users of the other clouds to access and utilize services provided by Cloud A 610. These are symbolically shown in Figure 6 as link 670 created between Cloud A 610 and Cloud B 640, and link 672 created between Cloud A 610 and Cloud C 660. Through link 670, customers of Cloud B 640 can access or use one or more services 614 provided by Cloud A 610. Similarly, through link 672, customers of Cloud C 660 can access or use one or more services 614 provided by Cloud A 610.

[0127] There are various ways in which MEI612 can be implemented. In one embodiment, MEI612 may include components that enable the establishment of links with different clouds. For example, in Figure 6, MEI622 includes infrastructure component 624, which is responsible for enabling a link 670 with cloud B 640, and infrastructure component 626, which is responsible for enabling a link 672 with cloud C 660. In a similar manner, MEI622 may include other components that enable and facilitate links with other clouds. In some implementations, the components of MEI622 may facilitate links with multiple different clouds.

[0128] There are several reasons why a customer of one cloud might want to use, or desire to use, cloud services provided by a different cloud provider. Using Figure 6 as an example, there are several reasons why customer B1 646-1 of Cloud B 640 might want to use cloud service 614 provided by Cloud A 610. In one use case scenario, this reason might arise because Cloud A 610 provides a cloud service with features not provided by Cloud B 640. In another use case scenario, Clouds A and B can provide similar services, but the service provided by Cloud A 610 might be better than the corresponding service provided by Cloud B 640 (e.g., more features / functions, faster speed, etc.). In yet another use case scenario, customer B1 646-1 of Cloud B 640 might want to use the cloud service provided by Cloud A 610 because it is offered at a lower price than the cloud service provided by Cloud B 640. In some cases, a customer B1 646-1 of Cloud B 640 may want to use cloud services provided by Cloud A 610 due to geographical constraints or other reasons. For example, Cloud A 610 may provide a desired service within a geographical area that is not provided by Cloud B 640, or a particular service may not be provided by Cloud B 640 within the geographical area where the customer desires the service. There are also several other possible use cases regarding reasons why a customer of one cloud might want to use services provided by a different cloud.

[0129] In one embodiment, MEI622 provides and performs the ability to create a link between Cloud A 610 and another cloud, through which users associated with customers of the other cloud can seamlessly access and use services provided by Cloud A 610 from the other cloud itself. For example, MEI622 allows user 648-1 associated with customer B1 646-1 of Cloud 640 to seamlessly access services from service A 614 provided by Cloud A 610. In one implementation, a user interface (e.g., a console) accessible from within Cloud B 640 may be provided, which allows the user to view a list of services 614 provided by Cloud A 610 and select a specific service that user 648-1 wishes to access. In response to the user's selection, MEI622 takes on the role of performing the process of establishing a link 670 between Clouds A and B to enable access to the requested service. The process of setting up link 670 is performed substantially automatically by MEI622. Customer B1 646-1 or associated user 648-1 will not have to worry about making any system, network, or other configuration changes necessary to facilitate the creation, maintenance, and use of the link 670 between clouds A 610 and B 640. Creating the link between the clouds will not impose any burden on users or customers. The link will be created in a fast and efficient manner using the technology described in this disclosure.

[0130] MEI622 can use various technologies to create and use seamless links for users and customers, thus providing an improved user experience. In one implementation, MEI622 makes the user interface (e.g., a graphical user interface GUI) and process flow with which customer B1 and associated user 648-1 interact to request services from cloud A 610 and access the requested services from cloud A 610 substantially similar to the interface and process flow experienced by the customer / user within cloud B 640. In this way, a customer or user who may be familiar with the interface and process flow of cloud B 640 does not need to learn a new interface and process flow to access service 614 from cloud A 610. MEI622 may present different interfaces and process flows to users in different cloud environments. For example, a first set of user interfaces and process flows substantially similar to those of Cloud B may be presented to a user from Cloud B 640, while a different set of user interfaces and process flows substantially similar to those of Cloud C may be presented to a user accessing Cloud A 610 from Cloud C 660. This is done to simplify and, consequently improve, the user experience for accessing service 614 of Cloud A 610 from other clouds.

[0131] As another example, each cloud environment typically includes an identity management system configured to provide security to the cloud environment. The identity management system is configured to protect resources within the cloud environment, including resources provided by the CSP and resources of the cloud's customers that are subscribed to the cloud. The functions performed by the identity management system include, for example, managing identity credentials (e.g., username, password, etc.) associated with the cloud's subscribers and associated users, using identity credentials to regulate user access to cloud resources and services based on permission / access policies configured for the cloud environment, and other functions. Different clouds may use different identity management systems and associated technologies. For example, the identity management system and associated procedures in Cloud A 610 may be completely different from the identity management system and associated procedures in Cloud B 640, and furthermore, the identity management system and associated procedures in Cloud B 640 may be completely different from the identity management system and associated procedures in Cloud C 660. In one implementation, despite these differences in identity management systems and associated procedures between different cloud environments, the technology described herein enables a user associated with a customer in the first cloud to access cloud services provided by a different cloud using the same identity credentials associated with the customer and user within the first cloud.

[0132] For example, in the embodiment shown in Figure 6, Cloud B 640 provided by CSP B may include an identity management system that assigns or allocates identity credentials to enrolled customers and associated users, such as customer B1 646-1 and associated user 648-1. These identity credentials are associated with a tenancy created for customer B1 646-1 within Cloud B 640. In one implementation, MEI 622 provided by Cloud A 610 allows user 648-1 associated with customer B1 646-1 in Cloud B to access services from service A 614 in Cloud A 610 using user 648-1 in Cloud B 640 and the identity credentials associated with customer B1 646-1. This significantly improves the user experience for user 648-1 because user 648-1 does not need to create new identity credentials specific to Cloud A 610 simply for the purpose of accessing service 614 in Cloud A 610. MEI622 facilitates such access.

[0133] For example, customer B1 of cloud B 640 may choose to use a service such as DBaaS (Database-as-a-Service) from the set of services 614 provided by cloud A 610. In response to such a choice, MEI 622 causes a link 670 to be automatically created between cloud A 610 and cloud B 640 so that user 648-1 associated with customer B1 646-1 can use the DBaaS service provided by cloud A 610. The automatic setup of link 670 is facilitated by MEI 622. After link 670 is set up, user 648-1 can use the DBaaS service in cloud A 610 via cloud B 640. As part of using this service, user 648-1 can send a request to cloud A 610 via cloud B 640 to create a database resource. Accordingly, CSPI_A 612 may create the requested database in cloud A 610. In one implementation, the created database may be provisioned within a virtual network (e.g., a virtual cloud network or VCN) created for customer B1 within cloud A 610 and accessible by user 648-1 via cloud B 640. User 648-1 may then send requests to cloud A 610 from cloud B 640 to use the provisioned database. These requests may include, for example, requests to write data to the database, update data stored in the database, delete data in the database, delete the database, or create additional databases. In some use cases, these requests may originate from user 648-1 via cloud B 640, or from a service 644 provided by cloud B 640. In this way, the MEI 622 provided by cloud A 610 allows users associated with customers in different clouds provided by different CSPs to seamlessly access the services provided by cloud A 610.

[0134] The distributed environment 600 shown in Figure 6 is merely an example and is not intended to unduly limit the scope of the claimed embodiments. Many variations, alternatives, and modifications are possible. For example, in an alternative embodiment, the distributed environment 600 may include more or fewer cloud environments. The cloud environments may include more or fewer systems and components, or have different configurations or arrangements of systems and components. The systems and components shown in Figure 6 may be implemented using hardware, or a combination thereof, in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of each system. The software may be stored on non-temporary storage media (e.g., memory devices).

[0135] Multi-Cloud Control Plane (MCCP) Figure 7 shows a high-level architecture of a multi-cloud control plane (MCCP) according to one embodiment. As shown in Figure 7, the high-level architecture 700 includes a first cloud environment provided by a first cloud service provider (e.g., OCI) 720, and a second cloud environment provided by a second cloud service provider 710 (i.e., Azure). The first cloud environment 720 includes a first cloud infrastructure 720A that provides multiple services to users of the first cloud environment 720. Furthermore, the first cloud environment 720 includes a multi-cloud infrastructure 720B that provisions the ability to deliver services of the first cloud infrastructure 720A (e.g., Oracle Cloud Infrastructure (OCI)) to users of other cloud environments (e.g., a second cloud infrastructure 710A). The multi-cloud infrastructure 720B enables users to access services on other clouds (e.g., Oracle PaaS services) with a user experience as close as possible to the user experience of their native cloud environment, while providing simple integration between cloud environments.

[0136] The second cloud infrastructure 710A includes a second cloud portal 711, Active Directory 712, a resource manager 713, customer subscriptions 715, and a subscription 717 for the first cloud provider (i.e., a subscription for the multi-cloud infrastructure 720B within the second cloud environment). The second cloud portal 711 is a centralized access point from which customers of the second environment 710 can log in and manage cloud deployments and instances. Note that the second cloud portal may provide options for both monitoring and operational services provided by the second cloud infrastructure. Active Directory 712 is a service provided by the second cloud infrastructure 710A that provides administrators with the ability to manage end-user identities and access rights. Its services may include core directory, access management, and identity protection. The resource manager 713 is a management layer provided to users for performing operations (e.g., create, update, delete, etc.) on resources deployed in the customer subscription 715, i.e., deployment and management services of the second cloud infrastructure. Note that a customer subscription is sometimes referred to as a virtual network (VNET) on which the customer's applications are deployed and run. A subscription to the first cloud provider 717 within the second cloud infrastructure 710A includes express routes as well as hub and spoke VNETs that provision network connectivity established with the second cloud infrastructure 710A (e.g., from on-premises locations, from external cloud environments). Note that such connectivity does not need to be routed over the public internet, thereby providing users with greater reliability, faster speeds, consistent latency, and higher security.

[0137] The first cloud infrastructure 720A includes a control plane 724, customer tenancy 726, and multi-cloud infrastructure 720B. The control plane 724 of the first cloud infrastructure 720A is the native control plane of the first cloud environment, providing management and orchestration across the cloud environments. The control plane 724 sets the configuration baseline, provisions user and role access, and has applications so that applications can run along with associated services. The multi-cloud infrastructure 720B includes a multi-cloud platform data plane 722 and a multi-cloud platform data plane 728. As previously mentioned, the multi-cloud infrastructure 720B provides a simple integration between cloud environments while provisioning users in other cloud environments (e.g., a second cloud environment 710) to access services provided by the first cloud environment with a user experience as close as possible to the user experience of the user's native cloud environment (e.g., a second cloud environment 710).

[0138] The MCCP architecture in Figure 7 further includes a multi-cloud console 721 (different from the second cloud portal 711) which allows authenticated users within the second cloud infrastructure 710 to perform control plane operations on resources of the first cloud infrastructure 720 exposed via the multi-cloud infrastructure 720B. In some implementations, as shown in Figure 7, all user 705 requests sent to the multi-cloud console 721 are directed to the multi-cloud infrastructure contained within the first cloud environment. It is understood that user 705 can directly send requests (e.g., CRUD requests) regarding resources provided by the first cloud infrastructure to the multi-cloud console 721. The multi-cloud console 721 is configured to process requests having a syntax or format similar to the syntax or format used locally within the second cloud infrastructure. In other words, the multi-cloud console 721 not only has a look and feel similar to the second cloud portal 711 contained within the second cloud infrastructure 710A, but also uses similar terminology. In some implementations, a link (e.g., a web link) directing the user to the multi-cloud console 721 may be provided within the second cloud console 711.

[0139] The multi-cloud infrastructure 720B, included in the first cloud infrastructure 720A, includes multiple microservices such as the authorization module 722A, the proxy module 722B, the platform services module 722C, the cloud link adapter 722D, the adapter pool 722E which includes adapter 1, adapter 2, adapter 3, and adapter 4, and the network link adapter 722F. The adapter pool 722E may include adapters such as the Exadata Cloud Services Adapter, the Autonomous Database Share Adapter, the Autonomous Database Dedicated Adapter, and the Virtual Machine Database Adapter.

[0140] Each adapter in the adapter pool 722E is responsible for exposing a unique set of underlying resources (provided by the first cloud infrastructure 720A) to users in other cloud environments (e.g., the second cloud environment). In particular, each adapter in the adapter pool 722E is mapped to a specific product or resource provided by the first cloud infrastructure 720A. Note that the actual resources are created by the native control plane 724 of the first cloud infrastructure. For example, with respect to DBaaS (database as a service), the DBaaS control plane included in control plane 724 is configured to instantiate exadatabase resources within the customer tenancy 726 of the first cloud environment.

[0141] Incoming requests received by the multi-cloud infrastructure 720B are processed by the authorization module 722A to perform authentication and access control. Each request includes a token associated with the user's account in the second cloud infrastructure. The authorization module extracts this token and validates it in conjunction with Active Directory 712 (i.e., the identity provider system of the second cloud infrastructure 710A). Upon successful validation, the authorization module 722A may check the roles (i.e., sets of permissions) associated with the user. Note that a role may be associated with one or more tasks / operations permitted for that role. According to one embodiment, the authorization module 722A is responsible for authenticating incoming requests to the MCCP and, based on the roles associated with the token, for authorizing the user if they are permitted to perform the requested operation. In some implementations, the authorization module 722A may perform the authentication process described above by leveraging the custom authentication capabilities of the service platform (i.e., SPLAT associated with the first cloud infrastructure). SPLAT receives the incoming request and forwards it to the authorization module 722A, which further parses the incoming request to determine whether to grant permission and returns a success or failure message to SPLAT. If successful, the request is passed by SPLAT to the routing proxy 722B; if unsuccessful, SPLAT returns an error response directly to the caller.

[0142] The proxy module 722B (also known as the routing proxy module), included in the multi-cloud infrastructure 720B, is a component that receives incoming requests from the multi-cloud console 721 and routes these requests to specific adapters in the adapter pool 722E. In one embodiment, the proxy module 722B receives a pre-authenticated request from the service platform of the first cloud infrastructure (i.e., SPLAT) and routes this request to the appropriate adapter based on the routing information contained in the incoming request. In some implementations, the proxy module 722B extracts an identifier corresponding to the service provider (from the incoming request) and routes this request to the appropriate adapter in the adapter pool 722E.

[0143] The CloudLink Adapter 722D, included in the Multi-Cloud Infrastructure 720B, is responsible for handling the lifecycle operations of resources provided by the first cloud infrastructure. The CloudLink Adapter 722D is configured to create a mapping (or relationship created during the signup process) between the Active Directory tenant (and associated subscription) of the second cloud infrastructure and the corresponding tenancy / account of the user in the first cloud infrastructure. In other words, the CloudLink Adapter 722D generates a mapping of the first identifier associated with the user's tenancy in the first cloud infrastructure to the second identifier associated with the user's account in the second cloud infrastructure.

[0144] In some implementations, the CloudLink Adapter 722D performs a conversion between an external cloud identifier (e.g., a second identifier associated with a user's account in the second cloud infrastructure) and a first identifier (associated with the user's tenancy in the first cloud infrastructure), enabling operations passing through the MultiCloud Control Plane 722 to be mapped to appropriate underlying resources in the first cloud infrastructure. In some embodiments, the CloudLink Adapter generates a data object to store the aforementioned mapping information. Furthermore, the CloudLink Adapter 722D also generates a resource principal associated with the data object. The resource principal is assigned one or more permissions based on the token (and its associated role) included in the request. Access to downstream services provided by the first cloud infrastructure is achieved by the user from the second cloud infrastructure based on the resource principal. The CloudLink Adapter 722D may store the data object and its associated resource principal in the root partition of the user's tenancy in the first cloud infrastructure. Alternatively or additionally, the CloudLink adapter 722D may maintain data objects and resource principals locally on the platform service module 722C of the multi-cloud infrastructure for seamless access by other adapters included in the multi-cloud infrastructure.

[0145] The network link module (also called a network adapter) 722F is responsible for creating a network link between a customer subscription 715 (in the second cloud infrastructure) and the corresponding customer tenancy / account 726 (in the first cloud infrastructure). In some embodiments, the network link module 722F obtains a token (from the platform service module 722C) and creates (1) a first peering relationship (in the first cloud environment) between the multi-cloud platform data plane 728 and the customer tenancy 726, and (2) a second peering relationship (in the second cloud environment) between the customer subscription 715 included in the second cloud infrastructure and the subscription 717 of the first cloud service provider.

[0146] The network link module 722F is also configured to establish a network connection between the first cloud environment and the second cloud environment; that is, the network link module 722F can be configured to connect the interconnect 719 in a communicative manner. Once a network link is formed between the two cloud environments, it is understood that applications running in a customer subscription (e.g., within a VNET of the second cloud infrastructure) can access resources (e.g., exadatabase resources deployed in the customer's tenancy 726 of the first cloud infrastructure). Furthermore, telemetry capabilities are provided within the tenancy 726, as shown in Figure 7. Such capabilities relate to maintaining logs, metrics, and other performance parameters related to resources deployed in the customer's tenancy and their respective usage. According to some embodiments, the MCCP mirrors logs and metrics associated with different resources in the first cloud environment and publishes the metrics, logs, events, etc., for further processing, for example, to a dashboard (e.g., to an application insight module included in the customer subscription 715 of the second cloud environment).

[0147] In some embodiments, the platform service module 722C included in the multi-cloud infrastructure is configured to store authentication information associated with services of the first cloud infrastructure provided to the second cloud infrastructure. The platform service module 722C provides token / resource principals for various adapters included in a pool of adapters 722E, for example, so that the adapters can communicate with the native control plane 724 of the first cloud infrastructure. In some embodiments, the platform service module 722C exposes APIs that are called by various adapters to perform tasks such as: • Sell a minimally scoped access token (issued by the second cloud infrastructure) to the adapter. For example, network adapter 722F requires an access token to perform the aforementioned network peering operation. • Provides a resource principal that the adapter uses to call downstream services and create resources in the customer's tenancy of the primary cloud infrastructure. • This triggers the replication of observable data (logs, metrics, events) from the first cloud infrastructure to the second cloud infrastructure.

[0148] As mentioned above, the adapter pool includes multiple adapters, each of which is responsible for exposing a set of resources that form the foundation of the first cloud infrastructure to users of the second cloud infrastructure; that is, each adapter is mapped to a specific product or resource provided by the first cloud environment. For example, the exadatabase adapter acts as a proxy for users of the second cloud infrastructure to create and utilize exadatabase resources. An exadatabase is a pre-configured combination of hardware and software that provides the infrastructure for running a database. In some embodiments, an exadatabase comprises a stack of resources including (a) exadata infrastructure (i.e., hardware), (b) a VM cloud cluster, (c) a container database, and (d) a pluggable database. According to some embodiments, the multi-cloud infrastructure provides (users of the second cloud infrastructure) the ability to analyze each of the stacked infrastructure levels. Furthermore, MCCP provides the flexibility for users to simply issue workflow creation commands (via the multi-cloud console 721), and MCCP then performs the automated creation of individual resources at each level of the stack. As shown in Figure 7, the adapter pool 722F includes four different adapters, but it is understood that this does not limit the scope of the MCCP architecture 700 in any way. The MCCP architecture may include other adapters, for example, dedicated adapters intended for use by specific cloud service providers based on the requirements of those cloud service providers.

[0149] Figure 8 shows an exemplary system diagram illustrating the components of a Multi-Cloud Control Plane (MCCP) in one embodiment. The MCCP 800 includes a Service Platform (SPLAT) 810, a Routing Proxy 815, a Cloud Link Adapter 820, a Database Adapter 825, a Network Adapter 830, and an MCCP Platform 835. Once a user successfully completes the sign-up process, they may use the Multi-Cloud Console 805 to access, create, or update resources in their tenancy within the first cloud infrastructure. For illustrative purposes, the following describes a scenario in which a user uses the Multi-Cloud Console to issue a request to create an exadatabase resource.

[0150] In some implementations, the user accesses the multi-cloud console 805 and provides login information, such as user credentials in the second cloud infrastructure. The multi-cloud console 805 provides multiple options, such as creating resources, accessing resources, and updating resources. Such options may be provided to the user in the form of selectable icons (e.g., buttons) within the multi-cloud console 805. When the user makes a selection (e.g., to create a resource), an API call is triggered to the service platform 810. It is understood that the request made to the service platform 810 is not a native call to the first cloud infrastructure. Rather, this call is a REST-type call that includes an authorization header containing a token associated with the user in the second cloud infrastructure.

[0151] The REST call containing the token is further forwarded to the routing proxy module 815, which performs authentication and access control operations. According to some embodiments, the routing proxy module 815 performs the authentication operation by extracting the token contained in the REST call. The routing proxy module 815 verifies the validity of the token by comparing the signature (used to sign the request) with a publicly available signature of the second cloud infrastructure, ensuring that the request originates from a valid customer associated with the second cloud infrastructure. Furthermore, the routing proxy module 815 may also check the role, i.e., the authority associated with the token, for example, whether the role corresponds to an Exadata DB administrator. Based on the role, the routing proxy module 815 may route the request to the appropriate adapter included in the MCCP framework 800.

[0152] In one embodiment, the routing proxy module 815 compares the role (associated with the token) with a pre-configured list of roles that are publicly available and assigned to each adapter (as part of the API specification). For example, if the role associated with the token corresponds to "Exadata DB Administrator," the request may be understood as a request to create an exadatabase, and therefore the request is forwarded to the database adapter 825. Furthermore, in some embodiments, the routing proxy module 815 may analyze information contained in the REST call, such as the provider ID and the type of resource requested, and based on the analyzed information, the routing proxy module 815 may forward the request to the appropriate adapter.

[0153] In some implementations, the request obtained by the routing proxy module 815 does not need to include information identifying the user tenancy in the first cloud infrastructure where the resource should be deployed. Therefore, the routing proxy module 815 communicates with the cloud link adapter 820 to obtain mapping information of the user's account in the second cloud infrastructure to the user tenancy in the first cloud infrastructure. If mapping information exists, the routing proxy module 815 obtains information about the user tenancy in the first cloud infrastructure and passes this information to the database adapter 825. In this way, the database adapter 825 knows the user tenancy in the first cloud infrastructure where the resource should be created / deployed. However, if the cloud link adapter 820 determines that no mapping information exists, the routing proxy module 815 may simply issue an "unauthorized access" message, which is returned to the user in response to the request to create the database resource.

[0154] Note that in some implementations, the CloudLink adapter 820 creates a data object (referred to herein as a CloudLink resource object) to store metadata information that identifies the two linked accounts. For example, the data object stores metadata information that includes a mapping of a first identifier associated with a tenancy (i.e., an account) in the first cloud infrastructure and a second identifier associated with a user's account using a second cloud service provider. Furthermore, the CloudLink adapter 820 also creates a resource principal (referred to herein as a CloudLink resource principal) for resources (e.g., a database) that are desirable to be created / managed by the user in the second cloud infrastructure. The CloudLink adapter 820 may maintain the data object and resource principal within the root partition of the user's tenancy in the first cloud infrastructure. In some embodiments, the CloudLink adapter 820 may maintain the data object and / or resource principal locally within the MCCP platform 835.

[0155] In some embodiments, the database adapter 825 may communicate with (or instruct the network adapter 830) to create a network link between the user's account in the second cloud infrastructure and the user's tenancy in the first cloud infrastructure. For example, the network adapter 830 may obtain a token associated with the user from a native service 845 in the second cloud infrastructure module and create (1) a first peering relationship (in the first cloud environment) between the MCCP data plane and the user's tenancy in the first cloud infrastructure, and (2) a second peering relationship (in the second cloud environment) between the user's account included in the second cloud infrastructure and the subscription of the first cloud provider.

[0156] The network adapter 830 is also configured to establish a network connection between the first cloud infrastructure and the second cloud infrastructure; that is, the network adapter 830 can configure an interconnect (e.g., interconnect 719 in Figure 7) that connects the two cloud environments in a communicative manner. Once a network link is formed between a user's tenancy in the first cloud infrastructure and a user's account in the second cloud infrastructure, it is understood that applications running in the user's subscription / account can access resources, such as an exadatabase deployed in the tenancy of the first cloud infrastructure. Furthermore, the creation of a peering relationship provisions metrics, such as database usage metrics, so that they can be accessed in the second cloud infrastructure, for example, in a dashboard application running in the user's subscription in the second cloud infrastructure.

[0157] In some implementations, the database adapter 825 may obtain a resource principal that is maintained locally within the MCCP platform 835. The database adapter 825 may send a request (including the resource principal) to one or more downstream services 840 included in the first cloud infrastructure and create the resource in the user's tenancy within the first cloud infrastructure. In other words, the downstream services 840 included in the first cloud infrastructure utilize the identity, i.e., the resource principal obtained from the MCCP platform 835 to create / deploy the required resource (e.g., an exadatabase in the user's tenancy within the first cloud infrastructure). When a user issues a request to create an exadatabase, the user may intermittently poll the MCCP 800 to obtain the status of the request. Once the downstream services 840 in the first cloud infrastructure have created the resource in the user's tenancy within the first cloud infrastructure and the network adapter 830 has established a peering relationship, the MCCP 800 may notify the user of the successful completion of the request.

[0158] Multi-cloud network service - network adapter Referring to Figure 7, the architecture of the multi-cloud control plane 700 enables customers of an external cloud environment (provided by an external cloud service provider), such as a second cloud environment 710, to deploy resources (e.g., database resources) provided within the first cloud environment 720, run services, etc., by utilizing the multi-cloud console 721 and multi-cloud infrastructure 720B (included in the first cloud environment). In some implementations, the network connectivity between the first and second cloud environments should be configured and maintained to expose the service offerings of the first cloud environment (e.g., PaaS offerings) to customers in the second cloud environment. As described in detail below, a network adapter (e.g., the network link component 722F in Figure 7) is responsible for handling resources associated with such a network.

[0159] In one embodiment, a multi-cloud network (MCN) service is provided, which is responsible for configuring and maintaining network connectivity between a first cloud environment and other cloud environments, such as a second cloud environment. The MCN seamlessly exposes services, such as PaaS provision, to customers in other cloud environments. It is understood that the MCN service fits into the architecture of the multi-cloud control plane 700 in Figure 7, such as the network adapter component 722F. According to some embodiments, the MCN service plays the following roles: 1. We expose customers to customer-facing API objects called network links, which integrate with multi-cloud platforms and function as an abstraction for inter-cloud virtual network interconnections. 2. Configure and maintain a direct interconnection link (e.g., ExpressRoute, FastConnect, etc.) between the external cloud environment and the first cloud environment, which will be shared among N customers. 3. Launch and monitor a packet processor instance for each customer, enabling end-to-end packet flow between the external cloud environment and the primary cloud environment. 4. Configure resources related to any domain name system (DNS) within the external cloud environment and the first cloud environment to enable seamless name resolution.

[0160] As the name suggests, a network link is an abstraction that interconnects two cloud environments. For example, a network link may interconnect a customer's virtual network in a second cloud environment to a customer's virtual cloud network (VCN) in a first cloud environment. If a customer needs to manually configure a network link, it is understood that the customer is responsible for numerous tasks, such as configuring the interconnection for the second cloud environment (e.g., Express Route), configuring another interconnection for the first cloud environment (e.g., Fast Connect), configuring a dynamic routing gateway, and configuring gateway connections and route tables. Such tasks are by no means trivial and, furthermore, lead to an unsatisfactory customer experience. As described by embodiments of this disclosure, a multi-cloud network (MCN) service is provided that automatically configures network links (and associated network resources) with minimal or no customer interaction. Therefore, by using the MCN service, customers do not need to worry about the internals of the network link components provided by the multi-cloud infrastructure of the first cloud environment, and can simply use the network link as a black box that enables traffic between the customer's virtual networks in different cloud environments. Furthermore, it should be noted that network links established between different cloud environments may need to have extremely low transmission latency. This is due to the fact that resources deployed in the first cloud environment may be resources such as exadatabase resources that process online transaction processing events. Because such events require minimal latency, the Multi-Cloud Network (MCN) service of this disclosure is relied upon to establish high-performance, highly available end-to-end network links with low transmission latency between different cloud environments.

[0161] Figure 9 shows a high-level block diagram of a network link component according to one embodiment. In particular, Figure 9 shows a network link component 907 (also referred to herein as a multi-cloud network infrastructure (MCNI)) established between a first cloud environment 905 and a second cloud environment 910. As shown in Figure 9, the network link component 907 is established between the customer's virtual private cloud (VPC), for example, VPC 922 contained in the second cloud environment 910, and the customer's virtual cloud network (VCN) 912, i.e., an account in the first cloud environment 905. Thus, a resource 913 (e.g., an exadatabase) hosted in the customer's VCN 912 (in the first cloud environment 905) may be accessed / utilized by a virtual machine 923 hosted in the customer's VPC 922 in the second cloud environment 910.

[0162] As shown in Figure 9, on the second cloud environment side, network link component 907 includes VPC 919 (labeled as a spoke VPC) which peers with the customer's VPC 922, which is located in the second cloud environment 910. Any traffic between the customer's VCN 912 and the spoke VPC 919 is routed through this VPC peering. Note that the spoke VPC 919 appears to the customer as a generic "network link VPC". On the first cloud environment side, network link component 907 includes a dynamic routing gateway (DRG) connection named multi-cloud connection 917. In some implementations, the customer may connect their VCN (e.g., customer's VCN 912) to a customer-owned DRG 915 using a VCN connection, and MCNI 907 is linked to this DRG 915 via multi-cloud connection 917. Note that any traffic to or from the customer's VPC 922 (which is included in the second cloud environment 910) is routed through the VCN connection, and then through the multi-cloud connection 917 within the DRG.

[0163] As explained below with reference to Figure 11, a portion of the network link component 907 is included in the first cloud environment 905, while another portion of the network link component 907 is included in the second cloud environment 910. Furthermore, it is understood that one of the purposes of the network link component 907 is to build a multi-tenant tunnel infrastructure to share the interconnection between the first and second cloud environments. Note that the network link component 907 provisions tunnels so that multi-tenant traffic shares a single inter-cloud interconnection, for example by using GENEVE tunnels to encapsulate / decapsulate customer traffic.

[0164] Referring to Figure 10, a detailed architecture 1000 of a network link configuration according to one embodiment is shown. As shown in Figure 10, the network link connects a region of the second cloud environment 1005 to a region of the first cloud environment 1035 in a communicative manner. The region of the second cloud environment 1005 may contain one or more customer virtual private cloud (VPC) networks (also called "customer virtual networks"). For example, the region of the second cloud environment 1005 contains two customer VPCs, namely customer 1's VPC 1001 and customer 2's VPC 1002. Thus, the portion of the second cloud environment 1005 hosting the customer VPCs is represented as customer domain 1050A. Similarly, the region of the first cloud environment 1035 may contain one or more customer VCNs (i.e., customer virtual networks). For example, the region of the first cloud environment 1035 includes the VCNs of two customers, namely VCN 1031 for customer 1 and VCN 1032 for customer 2. Therefore, the portion of the first cloud environment 1035 hosting the customers' VCNs is represented as customer domain 1050C.

[0165] According to some embodiments, a multi-cloud network infrastructure (MCNI) domain 1050B, i.e., a network link, connects pairs of customer virtual networks in a communicative manner. For example, as shown in Figure 10, customer 1's VPC 1001 (located in the region of the second cloud environment 1005) is communicatively connected to customer 1's VCN 1031 (located in the region of the first cloud environment 1035). In a similar manner, customer 2's VPC 1002 (located in the region of the second cloud environment 1005) is communicatively connected to customer 2's VCN 1032 (located in the region of the first cloud environment 1035). Note that the MCNI domain 1050 corresponds to the network link domain and is located between the dotted lines 1060 and 1070. More specifically, the MCNI domain 1050B includes a first portion located in the region of the second cloud environment 1005 and a second portion located in the region of the first cloud environment 1035. It is understood that MCNI domain 1050B is controlled by the cloud service provider of the first cloud environment 1035.

[0166] In some implementations, each pair of customer virtual networks (e.g., Customer 1's VPC 1001 in the region of the second cloud environment 1005 and Customer 1's VCN 1031 in the region of the first cloud environment 1035) is connected in a communicable manner using multiple virtual networks (referred to herein as link-enabled virtual networks) deployed (in the first and second cloud environments) by a Multi-Cloud Network Infrastructure (MCNI) service. For example, the first link-enabled virtual network 1007 (labeled as Spoke 1 VPC) and the second link-enabled virtual network 1023 (labeled as Spoke 1 VPC) are deployed in the second cloud environment 1005 and the first cloud environment 1035, respectively. It is understood that each pair of customer virtual networks (i.e., 1001 and 1031) is associated with the pair of link-enabled virtual networks, i.e., the first link-enabled virtual network 1007 and the second link-enabled virtual network 1023. In a similar manner, customer 2's VPC 1002 is associated with link-enabled virtual network 1008 (labeled as Spoke2VPC) in the second cloud environment, and another link-enabled virtual network 1024 (labeled as Spoke2VCN) deployed in the first cloud environment 1035. Thus, in the architecture of Figure 10, each pair of customer virtual networks is associated with a unique pair of link-enabled virtual networks.

[0167] According to some embodiments, a data plane hub virtual network (VN) 1013 and a control plane hub VN 1012 are deployed in a region of the second cloud environment 1005. The data plane hub virtual network (VN) 1013 is shared among the virtual networks of different customers included in the second cloud environment. In other words, the data plane hub virtual network (VN) 1013 handles the traffic of multiple customer tenancies (i.e., 1001, 1002) included in the region of the second cloud environment 1005. Similarly, a region of the first cloud environment 1035 includes another data plane hub VN 1022 and a control plane hub VN 1025. The data plane hub VN1022 in the first cloud environment 1035 is shared among different customer virtual cloud networks (e.g., 1031 and 1032) included in the first cloud environment; that is, the data plane hub VN1022 handles traffic for multiple customer VCNs included in the region of the first cloud environment 1035. As shown in Figure 10, the data plane hub VN1013 included in the region of the second cloud environment 1005 is communicably connected to the data plane hub VN1022 included in the region of the first cloud environment 1035 by a high-bandwidth network interconnect 1015A (i.e., a private communication channel). The control plane hub VN1025 included in the first cloud environment 1035 is communicably coupled to the control plane hub VN1012 included in the second cloud environment 1035 via a communication link 1015B (e.g., a public communication channel such as the internet).

[0168] According to some embodiments, the first link-enabled virtual network 1007 is associated with a first virtual network interface card (VNIC) that connects the first link-enabled virtual network 1007 to a data plane hub virtual network 1013 in a communicable manner, a second VNIC that connects the first link-enabled virtual network 1007 to a control plane hub virtual network 1012 in a communicable manner, and a third VNIC that connects the first link-enabled virtual network 1007 to a customer's virtual network 1001 in a second cloud environment 1005. The control plane hub VN 1025 included in the first cloud environment 1035 includes one or more distribution service nodes 1025A, each of which is configured to transmit network configuration information to the control plane hub virtual network 1012 included in the second cloud environment 1005. The control plane hub virtual network 1012 can then transfer network configuration information to the first link-enabled virtual network 1007, since the control plane hub virtual network 1012 is coupled to the first link-enabled virtual network 1007 via a VNIC connection (e.g., VNIC 1016). Note that the control plane hub virtual network 1012 can function as a gateway for distributing network configuration information (obtained from one or more distribution service nodes 1025A) to other link-enabled virtual networks (e.g., spoke 2VPC 1008) associated with other customers of the second cloud environment 1005. Furthermore, the control plane hub VN 1025 included in the first cloud environment 1035 includes a service gateway (SGW), which provides an endpoint for customers of the second cloud environment 1005 to access other services 1027 (e.g., telemetry, identity services, etc.) provided by the first cloud environment 1035.

[0169] According to some embodiments, network configuration information includes at least (i) tunnel encapsulation-decapsulation parameters, and (ii) health status information of a second link-enabled virtual network deployed in a first cloud environment. Such information is transmitted to the link-enabled virtual network in the second cloud environment 1005 to enable the link-enabled virtual network to configure an end-to-end network path between a customer's virtual network in the second cloud environment 1005 (e.g., customer VN1001) and another customer's virtual network in the first cloud environment 1035 (e.g., customer VN1031).

[0170] As shown in Figure 10, the first link-enabled virtual network 1007 (i.e., spoke 1 VPC) is peered (i.e., via a VNIC connection) to customer 1's VPC 1001 to receive traffic from a virtual machine running within customer 1's VPC 1001 (e.g., operated by user 1001A). Because the data plane hub VN 1013 is shared among multiple customer VPCs within the region of the second cloud environment 1005, some implementations of this disclosure utilize encapsulation and tunneling frameworks to distinguish traffic from different customers originating from different customer virtual networks within the second cloud environment 1005. The first link-enabled virtual network 1007 includes a pair of network adapters 1007A and 1007B (referred to herein as remote virtual network adaptors (RVNAs)). Each of the network adapters included in the first link-enabled virtual network 1007 is configured to encapsulate traffic received from customer 1's VPC 1001 to generate encapsulated traffic. In a similar manner, the link-enabled virtual network 1008 (corresponding to the customer's virtual network 1002) includes a pair of RVNAs 1008A and 1008B configured to encapsulate traffic received from customer 2's VPC 1002 and generate encapsulated traffic that is sent over the shared data plane hub virtual network 1013.

[0171] In some implementations, a pair of network adapters (e.g., 1007A, 1007B) is used to achieve high availability and is kept in an active-active state. In other words, each adapter included in the pair of network adapters is functional and encapsulates traffic received from customer 1's VPC 1001. In some embodiments, the pair of network adapters may utilize a service provided by a second cloud environment 1005 for the purpose of performing BGP peering. The pair of network adapters may instruct such a service to distribute traffic (originating from the customer's VPC) in a uniform manner by using a mechanism such as equal-cost multi-path routing (ECMP).

[0172] Traffic encapsulated by the network adapter included in the first link-enabled virtual network 1007 is forwarded to the data plane hub VN1013 (for example, to the VNIC 1017 included in the data plane hub VN1013). Thus, the data plane hub VN1013 receives encapsulated traffic from different link-enabled virtual networks (associated with different customer virtual networks in the second cloud environment 1005) and forwards this traffic to the data plane hub VN1022 included in the first cloud environment 1035 via the high-speed dedicated interconnection channel 1015A. The data plane hub VN1022 then forwards the encapsulated traffic to the second link-enabled virtual network 1023 deployed in the first cloud environment 1035 (for example, via the VNIC 1022A included in the data plane hub VN1022).

[0173] The second link-enabled virtual network 1023 includes a pair of virtual network adapters 1023A and 1023B (labeled as local virtual network adaptors (LVNAs)), each of which is configured to decapsulate encapsulated traffic received from VNIC 1022A, which is contained within the data plane hub VN 1022. Furthermore, as shown in Figure 10, the pair of virtual network adapters 1023A / 1023B contained within the second link-enabled virtual network 1023 of the first cloud environment 1035 sends the decapsulated traffic to customer 1's VN 1031 (for example, to resource 1031A deployed in customer 1's VCN 1031) via a dynamic routing gateway (DRG) connection.

[0174] According to some embodiments of this disclosure, in the process of establishing a network link between a customer's virtual network in a second cloud environment 1005 and a customer's virtual network 1031 in a first cloud environment 1035, a route table (RT) is configured for each of the virtual networks deployed in the multi-cloud network infrastructure domain 1050B, in addition to the virtual networks included in customer domains 1050A and 1050C. For example, as shown in Figure 10, referring to the second cloud environment 1005, RT1003 is configured for customer 1's VPC 1001, RT1004 is configured for customer 2's VPC 1002, RT1009 is configured for the first link-enabling virtual network 1007, RT1014A is configured for the control plane hub VN 1012, and RT1014B is configured for the data plane hub VN 1013.

[0175] Similarly, referring to the first cloud environment 1035 in Figure 10, note that RT1026 is configured with respect to the control plane hub VN1025, RT1022B is configured with respect to the data plane hub VN1022, RT1023B is configured with respect to the second link-enabled virtual network 1023, and RT1031B is configured with respect to customer 1's VN1031. The entries within RT are understood to correspond to the next-hop address in sending data (i.e., packets) from the second cloud environment 1005 to the first cloud environment 1035, or vice versa. Furthermore, Figure 10 also shows routing information (e.g., RT1010 and RT1034) for the tunnel formed between the virtual network adapters contained in the first link-enabled virtual network 1007 and the virtual network adapters contained in the second link-enabled virtual network 1023. For example, RT1010 indicates that the tunnel formed from virtual network adapter 1007A (included in the first link-enabled virtual network 1007) to virtual network adapter 1023A (included in the second link-enabled virtual network 1023) is designated as the first peer tunnel between address 100.70.128.2 (i.e., the address of RVNA1 1007A) and address 100.65.128.97 (i.e., the address of LVNA1 1023A).

[0176] In this way, an end-to-end network link is established between Customer 1's VPC 1001 in the region of the second cloud environment 1005 and Customer 1's VN 1031 in the region of the first cloud environment 1035, via the first link-enabled virtual network 1003, the second link-enabled virtual network 1023, the data plane hub VN 1013 (included in the second cloud environment 1005), and the data plane hub VN 1022 (in the first cloud environment). It is understood that a network link can be established between Customer 2's VPC 1002 (in the region of the second cloud environment 1005) and Customer 2's VN 1032 (in the region of the first cloud environment 1035) in a similar manner to the method described above for establishing a network link between Customer 1's VPC and Customer 1's VN. Furthermore, while customer virtual networks within the first and second cloud environments may share an IP address space, it should be noted that, to prevent traffic collisions, each of the first and second link-enabled virtual networks, as well as the hub virtual network, contained within the first and second cloud environments will be assigned its own unique classless inter-domain routing IP address space.

[0177] As explained above with respect to Figure 10, the architecture of the network link created between the first and second cloud environments relates to the location and placement of packet processors within the link-enabled virtual network created to provision the network link. It is understood that the network link created by the embodiments of this disclosure will support high-throughput, latency-sensitive applications such as exadatabase applications.

[0178] Figure 11 shows an exemplary flowchart illustrating the process of establishing a network link according to one embodiment. The process shown in Figure 11 may be implemented in software (e.g., code, instructions, programs), hardware, or a combination thereof, executed by one or more processing units (e.g., processors, cores) of each system. The software may be stored in a non-temporary storage medium (e.g., a memory device). The methods presented in Figure 11 and described below are intended to be exemplary and non-limiting. Figure 11 shows various processing steps occurring in a particular sequence or order, but this is not intended to be limiting. In some alternative embodiments, these steps may be performed in some different order, or some steps may be performed in parallel.

[0179] Process 1100 is initiated in step 1105, in which the multi-cloud infrastructure contained in the first cloud environment receives a request to create a network link between the first customer's virtual network in the first cloud environment and the second customer's virtual network in the second cloud environment. The first cloud environment (e.g., OCI) is provided by the first cloud service provider and is understood to be different from the second cloud environment (e.g., GCP) provided by the second cloud service provider.

[0180] Next, the process moves to step 1110, in which the first link-enabled virtual network is deployed to the second cloud environment. The first link-enabled virtual network is communicatively coupled to the first data plane hub virtual network, the first control plane hub virtual network, and the second customer's virtual network. Note that the first data plane hub virtual network and the first control plane hub virtual network are also deployed to the second cloud environment. Note that in order to establish such connections, the first link-enabled virtual network has three VNICs: a first VNIC that communicatively couples the first link-enabled virtual network to the first data plane hub virtual network, a second VNIC that communicatively couples the first link-enabled virtual network to the first control plane hub virtual network, and a third VNIC that communicatively couples the first link-enabled virtual network to the second customer's virtual network.

[0181] Next, the process moves to step 1115, in which the first link-enabling virtual network in the second cloud environment receives network configuration information from the first control plane hub virtual network in the second cloud environment. As previously mentioned, such network configuration information may be generated (and distributed) by one or more distribution service nodes (e.g., node 1025A in Figure 10) included in the control plane hub virtual network in the first cloud environment and transmitted to the first control plane hub virtual network in the second cloud environment.

[0182] Furthermore, in step 1120, traffic is enabled to be transmitted from the second customer's virtual network in the second cloud environment to the first customer's virtual network in the first cloud environment via the first link-enabling virtual network, and the first link-enabling virtual network sends the traffic to the first cloud environment via the first data plane hub virtual network based on the network configuration information.

[0183] Hub and spoke network topology Various cloud service providers may offer hub-and-spoke network architectures that include a central hub network that is communicatively coupled to multiple spoke networks. Such hub-and-spoke architectures are commonly used in transmission routing, where the central hub network provides routes to numerous spoke networks for traffic to pass through. The hub-and-spoke topology is a powerful architecture for building solutions when many isolated spoke networks need to pass network traffic through the hub network. The hub acts as a central gateway network, providing routing (usually dynamic routes via BGP) to all connected spoke networks. In virtual networking scenarios, i.e., cloud environments, such a topology allows all spoke networks to communicate with some network outside the cloud. In such scenarios, all spoke networks always have the same route (i.e., through the hub), and routing changes at the hub are applied to all spoke networks simultaneously.

[0184] However, a limitation in implementing such hub-and-spoke networks in a cloud environment is a limitation of scalability. For example, some cloud service providers offer a solution called "peering," in which spoke networks can be configured to peer with a central hub network. Such peering solutions limit the number of spoke networks that can be communicatively coupled with the hub network to a maximum of several dozen spoke networks (e.g., 50 spoke networks). Such limitations usually arise from the provision of specific cloud service providers. Below, we provide our own solutions to address the scalability issues in deploying hub-and-spoke network architectures in a cloud environment.

[0185] In a cloud environment, a hub virtual network is understood to function as a central relay network to other destinations. These destinations may be other networks within the cloud or on-premises locations. The hub network can dynamically learn routes to such networks and / or endpoints, for example via the BGP protocol, and can instantiate security rules in place to enable such connections. Spoke virtual networks, on the other hand, contain networks required to perform some local routing within the network.

[0186] Referring to Figure 12, a schematic diagram of a compute virtual machine (VM) containing multiple VNICs according to one embodiment is shown. For example, Figure 12 shows compute VM 1200 containing two VNICs 1205 and 1210. Compute VM 1200 is created with the first VNIC (e.g., VNIC 1205) associated with the customer's virtual cloud network (VCN) 1215 and the second VNIC (e.g., VNIC 1210) associated with a hub network 1220 (also referred to as simply the hub herein). Compute VM 1200 allows workloads to connect to the customer's VCN 1215 and hub 1220. Note that the customer's VCN and hub network are not connected at all within the virtual network layer, and the connection between the two exists only within the compute VM. Furthermore, note that even though compute VM 1200 contains multiple VNICs (e.g., VNICs 1205 and 1210), the multiple VNICs share the same network namespace. Furthermore, it should be noted that even the basic architecture of the compute virtual machine 1200 shown in Figure 12 may not be permitted in some cloud environments (e.g., Azure) due to restrictions imposed by cloud service providers. For example, some cloud service providers prohibit virtual machines from having multiple VNICs if each VNIC is associated with a different network.

[0187] Figure 13 shows a schematic diagram of a compute virtual machine including multiple network namespaces according to one embodiment of the present disclosure. A network namespace can be defined as a logical copy of the host system's network stack. Network namespaces are useful for configuring containers or virtual environments. Network namespaces isolate system resources associated with a network. In other words, independent network stacks can be created, in which case each network namespace has its own interface, IPv4 and IPv6 protocol stacks, routing tables, address resolution protocol (ARP) tables, firewall rules, etc.

[0188] Referring to Figure 13, the compute virtual machine 1300 includes an application layer 1301 and is configured using multiple network namespaces, e.g., a first network namespace (shown as the customer's VCN namespace 1305) and a second network namespace (shown as the hub network namespace 1310). Note that the customer's VCN network and the hub network are isolated from each other and each has its own interface, routing table, ARP entries, next-hop destinations, etc. Furthermore, the first VNIC (e.g., VNIC 1315) is associated with the first network namespace, i.e., the customer's VCN network namespace 1305, enabling communication between the customer VCN and the compute virtual machine 1300. The compute virtual machine 1300 further includes a second VNIC (e.g., VNIC 1320) associated with the second network namespace, i.e., the hub network namespace 1310, enabling communication between the compute virtual machine 1300 and the hub. Note that the first network namespace differs from the second network namespace, and each network namespace may be configured with unique configuration information (i.e., IPv4 and IPv6 protocol stacks, routing tables, Address Resolution Protocol (ARP) tables, firewall rules, DNS resolvers, etc.). In other words, the first network namespace is programmed using the first configuration information, and the second network namespace is programmed using second configuration information that differs from the first configuration information. Furthermore, it is understood that the compute virtual machine 1300 is not at all limited to containing only two network namespaces and two corresponding VNICs. Rather, the compute virtual machine 1300 may contain more network namespaces. For example, the compute virtual machine 1300 may include a third VNIC associated with a third network namespace that enables communication between the compute virtual machine 1300 and another virtual network (e.g., a control plane hub network).

[0189] Figure 14 shows a spoke and hub network architecture deployed in a cloud environment according to one embodiment. Figure 14 shows a first cloud environment 1405 containing the deployed hub and spoke topology architecture, and a second cloud environment 1410 containing one or more resources that are desired to be used by a customer of the first cloud environment 1405. As shown in Figure 14, the first cloud environment 1405 includes a plurality of spoke networks 1430, 1432, 1434, and a hub network (also referred to herein as a hub) 1450. Each spoke network includes a compute virtual machine, for example, spoke network 1 1430 includes compute VM 1420, spoke network 2 1432 includes compute VM 1422, and spoke network 1434 includes compute VM 1424. Each of compute VMs 1420, 1422, and 1424 is configured in a manner similar to the compute VMs described earlier with reference to Figure 13.

[0190] In detail, considering compute VM 1, 1420 (also called the first compute VM), note that such compute VMs are provided within the first cloud environment 1405. The first compute VM 1420 is connected to the first customer's virtual cloud network (VCN) 1442 and hub 1450 within the first cloud environment. The first compute VM 1420 is associated with a first set of virtual network interface cards (VNICs). In detail, the set of VNICs includes (i) a first VNIC (e.g., VNIC 1421) associated with a first network namespace (i.e., the network namespace of customer 1's VCN 1442) that enables communication between customer 1's VCN 1442 and the first compute virtual machine, and (ii) a second VNIC (e.g., VNIC 1423) associated with a second network namespace (i.e., the network namespace of hub 1450) that enables communication between the first compute virtual machine and the hub. As mentioned earlier, the first network namespace is different from the second network namespace.

[0191] Similarly, as shown in Figure 14, in the first cloud environment 1405, a second compute virtual machine 1422 connected to the hub 1450 is provided to the second customer's VCN 1444 within the first cloud environment. The second compute virtual machine 1422 is associated with a second set of VNICs. In detail, the second set of VNICs includes (i) a first VNIC (e.g., VNIC 1425) associated with a third network namespace (i.e., the network namespace of customer 2's VCN 1444) that enables communication between the second customer's VCN 1444 and the second compute virtual machine 1422, and (ii) a second VNIC (e.g., VNIC 1427) associated with a second network namespace (i.e., the network namespace of the hub 1450) that enables communication between the second compute virtual machine 1422 and the hub 1450. Note that the third network namespace is distinct from the second network namespace. Furthermore, the first network namespace (associated with customer 1's VCN1442 for compute VM1 1420) is understood to be different from the third network namespace (associated with customer 2's VCN1444 for compute VM2 1422). The second network namespaces of compute VM1 and 2, i.e., the network namespaces associated with the hub, may be the same.

[0192] Furthermore, compute VM N1424 is configured in a manner similar to that described above with reference to compute VMs 1420 and 1422. More specifically, compute VM N1424 includes a third set of VNICs, which are (i) a first VNIC (e.g., VNIC 1429) associated with a fourth network namespace (i.e., the network namespace of customer N's ​​VCN 1446) and enabling communication between customer VCN 1444 and compute virtual machine 1424, and (ii) a second VNIC (e.g., VNIC 1431) associated with a second network namespace (i.e., the network namespace of hub 1450) and enabling communication between compute virtual machine 1424 and hub 1450. Note that the fourth network namespace is distinct from the first, second, and third network namespaces.

[0193] In this way, all customer VCN networks, namely 1442, 1444, and 1446, are isolated from one another while sharing the same route (i.e., via hub 1450) to the on-premises or cloud network 1460 (located in the second cloud environment 1410). Thus, customers of the first cloud environment 1405 associated with different customer VCNs (i.e., 1442, 1444, and 1446) can access resources provided by the second cloud environment 1410 in a seamless manner, for example, via hub 1450. Each compute VM shown in Figure 14 is understood to contain two different network namespaces. However, this is understood not to limit the scope of this disclosure in any way. Rather, each VM may contain more network namespaces, and each compute VM may have a different number of network namespaces compared to the number of network namespaces contained in another compute VM. Thus, it is understood that the hub-and-spoke topology architecture described above with reference to Figures 13 and 14 allows the number of spoke nodes to be scaled up to tens of thousands of spoke nodes.

[0194] Referring to Figure 15, an exemplary flowchart is shown illustrating the process performed to enable communication in the spoke and hub architecture of Figure 14 according to one embodiment. The process shown in Figure 15 may be implemented in software (e.g., code, instructions, programs), hardware, or a combination thereof, performed by one or more processing units (e.g., processors, cores) of each system. The software may be stored in a non-temporary storage medium (e.g., a memory device). The methods presented in Figure 15 and described below are intended to be exemplary and non-limiting. Figure 15 shows various processing steps occurring in a particular sequence or order, but this is not intended to be limiting. In some alternative embodiments, these steps may be performed in some different order, or some steps may be performed in parallel.

[0195] The process starts at step 1505, in which step 1505, a first compute virtual machine is provided within the first cloud environment. The first compute virtual machine is connected to the first customer's virtual cloud network (VCN) and the hub within the first cloud environment. The first compute virtual machine is associated with a first set of virtual network interface cards (VNICs), which include (i) a first VNIC associated with a first network namespace, enabling communication between the first customer's VCN and the first compute virtual machine, and (ii) a second VNIC associated with a second network namespace, enabling communication between the first compute virtual machine and the hub. Note that the first network namespace is distinct from the second network namespace.

[0196] The process then moves to step 1510, in which a second compute virtual machine is provided within the first cloud environment. The second compute virtual machine is connected to the second customer's VCN and the hub within the first cloud environment. The second compute virtual machine is associated with a second set of VNICs, which include (i) a first VNIC associated with a third network namespace that enables communication between the second customer's VCN and the second compute virtual machine, and (ii) a second VNIC associated with a second network namespace that enables communication between the second compute virtual machine and the hub.

[0197] Furthermore, in step 1515, the process enables communication between the first customer's VCN in the first cloud environment and the second cloud environment using the first compute virtual machine and hub, and in step 1520, the process enables communication between the second customer's VCN in the first cloud environment and the second cloud environment using the second compute virtual machine and hub.

[0198] Examples of cloud infrastructure As mentioned earlier, IaaS (Infrastructure as a Service) is a specific type of cloud computing. IaaS can be configured to provide virtualized computing resources over a public network (e.g., the internet). In the IaaS model, a cloud computing provider can host infrastructure components (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer)). In some cases, the IaaS provider may also provide various services (e.g., billing, monitoring, logging, security, load balancing, and clustering) as they arise in conjunction with those infrastructure components. Therefore, since these services can be policy-driven, IaaS users may implement policies to drive load balancing to maintain application availability and performance.

[0199] In some cases, IaaS customers may access resources and services over a wide area network (WAN), such as the internet, and use the cloud provider's services to install the rest of their application stack. For example, a user might log into an IaaS platform, create virtual machines (VMs), install an operating system (OS) on each VM, deploy middleware such as databases, create storage buckets for workloads and backups, and install enterprise software on the VMs. The customer can then use the provider's services to perform a variety of functions, including balancing network traffic, troubleshooting application issues, monitoring performance, and managing disaster recovery.

[0200] In most cases, the cloud computing model requires the participation of a cloud provider. A cloud provider may, but does not have to, be a third-party service specializing in providing IaaS (e.g., offering, leasing, or selling). The entity may choose to deploy a private cloud and become its own provider of infrastructure services.

[0201] In some cases, IaaS deployment is the process of connecting a new application or a new version of an application to a prepared application server, etc. This process may include preparing the server (e.g., installing libraries, daemons, etc.). This process is often managed by the cloud provider under the hypervisor layer (e.g., servers, storage, network hardware, and virtualization). Therefore, the customer may be responsible for handling the deployment of the OS, middleware, and / or application (e.g., on self-service virtual machines, etc., which can be spun up on demand).

[0202] In some examples, IaaS provisioning may also refer to acquiring computers or virtual hosts for use and installing the necessary libraries or services on those computers or virtual hosts. In most cases, deployment does not include provisioning, and provisioning may need to be performed first.

[0203] In some cases, IaaS provisioning presents two distinct challenges. First, there is the initial challenge of provisioning an initial set of infrastructure before anything is executed. Second, there is the challenge of developing the existing infrastructure after everything has been provisioned (e.g., adding new services, modifying services, removing services, etc.). In some cases, these two challenges can be addressed by allowing the infrastructure configuration to be defined declaratively. In other words, the infrastructure (e.g., which components are needed and how those components interact) can be defined by one or more configuration files. In this way, the entire infrastructure topology (e.g., which resources depend on which resources and how each of those resources works together) can be described declaratively. In some cases, after the topology is defined, a workflow may be generated to create and / or manage the various components described in the configuration files.

[0204] In some examples, the infrastructure may include many interconnected elements. For example, there may be one or more virtual private clouds (VPCs), also known as core networks (e.g., a potentially on-demand pool of configurable and / or shared computing resources). In some examples, there may be one or more security group rules and one or more virtual machines (VMs) provisioned to define how network security is configured. Other infrastructure elements such as load balancers and databases may be provisioned. The infrastructure can evolve gradually as more infrastructure elements are desired and / or added.

[0205] In some cases, continuous deployment techniques may be employed to enable the deployment of infrastructure code across various virtual computing environments. Furthermore, the techniques described can enable infrastructure management within these environments. In some examples, a service team may write code that is desirable to be deployed to one or more, but often many, different production environments (e.g., across various geographical locations, sometimes even across the entire world). However, in some examples, the infrastructure to which the code is deployed must be configured first. In some cases, provisioning can be done manually, and provisioning tools may be used to provision resources, and / or deployment tools may be used to deploy the code after the infrastructure has been provisioned.

[0206] Figure 16 is a block diagram 1600 showing an exemplary pattern of an IaaS architecture according to at least one embodiment. A service operator 1602 may be communicably coupled to a secure host tenancy 1604 which may include a virtual cloud network (VCN) 1606 and a secure host subnet 1608. In some examples, the service operator 1602 may use one or more client computing devices, which may be portable handheld devices (e.g., iPhone®, mobile phones, iPad®, computing tablets, personal digital assistants (PDAs)) or wearable devices (e.g., Google Glass® head-mounted displays) with the Internet, email, short message service (SMS), Blackberry®, or other communication protocols enabled, running software such as Microsoft Windows Mobile® and / or various mobile operating systems such as iOS, Windows Phone, Android, BlackBerry 8, Palm OS. Alternatively, a client computing device can be a general-purpose personal computer, including, for example, personal computers and / or laptop computers running various versions of Microsoft Windows®, Apple Macintosh®, and / or Linux® operating systems. A client computing device can also be a workstation computer running any of various commercially available UNIX® or UNIX-like operating systems, including, but not limited to, various GNU / Linux operating systems such as Google Chrome OS.Alternatively or additionally, the client computing device may be any other electronic device, such as a thin client computer, an internet-enabled gaming system (e.g., a Microsoft Xbox game console with or without a Kinect® gesture input device), and / or a personal messaging device, that can communicate over a network and / or the Internet that has access to VCN1606.

[0207] VCN1606 may include a local peering gateway (LPG) 1610, which can be communicatively coupled to Secure Shell (SSH) VCN1612 via LPG1610 contained within SSH VCN1612. SSH VCN1612 may include an SSH subnet 1614, and SSH VCN1612 may be communicatively coupled to control plane VCN1616 via LPG1610 contained within control plane VCN1616. Additionally, SSH VCN1612 may be communicatively coupled to data plane VCN1618 via LPG1610. Control plane VCN1616 and data plane VCN1618 may be contained within a service tenancy 1619, which may be owned and / or operated by an IaaS provider.

[0208] The control plane VCN 1616 may include a control plane demilitarized zone (DMZ) layer 1620 that functions as a perimeter network (e.g., part of the corporate network between the corporate intranet and the external network). Servers based in the DMZ have limited responsibilities and can help keep security breaches contained. Furthermore, the DMZ layer 1620 may include a control plane application layer 1624 that may include one or more load balancer (LB) subnets 1622, an application subnet 1626, and a control plane data layer 1628 that may include database (DB) subnets 1630 (e.g., a frontend DB subnet and / or a backend DB subnet). The LB subnet 1622 included in the control plane DMZ layer 1620 can be communicatively coupled to the application subnet 1626 included in the control plane application layer 1624 and to the Internet gateway 1634 which may be included in the control plane VCN 1616. The application subnet 1626 can be communicatively coupled to the DB subnet 1630 included in the control plane data layer 1628, as well as to the service gateway 1636 and the network address translation (NAT) gateway 1638. The control plane VCN 1616 may include the service gateway 1636 and the NAT gateway 1638.

[0209] The control plane VCN1616 may include a data plane mirror application layer 1640 which may include an application subnet 1626. The application subnet 1626 included in the data plane mirror application layer 1640 may include a virtual network interface controller (VNIC) 1642 which can run compute instance 1644. Compute instance 1644 can communicatively connect the application subnet 1626 of the data plane mirror application layer 1640 to an application subnet 1626 which may be included in the data plane application layer 1646.

[0210] The data plane VCN1618 may include a data plane application layer 1646, a data plane DMZ layer 1648, and a data plane data layer 1650. The data plane DMZ layer 1648 may include an LB subnet 1622 that can be communicatively coupled to the application subnet 1626 of the data plane application layer 1646 and the internet gateway 1634 of the data plane VCN1618. The application subnet 1626 may be communicatively coupled to the service gateway 1636 of the data plane VCN1618 and the NAT gateway 1638 of the data plane VCN1618. The data plane data layer 1650 may also include a DB subnet 1630 that can be communicatively coupled to the application subnet 1626 of the data plane application layer 1646.

[0211] The Internet gateways 1634 of the control plane VCN1616 and the data plane VCN1618 can be communicatively coupled to a metadata management service 1652, which can be communicatively coupled to the public internet 1654. The public internet 1654 can be communicatively coupled to the NAT gateways 1638 of the control plane VCN1616 and the data plane VCN1618. The service gateways 1636 of the control plane VCN1616 and the data plane VCN1618 can be communicatively coupled to a cloud service 1656.

[0212] In some cases, a service gateway 1636 of the control plane VCN1616 or data plane VCN1618 can make application programming interface (API) calls to a cloud service 1656 without going through the public internet 1654. API calls from the service gateway 1636 to the cloud service 1656 can be one-way, with the service gateway 1636 making the API call to the cloud service 1656, and the cloud service 1656 sending the requested data to the service gateway 1636. However, the cloud service 1656 does not need to initiate the API call to the service gateway 1636.

[0213] In some examples, the secure host tenancy 1604 can be directly connected to the service tenancy 1619, or otherwise may be isolated. The secure host subnet 1608 can communicate with the SSH subnet 1614 via the LPG 1610, which can enable bidirectional communication on otherwise isolated systems. Connecting the secure host subnet 1608 to the SSH subnet 1614 may give the secure host subnet 1608 access to other entities within the service tenancy 1619.

[0214] The control plane VCN1616 may allow users of service tenancy 1619 to configure or otherwise provision desired resources. Desired resources provisioned within the control plane VCN1616 may be deployed or otherwise used in the data plane VCN1618. In some examples, the control plane VCN1616 can be isolated from the data plane VCN1618, and the data plane mirror application layer 1640 of the control plane VCN1616 can communicate with the data plane application layer 1646 of the data plane VCN1618 via a VNIC 1642 which may be included in the data plane mirror application layer 1640 and the data plane application layer 1646.

[0215] In some cases, a system user or customer may perform a request, such as a create, read, update, or delete (CRUD) operation, via the public internet 1654, which can then transmit the request to the metadata management service 1652. The metadata management service 1652 can then transmit the request to the control plane VCN 1616 via the internet gateway 1634. The request may be received by the LB subnet 1622, which is included in the control plane DMZ layer 1620. The LB subnet 1622 may determine that the request is valid, and in response to this determination, it may send the request to the application subnet 1626, which is included in the control plane application layer 1624. If the request is validated and requires a call to the public internet 1654, the call to the public internet 1654 may be sent to the NAT gateway 1638, which can make calls to the public internet 1654. Memory that may be desirable to be stored by the request may be stored in the DB subnet 1630.

[0216] In some cases, the data plane mirror application layer 1640 can facilitate direct communication between the control plane VCN1616 and the data plane VCN1618. For example, it may be desirable that changes, updates, or other appropriate modifications to the configuration be applied to resources contained in the data plane VCN1618. Through VNIC1642, the control plane VCN1616 can communicate directly with the resources contained in the data plane VCN1618, thereby enabling it to perform changes, updates, or other appropriate modifications to the configuration of those resources.

[0217] In some embodiments, the control plane VCN1616 and data plane VCN1618 may be included in the service tenancy 1619. In this case, the system user or customer does not have to own or operate either the control plane VCN1616 or the data plane VCN1618. Instead, the IaaS provider may own or operate the control plane VCN1616 and data plane VCN1618, both of which may be included in the service tenancy 1619. This embodiment can enable network isolation, which can prevent the user or customer from interacting with the resources of other users or other customers. This embodiment may also enable the system user or customer to store databases privately without requiring them to rely on the public internet 1654 for storage, which may not have the desired level of security.

[0218] In another embodiment, the LB subnet 1622 included in the control plane VCN 1616 may be configured to receive signals from the service gateway 1636. In this embodiment, the control plane VCN 1616 and the data plane VCN 1618 may be configured to be invoked by the IaaS provider's customer without calling the public internet 1654. The IaaS provider's customer may prefer this embodiment because the database used by the customer may be stored in a service tenancy 1619 that may be controlled by the IaaS provider and isolated from the public internet 1654.

[0219] Figure 17 is a block diagram 1700 showing another exemplary pattern of an IaaS architecture according to at least one embodiment. A service operator 1702 (e.g., service operator 1602 in Figure 16) may be communicatively coupled to a secure host tenancy 1704 (e.g., secure host tenancy 1604 in Figure 16), which may include a virtual cloud network (VCN) 1706 (e.g., VCN1606 in Figure 16) and a secure host subnet 1708 (e.g., secure host subnet 1608 in Figure 16). The VCN 1706 may include an LPG 1710 (e.g., LPG1610 in Figure 16), which may be communicatively coupled to an SSH VCN 1712 (e.g., SSH VCN1612 in Figure 16) via a local peering gateway (LPG) 1710 contained within a secure shell (SSH) VCN 1712. SSH VCN1712 may contain SSH subnet 1714 (e.g., SSH subnet 1614 in Figure 16), and SSH VCN1712 may be communicably coupled to control plane VCN1716 (e.g., control plane VCN1616 in Figure 16) via LPG1710 contained within control plane VCN1716. Control plane VCN1716 may contain service tenancy 1719 (e.g., service tenancy 1619 in Figure 16), and data plane VCN1718 (e.g., data plane VCN1618 in Figure 16) may contain customer tenancy 1721, which may be owned or operated by a user or customer of the system.

[0220] The control plane VCN1716 may include a control plane DMZ layer 1720 (e.g., control plane DMZ layer 1620 in Figure 16) which may include an LB subnet 1722 (e.g., LB subnet 1622 in Figure 16), a control plane application layer 1724 (e.g., control plane application layer 1624 in Figure 16) which may include an application subnet 1726 (e.g., application subnet 1626 in Figure 16), and a control plane data layer 1728 (e.g., control plane data layer 1628 in Figure 16) which may include a database (DB) subnet 1730 (e.g., similar to DB subnet 1630 in Figure 16). The LB subnet 1722 included in the control plane DMZ layer 1720 can be communicatively coupled to the application subnet 1726 included in the control plane application layer 1724 and to the Internet gateway 1734 (e.g., Internet gateway 1634 in Figure 16) which may be included in the control plane VCN 1716. The application subnet 1726 can be communicatively coupled to the DB subnet 1730 included in the control plane data layer 1728 and to the service gateway 1736 (e.g., the service gateway in Figure 16) and the network address translation (NAT) gateway 1738 (e.g., NAT gateway 1638 in Figure 16). The control plane VCN 1716 may include the service gateway 1736 and the NAT gateway 1738.

[0221] The control plane VCN 1716 may include a data plane mirror application layer 1740 (e.g., data plane mirror application layer 1640 in Figure 16) which may include an application subnet 1726. The application subnet 1726 included in the data plane mirror application layer 1740 may include a virtual network interface controller (VNIC) 1742 (e.g., VNIC 1642) which can run a compute instance 1744 (e.g., similar to compute instance 1644 in Figure 16). The compute instance 1744 can facilitate communication between the application subnet 1726 of the data plane mirror application layer 1740 and the application subnet 1726 that may be included in the data plane application layer 1746 (e.g., data plane application layer 1646 in Figure 16) via the VNIC 1742 included in the data plane mirror application layer 1740 and the VNIC 1742 included in the data plane application layer 1746.

[0222] The Internet gateway 1734 included in the control plane VCN 1716 can be communicably coupled to a metadata management service 1752 (e.g., metadata management service 1652 in Figure 16), which can be communicably coupled to the public internet 1754 (e.g., public internet 1654 in Figure 16). The public internet 1754 can be communicably coupled to a NAT gateway 1738 included in the control plane VCN 1716. The service gateway 1736 included in the control plane VCN 1716 can be communicably coupled to a cloud service 1756 (e.g., cloud service 1656 in Figure 16).

[0223] In some examples, the data plane VCN1718 may be included in the customer's tenancy 1721. In this case, the IaaS provider may provide a control plane VCN1716 for each customer, and the IaaS provider may configure a unique compute instance 1744 for each customer, included in the service tenancy 1719. Each compute instance 1744 may enable communication between the control plane VCN1716 included in the service tenancy 1719 and the data plane VCN1718 included in the customer's tenancy 1721. The compute instance 1744 may enable resources provisioned within the control plane VCN1716 included in the service tenancy 1719 to be deployed, or otherwise used, in the data plane VCN1718 included in the customer's tenancy 1721.

[0224] In another example, an IaaS provider's customer may have a database that persists in the customer's tenancy 1721. In this example, the control plane VCN 1716 may include a data plane mirror app layer 1740 that can include an app subnet 1726. The data plane mirror app layer 1740 may reside in the data plane VCN 1718, but does not have to reside in the data plane VCN 1718. That is, the data plane mirror app layer 1740 may have access rights to the customer's tenancy 1721, but does not have to reside in the data plane VCN 1718, and does not have to be owned or operated by the IaaS provider's customer. The data plane mirror app layer 1740 may be configured to make calls to the data plane VCN 1718, but does not have to be configured to make calls to any entities contained in the control plane VCN 1716. Customers may wish to deploy or otherwise use resources in the data plane VCN1718 that are provisioned within the control plane VCN1716, and the data plane mirror application layer 1740 can facilitate the customer's desired deployment or other use of resources.

[0225] In some embodiments, a customer of the IaaS provider can apply filters to the data plane VCN1718. In this embodiment, the customer can determine which data plane VCN1718s are accessible, and the customer may restrict access from the data plane VCN1718 to the public internet 1754. The IaaS provider does not need to be able to apply filters or otherwise control access of the data plane VCN1718 to any external network or database. Applying filters and controls to the data plane VCN1718 included in the customer's tenancy 1721 can help isolate the data plane VCN1718 from other customers and from the public internet 1754.

[0226] In some embodiments, cloud service 1756 may be invoked by service gateway 1736 to access services that may not exist on the public internet 1754, control plane VCN 1716, or data plane VCN 1718. The connection between cloud service 1756 and control plane VCN 1716 or data plane VCN 1718 may not be operational or continuous. Cloud service 1756 may reside on different networks owned or operated by the IaaS provider. Cloud service 1756 may be configured to receive calls from service gateway 1736 and not to receive calls from the public internet 1754. Some cloud services 1756 may be isolated from other cloud services 1756, and control plane VCN 1716 may be isolated from cloud services 1756 that may not be in the same region as control plane VCN 1716. For example, the control plane VCN1716 may be located in "Region 1," and the cloud service "Deployment 16" may be located in both Region 1 and "Region 2." If a call to Deployment 16 is made by a service gateway 1736 included in the control plane VCN1716 located in Region 1, this call may be sent to Deployment 16 in Region 1. In this example, the control plane VCN1716, or Deployment 16 in Region 1, may be communicatively coupled to Deployment 16 in Region 2, or otherwise not have to communicate with Deployment 16 in Region 2.

[0227] Figure 18 is a block diagram 1800 showing another exemplary pattern of an IaaS architecture according to at least one embodiment. A service operator 1802 (e.g., service operator 1602 in Figure 16) may be communicatively coupled to a secure host tenancy 1804 (e.g., secure host tenancy 1604 in Figure 16) which may include a virtual cloud network (VCN) 1806 (e.g., VCN1606 in Figure 16) and a secure host subnet 1808 (e.g., secure host subnet 1608 in Figure 16). VCN 1806 may include an LPG 1810 (e.g., LPG1610 in Figure 16) which may be communicatively coupled to SSH VCN 1812 (e.g., SSH VCN1612 in Figure 16) via an LPG 1810 contained in SSH VCN 1812. SSH VCN1812 may contain SSH subnet 1814 (e.g., SSH subnet 1614 in Figure 16), and SSH VCN1812 may be communicably coupled to control plane VCN1816 (e.g., control plane VCN1616 in Figure 16) via LPG1810 contained in control plane VCN1816, and to data plane VCN1818 (e.g., data plane 1618 in Figure 16) via LPG1810 contained in data plane VCN1818. Control plane VCN1816 and data plane VCN1818 may be contained in service tenancy 1819 (e.g., service tenancy 1619 in Figure 16).

[0228] The control plane VCN1816 may include a control plane DMZ layer 1820 (e.g., control plane DMZ layer 1620 in Figure 16) which may include a load balancer (LB) subnet 1822 (e.g., LB subnet 1622 in Figure 16), a control plane application layer 1824 (e.g., control plane application layer 1624 in Figure 16) which may include an application subnet 1826 (e.g., similar to application subnet 1626 in Figure 16), and a control plane data layer 1828 (e.g., control plane data layer 1628 in Figure 16) which may include a DB subnet 1830. The LB subnet 1822 included in the control plane DMZ layer 1820 can be communicatively coupled to the application subnet 1826 included in the control plane application layer 1824, and to an internet gateway 1834 (e.g., internet gateway 1634 in Figure 16) which may be included in the control plane VCN 1816. The application subnet 1826 can be communicatively coupled to the DB subnet 1830 included in the control plane data layer 1828, and to a service gateway 1836 (e.g., service gateway in Figure 16) and a network address translation (NAT) gateway 1838 (e.g., NAT gateway 1638 in Figure 16). The control plane VCN 1816 may include the service gateway 1836 and the NAT gateway 1838.

[0229] The data plane VCN1818 may include a data plane application layer 1846 (e.g., data plane application layer 1646 in Figure 16), a data plane DMZ layer 1848 (e.g., data plane DMZ layer 1648 in Figure 16), and a data plane data layer 1850 (e.g., data plane data layer 1650 in Figure 16). The data plane DMZ layer 1848 may include a trusted application subnet 1860 and an untrusted application subnet 1862 of the data plane application layer 1846, as well as an LB subnet 1822 that can be communicatively coupled to an internet gateway 1834 included in the data plane VCN1818. The trusted application subnet 1860 may be communicatively coupled to a service gateway 1836 included in the data plane VCN1818, a NAT gateway 1838 included in the data plane VCN1818, and a DB subnet 1830 included in the data plane data layer 1850. The untrusted application subnet 1862 can be communicatively coupled to the service gateway 1836, which is included in the data plane VCN 1818, and to the DB subnet 1830, which is included in the data plane data layer 1850. The data plane data layer 1850 may include the DB subnet 1830, which can be communicatively coupled to the service gateway 1836, which is included in the data plane VCN 1818.

[0230] An untrusted application subnet 1862 may include one or more primary VNICs 1864(1)-(N) that can be communicatively coupled to tenant virtual machines (VMs) 1866(1)-(N). Each tenant VM 1866(1)-(N) may be communicatively coupled to each application subnet 1867(1)-(N) that may be included in each container exit VCN 1868(1)-(N) that may be included in each customer tenancy 1870(1)-(N). Each secondary VNIC 1872(1)-(N) can facilitate communication between the untrusted application subnet 1862 included in the data plane VCN 1818 and the application subnets included in the container exit VCN 1868(1)-(N). Each container exit VCN 1868(1)-(N) may include a NAT gateway 1838 that can be communicatively coupled to the public internet 1854 (e.g., public internet 1654 in Figure 16).

[0231] The Internet gateway 1834, which is included in the control plane VCN1816 and the data plane VCN1818, can be communicatively coupled to a metadata management service 1852 (e.g., the metadata management system 1652 in Figure 16), which can be communicatively coupled to the public internet 1854. The public internet 1854 can be communicatively coupled to a NAT gateway 1838, which is included in the control plane VCN1816 and the data plane VCN1818. The service gateway 1836, which is included in the control plane VCN1816 and the data plane VCN1818, can be communicatively coupled to a cloud service 1856.

[0232] In some embodiments, the data plane VCN1818 may be integrated with the customer's tenancy 1870. This integration may be useful or desirable for the IaaS provider's customer in certain situations, such as when they may want support when executing code. The customer may provide code that may be destructive, communicate with other customers' resources, or otherwise cause undesirable effects. In response to this, the IaaS provider may determine whether it should execute code provided to the IaaS provider by the customer.

[0233] In some cases, an IaaS provider's customer might request the ability to grant the IaaS provider temporary network access privileges and connect to the data plane application layer 1846. The code to perform this function may run in VMs 1866(1) to (N), and this code does not need to be configured to run elsewhere on the data plane VCN 1818. Each VM 1866(1) to (N) may be connected to a single customer's tenancy 1870. Each container 1871(1) to (N) contained within VMs 1866(1) to (N) may be configured to run the code. In this case, a double isolation may exist (for example, container 1871(1)~(N) running the code, and container 1871(1)~(N) may be contained in at least VM1866(1)~(N) which is contained in the untrusted app subnet 1862), which can help prevent incorrect or otherwise unwanted code from damaging the IaaS provider's network or the network of a different customer. Container 1871(1)~(N) may be communicatively coupled to customer tenancy 1870 and may be configured to send or receive data to or from customer tenancy 1870. Container 1871(1)~(N) does not have to be configured to send or receive data to or from any other entity in the data plane VCN1818. Upon completion of code execution, the IaaS provider may terminate or otherwise discard container 1871(1)~(N).

[0234] In some embodiments, a trusted application subnet 1860 may execute code that may be owned or operated by the IaaS provider. In this embodiment, the trusted application subnet 1860 may be communicatively coupled to a DB subnet 1830 and configured to perform CRUD operations within the DB subnet 1830. An untrusted application subnet 1862 may be communicatively coupled to a DB subnet 1830, but in this embodiment, the untrusted application subnet may be configured to perform read operations within the DB subnet 1830. Containers 1871(1)~(N) that may be contained within each customer's VM1866(1)~(N) and capable of executing code from the customer do not need to be communicatively coupled to the DB subnet 1830.

[0235] In other embodiments, the control plane VCN1816 and the data plane VCN1818 do not need to be directly and communicatively coupled. In this embodiment, direct communication between the control plane VCN1816 and the data plane VCN1818 is not required. However, communication can occur indirectly by at least one method. The LPG1810 may be established by the IaaS provider and can facilitate communication between the control plane VCN1816 and the data plane VCN1818. In another example, the control plane VCN1816 or the data plane VCN1818 can make a call to the cloud service 1856 via the service gateway 1836. For example, a call from the control plane VCN1816 to the cloud service 1856 may include a request for a service that can communicate with the data plane VCN1818.

[0236] Figure 19 is a block diagram 1900 showing another exemplary pattern of an IaaS architecture according to at least one embodiment. A service operator 1902 (e.g., service operator 1602 in Figure 16) may be communicatively coupled to a secure host tenancy 1904 (e.g., secure host tenancy 1604 in Figure 16) which may include a virtual cloud network (VCN) 1906 (e.g., VCN1606 in Figure 16) and a secure host subnet 1908 (e.g., secure host subnet 1608 in Figure 16). VCN1906 may include an LPG1910 (e.g., LPG1610 in Figure 16) which may be communicatively coupled to SSH VCN1912 (e.g., SSH VCN1612 in Figure 16) via an LPG1910 contained in SSH VCN1912. SSH VCN1912 may contain SSH subnet 1914 (e.g., SSH subnet 1614 in Figure 16), and SSH VCN1912 may be communicably coupled to control plane VCN1916 (e.g., control plane VCN1616 in Figure 16) via LPG1910 contained in control plane VCN1916, and to data plane VCN1918 (e.g., data plane 1618 in Figure 16) via LPG1910 contained in data plane VCN1918. Control plane VCN1916 and data plane VCN1918 may be contained in service tenancy 1919 (e.g., service tenancy 1619 in Figure 16).

[0237] The control plane VCN1916 may include a control plane DMZ layer 1920 (e.g., control plane DMZ layer 1620 in Figure 16) which may include an LB subnet 1922 (e.g., LB subnet 1622 in Figure 16), a control plane application layer 1924 (e.g., control plane application layer 1624 in Figure 16) which may include an application subnet 1926 (e.g., application subnet 1626 in Figure 16), and a control plane data layer 1928 (e.g., control plane data layer 1628 in Figure 16) which may include a DB subnet 1930 (e.g., DB subnet 1830 in Figure 18). The LB subnet 1922 included in the control plane DMZ layer 1920 can be communicatively coupled to the application subnet 1926 included in the control plane application layer 1924, and to an internet gateway 1934 (e.g., internet gateway 1634 in Figure 16) which may be included in the control plane VCN 1916. The application subnet 1926 can be communicatively coupled to the DB subnet 1930 included in the control plane data layer 1928, and to a service gateway 1936 (e.g., service gateway in Figure 16) and a network address translation (NAT) gateway 1938 (e.g., NAT gateway 1638 in Figure 16). The control plane VCN 1916 may include the service gateway 1936 and the NAT gateway 1938.

[0238] The data plane VCN1918 may include a data plane application layer 1946 (e.g., data plane application layer 1646 in Figure 16), a data plane DMZ layer 1948 (e.g., data plane DMZ layer 1648 in Figure 16), and a data plane data layer 1950 (e.g., data plane data layer 1650 in Figure 16). The data plane DMZ layer 1948 may include trusted application subnets 1960 (e.g., trusted application subnet 1860 in Figure 18) and untrusted application subnets 1962 (e.g., untrusted application subnet 1862 in Figure 18) of the data plane application layer 1946, as well as an LB subnet 1922 that can be communicatively coupled to an internet gateway 1934 included in the data plane VCN1918. A trusted application subnet 1960 may be communicatively coupled to a service gateway 1936 included in data plane VCN 1918, a NAT gateway 1938 included in data plane VCN 1918, and a DB subnet 1930 included in data plane data layer 1950. An untrusted application subnet 1962 may be communicatively coupled to a service gateway 1936 included in data plane VCN 1918, and a DB subnet 1930 included in data plane data layer 1950. The data plane data layer 1950 may include a DB subnet 1930 that can be communicatively coupled to a service gateway 1936 included in data plane VCN 1918.

[0239] An untrusted application subnet 1962 may include primary VNICs 1964(1)-(N) that can be communicatively coupled to tenant virtual machines (VMs) 1966(1)-(N) residing within the untrusted application subnet 1962. Each tenant VM 1966(1)-(N) can execute code within its respective container 1967(1)-(N) and may be communicatively coupled to an application subnet 1926 that may be contained in a dataplane application layer 1946 that may be contained in a container exit VCN 1968. Each secondary VNIC 1972(1)-(N) can facilitate communication between the untrusted application subnet 1962 contained in a dataplane VCN 1918 and the application subnet contained in a container exit VCN 1968. The container exit VCN may include a NAT gateway 1938 that can be communicatively coupled to the public internet 1954 (e.g., public internet 1654 in Figure 16).

[0240] The Internet gateway 1934, included in the control plane VCN1916 and the data plane VCN1918, can be communicatively coupled to a metadata management service 1952 (e.g., the metadata management system 1652 in Figure 16), which can be communicatively coupled to the public internet 1954. The public internet 1954 can be communicatively coupled to a NAT gateway 1938, included in the control plane VCN1916 and the data plane VCN1918. The service gateway 1936, included in the control plane VCN1916 and the data plane VCN1918, can be communicatively coupled to a cloud service 1956.

[0241] In some examples, the pattern shown by the architecture in block diagram 1900 of Figure 19 may be considered an exception to the pattern shown by the architecture in block diagram 1800 of Figure 18, which may be desirable for the IaaS provider's customers when the IaaS provider cannot communicate directly with the customers (e.g., disconnected regions). Each container 1967(1)-(N) contained within VM1966(1)-(N) for each customer may be accessible in real time by the customer. Each container 1967(1)-(N) may be configured to make calls to each secondary VNIC 1972(1)-(N) contained in the application subnet 1926 of the data plane application layer 1946, which may be contained in the container exit VCN 1968. The secondary VNICs 1972(1)-(N) may send calls to the NAT gateway 1938, which may send calls to the public internet 1954. In this example, containers 1967(1)-(N), which can be accessed in real time by customers, can be isolated from the control plane VCN1916 and from other entities contained in the data plane VCN1918. Containers 1967(1)-(N) may also be isolated from other customer resources.

[0242] In another example, a customer can use containers 1967(1)-(N) to invoke cloud service 1956. In this example, the customer may execute code within containers 1967(1)-(N) that requests a service from cloud service 1956. Containers 1967(1)-(N) can send this request to secondary VNICs 1972(1)-(N), which can send this request to the NAT gateway, which can send this request to the public internet 1954. The public internet 1954 can send this request via internet gateway 1934 to LB subnet 1922, which is included in control plane VCN 1916. In response to determining that this request is valid, the LB subnet can send this request to application subnet 1926, which can send this request via service gateway 1936 to cloud service 1956.

[0243] It should be understood that the IaaS architectures 1600, 1700, 1800, and 1900 shown in the figures may include components other than those shown. Furthermore, the embodiments shown in the figures are merely examples of cloud infrastructure systems that may incorporate embodiments of this disclosure. In some other embodiments, the IaaS system may include more or fewer components than those shown in the figures, may combine two or more components, or may have different configurations or arrangements of components.

[0244] In some embodiments, the IaaS system described herein may include the provision of a set of applications, middleware, and database services delivered to customers in a self-service, subscription-based, elastically scalable, reliable, highly available, and secure manner. An example of such an IaaS system is the Oracle Cloud Infrastructure (OCI) provided by the Assignee.

[0245] Figure 20 shows an exemplary computer system 2000 in which various embodiments may be implemented. System 2000 may be used to implement any of the computer systems described above. As shown in the figure, computer system 2000 includes a processing unit 2004 that communicates with several peripheral subsystems via a bus subsystem 2002. These peripheral subsystems may include a processing acceleration unit 2006, an I / O subsystem 2008, a storage subsystem 2018, and a communication subsystem 2024. The storage subsystem 2018 includes a tangible computer-readable storage medium 2022 and system memory 2010.

[0246] The bus subsystem 2002 provides a mechanism for various components and subsystems of the computer system 2000 to communicate with each other as intended. Although the bus subsystem 2002 is schematically shown as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses. The bus subsystem 2002 may be one of several types of bus structures, including a memory bus or memory controller, a peripheral bus, and a local bus using any of the various bus architectures. For example, such architectures may include the ISA (Industry Standard Architecture) bus, the MCA (Micro Channel Architecture) bus, the EISA (Enhanced ISA) bus, the VESA (Video Electronics Standards Association) local bus, and the PCI (Peripheral Component Interconnect) bus, which can be implemented as a mezzanine bus manufactured according to the IEEE P1386.1 standard.

[0247] A processing unit 2004, which can be implemented as one or more integrated circuits (e.g., conventional microprocessors or microcontrollers), controls the operation of the computer system 2000. One or more processors may be included in the processing unit 2004. These processors may include single-core processors or multi-core processors. In one embodiment, the processing unit 2004 may be implemented as one or more independent processing units 2032 and / or 2034, each containing a single-core processor or a multi-core processor. In another embodiment, the processing unit 2004 may be implemented as a quad-core processing unit formed by integrating two dual-core processors onto a single chip.

[0248] In various embodiments, the processing unit 2004 can execute various programs depending on the program code and can maintain multiple programs or processes running simultaneously. At any given time, some or all of the program code to be executed may reside in the processor 2004 and / or the storage subsystem 2018. With appropriate programming, the processor 2004 can provide the various functions described above. The computer system 2000 may further include a processing acceleration unit 2006 which may include a digital signal processor (DSP), a dedicated processor, and / or similar.

[0249] The I / O subsystem 2008 may include user interface input devices and user interface output devices. User interface input devices may include pointing devices such as keyboards, mice or trackballs, touchpads or touchscreens integrated into displays, scroll wheels, click wheels, dials, buttons, switches, keypads, voice input devices with voice command recognition systems, microphones, and other types of input devices. User interface input devices may include motion detection devices and / or gesture recognition devices, such as Microsoft Kinect® motion sensors, which enable users to control and interact with input devices, such as Microsoft Xbox® 360 game controllers, through a natural user interface using gestures and spoken commands. User interface input devices may also include eye gesture recognition devices, such as Google Glass® blink detectors, which detect the user's eye activity (e.g., blinking when taking a picture and / or selecting a menu) and translate eye gestures into input to an input device (e.g., Google Glass®). Furthermore, the user interface input device may include a voice recognition detection device that allows the user to interact with a voice recognition system (e.g., Siri® Navigator) via voice commands.

[0250] User interface input devices may include, but are not limited to, three-dimensional (3D) mice, joysticks or pointing sticks, gamepads, and graphic tablets, as well as audio / visual devices such as speakers, digital cameras, digital video cameras, portable media players, webcams, image scanners, fingerprint scanners, barcode readers, 3D scanners, 3D printers, laser rangefinders, and eye-tracking devices. Furthermore, user interface input devices may include medical imaging input devices such as computed tomography, magnetic resonance imaging, positional emission tomography, and medical ultrasound imaging devices. User interface input devices may also include audio input devices such as MIDI keyboards and digital musical instruments.

[0251] User interface output devices may include non-visual displays such as display subsystems, indicator lights, or audio output devices. Display subsystems may include flat panel devices such as flat panel devices using cathode ray tubes (CRTs), liquid crystal displays (LCDs), or plasma displays, projection devices, touchscreens, etc. Generally, the use of the term “output device” is intended to include all possible types of devices and mechanisms for outputting information from the computer system 2000 to a user or another computer. For example, user interface output devices may include, but are not limited to, various display devices that visually convey text information, graphics information, and audio / video information, such as monitors, printers, speakers, headphones, car navigation systems, plotters, audio output devices, and modems.

[0252] The computer system 2000 may include a storage subsystem 2018 containing software elements as currently located in system memory 2010. System memory 2010 may store program instructions that are readable and executable by processing unit 2004, as well as data generated during the execution of these programs.

[0253] Depending on the configuration and type of computer system 2000, system memory 2010 may be volatile (such as random-access memory (RAM)) and / or non-volatile (such as read-only memory (ROM) or flash memory). RAM typically contains data and / or program modules that are immediately accessible by the processing unit 2004 and / or are currently being manipulated and executed. In some implementations, system memory 2010 may contain several different types of memory, such as static random-access memory (SRAM) or dynamic random-access memory (DRAM). In some implementations, a basic input / output system (BIOS), which contains basic routines that help transfer information between elements within computer system 2000 during startup and other times, may typically be stored in ROM. As an example, and not an limitation, System Memory 2010 also includes Application Programs 2012, Program Data 2014, and Operating Systems 2016, which may include client applications, web browsers, mid-tier applications, relational database management systems (RDBMS), etc.Examples of operating systems in 2016 include Microsoft Windows®, Apple Macintosh®, and / or Linux operating systems, various commercially available UNIX® or UNIX-like operating systems (including, but not limited to, various GNU / Linux operating systems, Google Chrome® OS, etc.), and / or various versions of mobile operating systems such as iOS, Windows® Phone, Android® OS, BlackBerry® OS, and Palm® OS.

[0254] The storage subsystem 2018 may provide a tangible, computer-readable storage medium for storing basic programming and data configurations that provide the functionality of certain embodiments. Software (programs, code modules, instructions) that provides the aforementioned functionality when executed by the processor may be stored in the storage subsystem 2018. These software modules or instructions may be executed by the processing unit 2004. The storage subsystem 2018 may provide a repository for storing data used in accordance with this disclosure.

[0255] The storage subsystem 2000 may include a computer-readable storage medium reader 2020 which may be further connected to the computer-readable storage medium 2022. In combination with the system memory 2010, together, optionally, the computer-readable storage medium 2022 may comprehensively represent storage media for temporarily and / or more permanently containing, storing, transmitting, and retrieving computer-readable information, in addition to remote, local, fixed, and / or removable storage devices.

[0256] Computer-readable storage medium 2022 containing code or a portion of code may also include any suitable medium known or used in the art, including, but not limited to, storage and communication media, such as volatile and non-volatile, removable and non-removable media, implemented in any way or technique for storing and / or transmitting information. Computer-readable storage medium 2022 may include tangible computer-readable storage media such as RAM, ROM, electronically erasable programmable ROM (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile disk (DVD), or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage, or other magnetic storage devices, or other tangible computer-readable media. Computer-readable storage medium 2022 may also include non-tangible computer-readable media such as data signals, data transmissions, or any other media that can be used to transmit desired information and can be accessed by the computing system 2000.

[0257] For example, Computer-Readable Storage Media 2022 may include hard disk drives that read from or write to non-removable non-volatile magnetic media, magnetic disk drives that read from or write to removable non-volatile magnetic disks, and optical disk drives that read from or write to removable non-volatile optical disks such as CD-ROMs, DVDs, and Blu-ray® discs, or other optical media. Computer-Readable Storage Media 2022 may also include, but is not limited to, Zip® drives, flash memory cards, Universal Serial Bus (USB) flash drives, Secure Digital (SD) cards, DVD discs, and digital videotapes. Computer-readable storage media 2022 may include solid-state drives (SSDs) based on non-volatile memory, such as flash memory-based SSDs, enterprise flash drives, and semiconductor ROMs; SSDs based on volatile memory, such as semiconductor RAM, dynamic RAM, static RAM, DRAM-based SSDs, and magneto-resistive RAM (MRAM) SSDs; and hybrid SSDs using a combination of DRAM and flash memory-based SSDs. Disk drives and associated computer-readable media may provide non-volatile storage for computer-readable instructions, data structures, program modules, and other data of the computer system 2022.

[0258] The communications subsystem 2024 provides interfaces to other computer systems and networks. It functions as an interface for computer system 2000 to receive data from other systems and to transmit data to other systems. For example, the communications subsystem 2024 may enable computer system 2000 to connect to one or more devices via the Internet. In some embodiments, the communications subsystem 2024 may include radio frequency (RF) transceiver components for accessing wireless voice and / or data networks (e.g., cellular technology, 3G, 4G, or EDGE (enhanced data rates for global evolution)), Wi-Fi (using IEEE 802.11 group standards, other mobile communication technologies, or any combination thereof), global positioning system (GPS) receiver components, and / or other components. In some embodiments, the communications subsystem 2024 may provide wired network connectivity (e.g., Ethernet) in addition to, or instead of, wireless interfaces.

[0259] In some embodiments, the communication subsystem 2024 may receive input communications on behalf of one or more users who may be using the computer system 2000, in the form of structured and / or unstructured data feeds 2026, event streams 2028, event updates 2030, etc.

[0260] For example, the communications subsystem 2024 may be configured to receive data feed 2026 in real time from users of social networks and / or other communications services such as Twitter® feeds and Facebook® updates, web feeds such as Rich Site Summary (RSS) feeds, and / or real-time updates from one or more third-party sources.

[0261] Furthermore, the communication subsystem 2024 may be configured to receive data in the form of a continuous data stream, which may include an event stream 2028 and / or event update 2030 of real-time events that are inherently continuous, without explicit ends, or without boundaries. Examples of applications that generate continuous data include, for example, sensor data applications, financial tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, and automotive traffic monitoring.

[0262] The communication subsystem 2024 may be configured to output structured and / or unstructured data feeds 2026, event streams 2028, event updates 2030, etc., to one or more databases that can communicate with one or more streaming data source computers coupled to the computer system 2000.

[0263] Computer System 2000 can be one of a variety of types, including handheld portable devices (e.g., iPhone® mobile phones, iPad® computing tablets, PDAs), wearable devices (e.g., Google Glass® head-mounted displays), PCs, workstations, mainframes, automated ticket machines, server racks, or any other data processing systems.

[0264] Due to the constantly changing nature of computers and networks, the description of the computer system 2000 shown in the figure is intended to be merely an example. Many other configurations are possible, including more or fewer components than the system shown in the figure. For example, customized hardware may be used, and / or certain elements may be implemented in hardware, firmware, software (including applets), or a combination thereof. Furthermore, connections to other computing devices, such as network input / output devices, may be employed. Based on the disclosures and teachings provided herein, those skilled in the art will understand other ways and / or methods for carrying out various embodiments.

[0265] While specific embodiments have been described, various modifications, changes, alternative structures, and equivalents are also included within the scope of this disclosure. The embodiments are not limited to operating within a particular data processing environment, but can freely operate within multiple data processing environments. Furthermore, while the embodiments have been described using a specific set of transactions and steps, it should be apparent to those skilled in the art that the scope of this disclosure is not limited to the described set of transactions and steps. The various features and aspects of the embodiments described above may be used individually or in combination.

[0266] Furthermore, while embodiments have been described using specific combinations of hardware and software, it should be recognized that other combinations of hardware and software are also within the scope of this disclosure. Embodiments may be implemented using hardware alone, software alone, or a combination thereof. Various processes described herein may be performed on different processors, either on the same processor or in any combination thereof. Thus, where a component or module is described as being configured to perform a certain operation, such configuration may be realized, for example, by designing electronic circuitry to perform this operation, by programming programmable electronic circuitry (such as a microprocessor) to perform this operation, or by any combination thereof. Processes may communicate using a variety of techniques, including but not limited to prior art for inter-process communication, and different pairs of processes may use different techniques, or the same pair of processes may use different techniques at different times.

[0267] Therefore, this specification and drawings should be considered illustrative, not limiting. However, it is clear that additions, reductions, deletions, and other modifications and changes may be made to this specification and drawings without departing from the broader idea and scope set forth in the claims. Thus, while certain embodiments of the disclosure have been described, they are not intended to be limiting. Various modifications and equivalents are within the scope of the appended claims.

[0268] The use of the terms “a,” “an,” and “the” and similar reference subjects in the context describing the disclosed embodiments (in particular, in the context of the appended claims) should be construed to apply to both singular and plural nouns unless otherwise specifically indicated herein or unless it is clearly inconsistent with the context. The terms “equipped,” “having,” “including,” and “containing” should be construed to be unrestricted terms (i.e., “including, but not limited to”) unless otherwise specifically noted. The term “connected” should be construed to mean partially or completely contained, connected, or joined together, even if there is something intervening. The enumeration of ranges of values ​​herein is intended simply as a way to refer individually to each separate value contained within the range unless otherwise specifically indicated herein, and each separate value is incorporated herein as if it were individually enumerated herein. All methods described herein may be performed in any appropriate order unless otherwise specifically indicated herein or unless it is clearly inconsistent with the context. Any use of any examples or illustrative language (e.g., "etc.") provided herein is intended solely to better illustrate the embodiments and, unless otherwise claimed, does not limit the scope of this disclosure. The language herein should not be construed as indicating that any unclaimed element is essential to the practice of this disclosure.

[0269] Disjunctive language, such as the phrase "at least one of X, Y, or Z," is generally intended to be understood in context as indicating that an item, condition, etc., may be any one of X, Y, or Z, or any combination thereof (e.g., X, Y, and / or Z), unless otherwise explicitly stated. Therefore, such disjunctive language is generally not intended, and should not, mean that a particular embodiment requires the presence of at least one of X, at least one of Y, or at least one of Z, respectively.

[0270] This specification describes preferred embodiments of the Disclosure, including known best modes for carrying out the Disclosure. As a person skilled in the art reads the foregoing description, variations of such preferred embodiments may become apparent. A person skilled in the art should be able to adopt such variations as needed, and the Disclosure may be practiced in ways other than those described in detail herein. Accordingly, this Disclosure includes all modifications and equivalents of the subject matter enumerated in the claims appended herein, as permitted by applicable law. Furthermore, any combination of the aforementioned elements in all possible variations of the embodiments is encompassed by this Disclosure unless specifically indicated herein.

[0271] All references cited herein, including published documents, patent applications, and patents, are individually and in detail incorporated by reference, and are incorporated herein by reference to the same extent as they would be incorporated as a whole. While aspects of this disclosure have been described in the preceding specification with reference to specific embodiments thereof, those skilled in the art will recognize that this disclosure is not limited thereto. The various features and aspects of the preceding disclosure may be used individually or together. Furthermore, embodiments may be used in any number of environments and applications beyond those described herein, without departing from the broader idea and scope of this specification. Thus, this specification and the drawings should be considered illustrative, not restrictive.

Claims

1. It is a method, In a first cloud environment, the provision includes providing a first compute virtual machine connected to a first customer's virtual cloud network (VCN) and a hub, wherein the first compute virtual machine is associated with a first plurality of virtual network interface cards (VNICs), and the first plurality of VNICs include (i) a first VNIC associated with a first network namespace that enables communication between the first customer's VCN and the first compute virtual machine, and (ii) a second VNIC associated with a second network namespace that enables communication between the first compute virtual machine and the hub, wherein the first network namespace differs from the second network namespace. The method includes providing a second compute virtual machine connected to a hub and to a VCN of a second customer in the first cloud environment, wherein the second compute virtual machine is associated with a second plurality of VNICs, the second plurality of VNICs including (i) a first VNIC associated with a third network namespace that enables communication between the second customer's VCN and the second compute virtual machine, and (ii) a second VNIC associated with the second network namespace that enables communication between the second compute virtual machine and the hub. The method enables communication between the VCN of the first customer in the first cloud environment and the second cloud environment using the first computing virtual machine and the hub, A method further comprising enabling communication between the second customer's VCN in the first cloud environment and the second cloud environment using the second computing virtual machine and the hub.

2. The method according to claim 1, wherein the first network namespace is programmed using first configuration information, and the second network namespace is programmed using second configuration information different from the first configuration information.

3. The method according to claim 1, wherein the third network namespace is different from the first network namespace.

4. The method according to claim 1, wherein the third network namespace is different from the second network namespace.

5. Each of the first configuration information and the second configuration information is, Routing table, IP address, Address Resolution Protocol Entry, The next hop destination, and, The method according to claim 2, including firewall rules.

6. The method according to claim 5, wherein the first configuration information includes a first set of DNS resolvers, and the second configuration information includes a second set of DNS resolvers different from the first set of DNS resolvers.

7. The method according to claim 1, wherein the first plurality of VNICs associated with the first computing virtual machine further include a third VNIC associated with a third network namespace, enabling communication between the first computing virtual machine and a control plane hub network.

8. The method according to claim 7, wherein the second plurality of VNICs associated with the second computing virtual machine further include a third VNIC associated with the third network namespace, enabling communication between the second computing virtual machine and the control plane hub network.

9. The method according to claim 7, wherein the third network namespace is different from each of the first network namespace and the second network namespace.

10. One or more computer-readable non-temporary media for storing computer executable instructions, wherein the computer executable instructions, when executed by one or more processors, In a first cloud environment, the provision of a first compute virtual machine connected to a hub and to a first customer's virtual cloud network (VCN) within the first cloud environment, wherein the first compute virtual machine is associated with a first plurality of virtual network interface cards (VNICs), and the first plurality of VNICs include (i) a first VNIC associated with a first network namespace that enables communication between the first customer's VCN and the first compute virtual machine, and (ii) a second VNIC associated with a second network namespace that enables communication between the first compute virtual machine and the hub, wherein the first network namespace differs from the second network namespace. The computer executable instructions, when executed by one or more processors, further cause the first cloud environment to provide a second computing virtual machine connected to the hub and to a second customer's VCN within the first cloud environment, wherein the second computing virtual machine is associated with a second plurality of VNICs, the second plurality of VNICs including (i) a first VNIC associated with a third network namespace that enables communication between the second customer's VCN and the second computing virtual machine, and (ii) a second VNIC associated with the second network namespace that enables communication between the second computing virtual machine and the hub. The computer executable instructions, when executed by one or more processors, enable communication between the first customer's VCN in the first cloud environment and the second cloud environment using the first computing virtual machine and the hub. One or more computer-readable non-temporary media for storing computer executable instructions, which further enable communication between the second customer's VCN in the first cloud environment and the second cloud environment using the second computing virtual machine and the hub.

11. One or more computer-readable non-temporary media for storing computer executable instructions according to claim 10, wherein the first network namespace is programmed using first configuration information, and the second network namespace is programmed using second configuration information different from the first configuration information.

12. The third network namespace is different from the first network namespace and is one or more computer-readable non-temporary media that store the computer-executable instructions according to claim 10.

13. The third network namespace is different from the second network namespace and is one or more computer-readable non-temporary media for storing the computer-executable instructions according to claim 10.

14. Each of the first configuration information and the second configuration information is, Routing table, IP address, Address Resolution Protocol Entry, The next hop destination, and, One or more computer-readable non-temporary media for storing computer-executable instructions according to claim 11, including firewall rules.

15. One or more computer-readable non-temporary media for storing computer-executable instructions according to claim 14, wherein the first configuration information includes a first set of DNS resolvers, and the second configuration information includes a second set of DNS resolvers different from the first set of DNS resolvers.

16. One or more computer-readable non-temporary media for storing computer-executable instructions according to claim 10, wherein the first plurality of VNICs associated with the first computing virtual machine further include a third VNIC associated with a third network namespace, enabling communication between the first computing virtual machine and a control plane hub network.

17. One or more processors, A computing device comprising memory containing instructions, wherein, when the instructions are executed using one or more processors, the computing device receives at least, In a first cloud environment, the system provides a first compute virtual machine connected to a hub and to a first customer's virtual cloud network (VCN) within the first cloud environment, the first compute virtual machine being associated with a first plurality of virtual network interface cards (VNICs), the first plurality of VNICs including (i) a first VNIC associated with a first network namespace that enables communication between the first customer's VCN and the first compute virtual machine, and (ii) a second VNIC associated with a second network namespace that enables communication between the first compute virtual machine and the hub, the first network namespace being different from the second network namespace, The instruction, when executed using the one or more processors, further causes the computing device to provide, in the first cloud environment, a second computing virtual machine connected to the hub and to the VCN of a second customer in the first cloud environment, wherein the second computing virtual machine is associated with a second plurality of VNICs, the second plurality of VNICs including (i) a first VNIC associated with a third network namespace that enables communication between the second customer's VCN and the second computing virtual machine, and (ii) a second VNIC associated with the second network namespace that enables communication between the second computing virtual machine and the hub. When the instruction is executed using the one or more processors, the computing device enables communication between the first customer's VCN in the first cloud environment and the second cloud environment using the first computing virtual machine and the hub. A computing device that further enables communication between the second customer's VCN in the first cloud environment and the second cloud environment using the second computing virtual machine and the hub.

18. The computing device according to claim 17, wherein the first network namespace is programmed using first configuration information, and the second network namespace is programmed using second configuration information different from the first configuration information.

19. The computing device according to claim 17, wherein the third network namespace is different from the first network namespace.

20. The computing device according to claim 17, wherein the third network namespace is different from the second network namespace.

Images