Provisioning and management of resources within the first cloud service provider's cloud infrastructure for cloud services provided by the second cloud service provider.

Abstract

Techniques are disclosed for provisioning and managing resources within the cloud infrastructure of a first cloud service provider for cloud services provided by a second cloud service provider. Cross-cloud services may be provisioned and managed by the cloud service provider's private cloud and between the cloud service provider's private clouds. These techniques include a component of the first private cloud in the first cloud environment receiving requests for cloud services from a component of the second private cloud in the second cloud environment. These techniques further include a component of the first private cloud performing one or more actions to establish prerequisites for network connectivity for network connectivity between the first private cloud and the second private cloud, and causing one or more components of the first private cloud to provision cloud services in the second private cloud.

Description

Technical Field

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of and priority to U.S. Provisional Patent Application No. 63 / 534,071 filed on August 22, 2023, U.S. Provisional Patent Application No. 63 / 538,254 filed on September 13, 2023, and U.S. Provisional Patent Application No. 63 / 608,036 filed on December 8, 2023, the entire contents of which are hereby incorporated by reference into this specification for all purposes.

Background Art

[0002] Background In recent years, there has been a dramatic increase in the adoption of cloud services, and this trend continues to grow. Various cloud environments are provided by different cloud service providers (CSPs), and each cloud environment provides a set of one or more cloud services. The set of cloud services provided by a cloud environment may include one or more different types of services including, but not limited to, SaaS (Software-as-a-Service) services, IaaS (Infrastructure-as-a-Service) services, PaaS (Platform-as-a-Service) services, and the like.

[0003] Although various cloud environments are currently available, each cloud environment provides a closed ecosystem to its subscribed customers. As a result, customers of a cloud environment are restricted to using the services provided by that cloud environment. There is no convenient way for a customer subscribed to a cloud environment provided by one CSP to use services provided by a different CSP through that cloud environment. The embodiments described herein address these and other problems.

Summary of Invention

Means for Solving the Problems

[0004] Brief Overview This specification discloses techniques for provisioning and managing resources within the cloud infrastructure of a first cloud service provider for cloud services provided by a second cloud service provider.

[0005] In some embodiments, the method includes a first component of a first private cloud in a first cloud environment receiving a request for a cloud service from a component of a second private cloud in a second cloud environment, and in response to receiving the request, the first component performing a set of operations associated with provisioning the cloud service in the second cloud environment, wherein at least one of the operations includes establishing prerequisites for network connectivity for network connectivity between the first private cloud and the second private cloud, and, after performing the set of operations, the first component causing one or more second components of the first private cloud to provision the cloud service in the second private cloud.

[0006] In some embodiments, the request is initiated in response to a customer device interacting with a portal in a second private cloud, and the customer device is associated with a customer account corresponding to the second private cloud.

[0007] In some embodiments, the portal is configured to present a portion of a graphical user interface provided by a first component, and a request is initiated in response to a customer device interacting with this portion.

[0008] In some embodiments, the first component allows one or more second components of the first private cloud to provision cloud services in the second private cloud, which includes allowing one or more second components to establish network connectivity between the first private cloud and the second private cloud based on network connectivity prerequisites.

[0009] In some embodiments, the first component provisioning cloud services in the second private cloud to one or more second components of the first private cloud includes provisioning virtual machines to another component of the second private cloud.

[0010] In some embodiments, establishing the prerequisites for network connectivity includes creating a first subnet in a first private cloud and a second subnet in a second private cloud, wherein at least one Internet Protocol address associated with the first subnet and at least one Internet Protocol address associated with the second subnet are within a Classless Inter-Domain Routing (CIDR) range.

[0011] In some embodiments, this method further includes integrating the first private cloud with the second private cloud by linking an identifier associated with the first component to an identifier associated with another component of the second private cloud, before the first component of the first private cloud in the first cloud environment receives a request for cloud services from a component of the second private cloud in the second cloud environment.

[0012] Some embodiments include a system comprising one or more processing systems and one or more computer-readable media storing instructions, which, when executed by one or more processing systems, cause the systems to perform some or all of the operations and / or methods disclosed herein.

[0013] Some embodiments include one or more non-temporary computer-readable media storing instructions, which, when executed by one or more processing systems, cause the systems to perform some or all of the operations and / or methods disclosed herein.

[0014] The techniques described above and below may be implemented in multiple ways and in multiple situations. Several exemplary implementations and situations are provided with reference to the following diagrams, which are described in more detail below. However, the following implementations and situations represent only a fraction of the many implementations and situations.

[0015] The features, embodiments, and advantages of this disclosure will be better understood when the following detailed description is read with reference to the accompanying drawings. [Brief explanation of the drawing]

[0016] [Figure 1] This is a high-level diagram of a distributed environment showing a virtual cloud network or overlay cloud network hosted by a cloud service provider infrastructure, according to one embodiment. [Figure 2] This figure shows a simplified architectural diagram of physical components in a physical network within a cloud service provider infrastructure (CSPI) according to one embodiment. [Figure 3]This figure shows an exemplary configuration within a CSPI in which a host machine is connected to multiple network virtualization devices (NVDs), according to one embodiment. [Figure 4] This figure shows the connection between the host machine and the NVD to implement I / O virtualization to support multi-tenancy functionality, according to one embodiment. [Figure 5] This figure shows a simplified block diagram of a physical network provided by CSPI according to one embodiment. [Figure 6] This figure shows a simplified high-level diagram of a distributed environment comprising multiple cloud environments provided by different cloud service providers (CSPs) according to one embodiment. [Figure 7] This figure shows an exemplary physical architecture for providing cross-cloud services based on a distributed infrastructure across multiple CSPs, according to some embodiments. [Figure 8] This figure shows an exemplary virtual architecture for providing cross-cloud services based on a distributed infrastructure across multiple CSPs, according to some embodiments. [Figure 9] This figure shows exemplary virtual resources provisioned by the first CSP for a customer of the second CSP, according to one embodiment. [Figure 10] This figure shows an exemplary architecture for provisioning and managing cross-cloud services based on a distributed infrastructure across multiple CSPs, according to some embodiments. [Figure 11] This figure shows an exemplary user experience flow for provisioning resources in some embodiments. [Figure 12] This figure shows an exemplary control plane provisioning flow according to some embodiments. [Figure 13]FIG. 1 is a diagram illustrating an example architecture including a resource management mechanism for provisioning and managing cross-cloud services between a plurality of cloud environments, in accordance with some embodiments. [Figure 14] FIG. 2 is a diagram illustrating an exemplary process for provisioning and managing cross-cloud services between a plurality of cloud environments, in accordance with some embodiments. [Figure 15] FIG. 3 is a block diagram illustrating one pattern for implementing a cloud infrastructure as a service system, in accordance with an embodiment. [Figure 16] FIG. 4 is a block diagram illustrating another pattern for implementing a cloud infrastructure as a service system, in accordance with an embodiment. [Figure 17] FIG. 5 is a block diagram illustrating another pattern for implementing a cloud infrastructure as a service system, in accordance with an embodiment. [Figure 18] FIG. 6 is a block diagram illustrating another pattern for implementing a cloud infrastructure as a service system, in accordance with an embodiment. [Figure 19] FIG. 7 is a block diagram illustrating an exemplary computer system, in accordance with an embodiment. MODE FOR CARRYING OUT THE INVENTION

[0017] Detailed Description In the following description, for purposes of explanation, specific details are set forth in order to provide a thorough understanding of certain embodiments. However, it will be apparent that various embodiments may be practiced without these specific details. The figures and description are not intended to be restrictive. The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment or design described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments or designs.

[0018] This disclosure relates in general to an improved cloud architecture, and more specifically to a technology for providing services based on a distributed infrastructure across multiple cloud service providers (CSPs). In one example, a first cloud service provider (CSP) (e.g., Oracle® Cloud Infrastructure (OCI)) provides a first service (e.g., a database service such as Oracle's Exadata service), some of which may be available from the first cloud service provider infrastructure (CSPI) of the first CSP. A second CSP (e.g., Microsoft® Azure, Google Cloud®, Amazon Web Services (AWS)®) provides a second service (e.g., Azure Virtual Machines), some of which may be available from the second cloud service provider infrastructure (CSPI) of the second CSP. At least one of the first services may be made available to the second CSP's customers via the second CSPI. In one example, hardware and / or software configured by the first CSPI to provide the first service is deployed on the second CSPI.

[0019] For example, at the physical level, a second CSPI may include a first set of computing resources from the first CSPI (e.g., racks of servers optimized for the first CSPI's services). The first CSPI may also include a second set of computing resources from the first CSPI (e.g., servers within a region). The second CSPI may also include a third set of computing resources from the second CSPI (e.g., servers within an availability zone). The first set of computing resources may be connected to the second and third sets of computing resources.

[0020] At the virtual level, a first set of resources and a second set of resources can host a first cloud for customers by the first CSP. By comparison, a third set of resources can host a second cloud for customers by the second CSP. These two clouds can be connected (for example, via a peering connection using a virtual router). The services provided to customers by the first CSP may have a first resource hosted in the first set of computing resources and a second resource hosted in the second set of computing resources. This service may be available to customers via the second cloud. The use of the first resources enables reduced latency. The provision of services by the first CSP through the second CSP can enable a better experience by extending the service to customers of the second CSP who may be more familiar with the cloud configuration and management by the second CSP.

[0021] To illustrate, consider the example of an Exadata service (i.e., an OCI database service) made available by Azure. In this example, at the physical level, an OCI child site belonging to an OCI region is located at the same location within an Azure data center. The OCI child site connects to the Azure data center via a FastConnect router (and a Microsoft MeetMe ToR router on the Azure side). The OCI child site also connects to the parent OCI region via a fiber optic connection. At the virtual level, the Azure data center hosts the customer's cloud (e.g., a VNET), while the OCI parent region and child site jointly host the customer's virtual cloud network (VCN). Latency-critical resources of the OCI service (e.g., data plane resources for the Exadata service) are hosted in the child site, while other support resources (e.g., control plane resources, customer console resources, etc.) are hosted in the parent region.

[0022] On the OCI side, VNETs and VCNs are connected via virtual routers (e.g., dynamic routing gateways (DRGs) that support FastConnect, and virtual routers provided by Azure that support MeetMe). A delegate subnet may be configured in the VNET. An Exadata service subnet may be configured in the VCN. Both of these subnets use the same Internet Protocol (IP) address range. The VNET resides within the IP address range and can store mapping information that maps the IP addresses used in the VNET to the IP addresses of the Exadata services in the VCN. Traffic to IP addresses (e.g., from compute instances in the VNET) is sent from the VNET to the VCN, assuming a one-to-one IP address mapping.

[0023] Therefore, from the customer's perspective, they perceive that the Exadata service within the VNET has an IP address. However, in reality, this Exadata service resides within the VCN under that IP address. Furthermore, by hosting at least a portion of the Exadata service in a child site (e.g., the data plane), latency associated with the Exadata service can be reduced.

[0024] Referring again to the architecture described above, the first set of computing resources can be considered to belong to a child site, while the second set of computing resources can be considered to belong to a parent site. The third set of computing resources can be considered to be data center resources. The child site of the first CSP is deployed in the data center (or second CSPI) of the second CSP. In comparison, the parent site of the first CSP is deployed in the first CSPI of the first CSP.

[0025] A child site can enable the first CSP's low-latency services to be delivered to customers via the second CSP. However, a deployment mechanism may be needed to control the deployment of the first CSP's resources in both the child and parent sites.

[0026] Embodiments of this disclosure relate to such a deployment mechanism. In one example, the deployment mechanism includes a control plane of a first CSP, which is not hosted at the child site. Customer input at a second CSPI can be received by the control plane and can be mapped to a customer ID by the first CSPI. Based on the customer ID, the control plane can provision resources at the first CSPI, for example, by creating a cloud network for the customer and connectivity resources associated with the cloud network (e.g., a dynamic routing gateway (DRG)). The cloud network can also be connected to a virtual machine (VM) cluster that hosts services. The VM cluster can be hosted at the child site, while other resources of the cloud network can be hosted at the parent site. The control plane can then interconnect the cloud network to the customer's cloud network by the second CSP so that the customer's two cloud networks are interconnected. As part of this interconnection, the control plane can provide the Internet Protocol (IP) addresses of the VM cluster, and potentially the corresponding DNS records, to a second CSP, which then makes these IP addresses, and potentially the DNS records, available to customers on the cloud network via the second CSP.

[0027] To illustrate, let's revisit the OCI and Azure examples. Azure customer input (e.g., VNET and subnet selection) can be received from the Azure Resource Manager (ARM). The Oracle Resource Provider (ORP) can translate this input into the customer's OCI ID and pass this information to the control plane. The control plane primarily performs two processes. The first process is to provision the relevant OCI resources. The second process is to connect these resources to the customer's VNET.

[0028] Following the first process, the control plane creates the customer's VCN in the customer tenancy (e.g., in the parent site) and creates subnets (and possibly backup subnets) within the VCN. The Azure and OCI subnets (subnets in the customer's Azure VNET and subnets in the customer's OCI VCN) use the same Classless Interdomain Routing (CIDR). The control plane also creates a DRG in the customer tenancy and connects this DRG to the VCN. The control plane also configures the routing information for the DRG and VCN (so that these two resources are interconnected). The ORP can also provision a VM cluster (e.g., in a child site). The IP address of this VM cluster from the CIDR is mapped to the corresponding DNS record.

[0029] Following the second process, the control plane registers these IP addresses with Azure (on the MeetMe router). The control plane also creates a virtual circuit between the DRG and the MeetMe router and sends DNS records so that a private DNS zone can be configured in Azure.

[0030] For clarity of explanation, embodiments of this disclosure are described in relation to specific CSPs (e.g., Oracle and Microsoft) and services (e.g., Exadata services). However, embodiments are not limited to such embodiments and instead apply similarly and equally to any CSP, CSPI, and service in a multi-cloud environment.

[0031] Examples of cloud networks The term "cloud service" is typically used to refer to services made available on demand (e.g., through a subscription model) by users or customers utilizing systems and infrastructure provided by the CSP (e.g., cloud infrastructure). Typically, the servers and systems that make up the CSP's infrastructure are separate from the customer's own on-premises systems and servers. Therefore, customers can utilize cloud services provided by the CSP without needing to purchase separate hardware and software resources for the service. Cloud services are designed to provide subscribers with easy and scalable access to a set of application and computing resources that do not require the customer to invest in procuring the infrastructure used to deliver the service.

[0032] There are multiple cloud service providers offering various types of cloud services. These include various types or models of cloud services, such as SaaS (Software-as-a-Service), IaaS (Infrastructure-as-a-Service), and PaaS (Platform-as-a-Service).

[0033] Customers can subscribe to one or more cloud services provided by the CSP. Customers can be any entity, such as an individual, organization, or company. When a customer subscribes to or registers for a service provided by the CSP, a tenancy or account is created for that customer. The customer can then use that account to access one or more subscribed cloud resources associated with the account.

[0034] As mentioned earlier, IaaS is a specific type of cloud computing service. In the IaaS model, the CSP provides infrastructure (called Cloud Service Provider Infrastructure or CSPI) that can be used by the customer to build their own customizable network and deploy their resources. Thus, the customer's resources and network are hosted in a distributed environment by the infrastructure provided by the CSP. This differs from traditional computing, where the customer's resources and network are hosted by infrastructure provided by the customer.

[0035] A CSPI can comprise interconnected, high-performance computing resources, including various host machines, memory resources, and network resources that form a physical network, also known as an underlay network. Resources in a CSPI may be distributed across one or more data centers, which may be geographically dispersed across one or more geographical regions. To provide a virtualized distributed environment, virtualization software may run on these physical resources. This virtualization creates an overlay network (also known as a software-based network, software-defined network, or virtual network) on top of the physical network (or underlay network). The physical network provides the foundation for creating one or more overlay or virtual networks on top of it. The physical network includes physical network devices such as physical switches, routers, computers, and host machines. An overlay network is a logical (or virtual) network that operates on top of the physical underlay network. A particular physical network can support one or more overlay networks. Overlay networks typically use encapsulation techniques to distinguish traffic belonging to different overlay networks. A virtual network or overlay network is also called a VCN. Virtual networks are implemented using software virtualization technologies (e.g., hypervisors, network virtualization devices (NVDs) (e.g., smart NICs), top-of-rack (ToR) switches, virtualization functions implemented by smart TORs that perform one or more functions performed by NVDs, and other mechanisms) to create a layer of network abstraction that can run on top of a physical network. Virtual networks can take many forms, including peer-to-peer networks and IP networks. Virtual networks are typically either Layer 3 IP networks or Layer 2 VLANs.This method of virtual networking or overlay networking is often called a virtual Layer 3 network or overlay Layer 3 network. Examples of protocols developed for virtual networks include IP-in-IP (or Generic Routing Encapsulation (GRE)), Virtual Extensible LAN (VXLAN - IETF RFC 7348), Virtual Private Networks (VPN) (e.g., MPLS Layer 3 Virtual Private Network (RFC 4364)), VMware's NSX, and GENEVE (Generic Network Virtualization Encapsulation).

[0036] In the case of IaaS, the CSPI provided by the CSP can be configured to deliver a set of virtualized computing resources over a public network (e.g., the internet). In the IaaS model, the cloud computing service provider can host infrastructure components (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer)). In some cases, the IaaS provider may also provide various services (e.g., billing, monitoring, logging, security, load balancing, and clustering) as they arise in conjunction with those infrastructure components. Since these services can be policy-driven, IaaS users may implement policies to drive load balancing to maintain application availability and performance. The CSPI provides infrastructure and a set of complementary cloud services that enable customers to build and run a wide range of applications and services in a highly available hosted distributed environment. The CSPI provides high-performance computing resources and computing power, as well as storage capacity, within a flexible virtual network that is securely accessible from a variety of networked locations, including the customer's on-premises network. When a customer subscribes to or registers for an IaaS service provided by a CSP, the tenancy created for that customer is a secure and isolated partition within the CSP where the customer can create, organize, and manage their own cloud resources.

[0037] Customers can build their own virtual networks using the computing, memory, and network resources provided by CSPI. One or more customer resources or workloads, such as compute instances, can be deployed to these virtual networks. For example, a customer can build one or more customizable private virtual networks called VCNs using the resources provided by CSPI. A customer can deploy one or more customer resources, such as compute instances, to their VCNs. Compute instances can take the form of virtual machines, bare metal instances, etc. Thus, CSPI provides the infrastructure and a set of complementary cloud services that enable customers to build and run a wide range of applications and services within a highly available hosted virtual environment. Customers do not manage or control the underlying physical resources provided by CSPI, but they can control the operating system, storage, and deployed applications, and, in some cases, have limited control over selected network components (e.g., firewalls).

[0038] The CSP may provide a console that enables customers and network administrators to configure, access, and manage resources deployed in the cloud using CSPI resources. In one embodiment, this console provides a web-based user interface that can be used to access and manage the CSPI. In some implementations, the console is a web-based application provided by the CSP.

[0039] CSPI may support single-tenancy or multi-tenancy architectures. In a single-tenancy architecture, software components (e.g., applications, databases) or hardware components (e.g., host machines or servers) work for a single customer or tenant. In a multi-tenancy architecture, software components or hardware components work for multiple customers or tenants. Therefore, in a multi-tenancy architecture, CSPI resources are shared among multiple customers or tenants. In a multi-tenancy situation, precautions are taken and preventative measures are implemented within CSPI to ensure that each tenant's data is isolated and remains invisible to other tenants.

[0040] In a physical network, a network endpoint ("endpoint") refers to a computing device or system that is connected to a physical network and communicates with the connected network. Network endpoints within a physical network may be connected to a local area network (LAN), a wide area network (WAN), or other types of physical networks. Examples of traditional endpoints within a physical network include modems, hubs, bridges, switches, routers, and other network devices, as well as physical computers (or host machines). Each physical device within a physical network has a fixed network address that can be used to communicate with the device. This fixed network address can be a Layer 2 address (e.g., a MAC address), a fixed Layer 3 address (e.g., an IP address), etc. In a virtual environment or virtual network, endpoints can include various virtual endpoints, such as virtual machines hosted by components of a physical network (e.g., hosted by a physical host machine). These endpoints within a virtual network are addressed by overlay addresses, such as overlay Layer 2 addresses (e.g., overlay MAC addresses) and overlay Layer 3 addresses (e.g., overlay IP addresses). Network overlays enable flexibility by allowing network administrators to move overlay addresses between network endpoints using software management (e.g., by software implementing the control plane of the virtual network). Therefore, unlike physical networks, in virtual networks, overlay addresses (e.g., overlay IP addresses) can be moved from one endpoint to another using network management software. Because virtual networks are built on top of physical networks, communication between components within a virtual network involves both the virtual network and the underlying physical network.To facilitate such communication, CSPI components are configured to learn and store mappings that map overlay addresses in the virtual network to actual physical addresses in the underlying network, and vice versa. These mappings are then used to facilitate communication. Customer traffic is encapsulated to facilitate routing within the virtual network.

[0041] Therefore, physical addresses (e.g., physical IP addresses) are associated with components within a physical network, while overlay addresses (e.g., overlay IP addresses) are associated with entities within a virtual network or overlay network. A physical IP address is an IP address associated with a physical device (e.g., a network device) within an underlying network or physical network. For example, each NVD has an associated physical IP address. An overlay IP address is an overlay address associated with an entity within an overlay network, such as being associated with a compute instance within a customer's VCN. Two different customers or tenants, each having their own private VCN, may be able to use the same overlay IP address within their VCNs without knowing anything about each other. Both physical IP addresses and overlay IP addresses are types of real IP addresses. These IP addresses exist separately from virtual IP addresses. A virtual IP address is typically a single IP address that represents or maps to multiple real IP addresses. A virtual IP address provides a one-to-many mapping between a virtual IP address and multiple real IP addresses. For example, a load balancer may use a VIP to map to or represent multiple servers, each server having its own actual IP address.

[0042] A CSPI is physically hosted in one or more data centers in one or more regions worldwide. A CSPI may include components within a physical network or underlying network, and virtual components (e.g., virtual networks, compute instances, virtual machines, etc.) in a virtual network built on top of the physical network components. In some embodiments, a CSPI is organized and hosted within realms, regions, and availability domains. A region is typically a localized geographical area containing one or more data centers. Regions are generally independent of each other and may be separated by vast distances, for example, across multiple countries or continents. For example, one region may be in Australia, another in Japan, and yet another in India. CSPI resources are partitioned across regions such that each region contains an independent subset of CSPI resources within that region itself. Each region may provide a set of core infrastructure services and resources, such as compute resources (e.g., bare metal servers, virtual machines, containers, and associated infrastructure), storage resources (e.g., block volume storage, file storage, object storage, archive storage), network resources (e.g., VCN, load balancing resources, connectivity to on-premises networks), database resources, edge network resources (e.g., DNS), and access management and monitoring resources. Each region generally has multiple paths to connect to other regions within the realm.

[0043] Because using nearby resources is faster than using distant resources, applications are generally deployed within the region where they are most frequently used (i.e., on the infrastructure associated with that region). Applications may also be deployed across different regions for various reasons, such as redundancy to mitigate the risk of region-wide events like major weather systems or earthquakes, to meet changing requirements regarding legal jurisdictions, tax areas, and other business or social standards.

[0044] Data centers within a region can be further organized and subdivided into availability domains (ADs). An availability domain may correspond to one or more data centers located within a region. A region may consist of one or more availability domains. In such a distributed environment, CSPI resources may be region-specific, such as VCNs, or availability domain-specific, such as compute instances.

[0045] Active Directory (AD) instances within a region are configured to be isolated from each other, fault-tolerant, and extremely unlikely to fail simultaneously. This is achieved by ADs that do not share critical infrastructure resources such as networks, physical cables, cable routes, and cable entry points, so that a failure in one AD within a region is unlikely to affect the availability of other ADs within the same region. ADs within the same region may be connected to each other by low-latency, high-bandwidth networks, enabling the provision of high-availability connectivity to other networks (e.g., the internet, customer on-premises networks) and the construction of replicated systems across multiple ADs for both high availability and disaster recovery. Cloud services use multiple ADs to ensure high availability and protect against resource failures. As the infrastructure provided by the IaaS provider grows, more regions and ADs with additional capacity may be added. Traffic between availability domains is typically encrypted.

[0046] In one embodiment, regions are grouped into realms. A realm is a logical collection of regions. Realms are isolated from each other and do not share any data. Regions within the same realm may communicate with each other, but regions in different realms cannot. A customer's tenancy or account, along with the CSP, resides within a single realm and may be distributed across one or more regions belonging to that realm. Typically, when a customer subscribes to an IaaS service, a tenancy or account for that customer is created within a region specified by the customer within a realm (referred to as the "home" region). A customer can extend their tenancy across one or more other regions within a realm. A customer cannot access regions that are not in the realm where their tenancy resides.

[0047] An IaaS provider can offer multiple realms, each catering to the specific needs of a particular set of customers or users. For example, a commercial realm may be offered to commercial customers. Another example is a realm offered to a specific country for customers within that country. Yet another example is a government realm offered to a government, for example. A government realm may cater to the specific needs of a particular government and may have a higher level of security than a commercial realm. For example, OCI currently offers two realms: one for commercial regions and another for government cloud regions (e.g., FedRAMP certified and IL5 certified).

[0048] In some embodiments, an Active Directory (AD) may be subdivided into one or more fault domains. A fault domain is a group of infrastructure resources within the AD to provide anti-affinity. Fault domains enable the distribution of compute instances so that multiple compute instances do not reside on the same physical hardware within a single AD. This distribution is known as anti-affinity. A fault domain refers to a set of hardware components (computers, switches, etc.) that share a single point of failure. The compute pool is logically divided into fault domains. Therefore, a hardware failure or maintenance event on compute hardware affecting one fault domain does not affect instances in other fault domains. Depending on the embodiment, the number of fault domains per AD may vary. For example, in some embodiments, each AD contains three fault domains. Fault domains function as logical data centers within the AD.

[0049] When a customer subscribes to an IaaS service, resources from CSPI are provisioned for the customer and associated with the customer's tenancy. The customer can use these provisioned resources to build private networks and deploy resources to these networks. A customer's network hosted in the cloud by CSPI is called a VCN. A customer can configure one or more VCNs using the CSPI resources allocated to them. A VCN is a virtual private network or software-defined private network. Customer resources deployed within a customer's VCN can include compute instances (e.g., virtual machines, bare metal instances) and other resources. These compute instances may represent various customer workloads, such as applications, load balancers, and databases. Compute instances deployed in a VCN can communicate with publicly accessible endpoints ("public endpoints") via public networks such as the internet, with other instances within the same VCN or other VCNs (e.g., other VCNs of the customer, or VCNs not belonging to the customer), with the customer's on-premises data centers or networks, and with service endpoints and other types of endpoints.

[0050] A CSP may use CSPI to provide various services. In some cases, a CSPI customer may act as a service provider themselves and provide services using CSPI resources. A service provider may expose service endpoints characterized by identification information (e.g., IP address, DNS name, and DNS port). A customer's resources (e.g., compute instances) can consume a particular service by accessing the service endpoint exposed by the service for that particular service. These service endpoints are generally publicly accessible by users using the public IP address associated with the endpoint over a public communication network such as the internet. Publicly accessible network endpoints are sometimes called public endpoints.

[0051] In one embodiment, a service provider may expose a service through an endpoint for the service (sometimes called a service endpoint). Customers of the service can then access the service using this service endpoint. In one implementation, the service endpoint provided for a service may be accessible to multiple customers who wish to consume that service. In another implementation, a dedicated service endpoint may be provided to a customer, allowing only that customer to access the service using that dedicated service endpoint.

[0052] In one embodiment, when a VCN is created, it is associated with a private overlay classless inter-domain routing (CIDR) address space, which is a range of private overlay IP addresses assigned to the VCN (e.g., 10.0 / 16). The VCN includes associated subnets, route tables, and gateways. The VCN resides within a single region but can span one, more, or all of the region's availability domains. A gateway is a virtual interface configured for the VCN that enables communication of traffic to and from the VCN and one or more endpoints outside the VCN. One or more different types of gateways may be configured for the VCN to enable communication with different types of endpoints.

[0053] A VCN can be subdivided into one or more subnets or subnets. Therefore, a subnet is a unit of configuration or subdivision that can be created within a VCN. A VCN can contain one or more subnets. Each subnet within a VCN is associated with a contiguous range of overlay IP addresses (e.g., 10.0.0.0 / 24 and 10.0.1.0 / 24) that represents a subset of the address space within the VCN's address space and does not overlap with other subnets within that VCN.

[0054] Each compute instance is associated with a virtual network interface card (VNIC) that allows the compute instance to join a subnet in the VCN. A VNIC is a logical representation of a physical network interface card (NIC). Generally, a VNIC is the interface between an entity (e.g., compute instance, service) and a virtual network. A VNIC resides in a subnet and has one or more associated IP addresses and associated security rules or security policies. A VNIC is equivalent to a Layer 2 port on a switch. A VNIC is connected to a compute instance and connects to a subnet within the VCN. The VNIC associated with a compute instance enables the compute instance to become part of a subnet in the VCN and allows the compute instance to communicate (e.g., send and receive packets) with endpoints on the same subnet as the compute instance, endpoints in different subnets within the VCN, or endpoints outside the VCN. Thus, the VNIC associated with a compute instance determines how the compute instance connects to endpoints inside and outside the VCN. A VNIC for a compute instance is created and associated with that compute instance when the compute instance is created and added to a subnet within the VCN. For a subnet containing a set of compute instances, the subnet contains a VNIC corresponding to the set of compute instances, and each VNIC connects to one of the compute instances in the set.

[0055] Each compute instance is assigned a private overlay IP address via the VNIC associated with it. This private overlay IP address is assigned to the VNIC associated with the compute instance when the compute instance is created and is used to route traffic to and from the compute instance. All VNICs within a given subnet use the same route table, security lists, and DHCP options. As mentioned earlier, each subnet within a VCN is associated with a contiguous range of overlay IP addresses (e.g., 10.0.0.0 / 24 and 10.0.1.0 / 24) that represents a subset of the address space within the VCN's address space and does not overlap with other subnets within that VCN. For a VNIC on a particular subnet of a VCN, the private overlay IP address assigned to the VNIC is an address from the contiguous range of overlay IP addresses assigned to that subnet.

[0056] In one embodiment, a compute instance may optionally be assigned additional overlay IP addresses in addition to its private overlay IP address, such as one or more public IP addresses if it is in a public subnet. These multiple addresses may be assigned to the same VNIC or across multiple VNICs associated with the compute instance. However, each instance has a primary VNIC, which is created during instance startup and associated with the overlay private IP address assigned to the instance, and this primary VNIC cannot be removed. Additional VNICs, called secondary VNICs, may be added to an existing instance in the same availability domain as the primary VNIC. All VNICs are in the same availability domain as the instance. Secondary VNICs may be in a subnet within the same VCN as the primary VNIC, or in a different subnet that is either in the same VCN or a different VCN.

[0057] If a compute instance is located in a public subnet, it may optionally be assigned a public IP address. When a subnet is created, it can be designated as either a public or private subnet. A private subnet means that resources within the subnet (e.g., compute instances) and associated VNICs cannot have public overlay IP addresses. A public subnet means that resources within the subnet and associated VNICs can have public IP addresses. Customers can specify subnets to reside within a single availability domain or across multiple availability domains within a region or realm.

[0058] As mentioned above, a VCN may be subdivided into one or more subnets. In one embodiment, a virtual router (VR) configured for the VCN (referred to as a VCN VR or simply a VR) enables communication between subnets within the VCN. For subnets within a VCN, the VR represents the logical gateway for that subnet, allowing the subnet (i.e., compute instances on that subnet) to communicate with endpoints on other subnets within the VCN and with other endpoints outside the VCN. A VCN VR is a logical entity configured to route traffic between VNICs within the VCN and virtual gateways ("gateways") associated with the VCN. A VCN VR is a Layer 3 / IP layer concept. In one embodiment, there is one VCN VR for one VCN, and the VCN VR has an unlimited number of ports, possibly addressed by IP addresses, with one port for each subnet in the VCN. Thus, the VCN VR has a different IP address for each subnet within the VCN to which the VCN VR is connected. The VR is also connected to various gateways configured for the VCN. In one embodiment, a specific overlay IP address from a subnet's overlay IP address range is reserved for a port on the VCN VR for that subnet. For example, consider a VCN that includes two subnets, each having the associated address ranges 10.0 / 16 and 10.1 / 16, respectively. For a first subnet in the VCN having the address range 10.0 / 16, an address from this range is reserved for a port on the VCN VR for that subnet. In some cases, the first IP address from this range may be reserved for the VCN VR. For example, for a subnet having the overlay IP address range 10.0 / 16, the IP address 10.0.0.1 may be reserved for a port on the VCN VR for that subnet. For a second subnet in the same VCN having the address range 10.1 / 16, the VCN VR may have a port on that second subnet having the IP address 10.1.0.1.A VCN VR has a different IP address for each subnet within the VCN.

[0059] In some other embodiments, each subnet within a VCN may include a VR associated with itself, which is addressable by the subnet using a reserved IP address or a default IP address associated with the VR. The reserved IP address or default IP address may be, for example, the first IP address from the range of IP addresses associated with that subnet. A VNIC within a subnet can communicate with the VR associated with the subnet using this default IP address or reserved IP address (e.g., send and receive packets). In such embodiments, the VR is the input / output point of that subnet. A VR associated with a subnet within a VCN can communicate with other VRs associated with other subnets within the VCN. A VR can also communicate with a gateway associated with the VCN. The VR functionality of a subnet operates on or is performed by one or more NVDs that are performing the VNIC functionality of the VNICs within the subnet.

[0060] Route tables, security rules, and DHCP options may be configured for the VCN. The route table is the VCN's virtual route table and contains rules for routing traffic from subnets within the VCN to destinations outside the VCN via a gateway or specially configured instance. The VCN's route table can be customized to control how packets are forwarded / routed to and from the VCN. DHCP options refer to configuration information that is automatically provided to an instance when it starts up.

[0061] Security rules configured for a VCN represent the VCN's overlay firewall rules. Security rules include ingress and egress rules and can specify the types of traffic (e.g., based on protocol and port) allowed to enter and exit instances within the VCN. Customers can choose whether specific rules are stateful or stateless. For example, a customer can allow incoming SSH traffic from any location to a set of instances by configuring a stateful ingress rule with source CIDR 0.0.0.0 / 0 and destination TCP port 22. Security rules can be implemented using network security groups or security lists. A network security group consists of a set of security rules that apply only to resources within that group. A security list, on the other hand, contains rules that apply to all resources within any subnet using the security list. A VCN may have a default security list containing default security rules. DHCP options configured for a VCN provide configuration information that is automatically provided to instances within the VCN when they start up.

[0062] In one embodiment, VCN configuration information is determined and stored by the VCN control plane (CP). VCN configuration information may include, for example, information about address ranges associated with the VCN, subnets within the VCN and associated information, one or more VRs associated with the VCN, compute instances within the VCN and associated VNICs, NVDs (e.g., VNICs, VRs, gateways) that perform various virtualized network functions associated with the VCN, VCN status information, and other VCN-related information. In one embodiment, a VCN distribution service exposes the configuration information stored by the VCN CP or a portion thereof to the NVD. The distributed information may be used to update information stored and used by the NVD to forward packets to and from compute instances within the VCN (e.g., forwarding tables, routing tables, etc.).

[0063] In one embodiment, the creation of VCNs and subnets is handled by the VCN CP, and the startup of compute instances is handled by the compute CP. The compute CP is responsible for allocating physical resources to compute instances, and then calls the VCN CP to create VNICs and connect them to the compute instances. The VCN CP also sends VCN data mappings to the VCN data plane, which is configured to perform packet forwarding and routing functions. In one embodiment, the VCN CP provides a distribution service, which is responsible for providing updates to the VCN data plane.

[0064] A customer may create one or more VCNs using resources hosted by CSPI. Compute instances deployed in a customer's VCN may communicate with various endpoints. These endpoints may include endpoints hosted by CSPI and endpoints outside of CSPI.

[0065] Various different architectures for implementing cloud-based services using CSPI are shown in Figures 1–12 and 15–19, and are described below. Figure 1 is a high-level diagram of a distributed environment 100 showing an overlay VCN or customer VCN hosted by CSPI according to one embodiment. The distributed environment shown in Figure 1 includes multiple components within the overlay network. The distributed environment 100 shown in Figure 1 is merely an example and is not intended to unduly limit the scope of the claimed embodiments. Many variations, alternatives, and modifications are possible. For example, in some implementations, the distributed environment shown in Figure 1 may include more or fewer systems or components than those shown in Figure 1, may combine two or more systems, or may include different configurations or arrangements of systems.

[0066] As shown in the example in Figure 1, the distributed environment 100 includes a CSPI 101 that provides services and resources that customers can subscribe to and use to build their own VCNs. In one embodiment, the CSPI 101 provides IaaS services to customers who are subscribing. The data centers within the CSPI 101 may be organized into one or more regions. One exemplary region, "Region US" 102, is shown in Figure 1. The customer has configured an Oracle International Corporation customer VCN with respect to Region 102. The customer may deploy various compute instances to the VCN 104, which may include virtual machines or bare metal instances. Examples of instances include applications, databases, load balancers, etc.

[0067] In the embodiment shown in Figure 1, the customer's VCN 104 includes two subnets, namely "Subnet 1" and "Subnet 2," each subnet having its own CIDR IP address range. In Figure 1, the overlay IP address range for Subnet 1 is 10.0 / 16, and the address range for Subnet 2 is 10.1 / 16. The VCN virtual router 105 represents the logical gateway of the VCN, enabling communication between subnets of VCN 104 and with other endpoints outside the VCN. The VCN VR 105 is configured to route traffic between VNICs within VCN 104 and gateways associated with VCN 104. The VCN VR 105 provides a port for each subnet of VCN 104. For example, the VCN VR 105 may provide a port with the IP address 10.0.0.1 for Subnet 1 and a port with the IP address 10.1.0.1 for Subnet 2.

[0068] Multiple compute instances may be deployed in each subnet, and compute instances can be virtual machine instances and / or bare metal instances. Compute instances within a subnet may be hosted by one or more host machines within CSPI101. Compute instances join the subnet via the VNIC associated with the compute instance. For example, as shown in Figure 1, compute instance C1 becomes part of subnet 1 via the VNIC associated with this compute instance. Similarly, compute instance C2 becomes part of subnet 1 via the VNIC associated with C2. In a similar manner, multiple compute instances, which may be virtual machine instances or bare metal instances, may become part of subnet 1. Each compute instance is assigned a private overlay IP address and MAC address via the VNIC associated with it. For example, in Figure 1, compute instance C1 has the overlay IP address 10.0.0.2 and the MAC address M1, while compute instance C2 has the private overlay IP address 10.0.0.3 and the MAC address M2. Each compute instance in subnet 1, including compute instances C1 and C2, has a default route to VCN VR105 using IP address 10.0.0.1, which is the IP address of the port of VCN VR105 in subnet 1.

[0069] Multiple compute instances, including virtual machine instances and / or bare metal instances, may be deployed in subnet 2. For example, as shown in Figure 1, compute instances D1 and D2 become part of subnet 2 via the VNICs associated with each compute instance. In the embodiment shown in Figure 1, compute instance D1 has the overlay IP address 10.1.0.2 and the MAC address of MM1, while compute instance D2 has the private overlay IP address 10.1.0.3 and the MAC address of MM2. Each compute instance in subnet 2, including compute instances D1 and D2, has a default route to VCN VR105 using the IP address 10.1.0.1, which is the IP address of the port of VCN VR105 in subnet 2.

[0070] VCN A104 may include one or more load balancers. For example, a load balancer may be provided to a subnet and configured to load balance traffic across multiple compute instances on the subnet. A load balancer may be provided to load balance traffic across multiple subnets within the VCN.

[0071] A specific compute instance deployed on VCN104 can communicate with various endpoints. These endpoints may include endpoints hosted by CSPI200 and endpoints outside of CSPI200. Endpoints hosted by CSPI101 may include endpoints on the same subnet as a particular compute instance (e.g., communication between two compute instances in subnet 1), endpoints on different subnets but within the same VCN (e.g., communication between a compute instance in subnet 1 and a compute instance in subnet 2), endpoints in different VCNs within the same region (e.g., communication between a compute instance in subnet 1 and an endpoint in the same VCN within the same region, and communication between a compute instance in subnet 1 and an endpoint in service network 110 within the same region), or endpoints in VCNs in different regions (e.g., communication between a compute instance in subnet 1 and an endpoint in the same VCN within a different region). A compute instance in a subnet hosted by CSPI101 may communicate with endpoints not hosted by CSPI101 (i.e., outside of CSPI101). These external endpoints include endpoints within the customer's on-premises network 116, endpoints within other remote cloud networks 118, public endpoints 112 accessible via public networks such as the internet, and other endpoints.

[0072] Communication between compute instances on the same subnet is facilitated using VNICs associated with the source and destination compute instances. For example, compute instance C1 in subnet 1 may want to send a packet to compute instance C2, also in subnet 1. For a packet originating from the source compute instance and destined for another compute instance in the same subnet, the packet is first processed by the VNIC associated with the source compute instance. The processing performed by the VNIC associated with the source compute instance may include determining the packet's destination information from the packet header, identifying any policies (e.g., security lists) configured for the VNIC associated with the source compute instance, determining the packet's next hop, performing any packet encapsulation / decapsulation functions as needed, and then forwarding / routing the packet to the next hop to facilitate communication of the packet with its intended destination. If the destination compute instance is in the same subnet as the source compute instance, the VNIC associated with the source compute instance is configured to identify the VNIC associated with the destination compute instance and forward the packet to that VNIC for processing. The VNIC associated with the destination compute instance then executes and forwards the packet to the destination compute instance.

[0073] For packets propagating from a compute instance within a subnet to an endpoint in a different subnet within the same VCN, this communication is facilitated by the VNICs and VCN VRs associated with the source and destination compute instances. For example, if compute instance C1 in subnet 1 in Figure 1 wants to send a packet to compute instance D1 in subnet 2, this packet is first processed by the VNIC associated with compute instance C1. The VNIC associated with compute instance C1 is configured to route the packet to VCN VR105 using the VCN VR's default route or port 10.0.0.1. VCN VR105 is configured to route the packet to subnet 2 using port 10.1.0.1. Next, the packet is received and processed by the VNIC associated with D1, and the VNIC forwards the packet to compute instance D1.

[0074] For packets propagated from compute instances within VCN104 to endpoints outside VCN104, communication is facilitated by the VNIC associated with the source compute instance, VCN VR105, and gateways associated with VCN104. One or more types of gateways may be associated with VCN104. A gateway is an interface between a VCN and another endpoint, which is outside the VCN. A gateway is a Layer 3 / IP layer concept that enables a VCN to communicate with endpoints outside the VCN. Thus, gateways facilitate traffic flow between a VCN and other VCNs or networks. Different types of gateways may be configured for a VCN to facilitate different types of communication with different types of endpoints. Depending on the gateway, communication may traverse a public network (e.g., the Internet) or a private network. Different communication protocols may be used for these communications.

[0075] For example, compute instance C1 may want to communicate with an endpoint outside of VCN104. The packet may first be processed by the VNIC associated with source compute instance C1. This VNIC processing determines that the packet's destination is outside of C1's subnet 1. The VNIC associated with C1 may forward the packet to VCN VR105 of VCN104. VCN VR105 then processes the packet and, as part of this processing, determines a specific gateway associated with VCN104 as the packet's next hop based on the packet's destination. VCN VR105 then may forward the packet to a specific identified gateway. For example, if the destination is an endpoint within a customer's on-premises network, VCN VR105 may forward the packet to DRG gateway 122 configured for VCN104. The packet may then be forwarded from the gateway to the next hop to facilitate the packet's delivery to its final intended destination.

[0076] Various types of gateways may be configured for the VCN. An example of a gateway that may be configured for the VCN is shown in Figure 1 and described below. An example of a gateway associated with the VCN is also shown in Figures 15–18 (for example, gateways referenced by reference numbers 1534, 1536, 1538, 1634, 1636, 1638, 1734, 1736, 1738, 1834, 1836, and 1838) and described below. As shown in the embodiment shown in Figure 1, DRG122 may be added to or associated with the customer's VCN104 to provide a route for private network traffic communication between the customer's VCN104 and another endpoint, which may be the customer's on-premises network 116, VCN108 in a different region of CSPI101, or another remote cloud network 118 not hosted by CSPI101. The customer's on-premises network 116 may be the customer's network built using the customer's resources or the customer's data center. Access to a customer's on-premises network 116 is typically extremely restricted. For a customer who has both their on-premises network 116 and one or more VCNs 104 deployed or hosted in the cloud by CSPI 101, the customer may want their on-premises network 116 and their cloud-based VCNs 104 to be able to communicate with each other. This would allow the customer to build an extended hybrid environment encompassing their VCNs 104 hosted by CSPI 101 and their on-premises network 116. DRG 122 enables this communication. To enable such communication, a communication channel 124 is configured, with one endpoint of this channel located within the customer's on-premises network 116 and the other endpoints located within CSPI 101 and connected to the customer's VCNs 104. The communication channel 124 can be via a public or private communication network, such as the internet.Various communication protocols may be used, such as IPsec VPN technology which uses a public communication network like the Internet, or Oracle's FastConnect technology which uses a private network instead of a public network. A device or equipment within the customer's on-premises network 116 that forms one endpoint of communication channel 124 is called customer premise equipment (CPE), such as CPE126 shown in Figure 1. On the CSPI101 side, the endpoint may be a host machine running DRG122.

[0077] In one embodiment, a Remote Peering Connection (RPC) may be added to the DRG, enabling a customer to peer one VCN with another VCN in a different region. Using such an RPC, a customer's VCN 104 can connect to a VCN 108 in a different region using the DRG 122. The DRG 122 may be used to communicate with other remote cloud networks 118 not hosted by the CSPI 101, such as the Microsoft Azure cloud or the Amazon AWS cloud.

[0078] As shown in Figure 1, an Internet Gateway (IGW) 120 may be configured for a customer's VCN104, enabling compute instances on VCN104 to communicate with a public endpoint 112 accessible via a public network such as the Internet. The IGW 120 is a gateway that connects the VCN to a public network such as the Internet. The IGW 120 enables direct access from public subnets within a VCN, such as VCN104 (resources within public subnets have public overlay IP addresses), to the public endpoint 112 on the public network 114, such as the Internet. Connections can be initiated using the IGW 120 from subnets within VCN104 or from the Internet.

[0079] A Network Address Translation (NAT) gateway 128 is configured for the customer's VCN 104 to enable internet access for cloud resources within the customer's VCN that do not have dedicated public overlay IP addresses. The NAT gateway 128 enables such access without directly exposing those resources to incoming internet connections (e.g., L4-L7 connections). This allows private access from private subnets within the VCN, such as private subnet 1 within VCN 104, to public endpoints on the internet. The NAT gateway can only initiate connections from private subnets to the public internet; it cannot initiate connections from the internet to private subnets.

[0080] In one embodiment, a Service Gateway (SGW) 126 may be configured for a customer's VCN 104, providing a route for private network traffic between the VCN 104 and supported service endpoints within a service network 110. In one embodiment, the service network 110 may be provided by a Customer Service Provider (CSP) and may provide a variety of services. An example of such a service network is Oracle's Services Network, which provides a variety of services that can be used by a customer. For example, compute instances (e.g., database systems) within a private subnet of a customer's VCN 104 can back up data to a service endpoint (e.g., object storage) without requiring a public IP address or internet access. In one embodiment, a VCN may contain only one SGW, and connections may be initiated only from subnets within the VCN and not from the service network 110. If a VCN is peered with another VCN, resources in the other VCN typically cannot access the SGW. Resources in an on-premises network connected to a VCN using FastConnect or VPN connections may also use the service gateway configured for that VCN.

[0081] In one implementation, SGW126 uses the concept of service-classless inter-domain routing (CIDR) labels, which are strings representing the public IP address range for all regions of the target service or group of services. Customers use service CIDR labels when configuring SGW and associated route rules to control traffic to their services. Optionally, customers can also utilize service CIDR labels when configuring security rules, eliminating the need to adjust those security rules if the public IP addresses of their services change in the future.

[0082] A Local Peering Gateway (LPG) 132 is a gateway that can be added to a customer's VCN 104, enabling VCN 104 to peer with another VCN within the same region. Peering means that VCNs communicate using private IP addresses without traffic traversing a public network such as the internet or routing traffic through the customer's on-premises network 116. In a preferred embodiment, a VCN includes a separate LPG for each peering it establishes. Local peering, or VCN peering, is a common method used to establish network connectivity between different application or infrastructure management functions.

[0083] Service providers, such as service providers within service network 110, may provide access to their services using various access models. According to the public access model, a service may be exposed as a public endpoint, which may be publicly accessible by compute instances in the customer's VCN via a public network such as the internet, and / or privately accessible via SGW126. According to a particular private access model, a service may be made accessible as a private IP endpoint in a private subnet within the customer's VCN. This access is called private endpoint (PE) access and allows service providers to expose their services as instances in the customer's private network. A private endpoint resource represents a service within the customer's VCN. Each PE appears as a VNIC (called a PE-VNIC, which has one or more private IPs) in a subnet selected by the customer within the customer's VCN. Thus, a PE provides a way to present a service in a private subnet of the customer's VCN using a VNIC. Because the endpoint is exposed as a VNIC, all features associated with the VNIC, such as routing rules and security lists, are available to the PE VNIC.

[0084] A service provider can register a service to enable access via a Public Access Point (PE). The provider can associate policies with the service that restrict its visibility to customer tenancies. A provider can register multiple services under a single virtual IP address (VIP), especially in the case of multi-tenant services. Multiple such private endpoints (within multiple VCNs) representing the same service may exist.

[0085] Next, compute instances within a private subnet can access the service using the private IP address of the PE VNIC or the DNS name of the service. Compute instances within a customer's VCN can access the service by sending traffic to the private IP address of the PE within the customer's VCN. The Private Access Gateway (PAGW) 130 is a gateway resource that can be connected to the service provider's VCN (e.g., a VCN within service network 110) and acts as the input / output point for all traffic to and from the private endpoints in the customer's subnet. The PAGW 130 allows the provider to scale the number of PE connections without utilizing internal IP address resources. The provider only needs to configure one PAGW for any number of services registered within a single VCN. The provider can represent a service as a private endpoint in multiple VCNs for one or more customers. From the customer's perspective, the PE VNIC appears to be connected to the service the customer wishes to interact with, rather than to the customer's instance. Traffic destined for the private endpoint is routed to the service via the PAGW 130. These are called customer-to-service (C2S) connections.

[0086] The PE concept can also be used to extend private access to a service to the customer's on-premises network and data center by allowing traffic to flow through FastConnect / IPsec links and private endpoints within the customer's VCN. Private access to the service can also be extended to the customer's peered VCN by allowing traffic to flow between LPG132 and the PE within the customer's VCN.

[0087] Customers can control routing within their VCN at the subnet level, allowing them to specify which subnets within their VCN, such as VCN104, use which gateways. The VCN's route table is used to determine whether traffic is allowed to exit the VCN through a particular gateway. For example, in a specific example, the route table for a public subnet within a customer's VCN104 may allow non-local traffic to be sent via IGW120. The route table for a private subnet within the same customer's VCN104 may allow traffic destined for CSP services to be sent via SGW126. All remaining traffic may be sent via NAT gateway 128. The route table controls only traffic leaving the VCN.

[0088] Security lists associated with a VCN are used to control traffic entering the VCN via a gateway through inbound connections. All resources within a subnet use the same route table and security lists. Security lists may be used to control specific types of traffic that are allowed to enter and leave instances within a VCN subnet. Security list rules may include inbound and outbound rules. For example, inbound rules may specify allowed source address ranges, while outbound rules may specify allowed destination address ranges. Security rules may specify specific protocols (e.g., TCP, ICMP), specific ports (e.g., 22 for SSH, 3389 for Windows RDP), etc. In some implementations, the instance's operating system may enforce its own firewall rules that match the security list rules. Rules may be stateful (e.g., connections are tracked and responses are automatically allowed without using explicit security list rules for response traffic) or stateless.

[0089] Access from a customer's VCN (i.e., by resources or compute instances deployed on VCN104) can be classified as public access, private access, or dedicated access. Public access refers to an access model in which a public IP address or NAT is used to access a public endpoint. Private access allows a customer's workload within VCN104 with a private IP address (e.g., resources in a private subnet) to access a service without traversing a public network such as the internet. In one embodiment, CSPI101 allows a customer's VCN workload with a private IP address to access a service (or its public service endpoint) using a service gateway. Thus, the service gateway provides a private access model by establishing a virtual link between the customer's VCN and the public endpoint of the service that resides outside the customer's private network.

[0090] Furthermore, CSPI may provide dedicated public access technology using technologies such as FastConnect public peering, which allows a customer's on-premises instances to access one or more services within the customer's VCN using FastConnect connectivity without traversing public networks such as the internet. CSPI may also provide dedicated private access using FastConnect private peering, which allows a customer's on-premises instances with private IP addresses to access workloads in the customer's VCN using FastConnect connectivity. FastConnect is a network connectivity alternative to using the public internet for connecting a customer's on-premises network to CSPI and its services. FastConnect provides an easy, resilient, and economical way to create dedicated private connectivity with higher bandwidth options and a more reliable and consistent network experience compared to internet-based connectivity.

[0091] Figure 1 and the accompanying explanation above illustrate the various virtual components within an exemplary virtual network. As previously mentioned, the virtual network is built on top of an underlying physical network or infrastructure network. Figure 2 shows a simplified architectural diagram of the physical components in the physical network within the underlying CSPI200 for the virtual network, according to one embodiment. As shown in the figure, the CSPI200 provides a distributed environment that includes components and resources (e.g., compute resources, memory resources, and network resources) provided by the CSP. These components and resources are used to deliver cloud services (e.g., IaaS services) to subscribers, i.e., customers who subscribe to one or more services provided by the CSP. Based on the services subscribed by the customer, a subset of the CSPI200's resources (e.g., compute resources, memory resources, and network resources) is provisioned for the customer. The customer can then use the physical compute resources, memory resources, and network resources provided by the CSPI200 to build their own cloud-based (i.e., CSPI-hosted) customizable private virtual network. As already shown, these customer networks are called VCNs. Customers can deploy one or more customer resources, such as compute instances, to these customers' VCNs. Compute instances can take the form of virtual machines, bare metal instances, etc. CSPI200 provides infrastructure and a suite of complementary cloud services that enable customers to build and run a wide range of applications and services in a highly available hosted environment.

[0092] In the embodiment shown in Figure 2, the physical components of the CSPI200 include one or more physical host machines or physical servers (e.g., 202, 206, 208), network virtualization devices (NVDs) (e.g., 210, 212), ToR switches (e.g., 214, 216), and a physical network (e.g., 218), and switches within the physical network 218. The physical host machines or physical servers may host and run various compute instances participating in one or more subnets of the VCN. Compute instances may include virtual machine instances and bare metal instances. For example, the various compute instances shown in Figure 1 may be hosted by the physical host machines shown in Figure 2. Virtual machine compute instances in the VCN may run by one host machine or by several different host machines. The physical host machines may host virtual host machines, container-based hosts or functions, etc. The VNIC and VCN VR shown in Figure 1 may run by the NVDs shown in Figure 2. The gateway shown in Figure 1 may be run by the host machine and / or by the NVD shown in Figure 2.

[0093] A host machine or server may run a hypervisor (also known as a virtual machine monitor or VMM) that creates and activates virtual environments on the host machine. Virtualization or virtual environments facilitate cloud-based computing. On a host machine, one or more compute instances may be created, run, and managed by the hypervisor on that host machine. The hypervisor on the host machine enables the sharing of the host machine's set of physical computing resources (e.g., compute resources, memory resources, and network resources) among the various compute instances running on the host machine.

[0094] For example, as shown in Figure 2, host machines 202 and 208 run hypervisors 260 and 266, respectively. These hypervisors may be implemented using software, firmware, hardware, or a combination thereof. Typically, a hypervisor is a process or software layer that resides on top of the host machine's operating system (OS), which runs on the host machine's hardware processors. A hypervisor provides a virtual environment by enabling the sharing of the host machine's set of physical computing resources (e.g., processing resources such as processors / cores, memory resources, and network resources) among various virtual machine computing instances run by the host machine. For example, in Figure 2, hypervisor 260 may reside on top of the host machine 202's OS, enabling the sharing of the host machine 202's set of computing resources (e.g., processing resources, memory resources, and network resources) among computing instances (e.g., virtual machines) run by the host machine 202. A virtual machine may have its own operating system (called a guest operating system), which may be the same as or different from the host machine's OS. The operating system of a virtual machine running on a host machine may be the same as or different from the operating system of another virtual machine running on the same host machine. Therefore, a hypervisor allows multiple operating systems to run in parallel with each other while sharing the same set of computing resources on the host machine. The host machines shown in Figure 2 may have the same or different types of hypervisors.

[0095] Compute instances can be virtual machine instances or bare metal instances. In Figure 2, compute instance 268 on host machine 202 and compute instance 274 on host machine 208 are examples of virtual machine instances. Host machine 206 is an example of a bare metal instance provided to a customer.

[0096] In one example, an entire host machine may be provisioned to a single customer, and all one or more compute instances (either virtual machines or bare metal instances) hosted by that host machine belong to that same customer. In another example, a host machine may be shared among multiple customers (i.e., multiple tenants). In such a multi-tenancy scenario, the host machine may host virtual machine compute instances belonging to different customers. These compute instances may be members of different VCNs of different customers. In one embodiment, bare metal compute instances are hosted by bare metal servers that do not have a hypervisor. When bare metal compute instances are provisioned, a single customer or tenant maintains control of the physical CPU, memory, and network interfaces of the host machine hosting the bare metal instances, and the host machine is not shared with other customers or tenants.

[0097] As mentioned above, each compute instance that is part of a VCN is associated with a VNIC that enables the compute instance to be a member of the VCN's subnet. The VNIC associated with a compute instance facilitates the communication of packets or frames to and from the compute instance. The VNIC is associated with a compute instance when the compute instance is created. In one embodiment, for a compute instance run by a host machine, the VNIC associated with that compute instance is run by an NVD connected to the host machine. For example, in Figure 2, host machine 202 runs a virtual machine compute instance 268 associated with VNIC 276, and VNIC 276 is run by an NVD 210 connected to host machine 202. In another example, a bare metal instance 272 hosted by host machine 206 is associated with VNIC 280, which is run by an NVD 212 connected to host machine 206. In yet another example, VNIC 284 is associated with compute instance 274 run by host machine 208, and VNIC 284 is run by an NVD 212 connected to host machine 208.

[0098] For compute instances hosted by a host machine, the NVDs connected to that host machine also run the VCN VRs corresponding to the VCNs to which those compute instances are members. For example, in the embodiment shown in Figure 2, NVD210 runs VCN VR277 corresponding to the VCN to which compute instance 268 is a member. NVD212 may run one or more VCN VR283 corresponding to the VCNs to which compute instances hosted by host machines 206 and 208 are associated.

[0099] The host machine may include one or more network interface cards (NICs) that enable the host machine to connect to other devices. The NICs on the host machine may provide one or more ports (or interfaces) that enable the host machine to connect communicatively to another device. For example, the host machine may be connected to the NVD using one or more ports (or interfaces) provided on the host machine and the NVD. The host machine may also be connected to other devices, such as another host machine.

[0100] For example, in Figure 2, host machine 202 is connected to NVD210 using a link 220 extending between port 234 provided by NIC 232 of host machine 202 and port 236 of NVD210. Host machine 206 is connected to NVD212 using a link 224 extending between port 246 provided by NIC 244 of host machine 206 and port 248 of NVD212. Host machine 208 is connected to NVD212 using a link 226 extending between port 252 provided by NIC 250 of host machine 208 and port 254 of NVD212.

[0101] Next, the NVDs are connected via communication links to ToR switches (also called switch fabrics) connected to the physical network 218. In some embodiments, the links between the host machines and the NVDs, and between the NVDs and the ToR switches, are Ethernet® links. For example, in Figure 2, links 228 and 230 are used to connect NVDs 210 and 212 to ToR switches 214 and 216, respectively. In some embodiments, links 220, 224, 226, 228, and 230 are Ethernet links. The collection of host machines and NVDs connected to the ToR is sometimes referred to as a rack.

[0102] The physical network 218 provides a communication fabric that allows the ToR switches to communicate with each other. The physical network 218 can be a multi-layer network. In one implementation, the physical network 218 is a multi-layer Clos network of switches, and the ToR switches 214 and 216 represent leaf-level nodes in a multi-layer multi-node physical switching network 218. Various Clos network configurations are possible, including but not limited to 2-layer, 3-layer, 4-layer, 5-layer networks, and generally "n"-layer networks. An example of a Clos network is shown in Figure 5 and described below.

[0103] Various connection configurations are possible between the host machine and the NVD, including one-to-one, many-to-one, and one-to-many configurations. In a one-to-one configuration, each host machine is connected to its own separate NVD. For example, in Figure 2, host machine 202 is connected to NVD210 via host machine 202's NIC232. In a many-to-one configuration, multiple host machines are connected to a single NVD. For example, in Figure 2, host machines 206 and 208 are connected to the same NVD212, respectively, via NIC244 and 250.

[0104] In a one-to-many configuration, one host machine is connected to multiple NVDs. Figure 3 shows an example within CSPI300 where a host machine is connected to multiple NVDs. As shown in Figure 3, the host machine 302 has a network interface card (NIC) 304 with multiple ports 306 and 308. The host machine 300 is connected to a first NVD 310 via port 306 and link 320, and to a second NVD 312 via port 308 and link 322. Ports 306 and 308 may be Ethernet ports, and links 320 and 322 between the host machine 302 and the NVDs 310 and 312 may be Ethernet links. Next, the NVD 310 is connected to a first ToR switch 314, and the NVD 312 is connected to a second ToR switch 316. The links between the NVDs 310 and 312 and the ToR switches 314 and 316 may be Ethernet links. ToR switches 314 and 316 represent Layer 0 switching devices in a multilayer physical network 318.

[0105] The configuration shown in Figure 3 provides two separate physical network paths to the host machine 302 across the physical switch network 318: a first path that crosses the ToR switch 314, passes through the NVD 310, and leads to the host machine 302, and a second path that crosses the ToR switch 316, passes through the NVD 312, and leads to the host machine 302. These separate paths result in improved availability (referred to as high availability) of the host machine 302. If there is a problem with one of the paths (e.g., one link in the path fails) or a device (e.g., a particular NVD is not functioning), the other path may be used for communication with the host machine 302.

[0106] In the configuration shown in Figure 3, the host machine connects to two different NVDs using two different ports provided by the host machine's NIC. In other embodiments, the host machine may include multiple NICs that enable the host machine to connect to multiple NVDs.

[0107] Referring again to Figure 2, the NVD is a physical device or component that performs one or more network and / or storage virtualization functions. The NVD may be any device having one or more processing units (e.g., a CPU, Network Processing Units (NPUs), FPGA, packet processing pipeline, etc.), memory including a cache, and ports. Various virtualization functions may be performed by software / firmware running on one or more processing units of the NVD.

[0108] NVDs can be implemented in various forms. For example, in one embodiment, an NVD is implemented as an interface card called a smart NIC or intelligent NIC that incorporates an embedded processor. A smart NIC is a separate device from the NIC on the host machine. In Figure 2, NVDs 210 and 212 may be implemented as smart NICs connected to host machine 202, and to host machines 206 and 208, respectively.

[0109] However, a smart NIC is merely one example of an NVD implementation. Various other implementations are possible. For example, in some other implementations, the NVD or one or more functions performed by the NVD may be incorporated into or performed by one or more host machines, one or more ToR switches, and other components of the CSPI200. For example, the NVD may be embodied in a host machine, and the functions performed by the NVD are performed by the host machine. Another example is that the NVD may be part of a ToR switch, or the ToR switch may be configured to perform the functions performed by the NVD, thereby enabling the ToR switch to perform various complex packet translations used in public clouds. A ToR that performs the functions of the NVD is sometimes called a smart ToR. In yet another implementation where virtual machine (VM) instances rather than bare metal (BM) instances are provided to customers, the functions performed by the NVD may be implemented within the hypervisor of the host machine. In some other implementations, some of the functions of the NVD may be offloaded to a centralized service running on a set of host machines.

[0110] In some embodiments, such as when implemented as a smart NIC as shown in Figure 2, the NVD may have multiple physical ports that enable the NVD to connect to one or more host machines and one or more ToR switches. The ports on the NVD may be classified as host-facing ports (also called “south ports”) or network-facing or ToR-facing ports (also called “north ports”). Host-facing ports on the NVD are the ports used to connect the NVD to a host machine. An example of host-facing ports in Figure 2 includes port 236 on the NVD210, and ports 248 and 254 on the NVD212. Network-facing ports on the NVD are the ports used to connect the NVD to a ToR switch. An example of network-facing ports in Figure 2 includes port 256 on the NVD210 and port 258 on the NVD212. As shown in Figure 2, the NVD210 connects to the ToR switch 214 using a link 228 extending from port 256 on the NVD210 to the ToR switch 214. Similarly, the NVD212 is connected to the ToR switch 216 using a link 230 that extends from port 258 of the NVD212 to the ToR switch 216.

[0111] The NVD may receive packets and frames from the host machine via its host-facing port (for example, packets and frames generated by compute instances hosted by the host machine), perform the necessary packet processing, and then forward those packets and frames to the ToR switch via the NVD's network-facing port. The NVD may also receive packets and frames from the ToR switch via its network-facing port, perform the necessary packet processing, and then forward those packets and frames to the host machine via the NVD's host-facing port.

[0112] In one embodiment, there may be multiple ports and associated links between the NVD and the ToR switch. These ports and links may be aggregated to form a link aggregator group (LAG) of multiple ports or links. Link aggregation allows multiple physical links between two endpoints (e.g., between the NVD and the ToR switch) to be treated as a single logical link. All physical links within a given LAG may operate in full-duplex mode at the same speed. LAGs help increase bandwidth and improve the reliability of the connection between the two endpoints. If one of the physical links in the LAG fails, traffic is dynamically and transparently reassigned to one of the other physical links in the LAG. Aggregated physical links provide higher bandwidth than individual links. Multiple ports associated with a LAG are treated as a single logical port. Traffic can be load-balanced across the multiple physical links of the LAG. One or more LAGs may be configured between two endpoints. The two endpoints may be between the NVD and the ToR switch, between a host machine and the NVD, etc.

[0113] NVD implements or performs network virtualization functions. These functions are performed by software / firmware run by NVD. Examples of network virtualization functions include, but are not limited to, packet encapsulation and decapsulation functions, functions for creating VCN networks, functions for enforcing network policies such as VCN security list (firewall) functions, and functions for facilitating the routing and forwarding of packets to and from compute instances within the VCN. In one embodiment, upon receiving a packet, NVD is configured to run a packet processing pipeline to process the packet and determine how the packet should be forwarded or routed. As part of this packet processing pipeline, NVD may run one or more virtual functions associated with the overlay network, such as running a VNIC associated with compute instances within the VCN, running a virtual router (VR) associated with the VCN, encapsulating and decapsulating packets to facilitate forwarding or routing within the virtual network, running a gateway (e.g., a local peering gateway), enforcing security lists and network security groups, network address translation (NAT) functions (e.g., translation from public IP to private IP on a per-host basis), throttling functions, and other functions.

[0114] In one embodiment, the packet processing data path within the NVD may comprise multiple packet pipelines, each consisting of a series of packet translation stages. In one implementation, upon receipt of a packet, it is parsed and classified into a single pipeline. The packet is then processed linearly, one stage at a time, until it is either discarded or transmitted through the NVD's interface. These stages provide basic functional packet processing components (e.g., header validation, bandwidth adjustment, insertion of new Layer 2 headers, L4 firewall implementation, VCN encapsulation / decapsulation, etc.) such that new pipelines can be constructed by assembling existing stages, and new functionality can be added by creating new stages and inserting them into existing pipelines.

[0115] The NVD may perform both control plane and data plane functions corresponding to the control plane and data plane of the VCN. Examples of VCN CPs are also shown in Figures 15–18 (see references 1516, 1616, 1716, and 1816) and are described below. Examples of data plane VCNs are also shown in Figures 15–18 (see references 1518, 1618, 1718, and 1818) and are described below. Control plane functions include functions used to configure the network that controls how data is forwarded (e.g., setting routes and route tables, configuring VNICs, etc.). In one embodiment, a VCN CP is provided that centrally calculates the mappings between all overlays and the board and exposes those mappings to the NVD and to various virtual network edge devices such as gateways such as DRG, SGW, and IGW. Firewall rules may be exposed using the same mechanism. In one embodiment, the NVD retrieves only the mappings relevant to that NVD. Data plane functions include functions for the actual routing / forwarding of packets based on the configuration settings that use the control plane. The VCN data plane is implemented by encapsulating customer network packets before they traverse the underlying network. The encapsulation / decapsulation function is performed by the NVD. In one embodiment, the NVD is configured to intercept all network packets entering and leaving the host machine and to perform network virtualization functions.

[0116] As shown above, NVD performs various virtualization functions, including VNICs and VCN VRs. NVD may run VNICs associated with compute instances hosted by one or more host machines connected to a VNIC. For example, as shown in Figure 2, NVD210 runs the functions of VNIC276 associated with compute instance 268 hosted by host machine 202 connected to NVD210. As another example, NVD212 runs VNIC280 associated with bare-metal compute instance 272 hosted by host machine 206 and VNIC284 associated with compute instance 274 hosted by host machine 208. Host machines may host compute instances belonging to different VCNs belonging to different customers, and NVDs connected to host machines may run VNICs corresponding to compute instances (i.e., perform functions associated with VNICs).

[0117] An NVD also runs a VCN virtual router corresponding to the VCN of a compute instance. For example, in the embodiment shown in Figure 2, NVD210 runs a VCN VR277 corresponding to the VCN to which compute instance 268 belongs. NVD212 runs one or more VCN VRs283 corresponding to one or more VCNs to which compute instances hosted by host machines 206 and 208 belong. In one embodiment, the VCN VR corresponding to that VCN is run by all NVDs connected to the host machine hosting at least one compute instance belonging to that VCN. If a host machine hosts compute instances belonging to different VCNs, the NVDs connected to that host machine may run VCN VRs corresponding to those different VCNs.

[0118] In addition to VNICs and VCN VRs, the NVD may run various software (e.g., daemons) and may include one or more hardware components that facilitate various network virtualization functions performed by the NVD. For simplicity, these various components are grouped together as a “packet processing component” as shown in Figure 2. For example, NVD210 has a packet processing component 286, and NVD212 has a packet processing component 288. For example, the NVD’s packet processing component may include a packet processor configured to interact with the NVD’s ports and hardware interfaces to monitor all packets received by the NVD and packets propagated using the NVD, and to store network information. Network information may include, for example, network flow information that identifies various network flows processed by the NVD and per-flow information (e.g., per-flow statistics). In one embodiment, network flow information may be stored per VNIC. In addition to performing per-packet operations, the packet processor may implement stateful NAT and L4 firewall (FW). As another example, the packet processing component may include a replication agent configured to replicate information stored by the NVD to one or more different replication target stores. As yet another example, the packet processing component may include a logging agent configured to perform the logging function of the NVD. The packet processing component may also include software for monitoring the performance and health of the NVD, and optionally software for monitoring the status and health of other components connected to the NVD.

[0119] Figure 1 shows components of an exemplary virtual network or overlay network, including a VCN, subnets within the VCN, compute instances deployed on the subnets, VNICs associated with the compute instances, a VCN VR, and a set of gateways configured for the VCN. The overlay components shown in Figure 1 may be run or hosted by one or more of the physical components shown in Figure 2. For example, compute instances within a VCN may be run or hosted by one or more host machines shown in Figure 2. In the case of compute instances hosted by a host machine, the VNICs associated with those compute instances are typically run by NVDs connected to that host machine (i.e., the VNIC functionality is provided by the NVDs connected to that host machine). The VCN VR functionality of the VCN is run by all NVDs connected to the host machines that host or run compute instances that are part of that VCN. Gateways associated with a VCN may be run by one or more different types of NVDs. For example, one gateway may be run by a smart NIC, while others may be run by one or more host machines or other implementations of NVDs.

[0120] As mentioned above, compute instances within a customer's VCN may communicate with various endpoints, which may be on the same subnet as the source compute instance, on a different subnet within the same VCN as the source compute instance, or outside the source compute instance's VCN. These communications are facilitated using the VNIC associated with the compute instance, the VCN VR, and the gateway associated with the VCN.

[0121] For communication between two compute instances on the same subnet within a VCN, communication is facilitated using VNICs associated with the source and destination compute instances. The source and destination compute instances may be hosted by the same host machine or by different host machines. Packets originating from the source compute instance may be forwarded from the host machine hosting the source compute instance to an NVD connected to that host machine. In the NVD, packets are processed using a packet processing pipeline, which may include running the VNIC associated with the source compute instance. Because the destination endpoint of the packet is within the same subnet, running the VNIC associated with the source compute instance causes the packet to be forwarded to an NVD running the VNIC associated with the destination compute instance, which then processes the packet and forwards it to the destination compute instance. The VNICs associated with source and destination compute instances may run on the same NVD (for example, if both source and destination compute instances are hosted by the same host machine) or on different NVDs (for example, if source and destination compute instances are hosted by different host machines connected to different NVDs). The VNICs may use routing / forwarding tables stored by the NVD to determine the next hop of a packet.

[0122] For packets propagated from a compute instance within a subnet to an endpoint in a different subnet within the same VCN, the packet originating from the source compute instance is propagated from the host machine hosting the source compute instance to the NVD connected to that host machine. The NVD processes the packet using a packet processing pipeline, which can include the execution of one or more VNICs and VRs associated with the VCN. For example, as part of the packet processing pipeline, the NVD executes or invokes (also called executing the VNIC) a function corresponding to the VNIC associated with the source compute instance. The function executed by the VNIC may include examining the VLAN tag on the packet. Because the packet's destination is outside the subnet, the VCN VR function is then invoked and executed by the NVD. The VCN VR then routes the packet to the NVD running the VNIC associated with the destination compute instance. The VNIC associated with the destination compute instance then processes the packet and forwards it to the destination compute instance. The VNICs associated with the source compute instance and the destination compute instance may run on the same NVD (for example, if both the source and destination compute instances are hosted by the same host machine) or on different NVDs (for example, if the source and destination compute instances are hosted by different host machines connected to different NVDs).

[0123] If the destination of a packet is outside the VCN of the source compute instance, the packet originating from the source compute instance is propagated from the host machine hosting the source compute instance to an NVD connected to that host machine. The NVD runs the VNIC associated with the source compute instance. Because the destination endpoint of the packet is outside the VCN, the packet is then processed by the VCN VR of that VCN. The NVD may invoke the VCN VR function, causing the packet to be forwarded to an NVD running the appropriate gateway associated with the VCN. For example, if the destination is an endpoint within the customer's on-premises network, the packet may be forwarded by the VCN VR to an NVD running the DRG gateway configured for the VCN. The VCN VR may run on the same NVD as the NVD running the VNIC associated with the source compute instance, or by a different NVD. The gateway may run by an NVD, which can be a smart NIC, a host machine, or another NVD implementation. The packet is then processed by the gateway and forwarded to the next hop, which facilitates the propagation of the packet to the intended destination endpoint. For example, in the embodiment shown in Figure 2, packets originating from compute instance 268 may be propagated from host machine 202 to NVD210 via link 220 (using NIC 232). On NVD210, VNIC 276 is invoked because it is the VNIC associated with source compute instance 268. VNIC 276 is configured to examine the encapsulated information in the packet, determine the next hop for forwarding the packet to facilitate its delivery to the intended destination endpoint, and then forward the packet to the determined next hop.

[0124] Compute instances deployed on a VCN can communicate with various endpoints. These endpoints may include endpoints hosted by CSPI200 and endpoints outside of CSPI200. Endpoints hosted by CSPI200 may include instances within the same VCN or other VCNs, which may be customer VCNs or VCNs not belonging to a customer. Communication between endpoints hosted by CSPI200 may be performed over the physical network 218. Compute instances may also communicate with endpoints that are not hosted by CSPI200 or are outside of CSPI200. Examples of these endpoints include endpoints within the customer's on-premises network or data center, or public endpoints accessible over a public network such as the Internet. Communication with endpoints outside of CSPI200 may be performed over a public network (e.g., the Internet) (not shown in Figure 2) or a private network (not shown in Figure 2) using various communication protocols.

[0125] The architecture of the CSPI200 shown in Figure 2 is merely an example and is not intended to be limiting. Alternative embodiments are possible, with variations, alternatives, and modifications. For example, in some implementations, the CSPI200 may include more or fewer systems or components than those shown in Figure 2, may combine two or more systems, or may include different configurations or arrangements of systems. The systems, subsystems, and other components shown in Figure 2 may be implemented using hardware, or a combination thereof, in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of each system. The software may be stored on non-temporary storage media (e.g., memory devices).

[0126] Figure 4 shows the connection between a host machine and an NVD to implement I / O virtualization to support multi-tenancy functionality in one embodiment. As shown in Figure 4, host machine 402 runs a hypervisor 404 that provides a virtual environment. Host machine 402 runs two virtual machine instances, VM1 406 belonging to customer / tenant 1 and VM2 408 belonging to customer / tenant 2. Host machine 402 has a physical NIC 410 connected to the NVD 412 via link 414. Each compute instance is connected to a VNIC run by the NVD 412. In the embodiment of Figure 4, VM1 406 is connected to VNIC-VM1 420 and VM2 408 is connected to VNIC-VM2 422.

[0127] As shown in Figure 4, NIC410 has two logical NICs: logical NIC A 416 and logical NIC B 418. Each virtual machine is connected to its own logical NIC and configured to work with that logical NIC. For example, VM1 406 is connected to logical NIC A 416, and VM2 408 is connected to logical NIC B 418. Even though the host machine 402 has only one physical NIC 410 shared by multiple tenants, each tenant's virtual machine is confident that it has its own host machine and NIC due to the logical NICs.

[0128] In one embodiment, each logical NIC is assigned its own VLAN ID. Thus, a specific VLAN ID is assigned to logical NIC A 416 for tenant 1, and another VLAN ID is assigned to logical NIC B 418 for tenant 2. When a packet is transmitted from VM1 406, the tag assigned to tenant 1 is attached to the packet by the hypervisor, and then the packet is transmitted from host machine 402 to NVD412 via link 414. In a similar manner, when a packet is transmitted from VM2 408, the tag assigned to tenant 2 is attached to the packet by the hypervisor, and then the packet is transmitted from host machine 402 to NVD412 via link 414. Thus, the packet 424 transmitted from host machine 402 to NVD412 has an associated tag 426 that identifies a specific tenant and associated VM. In NVD, for packet 424 received from host machine 402, the tag 426 associated with the packet is used to determine whether the packet should be processed by VNIC-VM1 420 or VNIC-VM2 422. The packet is then processed by the corresponding VNIC. The configuration shown in Figure 4 ensures that each tenant's compute instance owns its own host machine and NIC. The configuration shown in Figure 4 enables I / O virtualization to support multi-tenancy functionality.

[0129] Figure 5 shows a simplified block diagram of a physical network 500 according to one embodiment. The embodiment shown in Figure 5 is structured as a Clos network. A Clos network is a specific type of network topology designed to provide connectivity redundancy while maintaining high bimodal bandwidth and maximum resource utilization. A Clos network is a type of non-blocking multi-stage or multi-layer switching network, the number of stages or layers can be two, three, four, five, etc. The embodiment shown in Figure 5 is a three-layer network including layers 1, 2, and 3. The ToR switch 504 represents a layer 0 switch in the Clos network. One or more NVDs are connected to the ToR switch. Layer 0 switches are also called edge devices of the physical network. Layer 0 switches are connected to layer 1 switches, also called leaf switches. In the embodiment shown in Figure 5, a set of "n" layer 0 ToR switches is connected to a set of "n" layer 1 switches, together forming a pod. Each layer 0 switch in a pod is interconnected to all layer 1 switches in the pod, but there are no switch connections between pods. In one implementation, two pods are called a block. Each block is serviced by or connected to a set of n Layer 2 switches (sometimes called spine switches). Multiple blocks can exist in a physical network topology. The Layer 2 switches are then connected to n Layer 3 switches (sometimes called superspine switches). Packet communication over the physical network is typically performed using one or more Layer 3 communication protocols. Typically, all layers of the physical network except the TOR layer have n-way redundancy, thus enabling high availability. Policies may be specified for pods and blocks to control the visibility of switches to each other within the physical network, enabling scaling of the physical network.

[0130] A key feature of Clos networks is that the maximum number of hops required to reach a packet from one Layer 0 switch to another (or from one NVD connected to a Layer 0 switch to another NVD connected to a Layer 0 switch) is fixed. For example, in a Layer 3 Clos network, a packet may require a maximum of 7 hops to reach one NVD to another, with the source and target NVDs connected to the leaf layers of the Clos network. Similarly, in a Layer 4 Clos network, a packet may require a maximum of 9 hops to reach one NVD to another, with the source and target NVDs connected to the leaf layers of the Clos network. Therefore, the Clos network architecture maintains consistent latency throughout the network, which is crucial for communication within and between data centers. Clos topologies scale horizontally and are cost-effective. The network's bandwidth / throughput capacity can be easily increased by adding more switches (e.g., more leaf and spine switches) to different layers and by increasing the number of links between switches in adjacent layers.

[0131] In one embodiment, each resource within the CSPI is assigned a unique identifier called a Cloud Identifier (CID). This identifier is included as part of the resource information and can be used to manage the resource, for example, via the console or via an API. An example syntax for a CID is as follows:

[0132] ocid1.<Resource Type>.<Realm>.[Region][.Future Use].<Unique ID>, where, ocid1: A literal string indicating the CID version. Resource type: The type of resource (e.g., instance, volume, VCN, subnet, user, group, etc.), Realm: The realm containing the resource (example values ​​include "c1" for a commercial realm, "c2" for a government cloud realm, or "c3" for a federal cloud realm. Each realm may have its own domain name), Region: The region containing the resource (this part can be left blank if the region cannot be applied to the resource). Future use: (reserved for future use), and Unique ID: The unique part of the ID (this format may vary depending on the type of resource or service).

[0133] Multicloud Figure 6 shows a simplified high-level Figure 600 of a distributed environment with multiple cloud environments provided by different CSPs. As shown in Figure 6, various cloud environments (also called “clouds”) may be provided by different CSPs, and each cloud environment or cloud provides one or more cloud services that one or more customers of the corresponding CSP can subscribe to. The set of cloud services provided by a cloud environment provided by a CSP may include, but are not limited to, one or more different types of cloud services, such as SaaS services, IaaS services, PaaS services, and DBaaS (Database-as-a-Service) services. Examples of cloud environments provided by various CSPs include OCI, Azure, Google Cloud, and AWS. The set of cloud services provided by a particular cloud environment may differ from the set of cloud services provided by another cloud environment.

[0134] In a standard cloud environment, a CSP (Cloud Service Provider) provides a CSPI (Cloud Service Provider Interface) used to deliver a set of cloud services offered to customers by that cloud environment. The CSPI provided by the CSP may include various types of hardware and software resources, including computing resources, memory resources, network resources, and a console for accessing the cloud services. Customers of a cloud environment provided by a CSP may subscribe to one or more of the cloud services offered by that cloud environment. The CSP may offer customers various subscription models. After a customer subscribes to a cloud service provided by the cloud environment, one or more users may be associated with the subscribed customer, and such users may use the cloud services the customer has subscribed to. In one implementation, when a customer subscribes to a cloud service provided by a particular cloud environment, a customer account or customer tenancy is created for that customer. Then, one or more users may be associated with the customer tenancy, and such users may then use the services the customer has subscribed to under the customer tenancy. Information about the services the customer has subscribed to, the users associated with the customer tenancy, etc., is typically stored within the cloud environment and associated with the customer tenancy.

[0135] For example, Figure 6 shows two different cloud environments provided by two different CSPs (although a different number of cloud environments are possible). These cloud environments include Cloud Environment A (Cloud A) 610 provided by CSP A and Cloud Environment B (Cloud B) 640 provided by CSP B.

[0136] Cloud A610 includes infrastructure CSPI_A612 provided by CSP A. This infrastructure CSPI_A612 may be used to provide a set of in-cloud services 615 provided by Cloud A610. One or more customers (e.g., Cust_A1 616-1, Cust_A2 616-2) may subscribe to one or more of such services. One or more users 618-1 may be associated with customer Cust_A1 616-1 and can use the services that customer Cust_A1 616-1 has subscribed to within Cloud A610. Similarly, one or more users 618-2 may be associated with customer Cust_A2 616-2 and can use the services that customer Cust_A2 616-2 has subscribed to within Cloud A610. In various use cases, the service subscribed to by customer Cust_A1 616-1 may be different from the service subscribed to by customer Cust_A2 616-2.

[0137] Similarly, Cloud B640 includes infrastructure CSPI_B642 provided by CSP B. This infrastructure CSPI_B642 may be used to provide a set of services 644 provided by Cloud B640 (the set of services 644 may differ from, but do not necessarily have to differ from, the services provided by Cloud A610). One or more customers (e.g., Cust_B1 646-1) may subscribe to one or more services from the set of services 644. One or more users 648-1 may be associated with customer Cust_B1 646-1 and can use the services that customer Cust_B1 646-1 has subscribed to within Cloud B640.

[0138] As shown in Figure 6, customer Cust_A1 616-1 is also a customer of CSP B and subscribes to services available from CSPI_B642. Therefore, customer Cust_A1 616-1 has tenancy in both Cloud B640 and Cloud A610.

[0139] In one embodiment, CSP A and CSP B may agree to provide their respective cross-cloud services. In the example in Figure 6, CSP A provides one or more of its services to CSP B's customers via CSPI_B642 (referred to herein as cross-cloud services). These cross-cloud services include, for example, database services, storage services, and compute services. Thus, customer Cust_A1 616-1 (a customer of both Cloud A and Cloud B) can request, subscribe to, use, and / or manage one or more of CSP A's cross-cloud services via tenancy in CSPI_B642. In comparison, customer Cust_B1 646-1 does not have tenancy in CSPI_A612. Therefore, CSP A's cross-cloud services may not be available to customer Cust_B1 646-1 unless customer Cust_B1 646-1 requests provisioning of tenancy by CSP A. This request can be submitted and managed through the Cloud B640 portal, as further illustrated in the following diagram.

[0140] To enable the provision and availability of CSP A's cross-cloud services via cloud B640, cloud A610 can implement a cross-cloud service 614. The cross-cloud service 614 may, in particular, be configured to enable the use of CSP A's cross-cloud services via cloud B. For example, the cross-cloud service 614 can communicate with a service 644 in cloud B640 and translate such communication into information suitable for use by an in-cloud service 615 in cloud A610. More specifically, the set of services 644 can enable a portal for CSP B, as well as the deployment and management of resources for CSP B's customers within cloud B640. Through this portal, CSP A's cross-cloud services may be provided. Thus, a customer of CSP B (e.g., customer Cust_A1 616-1) can subscribe to and request CSP A's cross-cloud services through the portal. Such subscription and cloud operation requests can be received by the inter-cloud service 614, which then translates these requests into information specific to cloud A610. Such information is passed to one or more of the intra-cloud services 615, which then provision the services. The information returned from the intra-cloud services 615 can be translated by the inter-cloud service 614 and sent to cloud service 644 for use.

[0141] In one embodiment, the requested cross-cloud service may be provisioned by one or more in-cloud services 615 that span resources in both cloud A610 and cloud B640. This can support (or at least reduce) latency-sensitive operations.

[0142] As shown in the diagram, the infrastructure CSPI_A612 for Cloud A610 includes infrastructure 624 for customer Cust_A1 616-1's private cloud (e.g., a VCN that is part of this customer's tenancy in Cloud A610), infrastructure 626 for customer Cust_A2 616-2's private cloud in Cloud A610, and other infrastructure 620. Each of these infrastructures includes hardware and / or software provided by CSP A and installed in locations (organized as regions) under the management of CSP A.

[0143] For comparison, infrastructure CSPI_B642 in cloud B640 includes infrastructure 644 for customer Cust_A1 616-1's private cloud in cloud A (e.g., a VCN that is part of this customer's tenancy in cloud A610), infrastructure 646 for customer Cust_A1 616-1's private cloud in cloud B640 (e.g., a VNET that is part of this customer in cloud B640), infrastructure 648 for customer Cust_B1 646-1's private cloud in cloud B640, and other infrastructure 650. Infrastructure 644 includes hardware and / or software provided by CSP A and located in a location (organized as a region) under the management of CSP B (e.g., located in the same location as the components of CSPI_B642). For comparison, infrastructures 646, 648, and 650 include hardware and / or software provided by CSP B and located in a location (organized as a region) under the management of CSP B.

[0144] Infrastructure 644 and Infrastructure 646 can be networked together so that customer Cust_A1 616-1 can access them via a private cloud by CSP B and a private cloud by CSP A, with the private cloud by CSP A distributed between CSPI_A612 and CSPI_B642. This communication coupling of private clouds is initiated by an inter-cloud service 614 and can be executed by one or more intra-cloud services 615. In this way, customer A616-1 has two tenancies: a first tenancy by CSP A, which includes a first private cloud (e.g., VCN) distributed between CSPI_A612 and CSPI_B642, and a second tenancy by CSP B, which includes a second private cloud (e.g., VNET) local to CSPI_B642.

[0145] The requested cross-cloud service may actually be hosted on a first private cloud (e.g., at least partially on infrastructure 644 within CSPI_B642, and possibly within infrastructure 624 within CSPI_A612) and accessible via a second private cloud (e.g., by a workflow hosted by infrastructure 646). This distribution of components to enable the first private cloud of the first CSP to be at least partially hosted by the second cloud of the second CSP, and to be linked to the second private cloud hosted by the second cloud, is further illustrated in the following diagram.

[0146] Figure 7 shows an exemplary physical architecture for providing cross-cloud services based on a distributed infrastructure across multiple CSPs, according to one embodiment. In Figure 7, a customer of the second CSP manages the lifecycle of the cross-cloud service developed by the first CSP. For illustrative purposes, the Exadata service (sometimes referred to herein as the Oracle DB service) and the Oracle and Microsoft clouds are described. In this figure, Oracle corresponds to the first CSP, Microsoft corresponds to the second CSP, while the Exadata service corresponds to the cross-cloud service provided by the first CSP via the second CSP's cloud. However, embodiments are not limited in this way and instead apply equally to other CSPIs, CSPs, and / or cross-cloud services.

[0147] A customer of the second CSP (e.g., an Azure customer) can create and manage virtual resources (e.g., infrastructure and / or VM clusters) within the first CSP's cloud via the second CSP's portal (e.g., the Azure portal). These virtual resources may be provisioned to provide cross-cloud services. Support for such provisioning may be implemented, at least in part, using the cross-cloud services 614 in Figure 6. Furthermore, the infrastructure supporting the virtual resources may be distributed between the two CSPs' CSPIs (e.g., between an Azure data center and an OCI region). In detail, the first portion of the infrastructure provided by the first CSP resides in the second CSP's CSPI, while the remaining second portion of the infrastructure provided by the first CSP resides in the first CSP's CSPI. The first portion may be considered to be contained within the second CSP's cloud or to form a child site within the second CSP's cloud. The second portion may be considered to be contained within the parent region of the child site or to form the parent region of the child site, which resides within the first CSP's cloud.

[0148] On the left side of Figure 7, the second CSPI_B750 of the second CSP is shown. This CSPI_B750 could represent, for example, the data center of the second CSP (e.g., an Azure data center). Within CSPI_B750, the board network 730 is available and provided by the second CSP. The board network 730 includes a set of computing resources such as routers 732A and 732B (in the case of an Azure data center, these routers may include MeetMe ToR, which supports the MeetMe protocol for connectivity peering). CSPI_B750 also includes a child site 720 (the child site 720 is a set of computing resources such as server blades, racks, or other physical hardware of the first CSP that runs the CSP's software). Child site 720 includes, in particular, routers 722A, 722B, etc. (in the case of OCI, these routers may include FastConnect routers that support the FastConnect protocol for connectivity peering), connectivity fabric 724 (e.g., physical fabrics such as JFAB that provide connectivity with other physical fabrics and components), and physical resources 726 (e.g., racks such as OCI server blades optimized for cross-cloud services such as Exadata services).

[0149] On the right side of Figure 7, the first CSPI_A700 of the first CSP is shown. This CSPI_A700 can represent, for example, the data center of the first CSP (e.g., the OCI data center). Within CSPI_A700, the parent region 710 is shown. The parent region 710 may include an onboard network with multiple components. Among these components is a connectivity fabric 712 (e.g., a physical fabric such as JFAB). The connectivity fabric 712 is connected to a connectivity fabric 724 so that child sites 720 are coupled to the parent region 710 in a communicative manner. Optionally, the parent region 710 is located in a region physically close to child site 720 (or equivalently, CSPI_B750) to reduce network latency for communication between child site 720 and the parent region 710.

[0150] Router 732 in the board network 730 can be interconnected with Router 722 in the child site 720 (for example, via an Ethernet cable). The connectivity fabric 724 in the child site 720 provides interconnection between Router 722 and physical resources 726. Furthermore, this connectivity fabric 724 is connected to the connectivity fabric 714 in the parent region 710 (for example, using optical fiber such as dark fiber) so that data connectivity exists between the child site 720 and the parent region 710.

[0151] The board network 730 can host a set of resources for a second CSP for the customer (for example, to provide a VNET containing compute instances that can access Exadata services, as further shown in the following figure). These second CSP resources may be part of a proximity group that has a certain latency (e.g., 100 μs) to the cross-cloud service.

[0152] Child site 720 can host resources where latency is critical for the first CSP, and these first CSP resources support cross-cloud services (for example, child site 720 can host OCI database resources and data plane resources in support of Exadata services). Parent region 720 can host other resources that support cross-cloud services (for example, cross-cloud services 614 and intra-cloud services 715 in Figure 6) (for example, for Exadata services, parent region 720 can host ORP, OCI tools, OCI metrics and logging, OCI control plane, regional OCI services, and customer console). Some of these resources (for example, OCI database resources and data plane resources) can be deployed as part of the customer's first private cloud by the first CSP (for example, as a VCN in the case of OCI) and may be perceived by the customer as being available through the customer's second private cloud by the second CSP (for example, a VNET in the case of Azure). This recognition is made possible by using the same IP address range for cross-cloud services across two private clouds.

[0153] In one example, a customer may have multiple private clouds (e.g., multiple VNETs and / or multiple VCNs) under each CSP. In the case of multiple private clouds under the first CSP, the underlying physical resources do not have to be located within the same CSPI as the second CSP, but instead can be contained in or form different child sites. In this case, these resources do not have to be directly interconnected (e.g., there does not have to be a direct physical connection between different child sites). Instead, an indirect connection can exist through the parent region, with each of the different child sites being physically connected to the parent region (e.g., via fiber optic cable), and data flowing between the two child sites through the parent region.

[0154] Furthermore, child site 720 can support a multi-tenancy architecture. Specifically, multiple customers may each have one or more private clouds (e.g., one or more VNETs and one or more VCNs) provided by each CSP. Each of such customers can access the corresponding cross-cloud services separately through child site 720.

[0155] Figure 8 shows an exemplary virtual architecture for providing cross-cloud services based on a distributed infrastructure across multiple CSPs, according to one embodiment. In Figure 8, the customer's first private cloud 810 (e.g., a VCN hosted by OCI) by the first CSP and the customer's second cloud 820 (e.g., a VNET hosted by Azure) by the second CSP may be provisioned to provide the first CSP's cross-cloud services (e.g., Exadata services provided by Oracle and available to Azure customers via Azure) through the second CSP's cloud.

[0156] In one example, a second private cloud 820 (e.g., a VNET) is hosted by a physical computing resource (e.g., an Azure data center) of the second CSP's CSPI_B850. The second private cloud 820 may contain compute instances (not shown) that can access cross-cloud services. The second private cloud 820 includes subnet 822, which uses an IP address range. This range may be specified based on customer input through the second CSP's portal (e.g., the Azure portal). IP addresses may be assigned from this IP address range to each compute instance launched to provide cross-cloud services to the customer (e.g., a VM belonging to a VM cluster that provides a database instance). These compute instances may be hosted by the first private cloud 810 (more commonly shown as virtual resources 814A, 814B, 814C, and 814D in Figure 8). In Figure 8, four computed instances are illustrated and shown by rectangles labeled NIC824A, 824B, 824C, and 824D (or collectively, NIC824). While NIC stands for Network Interface Card, NIC824 in Figure 8 is not such a thing in itself. Instead, NIC824 represents mapping information for mapping each IP address to the corresponding computed instance of the cross-cloud service. This mapping information may be stored as part of the configuration information of the second private cloud 820. If a computed instance of the second private cloud 820 sends or requests traffic using an IP address and this IP address corresponds to a NIC, the mapping information indicates that the traffic is sent to or received between the first private cloud 810 (e.g., VCN) and is cross-cloud service traffic.

[0157] The first private cloud 810 (e.g., VCN) may be hosted by the physical computing resources of the first CSP, including, for example, the physical computing resources of child sites and parent regions (similar to those shown in Figure 7). The first private cloud 810 may include subnet 812 using the same IP address range. Each compute instance launched to provide cross-cloud services (e.g., each of virtual resources 814A, 814B, 814C, and 814D, collectively referred to as virtual resource 814) is hosted by a set of physical computing resources of the first CSP. The set of compute instances may include physical computing resources in child sites and physical computing resources in parent regions. The computing resources in child sites may include hardware (e.g., racks) running software for cross-cloud services that provide latency-sensitive virtual resources (e.g., database resources, data plane resources, etc.), for example, as shown in Figure 7. The physical computing resources of the parent region may include virtual resources where latency is not critical (e.g., control plane resources, tool resources, etc.), such as servers, server fleets, and network virtualization devices.

[0158] Each compute instance can have an IP address from an IP address range. A one-to-one mapping exists between the compute instance in the first private cloud 810 (e.g., virtual resource 814) and NICS824 in the second private cloud 820. Therefore, cross-cloud service traffic can be sent from the second private cloud 820 to the first private cloud 810 and vice versa using this one-to-one mapping.

[0159] The second private cloud 820 may be connected to the first private cloud 810 (e.g., peered) via one or more virtual routers, gateways (e.g., gateway 826 implemented in the child site and gateway 816 implemented in the parent region, for example, by being part of a connectivity fabric to connect the child site to the parent region), and connectivity protocols (e.g., MeetMe protocol and FastConnect protocol). Gateways 816 and 826 can collectively represent a dynamic routing gateway.

[0160] From the customer's perspective, the customer does not need to be aware of the underlying physical architecture and interconnects. Instead, it may suffice for the customer to have visibility at the virtual level, thereby enabling them to recognize and manage private clouds 810 and 820 (e.g., network configuration). Managing the second private cloud 820 can be done via the second portal of the second CSP. Managing the first private cloud 810 can be done via the second portal of the second CSP or the first portal of the first CSP. In embodiments, changes and / or operations related to cross-cloud services are enabled only via the second CSP (e.g., via the second CSP, the customer can expand or shrink, remove, terminate, add, etc., virtual resources in the first private cloud 810 launched for cross-cloud services). In comparison, changes and / or operations unrelated to cross-cloud services are enabled via (and in some cases, only via) the first CSP (for example, via the first CSP, a customer can expand or shrink, remove, terminate, add, etc., virtual resources in the first private cloud 810 launched for purposes other than cross-cloud services).

[0161] Figure 9 shows exemplary virtual resources provisioned for a customer in one embodiment. The right side of Figure 9 shows the first resource (e.g., OCI resource) of the first CSP deployed for the customer. The left side of Figure 9 shows the second resource of the second CSP (e.g., to an Azure data center) deployed for the customer. The first CSP (indicated as CSP A) provides cross-cloud services (e.g., Exadata services) available through the cloud of the second CSP (indicated as CSP B).

[0162] The first resource (which may be a combination of physical and virtual resources) may include a private cloud 920, a service tenancy 910, a VM cluster 926, and a gateway 928. The private cloud 920 (e.g., a VCN) may accommodate customer tenancies by the first CSP and may include a primary subnet 922 with an IP range (e.g., a classless inter-domain routing (CIDR) range), and optionally, a backup subnet 924 using the same IP range. The service tenancy 910 may be available to multiple customers of the first CSP and may include a gateway 912. The VM cluster 926 includes multiple VMs (or compute instances) launched by a client to provide cross-cloud services to the VMs. The VM cluster 926 may be hosted on physical resources within a child site. The gateway 928 (e.g., a service gateway) connects the virtual resources of the primary subnet 922 (and similarly, the backup subnet 924) hosted on physical resources within the parent region to the VM cluster 926. In comparison, gateway 912 connects such virtual resources to the customer's private cloud 960 by the second CSP. Gateway 912 may be implemented as a DRG connected to the private cloud 920 within the customer tenancy. This connection can be via gateway / router 914. Gateway / router 914 is provided at least partially within the child site and can connect the child site to the parent region via a first connection protocol (e.g., FastConnect) (thus providing gateway functionality) and connect the child site to the underlying network of the second CSP via a second connection protocol (e.g., MeetMe) (thus providing routing functionality). For example, gateway / router 914 may represent a FastConnect Virtual Circuit resource within service tenancy 910 for connecting the DRG to an Azure MeetMe router (example router 968).

[0163] The second resource (which may be a combination of physical and virtual resources) may include a private cloud 960 and a router 968. The private cloud 960 may include a primary subnet 922 with an IP range, and optionally a backup subnet 964 using the same IP range. The IP range may be the same as that of the primary subnet 922. The router 968 may be connected to a gateway / router 914 to provide data connectivity from the primary subnet 962 (and, for example, the backup subnet 964) to the private cloud 920. The primary subnet 962 may include NICs 966A-966K connected to the router 968.

[0164] The IP addresses used for different virtual resources to provide cross-cloud services may be selected from an IP range by the first CSP (e.g., by its cross-cloud or intra-cloud service). The first CSP (e.g., its cross-cloud or intra-cloud service) may create a domain name system (DNS) corresponding to the IP addresses and provide it to Private Cloud 960. Such DNS records can be used to configure a customer's private DNS zone, which may be used in conjunction with Private Cloud 960.

[0165] For illustrative purposes, subnets 922, 924, 962, and 964 use IP addresses from 10.0.10.0 / 24. Each NIC966 has an IP address from this range (for example, NICA966A uses 10.0.10.10 and NICK966K uses 10.0.10.11). VMs in a VM cluster use the same IP address (for example, the first VM uses 10.0.10.10, while the second VM uses 10.0.10.11).

[0166] Figure 10 illustrates an exemplary architecture for provisioning and managing cross-cloud services based on a distributed infrastructure across multiple CSPs, according to one embodiment. In Figure 10, a customer of a second CSP (shown as CSP B) manages the lifecycle of a cross-cloud service developed by a first CSP (shown as CSP A). For clarity and illustrative purposes, OCI and Azure are described as examples of CSPIs, and Exadata services are described as an example of a cross-cloud service. However, the embodiments are not limited in this way and instead apply equally to other CSPs and / or cross-cloud services.

[0167] A customer (for example, an Azure customer) can operate customer device 1000 and create and manage a set of virtual resources for cross-cloud services (for example, Oracle DB Infrastructure resources) through a portal of a second CSP (shown as CSP_B portal 1052). In one example, this portal 1052 interacts with one or more services of the second CSP (shown as CSP_B service 1054), which could include Azure Resource Manager (ARM) exposing APIs for managing the lifecycle of Oracle DB Infrastructure resources (DB Infrastructure resources include Exadata Infrastructure and Cloud VM Cluster), while other examples could include Azure Resource Provider as a Service (RPaaS) to simplify the development and operation of Azure Resource Providers, such as asynchronous operation and updates of their progress. The CSP_B service 1054 interacts with the first CSP's inter-cloud service 1012 so that customers are redirected to the inter-cloud service 1012 (e.g., ORP in the case of OCI) in order to manage resources built on a set of virtual resources (DBHome, Database, Pluggable Database).

[0168] The cross-cloud service 1012 is configured to perform (i) the conversion of identifiers assigned by a second CSP (e.g., Azure identifiers) to identifiers assigned by a first CSP (e.g., OCI identifiers), including identity, resource ID, and subscription ID; (ii) the conversion of the state of a second CSP (e.g., Azure state) to the state of a first CSP (e.g., OCI state), and vice versa; and / or (iii) the delegation of requests to perform to an in-cloud service 1014 of the first CSP (e.g., a resource control plane 1016 such as the OCI DBaaS control plane). The cross-cloud service 1012 is also configured to coordinate any integrations specific to a second CSP with other second CSP services (e.g., with Azure Resource Provider). The cross-cloud service 1012 may also be configured to perform or have other in-cloud services 1014 perform actions including linking cloud accounts, exposing observability information, and selling tokens for accessing customer environments in other clouds.

[0169] The in-cloud service 1014 can provide a portal to the first CSP (indicated as CSP_A portal 1018) accessible from the customer's customer device 1000 in order to manage other services of the customer by the first CSP. The two portals 1052 and 1018 may enable similar functionality (e.g., by presenting input and output fields), but may also be different. For example, portal 1052 (e.g., the Azure portal) may have a presentation format controlled by the second CSP. Furthermore, portal 1052 may enable functionality specific to the second CSP and independent of the first CSP, in addition to functionality related to cross-cloud services. In comparison, portal 1018 (e.g., the OCI portal) may have a presentation format controlled by the first CSP. Furthermore, portal 1018 may enable functionality specific to the first CSP and independent of the second CSP, in addition to functionality related to cross-cloud services. As far as cross-cloud services are concerned, portal 1052 can publish information available from the second CSP, and this information may be provided by the cross-cloud service 1012.

[0170] In an example of an OCI and Azure use case, the cross-cloud service 1012 exposes an Oracle DB product. The cross-cloud service 1012 can be registered with ARM via RPaaS and is configured to implement Azure Resource Provider Contracts (RPCs), a set of operations supported by all Azure Resource Providers. The internal services of Azure RPaaS are used to support most of the RPC operations.

[0171] In general, the cross-cloud service 1012 may be configured to have the ability to operate in the customer environment of the second CSP using the second CSP flow (e.g., the Azure OBO flow), in addition to the identity of the second CSP (e.g., the Azure RP identity equivalent to the OCI service principal). The cross-cloud service 1012 may also be configured to maintain metadata specific to the second CSP for the first CSP resources, such as mappings from Azure identifiers to OCIDs for DB resources. The cross-cloud service 1012 may also be configured to have the identity of the first CSP in order to obtain the scope to operate in the customer environment of the first CSP. The cross-cloud service 1012 may also be configured to function as a thin adapter layer, receiving a request formatted by the second CSP already authenticated by service 1054, translating it into a request for the first CSP, and delegating this request to the in-cloud service.

[0172] Therefore, the cross-cloud service 1012 performs several operations. These operations include translating identifiers from one cloud to another and vice versa. These operations also include obtaining the identity of the first CSP for an incoming request from service 1054 for use in calling the intra-cloud service 1014. These operations also include translating a request in the second CSP format into a request in the first CSP format and calling the intra-cloud service 1014. These operations also include passing or adjusting limit / quota / capacity validations, implicitly creating prerequisites for the first CSP for network connectivity of network-connected resources (e.g., creating the OCI DRG, VCN, and subnet to which the VMCluster is connected), creating private cloud links, and configuring DNS entries by the second CSP for resources to which DNS records are associated.

[0173] In one example, a cross-cloud service 1012 may be hosted in one or more child sites. The cross-cloud service 1012 may have a virtual resource 1020 hosted in a child site. In an Exadata service use case, the virtual resource 1020 may include a DBaaS data plane via a DBaaS control plane. This control plane is hosted in the parent region as part of an in-cloud service 1014. In this use case, compute instances (e.g., VMs) may be instantiated for the customer in the customer's private cloud (e.g., a VNET in Azure) by a second CSP. The compute instances can perform database operations by making calls to the DBaaS data plane. Such operations include querying, storing, or any other operations supported by the Exadata service. These calls and responses may reside within the private cloud by the second CSP. When in use, the DBaaS data plane can report usage information (e.g., metrics analysis, billing, etc.) to one or more of the in-cloud services 1014 (e.g., an observability service). This usage information can then be provided to the monitoring service of the second CSP.

[0174] Figure 11 shows an exemplary user experience flow 1100 for provisioning resources according to one embodiment. In one example, the user experience flow 1100 includes a customer device 1120, a second CSP's CSPI_B1150, and a first CSP's cross-cloud service. The first CSP can provide cross-cloud services through the second CSP. Virtual resources can be provisioned to provide at least a portion of the cross-cloud service, and this provisioning follows the user experience flow 1100.

[0175] User input is received from customer device 1120 by CSPI_B1150 (for example, by a portal similar to CSP_B portal 1052 in Figure 10) to create a subnet and to mark the subnet as delegated so that it can be used for cross-cloud services. The subnet and the customer's second private cloud by the second CSP are created, and instructions for this creation are provided to customer device 1100.

[0176] Next, user input is received from computer device 1100 by the cross-cloud service 1110 (this input is possible via the portal) and indicates a request to create a cross-cloud service infrastructure (e.g., Exadata service). This user input can indicate various provisioning parameters, such as region and availability zone. The cross-cloud service 1110 can indicate provisioning asterisks to computing device 1100 (e.g., via the portal) and execute a provisioning workflow in which the cross-cloud service infrastructure is created at the associated child site. While this infrastructure is being created, the cross-cloud service 1110 sets the infrastructure status to "provisioning". After creation, the cross-cloud service 1110 updates and provides the status to "successful". These status indications can be provided to computing device 1100 (e.g., via the portal). Similarly, computing device 1100 can check its current status via the portal, and status query requests can be made from CSPI_B1150 to the cross-cloud service 1110, which responds with status information.

[0177] Subsequently, user input is received by the inter-cloud service 1110 from CSPI_B1150 (this input is provided via the portal) and corresponds to a request to create VM cluster resources from the computing device 1100. This request may specify a subnet. The inter-cloud service 1110 then performs a series of checks and begins provisioning the VM cluster. This provisioning is further illustrated in the following diagram. The inter-cloud service 1110 sets the status to "Provisioning". After creation, the inter-cloud service 1110 updates the status to "Success". Again, these status indications can be provided to the computing device 1100 (for example, via the portal). Similarly, the computing device 1100 can check its current status via the portal, and status query requests can be made from CSPI_B1150 to the inter-cloud service 1110, which responds with status information.

[0178] Figure 12 shows an exemplary control plane provisioning flow 1260 according to one embodiment. In one example, the control plane provisioning flow 1260 includes a cross-cloud service 1200, a control plane 1202 (e.g., an example of an intra-cloud service), a customer's private cloud 1204 by a first CSP, a gateway 1206 (e.g., a DRG), a connectivity module 1208 (e.g., a connectivity module provided by a connectivity fabric and supporting connectivity protocols such as FastConnect), a cross-cloud service infrastructure 1210 by the first CSP, and a CSPI_B 1250 of a second CSP. The first CSP can provide cross-cloud services through the second CSP. Here, the control plane 1202 can execute part of the control plane provisioning flow 1260 when triggered by the cross-cloud service 1200 to provision VM cluster resources. The inter-cloud service 1200 can make such a call to the control plane 1202 upon receiving input from CSPI_B1250 (from ARM in Azure) in response to user input from the portal of the second CSP.

[0179] As previously stated herein, the cross-cloud service 1220 receives a request to create a VM cluster for the cross-cloud service (e.g., an Exadata VM cluster) and passes this request to the control plane 1202 along with other information (e.g., a first identifier assigned by the first CSP and mapped to a second identifier assigned by the second CSP). The control plane 1202 then retrieves details about the subnet, including the CIDR, from the CSPI_B1250 (e.g., from ARM in Azure) and verifies that the prerequisites are met (e.g., the subnet is provisioned in a region of the second private cloud by the second CSP (e.g., a VNET in Azure)). The control plane 1202 then creates the first private cloud (e.g., a VCN in OCI) by the first CSP in the customer tenancy (e.g., based on the first identifier) ​​and creates a subnet within the first private cloud. This subnet has the same CIDR as the subnet in the second private cloud.

[0180] Next, the control plane 1202 creates and configures a gateway 1206 (e.g., a DRG) in the service tenancy and requests that gateway 1206 be connected to the first private cloud within the customer tenancy. The control plane 1202 also configures routing information in gateway 1206 (e.g., by creating a DRG routing table for traffic between the two private clouds or MeetMe routers). Similarly, the control plane 1202 also configures routing information in the first private cloud (e.g., by creating a VCN routing table for traffic to the DRG and the VM cluster). After the routing information has been created, the control plane 1202 can notify the inter-cloud service 1200 of the success. Accordingly, the inter-cloud service 1200 can configure a network security group for the first private cloud and request that the VM cluster be created. The VM cluster is then provisioned at the associated child site, thereby assigning an IP address from the CIDR to the VM cluster. DNS records corresponding to the IP addresses are also generated. The inter-cloud service 1200 can receive such DNS records and request connectivity from the first private network to the second private network (for example, from a VCN to a VNET).

[0181] The control plane 1202 then calls the API of the second CSP, which enables the deployment of IP addresses to the customer subnet in the form of NICs. Each IP address can represent the IP of a VM cluster node. The control plane 1202 can register all the IP addresses that the VM cluster will have. The API call can include a device identifier (e.g., the GUID of the MeetMe router). This identifier can be based on a mapping of GUIDs mapped to physical locations (child sites). The call can also include a subnet resource identifier (e.g., a subnet of a private network within the customer tenancy), an IP address (e.g., the IP address of the NIC being created, which is within the subnet), and a resource name (the name of the NIC that will be visible to the user). The API can return a virtual local area network identifier (VLAN ID) that can be used when creating the virtual circuit. The control plane 1202 then creates a virtual circuit using the CSPI_B1250's connection to the router (e.g., to the MeetMe router) (e.g., a FastConnect connection) and provides DNS records so that the private DNS zone can be added to the customer's second private network. At this point, the VM cluster is successfully created. Next, compute instances for cross-cloud services (e.g., Exadata database instances) can be hosted within the VM cluster.

[0182] Resource provider As mentioned above, the first CSP may provide in-cloud services (e.g., database services, storage services, compute services, etc.) to its customers, and the second CSP may provide similar in-cloud services to its customers. The second CSP's customers may also be customers of the first CSP and may want to access the in-cloud services provided by the first CSP through tenancy in the cloud environment provided by the second CSP. Therefore, the first CSP may provide its in-cloud services as cross-cloud services to its customers. Similarly, the second CSP may provide in-cloud services, such as cross-cloud services, to its customers. At least one of the cross-cloud services provided by one CSP to another CSP's customers may be the same service as an in-cloud service provided by that CSP to its own customers. In this way, a customer of one CSP may be provided with a platform-level experience of another CSP from within the cloud environment of that one CSP. Furthermore, a customer of one CSP can access new features, releases, and resources from other CSPs without leaving the cloud environment of that single CSP.

[0183] For example, Oracle, as a CSP, can provide in-cloud services to its own customers via OCI, and Microsoft, as a CSP, can provide similar in-cloud services to its own customers via Azure. A Microsoft Azure customer may also be an Oracle OCI customer and may want to access in-cloud services provided by Oracle OCI through their Microsoft Azure tenancy. Therefore, Oracle OCI can provide in-cloud services, such as Oracle's Exadata database service, to Microsoft Azure customers as cross-cloud services. Similarly, Microsoft Azure can provide in-cloud services, such as Azure Synapse Analytics, to Oracle OCI customers as cross-cloud services. At least one of the cross-cloud services provided by Oracle OCI to Microsoft Azure customers may be the same service provided by Oracle OCI to one of its own customers. For example, Oracle OCI can provide Oracle's Exadata database service to its own customers and to Microsoft Azure customers via Microsoft's Azure environment. In this way, Microsoft Azure customers can access a platform-level experience of Oracle OCI from within Microsoft Azure. Furthermore, Microsoft Azure customers can access new features, releases, and resources from Oracle's OCI without leaving Microsoft Azure. While Oracle's OCI and Microsoft Azure are used as examples, the technologies described throughout are not limited to these CSPs and may be equally applicable to other CSPs such as Google Cloud® and AWS®.

[0184] To facilitate the provision of cross-cloud services, child sites may be provided in the respective CSPIs of different CSPs, and cross-cloud services provided by each of the other CSPs may be accessed using the child sites. For example, in the case of OCI and Azure as described above, a child site may be provided in the Azure CSPI, and may provide access to the Exadata database service provided by OCI from within the Azure cloud environment. Providing access to cross-cloud services using child sites can reduce latency and provide high-bandwidth access to those cross-cloud services compared to accessing them through the first CSP and / or other remote cloud environments. However, resources such as compute resources at each child site may be limited. Therefore, it may be desirable to provide one or more management mechanisms for provisioning and managing the lifecycle of cross-cloud services from one or more CSPIs and / or within the cloud environment. The techniques described herein relate to resource management mechanisms for provisioning and managing the lifecycle of cross-cloud services provided by one or more CSPs and / or between one or more CSPs. The resource management mechanisms described herein are dynamic in that the characteristics of the CSP's child sites, cloud environments, and / or CSPI may be taken into consideration, along with other factors, in provisioning and managing cross-cloud services.

[0185] Figure 13 shows an example of architecture 1300, which includes a resource management mechanism for provisioning and managing cross-cloud services between multiple cloud environments. As shown in Figure 13, architecture 1300 can include a first cloud environment 1302 (e.g., Oracle OCI) of a first CSP and a second cloud environment 1318 (e.g., Microsoft Azure) of a second CSP. The first cloud environment 1302 and the second cloud environment 1318 can be implemented according to the distributed environment described with respect to Figures 6-9. The first cloud environment 1302 and the second cloud environment 1318 can include one or more private clouds (e.g., a VCN in the case of Oracle OCI and a VNET in the case of Microsoft Azure). Furthermore, cross-cloud services between the first cloud environment 1302 and the second cloud environment 1318 can be provisioned according to the experience and provisioning flow described with respect to Figures 10-12.

[0186] The first cloud environment 1302 may be configured to receive requests for cross-cloud services, evaluate the authorization status of such requests, generate instructions to provision such services in one or more other cloud environments, such as the second cloud environment 1318, deploy such services to one or more other cloud environments, and manage the deployed services. In some implementations, the in-cloud services 1310 of the first cloud environment 1302 may be configured to provide one or more of the in-cloud services 1310 (e.g., Exadata in-cloud services) as one or more cross-cloud services (e.g., Exadata cross-cloud services) to customers with tenancy in the second cloud environment 1318. Provisioning the in-cloud services provided by the first cloud environment 1302 as cross-cloud services between the first cloud environment 1302 and the second cloud environment 1318 can be at least facilitated by the resource provider 1304 of the first cloud environment 1302. To provision a cross-cloud service, the resource provider 1304 can be configured to send the requested cross-cloud service provisioning request to the service control plane 1306 and network control plane 1308 of the first cloud environment 1302, and accordingly, the service control plane 1306 and network control plane 1308 can deploy the internal cloud of the in-cloud service 1310 as a cross-cloud service to the second cloud environment 1318 (for example, to the service data plane 1332 and network provider 1334 of the second cloud environment 1318).

[0187] In some implementations, the first cloud environment 1302 may contain multiple parent regions, and the second cloud environment 1318 may contain multiple child sites corresponding to multiple parent regions. In some implementations, the service data plane 1332 of the second cloud environment 1318 may function as one or more child sites of each parent region and / or form such child sites. In some implementations, each parent region of the first cloud environment 1302 may contain a resource provider, such as a resource provider 1304, which, together with one or more child sites associated with each parent region, facilitates the provisioning and lifecycle management of one or more cross-cloud services. One or more cross-cloud services may include one or more of the in-cloud services 1310 of the first cloud environment 1302. For example, resource provider 1304 may be a resource provider for a parent region within the first cloud environment 1302, and may provision and manage the lifecycle of one of the in-cloud services 1310 as a cross-cloud service between the service control plane 1306 of the first cloud environment 1302 and the service data plane 1332 of the second cloud environment 1318. Child sites may be implemented according to the physical architecture described above, at least with respect to Figure 7.

[0188] In some implementations, cross-cloud service provisioning between a first cloud environment 1302 and a second cloud environment 1318 can be facilitated by a resource provider 1304 in the first cloud environment 1302 and a resource manager 1324 in the second cloud environment 1318. For example, the resource manager 1324 in the second cloud environment 1318 may send a cross-cloud service provisioning request to the resource provider 1304 in the first cloud environment 1302, and accordingly, the resource provider 1304 can facilitate the provisioning of cross-cloud services together with the second cloud environment 1318.

[0189] Similarly, managing the lifecycle of provisioned cross-cloud services between the first cloud environment 1302 and the second cloud environment 1318 can be facilitated by the resource provider 1304 of the first cloud environment 1302 and the resource manager 1324 of the second cloud environment 1318. For example, the resource manager 1324 of the second cloud environment 1318 may send requests to the resource provider 1304 to manage the lifecycle of provisioned cross-cloud services (e.g., requests to terminate cross-cloud services), and in turn, the resource provider 1304 can facilitate the cross-cloud service lifecycle management function together with the second cloud environment 1318. In some implementations, to facilitate provisioning and lifecycle management, the resource manager 1324 may be configured to communicate with the resource provider 1304 (e.g., using APIs of the first cloud environment 1302 exposed to the resource manager 1324 by the resource provider 1304).

[0190] In some implementations, a customer of the second cloud environment 1318 and / or the second CSP who wishes to provision cross-cloud services from within the second cloud environment 1318 and / or manage cross-cloud services provisioned together with the second cloud environment 1318 can initiate the execution of request 1336. The customer can initiate request 1336 through the portal 1320 of the second cloud environment 1318. In some implementations, the portal 1320 may include one or more graphical user interfaces that can be accessed via a client device such as a computer (e.g., via an application, operating system, and / or software program running on the client device). One or more graphical user interfaces or a part thereof may be generated, entered, and / or otherwise provided by the resource manager 1324 of the second cloud environment 1318. Customers of the second cloud environment 1318 can access the portal 1320 to operate and / or interact with one or more graphical user interfaces to initiate a request 1336 and perform other functions such as managing tenancies within the second cloud environment 1318.

[0191] In some implementations, one or more graphical user interfaces or parts thereof may be generated, populated, and / or provided by a resource provider 1304 of the first cloud environment 1302. One or more graphical user interfaces or parts thereof may be provided to and / or declared in portal 1320 using blade 1322 of portal 1320. Blade 132 may serve and / or function as an extension, plug-in, add-on, etc., of portal 1320. Customers of both the first cloud environment 1302 and the second cloud environment 1318 can access portal 1320 to operate and / or interact with one or more graphical user interfaces to initiate a request 1336 and perform other functions such as managing tenancies in the first cloud environment 1302 and the second cloud environment 1318.

[0192] In some implementations, a request to provision a cross-cloud service received via portal 1320 can be routed to resource manager 1324 in a second cloud environment 1318, which can then route request 1336 to resource provider 1304 in the first cloud environment 1302 (for example, via a first set of APIs of the first cloud environment 1302 exposed within resource manager 1324). On the other hand, a request to manage the lifecycle of a provisioned cross-cloud service (for example, to view analytics, consumption, costs, logs, etc.) received via portal 1320 can be routed to console 1314 in the first cloud environment 1302, which can then route request 1336 within the first cloud environment 1302 (for example, via a second API exposed within the first cloud environment 1302). For example, console 1314 may be configured to route requests 1336 to analytics service 1316 in the first cloud environment 1302 to display analytics, consumption, costs, logs, etc., regarding provisioned cross-cloud services. In some implementations, each customer in the second cloud environment 1318 may be assigned a respective identifier so that each request 1336 initiated by each customer in the second cloud environment 1318 can be associated with the identifier of that respective customer. In this way, access to the portal and initiation of requests can be controlled and managed by resource manager 1324 based on the roles and / or permissions associated with each customer identifier.

[0193] To facilitate the provisioning of cross-cloud services and the lifecycle management of provisioned cross-cloud services, resource provider 1304 can be linked to resource manager 1324 and RPaaS (resource provider as a service) 1328 of a second cloud environment 1318. Linking resource provider 1304 to the second cloud environment 1318 enables resource provider 1304, resource manager 1324, and RPaaS 1328 to coordinate resources and operations. Resource provider 1304 can have an identity associated with the second cloud environment 1318. This identity can be configured to replicate the identity that resource provider 1304 has in the first cloud environment 1302. To facilitate the linking, resource provider 1304 can provide configuration 1326 to resource manager 1324 and configuration 1330 to RPaaS 1328. Configurations 1326 and 1330 may include an identifier for resource provider 1304 and may define the API specifications, connection endpoints, and / or locations of the in-cloud services associated with resource provider 1304.

[0194] The resource provider 1304 may be configured to provision cross-cloud services and / or manage the lifecycle of the provisioned cross-cloud services based on the operations performed by the multi-cloud platform 1312 in the first cloud environment 1302. The multi-cloud platform 1312 may be configured to perform operations common to linking and integrating the first cloud environment 1302 with the second cloud environment 1318 and other cloud environments of other CSPs. For example, the multi-cloud platform 1312 may be configured to link customer accounts of the second CSP to customer accounts of the first CSP, publish observations collected from the first cloud environment 1302 and / or the second cloud environment 1318, generate sales tokens for access to other cloud environments of customers, and so on. Furthermore, the multi-cloud platform 1312 may be configured to create, define, provide, and / or otherwise implement contracts between the resource provider 1304, the resource manager 1324, and the RPaaS 1328. This contract can identify the resources supported by resource provider 1304, resource manager 1324, and RPaaS 1328, as well as the operations to be performed. In some implementations, this contract can enable resource provider 1304, resource manager 1324, and RPaaS 1328 to operate on the same tenancy within a first cloud environment 1302. For example, this contract can define provisioning and / or lifecycle management events sent from resource manager 1324 and / or RPaaS 1328 when such events occur, and resource provider 1304 may be configured to asynchronously perform provisioning and / or lifecycle management operations based on the receipt of such events and / or changes in the state of such events.

[0195] The multi-cloud platform 1312 may be further configured to perform generalized cloud management operations, including, but not limited to, (i) mapping subscriptions and tenancies of a second cloud environment 1318 to subscriptions and tenancies of a first cloud environment 1302; (ii) generating and managing policy statements that control tenancies in each cloud environment (e.g., statements that facilitate the multi-cloud platform 1312 and resource provider 1304 operating within the same tenancy); (iii) generating and managing access tokens that facilitate cross-cloud access and / or communication between each cloud environment; and / or (iv) mapping observability information of the first cloud environment 1302 to the second cloud environment 1318 (e.g., writing an event to the second cloud environment 1318 when a backup is complete in the first cloud environment 1302, writing resource logs from the first cloud environment 1302 to the second cloud environment 1318). In this way, the first cloud environment 1302 and the second cloud environment 1318 can avoid redundant operations and resources, and as a result, efficiency can be improved.

[0196] In some implementations, upon receiving a request 1336 to provision a cross-cloud service, the resource provider 1304 may be configured to map the request 1336 to an identifier in the first cloud environment 1302 and pass this request to the service control plane 1306. The service control plane 1306 can be configured to perform two main processes: the first is to provision the relevant resources in the first cloud environment 1302, and the second is to connect these resources to the customer's tenancy in the second cloud environment 1318 (e.g., the customer's VNET in the second cloud environment 1318). Following the first process, the service control plane 1306 creates a VCN for the customer in the first cloud environment 1302 and creates one or more subnets within this VCN. In some implementations, this VCN can function as a shadow tenancy for the customer's tenancy in the second cloud environment 1318. One or more subnets within the VCN and one or more subnets within the VNET may use the same CIDR. Furthermore, the service control plane 1306 creates a DRG within the first cloud environment 1302, connects this DRG to the customer's VCN, and configures routing information for the DRG and VCN (for example, to interconnect these two resources). In addition, the service control plane 1306 provisions a VM cluster in the service data plane 1332. The IP addresses of this VM cluster are IP addresses derived from CIDR and are mapped to corresponding DNS records. Following the second process, the service control plane 1306 registers these IP addresses in the second cloud environment 1318 so that a private DNS zone can be configured within the second cloud environment 1318, creates a virtual circuit between the DRG and the network provider 1334, and sends DNS records.

[0197] Furthermore, or alternatively, upon receiving a request 1336 to provision a cross-cloud service, resource provider 1304 is configured to: (i) map the identifier assigned to the customer by the second CSP to the identifier assigned to the customer by the first CSP; (ii) retrieve the identity of resource provider 1304 associated with request 1336 (i.e., the second cloud environment 1318 from which request 1336 originates); (iii) convert the format of request 1336 from the format of the second cloud environment 1318 to the format of the first cloud environment 1302, and route the formatter request, along with the mapped identifier, to the service control plane 1306 and the network control plane 1308; (iv) the first cloud environment 1302 and the (v) establish the network connectivity prerequisites for network connectivity to the second cloud environment 1318 (e.g., establish a DRG, VCN, and subnet to connect to the network provider 1334 and the VMs of the service data plane 1332 in the second cloud environment 1318), and (v) establish network connectivity between the network control plane 1308 and the network provider 1334 based on the network connectivity prerequisites (e.g., link the delegated subnet of the second cloud environment 1318 to the VCN of the first cloud environment 1302 and configure DNS entries within the second cloud environment 1318). Furthermore, after the cross-cloud service has been provisioned, the resource provider 1304 may be configured to maintain metadata between the first cloud environment 1302 and the second cloud environment 1318 (e.g., a mapping between identifiers of the second cloud environment 1318 and the first cloud environment 1302), and to act as a thin adapter layer that receives requests formatted for the second cloud environment 1318 and already authenticated by the resource manager 1324 (e.g., requests to manage the lifecycle of the provisioned cross-cloud service), converts these requests into requests formatted for the first cloud environment 1302, and delegates the converted requests to the service control plane 1306.

[0198] Exemplary Method Figure 14 shows an exemplary process 1400 for provisioning and managing cross-cloud services between multiple cloud environments. The processes shown in Figure 14 may be implemented in software (e.g., code, instructions, programs), hardware, or a combination thereof, executed by one or more processing units (e.g., one or more processors, cores) of each system described throughout. The software may be stored in a non-temporary storage medium (e.g., a memory device). The method presented in Figure 14 illustrates various processing steps occurring in a particular sequence or order, but this is not intended to be limiting. In some alternative implementations, these steps may be executed in parallel and / or in a different order. In some implementations, such as the embodiments shown in Figures 6 to 13, the processes shown in Figure 14 may be executed by physical components of cloud environments, such as a first cloud environment 1302 and / or a second cloud environment 1318.

[0199] In block 1402, the first private cloud in the first cloud environment is integrated with the second private cloud in the second cloud environment. In some implementations, the first private cloud is integrated with the second private cloud by linking identifiers associated with components of the first private cloud to identifiers associated with components of the second private cloud. In some implementations, components of the first private cloud can be resource providers of the first private cloud, and components of the second private cloud can be resource managers and / or RPaaS of the second private cloud. In some implementations, linking resource providers of the first private cloud and / or cloud environment to the second private cloud and / or cloud environment allows resource providers, resource managers, and RPaaS to coordinate resources and operations. Resource providers can have an identity associated with the second cloud environment. This identity can be configured to replicate the identity that the resource provider has in the first cloud environment. To facilitate the linking, resource providers can provide configurations to resource managers and configurations to RPaaS. One or more of the configurations may include a resource provider identifier and define the API specifications, connection endpoints, and / or locations of the in-cloud services associated with the resource provider.

[0200] In block 1404, a determination is made as to whether a request for the cloud service has been received. In response to receiving the request, the flow can proceed to block 1406. If the request has not been received, another determination may be made as to whether a request for the cloud service has been received. In some implementations, this request may be received by a component of the first private cloud and / or cloud environment from a component of the second private cloud and / or cloud environment. In some implementations, this request is initiated in response to a customer device interacting with a portal in the second private cloud. The customer device may be associated with a customer account corresponding to the second private cloud. In some implementations, the portal is configured to present a portion of a graphical user interface provided by the first component. This request may be initiated in response to a customer device interacting with this portion. In some implementations, the cloud service may be a cross-cloud service provided by the first CSP, which provides the first cloud environment to customers of the second CSP providing the second cloud environment and / or end users having tenancy in the second cloud environment.

[0201] In some implementations, customers of a second cloud environment and / or a second CSP who wish to provision cross-cloud services from within the second cloud environment and / or manage cross-cloud services provisioned together with the second cloud environment can initiate the execution of a request through a portal in the second cloud environment. In some implementations, the portal may include one or more graphical user interfaces that can be accessed via a client device such as a computer (e.g., via an application, operating system, and / or software program running on the client device). One or more graphical user interfaces or a portion thereof may be generated, populated, and / or otherwise provided by the resource manager of the second cloud environment. Customers of the second cloud environment can access the portal to operate and / or interact with one or more graphical user interfaces to initiate a request and perform other functions such as managing tenancies within the second cloud environment.

[0202] In some implementations, one or more graphical user interfaces, or parts thereof, may be generated, populated, and / or otherwise provided by a resource provider in the first cloud environment. One or more graphical user interfaces, or parts thereof, may be provided to and / or declared in the portal using a portal blade. The blade may serve and / or function as an extension, plug-in, add-on, etc., of the portal. Customers in both the first and second cloud environments can access the portal to operate and / or interact with one or more graphical user interfaces to initiate requests and perform other functions such as managing tenancies in the first and second cloud environments.

[0203] In some implementations, requests to provision cross-cloud services received via the portal can be routed to a resource manager in the second cloud environment, which can then route the requests to resource providers in the first cloud environment (e.g., via a first set of APIs in the first cloud environment exposed within the resource manager). On the other hand, requests to manage the lifecycle of provisioned cross-cloud services (e.g., to view analytics, consumption, costs, logs, etc.) received via the portal can be routed to a console in the first cloud environment, which can then route the requests within the first cloud environment (e.g., via a second API exposed within the first cloud environment). For example, the console may be configured to route requests to an analytics service in the first cloud environment to view analytics, consumption, costs, logs, etc., regarding provisioned cross-cloud services. In some implementations, each customer in the second cloud environment may be assigned a unique identifier so that each request initiated by each customer in the second cloud environment can be associated with that customer's identifier. In this way, access to the portal and the initiation of requests can be controlled and managed by the resource manager based on the roles and / or permissions associated with each customer identifier.

[0204] In block 1406, a set of actions related to provisioning cloud services in the second private cloud and / or cloud environment are performed. In some implementations, the set of actions is performed by components of the first private cloud and / or cloud environment, and at least one of the actions includes establishing the prerequisites for network connectivity between the first private cloud and / or cloud environment and the second private cloud and / or cloud environment. In some implementations, establishing the prerequisites for network connectivity includes creating a first subnet in the first private cloud and creating a second subnet in the second private cloud. In some implementations, at least one Internet Protocol address associated with the first subnet and at least one Internet Protocol address associated with the second subnet are within a CIDR range.

[0205] In some implementations, upon receiving a request to provision cross-cloud services, the resource provider may be configured to: (i) map the identifier assigned to the customer by the second CSP to the identifier assigned to the customer by the first CSP; (ii) retrieve the identity of the resource provider associated with the request (i.e., the second cloud environment from which the request originates); (iii) convert the format of the request from the format of the second cloud environment to the format of the first cloud environment and route the formatter request, along with the mapped identifier, to the service control plane and network control plane; and (iv) establish the network connectivity prerequisites for network connectivity between the first and second cloud environments (e.g., establishing DRG, VCN, and subnets to connect to the network provider and service data plane VMs of the second cloud environment). In some implementations, upon receiving a request to provision cross-cloud services, the resource provider may be configured to map the request to an identifier in the first cloud environment and pass this request to the service control plane of the first cloud environment.

[0206] In block 1408, one or more second components of the first private cloud and / or cloud environment provision cloud services in the second private cloud and / or cloud environment. In some implementations, the first component of the first private cloud and / or cloud environment provision cloud services within the second private cloud and / or cloud environment. In some implementations, having one or more second components of the first private cloud and / or cloud environment provision cloud services in the second private cloud and / or cloud environment includes having one or more second components establish network connectivity between the first private cloud and / or cloud environment and the second private cloud and / or cloud environment based on network connectivity prerequisites. In some implementations, having one or more second components of the first private cloud and / or cloud environment provision cloud services in the second private cloud and / or cloud environment includes having another component of the second private cloud and / or cloud environment provision virtual machines.

[0207] In some implementations, after performing a series of actions, the resource provider may be configured to establish network connectivity between the network control plane of the first cloud environment and the network provider of the second cloud environment based on network connectivity prerequisites (e.g., linking delegated subnets in the second cloud environment to the VCN of the first cloud environment and configuring DNS entries within the second cloud environment). Furthermore, after cross-cloud services are provisioned, the resource provider may be configured to maintain metadata between the first and second cloud environments (e.g., mapping between identifiers in the second cloud environment and the first cloud environment), and to act as a thin adapter layer that receives requests formatted for the second cloud environment and already authenticated by the resource manager (e.g., requests to manage the lifecycle of provisioned cross-cloud services), translates these requests into requests formatted for the first cloud environment, and delegates the translated requests to the service control plane.

[0208] In some implementations, after performing a series of operations, the service control plane may be configured to perform a first process for provisioning relevant resources in the first cloud environment, and a second process for connecting those resources to the customer's tenancy in the second cloud environment (e.g., the customer's VNET in the second cloud environment). According to the first process, the service control plane creates a VCN for the customer in the first cloud environment and creates one or more subnets within this VCN. In some implementations, this VCN can function as a shadow tenancy for the customer's tenancy in the second cloud environment. One or more subnets within the VCN and one or more subnets within the VNET may use the same CIDR. The service control plane also creates a DRG within the first cloud environment 1302, connects this DRG to the customer's VCN, and configures routing information for the DRG and VCN (e.g., for interconnecting these two resources). Furthermore, the service control plane provisions a VM cluster in the service data plane. The IP address of this VM cluster is an IP address derived from the CIDR and is mapped to the corresponding DNS record. In the second process, the service control plane registers these IP addresses in the second cloud environment so that a private DNS zone can be configured within the second cloud environment, creates a virtual circuit between the DRG and the network provider, and sends DNS records.

[0209] In some implementations, a resource provider may be configured to provision cross-cloud services and / or manage the lifecycle of provisioned cross-cloud services based on the actions performed by the multi-cloud platform in the first cloud environment. The multi-cloud platform may be configured to perform actions common to linking and integrating the first cloud environment with the second cloud environment and other cloud environments of other CSPs. For example, the multi-cloud platform may be configured to link customer accounts of the second CSP to customer accounts of the first CSP, expose observations collected from the first and / or second cloud environments, and generate sales tokens for access to the customer's other cloud environments, etc. Furthermore, the multi-cloud platform may be configured to create, define, provide, and / or otherwise implement contracts between resource providers, resource managers, and RPaaS. These contracts can identify the resources supported by the resource provider, resource manager, and RPaaS, and / or the actions performed. In some implementations, these contracts can enable the resource provider, resource manager, and RPaaS to operate on the same tenancy within the first cloud environment. For example, this contract may define provisioning and / or lifecycle management events sent from a resource manager and / or RPaaS when such events occur, and the resource provider may be configured to perform provisioning and / or lifecycle management actions asynchronously based on the receipt of such events and / or changes in the state of such events.

[0210] The multi-cloud platform may be further configured to perform generalized cloud management operations, including, but not limited to, (i) mapping subscriptions and tenancies of a second cloud environment to subscriptions and tenancies of a first cloud environment; (ii) generating and managing policy statements that control tenancies in each cloud environment (e.g., statements that facilitate the multi-cloud platform and resource providers operating within the same tenancy); (iii) generating and managing access tokens that facilitate cross-cloud access and / or communication between each cloud environment; and / or (iv) mapping observability information of the first cloud environment to the second cloud environment (e.g., writing an event to the second cloud environment when a backup is complete in the first cloud environment, writing resource logs from the first cloud environment to the second cloud environment). In this way, the first and second cloud environments can avoid redundant operations and resources, resulting in improved efficiency.

[0211] Examples of cloud infrastructure As mentioned earlier, IaaS is a specific type of cloud computing. IaaS can be configured to provide virtualized computing resources over a public network (e.g., the internet). In the IaaS model, a cloud computing provider can host infrastructure components (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer)). In some cases, the IaaS provider may also provide various services that accompany those infrastructure components (e.g., services may include billing software, monitoring software, logging software, load balancing software, clustering software, etc.). Therefore, since these services can be policy-driven, IaaS users may implement policies to drive load balancing to maintain application availability and performance.

[0212] In some cases, IaaS customers may access resources and services over a wide area network (WAN), such as the internet, and use the cloud provider's services to install the rest of their application stack. For example, a user might log into an IaaS platform, create virtual machines (VMs), install an operating system (OS) on each VM, deploy middleware such as databases, create storage buckets for workloads and backups, and install enterprise software on the VMs. The customer can then use the provider's services to perform a variety of functions, including balancing network traffic, troubleshooting application issues, monitoring performance, and managing disaster recovery.

[0213] In most cases, the cloud computing model requires the participation of a cloud provider. A cloud provider may, but does not have to, be a third-party service specializing in providing IaaS (e.g., offering, leasing, or selling). The entity may choose to deploy a private cloud and become its own provider of infrastructure services.

[0214] In some cases, IaaS deployment is the process of connecting a new application or a new version of an application to a prepared application server, etc. This process may include preparing the server (e.g., installing libraries, daemons, etc.). This process is often managed by the cloud provider under the hypervisor layer (e.g., servers, storage, network hardware, and virtualization). Therefore, the customer may be responsible for handling the deployment of the OS, middleware, and / or application (e.g., on self-service virtual machines, etc., which can be spun up on demand).

[0215] In some examples, IaaS provisioning may also refer to acquiring computers or virtual hosts for use and installing the necessary libraries or services on those computers or virtual hosts. In most cases, deployment does not include provisioning, and provisioning may need to be performed first.

[0216] In some cases, IaaS provisioning presents two distinct challenges. First, there is the initial challenge of provisioning an initial set of infrastructure before anything is executed. Second, there is the challenge of developing the existing infrastructure after everything has been provisioned (e.g., adding new services, modifying services, removing services, etc.). In some cases, these two challenges can be addressed by allowing the infrastructure configuration to be defined declaratively. In other words, the infrastructure (e.g., which components are needed and how those components interact) can be defined by one or more configuration files. In this way, the entire infrastructure topology (e.g., which resources depend on which resources and how each of those resources works together) can be described declaratively. In some cases, after the topology is defined, a workflow may be generated to create and / or manage the various components described in the configuration files.

[0217] In some examples, the infrastructure may include many interconnected elements. For example, there may be one or more virtual private clouds (VPCs), also known as core networks (e.g., configurable and / or shared computing resources, possibly on-demand pools). In some examples, there may also be one or more inbound / outbound traffic group rules provisioned to define how inbound and / or outbound traffic on the network is configured, as well as one or more virtual machines (VMs). Other infrastructure elements such as load balancers and databases may also be provisioned. The infrastructure can evolve gradually as more infrastructure elements are desired and / or added.

[0218] In some cases, continuous deployment techniques may be employed to enable the deployment of infrastructure code across various virtual computing environments. Furthermore, the techniques described can enable infrastructure management within these environments. In some examples, a service team may write code that is desirable to be deployed to one or more, but often many, different production environments (e.g., across various geographical locations, sometimes even across the entire world). However, in some examples, the infrastructure to which the code is deployed must be configured first. In some cases, provisioning can be done manually, and provisioning tools may be used to provision resources, and / or deployment tools may be used to deploy the code after the infrastructure has been provisioned.

[0219] Figure 15 is a block diagram 1500 showing an exemplary pattern of an IaaS architecture according to at least one embodiment. A service operator 1502 may be communicably coupled to a secure host tenancy 1504 which may include a VCN 1506 and a secure host subnet 1508. In some examples, the service operator 1502 may use one or more client computing devices, which may be portable handheld devices (e.g., iPhone®, mobile phones, iPad®, computing tablets, personal digital assistants (PDAs)) or wearable devices (e.g., Google Glass® head-mounted displays) with the Internet, email, short message service (SMS), Blackberry®, or other communication protocols enabled, running software such as Microsoft Windows Mobile® and / or various mobile operating systems such as iOS®, Windows Phone, Android, BlackBerry 8, Palm OS. Alternatively, a client computing device can be a general-purpose personal computer, including, for example, personal computers and / or laptop computers running various versions of Microsoft Windows®, Apple Macintosh®, and / or Linux® operating systems. A client computing device can also be a workstation computer running any of various commercially available UNIX® or UNIX-like operating systems, including, but not limited to, various GNU / Linux operating systems such as Google Chrome OS.Alternatively or additionally, the client computing device may be any other electronic device, such as a thin client computer, an internet-enabled gaming system (e.g., a Microsoft Xbox game console with or without a Kinect® gesture input device), and / or a personal messaging device, that can communicate over a network and / or the Internet that has access to VCN1506.

[0220] VCN1506 may include a local peering gateway (LPG) 1510, which can be communicatively coupled to Secure Shell (SSH) VCN1512 via LPG1510 contained within SSH VCN1512. SSH VCN1512 may contain an SSH subnet 1514, and SSH VCN1512 may be communicatively coupled to control plane VCN1516 via LPG1510 contained within control plane VCN1516. Additionally, SSH VCN1512 may be communicatively coupled to data plane VCN1518 via LPG1510. Control plane VCN1516 and data plane VCN1518 may be contained within a service tenancy 1519, which may be owned and / or operated by an IaaS provider.

[0221] The control plane VCN 1516 may include a control plane demilitarized zone (DMZ) layer 1520 that functions as a perimeter network (e.g., part of the corporate network between the corporate intranet and the external network). Servers based in the DMZ have limited responsibilities and can help keep breaches contained. Furthermore, the DMZ layer 1520 may include a control plane application layer 1524 that may include one or more load balancer (LB) subnets 1522, an application subnet 1526, and a control plane data layer 1528 that may include database (DB) subnets 1530 (e.g., a frontend DB subnet and / or a backend DB subnet). The LB subnet 1522 included in the control plane DMZ layer 1520 can be communicatively coupled to the application subnet 1526 included in the control plane application layer 1524 and to the Internet gateway 1534 which may be included in the control plane VCN 1516. The application subnet 1526 can be communicatively coupled to the DB subnet 1530 included in the control plane data layer 1528, as well as to the service gateway 1536 and the network address translation (NAT) gateway 1538. The control plane VCN 1516 may include the service gateway 1536 and the NAT gateway 1538.

[0222] The control plane VCN 1516 may include a data plane mirror application layer 1540 that can include an application subnet 1526. The application subnet 1526 included in the data plane mirror application layer 1540 may include a virtual network interface controller (VNIC) 1542 that can run compute instances 1544. The compute instance 1544 can communicate with the application subnet 1526 of the data plane mirror application layer 1540 and with the application subnet 1526 that may be included in the data plane application layer 1546.

[0223] The data plane VCN1518 may include a data plane application layer 1546, a data plane DMZ layer 1548, and a data plane data layer 1550. The data plane DMZ layer 1548 may include an LB subnet 1522 that can be communicatively coupled to the application subnet 1526 of the data plane application layer 1546 and the internet gateway 1534 of the data plane VCN1518. The application subnet 1526 may be communicatively coupled to the service gateway 1536 of the data plane VCN1518 and the NAT gateway 1538 of the data plane VCN1518. The data plane data layer 1550 may also include a DB subnet 1530 that can be communicatively coupled to the application subnet 1526 of the data plane application layer 1546.

[0224] The Internet gateways 1534 of the control plane VCN1516 and the data plane VCN1518 can be communicatively coupled to a metadata management service 1552, which can be communicatively coupled to the public internet 1554. The public internet 1554 can be communicatively coupled to the NAT gateways 1538 of the control plane VCN1516 and the data plane VCN1518. The service gateways 1536 of the control plane VCN1516 and the data plane VCN1518 can be communicatively coupled to a cloud service 1556.

[0225] In some cases, a service gateway 1536 of the control plane VCN1516 or the data plane VCN1518 can make application programming interface (API) calls to a cloud service 1556 without going through the public internet 1554. API calls from the service gateway 1536 to the cloud service 1556 can be one-way, with the service gateway 1536 making the API call to the cloud service 1556, and the cloud service 1556 sending the requested data to the service gateway 1536. However, the cloud service 1556 does not need to initiate the API call to the service gateway 1536.

[0226] In some examples, the secure host tenancy 1504 can be directly connected to the service tenancy 1519, or otherwise may be isolated. The secure host subnet 1508 can communicate with the SSH subnet 1514 via the LPG 1510, which can enable bidirectional communication on otherwise isolated systems. Connecting the secure host subnet 1508 to the SSH subnet 1514 may give the secure host subnet 1508 access to other entities within the service tenancy 1519.

[0227] The control plane VCN1516 may allow users of service tenancy 1519 to configure or otherwise provision desired resources. Desired resources provisioned within the control plane VCN1516 may be deployed or otherwise used in the data plane VCN1518. In some examples, the control plane VCN1516 can be isolated from the data plane VCN1518, and the data plane mirror application layer 1540 of the control plane VCN1516 can communicate with the data plane application layer 1546 of the data plane VCN1518 via a VNIC 1542 which may be included in the data plane mirror application layer 1540 and the data plane application layer 1546.

[0228] In some cases, a system user or customer may perform a request, such as a create, read, update, or delete (CRUD) operation, via the public internet 1554, which can then transmit the request to the metadata management service 1552. The metadata management service 1552 can then transmit the request to the control plane VCN 1516 via the internet gateway 1534. The request may be received by an LB subnet 1522, which is included in the control plane DMZ layer 1520. The LB subnet 1522 may determine that the request is valid, and in response to this determination, it may send the request to an application subnet 1526, which is included in the control plane application layer 1524. If the request is validated and requires a call to the public internet 1554, the call to the public internet 1554 may be sent to a NAT gateway 1538, which can make a call to the public internet 1554. Metadata that may be desirable to be stored by the request may be stored in the DB subnet 1530.

[0229] In some cases, the data plane mirror application layer 1540 can facilitate direct communication between the control plane VCN 1516 and the data plane VCN 1518. For example, it may be desirable that changes, updates, or other appropriate modifications to the configuration be applied to resources contained in the data plane VCN 1518. Through VNIC 1542, the control plane VCN 1516 can communicate directly with the resources contained in the data plane VCN 1518, thereby enabling it to perform changes, updates, or other appropriate modifications to the configuration of those resources.

[0230] In some embodiments, the control plane VCN1516 and data plane VCN1518 may be included in the service tenancy 1519. In this case, the system user or customer does not have to own or operate either the control plane VCN1516 or the data plane VCN1518. Instead, the IaaS provider may own or operate the control plane VCN1516 and data plane VCN1518, both of which may be included in the service tenancy 1519. This embodiment can enable network isolation, which can prevent the user or customer from interacting with other users' resources or other customers' resources. This embodiment may also enable the system user or customer to store databases privately without requiring them to rely on the public internet 1554 for storage, which may not have the desired level of threat protection.

[0231] In another embodiment, the LB subnet 1522 included in the control plane VCN 1516 may be configured to receive signals from the service gateway 1536. In this embodiment, the control plane VCN 1516 and the data plane VCN 1518 may be configured to be invoked by the IaaS provider's customer without calling the public internet 1554. The IaaS provider's customer may prefer this embodiment because the database used by the customer may be stored in a service tenancy 1519 that may be controlled by the IaaS provider and isolated from the public internet 1554.

[0232] Figure 16 is a block diagram 1600 showing another exemplary pattern of an IaaS architecture according to at least one embodiment. A service operator 1602 (e.g., service operator 1502 in Figure 15) may be communicatively coupled to a secure host tenancy 1604 (e.g., secure host tenancy 1504 in Figure 15), which may include a VCN 1606 (e.g., VCN 1506 in Figure 15) and a secure host subnet 1608 (e.g., secure host subnet 1508 in Figure 15). VCN 1606 may include an LPG 1610 (e.g., LPG 1510 in Figure 15), which may be communicatively coupled to an SSH VCN 1612 (e.g., SSH VCN 1512 in Figure 15) via a local peering gateway (LPG) 1510 included in a secure shell (SSH) VCN 1612. SSH VCN1612 may contain SSH subnet 1614 (e.g., SSH subnet 1514 in Figure 15), and SSH VCN1612 may be communicably coupled to control plane VCN1616 (e.g., control plane VCN1516 in Figure 15) via LPG1610 contained within control plane VCN1616. Control plane VCN1616 may contain service tenancy 1619 (e.g., service tenancy 1519 in Figure 15), and data plane VCN1618 (e.g., data plane VCN1518 in Figure 15) may contain customer tenancy 1621, which may be owned or operated by a user or customer of the system.

[0233] The control plane VCN1616 may include a control plane DMZ layer 1620 (e.g., control plane DMZ layer 1520 in Figure 15) which may include an LB subnet 1622 (e.g., LB subnet 1522 in Figure 15), a control plane application layer 1624 (e.g., control plane application layer 1524 in Figure 15) which may include an application subnet 1626 (e.g., application subnet 1526 in Figure 15), and a control plane data layer 1628 (e.g., control plane data layer 1528 in Figure 15) which may include a database (DB) subnet 1630 (e.g., similar to DB subnet 1530 in Figure 15). The LB subnet 1622 included in the control plane DMZ layer 1620 can be communicatively coupled to the application subnet 1626 included in the control plane application layer 1624 and to the Internet gateway 1634 (e.g., Internet gateway 1534 in Figure 15) which may be included in the control plane VCN 1616. The application subnet 1626 can be communicatively coupled to the DB subnet 1630 included in the control plane data layer 1628 and to the service gateway 1636 (e.g., service gateway 1536 in Figure 15) and the network address translation (NAT) gateway 1638 (e.g., NAT gateway 1538 in Figure 15). The control plane VCN 1616 may include the service gateway 1636 and the NAT gateway 1638.

[0234] The control plane VCN 1616 may include a data plane mirror application layer 1640 (e.g., data plane mirror application layer 1540 in Figure 15) which may include an application subnet 1626. The application subnet 1626 included in the data plane mirror application layer 1640 may include a virtual network interface controller (VNIC) 1642 (e.g., VNIC 1542) which can run a compute instance 1644 (e.g., similar to compute instance 1544 in Figure 15). Compute instance 1644 can facilitate communication between the application subnet 1626 of the data plane mirror application layer 1640 and the application subnet 1626 that may be included in the data plane application layer 1646 (e.g., data plane application layer 1546 in Figure 15) via the VNIC 1642 included in the data plane mirror application layer 1640 and the VNIC 1642 included in the data plane application layer 1646.

[0235] The Internet gateway 1634 included in the control plane VCN1616 can be communicatively coupled to a metadata management service 1652 (e.g., metadata management service 1552 in Figure 15), which can be communicatively coupled to the public internet 1654 (e.g., public internet 1554 in Figure 15). The public internet 1654 can be communicatively coupled to a NAT gateway 1638 included in the control plane VCN1616. The service gateway 1636 included in the control plane VCN1616 can be communicatively coupled to a cloud service 1656 (e.g., cloud service 1556 in Figure 15).

[0236] In some examples, the data plane VCN1618 may be included in customer tenancy 1621. In this case, the IaaS provider may provide a control plane VCN1616 per customer, and the IaaS provider may configure a unique compute instance 1644 for each customer, included in service tenancy 1619. Each compute instance 1644 may enable communication between the control plane VCN1616 included in service tenancy 1619 and the data plane VCN1618 included in customer tenancy 1621. The compute instance 1644 may enable resources provisioned within the control plane VCN1616 included in service tenancy 1619 to be deployed, or otherwise used, in the data plane VCN1618 included in customer tenancy 1621.

[0237] In another example, an IaaS provider's customer may have a database that persists in customer tenancy 1621. In this example, control plane VCN 1616 may include a data plane mirror app layer 1640 that can include an app subnet 1626. The data plane mirror app layer 1640 may reside in data plane VCN 1618, but does not have to reside in data plane VCN 1618. That is, the data plane mirror app layer 1640 may have access rights to customer tenancy 1621, but does not have to reside in data plane VCN 1618, and does not have to be owned or operated by an IaaS provider's customer. The data plane mirror app layer 1640 may be configured to make calls to data plane VCN 1618, but does not have to be configured to make calls to any entities contained in control plane VCN 1616. Customers may wish to deploy or otherwise use resources in the data plane VCN1618 that are provisioned within the control plane VCN1616, and the data plane mirror application layer 1640 can facilitate the customer's desired deployment or other use of resources.

[0238] In some embodiments, a customer of the IaaS provider can apply filters to the data plane VCN1618. In this embodiment, the customer can determine which data plane VCN1618s are accessible, and the customer may restrict access from the data plane VCN1618 to the public internet 1654. The IaaS provider does not need to be able to apply filters or otherwise control access of the data plane VCN1618 to any external network or database. Applying filters and controls to the data plane VCN1618 included in the customer tenancy 1621 by the customer can help isolate the data plane VCN1618 from other customers and from the public internet 1654.

[0239] In some embodiments, cloud service 1656 may be invoked by service gateway 1636 to access services that may not exist on the public internet 1654, control plane VCN 1616, or data plane VCN 1618. The connection between cloud service 1656 and control plane VCN 1616 or data plane VCN 1618 may not be operational or continuous. Cloud service 1656 may reside on different networks owned or operated by the IaaS provider. Cloud service 1656 may be configured to receive calls from service gateway 1636 and not to receive calls from the public internet 1654. Some cloud services 1656 may be isolated from other cloud services 1656, and control plane VCN 1616 may be isolated from cloud services 1656 that may not be in the same region as control plane VCN 1616. For example, the control plane VCN1616 may be located in "Region 1," and the cloud service "Deployment 19" may be located in both Region 1 and "Region 2." If a call to Deployment 19 is made by a service gateway 1636 included in the control plane VCN1616 located in Region 1, this call may be sent to Deployment 19 in Region 1. In this example, the control plane VCN1616, or Deployment 19 in Region 1, may be communicatively coupled to Deployment 19 in Region 2, or otherwise not have to communicate with Deployment 19 in Region 2.

[0240] Figure 17 is a block diagram 1700 showing another exemplary pattern of an IaaS architecture according to at least one embodiment. A service operator 1702 (e.g., service operator 1502 in Figure 15) may be communicatively coupled to a secure host tenancy 1704 (e.g., secure host tenancy 1504 in Figure 15), which may include a VCN 1706 (e.g., VCN 1506 in Figure 15) and a secure host subnet 1708 (e.g., secure host subnet 1508 in Figure 15). VCN 1706 may include an LPG 1710 (e.g., LPG 1510 in Figure 15), which may be communicatively coupled to an SSH VCN 1712 (e.g., SSH VCN 1512 in Figure 15) via an LPG 1710 contained in an SSH VCN 1712. SSH VCN1712 may contain SSH subnet 1714 (e.g., SSH subnet 1514 in Figure 15), and SSH VCN1712 may be communicably coupled to control plane VCN1716 (e.g., control plane VCN1516 in Figure 15) via LPG1710 contained in control plane VCN1716, and to data plane VCN1718 (e.g., data plane 1518 in Figure 15) via LPG1710 contained in data plane VCN1718. Control plane VCN1716 and data plane VCN1718 may be contained in service tenancy 1719 (e.g., service tenancy 1519 in Figure 15).

[0241] The control plane VCN1716 may include a control plane DMZ layer 1720 (e.g., control plane DMZ layer 1520 in Figure 15) which may include a load balancer (LB) subnet 1722 (e.g., LB subnet 1522 in Figure 15), a control plane application layer 1724 (e.g., control plane application layer 1524 in Figure 15) which may include an application subnet 1726 (e.g., similar to application subnet 1526 in Figure 15), and a control plane data layer 1728 (e.g., control plane data layer 1528 in Figure 15) which may include a DB subnet 1730. The LB subnet 1722 included in the control plane DMZ layer 1720 can be communicatively coupled to the application subnet 1726 included in the control plane application layer 1724, and to an internet gateway 1734 (e.g., internet gateway 1534 in Figure 15) which may be included in the control plane VCN 1716. The application subnet 1726 can be communicatively coupled to the DB subnet 1730 included in the control plane data layer 1728, and to a service gateway 1736 (e.g., service gateway in Figure 15) and a network address translation (NAT) gateway 1738 (e.g., NAT gateway 1538 in Figure 15). The control plane VCN 1716 may include the service gateway 1736 and the NAT gateway 1738.

[0242] The data plane VCN1718 may include a data plane application layer 1746 (e.g., data plane application layer 1546 in Figure 15), a data plane DMZ layer 1748 (e.g., data plane DMZ layer 1548 in Figure 15), and a data plane data layer 1750 (e.g., data plane data layer 1550 in Figure 15). The data plane DMZ layer 1748 may include a trusted application subnet 1760 and an untrusted application subnet 1762 of the data plane application layer 1746, as well as an LB subnet 1722 that can be communicatively coupled to an internet gateway 1734 included in the data plane VCN1718. The trusted application subnet 1760 may be communicatively coupled to a service gateway 1736 included in the data plane VCN1718, a NAT gateway 1738 included in the data plane VCN1718, and a DB subnet 1730 included in the data plane data layer 1750. The untrusted application subnet 1762 can be communicatively coupled to the service gateway 1736 included in the data plane VCN 1718, and to the DB subnet 1730 included in the data plane data layer 1750. The data plane data layer 1750 may include the DB subnet 1730, which can be communicatively coupled to the service gateway 1736 included in the data plane VCN 1718.

[0243] An untrusted application subnet 1762 may include one or more primary VNICs 1764(1)-(N) that can be communicatively coupled to tenant virtual machines (VMs) 1766(1)-(N). Each tenant VM 1766(1)-(N) may be communicatively coupled to each application subnet 1767(1)-(N) that may be included in each container output VCN 1768(1)-(N) that may be included in each customer tenancy 1770(1)-(N). Each secondary VNIC 1772(1)-(N) can facilitate communication between the untrusted application subnet 1762 included in the data plane VCN 1718 and the application subnet included in the container output VCN 1768(1)-(N). Each container output VCN 1768(1)-(N) may include a NAT gateway 1738 that can be communicatively coupled to the public internet 1754 (e.g., public internet 1554 in Figure 15).

[0244] The Internet gateway 1734, included in the control plane VCN1716 and the data plane VCN1718, can be communicatively coupled to a metadata management service 1752 (e.g., the metadata management system 1552 in Figure 15), which can be communicatively coupled to the public internet 1754. The public internet 1754 can be communicatively coupled to a NAT gateway 1738, included in the control plane VCN1716 and the data plane VCN1718. The service gateway 1736, included in the control plane VCN1716 and the data plane VCN1718, can be communicatively coupled to a cloud service 1756.

[0245] In some embodiments, the data plane VCN1718 may be integrated with a customer tenancy 1770. This integration may be useful or desirable for the IaaS provider's customer in certain cases, such as when they may want support when executing code. The customer may provide code that may be destructive, communicate with other customers' resources, or otherwise cause undesirable effects. In response to this, the IaaS provider may determine whether it should execute code provided to the IaaS provider by the customer.

[0246] In some cases, an IaaS provider's customer might request the ability to grant the IaaS provider temporary network access privileges and connect to the data plane application layer 1746. The code to perform this function may run in VMs 1766(1) to (N), and this code does not need to be configured to run elsewhere on the data plane VCN 1718. Each VM 1766(1) to (N) may be connected to a single customer tenancy 1770. Each container 1771(1) to (N) contained within VMs 1766(1) to (N) may be configured to run the code. In this case, a double isolation may exist (for example, container 1771(1)~(N) running the code, and container 1771(1)~(N) may be contained in at least VM1766(1)~(N) which is contained in the untrusted app subnet 1762), which can help prevent incorrect or otherwise unwanted code from damaging the IaaS provider's network or the network of a different customer. Container 1771(1)~(N) may be communicably coupled to customer tenancy 1770 and may be configured to send or receive data to or from customer tenancy 1770. Container 1771(1)~(N) does not have to be configured to send or receive data to or from any other entity in the data plane VCN1718. Upon completion of code execution, the IaaS provider may terminate or otherwise discard container 1771(1)~(N).

[0247] In some embodiments, a trusted application subnet 1760 may execute code that may be owned or operated by the IaaS provider. In this embodiment, the trusted application subnet 1760 may be communicatively coupled to a DB subnet 1730 and configured to perform CRUD operations within the DB subnet 1730. An untrusted application subnet 1762 may be communicatively coupled to the DB subnet 1730, but in this embodiment, the untrusted application subnet may be configured to perform read operations within the DB subnet 1730. Containers 1771(1)~(N) that may be contained within each customer's VM 1766(1)~(N) and can execute code from the customer do not need to be communicatively coupled to the DB subnet 1730.

[0248] In other embodiments, the control plane VCN1716 and the data plane VCN1718 do not need to be directly and communicatively coupled. In this embodiment, direct communication between the control plane VCN1716 and the data plane VCN1718 is not required. However, communication can occur indirectly by at least one method. The LPG1710 may be established by the IaaS provider and can facilitate communication between the control plane VCN1716 and the data plane VCN1718. In another example, the control plane VCN1716 or the data plane VCN1718 can make a call to the cloud service 1756 via the service gateway 1736. For example, a call from the control plane VCN1716 to the cloud service 1756 may include a request for a service that can communicate with the data plane VCN1718.

[0249] Figure 18 is a block diagram 1800 showing another exemplary pattern of an IaaS architecture according to at least one embodiment. A service operator 1802 (e.g., service operator 1502 in Figure 15) may be communicatively coupled to a secure host tenancy 1804 (e.g., secure host tenancy 1504 in Figure 15), which may include a VCN 1806 (e.g., VCN 1506 in Figure 15) and a secure host subnet 1808 (e.g., secure host subnet 1508 in Figure 15). VCN 1806 may include an LPG 1810 (e.g., LPG 1510 in Figure 15), which may be communicatively coupled to an SSH VCN 1812 (e.g., SSH VCN 1512 in Figure 15) via an LPG 1810 contained in the SSH VCN 1812. SSH VCN1812 may contain SSH subnet 1814 (e.g., SSH subnet 1514 in Figure 15), and SSH VCN1812 may be communicably coupled to control plane VCN1816 (e.g., control plane VCN1516 in Figure 15) via LPG1810 contained in control plane VCN1816, and to data plane VCN1818 (e.g., data plane 1518 in Figure 15) via LPG1810 contained in data plane VCN1818. Control plane VCN1816 and data plane VCN1818 may be contained in service tenancy 1819 (e.g., service tenancy 1519 in Figure 15).

[0250] The control plane VCN1816 may include a control plane DMZ layer 1820 (e.g., control plane DMZ layer 1520 in Figure 15) which may include an LB subnet 1822 (e.g., LB subnet 1522 in Figure 15), a control plane application layer 1824 (e.g., control plane application layer 1524 in Figure 15) which may include an application subnet 1826 (e.g., application subnet 1526 in Figure 15), and a control plane data layer 1828 (e.g., control plane data layer 1528 in Figure 15) which may include a DB subnet 1830 (e.g., DB subnet 1730 in Figure 17). The LB subnet 1822 included in the control plane DMZ layer 1820 can be communicatively coupled to the application subnet 1826 included in the control plane application layer 1824, and to an internet gateway 1834 (e.g., internet gateway 1534 in Figure 15) which may be included in the control plane VCN 1816. The application subnet 1826 can be communicatively coupled to the DB subnet 1830 included in the control plane data layer 1828, and to a service gateway 1836 (e.g., service gateway in Figure 15) and a network address translation (NAT) gateway 1838 (e.g., NAT gateway 1538 in Figure 15). The control plane VCN 1816 may include the service gateway 1836 and the NAT gateway 1838.

[0251] The data plane VCN 1818 may include a data plane application layer 1846 (e.g., data plane application layer 1546 in Figure 15), a data plane DMZ layer 1848 (e.g., data plane DMZ layer 1548 in Figure 15), and a data plane data layer 1850 (e.g., data plane data layer 1550 in Figure 15). The data plane DMZ layer 1848 may include trusted application subnets 1860 (e.g., trusted application subnet 1760 in Figure 17) and untrusted application subnets 1862 (e.g., untrusted application subnet 1762 in Figure 17) of the data plane application layer 1846, as well as an LB subnet 1822 that can be communicatively coupled to the internet gateway 1834 included in the data plane VCN 1818. A trusted application subnet 1860 can be communicatively coupled to a service gateway 1836 included in data plane VCN 1818, a NAT gateway 1838 included in data plane VCN 1818, and a DB subnet 1830 included in data plane data layer 1850. An untrusted application subnet 1862 can be communicatively coupled to a service gateway 1836 included in data plane VCN 1818, and a DB subnet 1830 included in data plane data layer 1850. The data plane data layer 1850 may include a DB subnet 1830 that can be communicatively coupled to a service gateway 1836 included in data plane VCN 1818.

[0252] An untrusted application subnet 1862 may include primary VNICs 1864(1)-(N) that can be communicatively coupled to tenant virtual machines (VMs) 1866(1)-(N) residing within the untrusted application subnet 1862. Each tenant VM 1866(1)-(N) can execute code within its respective container 1867(1)-(N) and may be communicatively coupled to an application subnet 1826 that may be contained in a dataplane application layer 1846 that may be contained in a container output VCN 1868. Each secondary VNIC 1872(1)-(N) can facilitate communication between the untrusted application subnet 1862 contained in a dataplane VCN 1818 and the application subnet contained in a container output VCN 1868. The container output VCN may include a NAT gateway 1838 that can be communicatively coupled to the public internet 1854 (e.g., public internet 54 in Figure 15).

[0253] The Internet gateway 1834, included in the control plane VCN1816 and the data plane VCN1818, can be communicatively coupled to a metadata management service 1852 (e.g., the metadata management system 1552 in Figure 15), which can be communicatively coupled to the public internet 1854. The public internet 1854 can be communicatively coupled to a NAT gateway 1838, included in the control plane VCN1816 and the data plane VCN1818. The service gateway 1836, included in the control plane VCN1816 and the data plane VCN1818, can be communicatively coupled to a cloud service 1856.

[0254] In some examples, the pattern shown by the architecture in block diagram 1800 of Figure 18 may be considered an exception to the pattern shown by the architecture in block diagram 1700 of Figure 17, which may be desirable for the IaaS provider's customers when the IaaS provider cannot communicate directly with the customers (e.g., disconnected regions). Each container 1867(1)-(N) contained within VM 1866(1)-(N) for each customer may be accessible in real time by the customer. Each container 1867(1)-(N) may be configured to make calls to each secondary VNIC 1872(1)-(N) contained in the application subnet 1826 of the data plane application layer 1846, which may be contained within the container output VCN 1868. The secondary VNICs 1872(1)-(N) may send the calls to the NAT gateway 1838, which may send the calls to the public internet 1854. In this example, containers 1867(1)-(N), which can be accessed in real time by customers, can be isolated from the control plane VCN1816 and from other entities included in the data plane VCN1818. Containers 1867(1)-(N) may also be isolated from other customer resources.

[0255] In another example, a customer can use containers 1867(1)-(N) to invoke cloud service 1856. In this example, the customer may execute code within containers 1867(1)-(N) that requests a service from cloud service 1856. Containers 1867(1)-(N) can send this request to secondary VNICs 1872(1)-(N), which can send this request to the NAT gateway, which can send this request to the public internet 1854. The public internet 1854 can send this request to the LB subnet 1822, which is included in the control plane VCN 1816, via the internet gateway 1834. In response to determining that this request is valid, the LB subnet can send this request to the application subnet 1826, which can send this request to cloud service 1856 via the service gateway 1836.

[0256] It should be understood that the IaaS architectures 1500, 1600, 1700, and 1800 shown in the figures may include components other than those shown. Furthermore, the embodiments shown in the figures are merely examples of cloud infrastructure systems that may incorporate embodiments of this disclosure. In some other embodiments, the IaaS system may include more or fewer components than those shown in the figures, may combine two or more components, or may have different configurations or arrangements of components.

[0257] In some embodiments, the IaaS system described herein may include the provision of a set of applications, middleware, and database services delivered to customers in a self-service, subscription-based, elastically scalable, reliable, highly available, and secure manner. An example of such an IaaS system is the Oracle Cloud Infrastructure (OCI) provided by the Assignee.

[0258] Figure 19 shows an exemplary computer system 1900 in which various embodiments may be implemented. System 1900 may be used to implement any of the computer systems described above. As shown in the figure, computer system 1900 includes a processing unit 1904 that communicates with several peripheral subsystems via a bus subsystem 1902. These peripheral subsystems may include a processing acceleration unit 1906, an I / O subsystem 1908, a storage subsystem 1918, and a communication subsystem 1924. The storage subsystem 1918 includes a tangible computer-readable storage medium 1922 and system memory 1910.

[0259] The bus subsystem 1902 provides a mechanism for various components and subsystems of the computer system 1900 to communicate with each other as intended. Although the bus subsystem 1902 is schematically shown as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses. The bus subsystem 1902 may be one of several types of bus structures, including a memory bus or memory controller, a peripheral bus, and a local bus using any of the various bus architectures. For example, such architectures may include the ISA (Industry Standard Architecture) bus, the MCA (Micro Channel Architecture) bus, the EISA (Enhanced ISA) bus, the VESA (Video Electronics Standards Association) local bus, and the PCI (Peripheral Component Interconnect) bus, which can be implemented as a mezzanine bus manufactured according to the IEEE P1386.1 standard.

[0260] A processing unit 1904, which can be implemented as one or more integrated circuits (e.g., conventional microprocessors or microcontrollers), controls the operation of the computer system 1900. One or more processors may be included in the processing unit 1904. These processors may include single-core processors or multi-core processors. In one embodiment, the processing unit 1904 may be implemented as one or more independent processing units 1932 and / or 1934, each containing a single-core processor or a multi-core processor. In another embodiment, the processing unit 1904 may be implemented as a quad-core processing unit formed by integrating two dual-core processors onto a single chip.

[0261] In various embodiments, the processing unit 1904 can execute various programs depending on the program code and can maintain multiple programs or processes running simultaneously. At any given time, some or all of the program code to be executed may reside in the processor 1904 and / or the storage subsystem 1918. With appropriate programming, the processor 1904 can provide the various functions described above. The computer system 1900 may further include a processing acceleration unit 1906 which may include a digital signal processor (DSP), a dedicated processor, and / or similar.

[0262] The I / O subsystem 1908 may include user interface input devices and user interface output devices. User interface input devices may include pointing devices such as keyboards, mice or trackballs, touchpads or touchscreens integrated into displays, scroll wheels, click wheels, dials, buttons, switches, keypads, voice input devices with voice command recognition systems, microphones, and other types of input devices. User interface input devices may include motion detection devices and / or gesture recognition devices, such as Microsoft Kinect® motion sensors, which enable users to control and interact with input devices, such as Microsoft Xbox® 360 game controllers, through a natural user interface using gestures and spoken commands. User interface input devices may also include eye gesture recognition devices, such as Google Glass® blink detectors, which detect the user's eye activity (e.g., blinking when taking a picture and / or selecting a menu) and translate eye gestures into input to an input device (e.g., Google Glass®). Furthermore, the user interface input device may include a voice recognition detection device that allows the user to interact with a voice recognition system (e.g., Siri® Navigator) via voice commands.

[0263] User interface input devices may include, but are not limited to, three-dimensional (3D) mice, joysticks or pointing sticks, gamepads, and graphic tablets, as well as audio / visual devices such as speakers, digital cameras, digital video cameras, portable media players, webcams, image scanners, fingerprint scanners, barcode readers, 3D scanners, 3D printers, laser rangefinders, and eye-tracking devices. Furthermore, user interface input devices may include medical imaging input devices such as computed tomography, magnetic resonance imaging, positional emission tomography, and medical ultrasound imaging devices. User interface input devices may also include audio input devices such as MIDI keyboards and digital musical instruments.

[0264] User interface output devices may include non-visual displays such as display subsystems, indicator lights, or audio output devices. Display subsystems may include flat panel devices such as flat panel devices using cathode ray tubes (CRTs), liquid crystal displays (LCDs), or plasma displays, projection devices, touchscreens, etc. Generally, the use of the term “output device” is intended to include all possible types of devices and mechanisms for outputting information from the computer system 1900 to a user or another computer. For example, user interface output devices may include, but are not limited to, various display devices that visually convey text information, graphics information, and audio / video information, such as monitors, printers, speakers, headphones, car navigation systems, plotters, audio output devices, and modems.

[0265] The computer system 1900 may include a storage subsystem 1918 that provides a tangible, non-temporary, computer-readable storage medium for storing software and data structures that provide the functionality of the embodiments described herein. The software may include programs, code modules, instructions, scripts, etc., and, when executed by one or more cores or processors of the processing unit 1904, provides the aforementioned functionality. The storage subsystem 1918 may provide a repository for storing data used in accordance with this disclosure.

[0266] As shown in the example in Figure 19, the storage subsystem 1918 may include various components, including system memory 1910, computer-readable storage media 1922, and a computer-readable storage media reader 1920. System memory 1910 may store program instructions that can be read and executed by the processing unit 1904. System memory 1910 may also store data used during the execution of instructions and / or data generated during the execution of program instructions. Various different types of programs may be loaded into system memory 1910, including, but not limited to, client applications, web browsers, middle-tier applications, relational database management systems (RDBMS), virtual machines, and containers.

[0267] System memory 1910 may store an operating system 1916. Examples of operating systems 1916 include various versions of Microsoft Windows®, Apple Macintosh®, and / or Linux operating systems, various commercially available UNIX® or UNIX-like operating systems (including, but not limited to, various GNU / Linux operating systems, Google Chrome® OS, etc.), and / or mobile operating systems such as iOS, Windows® Phone, Android® OS, BlackBerry® OS, and Palm® OS. In one implementation in which computer system 1900 runs one or more virtual machines, the virtual machines may be loaded into system memory 1910 along with guest operating systems (GOS) and run by one or more processors or cores of processing unit 1904.

[0268] The system memory 1910 may be provided in different configurations depending on the type of computer system 1900. For example, the system memory 1910 may be volatile memory (such as random access memory (RAM)) and / or non-volatile memory (such as read-only memory (ROM) or flash memory). Various types of RAM configurations may be provided, including static random access memory (SRAM) and dynamic random access memory (DRAM). In some implementations, the system memory 1910 may include a basic input / output system (BIOS) that contains basic routines useful for transferring information between elements within the computer system 1900 during startup, etc.

[0269] In addition to a storage medium for temporarily and / or more permanently containing and storing computer-readable information for use by the computer system 1900, including instructions executable by the processing unit 1904 of the computer system 1900, the computer-readable storage medium 1922 may represent a remote storage device, a local storage device, a fixed storage device, and / or a removable storage device.

[0270] The computer-readable storage medium 1922 may include any suitable medium known or used in the art, including, but not limited to, storage and communication media, such as volatile and non-volatile, removable and non-removable media, implemented in any way or technique for storing and / or transmitting information. The computer-readable storage medium 1922 may also include tangible computer-readable storage media such as RAM, ROM, electronically erasable programmable ROM (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile disk (DVD), or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage, or other magnetic storage devices, or other tangible computer-readable media.

[0271] For example, computer-readable storage media 1922 may include hard disk drives that read from or write to non-removable non-volatile magnetic media, magnetic disk drives that read from or write to removable non-volatile magnetic disks, and optical disk drives that read from or write to removable non-volatile optical disks such as CD-ROMs, DVDs, and Blu-ray® discs, or other optical media. Computer-readable storage media 1922 may also include, but are not limited to, Zip® drives, flash memory cards, Universal Serial Bus (USB) flash drives, Secure Digital (SD) cards, DVD discs, and digital videotapes. Computer-readable storage media 1922 may include solid-state drives (SSDs) based on non-volatile memory, such as flash memory-based SSDs, enterprise flash drives, and semiconductor ROMs; SSDs based on volatile memory, such as semiconductor RAM, dynamic RAM, static RAM, DRAM-based SSDs, and magneto-resistive RAM (MRAM) SSDs; and hybrid SSDs using a combination of DRAM and flash memory-based SSDs. Disk drives and associated computer-readable media may provide non-volatile storage for computer-readable instructions, data structures, program modules, and other data of the computer system 1900.

[0272] Machine-readable instructions executable by one or more processors or cores of the processing unit 1904 may be stored in a non-temporary computer-readable storage medium. The non-temporary computer-readable storage medium may include physically tangible memory or storage devices, including volatile memory storage devices and / or non-volatile storage devices. Examples of non-temporary computer-readable storage media include magnetic storage media (e.g., disks or tapes), optical storage media (e.g., DVDs, CDs), various types of RAM, ROM, or flash memory, hard drives, floppy disk drives, removable memory drives (e.g., USB drives), or other types of storage devices.

[0273] The communication subsystem 1924 provides interfaces to other computer systems and networks. It functions as an interface for computer system 1900 to receive data from other systems and to transmit data to other systems. For example, the communication subsystem 1924 may enable computer system 1900 to connect to one or more devices via the Internet. In some embodiments, the communication subsystem 1924 may include radio frequency (RF) transceiver components for accessing wireless voice and / or data networks (e.g., cellular technology, 3G, 4G, or EDGE (enhanced data rates for global evolution)), Wi-Fi (using IEEE 802.11 group standards, other mobile communication technologies, or any combination thereof), global positioning system (GPS) receiver components, and / or other components. In some embodiments, the communication subsystem 1924 may provide wired network connectivity (e.g., Ethernet) in addition to, or instead of, wireless interfaces.

[0274] In some embodiments, the communication subsystem 1924 may receive input communications on behalf of one or more users who may be using the computer system 1900, in the form of structured and / or unstructured data feeds 1926, event streams 1928, event updates 1930, etc.

[0275] By way of example, communication subsystem 1924 may be configured to receive data feed 1926 in real time from users of social networks and / or other communication services such as Twitter® feeds, Facebook® updates, web feeds such as Rich Site Summary (RSS) feeds, and / or real-time updates from one or more third-party information sources.

[0276] Furthermore, communication subsystem 1924 may be configured to receive data in the form of a continuous data stream, which may have no explicit end, and may include an event stream 1928 and / or event updates 1930 of real-time events that may be substantially continuous or have no boundaries. Examples of applications that generate continuous data may include, for example, sensor data applications, financial tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, motor vehicle traffic monitoring, and the like.

[0277] Communication subsystem 1924 may be configured to output structured and / or unstructured data feeds 1926, event streams 1928, event updates 1930, and the like to one or more databases capable of communicating with one or more streaming data source computers coupled to computer system 1900.

[0278] Computer system 1900 may be one of various types, including handheld portable devices (e.g., iPhone® mobile phones, iPad® computing tablets, PDAs), wearable devices (e.g., Google Glass® head-mounted displays), PCs, workstations, mainframes, automatic ticket vending machines, server racks, or any other data processing system.

[0279] Due to the constantly changing nature of computers and networks, the description of the computer system 1900 illustrated in the drawings is merely intended to serve as a specific example. Many other configurations are possible that include more or fewer components than the system illustrated in the drawings. For example, customized hardware may be used, and / or certain elements may be implemented in hardware, firmware, software (including applets), or a combination thereof. Furthermore, connections to other computing devices such as network input / output devices may be employed. Based on the disclosure and teachings provided herein, those skilled in the art will understand other approaches and / or methods for implementing various embodiments.

[0280] While specific embodiments have been described, various modifications, changes, alternative constructions and equivalents are also encompassed within the scope of the present disclosure. Embodiments are not limited to operation in one particular data processing environment, but may operate freely in multiple data processing environments. Furthermore, while embodiments have been described using a specific sequence of transactions and steps, it should be apparent to those skilled in the art that the scope of the present disclosure is not limited to the described sequence of transactions and steps. Various features and aspects of the foregoing embodiments may be used individually or together.

[0281] Furthermore, while embodiments have been described using specific combinations of hardware and software, it should be recognized that other combinations of hardware and software are also within the scope of this disclosure. Embodiments may be implemented using hardware alone, software alone, or a combination thereof. Various processes described herein may be performed on different processors, either on the same processor or in any combination thereof. Thus, where a component or service is described as being configured to perform a certain operation, such configuration may be realized, for example, by designing electronic circuitry to perform this operation, by programming programmable electronic circuitry (such as a microprocessor) to perform this operation, or by any combination thereof. Processes may communicate using a variety of techniques, including but not limited to prior art for inter-process communication, and different pairs of processes may use different techniques, or the same pair of processes may use different techniques at different times.

[0282] Therefore, this specification and drawings should be considered illustrative, not limiting. However, it is clear that additions, reductions, deletions, and other modifications and changes may be made to this specification and drawings without departing from the broader idea and scope set forth in the claims. Thus, while certain embodiments of the disclosure have been described, they are not intended to be limiting. Various modifications and equivalents are within the scope of the appended claims.

[0283] The use of the terms “a,” “an,” and “the” and similar reference subjects in the context describing the disclosed embodiments (in particular, in the context of the appended claims) should be construed to apply to both singular and plural nouns unless otherwise specifically indicated herein or unless it is clearly inconsistent with the context. The terms “equipped,” “having,” “including,” and “containing” should be construed to be unrestricted terms (i.e., “including, but not limited to”) unless otherwise specifically noted. The term “connected” should be construed to mean partially or completely contained, connected, or joined together, even if there is something intervening. The enumeration of ranges of values ​​herein is intended simply as a way to refer individually to each separate value contained within the range unless otherwise specifically indicated herein, and each separate value is incorporated herein as if it were individually enumerated herein. All methods described herein may be performed in any appropriate order unless otherwise specifically indicated herein or unless it is clearly inconsistent with the context. Any use of any examples or illustrative language (e.g., "etc.") provided herein is intended solely to better illustrate the embodiments and, unless otherwise claimed, does not limit the scope of this disclosure. The language herein should not be construed as indicating that any unclaimed element is essential to the practice of this disclosure.

[0284] Disjunctive language, such as the phrase "at least one of X, Y, or Z," is generally intended to be understood in context as indicating that an item, condition, etc., may be any one of X, Y, or Z, or any combination thereof (e.g., X, Y, and / or Z), unless otherwise explicitly stated. Therefore, such disjunctive language is generally not intended, and should not, mean that a particular embodiment requires the presence of at least one of X, at least one of Y, or at least one of Z, respectively.

[0285] This specification describes preferred embodiments of the Disclosure, including known best modes for carrying out the Disclosure. As a person skilled in the art reads the foregoing description, variations of such preferred embodiments may become apparent. A person skilled in the art should be able to adopt such variations as needed, and the Disclosure may be practiced in ways other than those described in detail herein. Accordingly, this Disclosure includes all modifications and equivalents of the subject matter enumerated in the claims appended herein, as permitted by applicable law. Furthermore, any combination of the aforementioned elements in all possible variations of the embodiments is encompassed by this Disclosure unless specifically indicated herein.

[0286] All references cited herein, including published documents, patent applications, and patents, are individually and in detail incorporated by reference, and are incorporated herein by reference to the same extent as if they were presented as a whole.

[0287] While the embodiments of this disclosure have been described in the foregoing specification with reference to specific embodiments thereof, those skilled in the art will recognize that this disclosure is not limited thereto. The various features and embodiments of the foregoing disclosure may be used individually or together. Furthermore, embodiments may be used in any number of environments and applications beyond those described herein without departing from the broader idea and scope of this specification. Accordingly, this specification and the drawings should be considered illustrative rather than restrictive.

Claims

1. It is a method, The first component of the first private cloud within the first cloud environment receives a request for a cloud service from the second private cloud component within the second cloud environment. In response to receiving the request, the first component performs a set of operations associated with provisioning the cloud service within the second cloud environment, at least one of which includes establishing network connectivity prerequisites for network connectivity between the first private cloud and the second private cloud. The method further includes, after performing the series of operations, causing the first component to provision the cloud services in the second private cloud to one or more second components of the first private cloud.

2. The method according to claim 1, wherein the request is initiated in response to a customer device interacting with the portal of the second private cloud, and the customer device is associated with a customer account corresponding to the second private cloud.

3. The method according to claim 2, wherein the portal is configured to present a portion of a graphical user interface provided by the first component, and the request is initiated in response to the customer device interacting with the portion.

4. The method according to claim 1, wherein the first component causes one or more second components of the first private cloud to provision the cloud services in the second private cloud, which includes causing one or more second components to establish a network connection between the first private cloud and the second private cloud based on the network connection prerequisites.

5. The method according to claim 1, wherein the first component causes one or more second components of the first private cloud to provision the cloud services in the second private cloud, which includes causing another component of the second private cloud to provision a virtual machine.

6. The method according to claim 1, wherein establishing the prerequisites for the network connectivity includes creating a first subnet in the first private cloud and creating a second subnet in the second private cloud, wherein at least one Internet Protocol address associated with the first subnet and at least one Internet Protocol address associated with the second subnet are within a Classless Interdomain Routing (CIDR) range.

7. The aforementioned method, The method according to claim 1, further comprising integrating the first private cloud with the second private cloud by linking an identifier associated with the first component to an identifier associated with another component of the second private cloud, before the first component of the first private cloud in the first cloud environment receives the request for the cloud service from the component of the second private cloud in the second cloud environment.

8. One or more processing systems, A system comprising one or more computer-readable media storing instructions, wherein, when the instructions are executed by the one or more processing systems, the system causes the system to perform an operation, and the operation is The first component of the first private cloud within the first cloud environment receives a request for a cloud service from the second private cloud component within the second cloud environment. In response to receiving the request, the first component performs a set of operations associated with provisioning the cloud service within the second cloud environment, at least one of which includes establishing network connectivity prerequisites for network connectivity between the first private cloud and the second private cloud. A system further comprising, after performing the series of operations described above, the first component causing one or more second components of the first private cloud to provision the cloud services in the second private cloud.

9. The system according to claim 8, wherein the request is initiated in response to a customer device interacting with the portal of the second private cloud, and the customer device is associated with a customer account corresponding to the second private cloud.

10. The system according to claim 9, wherein the portal is configured to present a portion of a graphical user interface provided by the first component, and the request is initiated in response to the customer device interacting with the portion.

11. The system according to claim 8, wherein the first component causes one or more second components of the first private cloud to provision the cloud services in the second private cloud, and the first component causes one or more second components to establish a network connection between the first private cloud and the second private cloud based on the network connection prerequisites.

12. The system according to claim 8, wherein the first component causes one or more second components of the first cloud environment to provision the cloud services in the second cloud environment, which includes causing another component of the second cloud environment to provision a virtual machine.

13. The system according to claim 8, wherein establishing the prerequisites for the network connectivity includes creating a first subnet in the first private cloud and creating a second subnet in the second private cloud, wherein at least one Internet Protocol address associated with the first subnet and at least one Internet Protocol address associated with the second subnet are within a Classless Interdomain Routing (CIDR) range.

14. The aforementioned operation is, The system according to claim 8, further comprising integrating the first private cloud with the second private cloud by linking an identifier associated with the first component to an identifier associated with another component of the second private cloud, before the first component of the first private cloud in the first cloud environment receives the request for the cloud service from the component of the second private cloud in the second cloud environment.

15. One or more non-temporary computer-readable media storing instructions, wherein, when executed by the one or more processors, the instructions cause the system to perform an action, and the action is The first component of the first private cloud within the first cloud environment receives a request for a cloud service from the second private cloud component within the second cloud environment. In response to receiving the request, the first component performs a set of operations associated with provisioning the cloud service within the second cloud environment, at least one of which includes establishing network connectivity prerequisites for network connectivity between the first private cloud and the second private cloud. One or more non-temporary computer-readable media, further comprising the first component causing one or more second components of the first private cloud to provision the cloud services in the second private cloud after performing the series of operations described above.

16. The request is initiated in response to a customer device interacting with the portal of the second private cloud, and the customer device is associated with a customer account corresponding to the second private cloud, one or more non-temporary computer-readable media according to claim 15.

17. The portal is configured to present a portion of a graphical user interface provided by the first component, and the request is initiated in response to the customer device interacting with the portion, one or more non-temporary computer-readable media according to claim 16.

18. The first component provisioning the cloud services in the second private cloud to one or more second components of the first private cloud includes one or more second components establishing a network connection between the first private cloud and the second private cloud based on the network connection prerequisites, according to claim 15.

19. The first component provisioning the cloud services in the second cloud environment to one or more second components of the first cloud environment includes the virtual machine provisioning a virtual machine to another component of the second cloud environment, as described in claim 15.

20. Establishing the prerequisites for the network connection includes creating a first subnet in the first private cloud and creating a second subnet in the second private cloud, wherein at least one Internet Protocol address associated with the first subnet and at least one Internet Protocol address associated with the second subnet are within a Classless Interdomain Routing (CIDR) range, one or more non-temporary computer-readable media according to claim 15.

Images