Provisioning cloud service of first cloud service provider using control plane of second cloud service provider

Abstract

Techniques for provisioning cloud services of a first cloud service provider using a control plane of a second cloud service provider are disclosed. Technologies include detecting that a request for a cloud service provided by a first cloud service provider has been received from a second cloud environment of a second cloud service provider different from the first cloud service provider. The technique further includes, upon detecting that the request for the cloud service has been received, provisioning the first set of resources within the first cloud environment and linking the first set of resources to a second set of resources within the second cloud environment. Linking the first set of resources to the second set of resources enables data about the cloud service to be transferred from the second cloud environment to the first cloud environment.

Description

[0001] Cross-reference to related applications

[0002] This application claims the benefits and priority of U.S. Provisional Application No. 63 / 608,036, filed December 8, 2023; U.S. Provisional Application No. 63 / 538,254, filed September 13, 2023; U.S. Provisional Application No. 63 / 537,443, filed September 8, 2023; and U.S. Provisional Application No. 63 / 534,071, filed August 22, 2023, the entire contents of which are incorporated herein by reference for all purposes. Background Technology

[0003] Over the past few years, the adoption of cloud services has increased dramatically, and this trend is only expected to continue. Different cloud service providers (CSPs) offer a variety of different cloud environments, each providing a set of one or more cloud services. This set of cloud services offered by a cloud environment can include one or more different types of services, including but not limited to Software as a Service (SaaS), Infrastructure as a Service (IaaS), Platform as a Service (PaaS), etc.

[0004] While various cloud environments are currently available, each provides a closed ecosystem for its subscribers. Therefore, customers of a cloud environment are limited to the services offered by that specific cloud environment. There is no easy way for customers subscribed to a cloud environment provided by a CSP to access services offered by different CSPs through that cloud environment. The embodiments discussed herein address these and other problems. Summary of the Invention

[0005] This article discloses techniques for using the control plane of a second cloud service provider to supply cloud services to a first cloud service provider.

[0006] In some embodiments, a method includes: a first cloud environment of a first cloud service provider detecting that a request for a cloud service provided by the first cloud service provider has been received from a second cloud environment of a second cloud service provider different from the first cloud service provider; after detecting that the request for the cloud service has been received, the first cloud environment supplying a first set of resources within the first cloud environment; and the first cloud environment linking the first set of resources to a second set of resources within the second cloud environment, wherein linking the first set of resources to the second set of resources enables data about the cloud service to be transferred from the second cloud environment to the first cloud environment.

[0007] In some embodiments, the request is initiated in response to a portal interaction between the client device and the second cloud environment, and the client device is associated with a client account of the second cloud service provider.

[0008] In some embodiments, provisioning a first set of resources within a first cloud environment includes: receiving information identifying a first subnet in a second cloud environment and a first Internet Protocol (IP) address range associated with the first subnet; providing a private cloud within the first cloud environment; and providing a second subnet in the private cloud, wherein a second IP address range associated with the second subnet is within a Classless Inter-Domain Routing (CIDR) range of the first IP address range.

[0009] In some embodiments, the method further includes: creating resources comprising a cluster of virtual machines in a first cloud environment, wherein each IP address in a second IP address range is associated with a virtual machine in the cluster of virtual machines; and attaching the resources to a router in the first cloud environment.

[0010] In some embodiments, linking the first set of resources to the second set of resources includes: invoking an application programming interface (API) of the second cloud environment, wherein invoking the API causes IP addresses in the second IP address range to appear in the first subnet; registering IP addresses in the second IP address range with a router in the second cloud environment using the API; receiving a virtual LAN identifier from the second cloud environment; and creating a virtual circuit between the router in the first cloud environment and the router in the second cloud environment using the virtual LAN identifier.

[0011] In some embodiments, linking the first set of resources to the second set of resources further includes: accessing Domain Name System (DNS) records associated with a virtual machine cluster in the first cloud environment; and providing DNS records to the second cloud environment, wherein providing DNS records to the second cloud environment causes a private DNS zone to be created in a private cloud associated with a customer account of the second cloud service provider.

[0012] In some embodiments, linking the first set of resources to the second set of resources enables data about cloud services to be transferred from a first subnet associated with a first private cloud in a second cloud environment to a second subnet associated with a second private cloud in the first cloud environment.

[0013] Some embodiments include a system comprising one or more processing systems and one or more computer-readable media storing instructions that, when executed by the one or more processing systems, cause the system to perform some or all of the operations and / or methods disclosed herein.

[0014] Some embodiments include one or more non-transitory computer-readable media storing instructions that, when executed by one or more processing systems, cause the systems to perform some or all of the operations and / or methods disclosed herein.

[0015] The techniques described above and below can be implemented in a variety of ways and in a variety of contexts. Several example implementations and contexts are provided below in more detail, with reference to the accompanying figures. However, the implementations and contexts below are only a few of the many. Attached Figure Description

[0016] The features, embodiments, and advantages of this disclosure can be better understood by reading the following detailed description with reference to the accompanying drawings.

[0017] Figure 1 This is a high-level diagram illustrating a distributed environment of a virtual or overlay cloud network hosted by a cloud service provider infrastructure, according to certain embodiments.

[0018] Figure 2 A simplified architecture diagram of physical components in the physical network within a Cloud Service Provider Infrastructure (CSPI) according to certain embodiments is depicted.

[0019] Figure 3 An example arrangement within a CSPI according to certain embodiments is shown, in which a host is connected to multiple network virtualization devices (NVDs).

[0020] Figure 4 The diagram illustrates connectivity between multi-tenant hosts and NVDs, according to certain embodiments, for providing I / O virtualization to support multi-tenant hosts.

[0021] Figure 5 A simplified block diagram of a physical network provided by CSPI according to certain embodiments is depicted.

[0022] Figure 6 A simplified high-level diagram of a distributed environment comprising multiple cloud environments provided by different cloud service providers (CSPs) is depicted according to certain embodiments.

[0023] Figure 7 An exemplary physical architecture for providing cross-cloud services based on infrastructure distributed across multiple CSPs, according to some embodiments, is described.

[0024] Figure 8 An exemplary virtual architecture for providing cross-cloud services based on infrastructure distributed across multiple CSPs, according to some embodiments, is described.

[0025] Figure 9 Exemplary virtual resources provided by a first CSP to a customer of a second CSP, according to some embodiments, are described.

[0026] Figure 10 An exemplary architecture for provisioning and managing cross-cloud services based on infrastructure distributed across multiple CSPs is described according to some embodiments.

[0027] Figure 11 An exemplary user experience flow for provisioning resources is described according to some embodiments.

[0028] Figure 12 An exemplary control plane supply flow according to some embodiments is depicted.

[0029] Figure 13 An architectural example of a resource management mechanism for provisioning and managing cross-cloud services across multiple cloud environments is described according to some embodiments.

[0030] Figure 14 Examples of cloud services provided using resources from a cloud service provider, according to some embodiments, are described.

[0031] Figure 15 Another example of a control plane supply flow according to some embodiments is depicted.

[0032] Figure 16 Another example of a control plane supply flow according to some embodiments is depicted.

[0033] Figure 17 Example processes for provisioning and managing cross-cloud services across multiple cloud environments, according to some embodiments, are described.

[0034] Figure 18 An example process for supplying cloud services to a first cloud service provider using the control plane of a second cloud service provider, according to some embodiments, is described.

[0035] Figure 19 This is a block diagram illustrating a pattern for implementing a cloud infrastructure-as-a-service system according to certain embodiments.

[0036] Figure 20 This is a block diagram illustrating another pattern for implementing a cloud infrastructure-as-a-service system according to certain embodiments.

[0037] Figure 21 This is a block diagram illustrating another pattern for implementing a cloud infrastructure-as-a-service system according to certain embodiments.

[0038] Figure 22 This is a block diagram illustrating another pattern for implementing a cloud infrastructure-as-a-service system according to certain embodiments.

[0039] Figure 23 This is a block diagram illustrating an example computer system according to certain embodiments. Detailed Implementation

[0040] In the following description, specific details are set forth for illustrative purposes in order to provide a thorough understanding of certain embodiments. However, it will be clear that various embodiments can be practiced without these specific details. The accompanying drawings and descriptions are not intended to be limiting. The word “exemplary” is used herein to mean “serves as an example, instance, or illustration.” Any embodiment or design described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments or designs.

[0041] This disclosure generally relates to improved cloud architectures, and more specifically to technologies for providing services based on infrastructure distributed across multiple cloud service providers (CSPs). In the example, the first cloud service provider (CSP) (e.g., Oracle) ® The cloud infrastructure (OCI) provides primary services (e.g., database services, such as Exadata services available from Oracle), some of which may be available from the primary cloud service provider infrastructure (CSPI) of the primary CSP. The secondary CSP (e.g., Microsoft) provides the primary services. ® Azure, Google Cloud™, Amazon Web Services—AWS ® The system provides a second service (e.g., Azure virtual machines), some of which may be available from a second cloud service provider infrastructure (CSPI) of the second CSP. This makes at least one of the first services available to the customers of the second CSP via the second CSPI. In the example, the hardware and / or software configured by the first CSPI to provide the first service is deployed at the second CSPI.

[0042] For example, at the physical level, the second CSPI may include a first set of computing resources of the first CSPI (e.g., server racks optimized for the services of the first CSPI). The first CSPI may include a second set of computing resources of the first CSPI (e.g., servers within a region). The second CSPI may also include a third set of computing resources of the second CSPI (e.g., servers within an availability zone). The first set of computing resources may be connected to the second and third sets of computing resources.

[0043] At the virtualization level, the first and second sets of resources can be hosted in a first cloud for a first CSP and its customers. Conversely, the third set of resources can be hosted in a second cloud for a second CSP and its customers. These two clouds can be connected (e.g., via a peering connection using a virtual router). The services provided by the first CSP to its customers can have first resources hosted in the first set of compute resources and second resources hosted in the second set of compute resources. This service can be made available to customers via the second cloud. The use of the first resources allows for reduced latency. Providing services from the first CSP via the second CSP can enable a better experience by extending the service to customers of the second CSP who may be more familiar with configuring and managing the cloud using the second CSP.

[0044] For illustration, consider an example of an Exadata service (i.e., the OCI database service) made available via Azure. In this example, at the physical level, OCI subsites belonging to the OCI region are co-located in an Azure datacenter. The OCI subsites are connected to the Azure datacenter via a FastConnect router (and a Microsoft MeetMe ToR router on the Azure side). The OCI subsites are also connected to the parent OCI region via a fiber optic connection. At the virtual level, the Azure datacenter hosts the customer cloud (e.g., VNET), while the parent OCI region and the subsites co-host the customer virtual cloud network (VCN). Latency-critical resources for the OCI service (e.g., data plane resources for the Exadata service) are hosted in the subsites, while other supporting resources (e.g., control plane resources, customer-facing console resources, etc.) are hosted in the parent region.

[0045] On the OCI side, VNET and VCN are connected via virtual routers (e.g., a Dynamic Routing Gateway (DRG) supporting FastConnect and a virtual router provided by Azure supporting MeetMe). Within VNET, delegated subnets can be configured. Within VCN, Exadata service subnets can be configured. Both subnets use the same Internet Protocol (IP) address range. VNET can store mapping information that maps IP addresses within the IP address range used in VNET to IP addresses of the Exadata service at VCN. Given a one-to-one IP address mapping, traffic to the IP address (e.g., compute instances from VNET) is sent from VNET to VCN.

[0046] Therefore, from the customer's perspective, the customer perceives that their Exadata service within their VNET has an IP address. However, the Exadata service is actually located within the VCN at that IP address. Furthermore, by hosting at least a portion of the Exadata service (e.g., the data plane) in a subsite, the latency associated with the Exadata service can be reduced.

[0047] Referring back to the architecture above, the first group of computing resources can be referred to as belonging to the sub-site, and the second group of computing resources can be referred to as belonging to the parent site. The third group of computing resources can be referred to as data center resources. The sub-sites of the first CSP are deployed in the data center (or second CSPI) of the second CSP. In contrast, the parent site of the first CSP is deployed in the first CSPI of the first CSP.

[0048] Subsites allow low-latency services from the first CSP to be delivered to customers via a second CSP. However, deployment mechanisms may be needed to control the deployment of the first CSP's resources across subsites and the parent site.

[0049] Embodiments of this disclosure relate to a deployment mechanism in which a control plane for a first CSP is not hosted at a subsite. Customer input at a second CSPI can be received by the control plane and mapped to a customer ID related to the first CSPI. Based on the customer ID, the control plane can provision resources at the first CSPI, such as by creating a cloud network for the customer and associated connectivity resources (e.g., a Dynamic Routing Gateway – DRG) for the cloud network. The cloud network can also be connected to a cluster of virtual machines (VMs) hosting services. The VM cluster can be hosted at a subsite, while other resources of the cloud network can be hosted at a parent site. Subsequently, the control plane can interconnect the cloud network with the customer's cloud network related to the second CSP, thus interconnecting the customer's two cloud networks. As part of this interconnection, the control plane can provide the second CSPI with the Internet Protocol (IP) addresses of the VM clusters and, possibly, corresponding DNS records, making these IP addresses and, possibly, the DNS records, available to the customer at the customer's cloud network related to the second CSP.

[0050] To illustrate, consider the OCI and Azure example again. Azure customer input (e.g., selecting its VNET and subnet) is received from Azure Resource Manager (ARM). The Oracle Resource Provider (ORP) can convert this input into the customer's OCI ID and pass that information to the control plane. The control plane performs two main processes. The first process is provisioning the relevant OCI resources. The second process is connecting these resources to the customer's VNET.

[0051] In the first process, the control plane creates the customer's VCN in the customer's lease (e.g., at the parent site) and creates subnets (and possibly backup subnets) within the VCN. The Azure and OCI subnets (one in the customer's Azure VNET and one in the customer's OCI VCN) use the same Classless Inter-Domain Routing (CIDR). The control plane also creates a DRG in the customer's lease and attaches the DRG to the VCN. The control plane also establishes routing information for the DRG and VCN (enabling these two resources to interconnect). ORP can also be provisioned to the VM cluster (e.g., in the subsite). One or more IP addresses of this VM cluster from the CIDR are mapped to corresponding DNS records.

[0052] In the second process, the control plane registers these IP addresses with Azure (on the MeetMe router). It also creates a virtual circuit between the DRG and the MeetMe router and sends DNS records, enabling a private DNS zone to be established in Azure.

[0053] For clarity, embodiments of this disclosure have been described in relation to specific CSPs (e.g., Oracle and Microsoft) and services (e.g., Exadata services). However, the embodiments are not limited thereto, but rather, similarly and equivalently, apply to any CSP, CSPI, and service in a multi-cloud environment.

[0054] Cloud Network Example

[0055] The term cloud service is generally used to refer to services provided by a CSP (Consumer Service Provider) that make the systems and infrastructure (cloud infrastructure) offered by the CSP available to users or customers on demand (e.g., via a subscription model). Typically, the servers and systems that make up the CSP infrastructure are separate from the customer's own on-premises servers and systems. Therefore, customers can utilize cloud services provided by the CSP without having to purchase separate hardware and software resources for the service. Cloud services are designed to provide subscribers with easy, scalable access to a pool of applications and computing resources without requiring customers to invest in the infrastructure used to provide the service.

[0056] There are several cloud service providers that offer various types of cloud services. There are various different types or models of cloud services, including Software as a Service (SaaS), Infrastructure as a Service (IaaS), Platform as a Service (PaaS), etc.

[0057] A customer can subscribe to one or more cloud services provided by a CSP. A customer can be any entity, such as an individual, organization, or enterprise. When a customer subscribes to or registers for a service provided by a CSP, a lease or account is created for that customer. The customer can then access one or more cloud resources associated with that account through the subscription.

[0058] As mentioned above, IaaS is a specific type of cloud computing service. In the IaaS model, a CSP provides infrastructure (referred to as Cloud Service Provider Infrastructure or CSPI) that customers can use to build their own customizable networks and deploy customer resources. Therefore, the customer's resources and network are hosted in a distributed environment by the infrastructure provided by the CSP. This differs from traditional computing, where customer resources and networks are hosted by customer-provided infrastructure.

[0059] CSPI can include interconnected high-performance computing resources, including various host, memory, and network resources forming a physical network, also known as the base network or underlying network. Resources in CSPI can be distributed across one or more data centers, which can be geographically distributed across one or more geographic regions. Virtualization software can be executed by these physical resources to provide a virtualized distributed environment. Virtualization creates overlay networks (also known as software-based networks, software-defined networks, or virtual networks) on top of the physical network (or base network or underlying network). The physical network provides the underlying foundation for creating one or more overlay or virtual networks on top of it. The physical network includes physical network devices such as physical switches, routers, computers, and hosts. An overlay network is a logical (or virtual) network that runs on top of the physical base network. A given physical network can support one or more overlay networks. Overlay networks typically use encapsulation techniques to distinguish traffic belonging to different overlay networks. Virtual or overlay networks are also known as VCNs. Virtual networks utilize software virtualization technologies (e.g., hypervisors, functions implemented by network virtualization devices (NVDs) (e.g., smartNICs), top-of-rack (TOR) switches, intelligent TORs that implement one or more functions performed by NVDs, and other mechanisms) to create a network abstraction layer that can operate on top of the physical network. Virtual networks can take many forms, including peer-to-peer networks, IP networks, etc. Virtual networks are typically Layer-3 IP networks or Layer-2 VLANs. This method of virtual or overlay networking is often referred to as virtual or overlay Layer-3 networking. Examples of protocols developed for virtual networks include IP-in-IP (or Generic Routing Encapsulation (GRE)), Virtual Extensible LAN (VXLAN—IETF RFC 7348), Virtual Private Networks (VPNs) (e.g., MPLS Layer-3 Virtual Private Network (RFC 4364)), VMware's NSX, GENEVE (Generic Network Virtualization Encapsulation), and others.

[0060] For IaaS, CSPI provided by a CSP can be configured to provide virtualized groups of computing resources over a public network (e.g., the Internet). In the IaaS model, cloud service providers can host infrastructure components (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer), etc.). In some cases, IaaS providers can also provision a wide variety of services to accompany those infrastructure components (e.g., billing, monitoring, logging, security, load balancing, and clustering). Therefore, since these services can be policy-driven, IaaS users can implement policies to drive load balancing to maintain application availability and performance. CSPI provides infrastructure and a complementary set of cloud services, enabling customers to build and run a wide range of applications and services in a highly available, managed, distributed environment. CSPI provides high-performance computing resources and capabilities, as well as storage capacity, in a flexible virtual network securely accessible from various networked locations, such as from the customer's on-premises deployment network. When a customer subscribes to or registers for IaaS services provided by CSP, the lease created for that customer is a secure and isolated partition within CSP, in which the customer can create, organize, and manage their cloud resources.

[0061] Customers can use the compute, storage, and networking resources provided by CSPI to build their own virtual networks. One or more customer resources or workloads, such as compute instances, can be deployed on these virtual networks. For example, a customer can use the resources provided by CSPI to build one or more customizable and private virtual networks(s) known as VCNs. A customer can deploy one or more customer resources, such as compute instances, on a customer VCN. Compute instances can take the form of virtual machines, bare metal instances, etc. Therefore, CSPI provides the infrastructure and a complementary set of cloud services that enable customers to build and run a wide range of applications and services in a highly available virtual hosting environment. Customers do not manage or control the underlying physical resources provided by CSPI, but they have control over the operating system, storage devices, and deployed applications; and possibly limited control over chosen networking components (e.g., firewalls).

[0062] The CSP can provide a console that enables customers and network administrators to configure, access, and manage resources deployed in the cloud using CSPI resources. In some embodiments, the console provides a web-based user interface for accessing and managing the CSPI. In some implementations, the console is a web-based application provided by the CSP.

[0063] CSPI can support single-tenant or multi-tenant architectures. In a single-tenant architecture, software (e.g., applications, databases) or hardware components (e.g., hosts or servers) serve a single customer or tenant. In a multi-tenant architecture, the software or hardware components serve multiple customers or tenants. Therefore, in a multi-tenant architecture, CSPI resources are shared among multiple customers or tenants. In a multi-tenant scenario, precautions and safeguards are implemented within CSPI to ensure that each tenant's data is isolated and remains invisible to other tenants.

[0064] In a physical network, a network endpoint (“endpoint”) is a computing device or system that is connected to and communicates with the physical network. Network endpoints in a physical network can be connected to a local area network (LAN), a wide area network (WAN), or other types of physical networks. Examples of traditional endpoints in a physical network include modems, hubs, bridges, switches, routers and other networking devices, physical computers (or hosts), etc. Each physical device in a physical network has a fixed network address that can be used to communicate with the device. This fixed network address can be a Layer-2 address (e.g., a MAC address), a fixed Layer-3 address (e.g., an IP address), etc. In a virtualized environment or in a virtual network, endpoints can include various virtual endpoints, such as virtual machines hosted by components of the physical network (e.g., hosted by a physical host). These endpoints in a virtual network are addressed using overlay addresses, such as overlay Layer-2 addresses (e.g., overlay MAC addresses) and overlay Layer-3 addresses (e.g., overlay IP addresses). Network overlays enable flexibility by allowing network administrators to move the overlay addresses associated with network endpoints using software management (e.g., via software implementing a control plane for virtual networks). Therefore, unlike in a physical network, in a virtual network, overlay addresses (e.g., overlay IP addresses) can be moved from one endpoint to another using network management software. Since a virtual network is built on top of a physical network, communication between components within a virtual network involves both the virtual network and the underlying physical network. To facilitate such communication, CSPI components are configured to learn and store mappings that map overlay addresses in the virtual network to actual physical addresses in the base network, and vice versa. These mappings are then used to facilitate communication. Client traffic is encapsulated to facilitate routing within the virtual network.

[0065] Therefore, physical addresses (e.g., physical IP addresses) are associated with components in a physical network, and overlay addresses (e.g., overlay IP addresses) are associated with entities in a virtual or overlay network. A physical IP address is an IP address associated with a physical device (e.g., a network device) in the underlying or physical network. For example, each NVD has an associated physical IP address. An overlay IP address is an overlay address associated with an entity in the overlay network, such as a compute instance in a customer's VCN. Two different customers or tenants (each with its own private VCN) may potentially use the same overlay IP address in their VCNs without knowing about each other. Both physical IP addresses and overlay IP addresses are types of real IP addresses. These are separate from virtual IP addresses. A virtual IP address is typically a single IP address that represents or maps to multiple real IP addresses. A virtual IP address provides a one-to-many mapping between virtual IP addresses and multiple real IP addresses. For example, a load balancer can use a VIP to map to or represent multiple servers, each with its own real IP address.

[0066] CSPI is physically hosted in one or more data centers in one or more regions around the world. CSPI may include components in the physical or underlying network and virtualized components (e.g., virtual networks, compute instances, virtual machines, etc.) in a virtual network built on top of the physical network components. In some embodiments, CSPI is organized and hosted in domains, regions, and availability domains. A region is typically a local geographic area containing one or more data centers. Regions are typically independent of each other and can be geographically separated, for example, across countries or even continents. For example, one region might be in Australia, another in Japan, yet another in India, and so on. CSPI resources are partitioned between regions, such that each region has its own independent subset of CSPI resources. Each region can provide a set of core infrastructure services and resources, such as compute resources (e.g., bare metal servers, virtual machines, containers, and related infrastructure); storage resources (e.g., block volume storage, file storage, object storage, archive storage); networking resources (e.g., VCN, load balancing resources, connectivity to on-premises networks); database resources; edge networking resources (e.g., DNS); access management and monitoring resources, etc. Each region typically has multiple paths connecting it to other regions within its domain.

[0067] Generally, applications are deployed in the areas where they are used most frequently (i.e., on the infrastructure associated with that area) because using nearby resources is faster than using resources far away. Applications can also be deployed in different areas for various reasons, such as redundancy to mitigate the risk of regional events such as large weather systems or earthquakes, or to meet the requirements of changing legal jurisdictions, tax domains, and other business or social standards.

[0068] Data centers within a region can be further organized and subdivided into Availability Domains (ADs). An Availability Domain can correspond to one or more data centers located within a region. A region can consist of one or more Availability Domains. In such a distributed environment, CSPI resources are either region-specific (such as VCNs) or Availability Domain-specific (such as compute instances).

[0069] Availability Zones (ADs) within a region are isolated and fault-tolerant, and configured to make simultaneous failures unlikely. This is achieved by ensuring that ADs do not share critical infrastructure resources (such as networking, physical cabling, cable paths, cable entry points, etc.), making a failure at one AD within a region unlikely to affect the availability of other ADs in the same region. ADs within the same region can be interconnected via low-latency, high-bandwidth networks, making it possible to provide high-availability connectivity to other networks (e.g., the internet, customer on-premises networks, etc.) and to build replication systems across multiple ADs for both high availability and disaster recovery. Cloud services use multiple ADs to ensure high availability and prevent resource failures. As the infrastructure provided by IaaS providers grows, additional capacity can be added to more regions and ADs. Traffic between availability domains is typically encrypted.

[0070] In some embodiments, regions are grouped into domains. A domain is a logical collection of regions. Domains are isolated from each other and do not share any data. Regions within the same domain can communicate with each other, but regions in different domains cannot. A customer's lease or account with a CSP exists within a single domain and can be distributed across one or more regions belonging to that domain. Typically, when a customer subscribes to an IaaS service, the lease or account is created for that customer in the customer-designated region within a domain (referred to as the "home" region). A customer can extend their lease across one or more other regions within a domain. A customer cannot access regions that are not within the domain where their lease exists.

[0071] IaaS providers can offer multiple domains, each catering to a specific set of customers or users. For example, a business domain can be offered to business customers. As another example, a country-specific domain can be offered to customers within a specific country. And yet another example, a government domain can be offered to governments. For instance, a government domain can cater to a specific government and may offer a higher level of security compared to the business domain. For example, OCI currently offers one domain for its business region and two domains for its government cloud region (e.g., FedRAMP license and IL5 license).

[0072] In some embodiments, an Active Directory (AD) can be subdivided into one or more fault domains. A fault domain is a grouping of infrastructure resources within an AD to provide anti-affinity. Fault domains allow for the distribution of compute instances so that instances are not on the same physical hardware within a single AD. This is called anti-affinity. A fault domain refers to a group of hardware components (computers, switches, etc.) that share a single point of failure. Compute pools are logically divided into fault domains. Therefore, a hardware failure or compute hardware maintenance event affecting one fault domain does not affect instances in other fault domains. Depending on the embodiment, the number of fault domains per AD can vary. For example, in some embodiments, each AD contains three fault domains. Fault domains act as logical data centers within the AD.

[0073] When a customer subscribes to IaaS services, resources from CSPI are provisioned to the customer and associated with the customer's lease. The customer can use these provisioned resources to build private networks and deploy resources on those networks. A customer network hosted in the cloud by CSPI is called a VCN. A customer can use the CSPI resources allocated to them to establish one or more VCNs. A VCN is a virtual or software-defined private network. Customer resources deployed in a customer's VCN can include compute instances (e.g., virtual machines, bare metal instances) and other resources. These compute instances can represent various customer workloads, such as applications, load balancers, databases, etc. Compute instances deployed on a VCN can communicate with publicly accessible endpoints (“public endpoints”) via public networks such as the Internet, with other instances in the same VCN or other VCNs (e.g., other VCNs belonging to the customer or VCNs not belonging to the customer), with the customer's on-premises data center or network, and with service endpoints and other types of endpoints.

[0074] CSPs can use CSPIs to provide various services. In some instances, CSPI clients can act as service providers themselves and use CSPI resources to provide services. Service providers can expose service endpoints characterized by identifying information (e.g., IP address, DNS name, and port). Client resources (e.g., compute instances) can consume a specific service by accessing the service endpoints exposed by the service for that specific service. These service endpoints are typically publicly accessible to users via public communication networks (such as the Internet) using the public IP address associated with the endpoint. Publicly accessible network endpoints are sometimes referred to as public endpoints.

[0075] In some embodiments, a service provider may expose the service via an endpoint (sometimes referred to as a service endpoint). Clients of the service can then use this service endpoint to access the service. In some implementations, the service endpoint provided for the service can be accessed by multiple clients intending to consume the service. In other implementations, a dedicated service endpoint may be provided to a client, so that only that client can use that dedicated service endpoint to access the service.

[0076] In some embodiments, when a VCN is created, it is associated with a Private Overlay Classless Inter-Domain Routing (CIDR) address space, which refers to a range of private overlay IP addresses (e.g., 10.0 / 16) assigned to the VCN. A VCN includes associated subnets, routing tables, and gateways. A VCN resides within a single area but can span one or more or all availability domains within that area. A gateway is a virtual interface configured for the VCN and enables communication to and from the VCN to one or more endpoints outside the VCN. One or more different types of gateways can be configured for the VCN to enable communication to or from different types of endpoints.

[0077] A VCN can be subdivided into one or more subnets or subnetworks. Therefore, a subnet is a configuration unit or subdivision that can be created within a VCN. A VCN can have one or more subnets. Each subnet within a VCN is associated with a contiguous range of overlay IP addresses (e.g., 10.0.0.0 / 24 and 10.0.1.0 / 24) that do not overlap with other subnets within that VCN or subnets representing subsets of the address space within the VCN's address space.

[0078] Each compute instance is associated with a Virtual Network Interface Card (VNIC), which enables the compute instance to join a subnet within a VCN. A VNIC is the logical representation of a physical network interface card (NIC). Generally, a VNIC is the interface between an entity (e.g., a compute instance, a service) and a virtual network. A VNIC exists within a subnet and has one or more associated IP addresses and associated security rules or policies. A VNIC is equivalent to a Layer-2 port on a switch. A VNIC is attached to both the compute instance and the subnet within the VCN. The VNIC associated with a compute instance enables the compute instance to become part of a VCN's subnet and allows the compute instance to communicate with endpoints such as those on the same subnet, those in different subnets within the VCN, or those outside the VCN. Therefore, the VNIC associated with a compute instance determines how the compute instance connects to endpoints inside and outside the VCN. When a compute instance is created and added to a subnet within a VCN, a VNIC for the compute instance is created and associated with that compute instance. For a subnet that includes a set of compute instances, the subnet contains a VNIC corresponding to that set of compute instances, and each VNIC is attached to a compute instance within that set of compute instances.

[0079] Each compute instance is assigned a private overlay IP address via a VNIC associated with it. This private overlay IP address is assigned to the VNIC associated with the compute instance when the compute instance is created and used to route traffic to and from the compute instance. All VNICs within a given subnet use the same routing table, security list, and DHCP options. As described above, each subnet within a VCN is associated with a contiguous range of overlay IP addresses (e.g., 10.0.0.0 / 24 and 10.0.1.0 / 24) that do not overlap with other subnets within that VCN or subnets representing subsets of the address space within the VCN's address space. For a VNIC on a specific subnet of a VCN, the private overlay IP address assigned to the VNIC is an address derived from the contiguous range of overlay IP addresses allocated to that subnet.

[0080] In some embodiments, in addition to a private overlay IP address, a compute instance may optionally be assigned additional overlay IP addresses, such as one or more public IP addresses, for example, if in a public subnet. These multiple addresses are assigned on the same VNIC or on multiple VNICs associated with the compute instance. However, each instance has a primary VNIC created during instance startup and associated with the overlay private IP address assigned to the instance—this primary VNIC cannot be removed. Additional VNICs (referred to as secondary VNICs) may be added to an existing instance in the same availability domain as the primary VNIC. All VNICs are in the same availability domain as the instance. Secondary VNICs may be in a subnet within the same VCN as the primary VNIC, or in a different subnet within the same VCN or different VCNs.

[0081] If a compute instance is in a public subnet, it can be optionally assigned a public IP address. When a subnet is created, it can be specified as either a public or private subnet. A private subnet means that resources within the subnet (e.g., compute instances) and associated VNICs cannot have public overriding IP addresses. A public subnet means that resources within the subnet and associated VNICs can have public IP addresses. Customers can specify that a subnet exists within a single availability domain or across multiple availability domains across regions or domains.

[0082] As described above, a VCN can be subdivided into one or more subnets. In some embodiments, a virtual router (VR) configured for the VCN (referred to as a VCN VR or VR-only) enables communication between the VCN's subnets. For a subnet within the VCN, the VR represents the logical gateway for that subnet, enabling that subnet (i.e., the compute instances on that subnet) to communicate with endpoints on other subnets within the VCN and with other endpoints outside the VCN. A VCN VR is a logical entity configured to route traffic between a VNIC within the VCN and a virtual gateway (“gateway”) associated with the VCN. The VCN VR is a Layer-3 / IP layer concept. In one embodiment, there is a VCN VR for the VCN, where the VCN VR potentially has an unlimited number of ports addressed by IP addresses, with one port for each subnet of the VCN. In this way, the VCN VR has a different IP address for each subnet in the VCN to which the VCN VR is attached. The VR is also connected to various gateways configured for the VCN. In some embodiments, a specific overlay IP address from the overlay IP address range of the subnet is reserved for the port of the VCN VR for that subnet. For example, consider a VCN with two subnets, each with an associated address range of 10.0 / 16 and 10.1 / 16. For the first subnet within the VCN with an address range of 10.0 / 16, addresses from this range are reserved for ports on the VCN VR for that subnet. In some instances, the first IP address from the range can be reserved for the VCN VR. For example, for a subnet covering the 10.0 / 16 IP address range, IP address 10.0.0.1 might be reserved for a port on the VCN VR for that subnet. For a second subnet within the same VCN with an address range of 10.1 / 16, the VCN VR could have a port with the IP address 10.1.0.1 for that second subnet. The VCN VR has a different IP address for each subnet within the VCN.

[0083] In some other embodiments, each subnet within a VCN may have its own associated VR, which can be addressed by the subnet using a reserved or default IP address associated with the VR. The reserved or default IP address may, for example, be a first IP address from a range of IP addresses associated with the subnet. The VNICs within the subnet can communicate (e.g., send and receive packets) using the default or reserved IP address and the VR associated with the subnet. In such embodiments, the VR is the ingress / egress point of the subnet. The VR associated with a subnet within the VCN can communicate with other VRs associated with other subnets within the VCN. The VR can also communicate with a gateway associated with the VCN. The VR functionality of the subnet is running on or performed by one or more NVDs that perform the VNIC functionality of the VNICs within the subnet.

[0084] You can configure routing tables, security rules, and DHCP options for a VCN. The routing table is a virtual routing table for the VCN and contains rules that route traffic from subnets within the VCN to destinations outside the VCN via gateways or specially configured instances. The VCN's routing table can be customized to control how packets are forwarded / routed to or from the VCN. DHCP options refer to configuration information automatically provided to the instance when it boots up.

[0085] Security rules configured for a VCN represent the VCN's overriding firewall rules. Security rules can include ingress and egress rules and specify the types of traffic allowed to enter and exit instances within the VCN (e.g., based on protocol and port). Clients can choose whether a given rule is stateful or stateless. For example, a client can allow incoming SSH traffic from anywhere to a set of instances by establishing a stateful ingress rule using source CIDR 0.0.0.0 / 0 and destination TCP port 22. Security rules can be implemented using network security groups or security lists. A network security group consists of a set of security rules that apply only to resources within that group. A security list, on the other hand, includes rules that apply to all resources in any subnet using that security list. A VCN can be provided with a default security list containing default security rules. DHCP options configured for a VCN provide configuration information that is automatically provided to instances within the VCN at instance boot time.

[0086] In some embodiments, the configuration information of a VCN is determined and stored by the VCN control plane (CP). The VCN configuration information may include, for example, information about: the address range associated with the VCN, subnets and associated information within the VCN, one or more VRs associated with the VCN, compute instances and associated VNICs within the VCN, NVDs performing various virtualized network functions associated with the VCN (e.g., VNICs, VRs, gateways), VCN status information, and other VCN-related information. In some embodiments, a VCN distribution service publishes the configuration information or portions thereof stored by the VCN CP to the NVD. This distributed information can be used to update information stored and used by the NVD (e.g., forwarding tables, routing tables, etc.) to forward packets to and from compute instances within the VCN.

[0087] In some embodiments, the creation of the VCN and subnet is handled by the VCN CP, and the initiation of the compute instance is handled by the compute CP. The compute CP is responsible for allocating physical resources for the compute instance and then invoking the VCN CP to create the VNIC and attach it to the compute instance. The VCN CP also maps VCN data to the VCN data plane, which is configured to perform packet forwarding and routing functions. In some embodiments, the VCN CP provides a distribution service responsible for providing updates to the VCN data plane.

[0088] Customers can use resources hosted by CSPI to create one or more VCNs. Compute instances deployed on a customer's VCN can communicate with different endpoints. These endpoints can include CSPI-hosted endpoints and endpoints external to CSPI.

[0089] The various architectures used to implement cloud-based services using CSPI are described below. Figure 1 This is a high-level diagram illustrating a distributed environment 100 of an overlay or client VCN hosted by CSPI, according to certain embodiments. Figure 1 The distributed environment described includes multiple components in the overlay network. Figure 1 The distributed environment 100 depicted herein is merely an example and is not intended to unduly limit the scope of the claimed embodiments. Many variations, substitutions, and modifications are possible. For example, in some implementations, Figure 1 The distributed environment described in the text can have more than Figure 1 The more or fewer systems or components shown can be combined into two or more systems, or can have different system configurations or arrangements.

[0090] like Figure 1As illustrated in the example depicted, the distributed environment 100 includes a CSPI 101, which provides services and resources that customers can subscribe to and use to build their VCN. In some embodiments, CSPI 101 provides IaaS services to subscribing customers. Data centers within CSPI 101 can be organized into one or more regions. An example region, “Region US” 102, is... Figure 1 As shown in the diagram, the customer has already configured an Oracle International C / O customer VCN for region 102. The customer can deploy various compute instances on VCN 104, which can include virtual machines or bare metal instances. Examples of instances include applications, databases, load balancers, etc.

[0091] exist Figure 1 In the embodiment depicted, customer VCN 104 includes two subnets, namely "Subnet-1" and "Subnet-2", each with its own CIDR IP address range. Figure 1 In this configuration, subnet-1 covers the IP address range of 10.0 / 16, and subnet-2 covers the address range of 10.1 / 16. VCN Virtual Router 105 represents the logical gateway of the VCN, enabling communication between subnets of VCN 104 and with other endpoints outside the VCN. VCN VR 105 is configured to route traffic between the VNICs within VCN 104 and the gateway associated with VCN 104. VCN VR 105 provides a port for each subnet of VCN 104. For example, VCN VR 105 could provide a port with IP address 10.0.0.1 for subnet-1 and a port with IP address 10.1.0.1 for subnet-2.

[0092] Multiple compute instances can be deployed on each subnet, where compute instances can be virtual machine instances and / or bare metal instances. Compute instances within a subnet can be hosted by one or more hosts within CSPI 101. Compute instances join the subnet via a VNIC associated with the compute instance. For example, as... Figure 1 As shown, compute instance C1 is part of subnet-1 via the VNIC associated with it. Similarly, compute instance C2 is part of subnet-1 via the VNIC associated with it. In a similar manner, multiple compute instances (which can be virtual machine instances or bare metal instances) can be part of subnet-1. Each compute instance is assigned a private overlay IP address and MAC address via its associated VNIC. For example, in... Figure 1In this subnet-1, compute instance C1 has an overlay IP address of 10.0.0.2 and a MAC address of M1, while compute instance C2 has a private overlay IP address of 10.0.0.3 and a MAC address of M2. Each compute instance in subnet-1 (including compute instances C1 and C2) has a default route to the VCN VR 105 using IP address 10.0.0.1, which is the IP address used by the VCN VR 105 for the port in subnet-1.

[0093] Subnet-2 can have multiple compute instances deployed on it, including virtual machine instances and / or bare metal instances. For example, as Figure 1 As shown, compute instances D1 and D2 are part of subnet-2 via the VNIC associated with the respective compute instance. Figure 1 In the embodiment depicted, compute instance D1 has an overlay IP address 10.1.0.2 and a MAC address MM1, while compute instance D2 has a private overlay IP address 10.1.0.3 and a MAC address MM2. Each compute instance in subnet-2 (including compute instances D1 and D2) has a default route to VCN VR 105 using IP address 10.1.0.1, which is the IP address used for the port of VCN VR 105 for subnet-2.

[0094] VCN A 104 may also include one or more load balancers. For example, a load balancer may be provided for a subnet and can be configured to load balance traffic across multiple compute instances on the subnet. A load balancer may also be provided to load balance traffic across subnets in a VCN.

[0095] A specific compute instance deployed on VCN 104 can communicate with a variety of different endpoints. These endpoints can include endpoints hosted by CSPI 200 and endpoints outside of CSPI 200. Endpoints hosted by CSPI 101 can include: endpoints on the same subnet as the specific compute instance (e.g., communication between two compute instances in subnet-1); endpoints on different subnets but within the same VCN (e.g., communication between a compute instance in subnet-1 and a compute instance in subnet-2); endpoints in different VCNs within the same region (e.g., communication between a compute instance in subnet-1 and an endpoint in a VCN within the same region, or communication between a compute instance in subnet-1 and an endpoint in service network 110 within the same region); or endpoints in VCNs in different regions (e.g., communication between a compute instance in subnet-1 and an endpoint in a VCN in a different region). Compute instances in subnets hosted by CSPI 101 can also communicate with endpoints not hosted by CSPI 101 (i.e., outside of CSPI 101). These external endpoints include endpoints in the customer’s on-premises deployment network 116, endpoints in other remote cloud-hosted networks 118, public endpoints 112 that are accessible via public networks such as the Internet, and other endpoints.

[0096] VNICs associated with the source and destination compute instances are used to facilitate communication between compute instances on the same subnet. For example, compute instance C1 in subnet-1 might want to send packets to compute instance C2 in subnet-1. For packets originating from the source compute instance and destined for another compute instance in the same subnet, the packets are first processed by the VNIC associated with the source compute instance. The processing performed by the VNIC associated with the source compute instance may include determining the packet's destination information from the packet header, identifying any policies (e.g., security lists) configured for the VNIC associated with the source compute instance, determining the packet's next hop, performing any packet encapsulation / decapsulation functions as needed, and then forwarding / routing the packet to the next hop to facilitate communication to its intended destination. When the destination compute instance is in the same subnet as the source compute instance, the VNIC associated with the source compute instance is configured to identify the VNIC associated with the destination compute instance and forward packets to that VNIC for processing. The VNIC associated with the destination compute instance then performs the processing and forwards the packets to the destination compute instance.

[0097] For packets to be communicated from compute instances in a subnet to endpoints in different subnets within the same VCN, communication is facilitated by the VNIC associated with the source and destination compute instances, as well as the VCN VR. For example, if Figure 1If compute instance C1 in subnet-1 wants to send a packet to compute instance D1 in subnet-2, the packet is first processed by the VNIC associated with compute instance C1. The VNIC associated with compute instance C1 is configured to route the packet to VCN VR 105 using the default route or port 10.0.0.1 of the VCN VR. VCN VR 105 is configured to route the packet to subnet-2 using port 10.1.0.1. The packet is then received and processed by the VNIC associated with D1, and the VNIC forwards the packet to compute instance D1.

[0098] For packets to be communicated from compute instances within VCN 104 to endpoints outside VCN 104, communication is facilitated by the VNIC associated with the source compute instance, VCN VR 105, and the gateway associated with VCN 104. One or more types of gateways can be associated with VCN 104. A gateway is an interface between a VCN and another endpoint located outside the VCN. A gateway is a Layer 3 / IP concept and enables a VCN to communicate with endpoints outside the VCN. Therefore, a gateway facilitates traffic flow between a VCN and other VCNs or networks. Various different types of gateways can be configured for a VCN to facilitate different types of communication with different types of endpoints. Depending on the gateway, communication can be conducted over a public network (e.g., the Internet) or over a private network. Various communication protocols can be used for these communications.

[0099] For example, compute instance C1 might want to communicate with an endpoint outside of VCN 104. The packet can first be processed by the VNIC associated with the source compute instance C1. The VNIC processing determines that the packet's destination is outside subnet-1 of C1. The VNIC associated with C1 can then forward the packet to VCN VR 105 of VCN 104. VCN VR 105 then processes the packet and, as part of the processing, determines a specific gateway associated with VCN 104 as the packet's next hop based on the packet's destination. VCN VR 105 can then forward the packet to the specifically identified gateway. For example, if the destination is an endpoint within the customer's on-premises network, the packet can be forwarded by VCN VR 105 to the DRG gateway 122 configured for VCN 104. The packet can then be forwarded from the gateway to the next hop to facilitate communication to its final intended destination.

[0100] Various types of gateways can be configured for a VCN. Examples of gateways that can be configured for a VCN are available in [link to example]. Figure 1 It is depicted in the image and described below. An example of a gateway associated with a VCN is also shown below. Figures 19-22The gateways depicted (e.g., those referenced by reference numerals 1934, 1936, 1938, 2034, 2036, 2038, 2134, 2136, 2238, 2334, 2336, and 2338) are described below. Figure 1 As illustrated in the embodiments depicted, DRG 122 can be added to or associated with customer VCN 104 and provides a path for private network traffic communication between customer VCN 104 and another endpoint, which can be customer's on-premises network 116, VCN 108 in a different region of CSPI 101, or another remote cloud network 118 not hosted by CSPI 101. Customer on-premises network 116 can be a customer network or customer data center built using customer resources. Access to customer on-premises network 116 is typically very restricted. For customers with both customer on-premises network 116 and one or more VCNs 104 deployed or hosted in the cloud by CSPI 101, the customer may want their on-premises network 116 and their cloud-based VCN 104 to be able to communicate with each other. This allows customers to build extended hybrid environments encompassing customer VCNs 104 hosted by CSPI 101 and their on-premises network 116. DRG 122 enables such communication. To enable such communication, a communication channel 124 is established, with one endpoint in the customer's local deployment network 116 and the other endpoint in CSPI 101 connected to the customer's VCN 104. Communication channel 124 can be over a public or private communication network, such as the Internet. Various communication protocols can be used, such as IPsec VPN technology over a public communication network (such as the Internet), Oracle's FastConnect technology using a private network instead of a public network, and so on. The device or equipment in the customer's local deployment network 116 forming one endpoint of communication channel 124 is referred to as customer premises equipment (CPE), such as... Figure 1 The CPE 126 is depicted in the diagram. On the CSPI 101 side, the endpoint can be a host executing DRG 122.

[0101] In some embodiments, a remote peering connection (RPC) can be added to the DRG, which allows a customer to peer one VCN with another VCN in a different region. Using such an RPC, customer VCN 104 can use DRG 122 to connect to VCN 108 in another region. DRG 122 can also be used to communicate with other remote cloud networks 118 that are not hosted by CSPI 101, such as Microsoft Azure Cloud, Amazon AWS Cloud, etc.

[0102] like Figure 1As shown, an Internet Gateway (IGW) 120 can be configured for a client VCN 104, enabling compute instances on VCN 104 to communicate with a public endpoint 112 accessible via a public network such as the Internet. The IGW 120 is a gateway connecting the VCN to a public network such as the Internet. The IGW 120 enables public subnets within the VCN 104 (where resources within the public subnet have publicly overriding IP addresses) to directly access the public endpoint 112 on the public network 114, such as the Internet. Using the IGW 120, connections can originate from subnets within VCN 104 or from the Internet.

[0103] Network Address Translation (NAT) gateway 128 can be configured for a customer's VCN 104 and enable cloud resources within the customer's VCN that do not have private public overlay IP addresses to access the Internet, and NAT gateway 128 does so without exposing those resources to direct inbound Internet connections (e.g., L4-L7 connections). This allows private subnets within the VCN (such as private subnet-1 in VCN 104) to have private access to public endpoints on the Internet. In the NAT gateway, connections can only be initiated from private subnets to the public Internet, and not from the Internet to private subnets.

[0104] In some embodiments, a Service Gateway (SGW) 126 may be configured for a client VCN 104 and provide a path for private network traffic between VCN 104 and supported service endpoints in service network 110. In some embodiments, service network 110 may be provided by a CSP and may offer a variety of services. An example of such a service network is Oracle's service network, which provides a variety of services that can be used by clients. For example, compute instances (e.g., database systems) in a private subnet of client VCN 104 may back up data to service endpoints (e.g., object storage) without requiring a public IP address or access to the Internet. In some embodiments, a VCN may have only one SGW, and connections may only originate from subnets within the VCN and not from service network 110. If a VCN peers to another VCN, resources in the other VCN typically cannot access the SGW. Resources in an on-premises network connected to a VCN using FastConnect or VPN Connect may also use the service gateway configured for that VCN.

[0105] In some implementations, SGW 126 uses the concept of a Service Classless Inter-Domain Routing (CIDR) label, which is a string representing all regional ranges of public IP addresses for the service or group of services of interest. Customers use service CIDR labels to control traffic to services when configuring SGW and related routing rules. Customers can optionally use service CIDR labels when configuring security rules to avoid having to adjust them if the public IP addresses of services change in the future.

[0106] Local peering gateway (LPG) 132 is a gateway that can be added to customer VCN 104 and enable VCN 104 to peer with another VCN in the same area. Peering means that VCNs communicate using private IP addresses without traffic traversing public networks such as the Internet or routing traffic through the customer's locally deployed network 116. In a preferred embodiment, a VCN has a separate LPG for each peer it establishes. Local peering, or VCN peering, is a common practice for establishing network connectivity between different applications or infrastructure management functions.

[0107] Service providers (such as service providers in service network 110) can use different access models to provide access to services. Under the public access model, services can be exposed as public endpoints, publicly accessible by compute instances in a customer's VCN via public networks such as the Internet and / or privately accessible via SGW 126. Under a specific private access model, services can be accessed as private IP endpoints within a private subnet of the customer's VCN. This is called private endpoint (PE) access and allows service providers to expose their services as instances within the customer's private network. A private endpoint resource represents a service within the customer's VCN. Each PE is represented as a VNIC (referred to as a PE-VNIC, with one or more private IPs) within a subnet selected by the customer within the customer's VCN. Thus, a PE provides a method of presenting a service within a private customer VCN subnet using a VNIC. Because the endpoint is exposed as a VNIC, all characteristics associated with the VNIC (such as routing rules, security lists, etc.) are now available to the PE VNIC.

[0108] Service providers can register their services to make them accessible via a PE (Private Provider). Providers can associate policies with services, which restricts the visibility of services to customer leases. Providers can register multiple services under a single Virtual IP address (VIP), especially for multi-tenant services. Multiple such private endpoints (across multiple VCNs) can exist representing the same service.

[0109] Compute instances in a private subnet can then access the service using the private IP address or service DNS name of the PE VNIC. Compute instances in a customer VCN can access the service by sending traffic to the private IP address of the PE in the customer VCN. A Private Access Gateway (PAGW) 130 is a gateway resource that can be attached to a service provider VCN (e.g., a VCN in service network 110), acting as the ingress / egress point for all traffic originating from / to the private endpoint of the customer subnet. PAGW 130 allows providers to scale the number of PE connections without utilizing their internal IP address resources. A provider only needs to configure one PAGW for any number of services registered in a single VCN. A provider can represent a service as a private endpoint in multiple VCNs of one or more customers. From the customer's perspective, the PE VNIC is not an instance attached to the customer, but rather appears to be attached to the service the customer wishes to interact with. Traffic destined for the private endpoint is routed to the service via PAGW 130. These are referred to as customer-to-service private connections (C2S connections).

[0110] The PE concept can also be used to extend private access to services to a customer's on-premises network and data center by allowing traffic to flow through FastConnect / IPsec links and private endpoints within the customer's VCN. Private access to services can also be extended to the customer's peering VCN by allowing traffic to flow between LPG 132 and the PE within the customer's VCN.

[0111] Customers can control routing within a VCN at the subnet level, allowing them to specify which subnets within a customer's VCN (such as VCN104) use each gateway. The VCN's routing table determines whether traffic is allowed to leave the VCN via a specific gateway. For example, in a given instance, the routing table for the public subnet within customer VCN 104 might allow non-local traffic to be sent via IGW 120. The routing table for the private subnet within the same customer VCN 104 might allow traffic destined for CSP services to be sent via SGW 126. All remaining traffic can be sent via NAT gateway 128. The routing table only controls traffic leaving the VCN.

[0112] Security lists associated with a VCN are used to control traffic entering the VCN via a gateway through inbound connections. All resources within a subnet use the same routing tables and security lists. Security lists can be used to control specific types of traffic allowed to enter or leave instances within a VCN subnet. Security list rules can include inbound and outbound rules. For example, inbound rules can specify allowed ranges of source addresses, while outbound rules can specify allowed ranges of destination addresses. Security rules can specify specific protocols (e.g., TCP, ICMP), specific ports (e.g., 22 for SSH, 3389 for Windows RDP), etc. In some implementations, the instance's operating system can enforce its own firewall rules consistent with the security list rules. Rules can be stateful (e.g., connections are tracked and responses are automatically allowed in the absence of explicit security list rules for response traffic) or stateless.

[0113] Access from a customer's VCN (i.e., through resources or compute instances deployed on VCN 104) can be categorized as public access, private access, or dedicated access. Public access refers to an access model where a public IP address or NAT is used to access a public endpoint. Private access enables customer workloads (e.g., resources in a private subnet) within VCN 104 with private IP addresses to access services without traversing a public network such as the Internet. In some embodiments, CSPI 101 enables customer VCN workloads with private IP addresses to access the service (its public service endpoint) using a service gateway. Thus, the service gateway provides a private access model by establishing a virtual link between the customer's VCN and the public endpoint of the service residing outside the customer's private network.

[0114] Additionally, CSPI can use technologies such as FastConnect public peering to provide dedicated public access, where customer-locally deployed instances can use FastConnect connections to access one or more services within the customer's VCN without traversing public networks such as the Internet. CSPI can also use FastConnect private peering to provide dedicated private access, where customer-locally deployed instances with private IP addresses can use FastConnect connections to access the customer's VCN workloads. FastConnect is an alternative to using the public Internet to connect a customer's local network to CSPI and its services for network connectivity. Compared to Internet-based connections, FastConnect provides a simple, resilient, and cost-effective way to create dedicated and private connections with higher bandwidth options and a more reliable and consistent networking experience.

[0115] Figure 1The description attached above illustrates the various virtualization components in the example virtual network. As mentioned above, the virtual network is built on top of the underlying physical or base network. Figure 2 A simplified architecture diagram is depicted within the physical network of a CSPI 200, according to certain embodiments, illustrating the underlying physical components that provide the underlying physical network for virtual networks. As shown, CSPI 200 provides a distributed environment comprising components and resources (e.g., compute, memory, and networking resources) provided by a CSP. These components and resources are used to provide cloud services (e.g., IaaS services) to subscribing customers (i.e., customers who have subscribed to one or more services provided by the CSP). Based on the services subscribed to by the customer, a subset of the resources of CSPI 200 (e.g., compute, memory, and networking resources) is provisioned to the customer. The customer can then use the physical compute, memory, and networking resources provided by CSPI 200 to build their own customizable and private cloud-based (i.e., CSPI-hosted) virtual networks. As previously indicated, these customer networks are referred to as VCNs. Customers can deploy one or more customer resources, such as compute instances, on these customer VCNs. Compute instances can take the form of virtual machines, bare metal instances, etc. CSPI 200 provides infrastructure and a complementary set of cloud services that enable customers to build and run a wide range of applications and services in a highly available managed environment.

[0116] exist Figure 2 In the example embodiment depicted, the physical components of CSPI 200 include one or more physical hosts or physical servers (e.g., 202, 206, 208), network virtualization devices (NVDs) (e.g., 210, 212), ToR switches (e.g., 214, 216), and a physical network (e.g., 218), as well as switches within physical network 218. The physical hosts or servers can host and execute various compute instances that join one or more subnets of the VCN. Compute instances can include virtual machine instances and bare metal instances. For example, Figure 1 The various computational examples described in the text can be derived from... Figure 2 The physical host described in the text refers to virtual machine compute instances in a VCN that can be executed by a single host or multiple different hosts. Physical hosts can also host virtual machines, container-based hosts, or functionalities, etc. Figure 1 The VNIC and VCN VR described in the text can be generated by Figure 2 The NVD execution described in the text. Figure 1 The gateway described in the text can be... Figure 2 The host and / or NVD execution described in the document.

[0117] A host or server can execute a hypervisor (also known as a virtual machine monitor or VMM) that creates and enables virtualized environments on the host. Virtualization or virtualized environments facilitate cloud-based computing. One or more compute instances can be created, executed, and managed on the host by a hypervisor on the host. The hypervisor on the host enables groups of physical computing resources (e.g., compute, storage, and networking resources) of the host to be shared among various compute instances executed by the host.

[0118] For example, such as Figure 2 As depicted, hosts 202 and 208 execute hypervisors 260 and 266, respectively. These hypervisors can be implemented using software, firmware, or hardware, or a combination thereof. Typically, a hypervisor is a process or software layer that sits above the host operating system (OS), which in turn executes on the host's hardware processor. Hypervisors provide a virtualization environment by enabling the sharing of the host's physical computing resources (e.g., processing resources such as processors / cores, memory resources, networking resources) among various virtual machine computing instances executed by the host. For example, in... Figure 2 In this configuration, hypervisor 260 can reside on the operating system of host 202 and enable the computing resources (e.g., processing, memory, and networking resources) of host 202 to be shared among computing instances (e.g., virtual machines) executed by host 202. Virtual machines can have their own operating systems (called guest operating systems), which can be the same as or different from the host's operating system. The operating system of a virtual machine executed by the host can be the same as or different from the operating system of another virtual machine executed by the same host. Therefore, the hypervisor enables multiple operating systems to run simultaneously, sharing the same set of computing resources of the host. Figure 2 The hosts described in the text may have the same or different types of management programs.

[0119] A compute instance can be a virtual machine instance or a bare metal instance. Figure 2 In the diagram, compute instance 268 on host 202 and compute instance 274 on host 208 are examples of virtual machine instances. Host 206 is an example of a bare metal instance provided to a customer.

[0120] In some instances, an entire host can be provisioned to a single customer, and all compute instances (virtual machines or bare metal instances) hosted by that host belong to the same customer. In other instances, the host can be shared among multiple customers (i.e., multiple tenants). In such multi-tenant scenarios, the host can host virtual machine compute instances belonging to different customers. These compute instances can be members of different VCNs for different customers. In some embodiments, bare metal compute instances are hosted by bare metal servers without a hypervisor. When a bare metal compute instance is provisioned, a single customer or tenant maintains control over the physical CPU, memory, and network interfaces of the host hosting the bare metal instance, and the host is not shared with other customers or tenants.

[0121] As previously described, each compute instance, as part of a VCN, is associated with a VNIC that enables the compute instance to become a member of a subnet of the VCN. The VNIC associated with a compute instance facilitates communication to and from the compute instance for packets or frames. A VNIC is associated with a compute instance when it is created. In some embodiments, for compute instances executed by a host, the VNIC associated with that compute instance is executed by an NVD connected to the host. For example, in Figure 2 In this example, host 202 executes a virtual machine compute instance 268 associated with VNIC 276, and VNIC 276 is executed by NVD 210 connected to host 202. As another example, a bare metal instance 272 hosted by host 206 is associated with VNIC 280 executed by NVD 212 connected to host 206. As yet another example, VNIC 284 is associated with compute instance 274 executed by host 208, and VNIC 284 is executed by NVD 212 connected to host 208.

[0122] For compute instances hosted by a host, NVDs connected to that host also execute VCN VR corresponding to the VCN where the compute instance is a member. For example, in Figure 2 In the embodiment depicted, NVD 210 executes VCN VR 277 corresponding to the VCN of compute instance 268, which is a member of NVD 212. NVD 212 may also execute one or more VCN VR 283 corresponding to the VCNs of compute instances hosted by hosts 206 and 208.

[0123] A host may include one or more network interface cards (NICs) that enable the host to connect to other devices. The NIC on the host may provide one or more ports (or interfaces) that allow the host to communicatively connect to another device. For example, the host may use one or more ports (or interfaces) provided on the host and the NVD to connect to the NVD. The host may also connect to other devices, such as another host.

[0124] For example, in Figure 2 In this configuration, host 202 connects to NVD 210 via link 220, which extends between port 234 provided by NIC 232 of host 202 and port 236 of NVD 210. Host 206 connects to NVD 212 via link 224, which extends between port 246 provided by NIC 244 of host 206 and port 248 of NVD 212. Host 208 connects to NVD 212 via link 226, which extends between port 252 provided by NIC 250 of host 208 and port 254 of NVD 212.

[0125] The NVD is then connected to the ToR switch via a communication link, which is connected to physical network 218 (also known as a switch architecture). In some embodiments, the links between the host and the NVD, and between the NVD and the ToR switch, are Ethernet links. For example, in Figure 2 In this configuration, NVDs 210 and 212 are connected to ToR switches 214 and 216 via links 228 and 230, respectively. In some embodiments, links 220, 224, 226, 228, and 230 are Ethernet links. The collection of hosts and NVDs connected to the ToR is sometimes referred to as a rack.

[0126] Physical network 218 provides a communication structure that enables ToR switches to communicate with each other. Physical network 218 can be a multi-level network. In some implementations, physical network 218 is a multi-level Clos network of switches, where ToR switches 214 and 216 represent leaf-level nodes of the multi-level and multi-node physical switching network 218. Different Clos network configurations are possible, including but not limited to 2-level networks, 3-level networks, 4-level networks, 5-level networks, and general "n"-level networks. Examples of Clos networks are provided in... Figure 5 It is depicted in the middle and described below.

[0127] Various connection configurations between the host and the NVD, such as one-to-one, many-to-one, and one-to-many configurations, are possible. In a one-to-one configuration implementation, each host connects to its own individual NVD. For example, in... Figure 2 In this configuration, host 202 connects to NVD 210 via NIC 232. In a many-to-one configuration, multiple hosts connect to a single NVD. For example, in... Figure 2 In this configuration, hosts 206 and 208 are connected to the same NVD 212 via NICs 244 and 250, respectively.

[0128] In a one-to-many configuration, one host connects to multiple NVDs. Figure 3An example within the CSPI 300 is shown, where the host is connected to multiple NVDs. (Example follows) Figure 3 As shown, host 302 includes a network interface card (NIC) 304, which includes multiple ports 306 and 308. Host 300 is connected to a first NVD 310 via port 306 and link 320, and to a second NVD 312 via port 308 and link 322. Ports 306 and 308 may be Ethernet ports, and links 320 and 322 between host 302 and NVDs 310 and 312 may be Ethernet links. NVD 310 is then connected to a first ToR switch 314, and NVD 312 is connected to a second ToR switch 316. Links between NVDs 310 and 312 and between ToR switches 314 and 316 may be Ethernet links. ToR switches 314 and 316 represent tier 0 switching devices in a multi-level physical network 318.

[0129] Figure 3 The arrangement depicted provides two separate physical network paths to and from physical switch network 318 to host 302: a first path through ToR switch 314 to NVD 310 to host 302, and a second path through ToR switch 316 to NVD 312 to host 302. These separate paths provide enhanced availability (referred to as high availability) for host 302. If one of the paths (e.g., a link failure in one of the paths) or devices (e.g., a particular NVD not functioning) has a problem, the other path can be used for communication to / from host 302.

[0130] exist Figure 3 In the configuration depicted, the host uses two different ports provided by the host's NIC to connect to two different NVDs. In other embodiments, the host may include multiple NICs enabling host-to-multiple NVD connectivity.

[0131] Return to reference Figure 2 An NVD is a physical device or component that performs one or more network and / or storage virtualization functions. An NVD can be any device with one or more processing units (e.g., CPU, Network Processing Unit (NPU), FPGA, packet processing pipeline, etc.), memory including cache, and ports. Various virtualization functions can be executed by software / firmware performed by one or more processing units of the NVD.

[0132] NVDs can be implemented in various different forms. For example, in some embodiments, an NVD is implemented as an interface card called a smartNIC or intelligent NIC with an embedded processor. A smartNIC is a device separate from the NIC on the host machine. Figure 2In this context, NVD 210 and 212 can be implemented as smartNICs connected to host 202 and hosts 206 and 208, respectively.

[0133] However, smartNIC is just one example of an NVD implementation. Various other implementations are possible. For example, in some other implementations, the NVD, or one or more functions performed by the NVD, may be incorporated into or performed by one or more hosts, one or more ToR switches, and other components of the CSPI 200. For instance, the NVD may be embodied in a host, where the functions performed by the NVD are performed by the host. As another example, the NVD may be part of a ToR switch, or the ToR switch may be configured to perform functions performed by the NVD, enabling the ToR switch to perform various complex packet transformations for public clouds. A ToR that performs NVD functions is sometimes referred to as a smart ToR. In yet another implementation, where virtual machine (VM) instances, rather than bare metal (BM) instances, are provided to customers, the functions performed by the NVD may be implemented within the host's hypervisor. In some other implementations, some of the functions of the NVD may be offloaded to a centralized service running on a series of hosts.

[0134] In some embodiments, such as when implemented as Figure 2 When a smartNIC is shown, an NVD can include multiple physical ports that enable the NVD to connect to one or more hosts and one or more ToR switches. Ports on an NVD can be classified as host-facing ports (also known as "south ports") or network-facing or ToR-facing ports (also known as "north ports"). Host-facing ports on an NVD are those used to connect the NVD to a host. Figure 2 Examples of host-facing ports include port 236 on the NVD 210 and ports 248 and 254 on the NVD 212. Network-facing ports on the NVD are used to connect the NVD to the ToR switch. Figure 2 Examples of network-facing ports include port 256 on the NVD 210 and port 258 on the NVD 212. Figure 2 As shown, NVD 210 is connected to ToR switch 214 via link 228, which extends from port 256 of NVD 210 to ToR switch 214. Similarly, NVD 212 is connected to ToR switch 216 via link 230, which extends from port 258 of NVD 212 to ToR switch 216.

[0135] The NVD receives packets and frames from the host (e.g., packets and frames generated by compute instances hosted on the host) via its host-facing port, and after performing the necessary packet processing, can forward the packets and frames to the ToR switch via the NVD's network-facing port. The NVD can also receive packets and frames from the ToR switch via its network-facing port, and after performing the necessary packet processing, can forward the packets and frames to the host via the NVD's host-facing port.

[0136] In some embodiments, there can be multiple ports and associated links between the NVD and the ToR switch. These ports and links can be aggregated to form a group of link aggregators (called a LAG) of multiple ports or links. Link aggregation allows multiple physical links between two endpoints (e.g., between the NVD and the ToR switch) to be treated as a single logical link. All physical links in a given LAG can operate at the same speed in full-duplex mode. LAGs help improve the bandwidth and reliability of the connection between two endpoints. If one physical link in the LAG fails, traffic is dynamically and transparently reassigned to one of the other physical links in the LAG. Aggregated physical links deliver higher bandwidth than each individual link. Multiple ports associated with an LAG are treated as a single logical port. Traffic can be load balanced across multiple physical links in the LAG. One or more LAGs can be configured between two endpoints. The two endpoints can be located between the NVD and the ToR switch, between a host and the NVD, etc.

[0137] The NVD implements or performs network virtualization functions. These functions are performed by the software / firmware executed by the NVD. Examples of network virtualization functions include, but are not limited to: packet encapsulation and decapsulation functions; functions for creating VCN networks; functions for implementing network policies such as VCN security lists (firewalls); functions for facilitating the routing and forwarding of packets to and from compute instances in the VCN; and so on. In some embodiments, upon receiving a packet, the NVD is configured to perform a packet processing pipeline to process the packet and determine how the packet is forwarded or routed. As part of this packet processing pipeline, the NVD may perform one or more virtual functions associated with the overlay network, such as performing VNICs associated with compute instances in the VCN, performing virtual routers (VRs) associated with the VCN, packet encapsulation and decapsulation to facilitate forwarding or routing in the virtual network, execution of certain gateways (e.g., local peer gateways), security lists, network security groups, Network Address Translation (NAT) functions (e.g., public IP to private IP translation on a per-host basis), throttling functions, and other functions.

[0138] In some embodiments, the packet processing data path in NVD may include multiple packet pipelines, each consisting of a series of packet transformation stages. In some implementations, upon receiving a packet, it is parsed and classified into a single pipeline. The packet is then processed linearly, stage by stage, until it is dropped or sent out through the NVD's interface. These stages provide basic functional packet processing building blocks (e.g., header verification, throttling, inserting new Layer-2 headers, implementing L4 firewalls, VCN encapsulation / decapsulation, etc.), allowing new pipelines to be constructed by combining existing stages, and new functionality to be added by creating new stages and inserting them into existing pipelines.

[0139] NVD can perform both control plane and data plane functions corresponding to the control plane and data plane of VCN. Examples of VCN CP are also available. Figures 19-22 (See figures 1916, 2016, 2116 and 2216) depicted and described below. An example of the VCN data plane is shown in... Figures 19-22 (See figures 1918, 2018, 2118, and 2218) as depicted and described below. Control plane functions include those for configuring how control data is forwarded on the network (e.g., establishing routes and routing tables, configuring VNICs, etc.). In some embodiments, a VCN CP is provided, which centrally computes all overlay mappings to the base layer and publishes them to the NVD and to virtual network edge devices such as various gateways (e.g., DRGs, SGWs, IGWs, etc.). Firewall rules can also be published using the same mechanism. In some embodiments, the NVD only acquires mappings relevant to that NVD. Data plane functions include those for actual routing / forwarding of packets based on the configuration established using the control plane. The VCN data plane is implemented by encapsulating client network packets before they traverse the base layer network. Encapsulation / decapsulation functionality is implemented on the NVD. In some embodiments, the NVD is configured to intercept all network packets entering and leaving the host and perform network virtualization functions.

[0140] As indicated above, the NVD performs various virtualization functions, including VNICs and VCN VR. The NVD can execute VNICs associated with compute instances hosted by one or more hosts connected to the VNIC. For example, as... Figure 2As depicted, NVD 210 performs the functions of VNIC 276 associated with compute instance 268 hosted by host 202 connected to NVD 210. As another example, NVD 212 performs VNIC 280 associated with bare-metal compute instance 272 hosted by host 206, and VNIC 284 associated with compute instance 274 hosted by host 208. Hosts can host compute instances belonging to different VCNs belonging to different clients, and NVDs connected to hosts can perform VNICs corresponding to the compute instances (i.e., perform functions associated with the VNICs).

[0141] NVD also executes a VCN virtual router corresponding to the VCN of the compute instance. For example, in Figure 2 In the embodiments depicted, NVD 210 executes VCN VR 277 corresponding to the VCN to which compute instance 268 belongs. NVD 212 executes one or more VCN VR 283 corresponding to one or more VCNs to which the compute instances hosted by hosts 206 and 208 belong. In some embodiments, the VCN VR corresponding to that VCN is executed by all NVDs connected to hosts hosting at least one compute instance belonging to that VCN. If a host hosts compute instances belonging to different VCNs, the NVDs connected to that host can execute VCN VR corresponding to those different VCNs.

[0142] In addition to VNIC and VCN VR, NVD can execute various software (e.g., daemons) and includes one or more hardware components that facilitate the various network virtualization functions performed by NVD. For simplicity, these individual components are grouped together as... Figure 2The “packet processing component” is described in the diagram. For example, NVD 210 includes packet processing component 286, and NVD 212 includes packet processing component 288. For instance, the packet processing component of an NVD may include a packet processor configured to interact with the NVD’s ports and hardware interfaces to monitor all packets received by and used by the NVD and to store network information. Network information may include, for example, network flow information identifying different network flows handled by the NVD and per-flow information (e.g., per-flow statistics). In some embodiments, network flow information may be stored on a per-VNIC basis. The packet processor may perform packet-by-packet manipulation and implement stateful NAT and L4 firewall (FW). As another example, the packet processing component may include a replication agent configured to replicate information stored by the NVD to one or more different replication target stores. As yet another example, the packet processing component may include a logging agent configured to perform logging functions of the NVD. The packet processing component may also include software for monitoring the performance and health of the NVD and may also monitor the status and health of other components connected to the NVD.

[0143] Figure 1 The components of an example virtual or overlay network are shown, including a VCN, subnets within the VCN, compute instances deployed on the subnets, VNICs associated with the compute instances, the VCN's VR, and a set of gateways configured for the VCN. Figure 1 The overlay component described in the text can be made by Figure 2 One or more physical components described in the diagram are executed or managed. For example, a compute instance in a VCN can be executed or managed by... Figure 2 The VNIC described herein is executed or hosted by one or more hosts. For compute instances hosted by a host, the VNIC associated with that compute instance is typically executed by an NVD connected to that host (i.e., VNIC functionality is provided by an NVD connected to that host). The VCN VR functionality of a VCN is executed by all NVDs connected to the host that hosts or executes a compute instance that is part of that VCN. Gateways associated with a VCN can be executed by one or more different types of NVDs. For example, some gateways can be executed by smartNICs, while others can be executed by one or more hosts or other implementations of NVDs.

[0144] As described above, compute instances in a client VCN can communicate with a variety of different endpoints, which can be within the same subnet as the source compute instance, in a different subnet but within the same VCN as the source compute instance, or with endpoints outside the source compute instance's VCN. These communications are facilitated using the VNIC, VCN VR, and gateway associated with the VCN, all of which are linked to the compute instance.

[0145] For communication between two compute instances on the same subnet within a VCN, this communication is facilitated using VNICs associated with the source and destination compute instances. The source and destination compute instances can be hosted by the same host or different hosts. Packets originating from the source compute instance can be forwarded from the host hosting the source compute instance to an NVD connected to that host. On the NVD, packets are processed using a packet processing pipeline that may include the execution of the VNIC associated with the source compute instance. Because the destination endpoint of the packet is within the same subnet, the execution of the VNIC associated with the source compute instance results in the packet being forwarded to the NVD executing the VNIC associated with the destination compute instance, which then processes the packet and forwards it to the destination compute instance. The VNICs associated with the source and destination compute instances can execute on the same NVD (e.g., when both the source and destination compute instances are hosted by the same host) or on different NVDs (e.g., when the source and destination compute instances are hosted by different hosts connected to different NVDs). The VNIC can use the routing / forwarding tables stored in the NVD to determine the next hop for the packet.

[0146] For packets destined for endpoints in different subnets within the same VCN, the packet originating from the source compute instance is forwarded from the host hosting the source compute instance to the NVD connected to that host. On the NVD, a packet processing pipeline is used to process the packet, which may include the execution of one or more VNICs and a VR associated with the VCN. For example, as part of the packet processing pipeline, the NVD executes or invokes a function corresponding to the VNIC associated with the source compute instance (also known as executing the VNIC). The function executed by the VNIC may include viewing the VLAN tag on the packet. Since the packet's destination is outside the subnet, the VCN VR function is then invoked and executed by the NVD. The VCN VR then routes the packet to the NVD executing the VNIC associated with the destination compute instance. The VNIC associated with the destination compute instance then processes the packet and forwards it to the destination compute instance. The VNICs associated with the source and destination compute instances may execute on the same NVD (e.g., when both the source and destination compute instances are hosted by the same host) or on different NVDs (e.g., when the source and destination compute instances are hosted by different hosts connected to different NVDs).

[0147] If the packet's destination is outside the source compute instance's VCN, the packet originating from the source compute instance is passed from the host hosting the source compute instance to the NVD connected to that host. The NVD executes the VNIC associated with the source compute instance. Since the packet's destination endpoint is outside the VCN, the packet is then processed by the VCN VR of that VCN. The NVD invokes VCN VR functionality, which allows the packet to be forwarded to the NVD executing the appropriate gateway associated with the VCN. For example, if the destination is an endpoint within the customer's on-premises network, the packet may be forwarded by the VCN VR to the NVD executing the DRG gateway configured for the VCN. The VCN VR may be executed on the same NVD as the NVD executing the VNIC associated with the source compute instance or by a different NVD. The gateway may be executed by an NVD, which can be a smartNIC, a host, or another NVD implementation. The packet is then processed by the gateway and forwarded to the next hop that facilitates communication to its intended destination endpoint. For example, in Figure 2 In the embodiment depicted, packets originating from compute instance 268 can be transmitted from host 202 to NVD 210 via link 220 (using NIC 232). On NVD 210, VNIC 276 is invoked because it is the VNIC associated with the source compute instance 268. VNIC 276 is configured to examine the encapsulation information in the packets and determine the next hop for forwarding the packets, with the aim of facilitating communication of the packets to their intended destination endpoint, and then forward the packets to the determined next hop.

[0148] Compute instances deployed on a VCN can communicate with a variety of different endpoints. These endpoints can include endpoints hosted by CSPI 200 and endpoints outside of CSPI 200. Endpoints hosted by CSPI 200 can include instances within the same VCN or other VCNs, which can be the customer's VCN or a VCN not belonging to the customer. Communication between endpoints hosted by CSPI 200 can be performed via physical network 218. Compute instances can also communicate with endpoints not hosted by CSPI 200 or outside of CSPI 200. Examples of these endpoints include endpoints within the customer's on-premises deployment network or data center, or public endpoints accessible via public networks such as the Internet. Communication with endpoints outside of CSPI 200 can use various communication protocols over public networks (e.g., the Internet). Figure 2 (not shown in the image) or private network ( Figure 2 (Not shown in the image) to execute.

[0149] Figure 2 The architecture of the CSPI 200 depicted herein is merely illustrative and not intended to be limiting. Variations, substitutions, and modifications are possible in alternative embodiments. For example, in some implementations, the CSPI 200 may have more advanced features than... Figure 2 The systems or components shown may include more or fewer systems or components, and may combine two or more systems, or may have different system configurations or arrangements. Figure 2 The systems, subsystems, and other components described herein are implemented using hardware or a combination thereof, with software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective systems. The software may be stored on a non-transitory storage medium (e.g., a memory device).

[0150] Figure 4 The diagram describes connectivity between multi-tenant hosts and NVDs, according to certain embodiments. Figure 4 As depicted, host 402 executes a hypervisor 404 that provides a virtualization environment. Host 402 executes two virtual machine instances, VM1 406 belonging to customer / tenant #1 and VM2 408 belonging to customer / tenant #2. Host 402 includes a physical NIC 410 connected to NVD 412 via link 414. Each compute instance in the compute instances is attached to a VNIC executed by NVD 412. Figure 4 In one embodiment, VM1 406 is attached to VNIC-VM1 420 and VM2 408 is attached to VNIC-VM2 422.

[0151] like Figure 4 As shown, NIC 410 includes two logical NICs, logical NIC A 416 and logical NIC B 418. Each virtual machine is attached to its own logical NIC and configured to work with its own logical NIC. For example, VM1 406 is attached to logical NIC A 416, and VM2 408 is attached to logical NIC B 418. Although host 402 includes only one physical NIC 410 shared by multiple tenants, each tenant's virtual machines believe they have their own host and NIC due to the logical NICs.

[0152] In some embodiments, each logical NIC is assigned its own VLAN ID. Thus, a specific VLAN ID is assigned to logical NIC A 416 for tenant #1, and a separate VLAN ID is assigned to logical NIC B 418 for tenant #2. When a packet is delivered from VM1 406, the tag assigned to tenant #1 is attached to the packet by the hypervisor, and the packet is then delivered from host 402 to NVD 412 via link 414. Similarly, when a packet is delivered from VM2 408, the tag assigned to tenant #2 is attached to the packet by the hypervisor, and the packet is then delivered from host 402 to NVD 412 via link 414. Therefore, the packet 424 delivered from host 402 to NVD 412 has an associated tag 426 that identifies the specific tenant and associated VM. On the NVD, for a packet 424 received from host 402, the tag 426 associated with the packet is used to determine whether the packet should be processed by VNIC-VM1 420 or VNIC-VM2 422. The packet is then processed by the corresponding VNIC. Figure 4 The configuration described in the document allows each tenant's compute instance to believe that it owns its own host and NIC. Figure 4 The settings described in [the document] provide I / O virtualization for supporting multi-tenancy.

[0153] Figure 5 A simplified block diagram of a physical network 500 according to certain embodiments is depicted. Figure 5 The embodiments depicted are structured as Clos networks. A Clos network is a specific type of network topology designed to provide connectivity redundancy while maintaining high distributed bandwidth and maximum resource utilization. A Clos network is a non-blocking, multi-stage or multi-level switching network, wherein the number of stages or levels can be two, three, four, five, etc. Figure 5 The embodiment depicted is a three-tier network including tiers 1, 2, and 3. The ToR switch 504 represents a tier-0 switch in the Clos network. One or more NVDs are connected to the ToR switch. The tier-0 switch is also referred to as the edge device of the physical network. The tier-0 switch connects to a tier-1 switch, also referred to as a leaf switch layer. Figure 5In the embodiment depicted, a set of n Layer-0 ToR switches is connected to a set of n Layer-1 switches, forming a pod. Each Layer-0 switch in a pod is interconnected with all Layer-1 switches in that pod, but there is no switch connectivity between pods. In some implementations, two pods are referred to as blocks. Each block is served by or connected to a set of n Layer-2 switches (sometimes called backbone switches). Several blocks may exist in the physical network topology. The Layer-2 switches are then connected to n Layer-3 switches (sometimes called super backbone switches). Packet communication on the physical network 500 is typically performed using one or more Layer-3 communication protocols. Typically, all layers of the physical network, except the TOR layer, are n-way redundant, allowing for high availability. Policies can be specified for pods and blocks to control the visibility of switches to each other in the physical network, enabling scaling of the physical network.

[0154] A key characteristic of Clos networks is that the maximum number of hops from one Layer-0 switch to another Layer-0 switch (or from an NVD connected to a Layer-0 switch to another NVD connected to a Layer-0 switch) is fixed. For example, in a Layer 3 Clos network, a maximum of seven hops are required for a packet to travel from one NVD to another, where the source and destination NVDs are connected to the leaf layers of the Clos network. Similarly, in a Layer 4 Clos network, a maximum of nine hops are required for a packet to travel from one NVD to another, where the source and destination NVDs are connected to the leaf layers of the Clos network. Therefore, the Clos network architecture maintains consistent latency throughout the network, which is important for communication within and between data centers. Clos topologies are horizontally scalable and cost-effective. Network bandwidth / throughput can be easily increased by adding more switches at each layer (e.g., more leaf and backbone switches) and by increasing the number of links between switches at adjacent layers.

[0155] In some embodiments, each resource within the CSPI is assigned a unique identifier called a Cloud Identifier (CID). This identifier is included as part of the resource's information and can be used to manage the resource, for example, via a console or API. An example syntax for a CID is:

[0156] ocid1.<RESOURCE TYPE> . <realm>[REGION][FUTURE USE].<UNIQUE ID>

[0157] in,

[0158] ocid1: A literal string indicating the CID version;

[0159] Resource type: The type of resource (e.g., instance, volume, VCN, subnet, user, group, etc.).

[0160] Realm: The domain in which the resource resides (example values ​​"c1" for the commercial domain, "c2" for the government cloud domain, or "c3" for the federal government cloud domain, etc. Each domain can have its own domain name).

[0161] Region: The region where the resource is located (this part can be empty if the region is not applicable to the resource).

[0162] Future use: (reserved for future use); and.

[0163] Unique ID: The unique part of the ID (the format may vary depending on the type of resource or service).

[0164] Multi-cloud

[0165] Figure 6 A simplified high-level diagram 600 depicts a distributed environment comprising multiple cloud environments provided by different CSPs. (See diagram 600.) Figure 6 As described herein, various cloud environments (also referred to as "clouds") can be provided by different CSPs. Each cloud environment or cloud provides one or more cloud services, which can be subscribed to by one or more customers of the corresponding CSP. The cloud service group provided by a cloud environment offered by a CSP can include one or more different types of cloud services, including but not limited to SaaS services, IaaS services, PaaS services, Database as a Service (DBaaS) services, etc. Examples of cloud environments provided by various CSPs include OCI, Azure, Google Cloud, AWS, etc. The cloud services provided by a particular cloud environment may differ from the cloud service group provided by another cloud environment.

[0166] In a typical cloud environment, a Cloud Service Provider (CSP) provides a Cloud Service Provider Interface (CSPI) to its customers, offering a group of cloud services provided by that cloud environment. The CSPI provided by the CSP can include various types of hardware and software resources, including consoles for accessing cloud services, computing resources, storage resources, networking resources, etc. Customers of a cloud environment provided by a CSP can subscribe to one or more cloud services offered by that environment. The CSP can offer various subscription models to its customers. After a customer subscribes to a cloud service provided by the cloud environment, one or more users can be associated with the subscribed customer, and such users can use the cloud services subscribed to by the customer. In some implementations, when a customer subscribes to a cloud service provided by a specific cloud environment, a customer account or customer lease is created for that customer. One or more users can then be associated with the customer lease, and such users can then use the services subscribed to by the customer under the customer lease. Information about the services subscribed to by the customer, the users associated with the customer lease, etc., is typically stored within the cloud environment and associated with the customer lease.

[0167] For example, Figure 6 The document describes two different cloud environments provided by two different CSPs (although different numbers of cloud environments are possible). These include cloud environment A (Cloud A) 610 provided by CSP A and cloud environment B (Cloud B) 640 provided by CSP B.

[0168] Cloud A 610 includes infrastructure CSPI_A 612 provided by CSP A. This infrastructure CSPI_A 612 can be used to provide a set of in-cloud services 615 provided by Cloud A 610. One or more customers (e.g., Cust_A1 616-1, Cust_A2 616-2) can subscribe to one or more of these services. One or more users 618-1 can be associated with customer Cust_A1 616-1 and can use the services subscribed to by customer Cust_A1 616-1 in Cloud A 610. Similarly, one or more users 618-2 can be associated with customer Cust_A2 616-2 and can use the services subscribed to by customer Cust_A2 616-2 in Cloud A 610. In various use cases, the services subscribed to by customer Cust_A1 616-1 can differ from the services subscribed to by customer Cust_A2 616-2.

[0169] Similarly, Cloud B 640 includes infrastructure CSPI_B 642 provided by CSP B. This infrastructure CSPI_B 642 can be used to provide a set of services 644 provided by Cloud B 640 (this set of services may, but does not need to, be different from the services provided by Cloud A 610). One or more customers (e.g., Cust_B1 646-1) can subscribe to one or more services from this set of services 644. One or more users 648-1 can be associated with customer Cust_B1 646-1 and can use the services subscribed to by customer Cust_B1 646-1 in Cloud B 640.

[0170] like Figure 6 As described, customer Cust_A1 616-1 is also a customer of CSP B and has subscribed to services available from CSPI_B 642. Therefore, customer Cust_A1 616-1 has leases in both Cloud B 640 and Cloud A 610.

[0171] In some embodiments, CSP A and CSP B may agree to provide each other's cross-cloud services. Figure 6 In the diagram, CSP A provides one or more of its services (referred to herein as cross-cloud services) to CSP B's customers via CSPI_B 642. These cross-cloud services include, for example, database services, storage services, compute services, etc. Therefore, customer Cust_A1616-1 (a customer of both Cloud A and Cloud B) can request, subscribe to, use, and / or manage one or more cross-cloud services of CSP A via its lease at CSPI_B 642. In contrast, customer Cust_B1 646-1 has no lease at CSPI_A 612. Therefore, cross-cloud services of CSP A may not be available to customer Cust_B1 646-1 unless customer Cust_B1 646-1 requests a lease provided by CSP A. This request can be submitted and managed via the portal of Cloud B 640, as further described in the following figures.

[0172] To enable cross-cloud service provisioning and availability for CSP A via Cloud B 640, Cloud A 610 can implement Inter-Cloud Service 614. Inter-Cloud Service 614 can be configured to, among other things, enable the use of cross-cloud services for CSP A via B. For example, Inter-Cloud Service 614 can communicate with Service 644 of Cloud B 640 and translate such communication into information suitable for use by In-Cloud Services 615 of Cloud A 610. More specifically, this set of Services 644 can enable the portal for CSP B and the deployment and management resources within Cloud B 640 for CSP B's customers. Through this portal, cross-cloud services for CSP A can be provided. Therefore, customers of CSP B (e.g., customer Cust_A1 616-1) can subscribe to and request cross-cloud services for CSP A via the portal. Such subscription and cloud operation requests can be received by Inter-Cloud Service 614 and then translated into information specific to Cloud A 610. Such information can then be passed to one or more of the In-Cloud Services 615 that are then provisioned. Information returned from cloud service 615 can be transformed by cloud service 614 and sent to cloud service 644 for use.

[0173] In some embodiments, the requested cross-cloud service can be provisioned by one or more services from cloud service 615 across resources in both cloud A 610 and cloud B 640. Doing so can support latency-sensitive operations (or, at least, reduce processing latency).

[0174] As illustrated, the infrastructure CSPI_A 612 of Cloud A 610 includes infrastructure 624 for the private cloud of customer Cust_A1616-1 in Cloud A 610 (e.g., a VCN for a portion of the customer's lease at Cloud A 610), infrastructure 626 for the private cloud of customer Cust_A2 616-2 in Cloud A 610, and other infrastructure 620. Each of these infrastructures includes hardware and / or software provided by CSP A and installed at locations (organized by region) under the control of CSP A.

[0175] In contrast, the infrastructure CSPI_B 642 of Cloud B 640 includes infrastructure 644 for the private cloud of customer Cust_A1 616-1 in Cloud A (e.g., a VCN for the customer as part of a lease at Cloud A 610), infrastructure 646 for the private cloud of customer Cust_A1 616-1 in Cloud B 640 (e.g., a VNET for the customer as part of Cloud B 640), infrastructure 648 for the private cloud of customer Cust_B1 646-1 in Cloud B 640, and other infrastructure 650. Infrastructure 644 includes hardware and / or software (e.g., co-located with components of CSPI_B 642) provided by CSP A and installed at locations (organized by region) under the control of CSP B. In contrast, infrastructures 646, 648, and 650 include hardware and / or software provided by CSP B and installed at locations (organized by region) under the control of CSP B.

[0176] Infrastructures 644 and 646 can be networked together, enabling customer Cust_A1 616-1 to access customer's private cloud regarding CSP A via customer's private cloud regarding CSP B, wherein the private cloud regarding CSP A is distributed between CSPI_A 612 and CSPI_B 642. This communication coupling of the private clouds can be initiated by inter-cloud service 614 and executed by one or more services in intra-cloud service 615. In this manner, customer A 616-1 has two leases: a first lease regarding CSP A including a first private cloud (e.g., VCN) distributed between CSPI_A 612 and CSPI_B 642, and a second lease regarding CSP B including a second private cloud (e.g., VNET) local to CSPI_B 642.

[0177] The requested cross-cloud service can actually be hosted on a first private cloud (e.g., at least partially on infrastructure 644 within CSPI_B642, and possibly on infrastructure 624 within CSPI_A 612) and is accessible via a second private cloud (e.g., for workflows hosted by infrastructure 646). Such a distribution of components is further described in the following figures, which enables the first private cloud of the first CSP to be hosted at least partially by the second cloud of the second CSP and linked to the second private cloud hosted by the second cloud.

[0178] Figure 7 An exemplary physical architecture for providing cross-cloud services based on infrastructure distributed across multiple CSPs, according to some embodiments, is described. Figure 7 In this illustration, the second CSP's customer will manage the lifecycle of the cross-cloud service developed by the first CSP. For illustrative purposes, the Exadata service (which may also be referred to herein as the Oracle DB service) and the Oracle and Microsoft clouds are described. In this illustration, Oracle corresponds to the first CSP and Microsoft corresponds to the second CSP, while the Exadata service corresponds to the cross-cloud service provided by the first CSP via the cloud of the second CSP. However, this embodiment is not so limited; rather, it is similarly applicable to other CSPIs, CSPs, and / or cross-cloud services.

[0179] A second CSP's customer (e.g., an Azure customer) can create and manage virtual resources (e.g., infrastructure and / or VM clusters) within the first CSP's cloud via the second CSP's portal (e.g., the Azure portal). These virtual resources can be provisioned to offer cross-cloud services. Support for such provisioning can be at least partially available through... Figure 6 This is achieved through inter-cloud services 614. Furthermore, the infrastructure supporting virtual resources can be distributed between the CSPIs of two CSPs (e.g., between an Azure data center and an OCI region). Specifically, a first portion of the infrastructure provided by the first CSP is installed at the CSPI of the second CSP, while the remaining second portion of the infrastructure provided by the first CSP is installed at the CSPI of the first CSP. The first portion may be referred to as a subsite included within or forming the cloud of the second CSP. The second portion may be referred to as a parent-child region including or forming the subsite, wherein the parent region is within the cloud of the first CSP.

[0180] exist Figure 7 On the left, a second CSPI_B 750 of the second CSP is shown. This CSPI_B 750 can represent, for example, the data center of the second CSP (e.g., an Azure data center). Within the CSPI_B 750, a base network 730 is available and provided by the second CSP. The base network 730 includes a set of computing resources, such as routers 732A, 732B, etc. (In the case of an Azure data center, these routers may include MeetMe ToRs supporting the MeetMe protocol for peer-to-peer connections). The CSPI_B 750 also includes a subsite 720 (which is a set of computing resources, such as server blades, racks, or other physical hardware of the first CSP that executes the CSP's software). Subsite 720 includes, among other things, routers 722A, 722B, etc. (in the case of OCI, these routers may include FastConnect routers that support the FastConnect protocol for connecting peers), connectivity infrastructure 724 (e.g., physical infrastructure that provides connectivity to other physical structures and components, such as JFAB), and physical resources 726 (e.g., racks, such as OCI server blades optimized for cross-cloud services, such as Exadata services).

[0181] exist Figure 7 On the right, a first CSPI_A 700 of the first CSP is shown. This CSPI_A 700 may represent, for example, the data center of the first CSP (e.g., an OCI data center). Within CSPI_A 700, a parent region 710 is illustrated. The parent region 710 may include a base network with multiple components. Among these components is a connectivity structure 712 (e.g., a physical structure such as a JFAB). Connectivity structure 712 is connected to connectivity structure 724, such that the child site 720 is communicatively coupled to the parent region 710. Optionally, the parent region 710 is located in an area physically adjacent to the child site 720 (or equivalently, CSPI_B 750), such that network latency for communication between the child site 720 and the parent region 710 is reduced.

[0182] Router 732 within base network 730 can be interconnected (e.g., via Ethernet cable) to router 722 within subsite 720. Connectivity structure 724 of subsite 720 provides interconnectivity between router 722 and physical resource 726. Furthermore, this connectivity structure 724 is connected to connectivity structure 714 of parent area 710 (e.g., using fiber optic cable, such as dark fiber), allowing data connectivity to exist between subsite 720 and parent area 710.

[0183] The base network 730 can host a set of resources for a second CSP for a customer (e.g., to provide a VNET that includes compute instances with access to the Exadata service, as further illustrated in the figures below). These second CSP resources can be part of a neighboring group across cloud services within a certain latency (e.g., 100µs).

[0184] Subsite 720 can host latency-critical resources of the first CSP, where these resources support cross-cloud services (e.g., subsite 720 can host OCI database resources and data plane resources to support the Exadata service). Parent region 720 can host other resources that support cross-cloud services (e.g., Figure 6 (e.g., for the Exadata service, parent region 720 can host ORP, OCI tools, OCI metrics and logging, the OCI control plane, regional OCI services, and the customer-facing console). Some of these resources (e.g., OCI database resources and data plane resources) can be deployed as part of a first private cloud for a customer of the first CSP (e.g., VCN in the case of OCI) and can be perceived by the customer as being available to them via a second private cloud for a customer of the second CSP (e.g., VNET in the case of Azure). This perception is possible by using the same IP address range for cross-cloud services in both private clouds.

[0185] In the example, a customer may have multiple private clouds (e.g., multiple VNETs and / or multiple VCNs) for each CSP. In the case of multiple private clouds for the first CSP, the underlying physical resources may not co-located in the same CSPI for the second CSP; instead, they may be included in different subsites or form different subsites. In this case, these resources may not be directly interconnected (e.g., there may be no direct physical connection between different subsites). Instead, indirect connections may exist via a parent region, where each of the different subsites is physically connected to the parent region (e.g., via fiber optic cable), and data flows between the two subsites through the parent region.

[0186] Furthermore, Subsite 720 can support multi-tenant architecture. Specifically, multiple customers can each have one or more private clouds (e.g., one or more VNETs and one or more VCNs) for each CSP. Each such customer can have separate access to the corresponding cross-cloud service via Subsite 720.

[0187] Figure 8 An exemplary virtual architecture for providing cross-cloud services based on infrastructure distributed across multiple CSPs, according to some embodiments, is described. Figure 8 In this context, a first private cloud 810 (e.g., a VCN hosted by OCI) for the first CSP's customers and a second cloud 820 (e.g., a VNET hosted by Azure) for the second CSP's customers can be provisioned to offer cross-cloud services of the first CSP (e.g., Exadata services provided by Oracle and available to Azure customers via Azure) through the second CSP's cloud.

[0188] In the example, the second private cloud 820 (e.g., VNET) is hosted by the physical compute resources of the second CSP's CSPI_B 850 (e.g., Azure data center). The second private cloud 820 may include compute instances (not shown) with access to cross-cloud services. The second private cloud 820 includes a subnet 822. Subnet 822 uses an IP address range. This range can be specified based on customer input via a portal of the second CSP (e.g., the Azure portal). IP addresses can be assigned from the IP address range to each compute instance launched to provide cross-cloud services to customers (e.g., VMs belonging to a VM cluster and providing database instances). These compute instances may be hosted by the first private cloud 810 (in... Figure 8 More generally, these are referred to as virtual resources 814A, 814B, 814C, and 814D. Figure 8 The diagram illustrates four computational examples, labeled with rectangular representations of NIC 824A, 824B, 824C, and 824D (or collectively referred to as NIC 824). While NIC stands for Network Interface Card, Figure 8 The NIC 824 itself is not like that. Instead, the NIC 824 represents mapping information that maps each IP address to a corresponding compute instance across cloud services. This mapping information can be stored as part of the configuration information of the second private cloud 820. If a compute instance of the second private cloud 820 uses an IP address to send or request traffic, and that IP address corresponds to a NIC, then the mapping information indicates that the traffic will be sent to or will be received from the first private cloud 810 (e.g., VCN), and is cross-cloud service traffic.

[0189] The first private cloud 810 (e.g., VCN) can be hosted by the physical computing resources of the first CSP, including, for example, subsites and parent regions (similar to...). Figure 7 The physical computing resources (those shown in the diagram) of the first private cloud 810 may include subnets 812 using the same IP address range. Each compute instance launched to provide cross-cloud services (e.g., each of virtual resources 814A, 814B, 814C, and 814D, collectively referred to as virtual resource 814) is hosted by a set of physical computing resources of the first CSP. The set of compute instances may include physical computing resources in subsites and physical computing resources in the parent region. The computing resources of the subsites may include, for example, those performing tasks such as... Figure 7 The hardware (e.g., racks) of the cross-cloud service software shown herein provides latency-critical virtual resources (e.g., database resources, data plane resources, etc.). The physical computing resources of the parent region may include, for example, servers, server clusters, network virtualization devices, etc., that provide non-latency-critical virtual resources (e.g., control plane resources, tooling resources, etc.).

[0190] Each compute instance can have an IP address from a range of IP addresses. A one-to-one mapping exists between compute instances in the first private cloud 810 (e.g., virtual resource 814) and the NIC 824 in the second private cloud 820. Therefore, cross-cloud service traffic can be sent from the second private cloud 820 to the first private cloud 810 and vice versa using this one-to-one mapping.

[0191] The second private cloud 820 can be connected (e.g., peer-to-peer) to the first private cloud 810 via one or more virtual routers (e.g., coupling the base network of the second CSP to the sub-sites), gateways (e.g., gateway 826 implemented at the sub-sites and gateway 816 implemented at the parent area, such as as part of a connectivity structure) and connectivity protocols (e.g., MeetMe and FastConnect protocols). Gateways 816 and 826 can collectively represent a dynamic routing gateway.

[0192] From the customer's perspective, the customer does not need to understand the underlying physical architecture and interconnects. Instead, the customer can have visibility at the virtual level, allowing them to perceive and manage their private clouds 810 and 820 (e.g., their network configuration), which is sufficient. Managing the second private cloud 820 can be done via a second portal of the second CSP. Managing the first private cloud 810 can be done via a second portal of the second CSP or a first portal of the first CSP. In this embodiment, changes and / or operations related to cross-cloud services are enabled only via the second CSP (e.g., via the second CSP, a customer can scale up or down, remove, terminate, add, etc., virtual resources in the first private cloud 810 launched for cross-cloud services). In contrast, changes and / or operations unrelated to cross-cloud services may be enabled only via the first CSP (e.g., via the first CSP, a customer can scale up or down, remove, terminate, add, etc., virtual resources in the first private cloud 810 launched for non-cross-cloud services).

[0193] Figure 9 Exemplary virtual resources provided to customers according to some embodiments are described. Figure 9 The right side shows the first resource (e.g., OCI resource) of the first CSP deployed for the customer. Figure 9 The left side shows a second resource (e.g., to an Azure data center) deployed for a second CSP for a customer. The first CSP (shown as CSP A) provides cross-cloud services (e.g., Exadata services) available via the second CSP (shown as CSPB).

[0194] The first resource (which may be a combination of physical and virtual resources) may include a private cloud 920, a service lease 910, a VM cluster 926, and a gateway 928. The private cloud 920 (e.g., a VCN) may correspond to a customer's lease to a first CSP and may include a primary subnet 922 with an IP range (e.g., a Classless Inter-Domain Routing (CIDR) range) and, optionally, a backup subnet 924 using the same IP range. The service lease 910 may be used by multiple customers of the first CSP and may include a gateway 912. The VM cluster 926 includes multiple VMs (or compute instances) launched for clients to provide cross-cloud services. The VM cluster 926 may be hosted on physical resources within a subsite. The gateway 928 (e.g., a service gateway) connects the virtual resources of the primary subnet 922 (and, similarly, the backup subnet 924) hosted on physical resources within a parent region to the VM cluster 926. In contrast, the gateway 912 connects such virtual resources to a customer's private cloud 960 for a second CSP. Gateway 912 can be implemented as a DRG attached to a private cloud 920 in a customer lease. Connectivity can be via gateway / router 914. Gateway / router 914 can provide, at least partially, connectivity within a subsite to the parent region via a first connectivity protocol (e.g., FastConnect) (thus providing gateway functionality) and connectivity to the subsite's base network of a second CSP via a second connectivity protocol (e.g., MeetMe) (thus providing routing functionality). For example, gateway / router 914 can represent a FastConnect virtual circuit resource in service lease 910 to connect the DRG to an Azure MeetMe router (example router 968).

[0195] The second resource (which may be a combination of physical and virtual resources) may include a private cloud 960 and a router 968. The private cloud 960 may include a main subnet 922 with an IP range, and optionally a backup subnet 964 using the same IP range. The IP range may be the same as one of the IP ranges in the main subnet 922. The router 968 may be connected to a gateway / router 914 to provide data connectivity from the main subnet 962 (e.g., and the backup subnet 964) to the private cloud 920. The main subnet 962 may include NICs 966A to 966K connected to the router 968.

[0196] The first CSP (e.g., its inter-cloud or intra-cloud service) can select IP addresses from an IP range for different virtual resources to provide cross-cloud services. The first CSP (e.g., its inter-cloud or intra-cloud service) can create Domain Name System (DNS) records corresponding to these IP addresses, which can then be provided to Private Cloud 960. Such DNS records can be used to establish private DNS zones for customers, which can be used in conjunction with Private Cloud 960.

[0197] For illustration, subnets 922, 924, 962, and 964 use IP addresses from 10.0.10.0 / 24. Each NIC 966 has an IP address from this range (e.g., NICA 966 A uses 10.0.10.10, and NIC K 966 K uses 10.0.10.11). VMs in the VM cluster use the same IP addresses (e.g., the first VM uses 10.0.10.10, while the second VM uses 10.0.10.11).

[0198] Figure 10 Exemplary architectures for provisioning and managing cross-cloud services based on infrastructure distributed across multiple CSPs, according to some embodiments, are described. Figure 10 In this example, the customer of the second CSP (shown as CSP B) will manage the lifecycle of a cross-cloud service developed by the first CSP (shown as CSP A). For clarity and for illustrative purposes, OCI and Azure are described as examples of CSPI, and the Exadata service is described as an example of a cross-cloud service. However, the embodiment is not so limited, and rather, similarly applies to other CSPs and / or cross-cloud services.

[0199] A customer (e.g., an Azure customer) can operate customer device 1000 to create and manage a set of virtual resources (e.g., Oracle DB infrastructure resources) across cloud services via a portal of a second CSP (shown as CSP_B portal 1052). In the example, portal 1052 interacts with one or more services of the second CSP (shown as CSP_B service 1054, examples of which may include Azure Resource Manager (ARM), which exposes APIs to manage the lifecycle of Oracle DB infrastructure resources (DB infrastructure resources include Exadata infrastructure and cloud VM clusters); other examples include Azure Resource Provider as a Service (RPaaS) to simplify the development and operation of Azure resource providers, such as asynchronous operations and their progress updates). To manage resources built on top of the virtual resource group (DB Home, database, pluggable database), CSP_B service 1054 interacts with the inter-cloud service 1012 of the first CSP to redirect customers to inter-cloud service 1012 (e.g., ORP in the case of OCI).

[0200] Inter-cloud service 1012 is configured to: i. handle the translation of an identifier assigned by a second CSP (e.g., an Azure identifier) ​​to an identifier assigned by a first CSP (e.g., an OCI identifier), including identity, resource ID, and subscription ID; ii. handle the translation of a second CSP state (e.g., an Azure state) to a first CSP state (e.g., an OCI state) and vice versa; and / or iii. delegate requests to an in-cloud service 1014 of the first CSP for execution (e.g., delegate to a resource control plane 1016, such as the OCIDBaS control plane). Inter-cloud service 1012 is also configured to coordinate any second CSP-specific integrations with other second CSP services (e.g., with Azure resource providers). Inter-cloud service 1012 may also be configured to perform or cause other in-cloud services 1014 to perform operations, including linking cloud accounts, publishing observability information, and issuing vending tokens to access other cloud customer environments.

[0201] Cloud-based service 1014 can provide a portal (shown as CSP_A portal 1018) accessible to the customer's client device 1000 to manage other services provided by the first CSP. Two portals, 1052 and 1018, can enable similar functionality (e.g., by presenting input and output fields), however, they are different. For example, portal 1052 (e.g., the Azure portal) can have a presentation format controlled by the second CSP. Additionally, in addition to functionality related to cross-cloud services, portal 1052 can also enable functionality specific to the second CSP and independent of the first CSP. In contrast, portal 1018 (e.g., the OCI portal) can have a presentation format controlled by the first CSP. Additionally, in addition to functionality related to cross-cloud services, portal 1018 can also enable functionality specific to the first CSP and independent of the second CSP. Regarding cross-cloud services, portal 1052 can expose information available from the second CSP, which can be provided by inter-cloud service 1012.

[0202] In the OCI and Azure use case examples, Cloud Service 1012 exposes the Oracle DB product. Cloud Service 1012 can be registered with ARM via RPaaS and configured to implement Azure Resource Provider Contracts (RPCs), which are a set of operations supported by all Azure resource providers. Azure RPaaS internal services are used to support a large number of RPC operations.

[0203] Typically, Cloud Service 1012 can be configured to have a second CSP identity (e.g., an Azure RP identity equivalent to an OCI service principal) and the ability to operate on a second CSP customer environment using a second CSP stream (e.g., an Azure OBO stream). Cloud Service 1012 can also be configured to persist second CSP-specific metadata for first CSP resources, such as Azure identifier-to-OCID mappings for DB resources. Cloud Service 1012 can also be configured to have a first CSP identity to gain scope for operating on a first CSP customer environment. Cloud Service 1012 can also be configured to act as a thin adapter layer, accepting requests in a second CSP format already authenticated by Service 1054, converting them to a first CSP request, and delegating the request to an in-cloud service.

[0204] Therefore, cloud service 1012 performs multiple operations. These operations include converting identifiers from one cloud to another and vice versa. Operations also include obtaining the first CSP identity from an incoming request from service 1054 for invoking cloud service 1014. These operations also include converting a second CSP format request into a first CSP format request and invoking cloud service 1014. These operations also include limiting / quota / capacity verification delivery or coordination, implicitly creating first CSP prerequisites for network connectivity resources (e.g., creating the subnet, OCI DRG, and VCN to which the VMCluster is attached), resulting in private cloud links, and configuring DNS entries for resources with associated DNS records regarding the second CSP.

[0205] In the example, inter-cloud service 1012 can be hosted in one or more sub-sites. Inter-cloud service 1012 allows virtual resource 1020 to be hosted in a sub-site. In the Exadata service use case, virtual resource 1020 can include the DBaaS data plane via the DBaaS control plane. This control plane is hosted in a parent region as part of in-cloud service 1014. In this use case, compute instances (e.g., VMs) can be instantiated for the customer on the customer's private cloud regarding the second CSP (e.g., VNET in Azure). The compute instances can perform database operations by making calls to the DBaaS data plane. Such operations include queries, storage, etc., or any operations supported by the Exadata service. Calls and responses to it can be made within the private cloud regarding the second CSP. When in use, the DBaaS data plane can report usage information (e.g., for metrics analysis, billing, etc.) to one or more services in in-cloud service 1014 (e.g., to the observability service). This usage information can then be provided to the monitoring service of the second CSP.

[0206] Figure 11 An exemplary user experience flow 1100 for provisioning resources according to some embodiments is depicted. In the example, user experience flow 1100 relates to client device 1120, CSPI_B 1150 of a second CSP, and inter-cloud services of a first CSP. The first CSP may provide cross-cloud services via the second CSP. Virtual resources may be provisioned to provide at least a portion of the cross-cloud services, wherein provisioning follows user experience flow 1100.

[0207] User input is provided by CSPI_B 1150 (e.g., via a similar method). Figure 10 The CSP_B portal 1052 receives instructions from client device 1120 to create a subnet and mark it as being delegated, making the subnet available for cross-cloud services. The subnet and the client's second private cloud with respect to the second CSP are created, and the instructions for its creation are provided to client device 1100.

[0208] Next, user input is received by inter-cloud service 1110 from computing device 1100 (where this input can be at the portal), indicating a request to create cross-cloud service infrastructure (e.g., an Exadata service). User input can indicate various parameters for provisioning, such as region and availability zone. Inter-cloud service 1110 can (e.g., via the portal) indicate the start of provisioning to computing device 1100 and can execute a provisioning workflow through which the cross-cloud service infrastructure is created in the relevant subsites. Simultaneously with the creation of the infrastructure, inter-cloud service 1110 sets its status to "Provisioning". After creation, inter-cloud service 1110 updates the status to "Successfully" and provides it. Indications to these statuses can (e.g., via the portal) be provided to computing device 1100. Similarly, computing device 1100 can check the current status via the portal, where a status query request can be made to inter-cloud service 1110 from CSPI_B 1150, and inter-cloud service 1110 will respond by returning status information.

[0209] Subsequently, user input is received by cloud service 1110 from CSPI_B 1150 (where this input is provided at the portal) and corresponds to a request from computing device 1100 to create VM cluster resources. The request may indicate a subnet. Cloud service 1110 then performs a set of checks and begins provisioning the VM cluster. The following figures further illustrate this provisioning. Cloud service 1110 sets the status to "Provisioning". After creation, cloud service 1110 updates the status to "Successful". Here, these status indications may also be provided to computing device 1100 (e.g., via the portal). Similarly, computing device 1100 may check the current status via the portal, where a status query request may be made from CSPI_B 1150 to cloud service 1110, and cloud service 1110 will respond by returning status information.

[0210] Figure 12 An exemplary control plane provisioning flow 1260 according to some embodiments is depicted. In the example, control plane provisioning flow 1260 relates to inter-cloud service 1200, control plane 1202 (e.g., an example of an in-cloud service), a customer's private cloud 1204 for a first CSP, gateway 1206 (e.g., DRG), connectivity module 1208 (e.g., a connectivity module provided by a connectivity infrastructure and supporting connectivity protocols such as FastConnect), cross-cloud service infrastructure 1210 for the first CSP, and CSPI_B 1250 for the second CSP. The first CSP can provide cross-cloud services via the second CSP. Here, control plane 1202 can execute a portion of control plane provisioning flow 1260 to provision VM cluster resources when triggered by inter-cloud service 1200. In response to user input on the portal of the second CSP, inter-cloud service 1200 can make such a call to control plane 1202 after receiving input from CSPI_B 1250 (from ARM in Azure).

[0211] As described above, the inter-cloud service 1220 receives a request to create a VM cluster (e.g., an Exadata VM cluster) for cross-cloud services and passes this request, along with other information (e.g., a first identifier assigned by a first CSP and mapped to a second identifier assigned by a second CSP), to the control plane 1202. The control plane 1202 then retrieves details about the subnet from CSPI_B 1250 (e.g., from ARM in Azure), including the CIDR, and verifies that prerequisites are met (e.g., the subnet has been provisioned, the region of the second private cloud (e.g., a VNET in Azure) for the second CSP, etc.). The control plane 1202 then (e.g., based on the first identifier) ​​creates a first private cloud (e.g., a VCN in OCI) for the first CSP in the customer lease and creates the subnet within the first private cloud. This subnet has the same CIDR as the subnet in the second private cloud.

[0212] Next, control plane 1202 creates and configures gateway 1206 (e.g., DRG) in the service lease and requests that gateway 1206 be attached to the first private cloud in the customer's lease. Control plane 1202 also configures routing information in gateway 1206 (e.g., by creating a DRG routing table for traffic between the two private clouds or MeetMe routers). Similarly, control plane 1202 also configures routing information in the first private cloud (e.g., by creating a VCN routing table for traffic to the DRG and VM clusters). After the routing information has been created, control plane 1202 can notify inter-cloud service 1200 of success. In response, inter-cloud service 1200 can configure a network security group for the first private cloud and request the creation of a VM cluster. The VM cluster is then provisioned in the relevant subsite, thereby assigning IP addresses from CIDR to the VM cluster. DNS records corresponding to the IP addresses are also generated. Inter-cloud service 1200 can receive such DNS records and request attachment from the first private network to the second private network (e.g., VCN to VNET).

[0213] Control plane 1202 then invokes the API of the second CSP, which allows IP addresses to be injected into the customer's subnet in the form of NICs. Each IP address can represent a VM cluster node IP. Control plane 1202 can register all the IP addresses that the VM cluster will have. The API call can include a device identifier (e.g., the GUID of the MeetMe router). This identifier can be based on a mapping of the GUID to a physical location (subsite). The call can also include a subnet resource identifier (e.g., the subnet of the private network in the customer's lease), an IP address (e.g., the IP address of the NIC being created, where the address is within the subnet range), and a resource name (the name of the NIC that will be visible to the user). The API can return a Virtual LAN identifier (VLAN ID) available during virtual circuit creation. Control plane 1202 then creates a virtual circuit that uses a connection to a router (e.g., to the MeetMe router) (e.g., a FastConnect connection) and provides DNS records so that a private DNS zone can be added to the customer's second private network. At this point, the VM cluster is successfully created. Then, compute instances across cloud services (e.g., Exadata database instances) can be hosted in a VM cluster.

[0214] resource providers

[0215] As discussed above, a first CSP can provide in-cloud services (e.g., database services, storage services, computing services, etc.) to its customers, and a second CSP can provide similar in-cloud services to its customers. The second CSP's customers can also be customers of the first CSP and may wish to access the in-cloud services provided by the first CSP via their lease within the cloud environment provided by the second CSP. Therefore, the first CSP can provide in-cloud services to the second CSP's customers as cross-cloud services. Similarly, the second CSP can provide in-cloud services, such as cross-cloud services, to the first CSP's customers. At least one of the cross-cloud services provided by one CSP to another CSP's customers can be the same service as the in-cloud service provided by that CSP to its own customers. In this way, a CSP's customers can be provided with a platform-level experience from within that CSP's cloud environment. Additionally, a CSP's customers can expose new features, versions, and resources to another CSP without leaving that CSP's cloud environment.

[0216] For example, Oracle, as a CSP, provides in-cloud services to its own customers via OCI, and Microsoft, as a CSP, can provide similar in-cloud services to its own users via Azure. Microsoft Azure customers may also be Oracle OCI customers and may wish to access Oracle's in-cloud services offered by OCI through their Microsoft Azure leases. Therefore, Oracle's OCI can offer in-cloud services such as Oracle's Exadata database service as a cross-cloud service to Microsoft Azure customers. At least one of the cross-cloud services offered by Oracle's OCI to Microsoft Azure customers can be the same service offered by Oracle's OCI to one of its own customers. For example, Oracle's OCI can offer Oracle's Exadata database service to both its own customers and Microsoft Azure customers through Microsoft's Azure environment. In this way, Microsoft Azure customers can be provided with a platform-level experience from Oracle's OCI within Microsoft Azure. Additionally, Microsoft Azure customers can expose new features, versions, and resources to Oracle's OCI without leaving Microsoft Azure. While Oracle's OCI and Microsoft's Azure have been used as examples, the techniques described are not limited to these CSPs and can be similarly applied to other CSPs such as Google Cloud™ and AWS®.

[0217] To facilitate the provision of cross-cloud services, subsites can be provided within the corresponding CSPIs of different CSPs, and cross-cloud services provided by other corresponding CSPs can be accessed using the subsite. For example, in the case of OCI and Azure mentioned above, a subsite can be provided within the Azure CSPI and can provide access to the Exadata database service provided by OCI from within the Azure cloud environment. Using subsites to provide access to cross-cloud services can provide high-bandwidth access to those cross-cloud services with reduced latency compared to cross-cloud services accessed through a first CSP and / or other remote cloud environments. However, resources such as compute resources at each subsite may be limited. Therefore, it may be desirable to provide one or more management mechanisms to provision and manage the lifecycle of cross-cloud services from one or more CSPIs and / or cloud environments. The techniques described herein relate to resource management mechanisms for provisioning and managing the lifecycle of cross-cloud services provided by one or more CSPs and between them. The resource management mechanisms described herein are dynamic because the characteristics of the CSP's CSPI, the subsite and / or cloud environment, and other factors can be considered when provisioning and managing cross-cloud services.

[0218] Figure 13 An example architecture 1300 is described, including a resource management mechanism for provisioning and managing cross-cloud services across multiple cloud environments. For example... Figure 13 As shown, architecture 1300 may include a first cloud environment 1302 of a first CSP (e.g., Oracle's OCI) and a second cloud environment 1318 of a second CSP (e.g., Microsoft's Azure). The first cloud environment 1302 and the second cloud environment 1318 can be configured according to... Figures 6-9 The distributed environment described is used for implementation. The first cloud environment 1302 and the second cloud environment 1318 may include one or more private clouds (e.g., VCN in the case of Oracle's OCI and VNET in the case of Microsoft's Azure). Additionally, it can be implemented according to... Figures 10-12 The described experience and supply flow are used to supply cross-cloud services between the first cloud environment 1302 and the second cloud environment 1318.

[0219] The first cloud environment 1302 can be configured to receive requests for cross-cloud services, evaluate the permission status of such requests, generate instructions for provisioning such services in one or more other cloud environments (such as the second cloud environment 1318), deploy such services in one or more other cloud environments, and manage the deployed services. In some implementations, cloud-based services 1310 of the first cloud environment 1302 can be configured to provide one or more services (e.g., Exadata cloud-based services) in cloud-based services 1310 as one or more cross-cloud services (e.g., Exadata cross-cloud services) to customers with leases in the second cloud environment 1318. Provisioning cloud-based services provided by the first cloud environment 1302 as cross-cloud services between the first cloud environment 1302 and the second cloud environment 1318 can be facilitated at least by the resource provider 1304 of the first cloud environment 1302. To provide cross-cloud services, resource provider 1304 can be configured to send a provisioning request for the requested cross-cloud services to the service control plane 1306 and network control plane 1308 of the first cloud environment 1302, and in response, the service control plane 1306 and network control plane 1308 can deploy the cloud-based service 1310 as a cross-cloud service to the second cloud environment 1318 (e.g., to the service data plane 1332 and network provider 1334 of the second cloud environment 1318).

[0220] In some implementations, the first cloud environment 1302 may include multiple parent regions, and the second cloud environment 1318 may include multiple sub-sites corresponding to the multiple parent regions. In some implementations, the service data plane 1332 of the second cloud environment 1318 may be used as and / or form one or more sub-sites of the corresponding parent regions. In some implementations, each corresponding parent region of the first cloud environment 1302 may include a resource provider (such as resource provider 1304) that, together with one or more sub-sites associated with the corresponding parent region, facilitates the provisioning and lifecycle management of one or more cross-cloud services. The one or more cross-cloud services may include one or more services in the cloud-based services 1310 of the first cloud environment 1302. For example, resource provider 1304 may be a resource provider in a parent region of the first cloud environment 1302 and may provision and manage the lifecycle of the cloud-based services 1310 as a cross-cloud service between the service control plane 1306 of the first cloud environment 1302 and the service data plane 1332 of the second cloud environment 1318. Sub-sites can at least be configured according to the above-mentioned... Figure 7 The physical architecture described is used to implement this.

[0221] In some implementations, the resource provider 1304 of the first cloud environment 1302 and the resource manager 1324 of the second cloud environment 1318 can facilitate the provisioning of cross-cloud services between the first cloud environment 1302 and the second cloud environment 1318. For example, the resource manager 1324 of the second cloud environment 1318 can send a cross-cloud service provisioning request to the resource provider 1304 of the first cloud environment 1302, and in response, the resource provider 1304 can work with the second cloud environment 1318 to facilitate the provisioning of cross-cloud services.

[0222] Similarly, the resource provider 1304 of the first cloud environment 1302 and the resource manager 1324 of the second cloud environment 1318 can facilitate the management of the lifecycle of the provisioned cross-cloud services between the first cloud environment 1302 and the second cloud environment 1318. For example, the resource manager 1324 of the second cloud environment 1318 can send a request to the resource provider 1304 for managing the lifecycle of the provisioned cross-cloud services (e.g., terminating a cross-cloud service request), and in response, the resource provider 1304 can work with the second cloud environment 1318 to facilitate lifecycle management functionality for the cross-cloud services. In some implementations, to facilitate provisioning and lifecycle management, the resource manager 1324 can be configured to communicate with the resource provider 1304 (e.g., using the API of the first cloud environment 1302 exposed by the resource provider 1304 to the resource manager 1324).

[0223] In some implementations, a customer of the second cloud environment 1318 and / or the second CSP, who wishes to provision cross-cloud services from within the second cloud environment 1318 and / or manage cross-cloud services already provided in conjunction with the second cloud environment 1318, may initiate request 1336 to do so. The customer may initiate request 1336 via portal 1320 of the second cloud network 1318. In some implementations, portal 1320 may include one or more graphical user interfaces (GUIs) accessible via client devices such as computers (e.g., via applications, operating systems, and / or software programs running on the client devices). One or more GUIs, or portions thereof, may be generated, populated, and / or otherwise provisioned by the resource manager 1324 of the second cloud environment 1318. Customers of the second cloud environment 1318 may access portal 1320 to manipulate one or more GUIs and / or interact with them to initiate request 1336 and perform other functions, such as managing their leases within the second cloud environment 1318.

[0224] In some implementations, one or more graphical user interfaces (GUIs) or portions thereof may be generated, populated, and / or otherwise supplied by the resource provider 1304 of the first cloud environment 1302. One or more GUIs or portions thereof may be provided and / or declared to the portal 1320 using the blade 1322 of the portal 1320. The blade 132 may serve as an extension, plugin, add-on, etc., of the portal 1320, or function as such. Clients of both the first cloud environment 1302 and the second cloud environment 1318 may access the portal 1320 to manipulate one or more GUIs and / or interact with them to initiate requests 1336 and perform other functions, such as managing their rentals within the first cloud environment 1302 and the second cloud environment 1318.

[0225] In some implementations, a request for provisioning cross-cloud services received via portal 1320 can be routed to resource manager 1324 of a second cloud environment 1318, which can then route request 1336 to resource provider 1304 of a first cloud environment 1302 (e.g., via a first set of APIs of the first cloud environment 1302 exposed within resource manager 1324). On the other hand, a request received via portal 1320 to manage the lifecycle of the provisioned cross-cloud services (e.g., viewing analytics, consumption, costs, logs, etc.) can be routed to console 1314 of the first cloud environment 1302, which can then route request 1336 within the first cloud environment 1302 (e.g., via a second API exposed within the first cloud environment 1302). For example, console 1314 can be configured to route request 1336 to analytics service 1316 of the first cloud environment 1302 for viewing analytics, consumption, costs, logs, etc., regarding the provisioned cross-cloud services. In some implementations, corresponding clients of the second cloud environment 1318 can be assigned corresponding identifiers, such that each request 1336 initiated by a corresponding client of the second cloud environment 1318 can be associated with that corresponding client's identifier. In this way, the resource manager 1324 can control and manage access to the portal and request initiation based on the roles and / or permissions associated with each client identifier.

[0226] To facilitate the provisioning of cross-cloud services and the lifecycle management of the provided cross-cloud services, resource provider 1304 can be linked to resource manager 1324 and resource provider-as-a-service (RPaaS) 1328 of the second cloud environment 1318. Linking resource provider 1304 to the second cloud environment 1318 enables resource provider 1304, resource manager 1324, and RPaaS 1328 to coordinate resources and operations. Resource provider 1304 can have an identity associated with the second cloud environment 1318. The identity can be configured to replicate the identity that resource provider 1304 has with respect to the first cloud environment 1302. To facilitate the linking, resource provider 1304 can provide configuration 1326 to resource manager 1324 and configuration 1330 to RPaaS 1328. Configuration 1326 and configuration 1330 can include an identifier for resource provider 1304 and can define the location, API specifications, and / or connection endpoints of the in-cloud services associated with resource provider 1304.

[0227] Resource provider 1304 can be configured to supply cross-cloud services and / or manage the lifecycle of supplied cross-cloud services based on operations performed by multi-cloud platform 1312 of first cloud environment 1302. Multi-cloud platform 1312 can be configured to perform operations common to linking and integrating first cloud environment 1302 into second cloud environment 1318 and other CSPs. For example, multi-cloud platform 1312 can be configured to link a customer's account for the second CSP to a customer's account for the first CSP, publish observation information collected from first cloud environment 1302 and / or second cloud environment 1318, generate issuance tokens for accessing other cloud environments of the customer, etc. Additionally, multi-cloud platform 1312 can be configured to create, define, supply, and / or otherwise implement contracts between resource provider 1304, resource manager 1324, and RPaaS 1328. The contract can identify the resources supported by resource provider 1304, resource manager 1324, and RPaaS 1328 and / or the operations to be performed by them. In some implementations, the contract can allow resource provider 1304, resource manager 1324, and RPaaS 1328 to operate on the same lease within the first cloud environment 1302. For example, the contract can define provisioning and / or lifecycle management events to be sent from resource manager 1324 and / or RPaaS 1328 when such events occur, wherein resource provider 1304 can be configured to asynchronously perform provisioning and / or lifecycle management operations based on receipt and / or state changes of such events.

[0228] Multi-cloud platform 1312 can be further configured to perform general cloud management operations, including but not limited to: (i) mapping subscriptions and leases of the second cloud environment 1318 to subscriptions and leases of the first cloud environment 1302; (ii) generating and managing policy statements governing leases in the respective cloud environments (e.g., statements facilitating operation of multi-cloud platform 1312 and resource provider 1304 in the same lease); (iii) generating and managing access tokens facilitating cross-cloud access and / or communication between the respective cloud environments; and / or (iv) mapping observability information of the first cloud environment 1302 to the second cloud environment 1318 (e.g., writing events to the second cloud environment 1318 when a backup is completed on the first cloud environment 1302, writing resource logs of the first cloud environment 1302 to the second cloud environment 1318, etc.). In this way, the first cloud environment 1302 and the second cloud environment 1318 can avoid overlapping operations and resources, which in turn can improve efficiency.

[0229] In some implementations, upon receiving a request 1336 to provision cross-cloud services, resource provider 1304 can be configured to map request 1336 to an identifier in first cloud environment 1302 and pass the request to service control plane 1306. Service control plane 1306 can be configured to perform two main processes: a first process is to provision the relevant resources in first cloud environment 1302; and a second process is to connect these resources to a customer's lease in second cloud environment 1318 (e.g., a customer's VNET in second cloud environment 1318). In the first process, service control plane 1306 creates a VCN for the customer in first cloud environment 1302 and creates one or more subnets within the VCN. In some implementations, the VCN can act as a shadow lease for the customer's lease in second cloud environment 1318. One or more subnets within the VCN and one or more subnets in the VNET can use the same CIDR. Service control plane 1306 also creates a DRG in first cloud environment 1302, attaches the DRG to the customer's VCN, and configures routing information for the DRG and VCN (e.g., to interconnect the two resources). Service control plane 1306 also provides the VM cluster in service data plane 1332. The VM cluster's IP addresses(s) come from the CIDR and are mapped to corresponding DNS records. In the second process, service control plane 1306 registers these IP addresses(s)(s) with the second cloud environment 1318, creates a virtual circuit between the DRG and network provider 1334, and sends DNS records, enabling the establishment of a private DNS zone in the second cloud environment 1318.

[0230] Additionally, or alternatively, upon receiving request 1336 for providing cross-cloud services, resource provider 1304 is configured to: (i) map the identifier assigned to the customer by the second CSP to the identifier assigned to the customer by the first CSP; (ii) obtain the identity of resource provider 1304 associated with request 1336 (i.e., the second cloud environment 1318 from which request 1336 originates); and (iii) convert the format of request 1336 from the format of the second cloud environment 1318 to the format of the first cloud environment 1302, and route the formatter request along with the mapped identifier to service control plane 1. 306 and network control plane 1308; (iv) establish network connectivity prerequisites for network connectivity between the first cloud environment 1302 and the second cloud environment 1318 (e.g., establish DRGs, VCNs, and subnets for VMs to connect to the service data plane 1332 of the network provider 1334 and the second cloud environment 1318); (v) establish network connectivity between the network control plane 1308 and the network provider 1334 based on the network connectivity prerequisites (e.g., link the delegated subnet of the second cloud environment 1318 to the VCN of the first cloud environment 1302 and configure DNS entries in the second cloud environment 1318). Additionally, after the cross-cloud service is provisioned, resource provider 1304 can be configured to persist metadata (e.g., a mapping between identifiers of the first cloud environment 1302 and the second cloud environment 1318) between the first cloud environment 1302 and the second cloud environment 1318, and act as a thin adapter layer that accepts requests formatted for the second cloud environment 1318 and already authenticated by resource manager 1324 (e.g., requests that manage the lifecycle of the provisioned cross-cloud service), translates them into requests formatted for the first cloud environment 1302, and delegates the translated requests to service control plane 1306.

[0231] control plane

[0232] As described above, in order to provide cross-cloud services, resource provider 1304 can be configured to send a provisioning request for the requested cross-cloud service to service control plane 1306 of first cloud environment 1302, and in response, service control plane 1306 can provide the cloud-based service 1310 of cloud environment 1310 as a cross-cloud service to second cloud environment 1318. Also as described above, service control plane 1306 can be configured to perform two main processes: a first process for providing relevant resources to first cloud environment 1302; and a second process for connecting these resources to leases for customers in second cloud environment 1318.

[0233] Figure 14 Examples of cloud services provided using resources from cloud service providers are described. For example... Figure 14 As shown, a first cloud environment 1402, which may be a first cloud environment 1302, includes customer leases 1478 and service leases 1476 for one or more customers of a first CSP providing services to the first cloud environment 1402. Customer leases 1478 may include multiple private clouds (e.g., multiple VCNs) already deployed for one or more customers. For example, as... Figure 14 As shown, customer lease 1478 may include customer VCN A 1472 for a first customer and customer VCN B 1474 for a second customer. Customer VCN A 1472 may include primary subnet A 1452 and backup subnet A 1454, and customer VCN B 1474 may include primary subnet A 1456 and backup subnet 1458.

[0234] In this scenario, a first cloud environment 1402 provides one or more cross-cloud services (e.g., Exadata services) to one or more customers of a second CSP that may be the same as or different from one or more customers of the first CSP. The cross-cloud service associated with customer VCN A 1472 may be supported by virtual resources such as service gateway 1460 and VM cluster A, and the cross-cloud service associated with customer VCN B 1474 may be supported by virtual resources such as service gateway 1462 and VM cluster B 1466. VM cluster A 1464 includes multiple VMs (or compute instances) launched for customers of customer VCN A 1472 to provide them with cross-cloud services. Similarly, VM cluster B 1466 includes multiple VMs launched for customers of customer VCN B 1474 to provide them with cross-cloud services. VM clusters A 1464 and B 1466 may be hosted on physical resources within a subsite (as described above). Service gateways 1460 and 1462 can connect virtual resources of primary subnets A1452 and B1456 (and similarly, backup subnets A1454 and B1458) hosted on physical resources within the parent region to VM clusters A1464 and B1466. Primary subnet A1452 can have an IP address range (e.g., a Classless Inter-Domain Routing (CIDR) range), and optionally, backup subnet A1254 can use the same IP address range. Similarly, primary subnet B1456 can have an IP address range, and optionally, backup subnet B1458 can use the same IP address range. In some implementations, the IP address range of customer VCN A1472 and customer VCN B1474 can be the same IP address range.

[0235] Service lease 1476 can be used for one or more customers of the first CSP and may include gateways such as DRG1 1444 and DRG2 1446. DRG1 1444 and DRG2 1446 connect the virtual resources of customer VCN A 1472 and customer VCN B 1474 to the private cloud or customer VNET 1470 of the customer used by the second CSP. Customer VNET 1470 may form customer lease 1480 in a second cloud environment 1404, which may be a second cloud environment 1318. DRG1 1444 and DRG2 1446 may be attached to customer VNET 1470 in customer lease 1480 in the second cloud environment 1404. The connection may be via router 1440. Router 1140 may at least partially provide gateway functionality by connecting the sub-site to the parent area via a first connectivity protocol (e.g., FastConnect) and connecting the sub-site to the base network of a second CSP via a second connectivity protocol (e.g., MeetMe). For example, router 1140 may form FastConnect virtual circuits 11448 and DRG1 1444 and FastConnect virtual circuits 21450 and DRG2 1446 in service lease 1476 to connect DRG1 1444 and DRG2 1446 to the Azure MeetMe router.

[0236] Customer VNET 1470 may include a primary subnet A 1420 with an IP address range, and optionally, a backup subnet A 1422 using the same IP address range. The IP address range may be the same as one of the IP address ranges in primary subnet A 1452. Customer VNET 1470 may also include a primary subnet B 1424 with an IP address range, and optionally, a backup subnet B 1426 using the same IP address range. The IP address range may be the same as one of the IP address ranges in primary subnet B 1456. Router 1440 provides data connectivity from primary subnets A 1452 and B 1456 to customer VNET 1470, and from backup subnets A 1454 and B 1458 to customer VNET 1470. Primary subnet A 1420 may include vNICs 1428, 1430, and 1432 connected to router 1440. Similarly, main subnet B 1424 may include vNICs 1434, 1436 and 1439 connected to router 1440.

[0237] The IP addresses used for providing cross-cloud services to different virtual resources can be selected from an IP range by a first CSP (e.g., by its inter-cloud or intra-cloud service). The first CSP (e.g., its inter-cloud or intra-cloud service) can create a Domain Name System (DNS) record corresponding to the IP address, which can be provided to the customer VNET 1470. Such DNS records can be used to establish a private DNS zone 1442 for the customers of a second CSP, where this private DNS zone can be used with the customer VNET 1470.

[0238] For illustration, primary and backup subnets A 1452, 1454, 1420, and 1422 use IP addresses from 10.0.10.0 / 24. Each vNIC 1428, 1430, and 1432 has an IP address from this range (e.g., vNIC 1428 uses 10.0.10.10, vNIC 1430 uses 10.0.10.11, and so on). VMs in VM cluster A 1464 use the same IP addresses (e.g., the first VM uses 10.0.10.10, while the second VM uses 10.0.10.11). Similarly, primary and backup subnets B 1456, 1458, 1424, and 1426 use IP addresses from 10.0.10.0 / 24. Each vNIC 1434, 1436, and 1436 has an IP address from within this range (e.g., vNIC 1434 uses 10.0.10.10, vNIC 1436 uses 10.0.10.11, and so on). The VMs in VM cluster B 1466 use the same IP addresses (e.g., the first VM uses 10.0.10.10, while the second VM uses 10.0.10.11).

[0239] As described above, the control plane manages the provisioning of these network resources. The control plane performs two main processes. The first process is to provision the relevant resources for the first CSP, such as customer VCN A 1472, customer VCN B 1474, VM ​​cluster A 1464, and VM cluster B 1466. The second process is to connect these resources to customer VNET 1470.

[0240] Figure 15 An example of a control plane provisioning flow 1560 is depicted. In some implementations, the control plane provisioning flow 1560 may correspond to the first process described above (e.g., the network link creation phase). In the example, the control plane provisioning flow 1560 involves a resource provider 1500, a control plane 1502, a gateway 1506 (e.g., a DRG), a connectivity module 1508 (e.g., a connectivity module provided by a connectivity structure and supporting connectivity protocols such as FastConnect), a service infrastructure 1510 for the first CSP, and a tenant infrastructure 1550 for the second CSP. In some implementations, the resource provider 1500 may be a resource provider 1304, the control plane may be a service control plane 1306, the service infrastructure 1510 may be the infrastructure of the first cloud environment 1302, and the tenant infrastructure 1550 may be the infrastructure of the second cloud environment 1318. The first CSP may provide cross-cloud services via the second CSP. Here, the control plane 1502 may execute a portion of the control plane provisioning flow 1560 when triggered by the resource provider 1500 to provision VM cluster resources. Resource provider 1500 may make such a call to control plane 1502 in response to user input to a portal (such as portal 1320) when it receives input from tenant infrastructure 1550 (e.g., from resource manager 1324 in second cloud environment 1318).

[0241] Resource provider 1500 receives a request to create a VM cluster (e.g., an Exadata VM cluster) for a cross-cloud service and passes this request, along with other information (e.g., a first identifier assigned by a first CSP and mapped to a second identifier assigned by a second CSP), to control plane 1502. Control plane 1502 then retrieves details about the subnet from tenant infrastructure 1550, including, for example, the CIDR, and verifies that prerequisites are met (e.g., the subnet has been provisioned, the region of the second private cloud for the second CSP (e.g., customer VNET 1470), etc.). Control plane 1502 then creates a first private cloud for the first CSP (e.g., customer VCN A 1472) in the customer lease (e.g., customer lease 1478) (e.g., based on the first identifier), and creates a subnet (e.g., primary subnet A 1452) within the first private cloud. This subnet has the same CIDR as the subnet in the second private cloud (e.g., primary subnet A 1420).

[0242] exist Figure 15 As described in the process, control plane 1502 creates the customer's VCN in the customer's lease and creates subnets (and possibly backup subnets) within the VCN. The corresponding service provider subnets (e.g., a subnet in the customer's Azure VNET and a subnet in the customer's OCI VCN) use the same CIDR. Control plane 1502 can create DRGs such as DRG1 1444 in the customer's lease and attach the DRGs to the VCN. Control plane 1502 can also establish routing information for the DRGs and VCNs, enabling these two resources to interconnect. Resource provider 1500 can also provision VM clusters, and the IP addresses of these clusters from the CIDR can be mapped to corresponding DNS records.

[0243] Figure 16 Another example of control plane provisioning flow 1660 is depicted. In some implementations, control plane provisioning flow 1660 may correspond to the second process described above (e.g., the network link attachment phase). In this example, control plane provisioning flow 1660, control plane 1502 configures gateway 1506 (e.g., DRG) in the service lease and requests that gateway 1506 be attached to the first private cloud in the customer's lease. Control plane 1502 also configures routing information in gateway 1506 (e.g., by creating a DRG routing table for traffic between the two private clouds or MeetMe routers). Similarly, control plane 1502 also configures routing information in the first private cloud (e.g., by creating a VCN routing table for traffic to the DRG and the VM cluster). After the routing information has been created, control plane 1502 may notify resource provider 1500 of success. In response, resource provider 1500 may configure a network security group for the first private cloud and request the creation of a VM cluster. The VM cluster is then provisioned in the relevant subsite, thereby assigning IP addresses from CIDR to the VM cluster. It also generates DNS records corresponding to IP addresses. Resource provider 1500 can receive such DNS records and request attachments from the first private network to the second private network (e.g., VCN to VNET).

[0244] Control plane 1502 then invokes the API of the second CSP, which allows IP addresses to be injected into the customer's subnet in the form of a NIC such as vNIC 1428. Each IP address can represent a VM cluster node IP. Control plane 1502 can register all the IP addresses that the VM cluster will have. The API call can include a device identifier (e.g., the GUID of the MeetMe router). This identifier can be based on a mapping of the GUID to a physical location (subsite). The call can also include a subnet resource identifier (e.g., the subnet of the private network in the customer's lease), an IP address (e.g., the IP address of the NIC being created, where the address is within the subnet range), and a resource name (the name of the NIC that will be visible to the user). The API can return a Virtual LAN identifier (VLAN ID) available during virtual circuit creation. Control plane 1502 then creates a virtual circuit, such as Virtual Circuit 1 1448, which uses a connection to a router (e.g., to the MeetMe router) of the tenant infrastructure 1550 (e.g., a FastConnect connection) and provides DNS records so that a private DNS zone can be added to the customer's second private network. At this point, the VM cluster has been successfully created. Then, compute instances used for cross-cloud services (e.g., Exadata database instances) can be hosted within the VM cluster.

[0245] For example, during the process of connecting relevant resources to a customer's VNET, control plane 1502 registers IP addresses with Azure (on the MeetMe router). In some examples, control plane 1502 may create a virtual circuit between the DRG and the MeetMe router and send DNS records, enabling a private DNS zone to be established in Azure.

[0246] Explanatory methods

[0247] Figure 17 The illustration shows an example process 1700 for provisioning and managing cross-cloud services across multiple cloud environments. Figure 17 The processing described herein can be implemented in the form of software (e.g., code, instructions, programs), hardware, or a combination thereof, executed by one or more processing units (e.g., one or more processors, cores) throughout the corresponding system described. The software can be stored on a non-transitory storage medium (e.g., a memory device). Although Figure 17 The methods presented describe various processing steps that occur in a specific order or sequence, but this is not intended to be restrictive. In some alternative implementations, these steps may be executed in parallel and / or in a different order. In some implementations, such as in... Figures 6-16 In the embodiments depicted, Figure 17 The processing described herein can be performed by physical components such as cloud environments, such as the first cloud environment 1302 and / or the second cloud environment 1318.

[0248] At block 1702, a first private cloud within a first cloud environment is integrated with a second private cloud within a second cloud environment. In some implementations, the first private cloud is integrated with the second private cloud by linking identifiers associated with components of the first private cloud to identifiers associated with components of the second private cloud. In some implementations, components of the first private cloud may be resource providers of the first private cloud, and components of the second private cloud may be resource managers and / or RPaaS of the second private cloud. In some implementations, linking the resource providers of the first private cloud and / or cloud environment to the second private cloud and / or cloud environment enables the resource providers, resource managers, and RPaaS to coordinate resources and operations. The resource provider may have an identity associated with the second cloud environment. The identity can be configured to replicate the identity that the resource provider has with respect to the first cloud environment. To facilitate the linking, the resource provider may provide configuration to the resource manager and to the RPaaS. One or more configurations in the configuration may include the resource provider's identifier and may define the API specifications, connection endpoints, and / or the location of services within the cloud associated with the resource provider.

[0249] At block 1704, a determination is made regarding whether a request for the cloud service has been received. In response to receiving the request, the process can proceed to block 1706. If the request has not yet been received, another determination can be made regarding whether a request for the cloud service has been received. In some implementations, the request may be received by a component of the first private cloud and / or cloud environment from a component of the second private cloud and / or cloud environment. In some implementations, the request is initiated in response to an interaction between a client device and a portal of the second private cloud. The client device may be associated with a client account corresponding to the second private cloud. In some implementations, the portal is configured to present a portion of the graphical user interface provided by the first component. The request may be initiated in response to an interaction between the client device and this portion. In some implementations, the cloud service may be a cross-cloud service provided by a first CSP providing the first cloud environment to a customer of a second CSP providing the second cloud environment and / or to an end user with a lease in the second cloud environment.

[0250] In some implementations, customers of the second cloud environment and / or the second CSP may wish to provision cross-cloud services from within the second cloud environment and / or manage cross-cloud services already provided in conjunction with the second cloud environment. These customers may do so by initiating requests through a portal of the second cloud environment. In some implementations, the portal may include one or more graphical user interfaces (GUIs) accessible via client devices such as computers (e.g., through applications, operating systems, and / or software programs running on the client devices). One or more GUIs, or portions thereof, may be generated, populated, and / or otherwise provisioned by the resource manager of the second cloud environment. Customers of the second cloud environment may access the portal to manipulate one or more GUIs and / or interact with them to initiate requests and perform other functions, such as managing their leases within the second cloud environment.

[0251] In some implementations, one or more graphical user interfaces (GUIs) or portions thereof may be generated, populated, and / or otherwise provisioned by the resource provider of the first cloud environment. One or more GUIs or portions thereof may be provided and / or declared to the portal using the portal's blades. Blades may function as extensions, plug-ins, attachments, etc., of the portal, or serve as extensions, plug-ins, attachments, etc., of the portal. Customers of both the first and second cloud environments can access the portal to manipulate one or more GUIs and / or interact with them to make requests and perform other functions, such as managing their leases within the first and second cloud environments.

[0252] In some implementations, requests for provisioning cross-cloud services received via the portal can be routed to a resource manager in a second cloud environment, which can then route the request to a resource provider in the first cloud environment (e.g., via a first set of APIs exposed within the resource manager for the first cloud environment 1302). Conversely, requests for managing the lifecycle of the provisioned cross-cloud services received via the portal (e.g., viewing analytics, consumption, costs, logs, etc.) can be routed to a console in the first cloud environment, which can then route the request within the first cloud environment (e.g., via a second API exposed within the first cloud environment). For example, the console can be configured to route requests to an analytics service in the first cloud environment for viewing analytics, consumption, costs, logs, etc., related to the provisioned cross-cloud services. In some implementations, corresponding clients in the second cloud environment can be assigned corresponding identifiers, such that each request initiated by a corresponding client in the second cloud environment can be associated with that client's identifier. In this way, the resource manager can control and manage access to the portal and request initiation based on the roles and / or permissions associated with each client identifier.

[0253] At block 1706, a set of operations associated with provisioning cloud services in a second private cloud and / or cloud environment is performed. In some implementations, this set of operations is performed by components of the first private cloud and / or cloud environment, and at least one of the operations includes establishing network connectivity prerequisites for network connectivity between the first private cloud and / or cloud environment and the second private cloud and / or cloud environment. In some implementations, establishing network connectivity prerequisites includes creating a first subnet within the first private cloud and creating a second subnet within the second private cloud. In some implementations, at least one Internet Protocol address associated with the first subnet and at least one Internet Protocol address associated with the second subnet are within the CIDR range.

[0254] In some implementations, upon receiving a request to provision cross-cloud services, the resource provider is configured to: (i) map an identifier assigned to a customer by a second CSP to an identifier assigned to a customer by a first CSP; (ii) obtain the identity of the resource provider associated with the request (i.e., the second cloud environment from which the request originates); (iii) convert the format of the request from the format of the second cloud environment to the format of the first cloud environment, and route the formatted request along with the mapped identifier to the service control plane and the network control plane; and (iv) establish network connectivity prerequisites for network connectivity between the first and second cloud environments (e.g., establish DRGs, VCNs, and subnets for VMs connecting to the service data planes of the network provider and the second cloud environment). In some implementations, upon receiving a request to provision cross-cloud services, the resource provider may be configured to map the request to an identifier of the first cloud environment and pass the request to the service control plane of the first cloud environment.

[0255] At block 1708, one or more second components of the first private cloud and / or cloud environment enable cloud services to be provisioned in the second private cloud and / or cloud environment. In some implementations, the first component of the first private cloud and / or cloud environment enables one or more second components of the first private cloud and / or cloud environment to provision cloud services in the second private cloud and / or cloud environment. In some implementations, enabling one or more second components of the first private cloud and / or cloud environment to provision cloud services in the second private cloud and / or cloud environment includes enabling one or more second components to establish network connectivity between the first private cloud and / or cloud environment and the second private cloud and / or cloud environment based on network connectivity prerequisites. In some implementations, enabling one or more second components of the first private cloud and / or cloud environment to provision cloud services in the second private cloud and / or cloud environment includes enabling virtual machines to be provisioned in another component of the second private cloud and / or cloud environment.

[0256] In some implementations, after performing this set of operations, the resource provider can be configured to establish network connectivity between the network control plane of the first cloud environment and the network provider of the second cloud environment based on network connectivity prerequisites (e.g., linking a delegated subnet of the second cloud environment to the VCN of the first cloud environment and configuring DNS entries in the second cloud environment). Additionally, after the cross-cloud service is provisioned, the resource provider can be configured to persist metadata (e.g., a mapping between identifiers of the first and second cloud environments) between the first and second cloud environments and act as a thin adapter layer that accepts requests formatted for the second cloud environment and already authenticated by the resource manager (e.g., requests managing the lifecycle of the provisioned cross-cloud service), translates them into requests formatted for the first cloud environment, and delegates the translated requests to the service control plane.

[0257] In some implementations, after performing this set of operations, the service control plane can be configured to perform a first process to provision relevant resources in the first cloud environment, and a second process to connect these resources to the customer's lease in the second cloud environment (e.g., the customer's VNET in the second cloud environment). In the first process, the service control plane creates a VCN for the customer in the first cloud environment and creates one or more subnets within the VCN. In some implementations, the VCN can act as a shadow lease of the customer's lease in the second cloud environment. One or more subnets in the VCN and one or more subnets in the VNET can use the same CIDR. The service control plane also creates a DRG in the first cloud environment 1302, attaches the DRG to the customer's VCN, and configures routing information for the DRG and VCN (e.g., to interconnect these two resources). The service control plane also provisiones a VM cluster in the service data plane. The IP addresses(s) of this VM cluster come from the CIDR and are mapped to corresponding DNS records. In the second process, the service control plane registers these IP addresses(s) with the second cloud environment, creates a virtual circuit between the DRG and the network provider, and sends DNS records to enable the establishment of a private DNS zone in the second cloud environment.

[0258] In some implementations, resource providers can be configured to supply cross-cloud services and / or manage the lifecycle of supplied cross-cloud services based on operations performed by a multi-cloud platform within the first cloud environment. The multi-cloud platform can be configured to perform operations common to linking and integrating the first cloud environment into a second cloud environment and other cloud environments of other CSPs. For example, the multi-cloud platform can be configured to link a customer's account for the second CSP to the customer's account for the first CSP, publish observations collected from the first and / or second cloud environments, generate issuance tokens for accessing other cloud environments of the customer, etc. Additionally, the multi-cloud platform can be configured to create, define, provision, and / or otherwise implement contracts between resource providers, resource managers, and RPaaS. Contracts can identify the resources supported by the resource provider, resource manager, and RPaaS and / or the operations to be performed by them. In some implementations, contracts can allow the resource provider, resource manager, and RPaaS to operate on the same lease within the first cloud environment. For example, a contract can define provisioning and / or lifecycle management events to be sent from the resource manager and / or RPaaS when such events occur, where the resource provider can be configured to asynchronously perform provisioning and / or lifecycle management operations based on the receipt and / or state changes of such events.

[0259] The multi-cloud platform can be further configured to perform common cloud management operations, including but not limited to: (i) mapping subscriptions and leases of a second cloud environment to subscriptions and leases of a first cloud environment; (ii) generating and managing policy statements governing leases in the respective cloud environments (e.g., statements facilitating operations between the multi-cloud platform and resource providers in the same lease); (iii) generating and managing access tokens facilitating cross-cloud access and / or communication between the respective cloud environments; and / or (iv) mapping observability information of the first cloud environment to the second cloud environment (e.g., writing events to the second cloud environment when a backup is completed on the first cloud environment, writing resource logs from the first cloud environment to the second cloud environment, etc.). In this way, the first and second cloud environments can avoid overlapping operations and resources, which in turn can improve efficiency.

[0260] Figure 18 An example process is described for using the control plane of a second cloud service provider to supply cloud services to a first cloud service provider. Figure 18 The processing described herein can be implemented as software (e.g., code, instructions, programs), hardware, or a combination thereof, executed by one or more processing units (e.g., one or more processors, cores) throughout the corresponding system described. The software can be stored on a non-transitory storage medium (e.g., a memory device). Although Figure 18 The methods presented describe various processing steps that occur in a specific order or sequence, but this is not intended to be restrictive. In some alternative implementations, these steps may be executed in parallel and / or in a different order. In some implementations, such as in... Figures 6-16 In the embodiments depicted, Figure 18 The processing described herein can be performed by physical components such as cloud environments, such as the first cloud environment 1302 and / or the second cloud environment 1318.

[0261] At block 1802, a first private cloud within a first cloud environment is integrated with a second private cloud within a second cloud environment. In some implementations, the first and second private clouds are integrated by linking identifiers associated with components of the first private cloud to identifiers associated with components of the second private cloud. In some implementations, components of the first private cloud may be resource providers of the first private cloud, and components of the second private cloud may be resource managers and / or RPaaS of the second private cloud. In some implementations, linking the resource providers of the first private cloud and / or cloud environment to the second private cloud and / or cloud environment enables the resource providers, resource managers, and RPaaS to coordinate resources and operations. The resource provider may have an identity associated with the second cloud environment. The identity can be configured to replicate the resource provider's identity with respect to the first cloud environment. To facilitate the link, the resource provider may provide configuration to the resource manager and to the RPaaS. One or more configurations in the configuration may include the resource provider's identifier and may define the location, API specification, and / or connection endpoint of the cloud services associated with the resource provider.

[0262] At block 1804, a determination is made regarding whether a request for the cloud service has been received. In response to receiving the request, the process can proceed to block 1806. If the request has not yet been received, another determination can be made regarding whether a request for the cloud service has been received. In some implementations, the request may be received by a component of the first private cloud and / or cloud environment from a component of the second private cloud and / or cloud environment. In some implementations, the request is initiated in response to an interaction between a client device and a portal of the second private cloud. The client device may be associated with a client account corresponding to the second private cloud. In some implementations, the portal is configured to present a portion of the graphical user interface provided by the first component. The request may be initiated in response to an interaction between the client device and this portion. In some implementations, the cloud service may be a cross-cloud service provided by a first CSP providing the first cloud environment to a customer of a second CSP providing the second cloud environment and / or to an end user with a lease in the second cloud environment.

[0263] In some implementations, customers of the second cloud environment and / or the second CSP may wish to provision cross-cloud services from within the second cloud environment and / or manage cross-cloud services already provided in conjunction with the second cloud environment. These customers may do so by initiating requests through a portal of the second cloud environment. In some implementations, the portal may include one or more graphical user interfaces (GUIs) accessible via client devices such as computers (e.g., through applications, operating systems, and / or software programs running on the client devices). One or more GUIs, or portions thereof, may be generated, populated, and / or otherwise provisioned by the resource manager of the second cloud environment. Customers of the second cloud environment may access the portal to manipulate one or more GUIs and / or interact with them to initiate requests and perform other functions, such as managing their leases within the second cloud environment.

[0264] In some implementations, one or more graphical user interfaces (GUIs) or portions thereof may be generated, populated, and / or otherwise provisioned by the resource provider of the first cloud environment. One or more GUIs or portions thereof may be provided and / or declared to the portal using the portal's blades. Blades may serve as extensions, plugins, attachments, etc., of the portal, or function as extensions, plugins, attachments, etc. Customers of both the first and second cloud environments can access the portal to manipulate one or more GUIs and / or interact with them to make requests and perform other functions, such as managing their leases within the first and second cloud environments.

[0265] In some implementations, requests for provisioning cross-cloud services received via the portal can be routed to a resource manager in a second cloud environment, which can then route the request to a resource provider in the first cloud environment (e.g., via a first set of APIs exposed within the resource manager for the first cloud environment 1302). Conversely, requests for managing the lifecycle of the provisioned cross-cloud services received via the portal (e.g., viewing analytics, consumption, costs, logs, etc.) can be routed to a console in the first cloud environment, which can then route the request within the first cloud environment (e.g., via a second API exposed within the first cloud environment). For example, the console can be configured to route requests to an analytics service in the first cloud environment for viewing analytics, consumption, costs, logs, etc., related to the provisioned cross-cloud services. In some implementations, corresponding clients in the second cloud environment can be assigned corresponding identifiers, such that each request initiated by a corresponding client in the second cloud environment can be associated with that client's identifier. In this way, the resource manager can control and manage access to the portal and request initiation based on the roles and / or permissions associated with each client identifier.

[0266] In block 1806, a first set of resources is provisioned within a first cloud environment. In some implementations, the first set of resources may be provisioned by components of the first cloud environment, such as control plane 1502. In some implementations, provisioning the first resources within the first cloud environment includes receiving information identifying a first subnet in a second cloud environment and a first Internet Protocol (IP) address range associated with the first subnet; provisioning a private cloud within the first cloud environment; and provisioning a second subnet in the private cloud, wherein a second IP address range associated with the second subnet is within a Classless Inter-Domain Routing (CIDR) range of the first IP address range.

[0267] Provisioning a first set of resources within a first cloud environment may include receiving information identifying a first subnet in a second cloud environment and a first Internet Protocol (IP) address range associated with the first subnet. A second component then provides a private cloud within the first cloud environment and provides a second subnet within the private cloud, wherein a second IP address range associated with the second subnet is within the Classless Inter-Domain Routing (CIDR) range of the first IP address range. Additionally, the second component may create resources in the first cloud environment that include a cluster of virtual machines, wherein each IP address in the second IP address range is associated with a virtual machine in the virtual machine cluster, and attach the resources to a router in the first cloud environment.

[0268] At this step, the control plane creates the customer's VCN within the customer's lease and creates subnets (and possibly backup subnets) within the VCN. The corresponding service provider subnets (e.g., a subnet in the customer's Azure VNET and a subnet in the customer's OCI VCN) use the same CIDR. The control plane can create DRGs in the customer's lease and attach the DRGs to the VCN. The control plane can also establish routing information for the DRGs and VCNs (so that the two resources are interconnected). The resource provider can also provision VM clusters. One or more IP addresses of this cluster from the CIDR can be mapped to corresponding DNS records.

[0269] At block 1808, the first set of resources is linked to the second set of resources. In some implementations, a control plane such as control plane 1502 of the first cloud environment is used to link the first set of resources to the second set of resources. In some implementations, linking the first set of resources to the second set of resources enables data about cloud services to be transferred from the second cloud environment to the first cloud environment. In some implementations, linking the first set of resources to the second set of resources enables data about cloud services to be transferred from a first subnet associated with a first private cloud of the second cloud environment to a second subnet associated with a second private cloud of the first cloud environment.

[0270] In some implementations, linking the first set of resources to the second set of resources includes invoking an application programming interface (API) of the second cloud environment, wherein invoking the API causes IP addresses within the second IP address range to appear in the first subnet; registering IP addresses within the second IP address range with a router in the second cloud environment using the API; receiving a virtual LAN identifier from the second cloud environment; and using the virtual LAN identifier to create a virtual circuit between the router in the first cloud environment and the router in the second cloud environment. In some embodiments, linking the first set of resources to the second set of resources further includes accessing Domain Name System (DNS) records associated with a virtual machine cluster in the first cloud environment; and providing DNS records to the second cloud environment, wherein providing DNS records to the second cloud environment causes a private DNS zone to be created in a private cloud associated with a customer account of the second cloud service provider.

[0271] Cloud infrastructure examples

[0272] As indicated above, IaaS is a specific type of cloud computing. IaaS can be configured to provide virtualized computing resources over a public network (e.g., the Internet). In the IaaS model, cloud providers can host infrastructure components (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer), etc.). In some cases, IaaS providers can also provision a wide variety of services to accompany those infrastructure components (example services include billing software, monitoring software, logging software, load balancing software, clustering software, etc.). Therefore, because these services can be policy-driven, IaaS users can implement policies to drive load balancing to maintain application availability and performance.

[0273] In some cases, IaaS customers can access resources and services over a wide area network (WAN) (such as the Internet) and can use the cloud provider's services to install the remaining elements of the application stack. For example, a user can log in to the IaaS platform to create virtual machines (VMs), install an operating system (OS) on each VM, deploy middleware such as databases, create buckets for workloads and backups, and even install enterprise software into the VM. The customer can then use the provider's services to perform various functions, including balancing network traffic, troubleshooting application problems, monitoring performance, and managing disaster recovery.

[0274] In most cases, cloud computing models will require the involvement of cloud providers. Cloud providers can, but do not need to, be third-party providers specializing in (e.g., offering, renting, or selling) IaaS services. Entities can also choose to deploy private clouds, becoming providers of their own infrastructure services.

[0275] In some examples, IaaS deployment is the process of placing a new application or a new version of an application onto a prepared application server or similar. It may also include the process of preparing the server (e.g., installing libraries, daemons, etc.). This is typically managed by the cloud provider under a hypervisor layer (e.g., servers, storage devices, network hardware, and virtualization). Therefore, the customer can be responsible for disposal (OS), middleware, and / or application deployment (e.g., on self-service virtual machines that can be spun up on demand).

[0276] In some examples, IaaS provisioning can refer to acquiring a computer or virtual host for use, and even installing the necessary libraries or services on it. In most cases, deployment does not include provisioning, and provisioning may need to be performed first.

[0277] In some cases, there are two distinct challenges to IaaS provisioning. First, there is the initial challenge of provisioning the initial set of infrastructure before anything can run. Second, there is the challenge of evolving the existing infrastructure after everything has been provisioned (e.g., adding new services, changing services, removing services, etc.). In some cases, these challenges can be addressed by enabling the configuration of the infrastructure to be declaratively defined. In other words, the infrastructure (e.g., what components are needed and how they interact) can be defined by one or more configuration files. Therefore, the overall topology of the infrastructure can be declaratively described (e.g., which resources depend on which resources and how each of them works together). In some cases, after the topology is defined, workflows for creating and / or managing the different components described in the configuration files can be generated.

[0278] In some examples, the infrastructure can have many interconnected elements. For example, there may be one or more Virtual Private Clouds (VPCs) (e.g., potential on-demand pools of configurable and / or shared computing resources), also known as the core network. In some examples, there may also be one or more inbound / outbound traffic group rules, provisioned to define how inbound and / or outbound traffic will be structured for the network, and one or more virtual machines (VMs). Other infrastructure elements, such as load balancers, databases, etc., may also be provisioned. As more infrastructure elements are expected and / or added, the infrastructure can evolve incrementally.

[0279] In some cases, continuous deployment techniques can be used to enable the deployment of infrastructure code across various virtual computing environments. Additionally, the described techniques can enable infrastructure management within these environments. In some examples, service teams may write code that they expect to deploy to one or more (but often more) different production environments (e.g., across different geographical locations, sometimes across the world). However, in some examples, the infrastructure on which the code will be deployed must first be established. In some instances, provisioning can be done manually, provisioning tools can be used to provision resources, and / or, once the infrastructure is provisioned, deployment tools can be used to deploy the code.

[0280] Figure 19 This is a block diagram 1900 illustrating an example pattern of an IaaS architecture according to at least one embodiment. A service operator 1902 may be communicatively coupled to a secure host lease 1904, which may include a virtual cloud network (VCN) 1906 and a secure host subnet 1908. In some examples, the service operator 1902 may use one or more client computing devices, which may be portable handheld devices (e.g., iPhone®, cellular phone, iPad®, computing tablet, personal digital assistant (PDA)) or wearable devices (e.g., Google Glass® head-mounted display), running software (such as Microsoft Windows Mobile®) and / or a wide variety of mobile operating systems (such as iOS, Windows Phone, Android, BlackBerry 8, Palm OS, etc.), and enabled for the Internet, email, SMS, Blackberry®, or other communication protocols. Alternatively, the client computing device may be a general-purpose personal computer, for example including personal computers and / or laptops running various versions of Microsoft Windows®, Apple Macintosh®, and / or Linux operating systems. The client computing device can be a workstation computer running any of the various commercial UNIX® or UNIX-like operating systems (including, but not limited to, various GNU / Linux operating systems, such as, for example, Google Chrome OS). Alternatively or additionally, the client computing device can be any other electronic device capable of communicating over a network that can access VCN 1906 and / or the Internet, such as a thin client computer, an Internet-enabled gaming system (e.g., a Microsoft Xbox game console with or without Kinect® gesture input), and / or a personal messaging device.

[0281] VCN 1906 may include a local peering gateway (LPG) 1910, which may be communicatively coupled to a secure shell (SSH) VCN 1912 via LPG 1910 included in SSH VCN 1912. SSH VCN 1912 may include an SSH subnet 1914, and SSH VCN 1912 may be communicatively coupled to a control plane VCN 1916 via LPG 1910 included in control plane VCN 1916. Furthermore, SSH VCN 1912 may be communicatively coupled to a data plane VCN 1918 via LPG 1910. Control plane VCN 1916 and data plane VCN 1918 may be contained within a service lease 1919 that may be owned and / or operated by an IaaS provider.

[0282] The control plane VCN 1916 may include a control plane demilitarized zone (DMZ) layer 1920, which acts as a perimeter network (e.g., a portion of a corporate network between an intranet and an external network). DMZ-based servers can have limited liability and help keep violations contained. Additionally, the DMZ layer 1920 may include one or more load balancer (LB) subnets 1922, a control plane application layer 1924 that may include one or more application subnets 1926, and a control plane data layer 1928 that may include one or more database (DB) subnets 1930 (e.g., one or more front-end DB subnets and / or one or more back-end DB subnets). One or more LB subnets 1922 contained in the control plane DMZ layer 1920 may be communicatively coupled to one or more application subnets 1926 contained in the control plane application layer 1924 and an Internet gateway 1934 that may be contained in the control plane VCN 1916. The application subnets 1926 may be communicatively coupled to one or more DB subnets 1930 and service gateways 1936 and Network Address Translation (NAT) gateways 1938 contained in the control plane data layer 1928. The control plane VCN 1916 may include service gateway 1936 and NAT gateway 1938.

[0283] Control plane VCN 1916 may include a data plane mirroring application layer 1940, which may include one or more application subnets 1926. The one or more application subnets 1926 included in the data plane mirroring application layer 1940 may include a Virtual Network Interface Controller (VNIC) 1942, which can execute a compute instance 1944. The compute instance 1944 may communicatively couple the one or more application subnets 1926 of the data plane mirroring application layer 1940 to the one or more application subnets 1926 that may be included in the data plane application layer 1946.

[0284] Data plane VCN 1918 may include a data plane application layer 1946, a data plane DMZ layer 1948, and a data plane data layer 1950. Data plane DMZ layer 1948 may include one or more application subnets 1926 communicatively coupled to data plane application layer 1946 and one or more LB subnets 1922 of internet gateway 1934 of data plane VCN 1918. One or more application subnets 1926 may be communicatively coupled to service gateway 1936 and NAT gateway 1938 of data plane VCN 1918. Data plane data layer 1950 may also include one or more DB subnets 1930 communicatively coupled to one or more application subnets 1926 of data plane application layer 1946.

[0285] Internet gateway 1934 of control plane VCN 1916 and data plane VCN 1918 can communicatively couple to metadata management service 1952, which can communicatively couple to public internet 1954. Public internet 1954 can communicatively couple to NAT gateway 1938 of control plane VCN 1916 and data plane VCN 1918. Service gateway 1936 of control plane VCN 1916 and data plane VCN 1918 can communicatively couple to cloud service 1956.

[0286] In some examples, the service gateway 1936 of the control plane VCN 1916 or the data plane VCN 1918 can make application programming interface (API) calls to the cloud service 1956 without traversing the public internet 1954. API calls from the service gateway 1936 to the cloud service 1956 can be unidirectional: the service gateway 1936 can make API calls to the cloud service 1956, and the cloud service 1956 can send the requested data to the service gateway 1936. However, the cloud service 1956 may not initiate API calls to the service gateway 1936.

[0287] In some examples, secure host lease 1904 can be directly connected to service lease 1919, which would otherwise be isolated. Secure host subnet 1908 can communicate with SSH subnet 1914 via LPG 1910, which enables bidirectional communication over otherwise isolated systems. Connecting secure host subnet 1908 to SSH subnet 1914 grants secure host subnet 1908 access to other entities within service lease 1919.

[0288] Control plane VCN 1916 allows users of service lease 1919 to establish or otherwise provision desired resources. The desired resources provisioned in control plane VCN 1916 can be deployed or otherwise used in data plane VCN 1918. In some examples, control plane VCN 1916 can be isolated from data plane VCN 1918, and the data plane mirror application layer 1940 of control plane VCN 1916 can communicate with the data plane application layer 1946 of data plane VCN 1918 via a VNIC 1942 that can be included in both the data plane mirror application layer 1940 and the data plane application layer 1946.

[0289] In some examples, a user or client of the system may make a request (e.g., a Create, Read, Update, or Delete (CRUD) operation) via the public internet 1954, which can then forward the request to the metadata management service 1952. The metadata management service 1952 can then forward the request to the control plane VCN 1916 via internet gateway 1934. The request may be received by one or more LB subnets 1922 contained in the control plane DMZ layer 1920. The LB subnets 1922 can determine that the request is valid, and in response to this determination, they can forward the request to one or more application subnets 1926 contained in the control plane application layer 1924. If the request is authenticated and requires a call to the public internet 1954, the call to the public internet 1954 can be forwarded to a NAT gateway 1938 that can make the call to the public internet 1954. The request may expect that the metadata to be stored can be stored in one or more DB subnets 1930.

[0290] In some examples, the data plane mirroring application layer 1940 can facilitate direct communication between the control plane VCN 1916 and the data plane VCN 1918. For example, it may be desirable to apply configuration changes, updates, or other suitable modifications to resources contained in the data plane VCN 1918. Through VNIC 1942, the control plane VCN 1916 can communicate directly with the resources contained in the data plane VCN 1918, and thereby perform configuration changes, updates, or other suitable modifications to the resources contained in the data plane VCN 1918.

[0291] In some embodiments, the control plane VCN 1916 and data plane VCN 1918 may be included in service lease 1919. In this case, the system's users or customers may not own or operate the control plane VCN 1916 or data plane VCN 1918. Conversely, the IaaS provider may own or operate both the control plane VCN 1916 and data plane VCN 1918, both of which may be included in service lease 1919. This embodiment can enable network isolation that can prevent users or customers from interacting with resources of other users or other customers. Furthermore, this embodiment can allow the system's users or customers to privately store databases without relying on the public Internet 1954, which may not have the desired level of threat prevention.

[0292] In other embodiments, one or more LB subnets 1922 included in the control plane VCN 1916 may be configured to receive signals from the service gateway 1936. In this embodiment, the control plane VCN 1916 and the data plane VCN 1918 may be configured to be invoked by the IaaS provider's customers without invoking the public internet 1954. The IaaS provider's customers may expect this embodiment because the database(s) used by the customer can be controlled by the IaaS provider and can be stored on a service lease 1919 that can be isolated from the public internet 1954.

[0293] Figure 20 This is a block diagram 2000 illustrating another example pattern of an IaaS architecture according to at least one embodiment. Service operator 2002 (e.g., Figure 19 Service providers (1902) can communicatively couple to secure host rental (2004) (e.g., Figure 19 Secure hosting lease 1904), the secure hosting lease may include Virtual Cloud Network (VCN) 2006 (e.g., Figure 19 VCN1906) and Secure Host Subnet 2008 (e.g., Figure 19 The secure host subnet 1908). VCN 2006 can include Local Peer Gateway (LPG) 2010 (e.g., Figure 19 The LPG 1910 can be communicatively coupled to the Secure Shell (SSH) VCN 2012 (e.g., LPG 1910) contained in the SSH VCN 2012. Figure 19 (SSH) VCN 1912). SSH VCN 2012 can include SSH subnets 2014 (e.g., Figure 19 SSH subnet 1914), and SSH VCN 2012 can be communicatively coupled to control plane VCN 2016 via LPG 2010 included in control plane VCN 2016 (e.g., Figure 19 Control plane VCN 2016). Control plane VCN 2016 can be included in service lease 2019 (e.g., Figure 19 In the service lease 1919), and the data plane VCN 2018 (e.g., Figure 19 The data plane (VCN 1918) can be included in a customer lease 2021 that can be owned or operated by the system's users or customers.

[0294] The control plane VCN 2016 may include one or more LB subnets 2022 (e.g., Figure 19 The control plane DMZ layer of (one or more) LB subnets 1922) 2020 (e.g., Figure 19 The control plane DMZ layer 1920), may include one or more application subnets 2026 (e.g., Figure 19 The control plane application layer of (one or more) application subnets 1926) 2024 (e.g., Figure 19 The control plane application layer 1924) may include one or more database (DB) subnets 2030 (e.g., similar to...). Figure 19 The control plane data layer of (one or more) DB subnets 1930) 2028 (e.g., Figure 19 The control plane data layer 1928). One or more LB subnets 2022 contained in the control plane DMZ layer 2020 can be communicatively coupled to one or more application subnets 2026 contained in the control plane application layer 2024 and an Internet gateway 2034 that can be contained in the control plane VCN 2016 (e.g., Figure 19 Internet gateway 1934), and application subnet(s) 2026 can communicatively couple to DB subnet(s) 2030 and service gateway 2036 contained in control plane data layer 2028 (e.g., Internet gateway 1934), and application subnet(s) 2026 can communicatively couple to DB subnet(s) 2030 and service gateway(s) ... Figure 19 Service Gateway 1936) and Network Address Translation (NAT) Gateway 2038 (e.g., Figure 19 (NAT gateway 1938). The control plane VCN 2016 may include service gateway 2036 and NAT gateway 2038.

[0295] The control plane VCN 2016 may include a data plane mirror of the application layer 2040 (e.g., Figure 19 The data plane mirror application layer 1940 may include one or more application subnets 2026. The one or more application subnets 2026 included in the data plane mirror application layer 2040 may include computational instances 2044 (e.g., similar to...). Figure 19 The virtual network interface controller (VNIC) 2042 (e.g., the VNIC of 1942) of the computing instance 1944. The computing instance 2044 may facilitate the mirroring of the application layer 2040 (one or more) of the application subnets 2026 and may be included in the data plane application layer 2046 (e.g., Figure 19 Communication between one or more application subnets 2026 in the data plane application layer 1946 via VNIC 2042 contained in the data plane mirror application layer 2040 and VNIC 2042 contained in the data plane application layer 2046.

[0296] The Internet gateway 2034 included in the control plane VCN 2016 can be communicatively coupled to the metadata management service 2052 (e.g., Figure 19 Metadata management service (1952), which can communicatively couple to the public Internet (2054, e.g., Figure 19 The public internet 2054 can communicatively couple to a NAT gateway 2038 contained in the control plane VCN 2016. A service gateway 2036 contained in the control plane VCN 2016 can communicatively couple to a cloud service 2056 (e.g., ...). Figure 19 Cloud services (1956).

[0297] In some examples, the data plane VCN 2018 may be included in customer lease 2021. In this case, the IaaS provider may provide a control plane VCN 2016 for each customer, and the IaaS provider may establish a unique compute instance 2044 for each customer, included in service lease 2019. Each compute instance 2044 may allow communication between the control plane VCN 2016 included in service lease 2019 and the data plane VCN 2018 included in customer lease 2021. The compute instance 2044 may allow resources provisioned in the control plane VCN 2016 included in service lease 2019 to be deployed or otherwise used in the data plane VCN 2018 included in customer lease 2021.

[0298] In other examples, an IaaS provider's customer may have a database residing in customer lease 2021. In this example, control plane VCN 2016 may include a data plane mirroring application layer 2040, which may include one or more application subnets 2026. Data plane mirroring application layer 2040 may reside in data plane VCN 2018, but it may not reside in data plane VCN 2018. That is, data plane mirroring application layer 2040 may have access to customer lease 2021, but it may not reside in data plane VCN 2018 or be owned or operated by an IaaS provider's customer. Data plane mirroring application layer 2040 may be configured to make calls to data plane VCN 2018, but it may not be configured to make calls to any entities contained in control plane VCN 2016. Customers may expect to deploy or otherwise use resources provisioned in the control plane VCN 2016 in the data plane VCN 2018, and the data plane mirror application layer 2040 can facilitate the customer's desired deployment or other use of resources.

[0299] In some embodiments, IaaS provider customers can apply filters to the data plane VCN 2018. In this embodiment, customers can determine what the data plane VCN 2018 can access, and customers can restrict access from the data plane VCN 2018 to the public internet 2054. The IaaS provider may not be able to apply filters or otherwise control the data plane VCN 2018's access to any external networks or databases. Applying filters and controls by the customer to the data plane VCN 2018 included in the customer's lease 2021 can help isolate the data plane VCN 2018 from other customers and from the public internet 2054.

[0300] In some embodiments, cloud service 2056 can be invoked by service gateway 2036 to access services that may not exist on public internet 2054, control plane VCN 2016, or data plane VCN 2018. The connection between cloud service 2056 and control plane VCN 2016 or data plane VCN 2018 may not be active or continuous. Cloud service 2056 may reside on different networks owned or operated by an IaaS provider. Cloud service 2056 may be configured to receive calls from service gateway 2036 and may be configured not to receive calls from public internet 2054. Some cloud services 2056 may be isolated from other cloud services 2056, and control plane VCN 2016 may be isolated from cloud services 2056, meaning the cloud services may not be in the same region as control plane VCN 2016. For example, control plane VCN 2016 may be located in "Region 1," and cloud service "Deployment 19" may be located in both Region 1 and "Region 2." If a call to deployment 19 is made by a service gateway 2036 contained in a control plane VCN 2016 located in region 1, the call can be forwarded to deployment 19 in region 1. In this example, the control plane VCN 2016 or deployment 19 in region 1 may be non-communicatively coupled to deployment 19 in region 2, or may otherwise communicate with deployment 19 in region 2.

[0301] Figure 21 This is a block diagram 2100 illustrating another example pattern of an IaaS architecture according to at least one embodiment. Service operator 2102 (e.g., Figure 19 Service providers (1902) can communicatively couple to secure host leases (2104) (e.g., Figure 19 Secure hosting lease 1904), the secure hosting lease may include a Virtual Cloud Network (VCN) 2106 (e.g., Figure 19 VCN1906) and secure host subnet 2108 (e.g., Figure 19 The secure host subnet 1908). VCN 2106 can include LPG 2110 (e.g., Figure 19 The LPG 1910 can be communicatively coupled to the SSH VCN 2112 via the LPG 2110 contained in the SSH VCN 2112 (e.g., Figure 19 SSH VCN 2112 can include SSH subnet 2114 (e.g., SSH VCN 2112). Figure 19 SSH subnet 1914), and SSH VCN 2112 can be communicatively coupled to control plane VCN 2116 via LPG 2110 contained in control plane VCN 2116 (e.g., Figure 19 The control plane VCN 1916) and the LPG 2110 communicatively coupled to the data plane VCN 2118 (e.g., via the control plane VCN 1916) and the data plane VCN 2118 via the LPG 2110 contained in the data plane VCN 2118. Figure 19 Data plane 2116. Control plane VCN 2116 and data plane VCN 2118 may be contained in service lease 2119 (e.g., Figure 19 In the service rental of 1919.

[0302] The control plane VCN 2116 may include one or more load balancer (LB) subnets 2122 (e.g., Figure 19 The control plane DMZ layer 2120 of (one or more) LB subnets 1922) (e.g., Figure 19 The control plane DMZ layer 1920 may include one or more application subnets 2126 (e.g., similar to...). Figure 19 The control plane application layer 2124 of (one or more) application subnets 1926 (e.g., Figure 19 The control plane application layer 1924 may include (one or more) control plane data layers 2128 of the DB subnet 2130 (e.g., Figure 19 The control plane data layer 1928). One or more LB subnets 2122 contained in the control plane DMZ layer 2120 can be communicatively coupled to one or more application subnets 2126 contained in the control plane application layer 2124 and an Internet gateway 2134 that can be contained in the control plane VCN 2116 (e.g., Figure 19 Internet gateway 1934), and application subnet(s) 2126 can communicatively couple to DB subnet(s) 2130 contained in control plane data layer 2128 and service gateway 2136 (e.g., Figure 19 The service gateway) and the Network Address Translation (NAT) gateway 2138 (e.g., Figure 19 (NAT gateway 1938). The control plane VCN 2116 may include the service gateway 2136 and the NAT gateway 2138.

[0303] Data plane VCN 2118 may include data plane application layer 2146 (e.g., Figure 19 Data plane application layer 1946), data plane DMZ layer 2148 (e.g., Figure 19 Data plane DMZ layer 1948), and data plane data layer 2150 (e.g., Figure 19 The data plane data layer 1950). The data plane DMZ layer 2148 may include one or more LB subnets 2122, which may be communicatively coupled to one or more trusted application subnets 2160 and one or more untrusted application subnets 2162 of the data plane application layer 2146, and an Internet gateway 2134 contained in the data plane VCN 2118. One or more trusted application subnets 2160 may be communicatively coupled to the service gateway 2136 contained in the data plane VCN 2118, the NAT gateway 2138 contained in the data plane VCN 2118, and one or more DB subnets 2130 contained in the data plane data layer 2150. One or more untrusted application subnets 2162 may be communicatively coupled to the service gateway 2136 contained in the data plane VCN 2118 and the one or more DB subnets 2130 contained in the data plane data layer 2150. The data plane data layer 2150 may include one or more DB subnets 2130, which may be communicatively coupled to a service gateway 2136 contained in the data plane VCN 2118.

[0304] One or more untrusted application subnets 2162 may include one or more primary VNICs 2164(1)-(N) that can be communicatively coupled to tenant virtual machines (VMs) 2166(1)-(N). Each tenant VM 2166(1)-(N) may be communicatively coupled to a corresponding application subnet 2167(1)-(N) that may be contained in a corresponding container egress VCN 2168(1)-(N), which may be contained in a corresponding customer lease 2170(1)-(N). The corresponding secondary VNIC 2172(1)-(N) may facilitate communication between one or more untrusted application subnets 2162 contained in a data plane VCN 2118 and application subnets contained in container egress VCN 2168(1)-(N). Each container egress VCN 2168(1)-(N) may include a NAT gateway 2138 that can be communicatively coupled to the public Internet 2154 (e.g., Figure 19 The public internet (1954).

[0305] Internet gateway 2134, contained in control plane VCN 2116 and data plane VCN 2118, can be communicatively coupled to metadata management service 2152 (e.g., Figure 19 The metadata management system 1952 can communicatively couple to the public Internet 2154. The public Internet 2154 can communicatively couple to a NAT gateway 2138 contained in a control plane VCN 2116 and a data plane VCN 2118. The service gateway 2136 contained in the control plane VCN 2116 and the data plane VCN 2118 can communicatively couple to a cloud service 2156.

[0306] In some embodiments, the data plane VCN 2118 may be integrated with the customer lease 2170. This integration may be useful or desired by the IaaS provider's customer in certain situations, such as when support may be expected when executing code. The customer may provide code to run, which may be destructive, may communicate with other customer resources, or may otherwise cause adverse effects. In response, the IaaS provider may determine whether to run the code provided by the customer.

[0307] In some examples, an IaaS provider's customer may grant temporary network access to the IaaS provider and request functionality to be attached to the data plane application layer 2146. The code running this functionality may execute within VMs 2166(1)-(N) and may not be configured to run anywhere else on the data plane VCN 2118. Each VM 2166(1)-(N) may be connected to a customer lease 2170. A corresponding container 2171(1)-(N) contained within VMs 2166(1)-(N) may be configured to run the code. In this case, dual isolation may exist (e.g., container 2171(1)-(N) runs the code, where container 2171(1)-(N) may be contained at least within VM 2166(1)-(N), which is contained within one or more untrusted application subnets 2162), which can help prevent incorrect or otherwise unintended code from corrupting the IaaS provider's network or the networks of different customers. Containers 2171(1)-(N) may be communicatively coupled to customer lease 2170 and may be configured to send or receive data from customer lease 2170. Containers 2171(1)-(N) may not be configured to send or receive data from any other entity in data plane VCN 2118. After the running code is complete, the IaaS provider may terminate or otherwise dispose of containers 2171(1)-(N).

[0308] In some embodiments, one or more trusted application subnets 2160 may run code owned or operated by the IaaS provider. In this embodiment, one or more trusted application subnets 2160 may be communicatively coupled to one or more DB subnets 2130 and configured to perform CRUD operations in one or more DB subnets 2130. One or more untrusted application subnets 2162 may be communicatively coupled to one or more DB subnets 2130, but in this embodiment, one or more untrusted application subnets may be configured to perform read operations in one or more DB subnets 2130. Containers 2171 (1)-(N) that may be contained in each customer's VM 2166 (1)-(N) and may run code from the customer may not be communicatively coupled to one or more DB subnets 2130.

[0309] In other embodiments, the control plane VCN 2116 and the data plane VCN 2118 may be coupled without direct communication. In this embodiment, direct communication between the control plane VCN 2116 and the data plane VCN 2118 may not exist. However, communication can occur indirectly through at least one method. An LPG 2110 may be established by an IaaS provider, which can facilitate communication between the control plane VCN 2116 and the data plane VCN 2118. In another example, either the control plane VCN 2116 or the data plane VCN 2118 may make a call to the cloud service 2156 via the service gateway 2136. For example, a call from the control plane VCN 2116 to the cloud service 2156 may include a request for a service that can communicate with the data plane VCN 2118.

[0310] Figure 22 This is a block diagram 2200 illustrating another example pattern of an IaaS architecture according to at least one embodiment. Service operator 2202 (e.g., Figure 19 The service provider 1902) can communicatively couple to the secure host lease 2204 (e.g., Figure 19 Secure hosting lease 1904), the secure hosting lease may include a Virtual Cloud Network (VCN) 2206 (e.g., Figure 19 VCN1906) and Security Host Subnet 2208 (e.g., Figure 19 The secure host subnet 1908). VCN 2206 can include LPG 2210 (e.g., Figure 19 The LPG 1910 can be communicatively coupled to the SSH VCN 2212 via the LPG 2210 included in the SSH VCN 2212 (e.g., Figure 19 SSH VCN 1912). SSH VCN 2212 can include SSH subnet 2214 (e.g., Figure 19 SSH subnet 1914), and SSH VCN 2212 can be communicatively coupled to control plane VCN 2216 via LPG 2210 included in control plane VCN 2216 (e.g., Figure 19 The control plane VCN 1916) and the LPG 2210 communicatively coupled to the data plane VCN 2218 (e.g., via the control plane VCN 1916) and the data plane VCN 2218 via the LPG 2210 contained in the data plane VCN 2218. Figure 19 Data plane 2218). Control plane VCN 2216 and data plane VCN 2218 may be contained in service lease 2219 (e.g., Figure 19 In the service rental of 1919.

[0311] The control plane VCN 2216 may include one or more LB subnets 2222 (e.g., Figure 19 The control plane DMZ layer 2220 of (one or more) LB subnets 1922) (e.g., Figure 19 The control plane DMZ layer 1920 may include one or more application subnets 2226 (e.g., Figure 19 The control plane application layer 2224 of (one or more) application subnets 1926 (e.g., Figure 19 The control plane application layer 1924 may include one or more DB subnets 2230 (e.g., Figure 21 The control plane data layer 2228 of (one or more) DB subnets 2130 (e.g., Figure 19 The control plane data layer 1928). One or more LB subnets 2222 contained in the control plane DMZ layer 2220 can be communicatively coupled to one or more application subnets 2226 contained in the control plane application layer 2224 and an Internet gateway 2234 that can be contained in the control plane VCN 2216 (e.g., Figure 19 Internet gateway 1934), and application subnet(s) 2226 can communicatively couple to DB subnet(s) 2230 contained in control plane data layer 2228 and to service gateway 2236 (e.g., Figure 19 The service gateway) and the Network Address Translation (NAT) gateway 2238 (e.g., Figure 19 (NAT gateway 1938). The control plane VCN 2216 may include the service gateway 2236 and the NAT gateway 2238.

[0312] Data plane VCN 2218 may include data plane application layer 2246 (e.g., Figure 19 Data plane application layer 1946), data plane DMZ layer 2248 (e.g., Figure 19 Data plane DMZ layer 1948), and data plane data layer 2250 (e.g., Figure 19 The data plane data layer 1950). The data plane DMZ layer 2248 may include one or more LB subnets 2222, which may be communicatively coupled to one or more trusted application subnets 2260 of the data plane application layer 2246 (e.g., Figure 21 (one or more) trusted application subnets 2160 and (one or more) untrusted application subnets 2262 (e.g., Figure 21 The data plane VCN 2218 may include one or more untrusted application subnets 2262 and an Internet gateway 2234. One or more trusted application subnets 2260 may be communicatively coupled to a service gateway 2236, a NAT gateway 2238, and a DB subnet 2230 within the data plane VCN 2218. One or more untrusted application subnets 2262 may be communicatively coupled to a service gateway 2236 and a DB subnet 2230 within the data plane VCN 2218. The data plane VCN 2250 may include one or more DB subnets 2230 that may be communicatively coupled to a service gateway 2236 within the data plane VCN 2218.

[0313] One or more untrusted application subnets 2262 may include a primary VNIC 2264(1)-(N) that may be communicatively coupled to tenant virtual machines (VMs) 2266(1)-(N) residing within one or more untrusted application subnets 2262. Each tenant VM 2266(1)-(N) may run code in a corresponding container 2267(1)-(N) and may be communicatively coupled to an application subnet 2226 that may be contained in a data plane application layer 2246, which may be contained in a container egress VCN 2268. A corresponding secondary VNIC 2272(1)-(N) may facilitate communication between one or more untrusted application subnets 2262 contained in a data plane VCN 2218 and the application subnets contained in a container egress VCN 2268. The container egress VCN may include a NAT gateway 2238 that may be communicatively coupled to a public internet 2254 (e.g., Figure 19 The public internet (1954).

[0314] Internet gateway 2234, contained in control plane VCN 2216 and data plane VCN 2218, can be communicatively coupled to metadata management service 2252 (e.g., Figure 19 The metadata management system 1952 can communicatively couple to the public Internet 2254. The public Internet 2254 can communicatively couple to a NAT gateway 2238 contained in a control plane VCN 2216 and a data plane VCN 2218. The service gateway 2236 contained in the control plane VCN 2216 and the data plane VCN 2218 can communicatively couple to a cloud service 2256.

[0315] In some examples, by Figure 22 The architecture diagram of block diagram 2200 can be viewed as being composed of... Figure 21 The architecture illustrated in block diagram 2100 is an exception to the pattern, and may be what the IaaS provider's customers expect if the IaaS provider cannot communicate directly with the customer (e.g., in a disconnected region). The corresponding container 2267(1)-(N) contained in each customer's VM 2266(1)-(N) can be accessed by the customer in real time. Container 2267(1)-(N) can be configured to make calls to the corresponding secondary VNIC 2272(1)-(N) contained in one or more application subnets 2226 of the data plane application layer 2246, which may be contained in the container egress VCN 2268. The secondary VNIC 2272(1)-(N) can forward the calls to a NAT gateway 2238, which can forward the calls to the public internet 2254. In this example, containers 2267(1)-(N), which can be accessed by clients in real time, can be isolated from the control plane VCN 2216 and from other entities contained in the data plane VCN 2218. Containers 2267(1)-(N) can also be isolated from resources from other clients.

[0316] In other examples, a client may use container 2267(1)-(N) to invoke cloud service 2256. In this example, the client may run code requesting services from cloud service 2256 within container 2267(1)-(N). Container 2267(1)-(N) may forward the request to a secondary VNIC 2272(1)-(N), which may forward the request to a NAT gateway, which may forward the request to the public internet 2254. The public internet 2254 may forward the request to one or more LB subnets 2222 contained in control plane VCN 2216 via internet gateway 2234. In response to determining that the request is valid, one or more LB subnets may forward the request to one or more application subnets 2226, which may forward the request to cloud service 2256 via service gateway 2236.

[0317] It should be understood that the IaaS architectures 1900, 2000, 2100, and 2200 depicted in the accompanying drawings may have other components besides those depicted. Furthermore, the embodiments shown in the drawings are merely some examples of cloud infrastructure systems that can be incorporated into embodiments of this disclosure. In some other embodiments, the IaaS system may have more or fewer components than shown in the drawings, may combine two or more components, or may have different configurations or arrangements of components.

[0318] In some embodiments, the IaaS system described herein may include a suite of application, middleware, and database service offerings delivered to customers in a self-service, subscription-based, elastically scalable, reliable, highly available, and secure manner. An example of such an IaaS system is the Oracle Cloud Infrastructure (OCI) provided by this assignee.

[0319] Figure 23 An example computer system 2300, in which various embodiments can be implemented, is illustrated. System 2300 can be used to implement any computer system described above. As shown in the figure, computer system 2300 includes a processing unit 2304 that communicates with a plurality of peripheral subsystems via a bus subsystem 2302. These peripheral subsystems may include a processing acceleration unit 2306, an I / O subsystem 2308, a storage subsystem 2318, and a communication subsystem 2324. Storage subsystem 2318 includes a tangible computer-readable storage medium 2322 and system memory 2310.

[0320] Bus subsystem 2302 provides a mechanism for enabling the various components and subsystems of computer system 2300 to communicate with each other as intended. Although bus subsystem 2302 is schematically shown as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses. Bus subsystem 2302 can be any of several types of bus architectures, including memory buses or memory controllers, peripheral buses, and local buses using any of a variety of bus architectures. For example, such architectures may include Industry Standard Architecture (ISA) buses, Micro Channel Architecture (MCA) buses, Enhanced ISA (EISA) buses, Video Electronics Standards Association (VESA) local buses, and Peripheral Component Interconnect (PCI) buses that may be implemented as mezzanine buses manufactured according to the IEEE P1386.1 standard.

[0321] A processing unit 2304, which may be implemented as one or more integrated circuits (e.g., a conventional microprocessor or microcontroller), controls the operation of the computer system 2300. One or more processors may be included in the processing unit 2304. These processors may include single-core or multi-core processors. In some embodiments, the processing unit 2304 may be implemented as one or more independent processing units 2332 and / or 2334, each including a single-core or multi-core processor. In other embodiments, the processing unit 2304 may also be implemented as a quad-core processing unit formed by integrating two dual-core processors into a single chip.

[0322] In various embodiments, processing unit 2304 can execute various programs in response to program code and can maintain multiple concurrently executing programs or processes. At any given time, some or all of the program code to be executed may reside in processor(s) 2304 and / or storage subsystem 2318. With appropriate programming, processor(s) 2304 can provide the various functions described above. Additionally, computer system 2300 may include processing acceleration unit 2306, which may include a digital signal processor (DSP), a dedicated processor, and / or the like.

[0323] I / O subsystem 2308 may include user interface input devices and user interface output devices. User interface input devices may include keyboards, pointing devices such as mice or trackballs, touchpads or touchscreens integrated into the display, scroll wheels, click wheels, dial pads, buttons, switches, keypads, audio input devices with voice command recognition systems, microphones, and other types of input devices. User interface input devices may include, for example, motion sensing and / or gesture recognition devices, such as the Microsoft Kinect® motion sensor, which enables users to control and interact with input devices (such as the Microsoft Xbox® 360 game controller) using a natural user interface that employs gestures and verbal commands. User interface input devices may also include eye gesture recognition devices, such as the Google Glass® blink detector, which detects eye activity from the user (e.g., "blinking" when taking a photo and / or making menu selections) and translates the eye gesture into input on the input device (e.g., Google Glass®). Additionally, user interface input devices may include voice recognition sensing devices that enable users to interact with a voice recognition system (e.g., the Siri® navigator) via voice commands.

[0324] User interface input devices may also include, but are not limited to, 3D mice, joysticks or pointers, game controllers, and graphics tablets, as well as audio / video devices such as speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, barcode readers, 3D scanners, 3D printers, laser rangefinders, and eye-tracking devices. Additionally, user interface input devices may include, for example, medical imaging input devices such as computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), and medical ultrasound equipment. User interface input devices may also include, for example, audio input devices such as MIDI keyboards and digital musical instruments.

[0325] User interface output devices may include display subsystems, indicator lights, or non-visual displays such as audio output devices. Display subsystems may be cathode ray tubes (CRTs), flat panel devices such as those using liquid crystal displays (LCDs) or plasma displays, projection devices, touchscreens, etc. Generally, the term "output device" is used to encompass all possible types of devices and mechanisms for outputting information from computer system 2300 to the user or other computers. For example, user interface output devices may include, but are not limited to, various display devices that visually convey text, graphics, and audio / video information, such as monitors, printers, speakers, headphones, car navigation systems, plotters, voice output devices, and modems.

[0326] Computer system 2300 may include storage subsystem 2318, which may provide a tangible, non-transitory computer-readable storage medium for storing software and data constructs that provide the functionality of the embodiments described in this disclosure. The software may include programs, code modules, instructions, scripts, etc., which, when executed by one or more cores or processors of processing unit 2304, provide the aforementioned functionality. Storage subsystem 2318 may also provide a repository for storing data used according to this disclosure.

[0327] like Figure 23 As depicted in the example, storage subsystem 2318 may include various components, including system memory 2310, computer-readable storage medium 2322, and computer-readable storage medium reader 2320. System memory 2310 may store program instructions that can be loaded and executed by processing unit 2304. System memory 2310 may also store data used during the execution of instructions and / or data generated during the execution of program instructions. Various types of programs may be loaded into system memory 2310, including but not limited to client applications, web browsers, middleware applications, relational database management systems (RDBMS), virtual machines, containers, etc.

[0328] System memory 2310 may also store operating system 2316. Examples of operating system 2316 may include various versions of Microsoft Windows®, Apple Macintosh® and / or Linux operating systems, various commercial UNIX® or UNIX-like operating systems (including but not limited to various GNU / Linux operating systems, Google Chrome® OS, etc.) and / or mobile operating systems such as iOS, Windows® Phone, Android® OS, BlackBerry® OS, and Palm® OS. In some implementations of computer system 2300 that execute one or more virtual machines, the virtual machine, along with its guest operating system (GOS), may be loaded into system memory 2310 and executed by one or more processors or cores of processing unit 2304.

[0329] Depending on the type of computer system 2300, the system memory 2310 can have different configurations. For example, the system memory 2310 can be volatile memory (such as random access memory (RAM)) and / or non-volatile memory (such as read-only memory (ROM), flash memory, etc.). Different types of RAM configurations can be provided, including static random access memory (SRAM), dynamic random access memory (DRAM), etc. In some implementations, the system memory 2310 may include a basic input / output system (BIOS), which contains basic routines that facilitate the transfer of information between elements within the computer system 2300, such as during startup.

[0330] Computer-readable storage medium 2322 may represent remote, local, fixed and / or removable storage devices and storage media for temporarily and / or more permanently containing and storing computer-readable information used by computer system 2300, including instructions executable by processing unit 2304 of computer system 2300.

[0331] Computer-readable storage medium 2322 may include any suitable medium known or used in the art, including storage media and communication media, such as, but not limited to, volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing and / or transmitting information. This may include tangible computer-readable storage media or other tangible computer-readable media such as RAM, ROM, electronically erasable programmable ROM (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile disk (DVD) or other optical storage devices, magnetic tape cassettes, magnetic tape, disk storage devices or other magnetic storage devices.

[0332] For example, computer-readable storage medium 2322 may include hard disk drives that read from or write to non-removable non-volatile magnetic media, disk drives that read from or write to removable non-volatile disks, and optical disc drives that read from or write to removable non-volatile optical discs (such as CD ROMs, DVDs, and Blu-ray® discs) or other optical media. Computer-readable storage medium 2322 may include, but is not limited to, Zip® drives, flash memory cards, Universal Serial Bus (USB) flash memory drives, Secure Digital (SD) cards, DVD discs, digital video tapes, etc. Computer-readable storage medium 2322 may also include solid-state drives (SSDs) based on non-volatile memory (such as flash-based SSDs, enterprise flash drives, solid-state ROMs, etc.), volatile memory-based SSDs (such as solid-state RAM, dynamic RAM, static RAM, DRAM-based SSDs), magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs using a combination of DRAM and flash-based SSDs. Disk drives and their associated computer-readable media can provide non-volatile storage for computer-readable instructions, data structures, program modules and other data for computer system 2300.

[0333] Machine-readable instructions executable by one or more processors or cores of the processing unit 2304 may be stored on a non-transitory computer-readable storage medium. A non-transitory computer-readable storage medium may include physically tangible memory or storage devices that include volatile memory storage devices and / or non-volatile memory storage devices. Examples of non-transitory computer-readable storage media include magnetic storage media (e.g., magnetic disks or magnetic tapes), optical storage media (e.g., DVDs, CDs), various types of RAM, ROM, or flash memory, hard disk drives, floppy disk drives, removable memory drives (e.g., USB drives), or other types of storage devices.

[0334] The communication subsystem 2324 provides an interface to other computer systems and networks. The communication subsystem 2324 serves as an interface for receiving data from other systems and sending data to other systems from computer system 2300. For example, the communication subsystem 2324 may enable computer system 2300 to connect to one or more devices via the Internet. In some embodiments, the communication subsystem 2324 may include radio frequency (RF) transceiver components, global positioning system (GPS) receiver components, and / or other components for accessing wireless voice and / or data networks (e.g., using cellular phone technology, advanced data network technologies such as 3G, 4G, or EDGE (Enhanced Data Rate Global Evolution), WiFi (IEEE 802.11 series standards, or other mobile communication technologies, or any combination thereof)). In some embodiments, in addition to or instead of a wireless interface, the communication subsystem 2324 may provide wired network connectivity (e.g., Ethernet).

[0335] In some embodiments, the communication subsystem 2324 may also receive input communications on behalf of one or more users who may use the computer system 2300 in the form of structured and / or unstructured data feeds 2326, event streams 2328, event updates 2330, etc.

[0336] For example, the communication subsystem 2324 can be configured to receive data feeds 2326 in real time from users of social networks and / or other communication services, such as Twitter® feeds, Facebook® updates, web feeds such as Rich Site Summary (RSS) feeds, and / or real-time updates from one or more third-party information sources.

[0337] Additionally, the communication subsystem 2324 can also be configured to receive data in the form of a continuous data stream, which may include event streams 2328 and / or event updates 2330 of real-time events. This data may be continuous or unbounded in nature, without a definite end. Examples of applications that generate continuous data may include, for example, sensor data applications, financial stock analysis, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, vehicle traffic monitoring, etc.

[0338] The communication subsystem 2324 can also be configured to output structured and / or unstructured data feeds 2326, event streams 2328, event updates 2330, etc., to one or more databases that can communicate with one or more streaming data source computers coupled to the computer system 2300.

[0339] The computer system 2300 can be of a variety of types, including handheld portable devices (e.g., iPhone® cellular phones, iPad® computing tablets, PDAs), wearable devices (e.g., Google Glass® head-mounted displays), PCs, workstations, mainframes, kiosks, server racks, or any other data processing system.

[0340] Due to the constantly evolving nature of computers and networks, the description of the computer system 2300 depicted in the figures is intended only as a specific example. Many other configurations with more or fewer components than the system depicted in the figures are possible. For example, custom hardware may also be used and / or specific elements may be implemented in hardware, firmware, software (including applets), or a combination thereof. Furthermore, connectivity with other computing devices, such as network input / output devices, may be employed. Based on the disclosure and teachings provided herein, those skilled in the art will understand other ways and / or methods for implementing the various embodiments.

[0341] Although specific embodiments have been described, various modifications, alterations, alternative constructions, and equivalents are also covered within the scope of this disclosure. The embodiments are not limited to operation within certain specific data processing environments, but can freely operate within multiple data processing environments. Additionally, although embodiments have been described using a specific series of transactions and steps, it will be apparent to those skilled in the art that the scope of this disclosure is not limited to the described series of transactions and steps. The various features and aspects of the above embodiments can be used individually or in combination.

[0342] Furthermore, while embodiments have been described using specific combinations of hardware and software, it should be recognized that other combinations of hardware and software are also within the scope of this disclosure. Embodiments may be implemented solely in hardware, solely in software, or using a combination thereof. The various processes described herein may be implemented on the same or different processors in any combination. Thus, where a component or module is described as being configured to perform certain operations, such a configuration may be implemented, for example, by designing electronic circuits to perform operations, by programming programmable electronic circuits (such as microprocessors), or any combination thereof. Processes may communicate using a variety of techniques, including but not limited to conventional techniques for inter-process communication, and different pairs of processes may use different techniques, or the same pair of processes may use different techniques at different times.

[0343] Therefore, the specification and drawings are to be considered illustrative rather than restrictive. However, it will be clear that additions, deletions, omissions, and other modifications and alterations may be made thereto without departing from the broader spirit and scope set forth in the claims. Thus, although specific disclosed embodiments have been described, they are not intended to be limiting. Various modifications and equivalents are within the scope of the following claims.

[0344] In the context of describing the disclosed embodiments (particularly in the context of the following claims), the terms "a," "an," and "the," and similar pronouns, should be interpreted to cover both the singular and plural, unless otherwise stated herein or obviously contradicted by the context. Unless otherwise stated, the terms "comprising," "having," "owning," and "including" should be interpreted as open-ended terms (i.e., meaning "including, but not limited to"). The term "connected to" should be interpreted as partially or wholly included, attached to, or combined with, even if something else is involved. Unless otherwise indicated herein, the enumeration of numerical ranges herein is intended only as a concise way of expressing each separate value falling within that range, and each separate value is incorporated into the specification as if it were individually enumerated herein. Unless otherwise indicated herein or obviously contradicted by the context, all methods described herein may be performed in any suitable order. Unless otherwise claimed, the use of any and all examples or exemplary language (e.g., "such as") provided herein is intended only to better illustrate the embodiments and does not limit the scope of this disclosure. Nothing in the specification should be construed as indicating that any unclaimed element is essential to the practice of this disclosure.

[0345] Unless otherwise expressly stated, disjunctive language such as the phrase "at least one of X, Y, or Z" is intended to be understood in context as generally used to refer to items, terms, etc., which can be X, Y, or Z or any combination thereof (e.g., X, Y, and / or Z). Therefore, such disjunctive language is generally not intended and should not imply that certain embodiments require the presence of at least one of X, at least one of Y, or at least one of Z.

[0346] This document describes preferred embodiments of the present disclosure, including known best modes for carrying out the present disclosure. Variations of those preferred embodiments will become apparent to those skilled in the art after reading the foregoing description. Those skilled in the art should be able to appropriately employ such variations, and the present disclosure can be practiced in addition to those specifically described herein. Therefore, the present disclosure includes all modifications and equivalents of the subject matter recited in the appended claims as permitted by applicable law. Furthermore, unless otherwise indicated herein, any combination of the foregoing elements in all possible variations is included in this disclosure.

[0347] All references cited herein, including publications, patent applications and patents, are incorporated herein by reference to the extent that each reference is individually and specifically indicated as being incorporated and set forth herein by reference in its entirety.

[0348] In the foregoing description, aspects of this disclosure have been described with reference to specific embodiments thereof; however, those skilled in the art will recognize that this disclosure is not limited thereto. The various features and aspects of the foregoing disclosure may be used alone or in combination. Furthermore, embodiments may be utilized in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of this specification. Therefore, the description and drawings are to be considered illustrative rather than restrictive.< / realm>

Claims

1. A method comprising: The first cloud environment of the first cloud service provider detects that a request for cloud services provided by the first cloud service provider has been received from the second cloud environment of a second cloud service provider that is different from the first cloud service provider. After detecting that the request for the cloud service has been received, the first cloud environment supplies the first set of resources within the first cloud environment. as well as The first cloud environment links a first set of resources to a second set of resources within a second cloud environment, wherein linking the first set of resources to the second set of resources enables data about the cloud service to be transferred from the second cloud environment to the first cloud environment.

2. The method of claim 1, wherein the request is initiated in response to a portal interaction between a client device and a second cloud environment, and wherein the client device is associated with a client account of a second cloud service provider.

3. The method according to claim 1, wherein supplying the first set of resources within the first cloud environment comprises: Receive information identifying the first subnet in the second cloud environment and the first Internet Protocol (IP) address range associated with the first subnet; Providing private cloud within the first cloud environment; as well as A second subnet is provisioned in the private cloud, wherein the second IP address range associated with the second subnet is within the classless inter-domain routing (CIDR) range of the first IP address range.

4. The method of claim 3, further comprising: Resources comprising a virtual machine cluster are created in a first cloud environment, wherein each IP address in a second IP address range is associated with a virtual machine in the virtual machine cluster; as well as Attach the resource to the router of the first cloud environment.

5. The method of claim 3, wherein linking the first group of resources to the second group of resources comprises: Call the application programming interface (API) of the second cloud environment, wherein calling the API causes IP addresses in the second IP address range to appear in the first subnet; Use the API to register the IP address within the second IP address range with the router in the second cloud environment; Receive the virtual LAN identifier from the second cloud environment; as well as The virtual LAN identifier is used to create a virtual circuit between the router in the first cloud environment and the router in the second cloud environment.

6. The method of claim 5, wherein linking the first group of resources to the second group of resources further comprises: Access the Domain Name System (DNS) records associated with the virtual machine cluster in the first cloud environment; as well as The DNS record is provided to the second cloud environment, wherein providing the DNS record to the second cloud environment causes a private DNS zone to be created in a private cloud associated with a customer account of the second cloud service provider.

7. The method of claim 1, wherein linking the first set of resources to the second set of resources enables data concerning the cloud service to be transferred from a first subnet associated with a first private cloud in a second cloud environment to a second subnet associated with a second private cloud in the first cloud environment.

8. A system comprising: One or more processing systems; as well as One or more computer-readable media storing instructions that, when executed by the one or more processing systems, cause the systems to perform operations including: The first cloud environment of the first cloud service provider detects that a request for cloud services provided by the first cloud service provider has been received from the second cloud environment of a second cloud service provider that is different from the first cloud service provider. After detecting that the request for the cloud service has been received, the first cloud environment supplies the first set of resources within the first cloud environment. as well as The first cloud environment links a first set of resources to a second set of resources within a second cloud environment, wherein linking the first set of resources to the second set of resources enables data about the cloud service to be transferred from the second cloud environment to the first cloud environment.

9. The system of claim 8, wherein the request is initiated in response to a portal interaction between a client device and a second cloud environment, and wherein the client device is associated with a client account of a second cloud service provider.

10. The system of claim 8, wherein supplying the first set of resources within the first cloud environment comprises: Receive information identifying the first subnet in the second cloud environment and the first Internet Protocol (IP) address range associated with the first subnet; Providing private cloud within the first cloud environment; as well as A second subnet is provisioned in the private cloud, wherein the second IP address range associated with the second subnet is within the classless inter-domain routing (CIDR) range of the first IP address range.

11. The system of claim 10, wherein the operation further comprises: Resources comprising a virtual machine cluster are created in a first cloud environment, wherein each IP address in a second IP address range is associated with a virtual machine in the virtual machine cluster; as well as Attach the resource to the router of the first cloud environment.

12. The system of claim 10, wherein linking the first set of resources to the second set of resources comprises: Call the application programming interface (API) of the second cloud environment, wherein calling the API causes IP addresses in the second IP address range to appear in the first subnet; Use the API to register the IP address within the second IP address range with the router in the second cloud environment; Receive the virtual LAN identifier from the second cloud environment; as well as The virtual LAN identifier is used to create a virtual circuit between the router in the first cloud environment and the router in the second cloud environment.

13. The system of claim 12, wherein linking the first set of resources to the second set of resources further comprises: Access the Domain Name System (DNS) records associated with the virtual machine cluster in the first cloud environment; as well as The DNS record is provided to the second cloud environment, wherein providing the DNS record to the second cloud environment causes a private DNS zone to be created in a private cloud associated with a customer account of the second cloud service provider.

14. The system of claim 8, wherein linking the first set of resources to the second set of resources enables data concerning the cloud service to be transferred from a first subnet associated with a first private cloud in a second cloud environment to a second subnet associated with a second private cloud in the first cloud environment.

15. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processing systems, cause the systems to perform operations including: The first cloud environment of the first cloud service provider detects that a request for cloud services provided by the first cloud service provider has been received from the second cloud environment of a second cloud service provider that is different from the first cloud service provider. After detecting that the request for the cloud service has been received, the first cloud environment supplies the first set of resources within the first cloud environment. as well as The first cloud environment links a first set of resources to a second set of resources within a second cloud environment, wherein linking the first set of resources to the second set of resources enables data about the cloud service to be transferred from the second cloud environment to the first cloud environment.

16. The one or more non-transitory computer-readable media of claim 15, wherein the request is initiated in response to a portal interaction between a client device and a second cloud environment, and wherein the client device is associated with a client account of a second cloud service provider.

17. One or more non-transitory computer-readable media according to claim 15, wherein supplying the first set of resources within the first cloud environment comprises: Receive information identifying the first subnet in the second cloud environment and the first Internet Protocol (IP) address range associated with the first subnet; Providing private cloud within the first cloud environment; as well as A second subnet is provisioned in the private cloud, wherein the second IP address range associated with the second subnet is within the classless inter-domain routing (CIDR) range of the first IP address range.

18. The one or more non-transitory computer-readable media of claim 17, further comprising: Resources comprising a virtual machine cluster are created in a first cloud environment, wherein each IP address in a second IP address range is associated with a virtual machine in the virtual machine cluster; as well as Attach the resource to the router of the first cloud environment.

19. One or more non-transitory computer-readable media according to claim 17, wherein linking the first set of resources to the second set of resources comprises: Call the application programming interface (API) of the second cloud environment, wherein calling the API causes IP addresses in the second IP address range to appear in the first subnet; Use the API to register the IP address within the second IP address range with the router in the second cloud environment; Receive the virtual LAN identifier from the second cloud environment; as well as The virtual LAN identifier is used to create a virtual circuit between the router in the first cloud environment and the router in the second cloud environment.

20. One or more non-transitory computer-readable media of claim 15, wherein linking the first set of resources to the second set of resources enables data concerning the cloud service to be transferred from a first subnet associated with a first private cloud in a second cloud environment to a second subnet associated with a second private cloud in the first cloud environment.

Images