Providing services based on infrastructure distributed between multiple cloud service providers

Abstract

Techniques for providing services based on infrastructures distributed among multiple cloud service providers are disclosed. Low latency, high bandwidth, cross-cloud services may be provisioned and managed therebetween by a private cloud of a cloud service provider. The technique includes forming a cloud network between a first set of computing resources of a first infrastructure of a first cloud environment and a second set of computing resources of a second infrastructure of a second cloud environment. The first cloud environment is provided by a first cloud service provider and the second cloud environment is provided by a second cloud service provider different from the first cloud service provider.

Description

[0001] Cross-reference to related applications

[0002] This application claims the interests and priorities of U.S. Provisional Application No. 63 / 608,036, filed December 8, 2023; U.S. Provisional Application No. 63 / 538,254, filed September 13, 2023; U.S. Provisional Application No. 63 / 534,708, filed August 25, 2023; and U.S. Provisional Application No. 63 / 534,071, filed August 22, 2023, the entire contents of which are incorporated herein by reference for all purposes. Background Technology

[0003] Over the past few years, the adoption of cloud services has increased dramatically, and this trend is only expected to continue. Different cloud service providers (CSPs) offer a variety of different cloud environments, each providing a set of one or more cloud services. This set of cloud services offered by a cloud environment can include one or more different types of services, including but not limited to Software as a Service (SaaS), Infrastructure as a Service (IaaS), Platform as a Service (PaaS), etc.

[0004] While various cloud environments are currently available, each provides a closed ecosystem for its subscribers. Therefore, customers of a cloud environment are limited to the services offered by that specific cloud environment. There is no easy way for customers subscribed to a cloud environment provided by a CSP to access services offered by different CSPs through that cloud environment. The embodiments discussed herein address these and other problems. Summary of the Invention

[0005] This article discloses techniques for providing services based on infrastructure distributed across multiple cloud service providers.

[0006] In some embodiments, a system includes a first cloud environment, the first cloud environment including a first infrastructure, wherein: the first cloud environment is provided by a first cloud service provider; the first infrastructure is physically connected to a second infrastructure of a second cloud environment provided by a second cloud service provider different from the first cloud service provider; the first infrastructure includes a first set of computing resources located within the first cloud environment; the first infrastructure is configured to form a cloud network between the first set of computing resources and a second set of computing resources of the second infrastructure located within the second cloud environment; and the cloud network is configured to provide cloud services of the second cloud service provider to customers of the first cloud service provider using the first set of computing resources and the second set of computing resources.

[0007] In some embodiments, the first infrastructure is configured to form a cloud network in response to receiving instructions from the control plane of the second infrastructure, and to establish a network connection between a first subnet of a first set of computing resources and a second subnet of a second set of computing resources, the first subnet being provided by the data plane of the first infrastructure.

[0008] In some embodiments, the first infrastructure is configured to establish a network connection by associating an Internet Protocol (IP) address with a first subnet, the IP address being within the Classless Inter-Domain Routing (CIDR) IP range associated with a second subnet.

[0009] In some embodiments, a first router associated with a first cloud environment is connected to a second router associated with a second cloud environment, and the first router is also connected to a first infrastructure.

[0010] In some embodiments, the first infrastructure is configured to form a cloud network by connecting virtual resources of a first set of computing resources to a virtual machine cluster of a second set of computing resources.

[0011] In some embodiments, the first infrastructure is configured to form a cloud network by connecting a first network interface card of a first set of computing resources to a second network interface card of a second set of computing resources.

[0012] In some embodiments, the first set of computing resources includes one or more processors, wherein the one or more processors are connected to the physical resources of a second cloud service provider.

[0013] In some embodiments, a method includes receiving a request for a cloud service; and, upon receiving the request for the cloud service, forming a cloud network between a first set of computing resources and a second set of computing resources, wherein the first set of computing resources is located within a first infrastructure of a first cloud environment provided by a first cloud service provider, wherein the second set of computing resources is located within a second infrastructure of a second cloud environment provided by a second cloud service provider different from the first cloud service provider, wherein the first infrastructure is physically connected to the second infrastructure, and wherein the cloud network is configured to provide cloud services of the first cloud service provider to customers of the second cloud service provider using the first set of computing resources and the second set of computing resources.

[0014] In some embodiments, forming a cloud network includes receiving instructions from the control plane of a second infrastructure, and after receiving instructions from the control plane, establishing a network connection between a first subnet of a first set of computing resources and a second subnet of a second set of computing resources, the first subnet being provided by the data plane of the first infrastructure, and wherein forming a cloud network enables data about cloud services to be transferred from the data plane to an area in the second infrastructure.

[0015] In some embodiments, the Internet Protocol (IP) address associated with the first subnet is within the Classless Inter-Domain Routing (CIDR) IP range associated with the second subnet.

[0016] In some embodiments, a first router associated with a first cloud environment is connected to a second router associated with a second cloud environment, wherein the first router is also connected to a first infrastructure.

[0017] In some embodiments, forming a cloud network includes connecting virtual resources of a first set of computing resources to a virtual machine cluster of a second set of computing resources.

[0018] In some embodiments, forming a cloud network includes connecting a first network interface card of a first set of computing resources to a second network interface card of a second set of computing resources.

[0019] In some embodiments, the first set of computing resources includes one or more processors, wherein the one or more processors are connected to the physical resources of a second cloud service provider.

[0020] Some embodiments include one or more non-transitory computer-readable media storing instructions that, when executed by one or more processors of a computer system, cause the computer system to perform some or all of the operations and / or methods disclosed herein.

[0021] The techniques described above and below can be implemented in a variety of ways and in a variety of contexts. Several example implementations and contexts are provided below in more detail, with reference to the accompanying figures. However, the implementations and contexts below are only a few of the many. Attached Figure Description

[0022] The features, embodiments, and advantages of this disclosure can be better understood by reading the following detailed description with reference to the accompanying drawings.

[0023] Figure 1 This is a high-level diagram illustrating a distributed environment of a virtual or overlay cloud network hosted by a cloud service provider infrastructure, according to certain embodiments.

[0024] Figure 2 A simplified architecture diagram of physical components in the physical network within a Cloud Service Provider Infrastructure (CSPI) according to certain embodiments is depicted.

[0025] Figure 3 An example arrangement within a CSPI according to certain embodiments is shown, in which a host is connected to multiple network virtualization devices (NVDs).

[0026] Figure 4 The diagram illustrates connectivity between multi-tenant hosts and NVDs, according to certain embodiments, for providing I / O virtualization to support multi-tenant hosts.

[0027] Figure 5 A simplified block diagram of a physical network provided by CSPI according to certain embodiments is depicted.

[0028] Figure 6 A simplified high-level diagram of a distributed environment comprising multiple cloud environments provided by different cloud service providers (CSPs) is depicted according to certain embodiments.

[0029] Figure 7 An exemplary physical architecture for providing cross-cloud services based on infrastructure distributed across multiple CSPs, according to some embodiments, is described.

[0030] Figure 8 An exemplary virtual architecture for providing cross-cloud services based on infrastructure distributed across multiple CSPs, according to some embodiments, is described.

[0031] Figure 9 Exemplary virtual resources provided by a first CSP to a customer of a second CSP, according to some embodiments, are described.

[0032] Figure 10 An exemplary architecture for provisioning and managing cross-cloud services based on infrastructure distributed across multiple CSPs is described according to some embodiments.

[0033] Figure 11 An exemplary user experience flow for provisioning resources is described according to some embodiments.

[0034] Figure 12 An exemplary control plane supply flow according to some embodiments is depicted.

[0035] Figure 13 An example architecture for providing services based on infrastructure distributed across multiple cloud service providers, according to some embodiments, is described.

[0036] Figure 14 An example architecture of infrastructure distributed among multiple cloud service providers, according to some embodiments, is described.

[0037] Figure 15 An example process for providing services based on infrastructure distributed across multiple cloud service providers, according to some embodiments, is described.

[0038] Figure 16 This is a block diagram illustrating a pattern for implementing a cloud infrastructure-as-a-service system according to certain embodiments.

[0039] Figure 17 This is a block diagram illustrating another pattern for implementing a cloud infrastructure-as-a-service system according to certain embodiments.

[0040] Figure 18 This is a block diagram illustrating another pattern for implementing a cloud infrastructure-as-a-service system according to certain embodiments.

[0041] Figure 19 This is a block diagram illustrating another pattern for implementing a cloud infrastructure-as-a-service system according to certain embodiments.

[0042] Figure 20 This is a block diagram illustrating an example computer system according to certain embodiments. Detailed Implementation

[0043] In the following description, specific details are set forth for illustrative purposes in order to provide a thorough understanding of certain embodiments. However, it will be clear that various embodiments can be practiced without these specific details. The accompanying drawings and descriptions are not intended to be limiting. The word “exemplary” is used herein to mean “serves as an example, instance, or illustration.” Any embodiment or design described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments or designs.

[0044] This disclosure generally relates to improved cloud architectures, and more specifically to technologies for providing services based on infrastructure distributed across multiple cloud service providers (CSPs). In the example, the first cloud service provider (CSP) (e.g., Oracle) ® The cloud infrastructure (OCI) provides primary services (e.g., database services, such as Exadata services available from Oracle), some of which may be available from the primary cloud service provider infrastructure (CSPI) of the primary CSP. The secondary CSP (e.g., Microsoft) provides the primary services. ® Azure, Google Cloud™, Amazon Web Services—AWS ® The system provides a second service (e.g., Azure virtual machines), some of which may be available from a second cloud service provider infrastructure (CSPI) of the second CSP. This makes at least one of the first services available to the customers of the second CSP via the second CSPI. In the example, the hardware and / or software configured by the first CSPI to provide the first service is deployed at the second CSPI.

[0045] For example, at the physical level, the second CSPI may include a first set of computing resources of the first CSPI (e.g., server racks optimized for the services of the first CSPI). The first CSPI may include a second set of computing resources of the first CSPI (e.g., servers within a region). The second CSPI may also include a third set of computing resources of the second CSPI (e.g., servers within an availability zone). The first set of computing resources may be connected to the second and third sets of computing resources.

[0046] At the virtualization level, the first and second sets of resources can be hosted in a first cloud for a first CSP and its customers. Conversely, the third set of resources can be hosted in a second cloud for a second CSP and its customers. These two clouds can be connected (e.g., via a peering connection using a virtual router). The services provided by the first CSP to its customers can have first resources hosted in the first set of compute resources and second resources hosted in the second set of compute resources. This service can be made available to customers via the second cloud. The use of the first resources allows for reduced latency. Providing services from the first CSP via the second CSP can enable a better experience by extending the service to customers of the second CSP who may be more familiar with configuring and managing the cloud using the second CSP.

[0047] For illustration, consider an example of an Exadata service (i.e., the OCI database service) made available via Azure. In this example, at the physical level, OCI subsites belonging to the OCI region are co-located in an Azure datacenter. The OCI subsites are connected to the Azure datacenter via a FastConnect router (and a Microsoft MeetMe ToR router on the Azure side). The OCI subsites are also connected to the parent OCI region via a fiber optic connection. At the virtual level, the Azure datacenter hosts the customer cloud (e.g., VNET), while the parent OCI region and the subsites co-host the customer virtual cloud network (VCN). Latency-critical resources for the OCI service (e.g., data plane resources for the Exadata service) are hosted in the subsites, while other supporting resources (e.g., control plane resources, customer-facing console resources, etc.) are hosted in the parent region.

[0048] On the OCI side, VNET and VCN are connected via virtual routers (e.g., a Dynamic Routing Gateway (DRG) supporting FastConnect and a virtual router provided by Azure supporting MeetMe). Within VNET, delegated subnets can be configured. Within VCN, Exadata service subnets can be configured. Both subnets use the same Internet Protocol (IP) address range. VNET can store mapping information that maps IP addresses within the IP address range used in VNET to IP addresses of the Exadata service at VCN. Given a one-to-one IP address mapping, traffic to the IP address (e.g., compute instances from VNET) is sent from VNET to VCN.

[0049] Therefore, from the customer's perspective, the customer perceives that their Exadata service within their VNET has an IP address. However, the Exadata service is actually located within the VCN at that IP address. Furthermore, by hosting at least a portion of the Exadata service (e.g., the data plane) in a subsite, the latency associated with the Exadata service can be reduced.

[0050] Referring back to the architecture above, the first group of computing resources can be referred to as belonging to the sub-site, and the second group of computing resources can be referred to as belonging to the parent site. The third group of computing resources can be referred to as data center resources. The sub-sites of the first CSP are deployed in the data center (or second CSPI) of the second CSP. In contrast, the parent site of the first CSP is deployed in the first CSPI of the first CSP.

[0051] Subsites allow low-latency services from the first CSP to be delivered to customers via a second CSP. However, deployment mechanisms may be needed to control the deployment of the first CSP's resources across subsites and the parent site.

[0052] Embodiments of this disclosure relate to a deployment mechanism in which a control plane for a first CSP is not hosted at a subsite. Customer input at a second CSPI can be received by the control plane and mapped to a customer ID related to the first CSPI. Based on the customer ID, the control plane can provision resources at the first CSPI, such as by creating a cloud network for the customer and associated connectivity resources (e.g., a Dynamic Routing Gateway – DRG) for the cloud network. The cloud network can also be connected to a cluster of virtual machines (VMs) hosting services. The VM cluster can be hosted at a subsite, while other resources of the cloud network can be hosted at a parent site. Subsequently, the control plane can interconnect the cloud network with the customer's cloud network related to the second CSP, thus interconnecting the customer's two cloud networks. As part of this interconnection, the control plane can provide the second CSPI with the Internet Protocol (IP) addresses of the VM clusters and, possibly, corresponding DNS records, making these IP addresses and, possibly, the DNS records, available to the customer at the customer's cloud network related to the second CSP.

[0053] To illustrate, consider the OCI and Azure example again. Azure customer input (e.g., selecting its VNET and subnet) is received from Azure Resource Manager (ARM). The Oracle Resource Provider (ORP) can convert this input into the customer's OCI ID and pass that information to the control plane. The control plane performs two main processes. The first process is provisioning the relevant OCI resources. The second process is connecting these resources to the customer's VNET.

[0054] In the first process, the control plane creates the customer's VCN in the customer's lease (e.g., at the parent site) and creates subnets (and possibly backup subnets) within the VCN. The Azure and OCI subnets (one in the customer's Azure VNET and one in the customer's OCI VCN) use the same Classless Inter-Domain Routing (CIDR). The control plane also creates a DRG in the customer's lease and attaches the DRG to the VCN. The control plane also establishes routing information for the DRG and VCN (enabling these two resources to interconnect). ORP can also be provisioned to the VM cluster (e.g., in the subsite). One or more IP addresses of this VM cluster from the CIDR are mapped to corresponding DNS records.

[0055] In the second process, the control plane registers these IP addresses with Azure (on the MeetMe router). It also creates a virtual circuit between the DRG and the MeetMe router and sends DNS records, enabling a private DNS zone to be established in Azure.

[0056] For clarity, embodiments of this disclosure have been described in relation to specific CSPs (e.g., Oracle and Microsoft) and services (e.g., Exadata services). However, the embodiments are not limited thereto, but rather, similarly and equivalently, apply to any CSP, CSPI, and service in a multi-cloud environment.

[0057] Cloud Network Example

[0058] The term cloud service is generally used to refer to services provided by a CSP (Consumer Service Provider) that make the systems and infrastructure (cloud infrastructure) offered by the CSP available to users or customers on demand (e.g., via a subscription model). Typically, the servers and systems that make up the CSP infrastructure are separate from the customer's own on-premises servers and systems. Therefore, customers can utilize cloud services provided by the CSP without having to purchase separate hardware and software resources for the service. Cloud services are designed to provide subscribers with easy, scalable access to a pool of applications and computing resources without requiring customers to invest in the infrastructure used to provide the service.

[0059] There are several cloud service providers that offer various types of cloud services. There are various different types or models of cloud services, including Software as a Service (SaaS), Infrastructure as a Service (IaaS), Platform as a Service (PaaS), etc.

[0060] A customer can subscribe to one or more cloud services provided by a CSP. A customer can be any entity, such as an individual, organization, or enterprise. When a customer subscribes to or registers for a service provided by a CSP, a lease or account is created for that customer. The customer can then access one or more cloud resources associated with that account through the subscription.

[0061] As mentioned above, IaaS is a specific type of cloud computing service. In the IaaS model, a CSP provides infrastructure (referred to as Cloud Service Provider Infrastructure or CSPI) that customers can use to build their own customizable networks and deploy customer resources. Therefore, the customer's resources and network are hosted in a distributed environment by the infrastructure provided by the CSP. This differs from traditional computing, where customer resources and networks are hosted by customer-provided infrastructure.

[0062] CSPI can include interconnected high-performance computing resources, including various host, memory, and network resources forming a physical network, also known as the base network or underlying network. Resources in CSPI can be distributed across one or more data centers, which can be geographically distributed across one or more geographic regions. Virtualization software can be executed by these physical resources to provide a virtualized distributed environment. Virtualization creates overlay networks (also known as software-based networks, software-defined networks, or virtual networks) on top of the physical network (or base network or underlying network). The physical network provides the underlying foundation for creating one or more overlay or virtual networks on top of it. The physical network includes physical network devices such as physical switches, routers, computers, and hosts. An overlay network is a logical (or virtual) network that runs on top of the physical base network. A given physical network can support one or more overlay networks. Overlay networks typically use encapsulation techniques to distinguish traffic belonging to different overlay networks. Virtual or overlay networks are also known as VCNs. Virtual networks utilize software virtualization technologies (e.g., hypervisors, functions implemented by network virtualization devices (NVDs) (e.g., smartNICs), top-of-rack (TOR) switches, intelligent TORs that implement one or more functions performed by NVDs, and other mechanisms) to create a network abstraction layer that can operate on top of the physical network. Virtual networks can take many forms, including peer-to-peer networks, IP networks, etc. Virtual networks are typically Layer-3 IP networks or Layer-2 VLANs. This method of virtual or overlay networking is often referred to as virtual or overlay Layer-3 networking. Examples of protocols developed for virtual networks include IP-in-IP (or Generic Routing Encapsulation (GRE)), Virtual Extensible LAN (VXLAN—IETF RFC 7348), Virtual Private Networks (VPNs) (e.g., MPLS Layer-3 Virtual Private Network (RFC 4364)), VMware's NSX, GENEVE (Generic Network Virtualization Encapsulation), and others.

[0063] For IaaS, CSPI provided by a CSP can be configured to provide virtualized groups of computing resources over a public network (e.g., the Internet). In the IaaS model, cloud service providers can host infrastructure components (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer), etc.). In some cases, IaaS providers can also provision a wide variety of services to accompany those infrastructure components (e.g., billing, monitoring, logging, security, load balancing, and clustering). Therefore, since these services can be policy-driven, IaaS users can implement policies to drive load balancing to maintain application availability and performance. CSPI provides infrastructure and a complementary set of cloud services, enabling customers to build and run a wide range of applications and services in a highly available, managed, distributed environment. CSPI provides high-performance computing resources and capabilities, as well as storage capacity, in a flexible virtual network securely accessible from various networked locations, such as from the customer's on-premises deployment network. When a customer subscribes to or registers for IaaS services provided by CSP, the lease created for that customer is a secure and isolated partition within CSP, in which the customer can create, organize, and manage their cloud resources.

[0064] Customers can use the compute, storage, and networking resources provided by CSPI to build their own virtual networks. One or more customer resources or workloads, such as compute instances, can be deployed on these virtual networks. For example, a customer can use the resources provided by CSPI to build one or more customizable and private virtual networks(s) known as VCNs. A customer can deploy one or more customer resources, such as compute instances, on a customer VCN. Compute instances can take the form of virtual machines, bare metal instances, etc. Therefore, CSPI provides the infrastructure and a complementary set of cloud services that enable customers to build and run a wide range of applications and services in a highly available virtual hosting environment. Customers do not manage or control the underlying physical resources provided by CSPI, but they have control over the operating system, storage devices, and deployed applications; and possibly limited control over chosen networking components (e.g., firewalls).

[0065] The CSP can provide a console that enables customers and network administrators to configure, access, and manage resources deployed in the cloud using CSPI resources. In some embodiments, the console provides a web-based user interface for accessing and managing the CSPI. In some implementations, the console is a web-based application provided by the CSP.

[0066] CSPI can support single-tenant or multi-tenant architectures. In a single-tenant architecture, software (e.g., applications, databases) or hardware components (e.g., hosts or servers) serve a single customer or tenant. In a multi-tenant architecture, the software or hardware components serve multiple customers or tenants. Therefore, in a multi-tenant architecture, CSPI resources are shared among multiple customers or tenants. In a multi-tenant scenario, precautions and safeguards are implemented within CSPI to ensure that each tenant's data is isolated and remains invisible to other tenants.

[0067] In a physical network, a network endpoint (“endpoint”) is a computing device or system that is connected to and communicates with the physical network. Network endpoints in a physical network can be connected to a local area network (LAN), a wide area network (WAN), or other types of physical networks. Examples of traditional endpoints in a physical network include modems, hubs, bridges, switches, routers and other networking devices, physical computers (or hosts), etc. Each physical device in a physical network has a fixed network address that can be used to communicate with the device. This fixed network address can be a Layer-2 address (e.g., a MAC address), a fixed Layer-3 address (e.g., an IP address), etc. In a virtualized environment or in a virtual network, endpoints can include various virtual endpoints, such as virtual machines hosted by components of the physical network (e.g., hosted by a physical host). These endpoints in a virtual network are addressed using overlay addresses, such as overlay Layer-2 addresses (e.g., overlay MAC addresses) and overlay Layer-3 addresses (e.g., overlay IP addresses). Network overlays enable flexibility by allowing network administrators to move the overlay addresses associated with network endpoints using software management (e.g., via software implementing a control plane for virtual networks). Therefore, unlike in a physical network, in a virtual network, overlay addresses (e.g., overlay IP addresses) can be moved from one endpoint to another using network management software. Since a virtual network is built on top of a physical network, communication between components within a virtual network involves both the virtual network and the underlying physical network. To facilitate such communication, CSPI components are configured to learn and store mappings that map overlay addresses in the virtual network to actual physical addresses in the base network, and vice versa. These mappings are then used to facilitate communication. Client traffic is encapsulated to facilitate routing within the virtual network.

[0068] Therefore, physical addresses (e.g., physical IP addresses) are associated with components in a physical network, and overlay addresses (e.g., overlay IP addresses) are associated with entities in a virtual or overlay network. A physical IP address is an IP address associated with a physical device (e.g., a network device) in the underlying or physical network. For example, each NVD has an associated physical IP address. An overlay IP address is an overlay address associated with an entity in the overlay network, such as a compute instance in a customer's VCN. Two different customers or tenants (each with its own private VCN) may potentially use the same overlay IP address in their VCNs without knowing about each other. Both physical IP addresses and overlay IP addresses are types of real IP addresses. These are separate from virtual IP addresses. A virtual IP address is typically a single IP address that represents or maps to multiple real IP addresses. A virtual IP address provides a one-to-many mapping between virtual IP addresses and multiple real IP addresses. For example, a load balancer can use a VIP to map to or represent multiple servers, each with its own real IP address.

[0069] CSPI is physically hosted in one or more data centers in one or more regions around the world. CSPI may include components in the physical or underlying network and virtualized components (e.g., virtual networks, compute instances, virtual machines, etc.) in a virtual network built on top of the physical network components. In some embodiments, CSPI is organized and hosted in domains, regions, and availability domains. A region is typically a local geographic area containing one or more data centers. Regions are typically independent of each other and can be geographically separated, for example, across countries or even continents. For example, one region might be in Australia, another in Japan, yet another in India, and so on. CSPI resources are partitioned between regions, such that each region has its own independent subset of CSPI resources. Each region can provide a set of core infrastructure services and resources, such as compute resources (e.g., bare metal servers, virtual machines, containers, and related infrastructure); storage resources (e.g., block volume storage, file storage, object storage, archive storage); networking resources (e.g., VCN, load balancing resources, connectivity to on-premises networks); database resources; edge networking resources (e.g., DNS); access management and monitoring resources, etc. Each region typically has multiple paths connecting it to other regions within its domain.

[0070] Generally, applications are deployed in the areas where they are used most frequently (i.e., on the infrastructure associated with that area) because using nearby resources is faster than using resources far away. Applications can also be deployed in different areas for various reasons, such as redundancy to mitigate the risk of regional events such as large weather systems or earthquakes, or to meet the requirements of changing legal jurisdictions, tax domains, and other business or social standards.

[0071] Data centers within a region can be further organized and subdivided into Availability Domains (ADs). An Availability Domain can correspond to one or more data centers located within a region. A region can consist of one or more Availability Domains. In such a distributed environment, CSPI resources are either region-specific (such as VCNs) or Availability Domain-specific (such as compute instances).

[0072] Availability Zones (ADs) within a region are isolated and fault-tolerant, and configured to make simultaneous failures unlikely. This is achieved by ensuring that ADs do not share critical infrastructure resources (such as networking, physical cabling, cable paths, cable entry points, etc.), making a failure at one AD within a region unlikely to affect the availability of other ADs in the same region. ADs within the same region can be interconnected via low-latency, high-bandwidth networks, making it possible to provide high-availability connectivity to other networks (e.g., the internet, customer on-premises networks, etc.) and to build replication systems across multiple ADs for both high availability and disaster recovery. Cloud services use multiple ADs to ensure high availability and prevent resource failures. As the infrastructure provided by IaaS providers grows, additional capacity can be added to more regions and ADs. Traffic between availability domains is typically encrypted.

[0073] In some embodiments, regions are grouped into domains. A domain is a logical collection of regions. Domains are isolated from each other and do not share any data. Regions within the same domain can communicate with each other, but regions in different domains cannot. A customer's lease or account with a CSP exists within a single domain and can be distributed across one or more regions belonging to that domain. Typically, when a customer subscribes to an IaaS service, the lease or account is created for that customer in the customer-designated region within a domain (referred to as the "home" region). A customer can extend their lease across one or more other regions within a domain. A customer cannot access regions that are not within the domain where their lease exists.

[0074] IaaS providers can offer multiple domains, each catering to a specific set of customers or users. For example, a business domain can be offered to business customers. As another example, a country-specific domain can be offered to customers within a specific country. And yet another example, a government domain can be offered to governments. For instance, a government domain can cater to a specific government and may offer a higher level of security compared to the business domain. For example, OCI currently offers one domain for its business region and two domains for its government cloud region (e.g., FedRAMP license and IL5 license).

[0075] In some embodiments, an Active Directory (AD) can be subdivided into one or more fault domains. A fault domain is a grouping of infrastructure resources within an AD to provide anti-affinity. Fault domains allow for the distribution of compute instances so that instances are not on the same physical hardware within a single AD. This is called anti-affinity. A fault domain refers to a group of hardware components (computers, switches, etc.) that share a single point of failure. Compute pools are logically divided into fault domains. Therefore, a hardware failure or compute hardware maintenance event affecting one fault domain does not affect instances in other fault domains. Depending on the embodiment, the number of fault domains per AD can vary. For example, in some embodiments, each AD contains three fault domains. Fault domains act as logical data centers within the AD.

[0076] When a customer subscribes to IaaS services, resources from CSPI are provisioned to the customer and associated with the customer's lease. The customer can use these provisioned resources to build private networks and deploy resources on those networks. A customer network hosted in the cloud by CSPI is called a VCN. A customer can use the CSPI resources allocated to them to establish one or more VCNs. A VCN is a virtual or software-defined private network. Customer resources deployed in a customer's VCN can include compute instances (e.g., virtual machines, bare metal instances) and other resources. These compute instances can represent various customer workloads, such as applications, load balancers, databases, etc. Compute instances deployed on a VCN can communicate with publicly accessible endpoints (“public endpoints”) via public networks such as the Internet, with other instances in the same VCN or other VCNs (e.g., other VCNs belonging to the customer or VCNs not belonging to the customer), with the customer's on-premises data center or network, and with service endpoints and other types of endpoints.

[0077] CSPs can use CSPIs to provide various services. In some instances, CSPI clients can act as service providers themselves and use CSPI resources to provide services. Service providers can expose service endpoints characterized by identifying information (e.g., IP address, DNS name, and port). Client resources (e.g., compute instances) can consume a specific service by accessing the service endpoints exposed by the service for that specific service. These service endpoints are typically publicly accessible to users via public communication networks (such as the Internet) using the public IP address associated with the endpoint. Publicly accessible network endpoints are sometimes referred to as public endpoints.

[0078] In some embodiments, a service provider may expose the service via an endpoint (sometimes referred to as a service endpoint). Clients of the service can then use this service endpoint to access the service. In some implementations, the service endpoint provided for the service can be accessed by multiple clients intending to consume the service. In other implementations, a dedicated service endpoint can be provided to a client, so that only that client can use that dedicated service endpoint to access the service.

[0079] In some embodiments, when a VCN is created, it is associated with a Private Overlay Classless Inter-Domain Routing (CIDR) address space, which refers to a range of private overlay IP addresses (e.g., 10.0 / 16) assigned to the VCN. A VCN includes associated subnets, routing tables, and gateways. A VCN resides within a single area but can span one or more or all availability domains within that area. A gateway is a virtual interface configured for the VCN and enables communication to and from the VCN to one or more endpoints outside the VCN. One or more different types of gateways can be configured for the VCN to enable communication to or from different types of endpoints.

[0080] A VCN can be subdivided into one or more subnets or subnetworks. Therefore, a subnet is a configuration unit or subdivision that can be created within a VCN. A VCN can have one or more subnets. Each subnet within a VCN is associated with a contiguous range of overlay IP addresses (e.g., 10.0.0.0 / 24 and 10.0.1.0 / 24) that do not overlap with other subnets within that VCN or subnets representing subsets of the address space within the VCN's address space.

[0081] Each compute instance is associated with a Virtual Network Interface Card (VNIC), which enables the compute instance to join a subnet within a VCN. A VNIC is the logical representation of a physical network interface card (NIC). Generally, a VNIC is the interface between an entity (e.g., a compute instance, a service) and a virtual network. A VNIC exists within a subnet and has one or more associated IP addresses and associated security rules or policies. A VNIC is equivalent to a Layer-2 port on a switch. A VNIC is attached to both the compute instance and the subnet within the VCN. The VNIC associated with a compute instance enables the compute instance to become part of a VCN's subnet and allows the compute instance to communicate with endpoints such as those on the same subnet, those in different subnets within the VCN, or those outside the VCN. Therefore, the VNIC associated with a compute instance determines how the compute instance connects to endpoints inside and outside the VCN. When a compute instance is created and added to a subnet within a VCN, a VNIC for the compute instance is created and associated with that compute instance. For a subnet that includes a set of compute instances, the subnet contains a VNIC corresponding to that set of compute instances, and each VNIC is attached to a compute instance within that set of compute instances.

[0082] Each compute instance is assigned a private overlay IP address via a VNIC associated with it. This private overlay IP address is assigned to the VNIC associated with the compute instance when the compute instance is created and used to route traffic to and from the compute instance. All VNICs within a given subnet use the same routing table, security list, and DHCP options. As described above, each subnet within a VCN is associated with a contiguous range of overlay IP addresses (e.g., 10.0.0.0 / 24 and 10.0.1.0 / 24) that do not overlap with other subnets within that VCN or subnets representing subsets of the address space within the VCN's address space. For a VNIC on a specific subnet of a VCN, the private overlay IP address assigned to the VNIC is an address derived from the contiguous range of overlay IP addresses allocated to that subnet.

[0083] In some embodiments, in addition to a private overlay IP address, a compute instance may optionally be assigned additional overlay IP addresses, such as one or more public IP addresses, for example, if in a public subnet. These multiple addresses are assigned on the same VNIC or on multiple VNICs associated with the compute instance. However, each instance has a primary VNIC created during instance startup and associated with the overlay private IP address assigned to the instance—this primary VNIC cannot be removed. Additional VNICs (referred to as secondary VNICs) may be added to an existing instance in the same availability domain as the primary VNIC. All VNICs are in the same availability domain as the instance. Secondary VNICs may be in a subnet within the same VCN as the primary VNIC, or in a different subnet within the same VCN or different VCNs.

[0084] If a compute instance is in a public subnet, it can be optionally assigned a public IP address. When a subnet is created, it can be specified as either a public or private subnet. A private subnet means that resources within the subnet (e.g., compute instances) and associated VNICs cannot have public overriding IP addresses. A public subnet means that resources within the subnet and associated VNICs can have public IP addresses. Customers can specify that a subnet exists within a single availability domain or across multiple availability domains across regions or domains.

[0085] As described above, a VCN can be subdivided into one or more subnets. In some embodiments, a virtual router (VR) configured for the VCN (referred to as a VCN VR or VR-only) enables communication between the VCN's subnets. For a subnet within the VCN, the VR represents the logical gateway for that subnet, enabling that subnet (i.e., the compute instances on that subnet) to communicate with endpoints on other subnets within the VCN and with other endpoints outside the VCN. A VCN VR is a logical entity configured to route traffic between a VNIC within the VCN and a virtual gateway (“gateway”) associated with the VCN. The VCN VR is a Layer-3 / IP layer concept. In one embodiment, there is a VCN VR for the VCN, where the VCN VR potentially has an unlimited number of ports addressed by IP addresses, with one port for each subnet of the VCN. In this way, the VCN VR has a different IP address for each subnet in the VCN to which the VCN VR is attached. The VR is also connected to various gateways configured for the VCN. In some embodiments, a specific overlay IP address from the overlay IP address range of the subnet is reserved for the port of the VCN VR for that subnet. For example, consider a VCN with two subnets, each with an associated address range of 10.0 / 16 and 10.1 / 16. For the first subnet within the VCN with an address range of 10.0 / 16, addresses from this range are reserved for ports on the VCN VR for that subnet. In some instances, the first IP address from the range can be reserved for the VCN VR. For example, for a subnet covering the 10.0 / 16 IP address range, IP address 10.0.0.1 might be reserved for a port on the VCN VR for that subnet. For a second subnet within the same VCN with an address range of 10.1 / 16, the VCN VR could have a port with the IP address 10.1.0.1 for that second subnet. The VCN VR has a different IP address for each subnet within the VCN.

[0086] In some other embodiments, each subnet within a VCN may have its own associated VR, which can be addressed by the subnet using a reserved or default IP address associated with the VR. The reserved or default IP address may, for example, be a first IP address from a range of IP addresses associated with the subnet. The VNICs within the subnet can communicate (e.g., send and receive packets) using the default or reserved IP address and the VR associated with the subnet. In such embodiments, the VR is the ingress / egress point of the subnet. The VR associated with a subnet within the VCN can communicate with other VRs associated with other subnets within the VCN. The VR can also communicate with a gateway associated with the VCN. The VR functionality of the subnet is running on or performed by one or more NVDs that perform the VNIC functionality of the VNICs within the subnet.

[0087] You can configure routing tables, security rules, and DHCP options for a VCN. The routing table is a virtual routing table for the VCN and contains rules that route traffic from subnets within the VCN to destinations outside the VCN via gateways or specially configured instances. The VCN's routing table can be customized to control how packets are forwarded / routed to or from the VCN. DHCP options refer to configuration information automatically provided to the instance when it boots up.

[0088] Security rules configured for a VCN represent the VCN's overriding firewall rules. Security rules can include ingress and egress rules and specify the types of traffic allowed to enter and exit instances within the VCN (e.g., based on protocol and port). Clients can choose whether a given rule is stateful or stateless. For example, a client can allow incoming SSH traffic from anywhere to a set of instances by establishing a stateful ingress rule using source CIDR 0.0.0.0 / 0 and destination TCP port 22. Security rules can be implemented using network security groups or security lists. A network security group consists of a set of security rules that apply only to resources within that group. A security list, on the other hand, includes rules that apply to all resources in any subnet using that security list. A VCN can be provided with a default security list containing default security rules. DHCP options configured for a VCN provide configuration information that is automatically provided to instances within the VCN at instance boot time.

[0089] In some embodiments, the configuration information of a VCN is determined and stored by the VCN control plane (CP). The VCN configuration information may include, for example, information about: the address range associated with the VCN, subnets and associated information within the VCN, one or more VRs associated with the VCN, compute instances and associated VNICs within the VCN, NVDs performing various virtualized network functions associated with the VCN (e.g., VNICs, VRs, gateways), VCN status information, and other VCN-related information. In some embodiments, a VCN distribution service publishes the configuration information or portions thereof stored by the VCN CP to the NVD. This distributed information can be used to update information stored and used by the NVD (e.g., forwarding tables, routing tables, etc.) to forward packets to and from compute instances within the VCN.

[0090] In some embodiments, the creation of the VCN and subnet is handled by the VCN CP, and the initiation of the compute instance is handled by the compute CP. The compute CP is responsible for allocating physical resources for the compute instance and then invoking the VCN CP to create the VNIC and attach it to the compute instance. The VCN CP also maps VCN data to the VCN data plane, which is configured to perform packet forwarding and routing functions. In some embodiments, the VCN CP provides a distribution service responsible for providing updates to the VCN data plane.

[0091] Customers can use resources hosted by CSPI to create one or more VCNs. Compute instances deployed on a customer's VCN can communicate with different endpoints. These endpoints can include CSPI-hosted endpoints and endpoints external to CSPI.

[0092] The various architectures used to implement cloud-based services using CSPI are described below. Figure 1 This is a high-level diagram illustrating a distributed environment 100 of an overlay or client VCN hosted by CSPI, according to certain embodiments. Figure 1 The distributed environment described includes multiple components in the overlay network. Figure 1 The distributed environment 100 depicted herein is merely an example and is not intended to unduly limit the scope of the claimed embodiments. Many variations, substitutions, and modifications are possible. For example, in some implementations, Figure 1 The distributed environment described in the text can have more than Figure 1 The more or fewer systems or components shown can be combined into two or more systems, or can have different system configurations or arrangements.

[0093] like Figure 1As illustrated in the example depicted, the distributed environment 100 includes a CSPI 101, which provides services and resources that customers can subscribe to and use to build their VCN. In some embodiments, CSPI 101 provides IaaS services to subscribing customers. Data centers within CSPI 101 can be organized into one or more regions. An example region, “Region US” 102, is... Figure 1 As shown in the diagram, the customer has already configured an Oracle International C / O customer VCN for region 102. The customer can deploy various compute instances on VCN 104, which can include virtual machines or bare metal instances. Examples of instances include applications, databases, load balancers, etc.

[0094] exist Figure 1 In the embodiment depicted, customer VCN 104 includes two subnets, namely "Subnet-1" and "Subnet-2", each with its own CIDR IP address range. Figure 1 In this configuration, subnet-1 covers the IP address range of 10.0 / 16, and subnet-2 covers the address range of 10.1 / 16. VCN Virtual Router 105 represents the logical gateway of the VCN, enabling communication between subnets of VCN 104 and with other endpoints outside the VCN. VCN VR 105 is configured to route traffic between the VNICs within VCN 104 and the gateway associated with VCN 104. VCN VR 105 provides a port for each subnet of VCN 104. For example, VCN VR 105 could provide a port with IP address 10.0.0.1 for subnet-1 and a port with IP address 10.1.0.1 for subnet-2.

[0095] Multiple compute instances can be deployed on each subnet, where compute instances can be virtual machine instances and / or bare metal instances. Compute instances within a subnet can be hosted by one or more hosts within CSPI 101. Compute instances join the subnet via a VNIC associated with the compute instance. For example, as... Figure 1 As shown, compute instance C1 is part of subnet-1 via the VNIC associated with it. Similarly, compute instance C2 is part of subnet-1 via the VNIC associated with it. In a similar manner, multiple compute instances (which can be virtual machine instances or bare metal instances) can be part of subnet-1. Each compute instance is assigned a private overlay IP address and MAC address via its associated VNIC. For example, in... Figure 1In this subnet-1, compute instance C1 has an overlay IP address of 10.0.0.2 and a MAC address of M1, while compute instance C2 has a private overlay IP address of 10.0.0.3 and a MAC address of M2. Each compute instance in subnet-1 (including compute instances C1 and C2) has a default route to the VCN VR 105 using IP address 10.0.0.1, which is the IP address used by the VCN VR 105 for the port in subnet-1.

[0096] Subnet-2 can have multiple compute instances deployed on it, including virtual machine instances and / or bare metal instances. For example, as Figure 1 As shown, compute instances D1 and D2 are part of subnet-2 via the VNIC associated with the respective compute instance. Figure 1 In the embodiment depicted, compute instance D1 has an overlay IP address 10.1.0.2 and a MAC address MM1, while compute instance D2 has a private overlay IP address 10.1.0.3 and a MAC address MM2. Each compute instance in subnet-2 (including compute instances D1 and D2) has a default route to VCN VR 105 using IP address 10.1.0.1, which is the IP address used for the port of VCN VR 105 for subnet-2.

[0097] VCN A 104 may also include one or more load balancers. For example, a load balancer may be provided for a subnet and can be configured to load balance traffic across multiple compute instances on the subnet. A load balancer may also be provided to load balance traffic across subnets in a VCN.

[0098] A specific compute instance deployed on VCN 104 can communicate with a variety of different endpoints. These endpoints can include endpoints hosted by CSPI 200 and endpoints outside of CSPI 200. Endpoints hosted by CSPI 101 can include: endpoints on the same subnet as the specific compute instance (e.g., communication between two compute instances in subnet-1); endpoints on different subnets but within the same VCN (e.g., communication between a compute instance in subnet-1 and a compute instance in subnet-2); endpoints in different VCNs within the same region (e.g., communication between a compute instance in subnet-1 and an endpoint in a VCN within the same region, or communication between a compute instance in subnet-1 and an endpoint in service network 110 within the same region); or endpoints in VCNs in different regions (e.g., communication between a compute instance in subnet-1 and an endpoint in a VCN in a different region). Compute instances in subnets hosted by CSPI 101 can also communicate with endpoints not hosted by CSPI 101 (i.e., outside of CSPI 101). These external endpoints include endpoints in the customer’s on-premises network 116, endpoints in other remote cloud-hosted networks 118, public endpoints 112 that are accessible via public networks such as the Internet, and other endpoints.

[0099] VNICs associated with the source and destination compute instances are used to facilitate communication between compute instances on the same subnet. For example, compute instance C1 in subnet-1 might want to send packets to compute instance C2 in subnet-1. For packets originating from the source compute instance and destined for another compute instance in the same subnet, the packets are first processed by the VNIC associated with the source compute instance. The processing performed by the VNIC associated with the source compute instance may include determining the packet's destination information from the packet header, identifying any policies (e.g., security lists) configured for the VNIC associated with the source compute instance, determining the packet's next hop, performing any packet encapsulation / decapsulation functions as needed, and then forwarding / routing the packet to the next hop to facilitate communication to its intended destination. When the destination compute instance is in the same subnet as the source compute instance, the VNIC associated with the source compute instance is configured to identify the VNIC associated with the destination compute instance and forward packets to that VNIC for processing. The VNIC associated with the destination compute instance then performs the processing and forwards the packets to the destination compute instance.

[0100] For packets to be communicated from compute instances in a subnet to endpoints in different subnets within the same VCN, communication is facilitated by the VNIC associated with the source and destination compute instances, as well as the VCN VR. For example, if Figure 1If compute instance C1 in subnet-1 wants to send a packet to compute instance D1 in subnet-2, the packet is first processed by the VNIC associated with compute instance C1. The VNIC associated with compute instance C1 is configured to route the packet to VCN VR 105 using the default route or port 10.0.0.1 of the VCN VR. VCN VR 105 is configured to route the packet to subnet-2 using port 10.1.0.1. The packet is then received and processed by the VNIC associated with D1, and the VNIC forwards the packet to compute instance D1.

[0101] For packets to be communicated from compute instances within VCN 104 to endpoints outside VCN 104, communication is facilitated by the VNIC associated with the source compute instance, VCN VR 105, and the gateway associated with VCN 104. One or more types of gateways can be associated with VCN 104. A gateway is an interface between a VCN and another endpoint located outside the VCN. A gateway is a Layer 3 / IP concept and enables a VCN to communicate with endpoints outside the VCN. Therefore, a gateway facilitates traffic flow between a VCN and other VCNs or networks. Various different types of gateways can be configured for a VCN to facilitate different types of communication with different types of endpoints. Depending on the gateway, communication can be conducted over a public network (e.g., the Internet) or over a private network. Various communication protocols can be used for these communications.

[0102] For example, compute instance C1 might want to communicate with an endpoint outside of VCN 104. The packet can first be processed by the VNIC associated with the source compute instance C1. The VNIC processing determines that the packet's destination is outside subnet-1 of C1. The VNIC associated with C1 can then forward the packet to VCN VR 105 of VCN 104. VCN VR 105 then processes the packet and, as part of the processing, determines a specific gateway associated with VCN 104 as the packet's next hop based on the packet's destination. VCN VR 105 can then forward the packet to the specifically identified gateway. For example, if the destination is an endpoint within the customer's on-premises network, the packet can be forwarded by VCN VR 105 to the DRG gateway 122 configured for VCN 104. The packet can then be forwarded from the gateway to the next hop to facilitate communication to its final intended destination.

[0103] Various types of gateways can be configured for a VCN. Examples of gateways that can be configured for a VCN are available in [link / details]. Figure 1 It is depicted in the image and described below. An example of a gateway associated with a VCN is also shown below. Figures 16-19The gateways depicted (e.g., those referenced by reference numerals 1634, 1636, 1638, 1734, 1736, 1738, 1834, 1836, 1838, 1934, 1936 and 1938) are described below. Figure 1 As illustrated in the embodiments depicted, DRG 122 can be added to or associated with customer VCN 104 and provides a path for private network traffic communication between customer VCN 104 and another endpoint, which can be customer's on-premises network 116, VCN 108 in a different region of CSPI 101, or another remote cloud network 118 not hosted by CSPI 101. Customer on-premises network 116 can be a customer network or customer data center built using customer resources. Access to customer on-premises network 116 is typically very restricted. For customers with both customer on-premises network 116 and one or more VCNs 104 deployed or hosted in the cloud by CSPI 101, the customer may want their on-premises network 116 and their cloud-based VCN 104 to be able to communicate with each other. This allows customers to build extended hybrid environments encompassing customer VCNs 104 hosted by CSPI 101 and their on-premises network 116. DRG 122 enables such communication. To enable such communication, a communication channel 124 is established, with one endpoint in the customer's local deployment network 116 and the other endpoint in CSPI 101 connected to the customer's VCN 104. Communication channel 124 can be over a public or private communication network, such as the Internet. Various communication protocols can be used, such as IPsec VPN technology over a public communication network (such as the Internet), Oracle's FastConnect technology using a private network instead of a public network, and so on. The device or equipment in the customer's local deployment network 116 forming one endpoint of communication channel 124 is referred to as customer premises equipment (CPE), such as... Figure 1 The CPE 126 is depicted in the diagram. On the CSPI 101 side, the endpoint can be a host executing DRG 122.

[0104] In some embodiments, a remote peering connection (RPC) can be added to the DRG, which allows a customer to peer one VCN with another VCN in a different region. Using such an RPC, customer VCN 104 can use DRG 122 to connect to VCN 108 in another region. DRG 122 can also be used to communicate with other remote cloud networks 118 that are not hosted by CSPI 101, such as Microsoft Azure Cloud, Amazon AWS Cloud, etc.

[0105] like Figure 1As shown, an Internet Gateway (IGW) 120 can be configured for a client VCN 104, enabling compute instances on VCN 104 to communicate with a public endpoint 112 accessible via a public network such as the Internet. The IGW 120 is a gateway connecting the VCN to a public network such as the Internet. The IGW 120 enables public subnets within the VCN 104 (where resources within the public subnet have publicly overriding IP addresses) to directly access the public endpoint 112 on the public network 114, such as the Internet. Using the IGW 120, connections can originate from subnets within VCN 104 or from the Internet.

[0106] Network Address Translation (NAT) gateway 128 can be configured for a customer's VCN 104 and enable cloud resources within the customer's VCN that do not have private public overlay IP addresses to access the Internet, and NAT gateway 128 does so without exposing those resources to direct inbound Internet connections (e.g., L4-L7 connections). This allows private subnets within the VCN (such as private subnet-1 in VCN 104) to have private access to public endpoints on the Internet. In the NAT gateway, connections can only be initiated from private subnets to the public Internet, and not from the Internet to private subnets.

[0107] In some embodiments, a Service Gateway (SGW) 126 may be configured for a client VCN 104 and provide a path for private network traffic between VCN 104 and supported service endpoints in service network 110. In some embodiments, service network 110 may be provided by a CSP and may offer a variety of services. An example of such a service network is Oracle's service network, which provides a variety of services that can be used by clients. For example, compute instances (e.g., database systems) in a private subnet of client VCN 104 may back up data to service endpoints (e.g., object storage) without requiring a public IP address or access to the Internet. In some embodiments, a VCN may have only one SGW, and connections may only originate from subnets within the VCN and not from service network 110. If a VCN peers to another VCN, resources in the other VCN typically cannot access the SGW. Resources in an on-premises network connected to a VCN using FastConnect or VPN Connect may also use the service gateway configured for that VCN.

[0108] In some implementations, SGW 126 uses the concept of a Service Classless Inter-Domain Routing (CIDR) label, which is a string representing all regional ranges of public IP addresses for the service or group of services of interest. Customers use service CIDR labels to control traffic to services when configuring SGW and related routing rules. Customers can optionally use service CIDR labels when configuring security rules to avoid having to adjust them if the public IP addresses of services change in the future.

[0109] Local peering gateway (LPG) 132 is a gateway that can be added to customer VCN 104 and enable VCN 104 to peer with another VCN in the same area. Peering means that VCNs communicate using private IP addresses without traffic traversing public networks such as the Internet or routing traffic through the customer's locally deployed network 116. In a preferred embodiment, a VCN has a separate LPG for each peer it establishes. Local peering, or VCN peering, is a common practice for establishing network connectivity between different applications or infrastructure management functions.

[0110] Service providers (such as service providers in service network 110) can use different access models to provide access to services. Under the public access model, services can be exposed as public endpoints, publicly accessible by compute instances in a customer's VCN via public networks such as the Internet and / or privately accessible via SGW 126. Under a specific private access model, services can be accessed as private IP endpoints within a private subnet of the customer's VCN. This is called private endpoint (PE) access and allows service providers to expose their services as instances within the customer's private network. A private endpoint resource represents a service within the customer's VCN. Each PE is represented as a VNIC (referred to as a PE-VNIC, with one or more private IPs) within a subnet selected by the customer within the customer's VCN. Thus, a PE provides a method of presenting a service within a private customer VCN subnet using a VNIC. Because the endpoint is exposed as a VNIC, all characteristics associated with the VNIC (such as routing rules, security lists, etc.) are now available to the PE VNIC.

[0111] Service providers can register their services to make them accessible via a PE (Private Provider). Providers can associate policies with services, which restricts the visibility of services to customer leases. Providers can register multiple services under a single Virtual IP address (VIP), especially for multi-tenant services. Multiple such private endpoints (across multiple VCNs) can exist representing the same service.

[0112] Compute instances in a private subnet can then access the service using the private IP address or service DNS name of the PE VNIC. Compute instances in a customer VCN can access the service by sending traffic to the private IP address of the PE in the customer VCN. A Private Access Gateway (PAGW) 130 is a gateway resource that can be attached to a service provider VCN (e.g., a VCN in service network 110), acting as the ingress / egress point for all traffic originating from / to the private endpoint of the customer subnet. PAGW 130 allows providers to scale the number of PE connections without utilizing their internal IP address resources. A provider only needs to configure one PAGW for any number of services registered in a single VCN. A provider can represent a service as a private endpoint in multiple VCNs of one or more customers. From the customer's perspective, the PE VNIC is not an instance attached to the customer, but rather appears to be attached to the service the customer wishes to interact with. Traffic destined for the private endpoint is routed to the service via PAGW 130. These are referred to as customer-to-service private connections (C2S connections).

[0113] The PE concept can also be used to extend private access to services to a customer's on-premises network and data center by allowing traffic to flow through FastConnect / IPsec links and private endpoints within the customer's VCN. Private access to services can also be extended to the customer's peering VCN by allowing traffic to flow between LPG 132 and the PE within the customer's VCN.

[0114] Customers can control routing within a VCN at the subnet level, allowing them to specify which subnets within a customer's VCN (such as VCN104) use each gateway. The VCN's routing table determines whether traffic is allowed to leave the VCN via a specific gateway. For example, in a given instance, the routing table for the public subnet within customer VCN 104 might allow non-local traffic to be sent via IGW 120. The routing table for the private subnet within the same customer VCN 104 might allow traffic destined for CSP services to be sent via SGW 126. All remaining traffic can be sent via NAT gateway 128. The routing table only controls traffic leaving the VCN.

[0115] Security lists associated with a VCN are used to control traffic entering the VCN via a gateway through inbound connections. All resources within a subnet use the same routing tables and security lists. Security lists can be used to control specific types of traffic allowed to enter or leave instances within a VCN subnet. Security list rules can include inbound and outbound rules. For example, inbound rules can specify allowed ranges of source addresses, while outbound rules can specify allowed ranges of destination addresses. Security rules can specify specific protocols (e.g., TCP, ICMP), specific ports (e.g., 22 for SSH, 3389 for Windows RDP), etc. In some implementations, the instance's operating system can enforce its own firewall rules consistent with the security list rules. Rules can be stateful (e.g., connections are tracked and responses are automatically allowed in the absence of explicit security list rules for response traffic) or stateless.

[0116] Access from a customer's VCN (i.e., through resources or compute instances deployed on VCN 104) can be categorized as public access, private access, or dedicated access. Public access refers to an access model where a public IP address or NAT is used to access a public endpoint. Private access enables customer workloads (e.g., resources in a private subnet) within VCN 104 with private IP addresses to access services without traversing a public network such as the Internet. In some embodiments, CSPI 101 enables customer VCN workloads with private IP addresses to access the service (its public service endpoint) using a service gateway. Thus, the service gateway provides a private access model by establishing a virtual link between the customer's VCN and the public endpoint of the service residing outside the customer's private network.

[0117] Additionally, CSPI can use technologies such as FastConnect public peering to provide dedicated public access, where customer-locally deployed instances can use FastConnect connections to access one or more services within the customer's VCN without traversing public networks such as the Internet. CSPI can also use FastConnect private peering to provide dedicated private access, where customer-locally deployed instances with private IP addresses can use FastConnect connections to access the customer's VCN workloads. FastConnect is an alternative to using the public Internet to connect a customer's local network to CSPI and its services for network connectivity. Compared to Internet-based connections, FastConnect provides a simple, resilient, and cost-effective way to create dedicated and private connections with higher bandwidth options and a more reliable and consistent networking experience.

[0118] Figure 1The description attached above illustrates the various virtualization components in the example virtual network. As mentioned above, the virtual network is built on top of the underlying physical or base network. Figure 2 A simplified architecture diagram is depicted within the physical network of a CSPI 200, according to certain embodiments, illustrating the underlying physical components that provide the underlying physical network for virtual networks. As shown, CSPI 200 provides a distributed environment comprising components and resources (e.g., compute, memory, and networking resources) provided by a CSP. These components and resources are used to provide cloud services (e.g., IaaS services) to subscribing customers (i.e., customers who have subscribed to one or more services provided by the CSP). Based on the services subscribed to by the customer, a subset of the resources of CSPI 200 (e.g., compute, memory, and networking resources) is provisioned to the customer. The customer can then use the physical compute, memory, and networking resources provided by CSPI 200 to build their own customizable and private cloud-based (i.e., CSPI-hosted) virtual networks. As previously indicated, these customer networks are referred to as VCNs. Customers can deploy one or more customer resources, such as compute instances, on these customer VCNs. Compute instances can take the form of virtual machines, bare metal instances, etc. CSPI 200 provides infrastructure and a complementary set of cloud services that enable customers to build and run a wide range of applications and services in a highly available managed environment.

[0119] exist Figure 2 In the example embodiment depicted, the physical components of CSPI 200 include one or more physical hosts or physical servers (e.g., 202, 206, 208), network virtualization devices (NVDs) (e.g., 210, 212), ToR switches (e.g., 214, 216), and a physical network (e.g., 218), as well as switches within physical network 218. The physical hosts or servers can host and execute various compute instances that join one or more subnets of the VCN. Compute instances can include virtual machine instances and bare metal instances. For example, Figure 1 The various computational examples described in the text can be derived from... Figure 2 The physical host described in the text refers to virtual machine compute instances in a VCN that can be executed by a single host or multiple different hosts. Physical hosts can also host virtual machines, container-based hosts, or functionalities, etc. Figure 1 The VNIC and VCN VR described in the text can be generated by Figure 2 The NVD execution described in the text. Figure 1 The gateway described in the text can be... Figure 2 The host and / or NVD execution described in the document.

[0120] A host or server can execute a hypervisor (also known as a virtual machine monitor or VMM) that creates and enables virtualized environments on the host. Virtualization or virtualized environments facilitate cloud-based computing. One or more compute instances can be created, executed, and managed on the host by a hypervisor on the host. The hypervisor on the host enables groups of physical computing resources (e.g., compute, storage, and networking resources) of the host to be shared among various compute instances executed by the host.

[0121] For example, such as Figure 2 As depicted, hosts 202 and 208 execute hypervisors 260 and 266, respectively. These hypervisors can be implemented using software, firmware, or hardware, or a combination thereof. Typically, a hypervisor is a process or software layer that sits above the host operating system (OS), which in turn executes on the host's hardware processor. Hypervisors provide a virtualization environment by enabling the sharing of the host's physical computing resources (e.g., processing resources such as processors / cores, memory resources, networking resources) among various virtual machine computing instances executed by the host. For example, in... Figure 2 In this configuration, hypervisor 260 can reside on the operating system of host 202 and enable the computing resources (e.g., processing, memory, and networking resources) of host 202 to be shared among computing instances (e.g., virtual machines) executed by host 202. Virtual machines can have their own operating systems (called guest operating systems), which can be the same as or different from the host's operating system. The operating system of a virtual machine executed by the host can be the same as or different from the operating system of another virtual machine executed by the same host. Therefore, the hypervisor enables multiple operating systems to run simultaneously, sharing the same set of computing resources of the host. Figure 2 The hosts described in the text may have the same or different types of management programs.

[0122] A compute instance can be a virtual machine instance or a bare metal instance. Figure 2 In the diagram, compute instance 268 on host 202 and compute instance 274 on host 208 are examples of virtual machine instances. Host 206 is an example of a bare metal instance provided to a customer.

[0123] In some instances, an entire host can be provisioned to a single customer, and all compute instances (virtual machines or bare metal instances) hosted by that host belong to the same customer. In other instances, the host can be shared among multiple customers (i.e., multiple tenants). In such multi-tenant scenarios, the host can host virtual machine compute instances belonging to different customers. These compute instances can be members of different VCNs for different customers. In some embodiments, bare metal compute instances are hosted by bare metal servers without a hypervisor. When a bare metal compute instance is provisioned, a single customer or tenant maintains control over the physical CPU, memory, and network interfaces of the host hosting the bare metal instance, and the host is not shared with other customers or tenants.

[0124] As previously described, each compute instance, as part of a VCN, is associated with a VNIC that enables the compute instance to become a member of a subnet of the VCN. The VNIC associated with a compute instance facilitates communication to and from the compute instance for packets or frames. A VNIC is associated with a compute instance when it is created. In some embodiments, for compute instances executed by a host, the VNIC associated with that compute instance is executed by an NVD connected to the host. For example, in Figure 2 In this example, host 202 executes a virtual machine compute instance 268 associated with VNIC 276, and VNIC 276 is executed by NVD 210 connected to host 202. As another example, a bare metal instance 272 hosted by host 206 is associated with VNIC 280 executed by NVD 212 connected to host 206. As yet another example, VNIC 284 is associated with compute instance 274 executed by host 208, and VNIC 284 is executed by NVD 212 connected to host 208.

[0125] For compute instances hosted by a host, NVDs connected to that host also execute VCN VR corresponding to the VCN where the compute instance is a member. For example, in Figure 2 In the embodiment depicted, NVD 210 executes VCN VR 277 corresponding to the VCN of compute instance 268, which is a member of NVD 212. NVD 212 may also execute one or more VCN VR 283 corresponding to the VCNs of compute instances hosted by hosts 206 and 208.

[0126] A host may include one or more network interface cards (NICs) that enable the host to connect to other devices. The NIC on the host may provide one or more ports (or interfaces) that allow the host to communicatively connect to another device. For example, the host may use one or more ports (or interfaces) provided on the host and the NVD to connect to the NVD. The host may also connect to other devices, such as another host.

[0127] For example, in Figure 2 In this configuration, host 202 connects to NVD 210 via link 220, which extends between port 234 provided by NIC 232 of host 202 and port 236 of NVD 210. Host 206 connects to NVD 212 via link 224, which extends between port 246 provided by NIC 244 of host 206 and port 248 of NVD 212. Host 208 connects to NVD 212 via link 226, which extends between port 252 provided by NIC 250 of host 208 and port 254 of NVD 212.

[0128] The NVD is then connected to the ToR switch via a communication link, which is connected to physical network 218 (also known as a switch architecture). In some embodiments, the links between the host and the NVD, and between the NVD and the ToR switch, are Ethernet links. For example, in Figure 2 In this configuration, NVDs 210 and 212 are connected to ToR switches 214 and 216 via links 228 and 230, respectively. In some embodiments, links 220, 224, 226, 228, and 230 are Ethernet links. The collection of hosts and NVDs connected to the ToR is sometimes referred to as a rack.

[0129] Physical network 218 provides a communication structure that enables ToR switches to communicate with each other. Physical network 218 can be a multi-level network. In some implementations, physical network 218 is a multi-level Clos network of switches, where ToR switches 214 and 216 represent leaf-level nodes of the multi-level and multi-node physical switching network 218. Different Clos network configurations are possible, including but not limited to 2-level networks, 3-level networks, 4-level networks, 5-level networks, and general "n"-level networks. Examples of Clos networks are provided in... Figure 5 It is depicted in the middle and described below.

[0130] Various connection configurations between the host and the NVD, such as one-to-one, many-to-one, and one-to-many configurations, are possible. In a one-to-one configuration implementation, each host connects to its own individual NVD. For example, in... Figure 2 In this configuration, host 202 connects to NVD 210 via NIC 232. In a many-to-one configuration, multiple hosts connect to a single NVD. For example, in... Figure 2 In this configuration, hosts 206 and 208 are connected to the same NVD 212 via NICs 244 and 250, respectively.

[0131] In a one-to-many configuration, one host connects to multiple NVDs. Figure 3An example within the CSPI 300 is shown, where the host is connected to multiple NVDs. (Example follows) Figure 3 As shown, host 302 includes a network interface card (NIC) 304, which includes multiple ports 306 and 308. Host 300 is connected to a first NVD 310 via port 306 and link 320, and to a second NVD 312 via port 308 and link 322. Ports 306 and 308 may be Ethernet ports, and links 320 and 322 between host 302 and NVDs 310 and 312 may be Ethernet links. NVD 310 is then connected to a first ToR switch 314, and NVD 312 is connected to a second ToR switch 316. Links between NVDs 310 and 312 and between ToR switches 314 and 316 may be Ethernet links. ToR switches 314 and 316 represent tier 0 switching devices in a multi-level physical network 318.

[0132] Figure 3 The arrangement depicted provides two separate physical network paths to and from physical switch network 318 to host 302: a first path through ToR switch 314 to NVD 310 to host 302, and a second path through ToR switch 316 to NVD 312 to host 302. These separate paths provide enhanced availability (referred to as high availability) for host 302. If one of the paths (e.g., a link failure in one of the paths) or devices (e.g., a particular NVD not functioning) has a problem, the other path can be used for communication to / from host 302.

[0133] exist Figure 3 In the configuration depicted, the host uses two different ports provided by the host's NIC to connect to two different NVDs. In other embodiments, the host may include multiple NICs enabling host-to-multiple NVD connectivity.

[0134] Return to reference Figure 2 An NVD is a physical device or component that performs one or more network and / or storage virtualization functions. An NVD can be any device with one or more processing units (e.g., CPU, Network Processing Unit (NPU), FPGA, packet processing pipeline, etc.), memory including cache, and ports. Various virtualization functions can be executed by software / firmware performed by one or more processing units of the NVD.

[0135] NVDs can be implemented in various different forms. For example, in some embodiments, an NVD is implemented as an interface card called a smartNIC or intelligent NIC with an embedded processor. A smartNIC is a device separate from the NIC on the host machine. Figure 2In this context, NVD 210 and 212 can be implemented as smartNICs connected to host 202 and hosts 206 and 208, respectively.

[0136] However, smartNIC is just one example of an NVD implementation. Various other implementations are possible. For example, in some other implementations, the NVD, or one or more functions performed by the NVD, may be incorporated into or performed by one or more hosts, one or more ToR switches, and other components of the CSPI 200. For instance, the NVD may be embodied in a host, where the functions performed by the NVD are performed by the host. As another example, the NVD may be part of a ToR switch, or the ToR switch may be configured to perform functions performed by the NVD, enabling the ToR switch to perform various complex packet transformations for public clouds. A ToR that performs NVD functions is sometimes referred to as a smart ToR. In yet another implementation, where virtual machine (VM) instances, rather than bare metal (BM) instances, are provided to customers, the functions performed by the NVD may be implemented within the host's hypervisor. In some other implementations, some of the functions of the NVD may be offloaded to a centralized service running on a series of hosts.

[0137] In some embodiments, such as when implemented as Figure 2 When a smartNIC is shown, an NVD can include multiple physical ports that enable the NVD to connect to one or more hosts and one or more ToR switches. Ports on an NVD can be classified as host-facing ports (also known as "south ports") or network-facing or ToR-facing ports (also known as "north ports"). Host-facing ports on an NVD are those used to connect the NVD to a host. Figure 2 Examples of host-facing ports include port 236 on the NVD 210 and ports 248 and 254 on the NVD 212. Network-facing ports on the NVD are used to connect the NVD to the ToR switch. Figure 2 Examples of network-facing ports include port 256 on the NVD 210 and port 258 on the NVD 212. Figure 2 As shown, NVD 210 is connected to ToR switch 214 via link 228, which extends from port 256 of NVD 210 to ToR switch 214. Similarly, NVD 212 is connected to ToR switch 216 via link 230, which extends from port 258 of NVD 212 to ToR switch 216.

[0138] The NVD receives packets and frames from the host (e.g., packets and frames generated by compute instances hosted on the host) via its host-facing port, and after performing the necessary packet processing, can forward the packets and frames to the ToR switch via the NVD's network-facing port. The NVD can also receive packets and frames from the ToR switch via its network-facing port, and after performing the necessary packet processing, can forward the packets and frames to the host via the NVD's host-facing port.

[0139] In some embodiments, there can be multiple ports and associated links between the NVD and the ToR switch. These ports and links can be aggregated to form a group of link aggregators (called a LAG) of multiple ports or links. Link aggregation allows multiple physical links between two endpoints (e.g., between the NVD and the ToR switch) to be treated as a single logical link. All physical links in a given LAG can operate at the same speed in full-duplex mode. LAGs help improve the bandwidth and reliability of the connection between two endpoints. If one physical link in the LAG fails, traffic is dynamically and transparently reassigned to one of the other physical links in the LAG. Aggregated physical links deliver higher bandwidth than each individual link. Multiple ports associated with an LAG are treated as a single logical port. Traffic can be load balanced across multiple physical links in the LAG. One or more LAGs can be configured between two endpoints. The two endpoints can be located between the NVD and the ToR switch, between a host and the NVD, etc.

[0140] The NVD implements or performs network virtualization functions. These functions are performed by the software / firmware executed by the NVD. Examples of network virtualization functions include, but are not limited to: packet encapsulation and decapsulation functions; functions for creating VCN networks; functions for implementing network policies such as VCN security lists (firewalls); functions for facilitating the routing and forwarding of packets to and from compute instances in the VCN; and so on. In some embodiments, upon receiving a packet, the NVD is configured to perform a packet processing pipeline to process the packet and determine how the packet is forwarded or routed. As part of this packet processing pipeline, the NVD may perform one or more virtual functions associated with the overlay network, such as performing VNICs associated with compute instances in the VCN, performing virtual routers (VRs) associated with the VCN, packet encapsulation and decapsulation to facilitate forwarding or routing in the virtual network, execution of certain gateways (e.g., local peer gateways), security lists, network security groups, Network Address Translation (NAT) functions (e.g., public IP to private IP translation on a per-host basis), throttling functions, and other functions.

[0141] In some embodiments, the packet processing data path in NVD may include multiple packet pipelines, each consisting of a series of packet transformation stages. In some implementations, upon receiving a packet, it is parsed and classified into a single pipeline. The packet is then processed linearly, stage by stage, until it is dropped or sent out through the NVD's interface. These stages provide basic functional packet processing building blocks (e.g., header verification, throttling, inserting new Layer-2 headers, implementing L4 firewalls, VCN encapsulation / decapsulation, etc.), allowing new pipelines to be constructed by combining existing stages, and new functionality to be added by creating new stages and inserting them into existing pipelines.

[0142] NVD can perform both control plane and data plane functions corresponding to the control plane and data plane of VCN. Examples of VCN CP are also available. Figures 16-19 (See figures 1616, 1716, 1816 and 1916) depicted and described below. An example of the VCN data plane is shown in... Figures 16-19 (See figures 1618, 1718, 1818, and 1918) as depicted and described below. Control plane functions include those for configuring how control data is forwarded on the network (e.g., establishing routes and routing tables, configuring VNICs, etc.). In some embodiments, a VCN CP is provided, which centrally computes all overlay mappings to the base layer and publishes them to the NVD and to virtual network edge devices such as various gateways (e.g., DRGs, SGWs, IGWs, etc.). Firewall rules can also be published using the same mechanism. In some embodiments, the NVD only acquires mappings associated with that NVD. Data plane functions include those for actual routing / forwarding of packets based on the configuration established using the control plane. The VCN data plane is implemented by encapsulating client network packets before they traverse the base layer network. Encapsulation / decapsulation functionality is implemented on the NVD. In some embodiments, the NVD is configured to intercept all network packets entering and leaving the host and perform network virtualization functions.

[0143] As indicated above, the NVD performs various virtualization functions, including VNICs and VCN VR. The NVD can execute VNICs associated with compute instances hosted by one or more hosts connected to the VNIC. For example, as... Figure 2As depicted, NVD 210 performs the functions of VNIC 276 associated with compute instance 268 hosted by host 202 connected to NVD 210. As another example, NVD 212 performs VNIC 280 associated with bare-metal compute instance 272 hosted by host 206, and VNIC 284 associated with compute instance 274 hosted by host 208. Hosts can host compute instances belonging to different VCNs belonging to different clients, and NVDs connected to hosts can perform VNICs corresponding to the compute instances (i.e., perform functions associated with the VNICs).

[0144] NVD also executes a VCN virtual router corresponding to the VCN of the compute instance. For example, in Figure 2 In the embodiments depicted, NVD 210 executes VCN VR 277 corresponding to the VCN to which compute instance 268 belongs. NVD 212 executes one or more VCN VR 283 corresponding to one or more VCNs to which the compute instances hosted by hosts 206 and 208 belong. In some embodiments, the VCN VR corresponding to that VCN is executed by all NVDs connected to hosts hosting at least one compute instance belonging to that VCN. If a host hosts compute instances belonging to different VCNs, the NVDs connected to that host can execute VCN VR corresponding to those different VCNs.

[0145] In addition to VNIC and VCN VR, NVD can execute various software (e.g., daemons) and includes one or more hardware components that facilitate the various network virtualization functions performed by NVD. For simplicity, these individual components are grouped together as... Figure 2The “packet processing component” is described in the diagram. For example, NVD 210 includes packet processing component 286, and NVD 212 includes packet processing component 288. For instance, the packet processing component of an NVD may include a packet processor configured to interact with the NVD’s ports and hardware interfaces to monitor all packets received by and used by the NVD and to store network information. Network information may include, for example, network flow information identifying different network flows handled by the NVD and per-flow information (e.g., per-flow statistics). In some embodiments, network flow information may be stored on a per-VNIC basis. The packet processor may perform packet-by-packet manipulation and implement stateful NAT and L4 firewall (FW). As another example, the packet processing component may include a replication agent configured to replicate information stored by the NVD to one or more different replication target stores. As yet another example, the packet processing component may include a logging agent configured to perform logging functions of the NVD. The packet processing component may also include software for monitoring the performance and health of the NVD and may also monitor the status and health of other components connected to the NVD.

[0146] Figure 1 The components of an example virtual or overlay network are shown, including a VCN, subnets within the VCN, compute instances deployed on the subnets, VNICs associated with the compute instances, the VCN's VR, and a set of gateways configured for the VCN. Figure 1 The overlay component described in the text can be made by Figure 2 One or more physical components described in the diagram are executed or managed. For example, a compute instance in a VCN can be executed or managed by... Figure 2 The VNIC described herein is executed or hosted by one or more hosts. For compute instances hosted by a host, the VNIC associated with that compute instance is typically executed by an NVD connected to that host (i.e., VNIC functionality is provided by an NVD connected to that host). The VCN VR functionality of a VCN is executed by all NVDs connected to the host that hosts or executes a compute instance that is part of that VCN. Gateways associated with a VCN can be executed by one or more different types of NVDs. For example, some gateways can be executed by smartNICs, while others can be executed by one or more hosts or other implementations of NVDs.

[0147] As described above, compute instances in a client VCN can communicate with a variety of different endpoints, which can be within the same subnet as the source compute instance, in a different subnet but within the same VCN as the source compute instance, or with endpoints outside the source compute instance's VCN. These communications are facilitated using the VNIC, VCN VR, and gateway associated with the VCN, all of which are linked to the compute instance.

[0148] For communication between two compute instances on the same subnet within a VCN, this communication is facilitated using VNICs associated with the source and destination compute instances. The source and destination compute instances can be hosted by the same host or different hosts. Packets originating from the source compute instance can be forwarded from the host hosting the source compute instance to an NVD connected to that host. On the NVD, packets are processed using a packet processing pipeline that may include the execution of the VNIC associated with the source compute instance. Because the destination endpoint of the packet is within the same subnet, the execution of the VNIC associated with the source compute instance results in the packet being forwarded to the NVD executing the VNIC associated with the destination compute instance, which then processes the packet and forwards it to the destination compute instance. The VNICs associated with the source and destination compute instances can execute on the same NVD (e.g., when both the source and destination compute instances are hosted by the same host) or on different NVDs (e.g., when the source and destination compute instances are hosted by different hosts connected to different NVDs). The VNIC can use the routing / forwarding tables stored in the NVD to determine the next hop for the packet.

[0149] For packets destined for endpoints in different subnets within the same VCN, the packet originating from the source compute instance is forwarded from the host hosting the source compute instance to the NVD connected to that host. On the NVD, a packet processing pipeline is used to process the packet, which may include the execution of one or more VNICs and a VR associated with the VCN. For example, as part of the packet processing pipeline, the NVD executes or invokes a function corresponding to the VNIC associated with the source compute instance (also known as executing the VNIC). The function executed by the VNIC may include viewing the VLAN tag on the packet. Since the packet's destination is outside the subnet, the VCN VR function is then invoked and executed by the NVD. The VCN VR then routes the packet to the NVD executing the VNIC associated with the destination compute instance. The VNIC associated with the destination compute instance then processes the packet and forwards it to the destination compute instance. The VNICs associated with the source and destination compute instances may execute on the same NVD (e.g., when both the source and destination compute instances are hosted by the same host) or on different NVDs (e.g., when the source and destination compute instances are hosted by different hosts connected to different NVDs).

[0150] If the packet's destination is outside the source compute instance's VCN, the packet originating from the source compute instance is passed from the host hosting the source compute instance to the NVD connected to that host. The NVD executes the VNIC associated with the source compute instance. Since the packet's destination endpoint is outside the VCN, the packet is then processed by the VCN VR of that VCN. The NVD invokes VCN VR functionality, which allows the packet to be forwarded to the NVD executing the appropriate gateway associated with the VCN. For example, if the destination is an endpoint within the customer's on-premises network, the packet may be forwarded by the VCN VR to the NVD executing the DRG gateway configured for the VCN. The VCN VR may be executed on the same NVD as the NVD executing the VNIC associated with the source compute instance or by a different NVD. The gateway may be executed by an NVD, which can be a smartNIC, a host, or another NVD implementation. The packet is then processed by the gateway and forwarded to the next hop that facilitates communication to its intended destination endpoint. For example, in Figure 2 In the embodiment depicted, packets originating from compute instance 268 can be transmitted from host 202 to NVD 210 via link 220 (using NIC 232). On NVD 210, VNIC 276 is invoked because it is the VNIC associated with the source compute instance 268. VNIC 276 is configured to examine the encapsulation information in the packets and determine the next hop for forwarding the packets, with the aim of facilitating communication of the packets to their intended destination endpoint, and then forward the packets to the determined next hop.

[0151] Compute instances deployed on a VCN can communicate with a variety of different endpoints. These endpoints can include endpoints hosted by CSPI 200 and endpoints outside of CSPI 200. Endpoints hosted by CSPI 200 can include instances within the same VCN or other VCNs, which can be the customer's VCN or a VCN not belonging to the customer. Communication between endpoints hosted by CSPI 200 can be performed via physical network 218. Compute instances can also communicate with endpoints not hosted by CSPI 200 or outside of CSPI 200. Examples of these endpoints include endpoints within the customer's on-premises deployment network or data center, or public endpoints accessible via public networks such as the Internet. Communication with endpoints outside of CSPI 200 can use various communication protocols over public networks (e.g., the Internet). Figure 2 (not shown in the image) or private network ( Figure 2 (Not shown in the image) to execute.

[0152] Figure 2 The architecture of the CSPI 200 depicted herein is merely illustrative and not intended to be limiting. Variations, substitutions, and modifications are possible in alternative embodiments. For example, in some implementations, the CSPI 200 may have more advanced features than... Figure 2 The systems or components shown may include more or fewer systems or components, and may combine two or more systems, or may have different system configurations or arrangements. Figure 2 The systems, subsystems, and other components described herein are implemented using hardware or a combination thereof, with software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective systems. The software may be stored on a non-transitory storage medium (e.g., a memory device).

[0153] Figure 4 The diagram describes connectivity between multi-tenant hosts and NVDs, according to certain embodiments. Figure 4 As depicted, host 402 executes a hypervisor 404 that provides a virtualization environment. Host 402 executes two virtual machine instances, VM1 406 belonging to customer / tenant #1 and VM2 408 belonging to customer / tenant #2. Host 402 includes a physical NIC 410 connected to NVD 412 via link 414. Each compute instance in the compute instances is attached to a VNIC executed by NVD 412. Figure 4 In one embodiment, VM1 406 is attached to VNIC-VM1 420 and VM2 408 is attached to VNIC-VM2 422.

[0154] like Figure 4 As shown, NIC 410 includes two logical NICs, logical NIC A 416 and logical NIC B 418. Each virtual machine is attached to its own logical NIC and configured to work with its own logical NIC. For example, VM1 406 is attached to logical NIC A 416, and VM2 408 is attached to logical NIC B 418. Although host 402 includes only one physical NIC 410 shared by multiple tenants, each tenant's virtual machines believe they have their own host and NIC due to the logical NICs.

[0155] In some embodiments, each logical NIC is assigned its own VLAN ID. Thus, a specific VLAN ID is assigned to logical NIC A 416 for tenant #1, and a separate VLAN ID is assigned to logical NIC B 418 for tenant #2. When a packet is delivered from VM1 406, the tag assigned to tenant #1 is attached to the packet by the hypervisor, and the packet is then delivered from host 402 to NVD 412 via link 414. Similarly, when a packet is delivered from VM2 408, the tag assigned to tenant #2 is attached to the packet by the hypervisor, and the packet is then delivered from host 402 to NVD 412 via link 414. Therefore, the packet 424 delivered from host 402 to NVD 412 has an associated tag 426 that identifies the specific tenant and associated VM. On the NVD, for a packet 424 received from host 402, the tag 426 associated with the packet is used to determine whether the packet should be processed by VNIC-VM1 420 or VNIC-VM2 422. The packet is then processed by the corresponding VNIC. Figure 4 The configuration described in the document allows each tenant's compute instance to believe that it owns its own host and NIC. Figure 4 The settings described in [the document] provide I / O virtualization for supporting multi-tenancy.

[0156] Figure 5 A simplified block diagram of a physical network 500 according to certain embodiments is depicted. Figure 5 The embodiments depicted are structured as Clos networks. A Clos network is a specific type of network topology designed to provide connectivity redundancy while maintaining high distributed bandwidth and maximum resource utilization. A Clos network is a non-blocking, multi-stage or multi-level switching network, wherein the number of stages or levels can be two, three, four, five, etc. Figure 5 The embodiment depicted is a three-tier network including tiers 1, 2, and 3. The ToR switch 504 represents a tier-0 switch in the Clos network. One or more NVDs are connected to the ToR switch. The tier-0 switch is also referred to as the edge device of the physical network. The tier-0 switch connects to a tier-1 switch, also referred to as a leaf switch layer. Figure 5In the embodiment depicted, a set of n Layer-0 ToR switches is connected to a set of n Layer-1 switches, forming a pod. Each Layer-0 switch in a pod is interconnected with all Layer-1 switches in that pod, but there is no switch connectivity between pods. In some implementations, two pods are referred to as blocks. Each block is served by or connected to a set of n Layer-2 switches (sometimes called backbone switches). Several blocks may exist in the physical network topology. The Layer-2 switches are then connected to n Layer-3 switches (sometimes called super backbone switches). Packet communication on the physical network 500 is typically performed using one or more Layer-3 communication protocols. Typically, all layers of the physical network, except the TOR layer, are n-way redundant, allowing for high availability. Policies can be specified for pods and blocks to control the visibility of switches to each other in the physical network, enabling scaling of the physical network.

[0157] A key characteristic of Clos networks is that the maximum number of hops from one Layer-0 switch to another Layer-0 switch (or from an NVD connected to a Layer-0 switch to another NVD connected to a Layer-0 switch) is fixed. For example, in a Layer 3 Clos network, a maximum of seven hops are required for a packet to travel from one NVD to another, where the source and destination NVDs are connected to the leaf layers of the Clos network. Similarly, in a Layer 4 Clos network, a maximum of nine hops are required for a packet to travel from one NVD to another, where the source and destination NVDs are connected to the leaf layers of the Clos network. Therefore, the Clos network architecture maintains consistent latency throughout the network, which is important for communication within and between data centers. Clos topologies are horizontally scalable and cost-effective. Network bandwidth / throughput can be easily increased by adding more switches at each layer (e.g., more leaf and backbone switches) and by increasing the number of links between switches at adjacent layers.

[0158] In some embodiments, each resource within the CSPI is assigned a unique identifier called a Cloud Identifier (CID). This identifier is included as part of the resource's information and can be used to manage the resource, for example, via a console or API. An example syntax for a CID is:

[0159] ocid1.<RESOURCE TYPE> . <realm>[REGION][FUTURE USE].<UNIQUE ID>

[0160] in,

[0161] ocid1: A literal string indicating the CID version;

[0162] Resource type: The type of resource (e.g., instance, volume, VCN, subnet, user, group, etc.).

[0163] Realm: The domain in which the resource resides (example values ​​"c1" for the commercial domain, "c2" for the government cloud domain, or "c3" for the federal government cloud domain, etc. Each domain can have its own domain name).

[0164] Region: The region where the resource is located (this part can be empty if the region is not applicable to the resource).

[0165] Future use: (reserved for future use); and.

[0166] Unique ID: The unique part of the ID (the format may vary depending on the type of resource or service).

[0167] Multi-cloud

[0168] Figure 6 A simplified high-level diagram 600 depicts a distributed environment comprising multiple cloud environments provided by different CSPs. (See diagram 600.) Figure 6 As described herein, various cloud environments (also referred to as "clouds") can be provided by different CSPs. Each cloud environment or cloud provides one or more cloud services, which can be subscribed to by one or more customers of the corresponding CSP. The cloud service group provided by a cloud environment offered by a CSP can include one or more different types of cloud services, including but not limited to SaaS services, IaaS services, PaaS services, Database as a Service (DBaaS) services, etc. Examples of cloud environments provided by various CSPs include OCI, Azure, Google Cloud, AWS, etc. The cloud services provided by a particular cloud environment may differ from the cloud service group provided by another cloud environment.

[0169] In a typical cloud environment, a Cloud Service Provider (CSP) provides a Cloud Service Provider Interface (CSPI) to its customers, offering a group of cloud services provided by that cloud environment. The CSPI provided by the CSP can include various types of hardware and software resources, including consoles for accessing cloud services, computing resources, storage resources, networking resources, etc. Customers of a cloud environment provided by a CSP can subscribe to one or more cloud services offered by that environment. The CSP can offer various subscription models to its customers. After a customer subscribes to a cloud service provided by the cloud environment, one or more users can be associated with the subscribed customer, and such users can use the cloud services subscribed to by the customer. In some implementations, when a customer subscribes to a cloud service provided by a specific cloud environment, a customer account or customer lease is created for that customer. One or more users can then be associated with the customer lease, and such users can then use the services subscribed to by the customer under the customer lease. Information about the services subscribed to by the customer, the users associated with the customer lease, etc., is typically stored within the cloud environment and associated with the customer lease.

[0170] For example, Figure 6 The document describes two different cloud environments provided by two different CSPs (although different numbers of cloud environments are possible). These include cloud environment A (Cloud A) 610 provided by CSP A and cloud environment B (Cloud B) 640 provided by CSP B.

[0171] Cloud A 610 includes infrastructure CSPI_A 612 provided by CSP A. This infrastructure CSPI_A 612 can be used to provide a set of in-cloud services 615 provided by Cloud A 610. One or more customers (e.g., Cust_A1 616-1, Cust_A2 616-2) can subscribe to one or more of these services. One or more users 618-1 can be associated with customer Cust_A1 616-1 and can use the services subscribed to by customer Cust_A1 616-1 in Cloud A 610. Similarly, one or more users 618-2 can be associated with customer Cust_A2 616-2 and can use the services subscribed to by customer Cust_A2 616-2 in Cloud A 610. In various use cases, the services subscribed to by customer Cust_A1 616-1 can differ from the services subscribed to by customer Cust_A2 616-2.

[0172] Similarly, Cloud B 640 includes infrastructure CSPI_B 642 provided by CSP B. This infrastructure CSPI_B 642 can be used to provide a set of services 644 provided by Cloud B 640 (this set of services may, but does not need to, be different from the services provided by Cloud A 610). One or more customers (e.g., Cust_B1 646-1) can subscribe to one or more services from this set of services 644. One or more users 648-1 can be associated with customer Cust_B1 646-1 and can use the services subscribed to by customer Cust_B1 646-1 in Cloud B 640.

[0173] like Figure 6 As described, customer Cust_A1 616-1 is also a customer of CSP B and has subscribed to services available from CSPI_B 642. Therefore, customer Cust_A1 616-1 has leases in both Cloud B 640 and Cloud A 610.

[0174] In some embodiments, CSP A and CSP B may agree to provide each other's cross-cloud services. Figure 6 In the diagram, CSP A provides one or more of its services (referred to herein as cross-cloud services) to CSP B's customers via CSPI_B 642. These cross-cloud services include, for example, database services, storage services, compute services, etc. Therefore, customer Cust_A1616-1 (a customer of both Cloud A and Cloud B) can request, subscribe to, use, and / or manage one or more cross-cloud services of CSP A via its lease at CSPI_B 642. In contrast, customer Cust_B1 646-1 has no lease at CSPI_A 612. Therefore, cross-cloud services of CSP A may not be available to customer Cust_B1 646-1 unless customer Cust_B1 646-1 requests a lease provided by CSP A. This request can be submitted and managed via the portal of Cloud B 640, as further described in the following figures.

[0175] To enable cross-cloud service provisioning and availability for CSP A via Cloud B 640, Cloud A 610 can implement Inter-Cloud Service 614. Inter-Cloud Service 614 can be configured to, among other things, enable the use of cross-cloud services for CSP A via B. For example, Inter-Cloud Service 614 can communicate with Service 644 of Cloud B 640 and translate such communication into information suitable for use by In-Cloud Services 615 of Cloud A 610. More specifically, this set of Services 644 can enable the portal for CSP B and the deployment and management resources within Cloud B 640 for CSP B's customers. Through this portal, cross-cloud services for CSP A can be provided. Therefore, customers of CSP B (e.g., customer Cust_A1 616-1) can subscribe to and request cross-cloud services for CSP A via the portal. Such subscription and cloud operation requests can be received by Inter-Cloud Service 614 and then translated into information specific to Cloud A 610. Such information can then be passed to one or more of the In-Cloud Services 615 that are then provisioned. Information returned from cloud service 615 can be transformed by cloud service 614 and sent to cloud service 644 for use.

[0176] In some embodiments, the requested cross-cloud service can be provisioned by one or more services from cloud service 615 across resources in both cloud A 610 and cloud B 640. Doing so can support latency-sensitive operations (or, at least, reduce processing latency).

[0177] As illustrated, the infrastructure CSPI_A 612 of Cloud A 610 includes infrastructure 624 for the private cloud of customer Cust_A1616-1 in Cloud A 610 (e.g., a VCN for a portion of the customer's lease at Cloud A 610), infrastructure 626 for the private cloud of customer Cust_A2 616-2 in Cloud A 610, and other infrastructure 620. Each of these infrastructures includes hardware and / or software provided by CSP A and installed at locations (organized by region) under the control of CSP A.

[0178] In contrast, the infrastructure CSPI_B 642 of Cloud B 640 includes infrastructure 644 for the private cloud of customer Cust_A1 616-1 in Cloud A (e.g., a VCN for the customer as part of a lease at Cloud A 610), infrastructure 646 for the private cloud of customer Cust_A1 616-1 in Cloud B 640 (e.g., a VNET for the customer as part of Cloud B 640), infrastructure 648 for the private cloud of customer Cust_B1 646-1 in Cloud B 640, and other infrastructure 650. Infrastructure 644 includes hardware and / or software (e.g., co-located with components of CSPI_B 642) provided by CSP A and installed at locations (organized by region) under the control of CSP B. In contrast, infrastructures 646, 648, and 650 include hardware and / or software provided by CSP B and installed at locations (organized by region) under the control of CSP B.

[0179] Infrastructures 644 and 646 can be networked together, enabling customer Cust_A1 616-1 to access customer's private cloud regarding CSP A via customer's private cloud regarding CSP B, wherein the private cloud regarding CSP A is distributed between CSPI_A 612 and CSPI_B 642. This communication coupling of the private clouds can be initiated by inter-cloud service 614 and executed by one or more services in intra-cloud service 615. In this manner, customer A 616-1 has two leases: a first lease regarding CSP A including a first private cloud (e.g., VCN) distributed between CSPI_A 612 and CSPI_B 642, and a second lease regarding CSP B including a second private cloud (e.g., VNET) local to CSPI_B 642.

[0180] The requested cross-cloud service can actually be hosted on a first private cloud (e.g., at least partially on infrastructure 644 within CSPI_B642, and possibly on infrastructure 624 within CSPI_A 612) and is accessible via a second private cloud (e.g., for workflows hosted by infrastructure 646). Such a distribution of components is further described in the following figures, which enables the first private cloud of the first CSP to be hosted at least partially by the second cloud of the second CSP and linked to the second private cloud hosted by the second cloud.

[0181] Figure 7 An exemplary physical architecture for providing cross-cloud services based on infrastructure distributed across multiple CSPs, according to some embodiments, is described. Figure 7 In this illustration, the second CSP's customer will manage the lifecycle of the cross-cloud service developed by the first CSP. For illustrative purposes, the Exadata service (which may also be referred to herein as the Oracle DB service) and the Oracle and Microsoft clouds are described. In this illustration, Oracle corresponds to the first CSP and Microsoft corresponds to the second CSP, while the Exadata service corresponds to the cross-cloud service provided by the first CSP via the cloud of the second CSP. However, this embodiment is not so limited; rather, it is similarly applicable to other CSPIs, CSPs, and / or cross-cloud services.

[0182] A second CSP's customer (e.g., an Azure customer) can create and manage virtual resources (e.g., infrastructure and / or VM clusters) within the first CSP's cloud via the second CSP's portal (e.g., the Azure portal). These virtual resources can be provisioned to offer cross-cloud services. Support for such provisioning can be at least partially available through... Figure 6 This is achieved through inter-cloud services 614. Furthermore, the infrastructure supporting virtual resources can be distributed between the CSPIs of two CSPs (e.g., between an Azure data center and an OCI region). Specifically, a first portion of the infrastructure provided by the first CSP is installed at the CSPI of the second CSP, while the remaining second portion of the infrastructure provided by the first CSP is installed at the CSPI of the first CSP. The first portion may be referred to as a subsite included within or forming the cloud of the second CSP. The second portion may be referred to as a parent-child region including or forming the subsite, wherein the parent region is within the cloud of the first CSP.

[0183] exist Figure 7 On the left, a second CSPI_B 750 of the second CSP is shown. This CSPI_B 750 can represent, for example, the data center of the second CSP (e.g., an Azure data center). Within the CSPI_B 750, a base network 730 is available and provided by the second CSP. The base network 730 includes a set of computing resources, such as routers 732A, 732B, etc. (In the case of an Azure data center, these routers may include MeetMe ToRs supporting the MeetMe protocol for peer-to-peer connections). The CSPI_B 750 also includes a subsite 720 (which is a set of computing resources, such as server blades, racks, or other physical hardware of the first CSP that executes the CSP's software). Subsite 720 includes, among other things, routers 722A, 722B, etc. (in the case of OCI, these routers may include FastConnect routers that support the FastConnect protocol for connecting peers), connectivity infrastructure 724 (e.g., physical infrastructure that provides connectivity to other physical structures and components, such as JFAB), and physical resources 726 (e.g., racks, such as OCI server blades optimized for cross-cloud services, such as Exadata services).

[0184] exist Figure 7 On the right, a first CSPI_A 700 of the first CSP is shown. This CSPI_A 700 may represent, for example, the data center of the first CSP (e.g., an OCI data center). Within CSPI_A 700, a parent region 710 is illustrated. The parent region 710 may include a base network with multiple components. Among these components is a connectivity structure 712 (e.g., a physical structure such as a JFAB). Connectivity structure 712 is connected to connectivity structure 724, such that the child site 720 is communicatively coupled to the parent region 710. Optionally, the parent region 710 is located in an area physically adjacent to the child site 720 (or equivalently, CSPI_B 750), such that network latency for communication between the child site 720 and the parent region 710 is reduced.

[0185] Router 732 within base network 730 can be interconnected (e.g., via Ethernet cable) to router 722 within subsite 720. Connectivity structure 724 of subsite 720 provides interconnectivity between router 722 and physical resource 726. Furthermore, this connectivity structure 724 is connected to connectivity structure 714 of parent area 710 (e.g., using fiber optic cable, such as dark fiber), allowing data connectivity to exist between subsite 720 and parent area 710.

[0186] The base network 730 can host a set of resources for a second CSP for a customer (e.g., to provide a VNET that includes compute instances with access to the Exadata service, as further illustrated in the figures below). These second CSP resources can be part of a neighboring group across cloud services within a certain latency (e.g., 100µs).

[0187] Subsite 720 can host latency-critical resources of the first CSP, where these resources support cross-cloud services (e.g., subsite 720 can host OCI database resources and data plane resources to support the Exadata service). Parent region 720 can host other resources that support cross-cloud services (e.g., Figure 6 (e.g., for the Exadata service, parent region 720 can host ORP, OCI tools, OCI metrics and logging, the OCI control plane, regional OCI services, and the customer-facing console). Some of these resources (e.g., OCI database resources and data plane resources) can be deployed as part of a first private cloud for a customer of the first CSP (e.g., VCN in the case of OCI) and can be perceived by the customer as being available to them via a second private cloud for a customer of the second CSP (e.g., VNET in the case of Azure). This perception is possible by using the same IP address range for cross-cloud services in both private clouds.

[0188] In the example, a customer may have multiple private clouds (e.g., multiple VNETs and / or multiple VCNs) for each CSP. In the case of multiple private clouds for the first CSP, the underlying physical resources may not co-located in the same CSPI for the second CSP; instead, they may be included in different subsites or form different subsites. In this case, these resources may not be directly interconnected (e.g., there may be no direct physical connection between different subsites). Instead, indirect connections may exist via a parent region, where each of the different subsites is physically connected to the parent region (e.g., via fiber optic cable), and data flows between the two subsites through the parent region.

[0189] Furthermore, Subsite 720 can support multi-tenant architecture. Specifically, multiple customers can each have one or more private clouds (e.g., one or more VNETs and one or more VCNs) for each CSP. Each such customer can have separate access to the corresponding cross-cloud service via Subsite 720.

[0190] Figure 8 An exemplary virtual architecture for providing cross-cloud services based on infrastructure distributed across multiple CSPs, according to some embodiments, is described. Figure 8 In this context, a first private cloud 810 (e.g., a VCN hosted by OCI) for the first CSP's customers and a second cloud 820 (e.g., a VNET hosted by Azure) for the second CSP's customers can be provisioned to offer cross-cloud services of the first CSP (e.g., Exadata services provided by Oracle and available to Azure customers via Azure) through the second CSP's cloud.

[0191] In the example, the second private cloud 820 (e.g., VNET) is hosted by the physical compute resources of the second CSP's CSPI_B 850 (e.g., Azure data center). The second private cloud 820 may include compute instances (not shown) with access to cross-cloud services. The second private cloud 820 includes a subnet 822. Subnet 822 uses an IP address range. This range can be specified based on customer input via a portal of the second CSP (e.g., the Azure portal). IP addresses can be assigned from the IP address range to each compute instance launched to provide cross-cloud services to customers (e.g., VMs belonging to a VM cluster and providing database instances). These compute instances may be hosted by the first private cloud 810 (in... Figure 8 More generally, these are referred to as virtual resources 814A, 814B, 814C, and 814D. Figure 8 The diagram illustrates four computational examples, labeled with rectangular representations of NIC 824A, 824B, 824C, and 824D (or collectively referred to as NIC 824). While NIC stands for Network Interface Card, Figure 8 The NIC 824 itself is not like that. Instead, the NIC 824 represents mapping information that maps each IP address to a corresponding compute instance across cloud services. This mapping information can be stored as part of the configuration information of the second private cloud 820. If a compute instance of the second private cloud 820 uses an IP address to send or request traffic, and that IP address corresponds to a NIC, then the mapping information indicates that the traffic will be sent to or will be received from the first private cloud 810 (e.g., VCN), and is cross-cloud service traffic.

[0192] The first private cloud 810 (e.g., VCN) can be hosted by the physical computing resources of the first CSP, including, for example, subsites and parent regions (similar to...). Figure 7 The physical computing resources (those shown in the diagram) of the first private cloud 810 may include subnets 812 using the same IP address range. Each compute instance launched to provide cross-cloud services (e.g., each of virtual resources 814A, 814B, 814C, and 814D, collectively referred to as virtual resource 814) is hosted by a set of physical computing resources of the first CSP. The set of compute instances may include physical computing resources in subsites and physical computing resources in the parent region. The computing resources of the subsites may include, for example, those performing tasks such as... Figure 7 The hardware (e.g., racks) of the cross-cloud service software shown herein provides latency-critical virtual resources (e.g., database resources, data plane resources, etc.). The physical computing resources of the parent region may include, for example, servers, server clusters, network virtualization devices, etc., that provide non-latency-critical virtual resources (e.g., control plane resources, tooling resources, etc.).

[0193] Each compute instance can have an IP address from a range of IP addresses. A one-to-one mapping exists between compute instances in the first private cloud 810 (e.g., virtual resource 814) and the NIC 824 in the second private cloud 820. Therefore, cross-cloud service traffic can be sent from the second private cloud 820 to the first private cloud 810 and vice versa using this one-to-one mapping.

[0194] The second private cloud 820 can be connected (e.g., peer-to-peer) to the first private cloud 810 via one or more virtual routers (e.g., coupling the base network of the second CSP to the sub-sites), gateways (e.g., gateway 826 implemented at the sub-sites and gateway 816 implemented at the parent area, such as as part of a connectivity structure) and connectivity protocols (e.g., MeetMe and FastConnect protocols). Gateways 816 and 826 can collectively represent a dynamic routing gateway.

[0195] From the customer's perspective, the customer does not need to understand the underlying physical architecture and interconnects. Instead, the customer can have visibility at the virtual level, allowing them to perceive and manage their private clouds 810 and 820 (e.g., their network configuration), which is sufficient. Managing the second private cloud 820 can be done via a second portal of the second CSP. Managing the first private cloud 810 can be done via a second portal of the second CSP or a first portal of the first CSP. In this embodiment, changes and / or operations related to cross-cloud services are enabled only via the second CSP (e.g., via the second CSP, a customer can scale up or down, remove, terminate, add, etc., virtual resources in the first private cloud 810 launched for cross-cloud services). In contrast, changes and / or operations unrelated to cross-cloud services may be enabled only via the first CSP (e.g., via the first CSP, a customer can scale up or down, remove, terminate, add, etc., virtual resources in the first private cloud 810 launched for non-cross-cloud services).

[0196] Figure 9 Exemplary virtual resources provided to customers according to some embodiments are described. Figure 9 The right side shows the first resource (e.g., OCI resource) of the first CSP deployed for the customer. Figure 9 The left side shows a second resource (e.g., to an Azure data center) deployed for a second CSP for a customer. The first CSP (shown as CSP A) provides cross-cloud services (e.g., Exadata services) available via the second CSP (shown as CSPB).

[0197] The first resource (which may be a combination of physical and virtual resources) may include a private cloud 920, a service lease 910, a VM cluster 926, and a gateway 928. The private cloud 920 (e.g., a VCN) may correspond to a customer's lease to a first CSP and may include a primary subnet 922 with an IP range (e.g., a Classless Inter-Domain Routing (CIDR) range) and, optionally, a backup subnet 924 using the same IP range. The service lease 910 may be used by multiple customers of the first CSP and may include a gateway 912. The VM cluster 926 includes multiple VMs (or compute instances) launched for clients to provide cross-cloud services. The VM cluster 926 may be hosted on physical resources within a subsite. The gateway 928 (e.g., a service gateway) connects the virtual resources of the primary subnet 922 (and, similarly, the backup subnet 924) hosted on physical resources within a parent region to the VM cluster 926. In contrast, the gateway 912 connects such virtual resources to a customer's private cloud 960 for a second CSP. Gateway 912 can be implemented as a DRG attached to a private cloud 920 in a customer lease. Connectivity can be via gateway / router 914. Gateway / router 914 can provide, at least partially, connectivity within a subsite to the parent region via a first connectivity protocol (e.g., FastConnect) (thus providing gateway functionality) and connectivity to the subsite's base network of a second CSP via a second connectivity protocol (e.g., MeetMe) (thus providing routing functionality). For example, gateway / router 914 can represent a FastConnect virtual circuit resource in service lease 910 to connect the DRG to an Azure MeetMe router (example router 968).

[0198] The second resource (which may be a combination of physical and virtual resources) may include a private cloud 960 and a router 968. The private cloud 960 may include a main subnet 922 with an IP range, and optionally a backup subnet 964 using the same IP range. The IP range may be the same as one of the IP ranges in the main subnet 922. The router 968 may be connected to a gateway / router 914 to provide data connectivity from the main subnet 962 (e.g., and the backup subnet 964) to the private cloud 920. The main subnet 962 may include NICs 966A to 966K connected to the router 968.

[0199] The first CSP (e.g., its inter-cloud or intra-cloud service) can select IP addresses from an IP range for different virtual resources to provide cross-cloud services. The first CSP (e.g., its inter-cloud or intra-cloud service) can create Domain Name System (DNS) records corresponding to these IP addresses, which can then be provided to Private Cloud 960. Such DNS records can be used to establish private DNS zones for customers, which can be used in conjunction with Private Cloud 960.

[0200] For illustration, subnets 922, 924, 962, and 964 use IP addresses from 10.0.10.0 / 24. Each NIC 966 has an IP address from this range (e.g., NICA 966 A uses 10.0.10.10, and NIC K 966 K uses 10.0.10.11). VMs in the VM cluster use the same IP addresses (e.g., the first VM uses 10.0.10.10, while the second VM uses 10.0.10.11).

[0201] Figure 10 Exemplary architectures for provisioning and managing cross-cloud services based on infrastructure distributed across multiple CSPs, according to some embodiments, are described. Figure 10 In this example, the customer of the second CSP (shown as CSP B) will manage the lifecycle of a cross-cloud service developed by the first CSP (shown as CSP A). For clarity and for illustrative purposes, OCI and Azure are described as examples of CSPI, and the Exadata service is described as an example of a cross-cloud service. However, the embodiment is not so limited, and rather, similarly applies to other CSPs and / or cross-cloud services.

[0202] A customer (e.g., an Azure customer) can operate customer device 1000 to create and manage a set of virtual resources (e.g., Oracle DB infrastructure resources) across cloud services via a portal of a second CSP (shown as CSP_B portal 1052). In the example, portal 1052 interacts with one or more services of the second CSP (shown as CSP_B service 1054, examples of which may include Azure Resource Manager (ARM), which exposes APIs to manage the lifecycle of Oracle DB infrastructure resources (DB infrastructure resources include Exadata infrastructure and cloud VM clusters); other examples include Azure Resource Provider as a Service (RPaaS) to simplify the development and operation of Azure resource providers, such as asynchronous operations and their progress updates). To manage resources built on top of the virtual resource group (DB Home, database, pluggable database), CSP_B service 1054 interacts with the inter-cloud service 1012 of the first CSP to redirect customers to inter-cloud service 1012 (e.g., ORP in the case of OCI).

[0203] Inter-cloud service 1012 is configured to: i. handle the translation of an identifier assigned by a second CSP (e.g., an Azure identifier) ​​to an identifier assigned by a first CSP (e.g., an OCI identifier), including identity, resource ID, and subscription ID; ii. handle the translation of a second CSP state (e.g., an Azure state) to a first CSP state (e.g., an OCI state) and vice versa; and / or iii. delegate requests to an in-cloud service 1014 of the first CSP for execution (e.g., delegate to a resource control plane 1016, such as the OCIDBaS control plane). Inter-cloud service 1012 is also configured to coordinate any second CSP-specific integrations with other second CSP services (e.g., with Azure resource providers). Inter-cloud service 1012 may also be configured to perform or cause other in-cloud services 1014 to perform operations, including linking cloud accounts, publishing observability information, and issuing vending tokens to access other cloud customer environments.

[0204] Cloud-based service 1014 can provide a portal (shown as CSP_A portal 1018) accessible to the customer's client device 1000 to manage other services provided by the first CSP. Two portals, 1052 and 1018, can enable similar functionality (e.g., by presenting input and output fields), however, they are different. For example, portal 1052 (e.g., the Azure portal) can have a presentation format controlled by the second CSP. Additionally, in addition to functionality related to cross-cloud services, portal 1052 can also enable functionality specific to the second CSP and independent of the first CSP. In contrast, portal 1018 (e.g., the OCI portal) can have a presentation format controlled by the first CSP. Additionally, in addition to functionality related to cross-cloud services, portal 1018 can also enable functionality specific to the first CSP and independent of the second CSP. Regarding cross-cloud services, portal 1052 can expose information available from the second CSP, which can be provided by inter-cloud service 1012.

[0205] In the OCI and Azure use case examples, Cloud Service 1012 exposes the Oracle DB product. Cloud Service 1012 can be registered with ARM via RPaaS and configured to implement Azure Resource Provider Contracts (RPCs), which are a set of operations supported by all Azure resource providers. Azure RPaaS internal services are used to support a large number of RPC operations.

[0206] Typically, Cloud Service 1012 can be configured to have a second CSP identity (e.g., an Azure RP identity equivalent to an OCI service principal) and the ability to operate on a second CSP customer environment using a second CSP stream (e.g., an Azure OBO stream). Cloud Service 1012 can also be configured to persist second CSP-specific metadata for first CSP resources, such as Azure identifier-to-OCID mappings for DB resources. Cloud Service 1012 can also be configured to have a first CSP identity to gain scope for operating on a first CSP customer environment. Cloud Service 1012 can also be configured to act as a thin adapter layer, accepting requests in a second CSP format already authenticated by Service 1054, converting them to a first CSP request, and delegating the request to an in-cloud service.

[0207] Therefore, cloud service 1012 performs multiple operations. These operations include converting identifiers from one cloud to another and vice versa. Operations also include obtaining the first CSP identity from an incoming request from service 1054 for invoking cloud service 1014. These operations also include converting a second CSP format request into a first CSP format request and invoking cloud service 1014. These operations also include limiting / quota / capacity verification delivery or coordination, implicitly creating first CSP prerequisites for network connectivity resources (e.g., creating the subnet, OCI DRG, and VCN to which the VMCluster is attached), resulting in private cloud links, and configuring DNS entries for resources with associated DNS records regarding the second CSP.

[0208] In the example, inter-cloud service 1012 can be hosted in one or more sub-sites. Inter-cloud service 1012 allows virtual resource 1020 to be hosted in a sub-site. In the Exadata service use case, virtual resource 1020 can include the DBaaS data plane via the DBaaS control plane. This control plane is hosted in a parent region as part of in-cloud service 1014. In this use case, compute instances (e.g., VMs) can be instantiated for the customer on the customer's private cloud regarding the second CSP (e.g., VNET in Azure). The compute instances can perform database operations by making calls to the DBaaS data plane. Such operations include queries, storage, etc., or any operations supported by the Exadata service. Calls and responses to it can be made within the private cloud regarding the second CSP. When in use, the DBaaS data plane can report usage information (e.g., for metrics analysis, billing, etc.) to one or more services in in-cloud service 1014 (e.g., to the observability service). This usage information can then be provided to the monitoring service of the second CSP.

[0209] Figure 11 An exemplary user experience flow 1100 for provisioning resources according to some embodiments is depicted. In the example, user experience flow 1100 relates to client device 1120, CSPI_B 1150 of a second CSP, and inter-cloud services of a first CSP. The first CSP may provide cross-cloud services via the second CSP. Virtual resources may be provisioned to provide at least a portion of the cross-cloud services, wherein provisioning follows user experience flow 1100.

[0210] User input is provided by CSPI_B 1150 (e.g., via a similar method). Figure 10 The CSP_B portal 1052 receives instructions from client device 1120 to create a subnet and mark it as being delegated, making the subnet available for cross-cloud services. The subnet and the client's second private cloud with respect to the second CSP are created, and the instructions for its creation are provided to client device 1100.

[0211] Next, user input is received by inter-cloud service 1110 from computing device 1100 (where this input can be at the portal), indicating a request to create cross-cloud service infrastructure (e.g., an Exadata service). User input can indicate various parameters for provisioning, such as region and availability zone. Inter-cloud service 1110 can (e.g., via the portal) indicate the start of provisioning to computing device 1100 and can execute a provisioning workflow through which the cross-cloud service infrastructure is created in the relevant subsites. Simultaneously with the creation of the infrastructure, inter-cloud service 1110 sets its status to "Provisioning". After creation, inter-cloud service 1110 updates the status to "Successfully" and provides it. Indications to these statuses can (e.g., via the portal) be provided to computing device 1100. Similarly, computing device 1100 can check the current status via the portal, where a status query request can be made to inter-cloud service 1110 from CSPI_B 1150, and inter-cloud service 1110 will respond by returning status information.

[0212] Subsequently, user input is received by cloud service 1110 from CSPI_B 1150 (where this input is provided at the portal) and corresponds to a request from computing device 1100 to create VM cluster resources. The request may indicate a subnet. Cloud service 1110 then performs a set of checks and begins provisioning the VM cluster. The following figures further illustrate this provisioning. Cloud service 1110 sets the status to "Provisioning". After creation, cloud service 1110 updates the status to "Successful". Here, these status indications may also be provided to computing device 1100 (e.g., via the portal). Similarly, computing device 1100 may check the current status via the portal, where a status query request may be made from CSPI_B 1150 to cloud service 1110, and cloud service 1110 will respond by returning status information.

[0213] Figure 12 An exemplary control plane provisioning flow 1260 according to some embodiments is depicted. In the example, control plane provisioning flow 1260 relates to inter-cloud service 1200, control plane 1202 (e.g., an example of an in-cloud service), a customer's private cloud 1204 for a first CSP, gateway 1206 (e.g., DRG), connectivity module 1208 (e.g., a connectivity module provided by a connectivity infrastructure and supporting connectivity protocols such as FastConnect), cross-cloud service infrastructure 1210 for the first CSP, and CSPI_B 1250 for the second CSP. The first CSP can provide cross-cloud services via the second CSP. Here, control plane 1202 can execute a portion of control plane provisioning flow 1260 to provision VM cluster resources when triggered by inter-cloud service 1200. In response to user input on the portal of the second CSP, inter-cloud service 1200 can make such a call to control plane 1202 after receiving input from CSPI_B 1250 (from ARM in Azure).

[0214] As described above, the inter-cloud service 1220 receives a request to create a VM cluster (e.g., an Exadata VM cluster) for cross-cloud services and passes this request, along with other information (e.g., a first identifier assigned by a first CSP and mapped to a second identifier assigned by a second CSP), to the control plane 1202. The control plane 1202 then retrieves details about the subnet from CSPI_B 1250 (e.g., from ARM in Azure), including the CIDR, and verifies that prerequisites are met (e.g., the subnet has been provisioned, the region of the second private cloud (e.g., a VNET in Azure) for the second CSP, etc.). The control plane 1202 then (e.g., based on the first identifier) ​​creates a first private cloud (e.g., a VCN in OCI) for the first CSP in the customer lease and creates the subnet within the first private cloud. This subnet has the same CIDR as the subnet in the second private cloud.

[0215] Next, control plane 1202 creates and configures gateway 1206 (e.g., DRG) in the service lease and requests that gateway 1206 be attached to the first private cloud in the customer's lease. Control plane 1202 also configures routing information in gateway 1206 (e.g., by creating a DRG routing table for traffic between the two private clouds or MeetMe routers). Similarly, control plane 1202 also configures routing information in the first private cloud (e.g., by creating a VCN routing table for traffic to the DRG and VM clusters). After the routing information has been created, control plane 1202 can notify inter-cloud service 1200 of success. In response, inter-cloud service 1200 can configure a network security group for the first private cloud and request the creation of a VM cluster. The VM cluster is then provisioned in the relevant subsite, thereby assigning IP addresses from CIDR to the VM cluster. DNS records corresponding to the IP addresses are also generated. Inter-cloud service 1200 can receive such DNS records and request attachment from the first private network to the second private network (e.g., VCN to VNET).

[0216] Control plane 1202 then invokes the API of the second CSP, which allows IP addresses to be injected into the customer's subnet in the form of NICs. Each IP address can represent a VM cluster node IP. Control plane 1202 can register all the IP addresses that the VM cluster will have. The API call can include a device identifier (e.g., the GUID of the MeetMe router). This identifier can be based on a mapping of the GUID to a physical location (subsite). The call can also include a subnet resource identifier (e.g., the subnet of the private network in the customer's lease), an IP address (e.g., the IP address of the NIC being created, where the address is within the subnet range), and a resource name (the name of the NIC that will be visible to the user). The API can return a Virtual LAN identifier (VLAN ID) available during virtual circuit creation. Control plane 1202 then creates a virtual circuit that uses a connection to a router (e.g., to the MeetMe router) (e.g., a FastConnect connection) and provides DNS records so that a private DNS zone can be added to the customer's second private network. At this point, the VM cluster is successfully created. Then, compute instances across cloud services (e.g., Exadata database instances) can be hosted in a VM cluster.

[0217] Multi-cloud networking

[0218] As discussed above, a first CSP can provide in-cloud services (e.g., database services, storage services, computing services, etc.) to its customers, and a second CSP can provide similar in-cloud services to its customers. The second CSP's customers can also be customers of the first CSP and may wish to access the in-cloud services provided by the first CSP via their lease within the cloud environment provided by the second CSP. Therefore, the first CSP can provide in-cloud services to the second CSP's customers as cross-cloud services. Similarly, the second CSP can provide in-cloud services such as cross-cloud services to the first CSP's customers. At least one of the cross-cloud services provided by one CSP to another CSP's customers can be the same service as the in-cloud service provided by that CSP to its own customers. In this way, a CSP's customers can be provided with another CSP's platform-level experience from within that CSP's cloud environment. Additionally, a CSP's customers can be exposed to new features, versions, and resources of another CSP without leaving that CSP's cloud environment.

[0219] For example, Oracle, as a CSP, provides in-cloud services to its own customers via OCI, and Microsoft, as a CSP, can provide similar in-cloud services to its own users via Azure. Microsoft Azure customers may also be Oracle OCI customers and may wish to access Oracle's in-cloud services offered by OCI through their Microsoft Azure leases. Therefore, Oracle's OCI can offer in-cloud services such as Oracle's Exadata database service as a cross-cloud service to Microsoft Azure customers. Similarly, Microsoft Azure can offer in-cloud services such as Azure Synapse Analytics as a cross-cloud service to Oracle OCI customers. At least one of the cross-cloud services offered by Oracle's OCI to Microsoft Azure customers can be the same service that Oracle's OCI offers to one of its own customers. For example, Oracle's OCI can offer Oracle's Exadata database service to both its own customers and Microsoft Azure customers through a Microsoft Azure environment. In this way, Microsoft Azure customers can be provided with a platform-level experience with Oracle's OCI within Microsoft Azure. Additionally, Microsoft Azure customers can expose new features, versions, and resources to Oracle's OCI without leaving Microsoft Azure. While Oracle's OCI and Microsoft's Azure have been used as examples, the techniques described are not limited to these CSPs and can be similarly applied to other CSPs such as Google Cloud™ and AWS®.

[0220] To facilitate the provision of cross-cloud services, the infrastructure of the first CSP can be provided within the CSPI of the second CSP, and services provided by the first CSP can be offered as cross-cloud services using this infrastructure. The infrastructure in the first CSP's CSPI can serve as a parent region for the infrastructure provided in the second CSP's CSPI, and the infrastructure provided in the second CSP's CSPI can be considered a child site of the parent region. Providing the first CSP's infrastructure within the second CSP's CSPI enables the first CSP to offer low-latency, high-bandwidth cross-cloud services to its customers. In this way, the second CSP's customers can utilize services provided by the first CSP from their leases with the second CSP.

[0221] Using subsites to provide access to cross-cloud services can offer high-bandwidth access to those services with reduced latency compared to those accessed via a first CSP and / or other remote cloud environments. However, resources such as compute resources at each subsite may be limited. Therefore, managing the network between subsites and the parent region may be desirable. The techniques described herein overcome these and other challenges by providing technologies for managing cross-cloud networks to facilitate the provision of low-latency, high-bandwidth cross-cloud services between one or more CSPs.

[0222] Figure 13 An example architecture 1300 for providing services based on infrastructure distributed across multiple cloud service providers is described. For example... Figure 13 As described herein, various cloud environments (also referred to as "clouds") can be provided by different CSPs. Each cloud environment or cloud provides one or more cloud services that can be utilized by one or more customers of the corresponding CSP. The cloud service group provided by a cloud environment provided by a CSP can include one or more different types of cloud services, including but not limited to SaaS services, IaaS services, PaaS services, DBaaS services, etc. Examples of cloud environments provided by various CSPs include OCI, Azure, Google Cloud, AWS, etc. The cloud services provided by a particular cloud environment may differ from the cloud service group provided by another cloud environment.

[0223] like Figure 13 As shown, architecture 1300 may include a first cloud environment, cloud environment A (cloud A) 1310 (although a different number of cloud environments are possible). Cloud A 1310 may be provided by a first CSP A (e.g., CSP A). Architecture 1300 may further include a second cloud environment, cloud environment B (cloud B) 1340. Cloud B 1340 may be provided by a second CSP (e.g., CSP B). Cloud A 1310 and Cloud B 1340 may be configured according to... Figures 6-12 The multi-cloud environment described is used for implementation. Cloud A 1310 and Cloud B 1340 can host one or more private clouds for customers of the first CSP and / or the second CSP. For example, the cloud infrastructure of Cloud A 1310 can host private clouds for customers of the first CSP (e.g., VCN in the case of Oracle OCI), and the cloud infrastructure of Cloud B 1340 can host private clouds for customers of the second CSP (e.g., VNET in the case of Microsoft Azure). Additionally, cross-cloud services between Cloud A 1310 and Cloud B 1340 can be implemented according to... Figures 10-12 The described experience and supply process are used to supply.

[0224] Cloud A 1310 includes infrastructure CSPI_A 1312. Infrastructure CSPI_A 1312 can be used to provide a set of services (e.g., in-cloud services) to customers of CSP A. Similarly, Cloud B 1340 includes infrastructure CSPI_B 1342, which can be used to provide customers of CSP B with a set of services provided by Cloud B 1340 (this set of services may, but does not need to, be different from the services provided by Cloud A 1310). In some embodiments, CSP A and CSP B can provide their services as cross-cloud services to their respective customers or tenants. For example, CSP A can provide one of its in-cloud services as a cross-cloud service to customers of CSP B. CSPI_A 1312, together with CSPI_B 1342, can facilitate the provision of services of Cloud A as cross-cloud services. Similarly, although not shown, CSPI_B 1342, together with CSPI_A 1310, can facilitate the provision of services of Cloud B as cross-cloud services. Cross-cloud services may include, but are not limited to, database services, storage services, computing services, etc. Therefore, customers of both Cloud A and Cloud B can request, subscribe to, use and / or manage one or more cross-cloud services of CSP A and CSP B through their respective leases of Cloud A 1310 and Cloud B 1340.

[0225] CSPI_A 1312 (which may be a data center of CSP A) includes a set of CSP A computing resources 1314 supporting cloud services of cloud A 1310. Similarly, CSPI_B 1342 (which may be a data center of CSP B) includes a set of CSP A computing resources 1344 and a set of CSP B computing resources 1348 supporting cloud services of cloud A 1310 and cloud services of cloud B. The set of CSP A computing resources 1344 can be physically housed within CSPI_B 1342, allowing customers of CSP B to perceive these resources as part of a private cloud hosted for them by cloud B 1340. The set of CSP A computing resources 1314 can be connected to the set of CSP A computing resources 1344 using routers 1318 and 1320. Router 1318 is connected to the set of CSP A computing resources 1314, and router 1320 is connected to the set of CSP A computing resources 1344. Routers 1318 and 1320 can be connected using one or more physical connections (e.g., dark fiber) and provide data connectivity between the set of CSP A computing resources 1314 and 1344 based on a communication protocol. For example, in the case of Oracle OCI, routers 1318 and 1320 can be FastConnect routers supporting the FastConnect protocol for connecting peers. In some implementations, although not shown, each of routers 1318 and 1320 may include one or more routers and is part of a network connectivity structure. The set of CSP A computing resources 1314 can be a parent region of cloud A 1310, and the set of CSP A computing resources 1344 can be a child site of the parent region of cloud A 1310, and the set of CSP A computing resources 1314 and 1344 can together host a private cloud for customers of CSP A and / or CSP B. By positioning this set of CSP A computing resources 1344 within CSPI_B 1342 of Cloud B 1340, customers of Cloud B 1340 can be provided with low-latency and high-bandwidth access to Cloud A 1310 services.

[0226] Routers 1350 and 1360 can also be used to connect the group of CSP A compute resources 1344 to the group of CSP B compute resources 1348. Router 1350 is connected to the group of CSP A compute resources 1344, and router 1360 is connected to the group of CSP B compute resources 1348. Routers 1350 and 1360 can be connected using one or more physical connections (e.g., Ethernet connections) and provide data connectivity between the group of CSP A compute resources 1344 and the group of CSP B compute resources 1348 based on a communication protocol. In some implementations, the communication protocol supporting data connectivity between routers 1318 and 1320 is different from the communication protocol supporting data connectivity between routers 1350 and 1360. For example, in the case of Oracle's OCI and Microsoft's Azure, router 1350 could be a FastConnect router, and router 1360 could be a MeetMe ToR router supporting the MeetMe protocol for connecting peers. In some implementations, although not shown, each of routers 1350 and 1360 may include one or more routers. The set of CSP A computing resources 1344 and the set of CSP B computing resources 1348 may together host a private cloud for CSP B's customers. Each set of computing resources (i.e., CSP A computing resources 1314, 1344 and CSP B computing resources 1348) may include software computing resources and hardware computing resources, such as server blades, racks, and processors, to facilitate the hosting of the private cloud and provide cloud services.

[0227] In some implementations, a cloud network can be formed between the set of CSP A computing resources 1314 and the set of CSP A computing resources 1344. To form the cloud network, cloud A 1310 can reserve a service subnet 1316 within the set of CSP A computing resources 1314, and cloud B 1340 can reserve a delegated subnet 1346 corresponding to the service subnet 1316 within the set of CSP A computing resources 1344. The delegated subnet 1346 can be associated with an Internet Protocol (IP) address within the Classless Inter-Domain Routing (CIDR) IP range as the IP address associated with the service subnet 1316. In some implementations, the IP address can be specified based on client input received via the CSPB portal.

[0228] The group of CSP A computing resources 1314 can be configured to host a group of virtual resources 1334A, 1334B, 1334C, and 1334D. In some implementations, the group of virtual resources 1334A, 1334B, 1334C, and 1334D can be a group of computing instances within the first group of CSP A computing resources 1314, wherein each computing instance of the group is associated with a NIC of a group of network interface cards (or NICs) associated with the first group of CSP A computing resources 1314.

[0229] The set of CSP A computing resources 1346 can be configured to host a set of NICs 1324A, 1324B, 1324C, and 1324D corresponding to the set of virtual resources 1334A, 1334B, 1334C, and 1334D. In some implementations, each corresponding NIC (e.g., NIC 1324A) can correspond to each corresponding virtual resource (e.g., virtual resource 1334A). In some implementations, the set of NICs 1324A, 1324B, 1324C, and 1324D can correspond to a virtual machine cluster, where each corresponding virtual machine in the virtual machine cluster corresponds to a corresponding computing instance of the set of computing instances of the set of CSP A computing resources 1314. In some implementations, the cloud network is formed by connecting the set of virtual resources 1334A, 1334B, 1334C, and 1334D to the virtual machine cluster of the set of CSP A computing resources 1344. In some implementations, the cloud network is formed by connecting the NICs of the set of CSP A computing resources 1314 to the set of NICs 1324A, 1324B, 1324C, and 1324D of the set of CSP A computing resources 1344. In some implementations, an IP address within the CIDR IP address range can be assigned to each of the corresponding NICs 1324A, 1324B, 1324C, and 1324D, and this IP address can be used to map each virtual machine in the virtual machine cluster to the corresponding virtual resource in the set of virtual resources 1334A, 1334B, 1334C, and 1334D. The mapped IP address can be included in the mapping information stored as part of the configuration information of Cloud B 1340, so that network traffic received at the group of NICs 1324A, 1324B, 1324C and 1324D can be routed to the group of virtual resources 1334A, 1334B, 1334C and 1334D by accessing the mapping information.

[0230] Figure 14 An example architecture 1400 is depicted, illustrating infrastructure distributed across multiple cloud service providers. (Example:) Figure 14 As shown, CSPI_A 1312 may include CSP A physical resource 1410. In some implementations, CSP A physical resource 1410 may be included... Figure 13 The set of CSP A computing resources 1314 shown herein and / or supports the set of CSP A computing resources 1314. Similarly, as... Figure 14 As shown, CSPI_B 1342 may include a CSP A cage 1406 and a CSP B base network 1408. The CSP A cage 1406 may include CSP A physical resources 1412. In some implementations, the CSP A physical resources 1412 may be included in... Figure 13 The CSP A computing resources 1344 shown are included in and / or support the CSP A computing resources 1344. The CSP B base network 1408 may include CSP B physical resources 1420. CSP B physical resources 1420 may be included in and / or support the CSP B computing resources 1348. Each group of physical resources (i.e., CSP physical resources 1410, 1412 and CSP B physical resources 1420) may include software computing resources and hardware computing resources, such as server blades, racks, and processors, to facilitate hosting private clouds and providing cloud services. A cloud network may be formed between CSP A computing resources 1314 and CSP A physical resources 1410, and between CSP A computing resources 1344 and CSP A physical resources 1412. Similarly, a cloud network may be formed between CSP A computing resource groups 1344 and CSP A physical resources 1412, and between CSP B computing resources 1348 and CSP B physical resources 1420.

[0231] In some implementations, CSP B's customers (e.g., Microsoft Azure customers) can create and manage private clouds hosted and managed by CSP A (e.g., Oracle's OCI) via CSP B's portal (e.g., the Azure portal). Figure 14 As shown, the physical infrastructure supporting the private cloud (i.e., CSP A physical resource 1410 and CSP A physical resource 1412) can be distributed across the CSPIs of the two CSPs (e.g., CSPI_A 1312 and CSPI_B 1342). For example, CSP A physical resource 1412 can be used as the first part of the physical infrastructure, and CSP A physical resource 1410 can be used as the second part of the physical infrastructure. The first part can be considered a subsite, and the second part can be considered a parent region of the subsite.

[0232] In some implementations, CSP A cage 1406 can host latency-critical resources supporting cross-cloud services provided by CSP A (e.g., the data plane for database services used in Exadata services such as OCI). On the other hand, CSPI_A 1312 can host other resources supporting cross-cloud services (e.g., resource providers, control planes, tooling and metrics resources, consoles, etc.). These resources can be deployed as part of a first private cloud for a customer with a lease on CSP A, but can be perceived by the customer as being available to them via a second private cloud through a lease on CSP B.

[0233] In some implementations, CSP A cage 1406 can serve as a physical and virtual security boundary 1416 between the resources of CSP A and CSP B. This security boundary 1416, which can be established at a physical port at the edge of CSP A's secure footprint, ensures that only designated customer traffic traverses the corresponding leased CSP A. In some implementations, connectivity between CSP A cage 1406 and CSPI_A 1312 can be supported by one or more dark fiber connections, ensuring that internal resources are inaccessible outside of CSP A and that the core CSP A network remains secure. Therefore, customers of both CSP A and CSP B can seamlessly integrate and manage high-performance cloud services of CSP A within the leased CSP B without sacrificing security, bandwidth, or latency.

[0234] Explanatory methods

[0235] Figure 15 The illustration shows an example process 1500 for providing services based on infrastructure distributed across multiple cloud service providers. Figure 15 The processing described herein can be implemented as software (e.g., code, instructions, programs), hardware, or a combination thereof, executed by one or more processing units (e.g., one or more processors, cores) throughout the corresponding system described. The software can be stored on a non-transitory storage medium (e.g., a memory device). Although Figure 15 The methods presented describe various processing steps that occur in a specific order or sequence, but this is not intended to be restrictive. In some alternative implementations, these steps may be executed in parallel and / or in a different order. In some implementations, such as in... Figures 6-14 In the embodiments depicted, Figure 15 The processing described herein can be performed by physical components of a cloud environment such as cloud environment 1310 and / or 1340.

[0236] At block 1502, a request for a cloud service is received. In some implementations, the request is received from a cloud environment provided by a cloud service provider. In some implementations, although not shown, receiving a request for a cloud service may include determining whether a request for the cloud service has been received. In response to receiving a request, the process may proceed to block 1504. If a request has not yet been received, another determination may be made regarding whether a request for the cloud service has been received.

[0237] In some implementations, requests may be received by components of the cloud environment. In some implementations, requests are initiated in response to client device interactions with a portal of the cloud environment. The client device may be associated with a customer account corresponding to a cloud service provider. In some implementations, the portal is configured to present a portion of a graphical user interface provided by another cloud service provider. Requests may be initiated in response to client device interactions with that portion. In some implementations, customers of the first CSP providing the cloud environment and / or end users with leases in the cloud environment can access cross-cloud services corresponding to cloud services provided by a second CSP.

[0238] In some implementations, requests for provisioning cross-cloud services received via a portal can be routed to a resource manager in one cloud environment, which can then route the request to a resource provider in another cloud environment. On the other hand, requests for managing the lifecycle of provisioned cross-cloud services received via a portal (e.g., viewing analytics, consumption, costs, logs, etc.) can be routed to a console in another cloud environment, which can then route the request within that cloud environment.

[0239] At block 1504, a cloud network is formed. In some implementations, a cloud network is formed between a first set of computing resources and a second set of computing resources. In some implementations, the first set of computing resources resides within a first infrastructure of a first cloud environment, and the second set of computing resources resides within a second infrastructure of a second cloud environment provided by a second cloud service provider different from the first cloud service provider. In some implementations, the first infrastructure is physically connected to the second infrastructure. In some implementations, a first router is associated with the first cloud environment and connected to the first infrastructure and connected to a second router associated with the second cloud environment. In some implementations, the first set of computing resources includes one or more processors connected to physical resources of the second cloud service provider.

[0240] In some implementations, the cloud network is configured to provide cloud services of the first cloud service provider to customers of the second cloud service provider using a first set of computing resources and a second set of computing resources. In some implementations, forming the cloud network includes receiving instructions from the control plane of the second infrastructure.

[0241] In some implementations, at block 1504A, forming a cloud network includes establishing a network connection between a first subnet of the first set of computing resources and a second subnet of the second set of computing resources. In some implementations, the first subnet is provided by the data plane of the first infrastructure, and forming a cloud network enables data about cloud services to be transferred from the data plane to an area within the second infrastructure. In some implementations, the Internet Protocol (IP) address associated with the first subnet is within the Classless Inter-Domain Routing (CIDR) IP range associated with the second subnet. In some implementations, at block 1504B, forming a cloud network includes connecting virtual resources of the first set of computing resources to a cluster of virtual machines of the second set of computing resources. In some implementations, at block 1504C, forming a cloud network includes connecting a first network interface card of the first set of computing resources to a second network interface card of the second set of computing resources.

[0242] In some implementations, to form a cloud network, a first subnet, such as the service subnet described above, can be retained within a first set of computing resources, and a second subnet, such as the delegated subnet described above, corresponding to the first subnet, can be retained within a second set of computing resources. As mentioned above, the delegated subnet can be associated with an Internet Protocol (IP) address within the Classless Inter-Domain Routing (CIDR) IP range as the IP address associated with the service subnet. The first set of computing resources can be configured to host a set of virtual resources. In some implementations, the first set of virtual resources can be a set of computing instances of the first set of computing resources, wherein each computing instance of the set of computing instances is associated with a NIC of a set of network interface cards (or NICs) associated with the first set of computing resources. In some implementations, each corresponding NIC can correspond to each corresponding virtual resource. In some implementations, the cloud network is formed by connecting the set of virtual resources to a virtual machine cluster of the second set of computing resources. In some implementations, the cloud network is formed by connecting the NICs of the first set of computing resources to a set of NICs of the second set of computing resources. In some implementations, each corresponding NIC of the second set of computing resources can correspond to a corresponding virtual machine of the virtual machine cluster. In some implementations, IP addresses within the CIDR IP address range can be assigned to each corresponding NIC in the group of NICs of the second group of compute resources, and can be used to map each virtual machine in the virtual machine cluster to a corresponding virtual resource in the group of virtual resources of the first group of compute resources. The mapped IP addresses can be included in mapping information stored as part of the configuration information, such that network traffic received at the group of NICs of the second group of compute resources can be routed to the group of virtual resources of the first group of compute instances by accessing the mapping information.

[0243] In this way, cloud services can be provided by one cloud service provider to the customers of another cloud service provider based on infrastructure distributed between two cloud service providers. Specifically, the infrastructure of the first cloud service provider can be physically located in a cloud environment hosted by the first cloud service provider, and also physically located in a cloud environment hosted by the second cloud service provider. Using the distributed infrastructure of the first cloud service provider, low-latency, high-bandwidth cloud services provided by the first cloud service provider can be accessed from within the private cloud network of the second cloud environment.

[0244] Cloud infrastructure examples

[0245] As indicated above, IaaS is a specific type of cloud computing. IaaS can be configured to provide virtualized computing resources over a public network (e.g., the Internet). In the IaaS model, cloud providers can host infrastructure components (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer), etc.). In some cases, IaaS providers can also provision a wide variety of services to accompany those infrastructure components (example services include billing software, monitoring software, logging software, load balancing software, clustering software, etc.). Therefore, because these services can be policy-driven, IaaS users can implement policies to drive load balancing to maintain application availability and performance.

[0246] In some cases, IaaS customers can access resources and services over a wide area network (WAN) (such as the Internet) and can use the cloud provider's services to install the remaining elements of the application stack. For example, a user can log in to the IaaS platform to create virtual machines (VMs), install an operating system (OS) on each VM, deploy middleware such as databases, create buckets for workloads and backups, and even install enterprise software into the VM. The customer can then use the provider's services to perform various functions, including balancing network traffic, troubleshooting application problems, monitoring performance, and managing disaster recovery.

[0247] In most cases, cloud computing models will require the involvement of cloud providers. Cloud providers can, but do not need to, be third-party providers specializing in (e.g., offering, renting, or selling) IaaS services. Entities can also choose to deploy private clouds, becoming providers of their own infrastructure services.

[0248] In some examples, IaaS deployment is the process of placing a new application or a new version of an application onto a prepared application server or similar. It may also include the process of preparing the server (e.g., installing libraries, daemons, etc.). This is typically managed by the cloud provider under a hypervisor layer (e.g., servers, storage devices, network hardware, and virtualization). Therefore, the customer can be responsible for disposal (OS), middleware, and / or application deployment (e.g., on self-service virtual machines that can be spun up on demand).

[0249] In some examples, IaaS provisioning can refer to acquiring a computer or virtual host for use, and even installing the necessary libraries or services on it. In most cases, deployment does not include provisioning, and provisioning may need to be performed first.

[0250] In some cases, there are two distinct challenges to IaaS provisioning. First, there is the initial challenge of provisioning the initial set of infrastructure before anything can run. Second, there is the challenge of evolving the existing infrastructure after everything has been provisioned (e.g., adding new services, changing services, removing services, etc.). In some cases, these challenges can be addressed by enabling the configuration of the infrastructure to be declaratively defined. In other words, the infrastructure (e.g., what components are needed and how they interact) can be defined by one or more configuration files. Therefore, the overall topology of the infrastructure can be declaratively described (e.g., which resources depend on which resources and how each of them works together). In some cases, after the topology is defined, workflows for creating and / or managing the different components described in the configuration files can be generated.

[0251] In some examples, the infrastructure can have many interconnected elements. For example, there may be one or more Virtual Private Clouds (VPCs) (e.g., potential on-demand pools of configurable and / or shared computing resources), also known as the core network. In some examples, there may also be one or more inbound / outbound traffic group rules, provisioned to define how inbound and / or outbound traffic will be structured for the network, and one or more virtual machines (VMs). Other infrastructure elements, such as load balancers, databases, etc., may also be provisioned. As more infrastructure elements are expected and / or added, the infrastructure can evolve incrementally.

[0252] In some cases, continuous deployment techniques can be used to enable the deployment of infrastructure code across various virtual computing environments. Additionally, the described techniques can enable infrastructure management within these environments. In some examples, service teams may write code that they expect to deploy to one or more (but often more) different production environments (e.g., across different geographical locations, sometimes across the world). However, in some examples, the infrastructure on which the code will be deployed must first be established. In some instances, provisioning can be done manually, provisioning tools can be used to provision resources, and / or, once the infrastructure is provisioned, deployment tools can be used to deploy the code.

[0253] Figure 16 This is a block diagram 1600 illustrating an example pattern of an IaaS architecture according to at least one embodiment. Service operator 1602 may be communicatively coupled to a secure host lease 1604, which may include a virtual cloud network (VCN) 1606 and a secure host subnet 1608. In some examples, service operator 1602 may use one or more client computing devices, which may be portable handheld devices (e.g., iPhone®, cellular phone, iPad®, computing tablet, personal digital assistant (PDA)) or wearable devices (e.g., Google Glass® head-mounted display), running software (such as Microsoft Windows Mobile®) and / or a wide variety of mobile operating systems (such as iOS, Windows Phone, Android, BlackBerry 8, Palm OS, etc.), and enabled for the Internet, email, SMS, Blackberry®, or other communication protocols. Alternatively, client computing devices may be general-purpose personal computers, for example including personal computers and / or laptops running various versions of Microsoft Windows®, Apple Macintosh®, and / or Linux operating systems. The client computing device can be a workstation computer running any of the various commercial UNIX® or UNIX-like operating systems (including, but not limited to, various GNU / Linux operating systems, such as, for example, Google Chrome OS). Alternatively or additionally, the client computing device can be any other electronic device capable of communicating over a network that can access the VCN 1606 and / or the Internet, such as a thin client computer, an Internet-enabled gaming system (e.g., a Microsoft Xbox game console with or without Kinect® gesture input), and / or a personal messaging device.

[0254] VCN 1606 may include a local peering gateway (LPG) 1610, which may be communicatively coupled to a secure shell (SSH) VCN 1612 via LPG 1610 included in SSH VCN 1612. SSH VCN 1612 may include an SSH subnet 1614, and SSH VCN 1612 may be communicatively coupled to a control plane VCN 1616 via LPG 1610 included in control plane VCN 1616. Furthermore, SSH VCN 1612 may be communicatively coupled to a data plane VCN 1618 via LPG 1610. Control plane VCN 1616 and data plane VCN 1618 may be included in a service lease 1619 that may be owned and / or operated by an IaaS provider.

[0255] The control plane VCN 1616 may include a control plane demilitarized zone (DMZ) layer 1620, which acts as a portion of the perimeter network (e.g., a corporate network between an intranet and an external network). DMZ-based servers can have limited liability and help keep breaches contained. Additionally, the DMZ layer 1620 may include one or more load balancer (LB) subnets 1622, a control plane application layer 1624 that may include one or more application subnets 1626, and a control plane data layer 1628 that may include one or more database (DB) subnets 1630 (e.g., one or more front-end DB subnets and / or one or more back-end DB subnets). One or more LB subnets 1622 contained in the control plane DMZ layer 1620 may be communicatively coupled to one or more application subnets 1626 contained in the control plane application layer 1624 and an Internet gateway 1634 that may be contained in the control plane VCN 1616. The application subnets 1626 may be communicatively coupled to one or more DB subnets 1630 and service gateways 1636 and Network Address Translation (NAT) gateways 1638 contained in the control plane data layer 1628. The control plane VCN 1616 may include service gateway 1636 and NAT gateway 1638.

[0256] Control plane VCN 1616 may include a data plane mirroring application layer 1640, which may include one or more application subnets 1626. The one or more application subnets 1626 included in the data plane mirroring application layer 1640 may include a virtual network interface controller (VNIC) 1642, which can execute a compute instance 1644. The compute instance 1644 may communicatively couple the one or more application subnets 1626 of the data plane mirroring application layer 1640 to the one or more application subnets 1626 that may be included in the data plane application layer 1646.

[0257] Data plane VCN 1618 may include a data plane application layer 1646, a data plane DMZ layer 1648, and a data plane data layer 1650. Data plane DMZ layer 1648 may include one or more application subnets 1626 communicatively coupled to data plane application layer 1646 and one or more LB subnets 1622 of Internet gateway 1634 of data plane VCN 1618. One or more application subnets 1626 may be communicatively coupled to service gateway 1636 and NAT gateway 1638 of data plane VCN 1618. Data plane data layer 1650 may also include one or more DB subnets 1630 communicatively coupled to one or more application subnets 1626 of data plane application layer 1646.

[0258] Internet gateway 1634 of control plane VCN 1616 and data plane VCN 1618 can communicatively couple to metadata management service 1652, which can communicatively couple to public internet 1654. Public internet 1654 can communicatively couple to NAT gateway 1638 of control plane VCN 1616 and data plane VCN 1618. Service gateway 1636 of control plane VCN 1616 and data plane VCN 1618 can communicatively couple to cloud service 1656.

[0259] In some examples, the service gateway 1636 of the control plane VCN 1616 or the data plane VCN 1618 can make application programming interface (API) calls to the cloud service 1656 without traversing the public internet 1654. API calls from the service gateway 1636 to the cloud service 1656 can be unidirectional: the service gateway 1636 can make API calls to the cloud service 1656, and the cloud service 1656 can send the requested data to the service gateway 1636. However, the cloud service 1656 may not initiate API calls to the service gateway 1636.

[0260] In some examples, secure host lease 1604 can be directly connected to service lease 1619, which would otherwise be isolated. Secure host subnet 1608 can communicate with SSH subnet 1614 via LPG 1610, which enables bidirectional communication over otherwise isolated systems. Connecting secure host subnet 1608 to SSH subnet 1614 grants secure host subnet 1608 access to other entities within service lease 1619.

[0261] Control plane VCN 1616 may allow users of service lease 1619 to establish or otherwise provision desired resources. The desired resources provisioned in control plane VCN 1616 may be deployed or otherwise used in data plane VCN 1618. In some examples, control plane VCN 1616 may be isolated from data plane VCN 1618, and the data plane mirror application layer 1640 of control plane VCN 1616 may communicate with the data plane application layer 1646 of data plane VCN 1618 via a VNIC 1642 that may be included in both the data plane mirror application layer 1640 and the data plane application layer 1646.

[0262] In some examples, a user or client of the system may make a request (e.g., a create, read, update, or delete (CRUD) operation) via the public internet 1654, which can then forward the request to the metadata management service 1652. The metadata management service 1652 can then forward the request to the control plane VCN 1616 via internet gateway 1634. The request may be received by one or more LB subnets 1622 contained in the control plane DMZ layer 1620. The LB subnets 1622 can determine that the request is valid, and in response to this determination, they can forward the request to one or more application subnets 1626 contained in the control plane application layer 1624. If the request is authenticated and requires a call to the public internet 1654, the call to the public internet 1654 can be forwarded to a NAT gateway 1638 that can make the call to the public internet 1654. The request may expect that the metadata to be stored can be stored in one or more DB subnets 1630.

[0263] In some examples, the data plane mirroring application layer 1640 can facilitate direct communication between the control plane VCN 1616 and the data plane VCN 1618. For example, it may be desirable to apply configuration changes, updates, or other suitable modifications to resources contained in the data plane VCN 1618. Through VNIC 1642, the control plane VCN 1616 can communicate directly with the resources contained in the data plane VCN 1618, and thereby perform configuration changes, updates, or other suitable modifications to the resources contained in the data plane VCN 1618.

[0264] In some embodiments, the control plane VCN 1616 and data plane VCN 1618 may be included in service lease 1619. In this case, the system's users or customers may not own or operate the control plane VCN 1616 or data plane VCN 1618. Conversely, the IaaS provider may own or operate both the control plane VCN 1616 and data plane VCN 1618, both of which may be included in service lease 1619. This embodiment can enable network isolation that can prevent users or customers from interacting with resources of other users or other customers. Furthermore, this embodiment can allow the system's users or customers to privately store databases without relying on the public Internet 1654, which may not have the desired level of threat prevention.

[0265] In other embodiments, one or more LB subnets 1622 included in the control plane VCN 1616 may be configured to receive signals from the service gateway 1636. In this embodiment, the control plane VCN 1616 and the data plane VCN 1618 may be configured to be invoked by the IaaS provider's customers without invoking the public internet 1654. The IaaS provider's customers may expect this embodiment because the database(s) used by the customer can be controlled by the IaaS provider and can be stored on a service lease 1619 that can be isolated from the public internet 1654.

[0266] Figure 17 This is a block diagram 1700 illustrating another example pattern of an IaaS architecture according to at least one embodiment. Service operator 1702 (e.g., Figure 16 The service provider 1602) can communicatively couple to the secure host lease 1704 (e.g., Figure 16 Secure hosting lease 1604), the secure hosting lease may include a Virtual Cloud Network (VCN) 1706 (e.g., Figure 16 VCN1606) and Secure Host Subnet 1708 (e.g., Figure 16 The secure host subnet 1608). VCN 1706 may include a local peering gateway (LPG) 1710 (e.g., Figure 16 The LPG 1610 can be communicatively coupled to the Secure Shell (SSH) VCN 1712 (e.g., LPG 1610) contained in the SSH VCN 1712. Figure 16 (SSH) VCN 1612). SSH VCN 1712 can include SSH subnet 1714 (e.g., Figure 16 SSH subnet 1614), and SSH VCN 1712 can be communicatively coupled to control plane VCN 1716 via LPG 1710 contained in control plane VCN 1716 (e.g., Figure 16 Control plane VCN 1716. Control plane VCN 1716 may be included in service lease 1719 (e.g., Figure 16 In the service lease 1619), and the data plane VCN 1718 (e.g., Figure 16 The data plane VCN 1618 can be included in a customer lease 1721 that can be owned or operated by the system's user or customer.

[0267] The control plane VCN 1716 may include one or more LB subnets 1722 (e.g., Figure 16 The control plane DMZ layer 1720 of (one or more) LB subnets 1622) (e.g., Figure 16 The control plane DMZ layer 1620 may include one or more application subnets 1726 (e.g., Figure 16 The control plane application layer 1724 of (one or more) application subnets 1626 (e.g., Figure 16 The control plane application layer 1624 may include one or more database (DB) subnets 1730 (e.g., similar to...). Figure 16 The control plane data layer 1728 of (one or more) DB subnets 1630 (e.g., Figure 16 The control plane data layer 1628). One or more LB subnets 1722 contained in the control plane DMZ layer 1720 can be communicatively coupled to one or more application subnets 1726 contained in the control plane application layer 1724 and an Internet gateway 1734 that can be contained in the control plane VCN 1716 (e.g., Figure 16 Internet gateway 1634), and application subnet(s) 1726 can communicatively couple to DB subnet(s) 1730 and service gateway 1736 contained in control plane data layer 1728 (e.g., Internet gateway 1634), and application subnet(s) 1726 can communicatively couple to DB subnet(s) 1730 and service gateway(s) ... Figure 16 Service gateway 1636) and Network Address Translation (NAT) gateway 1738 (e.g., Figure 16 (NAT gateway 1638). The control plane VCN 1716 may include the service gateway 1736 and the NAT gateway 1738.

[0268] The control plane VCN 1716 may include a data plane mirror of the application layer 1740 (e.g., Figure 16 The data plane mirroring application layer 1640 may include one or more application subnets 1726. The application subnets 1726 included in the data plane mirroring application layer 1740 may include instances 1744 capable of performing computations (e.g., similar to...). Figure 16 The virtual network interface controller (VNIC) 1742 (e.g., the VNIC of 1642) of the computing instance 1644. The computing instance 1744 may facilitate the mirroring of one or more application subnets 1726 of the data plane application layer 1740 and may be included in the data plane application layer 1746 (e.g., Figure 16 Communication between one or more application subnets 1726 in the data plane application layer 1646 via VNIC 1742 contained in the data plane mirror application layer 1740 and VNIC 1742 contained in the data plane application layer 1746.

[0269] The Internet gateway 1734, included in the control plane VCN 1716, can be communicatively coupled to the metadata management service 1752 (e.g., Figure 16 Metadata management service 1652), which can communicatively couple to the public Internet 1754 (e.g., Figure 16 The public internet 1754 can communicatively couple to a NAT gateway 1738 contained in a control plane VCN 1716. The service gateway 1736 contained in the control plane VCN 1716 can communicatively couple to a cloud service 1756 (e.g., ...). Figure 16 Cloud services (1656).

[0270] In some examples, data plane VCN 1718 may be included in customer lease 1721. In this case, the IaaS provider may provide control plane VCN 1716 for each customer, and the IaaS provider may establish a unique compute instance 1744 for each customer, included in service lease 1719. Each compute instance 1744 may allow communication between control plane VCN 1716 included in service lease 1719 and data plane VCN 1718 included in customer lease 1721. Compute instance 1744 may allow resources provisioned in control plane VCN 1716 included in service lease 1719 to be deployed or otherwise used in data plane VCN 1718 included in customer lease 1721.

[0271] In other examples, an IaaS provider's customer may have a database residing in customer lease 1721. In this example, control plane VCN 1716 may include a data plane mirroring application layer 1740, which may include one or more application subnets 1726. Data plane mirroring application layer 1740 may reside in data plane VCN 1718, but it may not reside in data plane VCN 1718. That is, data plane mirroring application layer 1740 may have access to customer lease 1721, but it may not reside in data plane VCN 1718 or be owned or operated by an IaaS provider's customer. Data plane mirroring application layer 1740 may be configured to make calls to data plane VCN 1718, but it may not be configured to make calls to any entity contained in control plane VCN 1716. Customers may expect to deploy or otherwise use resources provided in the control plane VCN 1716 in the data plane VCN 1718, and the data plane mirroring application layer 1740 can facilitate the customer's desired deployment or other use of resources.

[0272] In some embodiments, an IaaS provider's customer may apply filters to data plane VCN 1718. In this embodiment, the customer may determine what data plane VCN 1718 can access, and the customer may restrict access from data plane VCN 1718 to the public Internet 1754. The IaaS provider may not be able to apply filters or otherwise control data plane VCN 1718's access to any external networks or databases. Applying filters and controls by the customer to data plane VCN 1718, which is included in customer lease 1721, can help isolate data plane VCN 1718 from other customers and from the public Internet 1754.

[0273] In some embodiments, cloud service 1756 can be invoked by service gateway 1736 to access services that may not exist on public internet 1754, control plane VCN 1716, or data plane VCN 1718. The connection between cloud service 1756 and control plane VCN 1716 or data plane VCN 1718 may not be active or continuous. Cloud service 1756 may reside on different networks owned or operated by an IaaS provider. Cloud service 1756 may be configured to receive calls from service gateway 1736 and may be configured not to receive calls from public internet 1754. Some cloud services 1756 may be isolated from other cloud services 1756, and control plane VCN 1716 may be isolated from cloud services 1756, meaning the cloud service may not be in the same region as control plane VCN 1716. For example, control plane VCN 1716 may be located in "Region 1," and cloud service "Deployment 16" may be located in both Region 1 and "Region 2." If a call to deployment 16 is made by a service gateway 1736 contained in a control plane VCN 1716 located in region 1, the call can be forwarded to deployment 16 in region 1. In this example, the control plane VCN 1716 or deployment 16 in region 1 may be non-communicatively coupled to deployment 16 in region 2, or may otherwise communicate with deployment 16 in region 2.

[0274] Figure 18 This is a block diagram 1800 illustrating another example pattern of an IaaS architecture according to at least one embodiment. Service operator 1802 (e.g., Figure 16 The service provider 1602) can communicatively couple to the secure host lease 1804 (e.g., Figure 16 Secure hosting lease 1604), the secure hosting lease may include a Virtual Cloud Network (VCN) 1806 (e.g., Figure 16 VCN1606) and Secure Host Subnet 1808 (e.g., Figure 16 The secure host subnet 1608). VCN 1806 can include LPG 1810 (e.g., Figure 16 The LPG 1610 can be communicatively coupled to the SSH VCN 1812 via the LPG 1810 included in the SSH VCN 1812 (e.g., Figure 16 SSH VCN 1612). SSH VCN 1812 can include SSH subnet 1814 (e.g., Figure 16 SSH subnet 1614), and SSH VCN 1812 can be communicatively coupled to control plane VCN 1816 via LPG 1810 contained in control plane VCN 1816 (e.g., Figure 16 The control plane VCN 1616) and the LPG 1810 communicatively coupled to the data plane VCN 1818 (e.g., via the control plane VCN 1616) and the data plane VCN 1818 via the LPG 1810 contained in the data plane VCN 1818. Figure 16 Data plane 1618). Control plane VCN 1816 and data plane VCN 1818 may be included in service lease 1819 (e.g., Figure 16 In the service rental (1619).

[0275] The control plane VCN 1816 may include one or more load balancer (LB) subnets 1822 (e.g., Figure 16 The control plane DMZ layer 1820 of (one or more) LB subnets 1622) (e.g., Figure 16 The control plane DMZ layer 1620 may include one or more application subnets 1826 (e.g., similar to...). Figure 16 The control plane application layer 1824 of (one or more) application subnets 1626 (e.g., Figure 16 The control plane application layer 1624 may include (one or more) control plane data layers 1828 of the DB subnet 1830 (e.g., Figure 16 The control plane data layer 1628). One or more LB subnets 1822 contained in the control plane DMZ layer 1820 can be communicatively coupled to one or more application subnets 1826 contained in the control plane application layer 1824 and an Internet gateway 1834 that can be contained in the control plane VCN 1816 (e.g., Figure 16 Internet gateway 1634), and application subnet(s) 1826 can communicatively couple to DB subnet(s) 1830 contained in control plane data layer 1828 and service gateway 1836 (e.g., Figure 16 The service gateway) and Network Address Translation (NAT) gateway 1838 (e.g., Figure 16 (NAT gateway 1638). The control plane VCN 1816 may include the service gateway 1836 and the NAT gateway 1838.

[0276] The data plane VCN 1818 may include the data plane application layer 1846 (e.g., Figure 16 Data plane application layer 1646), data plane DMZ layer 1848 (e.g., Figure 16 Data plane DMZ layer 1648), and data plane data layer 1850 (e.g., Figure 16 The data plane data layer 1650. The data plane DMZ layer 1848 may include one or more LB subnets 1822, which may be communicatively coupled to one or more trusted application subnets 1860 and one or more untrusted application subnets 1862 of the data plane application layer 1846, and an Internet gateway 1834 contained in the data plane VCN 1818. One or more trusted application subnets 1860 may be communicatively coupled to a service gateway 1836 contained in the data plane VCN 1818, a NAT gateway 1838 contained in the data plane VCN 1818, and one or more DB subnets 1830 contained in the data plane data layer 1850. One or more untrusted application subnets 1862 may be communicatively coupled to a service gateway 1836 contained in the data plane VCN 1818 and one or more DB subnets 1830 contained in the data plane data layer 1850. The data plane data layer 1850 may include one or more DB subnets 1830, which may be communicatively coupled to a service gateway 1836 contained in the data plane VCN 1818.

[0277] One or more untrusted application subnets 1862 may include one or more primary VNICs 1864(1)-(N) that can be communicatively coupled to tenant virtual machines (VMs) 1866(1)-(N). Each tenant VM 1866(1)-(N) may be communicatively coupled to a corresponding application subnet 1867(1)-(N) that may be contained in a corresponding container egress VCN 1868(1)-(N), which may be contained in a corresponding customer lease 1870(1)-(N). The corresponding secondary VNIC 1872(1)-(N) may facilitate communication between one or more untrusted application subnets 1862 contained in a data plane VCN 1818 and application subnets contained in a container egress VCN 1868(1)-(N). Each container egress VCN 1868(1)-(N) may include a NAT gateway 1838 that can be communicatively coupled to the public Internet 1854 (e.g., Figure 16 The public internet (1654).

[0278] Internet gateway 1834, contained in control plane VCN 1816 and data plane VCN 1818, can be communicatively coupled to metadata management service 1852 (e.g., Figure 16 A metadata management system 1652 is provided, which can communicatively couple to the public internet 1854. The public internet 1854 can communicatively couple to a NAT gateway 1838 contained in a control plane VCN 1816 and a data plane VCN 1818. A service gateway 1836 contained in a control plane VCN 1816 and a data plane VCN 1818 can communicatively couple to a cloud service 1856.

[0279] In some embodiments, the data plane VCN 1818 can be integrated with the customer-leased 1870. This integration may be useful or desired by the IaaS provider's customers in certain situations, such as when support might be expected when executing code. Customers may provide code to run, which may be destructive, may communicate with other customer resources, or may otherwise cause adverse effects. In response, the IaaS provider may determine whether to run the code provided by the customer.

[0280] In some examples, an IaaS provider's customer can grant temporary network access to the IaaS provider and request functionality to be attached to the data plane application layer 1846. The code running this functionality can execute within VMs 1866(1)-(N), and the code may not be configured to run anywhere else on the data plane VCN 1818. Each VM 1866(1)-(N) can be connected to a customer lease 1870. A corresponding container 1871(1)-(N) contained within VMs 1866(1)-(N) can be configured to run the code. In this case, dual isolation can exist (e.g., container 1871(1)-(N) runs the code, where container 1871(1)-(N) can be contained at least within VM 1866(1)-(N), which is contained within one or more untrusted application subnets 1862), which can help prevent incorrect or otherwise unintended code from corrupting the IaaS provider's network or the networks of different customers. Containers 1871(1)-(N) may be communicatively coupled to customer lease 1870 and may be configured to send or receive data from customer lease 1870. Containers 1871(1)-(N) may not be configured to send or receive data from any other entity in the data plane VCN 1818. After the running code has completed, the IaaS provider may terminate or otherwise dispose of containers 1871(1)-(N).

[0281] In some embodiments, one or more trusted application subnets 1860 may run code owned or operated by the IaaS provider. In this embodiment, one or more trusted application subnets 1860 may be communicatively coupled to one or more DB subnets 1830 and configured to perform CRUD operations in one or more DB subnets 1830. One or more untrusted application subnets 1862 may be communicatively coupled to one or more DB subnets 1830, but in this embodiment, one or more untrusted application subnets may be configured to perform read operations in one or more DB subnets 1830. Containers 1871(1)-(N) that may be contained in each customer's VM 1866(1)-(N) and may run code from the customer may not be communicatively coupled to one or more DB subnets 1830.

[0282] In other embodiments, the control plane VCN 1816 and the data plane VCN 1818 may be coupled without direct communication. In this embodiment, direct communication between the control plane VCN 1816 and the data plane VCN 1818 may not exist. However, communication can occur indirectly through at least one method. An LPG 1810, established by an IaaS provider, can facilitate communication between the control plane VCN 1816 and the data plane VCN 1818. In another example, either the control plane VCN 1816 or the data plane VCN 1818 can make a call to the cloud service 1856 via the service gateway 1836. For example, a call from the control plane VCN 1816 to the cloud service 1856 may include a request for a service that can communicate with the data plane VCN 1818.

[0283] Figure 19 This is a block diagram 1900 illustrating another example pattern of an IaaS architecture according to at least one embodiment. Service operator 1902 (e.g., Figure 16 The service provider 1602) can communicatively couple to the secure host lease 1904 (e.g., Figure 16 Secure hosting lease 1604), the secure hosting lease may include a Virtual Cloud Network (VCN) 1906 (e.g., Figure 16 VCN1606) and Secure Host Subnet 1908 (e.g., Figure 16 The secure host subnet 1608). VCN 1906 can include LPG 1910 (e.g., Figure 16 The LPG 1610 can be communicatively coupled to the SSH VCN 1912 via the LPG 1910 included in the SSH VCN 1912 (e.g., Figure 16 SSH VCN 1612). SSH VCN 1912 can include SSH subnet 1914 (e.g., Figure 16 SSH subnet 1614), and SSH VCN 1912 can be communicatively coupled to control plane VCN 1916 via LPG 1910 contained in control plane VCN 1916 (e.g., Figure 16 The control plane VCN 1616) and the LPG 1910 communicatively coupled to the data plane VCN 1918 (e.g., via the control plane VCN 1616) and the data plane VCN 1918 via the LPG 1910 contained in the data plane VCN 1918. Figure 16 Data plane 1916. Control plane VCN 1916 and data plane VCN 1918 may be contained in service lease 1919 (e.g., Figure 16 In the service rental (1619).

[0284] The control plane VCN 1916 may include (one or more) LB subnets 1922 (e.g., Figure 16 The control plane DMZ layer 1920 of (one or more) LB subnets 1622) (e.g., Figure 16 The control plane DMZ layer 1620), may include one or more application subnets 1926 (e.g., Figure 16 The control plane application layer 1924 of (one or more) application subnets 1626 (e.g., Figure 16 The control plane application layer 1624 may include one or more DB subnets 1930 (e.g., Figure 18 The control plane data layer 1928 of (one or more) DB subnets 1830 (e.g., Figure 16 The control plane data layer 1628). One or more LB subnets 1922 contained in the control plane DMZ layer 1920 can be communicatively coupled to one or more application subnets 1926 contained in the control plane application layer 1924 and an Internet gateway 1934 that can be contained in the control plane VCN 1916 (e.g., Figure 16 Internet gateway 1634), and application subnet(s) 1926 can communicatively couple to DB subnet(s) 1930 contained in control plane data layer 1928 and to service gateway 1936 (e.g., Figure 16 The service gateway) and Network Address Translation (NAT) gateway 1938 (e.g., Figure 16 (NAT gateway 1638). The control plane VCN 1916 may include the service gateway 1936 and the NAT gateway 1938.

[0285] Data plane VCN 1918 may include data plane application layer 1946 (e.g., Figure 16 Data plane application layer 1646), data plane DMZ layer 1948 (e.g., Figure 16 Data plane DMZ layer 1648), and data plane data layer 1950 (e.g., Figure 16 The data plane data layer 1650). The data plane DMZ layer 1948 may include one or more LB subnets 1922, which may be communicatively coupled to one or more trusted application subnets 1960 of the data plane application layer 1946 (e.g., Figure 18 (one or more) trusted application subnets (1860) and (one or more) untrusted application subnets (1962) (e.g., Figure 18 The data plane includes one or more untrusted application subnets 1962 and an Internet gateway 1934 contained in data plane VCN 1918. One or more trusted application subnets 1960 may communicatively couple to a service gateway 1936 contained in data plane VCN 1918, a NAT gateway 1938 contained in data plane VCN 1918, and one or more DB subnets 1930 contained in data plane data layer 1950. One or more untrusted application subnets 1962 may communicatively couple to a service gateway 1936 contained in data plane VCN 1918 and one or more DB subnets 1930 contained in data plane data layer 1950. Data plane data layer 1950 may include one or more DB subnets 1930 that may communicatively couple to a service gateway 1936 contained in data plane VCN 1918.

[0286] One or more untrusted application subnets 1962 may include a primary VNIC 1964(1)-(N) that may be communicatively coupled to tenant virtual machines (VMs) 1966(1)-(N) residing within one or more untrusted application subnets 1962. Each tenant VM 1966(1)-(N) may run code in a corresponding container 1967(1)-(N) and may be communicatively coupled to an application subnet 1926 that may be contained in a data plane application layer 1946, which may be contained in a container egress VCN 1968. A corresponding secondary VNIC 1972(1)-(N) may facilitate communication between one or more untrusted application subnets 1962 contained in a data plane VCN 1918 and the application subnets contained in a container egress VCN 1968. The container egress VCN may include a NAT gateway 1938 that may be communicatively coupled to the public internet 1954 (e.g., Figure 16 The public internet (1654).

[0287] Internet gateway 1934, contained in control plane VCN 1916 and data plane VCN 1918, can communicatively couple to metadata management service 1952 (e.g., Figure 16 A metadata management system 1652), which provides metadata management services, can communicatively couple to the public internet 1954. The public internet 1954 can communicatively couple to a NAT gateway 1938 contained in a control plane VCN 1916 and a data plane VCN 1918. A service gateway 1936 contained in a control plane VCN 1916 and a data plane VCN 1918 can communicatively couple to a cloud service 1956.

[0288] In some examples, by Figure 19 The block diagram of the 1900 architecture can be viewed as a pattern derived from... Figure 18 The architecture illustrated in block diagram 1800 is an exception to the pattern, and may be what the IaaS provider's customers expect if the IaaS provider cannot communicate directly with the customer (e.g., in a disconnected region). The corresponding container 1967(1)-(N) contained in each customer's VM 1966(1)-(N) can be accessed by the customer in real time. Container 1967(1)-(N) can be configured to make calls to the corresponding secondary VNIC 1972(1)-(N) contained in one or more application subnets 1926 of the data plane application layer 1946, which may be contained in the container egress VCN 1968. The secondary VNIC 1972(1)-(N) can forward the calls to a NAT gateway 1938, which can forward the calls to the public internet 1954. In this example, containers 1967(1)-(N), which can be accessed by clients in real time, can be isolated from the control plane VCN 1916 and from other entities contained in the data plane VCN 1918. Containers 1967(1)-(N) can also be isolated from resources from other clients.

[0289] In other examples, a client may use container 1967(1)-(N) to invoke cloud service 1956. In this example, the client may run code requesting a service from cloud service 1956 within container 1967(1)-(N). Container 1967(1)-(N) may forward the request to a secondary VNIC 1972(1)-(N), which may forward the request to a NAT gateway, which may forward the request to the public internet 1954. The public internet 1954 may forward the request via internet gateway 1934 to one or more LB subnets 1922 contained in control plane VCN 1916. In response to determining that the request is valid, one or more LB subnets may forward the request to one or more application subnets 1926, which may forward the request to cloud service 1956 via service gateway 1936.

[0290] It should be understood that the IaaS architectures 1600, 1700, 1800, and 1900 depicted in the accompanying drawings may have other components besides those depicted. Furthermore, the embodiments shown in the drawings are merely some examples of cloud infrastructure systems that can be incorporated into embodiments of this disclosure. In some other embodiments, the IaaS system may have more or fewer components than shown in the drawings, may combine two or more components, or may have different configurations or arrangements of components.

[0291] In some embodiments, the IaaS system described herein may include a suite of application, middleware, and database service offerings delivered to customers in a self-service, subscription-based, elastically scalable, reliable, highly available, and secure manner. An example of such an IaaS system is the Oracle Cloud Infrastructure (OCI) provided by this assignee.

[0292] Figure 20 An example computer system 2000, in which various embodiments can be implemented, is illustrated. System 2000 can be used to implement any computer system described above. As shown in the figure, computer system 2000 includes a processing unit 2004 that communicates with a plurality of peripheral subsystems via a bus subsystem 2002. These peripheral subsystems may include a processing acceleration unit 2006, an I / O subsystem 2008, a storage subsystem 2018, and a communication subsystem 2024. Storage subsystem 2018 includes a tangible computer-readable storage medium 2022 and system memory 2010.

[0293] The bus subsystem 2002 provides a mechanism for enabling the various components and subsystems of the computer system 2000 to communicate with each other as intended. Although the bus subsystem 2002 is schematically shown as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses. The bus subsystem 2002 can be any of several types of bus architectures, including memory buses or memory controllers, peripheral buses, and local buses using any of a variety of bus architectures. For example, such architectures may include Industry Standard Architecture (ISA) buses, Micro Channel Architecture (MCA) buses, Enhanced ISA (EISA) buses, Video Electronics Standards Association (VESA) local buses, and Peripheral Component Interconnect (PCI) buses that may be implemented as mezzanine buses manufactured according to the IEEE P1386.1 standard.

[0294] A processing unit 2004, which may be implemented as one or more integrated circuits (e.g., a conventional microprocessor or microcontroller), controls the operation of a computer system 2000. One or more processors may be included in the processing unit 2004. These processors may include single-core or multi-core processors. In some embodiments, the processing unit 2004 may be implemented as one or more independent processing units 2032 and / or 2034, each including a single-core or multi-core processor. In other embodiments, the processing unit 2004 may also be implemented as a quad-core processing unit formed by integrating two dual-core processors into a single chip.

[0295] In various embodiments, processing unit 2004 can execute various programs in response to program code and can maintain multiple concurrently executing programs or processes. At any given time, some or all of the program code to be executed may reside in processor(s) 2004 and / or storage subsystem 2018. With appropriate programming, processor(s) 2004 can provide the various functions described above. Additionally, computer system 2000 may include processing acceleration unit 2006, which may include a digital signal processor (DSP), a dedicated processor, and / or the like.

[0296] The I / O subsystem 2008 may include user interface input devices and user interface output devices. User interface input devices may include keyboards, pointing devices such as mice or trackballs, touchpads or touchscreens integrated into the display, scroll wheels, click wheels, dial pads, buttons, switches, keypads, audio input devices with voice command recognition systems, microphones, and other types of input devices. User interface input devices may include, for example, motion sensing and / or gesture recognition devices, such as the Microsoft Kinect® motion sensor, which enables users to control and interact with input devices (such as the Microsoft Xbox® 360 game controller) using a natural user interface that employs gestures and verbal commands. User interface input devices may also include eye gesture recognition devices, such as the Google Glass® blink detector, which detects eye activity from the user (e.g., "blinking" when taking a photo and / or making menu selections) and translates the eye gesture into input on the input device (e.g., Google Glass®). Additionally, user interface input devices may include voice recognition sensing devices that enable users to interact with a voice recognition system (e.g., the Siri® navigator) via voice commands.

[0297] User interface input devices may also include, but are not limited to, 3D mice, joysticks or pointers, game controllers, and graphics tablets, as well as audio / video devices such as speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, barcode readers, 3D scanners, 3D printers, laser rangefinders, and eye-tracking devices. Additionally, user interface input devices may include, for example, medical imaging input devices such as computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), and medical ultrasound equipment. User interface input devices may also include, for example, audio input devices such as MIDI keyboards and digital musical instruments.

[0298] User interface output devices may include display subsystems, indicator lights, or non-visual displays such as audio output devices. Display subsystems may be cathode ray tubes (CRTs), flat panel devices such as those using liquid crystal displays (LCDs) or plasma displays, projection devices, touchscreens, etc. Generally, the term "output device" is used to encompass all possible types of devices and mechanisms for outputting information from the computer system 2000 to the user or other computers. For example, user interface output devices may include, but are not limited to, various display devices that visually convey text, graphics, and audio / video information, such as monitors, printers, speakers, headphones, car navigation systems, plotters, voice output devices, and modems.

[0299] Computer system 2000 may include a storage subsystem 2018 that can provide a tangible, non-transitory computer-readable storage medium for storing software and data constructs that provide the functionality of the embodiments described in this disclosure. The software may include programs, code modules, instructions, scripts, etc., which, when executed by one or more cores or processors of processing unit 2004, provide the aforementioned functionality. Storage subsystem 2018 may also provide a repository for storing data used according to this disclosure.

[0300] like Figure 20 As depicted in the example, the storage subsystem 2018 may include various components, including system memory 2010, computer-readable storage medium 2022, and computer-readable storage medium reader 2020. System memory 2010 may store program instructions that can be loaded and executed by processing unit 2004. System memory 2010 may also store data used during the execution of instructions and / or data generated during the execution of program instructions. Various types of programs may be loaded into system memory 2010, including but not limited to client applications, web browsers, middleware applications, relational database management systems (RDBMS), virtual machines, containers, etc.

[0301] System memory 2010 may also store operating system 2016. Examples of operating system 2016 may include various versions of Microsoft Windows®, Apple Macintosh® and / or Linux operating systems, various commercial UNIX® or UNIX-like operating systems (including but not limited to various GNU / Linux operating systems, Google Chrome® OS, etc.) and / or mobile operating systems such as iOS, Windows® Phone, Android® OS, BlackBerry® OS, and Palm® OS. In some implementations of computer system 2000 that execute one or more virtual machines, the virtual machine, along with its guest operating system (GOS), may be loaded into system memory 2010 and executed by one or more processors or cores of processing unit 2004.

[0302] Depending on the type of computer system 2000, the system memory 2010 can have different configurations. For example, the system memory 2010 can be volatile memory (such as random access memory (RAM)) and / or non-volatile memory (such as read-only memory (ROM), flash memory, etc.). Different types of RAM configurations can be provided, including static random access memory (SRAM), dynamic random access memory (DRAM), etc. In some implementations, the system memory 2010 may include a basic input / output system (BIOS), which contains basic routines that facilitate the transfer of information between elements within the computer system 2000, such as during startup.

[0303] Computer-readable storage medium 2022 may represent remote, local, fixed and / or removable storage devices and storage media for temporarily and / or more permanently containing and storing computer-readable information used by computer system 2000, including instructions executable by processing unit 2004 of computer system 2000.

[0304] Computer-readable storage media 2022 may include any suitable media known or used in the art, including storage media and communication media, such as, but not limited to, volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing and / or transmitting information. This may include tangible computer-readable storage media or other tangible computer-readable media such as RAM, ROM, electronically erasable programmable ROM (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile disk (DVD) or other optical storage devices, magnetic tape cassettes, magnetic tape, disk storage devices or other magnetic storage devices.

[0305] For example, computer-readable storage media 2022 may include hard disk drives that read from or write to non-removable non-volatile magnetic media, disk drives that read from or write to removable non-volatile disks, and optical disc drives that read from or write to removable non-volatile optical discs (such as CD ROMs, DVDs, and Blu-ray® discs) or other optical media. Computer-readable storage media 2022 may include, but is not limited to, Zip® drives, flash memory cards, Universal Serial Bus (USB) flash memory drives, Secure Digital (SD) cards, DVD discs, digital video tapes, etc. Computer-readable storage media 2022 may also include solid-state drives (SSDs) based on non-volatile memory (such as flash-based SSDs, enterprise flash drives, solid-state ROMs, etc.), volatile memory-based SSDs (such as solid-state RAM, dynamic RAM, static RAM, DRAM-based SSDs), magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs using a combination of DRAM and flash-based SSDs. Disk drives and their associated computer-readable media can provide non-volatile storage for computer-readable instructions, data structures, program modules and other data for a computer system 2000.

[0306] Machine-readable instructions executable by one or more processors or cores of the processing unit 2004 may be stored on a non-transitory computer-readable storage medium. A non-transitory computer-readable storage medium may include physically tangible memory or storage devices that include volatile memory storage devices and / or non-volatile memory storage devices. Examples of non-transitory computer-readable storage media include magnetic storage media (e.g., magnetic disks or magnetic tapes), optical storage media (e.g., DVDs, CDs), various types of RAM, ROM, or flash memory, hard disk drives, floppy disk drives, removable memory drives (e.g., USB drives), or other types of storage devices.

[0307] The communication subsystem 2024 provides interfaces to other computer systems and networks. The communication subsystem 2024 serves as an interface for receiving data from other systems and sending data to other systems from computer system 2000. For example, the communication subsystem 2024 may enable computer system 2000 to connect to one or more devices via the Internet. In some embodiments, the communication subsystem 2024 may include radio frequency (RF) transceiver components, global positioning system (GPS) receiver components, and / or other components for accessing wireless voice and / or data networks (e.g., using cellular phone technology, advanced data network technologies such as 3G, 4G, or EDGE (Enhanced Data Rate Global Evolution), WiFi (IEEE 802.11 series standards, or other mobile communication technologies, or any combination thereof)). In some embodiments, in addition to or instead of the wireless interface, the communication subsystem 2024 may provide wired network connectivity (e.g., Ethernet).

[0308] In some embodiments, the communication subsystem 2024 may also receive input communications on behalf of one or more users who may use the computer system 2000 in the form of structured and / or unstructured data feeds 2026, event streams 2028, event updates 2030, etc.

[0309] For example, the communication subsystem 2024 can be configured to receive data feeds 2026 in real time from users of social networks and / or other communication services, such as Twitter® feeds, Facebook® updates, web feeds such as Rich Site Summary (RSS) feeds, and / or real-time updates from one or more third-party information sources.

[0310] Additionally, the communication subsystem 2024 can also be configured to receive data in the form of a continuous data stream, which may include event streams 2028 and / or event updates 2030 of real-time events. This data may be continuous or unbounded in nature, without a definite end. Examples of applications that generate continuous data may include, for example, sensor data applications, financial stock analysis, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, vehicle traffic monitoring, etc.

[0311] The communication subsystem 2024 can also be configured to output structured and / or unstructured data feeds 2026, event streams 2028, event updates 2030, etc., to one or more databases that can communicate with one or more streaming data source computers coupled to the computer system 2000.

[0312] The computer system 2000 can be one of a variety of types, including handheld portable devices (e.g., iPhone® cellular phones, iPad® computing tablets, PDAs), wearable devices (e.g., Google Glass® head-mounted displays), PCs, workstations, mainframes, kiosks, server racks, or any other data processing system.

[0313] Due to the constantly evolving nature of computers and networks, the description of the computer system 2000 depicted in the figures is intended only as a specific example. Many other configurations with more or fewer components than the system depicted in the figures are possible. For example, custom hardware may also be used and / or specific elements may be implemented in hardware, firmware, software (including applets), or a combination thereof. Furthermore, connectivity with other computing devices, such as network input / output devices, may be employed. Based on the disclosure and teachings provided herein, those skilled in the art will understand other ways and / or methods for implementing the various embodiments.

[0314] Although specific embodiments have been described, various modifications, alterations, alternative constructions, and equivalents are also covered within the scope of this disclosure. The embodiments are not limited to operation within certain specific data processing environments, but can freely operate within multiple data processing environments. Additionally, although embodiments have been described using a specific series of transactions and steps, it will be apparent to those skilled in the art that the scope of this disclosure is not limited to the described series of transactions and steps. The various features and aspects of the above embodiments can be used individually or in combination.

[0315] Furthermore, while embodiments have been described using specific combinations of hardware and software, it should be recognized that other combinations of hardware and software are also within the scope of this disclosure. Embodiments may be implemented solely in hardware, solely in software, or using a combination thereof. The various processes described herein may be implemented on the same or different processors in any combination. Thus, where a component or module is described as being configured to perform certain operations, such a configuration may be implemented, for example, by designing electronic circuits to perform operations, by programming programmable electronic circuits (such as microprocessors), or any combination thereof. Processes may communicate using a variety of techniques, including but not limited to conventional techniques for inter-process communication, and different pairs of processes may use different techniques, or the same pair of processes may use different techniques at different times.

[0316] Therefore, the specification and drawings are to be considered illustrative rather than restrictive. However, it will be clear that additions, deletions, omissions, and other modifications and alterations may be made thereto without departing from the broader spirit and scope set forth in the claims. Thus, although specific disclosed embodiments have been described, they are not intended to be limiting. Various modifications and equivalents are within the scope of the following claims.

[0317] In the context of describing the disclosed embodiments (particularly in the context of the following claims), the terms "a," "an," and "the," and similar pronouns, should be interpreted to cover both the singular and plural, unless otherwise stated herein or obviously contradicted by the context. Unless otherwise stated, the terms "comprising," "having," "owning," and "including" should be interpreted as open-ended terms (i.e., meaning "including, but not limited to"). The term "connected to" should be interpreted as partially or wholly included, attached to, or combined with, even if something else is involved. Unless otherwise indicated herein, the enumeration of numerical ranges herein is intended only as a concise way of expressing each separate value falling within that range, and each separate value is incorporated into the specification as if it were individually enumerated herein. Unless otherwise indicated herein or obviously contradicted by the context, all methods described herein may be performed in any suitable order. Unless otherwise claimed, the use of any and all examples or exemplary language (e.g., "such as") provided herein is intended only to better illustrate the embodiments and does not limit the scope of this disclosure. Nothing in the specification should be construed as indicating that any unclaimed element is essential to the practice of this disclosure.

[0318] Unless otherwise expressly stated, disjunctive language such as the phrase "at least one of X, Y, or Z" is intended to be understood in context as generally used to refer to items, terms, etc., which can be X, Y, or Z or any combination thereof (e.g., X, Y, and / or Z). Therefore, such disjunctive language is generally not intended and should not imply that certain embodiments require the presence of at least one of X, at least one of Y, or at least one of Z.

[0319] This document describes preferred embodiments of the present disclosure, including known best modes for carrying out the present disclosure. Variations of those preferred embodiments will become apparent to those skilled in the art after reading the foregoing description. Those skilled in the art should be able to appropriately employ such variations, and the present disclosure can be practiced in addition to those specifically described herein. Therefore, the present disclosure includes all modifications and equivalents of the subject matter recited in the appended claims as permitted by applicable law. Furthermore, unless otherwise indicated herein, any combination of the foregoing elements in all possible variations is included in this disclosure.

[0320] All references cited herein, including publications, patent applications and patents, are incorporated herein by reference to the extent that each reference is individually and specifically indicated as being incorporated and set forth herein by reference in its entirety.

[0321] In the foregoing description, aspects of this disclosure have been described with reference to specific embodiments thereof; however, those skilled in the art will recognize that this disclosure is not limited thereto. The various features and aspects of the foregoing disclosure may be used alone or in combination. Furthermore, embodiments may be utilized in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of this specification. Therefore, the description and drawings are to be considered illustrative rather than restrictive.< / realm>

Claims

1. A system comprising: The first cloud environment, which includes the first infrastructure, includes: The First Cloud environment is provided by First Cloud service provider; The first infrastructure is physically connected to a second infrastructure provided by a second cloud service provider that is different from the first cloud service provider; The first infrastructure includes the first set of computing resources located within the first cloud environment; The first infrastructure is configured to form a cloud network between the first set of computing resources and the second set of computing resources of the second infrastructure located within the second cloud environment; and The cloud network is configured to use a first set of computing resources and a second set of computing resources to provide cloud services of a second cloud service provider to the customers of a first cloud service provider.

2. The system of claim 1, wherein the first infrastructure is configured to form the cloud network in response to receiving instructions from the control plane of the second infrastructure, and to establish a network connection between a first subnet of a first set of computing resources and a second subnet of a second set of computing resources, the first subnet being provided by the data plane of the first infrastructure.

3. The system of claim 2, wherein the first infrastructure is configured to establish the network connection by associating an Internet Protocol (IP) address with a first subnet, the IP address being within a Classless Inter-Domain Routing (CIDR) IP range associated with a second subnet.

4. The system of claim 1, wherein a first router associated with a first cloud environment is connected to a second router associated with a second cloud environment, and wherein the first router is also connected to a first infrastructure.

5. The system of claim 1, wherein the first infrastructure is configured to form the cloud network by connecting virtual resources of a first set of computing resources to a virtual machine cluster of a second set of computing resources.

6. The system of claim 1, wherein the first infrastructure is configured to form the cloud network by connecting a first network interface card of a first set of computing resources to a second network interface card of a second set of computing resources.

7. The system of claim 1, wherein the first set of computing resources includes one or more processors, and wherein the one or more processors are connected to the physical resources of a second cloud service provider.

8. A method comprising: Receive requests for cloud services; as well as Upon receiving the request for the cloud service, a cloud network is formed between a first set of computing resources and a second set of computing resources, wherein the first set of computing resources is located within a first infrastructure of a first cloud environment provided by a first cloud service provider, and wherein the second set of computing resources is located within a second infrastructure of a second cloud environment provided by a second cloud service provider different from the first cloud service provider, wherein the first infrastructure is physically connected to the second infrastructure, and wherein the cloud network is configured to use the first set of computing resources and the second set of computing resources to provide the cloud services of the first cloud service provider to the customers of the second cloud service provider.

9. The method of claim 8, wherein forming the cloud network includes receiving instructions from a control plane of the second infrastructure, and after receiving the instructions from the control plane, establishing a network connection between a first subnet of a first set of computing resources and a second subnet of a second set of computing resources, the first subnet being provided by a data plane of the first infrastructure, and wherein forming the cloud network enables data concerning the cloud service to be transferred from the data plane to an area in the second infrastructure.

10. The method of claim 9, wherein the Internet Protocol (IP) address associated with the first subnet is within the Classless Inter-Domain Routing (CIDR) IP range associated with the second subnet.

11. The method of claim 8, wherein a first router associated with a first cloud environment is connected to a second router associated with a second cloud environment, wherein the first router is also connected to a first infrastructure.

12. The method of claim 8, wherein forming the cloud network includes connecting virtual resources of a first set of computing resources to a virtual machine cluster of a second set of computing resources.

13. The method of claim 8, wherein forming the cloud network includes connecting a first network interface card of a first set of computing resources to a second network interface card of a second set of computing resources.

14. The method of claim 8, wherein the first set of computing resources includes one or more processors, and wherein the one or more processors are connected to the physical resources of a second cloud service provider.

15. One or more non-transitory computer-readable storage media storing instructions that, when executed by one or more processors of a computer system, cause the computer system to perform operations including: Receive requests for cloud services; and Upon receiving the request for the cloud service, a cloud network is formed between a first set of computing resources and a second set of computing resources, wherein the first set of computing resources is located within a first infrastructure of a first cloud environment provided by a first cloud service provider, and wherein the second set of computing resources is located within a second infrastructure of a second cloud environment provided by a second cloud service provider different from the first cloud service provider, wherein the first infrastructure is physically connected to the second infrastructure, and wherein the cloud network is configured to use the first set of computing resources and the second set of computing resources to provide the cloud services of the first cloud service provider to the customers of the second cloud service provider.

16. The one or more non-transitory computer-readable storage media of claim 15, wherein forming the cloud network includes receiving instructions from the control plane of the second infrastructure, and, after receiving the instructions from the control plane, establishing a network connection between a first subnet of a first set of computing resources and a second subnet of a second set of computing resources, the first subnet being provided by the data plane of the first infrastructure.

17. One or more non-transitory computer-readable storage media according to claim 16, wherein forming the cloud network enables data concerning the cloud service to be transferred from the data plane to an area in a second infrastructure.

18. One or more non-transitory computer-readable storage media of claim 15, wherein forming the cloud network includes connecting a first router associated with a first cloud environment and connected to a first infrastructure to a second router associated with a second cloud environment and connected to a second infrastructure.

19. One or more non-transitory computer-readable storage media of claim 15, wherein forming the cloud network includes connecting virtual resources of a first set of computing resources to a cluster of virtual machines of a second set of computing resources.

20. The one or more non-transitory computer-readable storage media of claim 15, wherein the first set of computing resources includes one or more processors, and wherein forming the cloud network includes physical resources connecting the one or more processors to a second cloud service provider.

Images