Network architecture for customer-specific region clouds

Abstract

A network fabric architecture for DRCC is described. The fabric includes a plurality of switch blocks. A compute fabric (CFAB) block is provided that is communicatively coupled to the plurality of switch blocks. The CFAB block includes (i) a set of one or more racks, where each rack includes one or more servers configured to execute one or more workloads of a customer, and (ii) a first plurality of switches organized into a first plurality of levels. A first plurality of switches communicatively couple a set of one or more racks to the plurality of switch blocks. A network fabric block is provided that is communicatively coupled to the plurality of switch blocks and includes (i) one or more edge devices including a first edge device (providing connectivity to a first external resource to a workload), and (ii) a second plurality of switches organized into a second plurality of levels. A second plurality of switches communicatively couple the one or more edge devices to the plurality of switch blocks.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of the filing dates of:

[0003] U.S. Provisional Application No. 63 / 398,134, filed on August 15, 2022,

[0004] U.S. Provisional Application No. 63 / 381,262, filed on October 27, 2022, and

[0005] U.S. nonprovisional application No. 18 / 360,695 filed on July 27, 2023.

[0006] Each of the applications listed above is incorporated herein by reference in its entirety. Background Art

[0007] Organizations are continually moving business applications and databases to the cloud to reduce the cost of purchasing, updating, and maintaining on-premise hardware and software. High-performance computer applications continually consume all available computing power to achieve specific outcomes or results. Such applications require dedicated network performance, fast storage, high computing power, and large amounts of memory—resources that are in short supply in the virtualized infrastructure that makes up today's commodity clouds.

[0008] However, due to the inherent mismatch between traditional application architectures and cloud architectures, enterprises often find it costly and difficult to move to cloud infrastructure. These challenges are exacerbated for workloads that cannot be moved to the public cloud. Enterprises can only access a small subset of cloud services on-premises, and these services also have only a limited set of features and capabilities compared to those available in the public cloud.

[0009] Thus, a framework is desired that brings the full power of the public cloud to on-premises deployments so that enterprises can reduce infrastructure and operating costs, upgrade legacy applications on modern cloud services, and meet the most stringent regulatory, data residency, and latency requirements—all while being enabled by the cloud service provider's infrastructure, which provides enhanced performance and the highest levels of security. The embodiments discussed herein address these and other issues. Summary of the invention

[0010] The present disclosure relates generally to cloud networks. More particularly, the present disclosure relates to a dedicated regional cloud (DRCC) at a customer location that corresponds to a cloud service provider's (CSP) infrastructure deployed in a customer's own data center. Leveraging a DRCC, an enterprise can easily integrate mission-critical database systems with applications previously deployed on expensive hardware on a CSP's high availability and security infrastructure, thereby creating operational efficiencies and modernization opportunities.

[0011] The DRCC framework brings the full power of the public cloud to on-premises deployments, so enterprises can reduce infrastructure and operational costs, upgrade legacy applications on modern cloud services, and meet the most stringent regulatory, data residency, and latency requirements—all enabled by the CSP’s infrastructure, which provides enhanced performance and the highest levels of security. Customers get the choice and flexibility to run all of the CSP’s cloud services in their data centers. Customers can choose from all of the public cloud services offered by the CSP, including VMware Cloud, Autonomous Database, Container Engine for Kubernetes, Bare Metal Servers, Exadata Cloud Services, for example, and pay only for the services they consume. The DRCC framework is designed to keep data and customer operations completely isolated from the Internet—with control plane and data plane operations kept on-premises—to help customers meet their most stringent compliance and latency requirements. With a fully managed experience and access to new capabilities the moment they become available in the public cloud, the DRCC framework provides cloud-scale security, elasticity, and scale, as well as support for mission-critical workloads with tools to gradually modernize legacy workloads.

[0012] Various embodiments are described herein, including methods, systems, non-transitory computer-readable storage media storing programs, codes, or instructions that can be executed by one or more processors, etc. These illustrative embodiments are not mentioned to limit or define the present disclosure, but to provide examples to aid in understanding the present disclosure. Additional embodiments are discussed in the detailed description, and further description is provided therein.

[0013] One aspect of the present disclosure provides a method, the method comprising: providing a plurality of switch blocks; providing a computing fabric block communicatively coupled to the plurality of switch blocks, the computing fabric block comprising: (i) a set of one or more racks, each rack in the set of one or more racks comprising one or more servers configured to execute one or more workloads of a customer, and (ii) a first plurality of switches organized into a first plurality of levels, the first plurality of switches communicatively coupling the set of one or more racks to the plurality of switch blocks; and providing a network fabric block communicatively coupled to the plurality of switch blocks, the network fabric block comprising: (i) one or more edge devices, the one or more edge devices comprising a first edge device providing connectivity to a first external resource, the first edge device enabling a workload executed by a server included in a rack in the set of one or more racks to access the first external resource, and (ii) a second plurality of switches organized into a second plurality of levels, the second plurality of switches communicatively coupling the one or more edge devices to the plurality of switch blocks.

[0014] Another aspect of the present disclosure provides a computing device, the computing device comprising: a processor; and a memory, the memory comprising instructions, which when executed by the processor causes the computing device to at least: provide a plurality of switch blocks; provide a computing fabric block communicatively coupled to the plurality of switch blocks, the computing fabric block comprising: (i) a set of one or more racks, each rack in the set of one or more racks comprising one or more servers configured to execute one or more workloads of a customer, and (ii) a first plurality of switches organized into a first plurality of levels, the first plurality of switches communicatively coupling the set of one or more racks to the plurality of switch blocks; and provide a network fabric block communicatively coupled to the plurality of switch blocks, the network fabric block comprising: (i) one or more edge devices, the one or more edge devices comprising a first edge device providing connectivity to a first external resource, the first edge device enabling a workload executed by a server included in a rack in the set of one or more racks to access the first external resource, and (ii) a second plurality of switches organized into a second plurality of levels, the second plurality of switches communicatively coupling the one or more edge devices to the plurality of switch blocks.

[0015] Another aspect of the present disclosure provides a non-transitory computer-readable medium storing specific computer-executable instructions that, when executed by a processor, cause a computer system to perform operations including: providing a plurality of switch blocks; providing a computing fabric block communicatively coupled to the plurality of switch blocks, the computing fabric block comprising: (i) a set of one or more racks, each rack in the set of one or more racks comprising one or more servers configured to execute one or more workloads of a customer, and (ii) a first plurality of switches organized into a first plurality of levels, the first plurality of switches communicatively coupling the set of one or more racks to the plurality of switch blocks; and providing a network fabric block communicatively coupled to the plurality of switch blocks, the network fabric block comprising: (i) one or more edge devices, the one or more edge devices comprising a first edge device providing connectivity to a first external resource, the first edge device enabling a workload executed by a server included in a rack in the set of one or more racks to access the first external resource, and (ii) a second plurality of switches organized into a second plurality of levels, the second plurality of switches communicatively coupling the one or more edge devices to the plurality of switch blocks. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The features, embodiments, and advantages of the present disclosure may be better understood when the following detailed description is read with reference to the accompanying drawings.

[0017] Figure 1 is a high-level diagram illustrating a distributed environment of a virtual or overlay cloud network hosted by a cloud service provider infrastructure, according to certain embodiments.

[0018] Figure 2 Depicted is a simplified architectural diagram of physical components in a physical network within a CSPI in accordance with certain embodiments.

[0019] Figure 3 An example arrangement within a CSPI is shown in accordance with certain embodiments, where a host machine is connected to multiple network virtualization devices (NVDs).

[0020] Figure 4 Depicted is connectivity between a host machine and an NVD for providing I / O virtualization to support multitenancy, according to some embodiments.

[0021] Figure 5 Depicted is a simplified block diagram of a physical network provided by CSPI in accordance with certain embodiments.

[0022] Figure 6 Depicted are configurations of multiple TORs included in a rack according to at least one embodiment.

[0023] Figure 7 Depicted is another configuration of multiple TORs included in a rack according to some embodiments.

[0024] Figure 8 Depicted is a flow diagram illustrating steps performed in provisioning availability domains to customers in accordance with at least one embodiment.

[0025] Fig. 9 Depicted is an exemplary infrastructure of a DRCC in accordance with some embodiments.

[0026] Fig.10 Depicted is a flow chart illustrating a process of providing a DRCC to a customer on-premises location, in accordance with some embodiments.

[0027] Fig.11A Depicted is a flow diagram illustrating steps performed in the transmission of packets from a computing host included in a data center to a remote host outside of the data center, in accordance with some embodiments.

[0028] Fig. 11B Depicted is a flow diagram illustrating steps performed in the transmission of packets from a remote host outside of a data center to a computing host included in a data center, in accordance with some embodiments.

[0029] Fig.12 Another exemplary infrastructure of a DRCC is depicted in accordance with some embodiments.

[0030] Fig.13 Depicted is a flow chart illustrating another process for providing DRCC to a customer on-premises location, in accordance with some embodiments.

[0031] Fig.14 Depicted is an exemplary network fabric architecture for a DRCC in accordance with some embodiments.

[0032] Fig.15 Connections between a NFAB block and a plurality of switch blocks as well as connections within a NFAB block are illustrated according to some embodiments.

[0033] Fig.16 Illustrated is an exemplary dedicated backbone network for a customer region in accordance with some embodiments.

[0034] Fig.17 Depicted is a flow diagram of a process for constructing a network structure in accordance with some embodiments.

[0035] Fig.18 is a block diagram illustrating one mode for implementing a cloud infrastructure as a service system according to at least one embodiment.

[0036] Fig.19is a block diagram illustrating another mode for implementing a cloud infrastructure as a service system according to at least one embodiment.

[0037] Fig. 20 is a block diagram illustrating another mode for implementing a cloud infrastructure as a service system according to at least one embodiment.

[0038] Fig.21 is a block diagram illustrating another mode for implementing a cloud infrastructure as a service system according to at least one embodiment.

[0039] Fig. 22 is a block diagram illustrating an example computer system in accordance with at least one embodiment. DETAILED DESCRIPTION

[0040] In the following description, for the purpose of explanation, specific details are set forth in order to provide a thorough understanding of certain embodiments. However, it is apparent that various embodiments can be practiced without these specific details. The drawings and descriptions are not intended to be limiting. The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment or design described herein as "exemplary" is not necessarily to be construed as preferred or superior to other embodiments or designs.

[0041] Example architecture for cloud infrastructure

[0042] The term "cloud service" is generally used to refer to services provided on-demand (e.g., via a subscription model) by a cloud service provider (CSP) to users or customers using systems and infrastructure (cloud infrastructure) provided by the CSP. Typically, the servers and systems that make up the CSP's infrastructure are separate from the customer's own on-premises servers and systems. Therefore, customers can take advantage of the cloud services provided by the CSP themselves without having to purchase separate hardware and software resources for the service. Cloud services are designed to provide subscribing customers with simple, scalable access to applications and computing resources without requiring customers to invest in the infrastructure used to provide the service.

[0043] There are several cloud service providers that provide various types of cloud services. There are various different types or models of cloud services including Software as a Service (SaaS), Platform as a Service (PaaS), Infrastructure as a Service (IaaS), etc.

[0044] A customer can subscribe to one or more cloud services provided by a CSP. A customer can be any entity, such as an individual, organization, enterprise, etc. When a customer subscribes to or registers for a service provided by a CSP, a lease or account is created for the customer. The customer can then access one or more subscribed cloud resources associated with the account via this account.

[0045] As mentioned above, Infrastructure as a Service (IaaS) is a specific type of cloud computing service. In the IaaS model, the CSP provides infrastructure (called Cloud Service Provider Infrastructure or CSPI) that customers can use to build their own customizable networks and deploy customer resources. Therefore, the customer's resources and network are hosted in a distributed environment by the infrastructure provided by the CSP. This is different from traditional computing, in which the customer's resources and network are hosted by the infrastructure provided by the customer.

[0046] CSPI may include interconnected high-performance computing resources, including various host machines, memory resources, and network resources that form a physical network, which is also referred to as a base network or underlying network. The resources in CSPI may be spread across one or more data centers, which may be geographically spread across one or more geographic regions. Virtualization software may be executed by these physical resources to provide a virtualized distributed environment. Virtualization creates an overlay network (also referred to as a software-based network, a software-defined network, or a virtual network) on a physical network. The CSPI physical network provides an underlying foundation for creating one or more overlay or virtual networks on top of the physical network. A virtual or overlay network may include one or more virtual cloud networks (VCNs). A virtual network is implemented using software virtualization technology (e.g., a hypervisor, functions performed by a network virtualization device (NVD) (e.g., smartNIC), a top-of-rack (TOR) switch, an intelligent TOR that implements one or more functions performed by the NVD, and other mechanisms) to create a layer of network abstraction that can run on top of a physical network. A virtual network may take a variety of forms, including peer-to-peer networks, IP networks, and the like. A virtual network is typically either a layer 3 IP network or a layer 2 VLAN. This method of virtual or overlay networking is often referred to as virtual or overlay layer 3 networking. Examples of protocols developed for virtual networks include IP-in-IP (or Generic Routing Encapsulation (GRE)), Virtual Extensible LAN (VXLAN - IETF RFC7348), Virtual Private Network (VPN) (e.g., MPLS Layer 3 Virtual Private Network (RFC 4364)), VMware's NSX, GENEVE (Generic Network Virtualization Encapsulation), etc.

[0047] For IaaS, the infrastructure (CSPI) provided by the CSP can be configured to provide virtualized computing resources over a public network (e.g., the Internet). In the IaaS model, cloud computing service providers can host infrastructure components (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer), etc.). In some cases, IaaS providers can also supply various services to accompany those infrastructure components (e.g., billing, monitoring, logging, security, load balancing, and clustering, etc.). Therefore, since these services can be policy-driven, IaaS users can implement policies to drive load balancing to maintain application availability and performance. CSPI provides infrastructure and a set of complementary cloud services that enable customers to build and run a wide range of applications and services in a highly available, hosted, distributed environment. CSPI provides high-performance computing resources and capabilities and storage capacity in a flexible virtual network that can be securely accessed from a variety of networked locations (such as from a customer's on-premises network). When a customer subscribes to or signs up for an IaaS service provided by a CSP, the tenancy created for that customer is a secure and isolated partition within CSPI where the customer can create, organize, and manage their cloud resources.

[0048] Customers can build their own virtual networks using the computing, storage, and networking resources provided by CSPI. One or more customer resources or workloads, such as computing instances, can be deployed on these virtual networks. For example, customers can use the resources provided by CSPI to build (one or more) customizable and private virtual networks, called virtual cloud networks (VCNs). Customers can deploy one or more customer resources, such as computing instances, on customer VCNs. Computing instances can take the form of virtual machines, bare metal instances, etc. Therefore, CSPI provides infrastructure and a set of complementary cloud services that enable customers to build and run a wide range of applications and services in a highly available virtual hosted environment. Customers do not manage or control the underlying physical resources provided by CSPI, but have control over operating systems, storage, and deployed applications; and may have limited control over selected networking components (e.g., firewalls).

[0049] The CSP may provide a console that enables customers and network administrators to use CSPI resources to configure, access, and manage resources deployed in the cloud. In some embodiments, the console provides a web-based user interface that can be used to access and manage CSPI. In some implementations, the console is a web-based application provided by the CSP.

[0050] CSPI can support single-tenancy or multi-tenancy architecture. In a single-tenancy architecture, software (e.g., application, database) or hardware component (e.g., host machine or server) provides services to a single customer or tenant. In a multi-tenancy architecture, software or hardware component provides services to multiple customers or tenants. Therefore, in a multi-tenancy architecture, CSPI resources are shared among multiple customers or tenants. In the case of multi-tenancy, precautions are taken and protections are implemented in CSPI to ensure that each tenant's data is isolated from other tenants and remains invisible to other tenants.

[0051] In a physical network, a network endpoint ("endpoint") refers to a computing device or system that is connected to a physical network and communicates back and forth with the network to which it is connected. A network endpoint in a physical network can be connected to a local area network (LAN), a wide area network (WAN), or other types of physical networks. Examples of traditional endpoints in a physical network include modems, hubs, bridges, switches, routers and other network devices, physical computers (or host machines), etc. Each physical device in a physical network has a fixed network address that can be used to communicate with the device. This fixed network address can be a layer 2 address (e.g., a MAC address), a fixed layer 3 address (e.g., an IP address), etc. In a virtualized environment or virtual network, endpoints can include various virtual endpoints, such as virtual machines hosted by components of the physical network (e.g., hosted by a physical host machine). These endpoints in a virtual network are addressed by overlay addresses, such as overlay layer 2 addresses (e.g., overlay MAC addresses) and overlay layer 3 addresses (e.g., overlay IP addresses). Network overlays achieve flexibility by allowing network administrators to move around overlay addresses associated with network endpoints using software management (e.g., via software that implements a control plane for a virtual network). Thus, unlike in a physical network, in a virtual network, an overlay address (e.g., an overlay IP address) can be moved from one endpoint to another using network management software. Since the virtual network is built on top of the physical network, the communication between components in the virtual network involves both the virtual network and the underlying physical network. To facilitate such communication, the components of CSPI are configured to learn and store mappings that map overlay addresses in the virtual network to actual physical addresses in the base network, and vice versa. These mappings are then used to facilitate communication. Encapsulate customer traffic to facilitate routing in the virtual network.

[0052] Thus, a physical address (e.g., a physical IP address) is associated with a component in a physical network, and an overlay address (e.g., an overlay IP address) is associated with an entity in a virtual network. Both a physical IP address and an overlay IP address are types of real IP addresses. They are separate from virtual IP addresses, which are mapped to multiple real IP addresses. A virtual IP address provides a one-to-many mapping between a virtual IP address and multiple real IP addresses.

[0053] The cloud infrastructure or CSPI is physically hosted in one or more data centers in one or more regions of the world. The CSPI may include components in a physical or base network and virtualized components (e.g., virtual networks, computing instances, virtual machines, etc.) located in a virtual network built on top of the physical network components. In some embodiments, the CSPI is organized and hosted in domains, regions, and availability domains. A region is typically a localized geographic area containing one or more data centers. Regions are generally independent of each other and can be far apart, for example, across countries or even continents. For example, a first region may be in Australia, another in Japan, another in India, and so on. CSPI resources are divided between regions so that each region has its own independent subset of CSPI resources. Each region can provide a set of core infrastructure services and resources, such as computing resources (e.g., bare metal servers, virtual machines, containers and related infrastructure, etc.); storage resources (e.g., block volume storage, file storage, object storage, archive storage); networking resources (e.g., virtual cloud network (VCN), load balancing resources, connections to on-premises networks), database resources; edge networking resources (e.g., DNS); and access management and monitoring resources, etc. Each region generally has multiple paths connecting it to other regions in the domain.

[0054] Generally speaking, an application is deployed in the region where it is most frequently used (i.e., on the infrastructure associated with that region) because it is faster to use nearby resources than to use distant resources. Applications may also be deployed in different regions for various reasons, such as redundancy to mitigate the risk of region-wide events (such as large weather systems or earthquakes), to meet different requirements of legal jurisdictions, tax domains, and other business or social standards, etc.

[0055] Data centers within a region can be further organized and subdivided into availability domains (ADs). An availability domain can correspond to one or more data centers located within a region. A region can consist of one or more availability domains. In this distributed environment, CSPI resources are either region-specific (such as a virtual cloud network (VCN)) or availability domain-specific (such as a compute instance).

[0056] ADs within a region are isolated from each other, fault-tolerant, and configured so that they are extremely unlikely to fail at the same time. This is achieved by ADs not sharing key infrastructure resources (such as networking, physical cables, cable paths, cable entry points, etc.), making it unlikely that a failure at one AD within a region will affect the availability of other ADs in the same region. ADs within the same region can be connected to each other through a low-latency, high-bandwidth network, which makes it possible to provide high-availability connectivity to other networks (e.g., the Internet, a customer's on-premises network, etc.) and to build replication systems in multiple ADs to achieve both high availability and disaster recovery. Cloud services use multiple ADs to ensure high availability and prevent resource failures. As the infrastructure provided by the IaaS provider grows, more regions and ADs can be added, as well as additional capacity. Traffic between availability domains is typically encrypted.

[0057] In some embodiments, regions are grouped into domains. A domain is a logical collection of regions. Domains are isolated from each other and do not share any data. Regions in the same domain can communicate with each other, but regions in different domains cannot. A customer's lease or account with a CSP exists in a single domain and can be spread across one or more regions that belong to that domain. Typically, when a customer subscribes to an IaaS service, a lease or account is created for the customer in a customer-specified region within a domain (called the "home" region). A customer can extend the customer's lease across one or more other regions within a domain. A customer cannot access regions that are not in the domain where the customer's lease is located.

[0058] An IaaS provider may offer multiple domains, each domain catering to a specific set of customers or users. For example, a business domain may be provided for business customers. As another example, a domain may be provided for a specific country for customers within that country. As yet another example, a government domain may be provided for a government, etc. For example, a government domain may cater to a specific government and may have a higher level of security than a business domain. For example, Oracle Cloud Infrastructure (OCI) currently offers a domain for a business region and two domains (e.g., FedRAMP-authorized and IL5-authorized) for a government cloud region.

[0059] In some embodiments, an AD may be subdivided into one or more fault domains. A fault domain is a grouping of infrastructure resources within an AD to provide anti-affinity. Fault domains allow for the distribution of compute instances so that they are not located on the same physical hardware within a single AD. This is called anti-affinity. A fault domain refers to a group of hardware components (computers, switches, etc.) that share a single point of failure. The compute pool is logically divided into fault domains. Therefore, a hardware failure or computing hardware maintenance event that affects one fault domain does not affect instances in other fault domains. Depending on the embodiment, the number of fault domains used for each AD may vary. For example, in some embodiments, each AD contains three fault domains. Fault domains act as logical data centers within an AD.

[0060] When a customer subscribes to an IaaS service, resources from CSPI are supplied to the customer and associated with the customer's lease. Customers can use these supplied resources to build private networks and deploy resources on these networks. Customer networks hosted by CSPI in the cloud are called virtual cloud networks (VCNs). Customers can use CSPI resources allocated to customers to establish one or more virtual cloud networks (VCNs). VCNs are virtual or software-defined private networks. Customer resources deployed in a customer's VCN may include computing instances (e.g., virtual machines, bare metal instances) and other resources. These computing instances may represent various customer workloads, such as applications, load balancers, databases, and the like. Computing instances deployed on VCNs can communicate with publicly accessible endpoints ("public endpoints"), with other instances in the same VCN or other VCNs (e.g., other VCNs of customers or VCNs that do not belong to customers), with customers' on-premises data centers or networks, and with service endpoints and other types of endpoints through public networks (such as the Internet).

[0061] CSPs can use CSPI to provide various services. In some cases, customers of CSPI can themselves act like service providers and use CSPI resources to provide services. Service providers can expose service endpoints, which are characterized by identification information (e.g., IP addresses, DNS names, and ports). Customers' resources (e.g., computing instances) can consume a particular service by accessing the service endpoints exposed by the service for that particular service. These service endpoints are generally endpoints that are publicly accessible to users via public communication networks (such as the Internet) using public IP addresses associated with the endpoints. Publicly accessible network endpoints are sometimes also referred to as public endpoints.

[0062] In some embodiments, a service provider may expose a service via an endpoint for the service (sometimes referred to as a service endpoint). Customers of the service may then use this service endpoint to access the service. In some implementations, the service endpoint provided for a service may be accessed by multiple customers that intend to consume the service. In other implementations, a dedicated service endpoint may be provided to a customer so that only the customer can access the service using the dedicated service endpoint.

[0063] In certain embodiments, when a VCN is created, it is associated with a private overlay Classless Inter-Domain Routing (CIDR) address space, which is a range of private overlay IP addresses assigned to the VCN (e.g., 10.0 / 16). The VCN includes associated subnets, routing tables, and gateways. The VCN resides within a single region but can span one or more or all of the availability domains in the region. A gateway is a virtual interface configured for the VCN and enables communication of traffic to and from the VCN and between one or more endpoints external to the VCN. One or more different types of gateways can be configured for the VCN to enable communication to and from different types of endpoints.

[0064] A VCN may be subdivided into one or more subnetworks, such as one or more subnets. Thus, a subnet is a unit or subdivision of a configuration that can be created within a VCN. A VCN may have one or more subnets. Each subnet within a VCN is associated with a contiguous range of overlay IP addresses (e.g., 10.0.0.0 / 24 and 10.0.1.0 / 24) that do not overlap with other subnets in that VCN and represent a subset of the address space within the address space of the VCN.

[0065] Each computing instance is associated with a virtual network interface card (VNIC), which enables the computing instance to participate in the subnet of the VCN. VNIC is a logical representation of a physical network interface card (NIC). In general, a VNIC is an interface between an entity (e.g., a computing instance, a service) and a virtual network. VNICs exist in subnets, have one or more associated IP addresses, and associated security rules or policies. VNICs are equivalent to layer 2 ports on switches. VNICs are attached to subnets within computing instances and VCNs. The VNICs associated with computing instances make the computing instances part of the subnet of the VCN, and enable the computing instances to communicate (e.g., send and receive packets) with endpoints located on the same subnet as the computing instance, with endpoints in different subnets in the VCN, or with endpoints outside the VCN. Therefore, the VNICs associated with computing instances determine how the computing instances are connected to endpoints inside and outside the VCN. When a computing instance is created and added to a subnet within the VCN, a VNIC for the computing instance is created and associated with the computing instance. For a subnet that includes a group of computing instances, the subnet includes VNICs corresponding to the group of computing instances, and each VNIC is attached to a computing instance within the group of computing instances.

[0066] A private overlay IP address is assigned to each computing instance via the VNIC associated with the computing instance. This private overlay network IP address is assigned to the VNIC associated with the computing instance when the computing instance is created and is used to route traffic to and from the computing instance. All VNICs in a given subnet use the same routing table, security list, and DHCP options. As described above, each subnet within the VCN is associated with a continuous range of overlay IP addresses (e.g., 10.0.0.0 / 24 and 10.0.1.0 / 24), which do not overlap with other subnets in the VCN and represent a subset of the address space within the address space of the VCN. For a VNIC on a specific subnet of the VCN, the private overlay IP address assigned to the VNIC is an address from the continuous range of overlay IP addresses allocated for the subnet.

[0067] In some embodiments, in addition to the private overlay IP address, the computing instance may optionally be assigned additional overlay IP addresses, such as, for example, one or more public IP addresses if in a public subnet. These multiple addresses are assigned either on the same VNIC or on multiple VNICs associated with the computing instance. However, each instance has a primary VNIC, which is created during instance startup and is associated with the overlay private IP address assigned to the instance-this primary VNIC cannot be removed. Additional VNICs, called auxiliary VNICs, can be added to existing instances in the same availability domain as the primary VNIC. All VNICs are in the same availability domain as the instance. The auxiliary VNIC may be located in a subnet in the same VCN as the primary VNIC, or in a different subnet in the same VCN or in a different VCN.

[0068] If the compute instances are in a public subnet, they can optionally be assigned public IP addresses. When creating a subnet, you can specify the subnet as either a public subnet or a private subnet. A private subnet means that the resources (e.g., compute instances) and associated VNICs in the subnet cannot have public overlay IP addresses. A public subnet means that the resources and associated VNICs in the subnet can have public IP addresses. Customers can specify that a subnet exists in a single availability domain or across multiple availability domains in a region or domain.

[0069] As described above, a VCN can be subdivided into one or more subnets. In some embodiments, a virtual router (VR) configured for the VCN (referred to as a VCN VR or simply a VR) enables communication between subnets of the VCN. For a subnet within a VCN, the VR represents a logical gateway for the subnet, which enables the subnet (i.e., the computing instances on the subnet) to communicate with endpoints on other subnets within the VCN and other endpoints outside the VCN. The VCN VR is configured as a logical entity that routes traffic between VNICs in the VCN and a virtual gateway ("gateway") associated with the VCN. Figure 1The gateway is further described. The VCN VR is a layer 3 / IP layer concept. In one embodiment, there is one VCN VR for the VCN, where the VCN VR has a potentially unlimited number of ports addressed by IP addresses, one port for each subnet of the VCN. In this way, the VCN VR has a different IP address for each subnet in the VCN to which the VCN VR is attached. The VR is also connected to various gateways configured for the VCN. In some embodiments, a specific overlay IP address in the overlay IP address range for a subnet is reserved for the port of the VCN VR for that subnet. For example, consider that a VCN has two subnets with associated address ranges 10.0 / 16 and 10.1 / 16, respectively. For the first subnet in the VCN with an address range of 10.0 / 16, addresses in this range are reserved for the ports of the VCN VR for that subnet. In some cases, the first IP address in the range can be reserved for the VCN VR. For example, for a subnet with an overlay IP address range of 10.0 / 16, the IP address 10.0.0.1 can be reserved for the port of the VCN VR for that subnet. For a second subnet in the same VCN with an address range of 10.1 / 16, the VCN VR can have a port for the second subnet with an IP address of 10.1.0.1. The VCN VR has a different IP address for each subnet in the VCN.

[0070] In some other embodiments, each subnet within a VCN may have its own associated VR that is addressable by the subnet using a reserved or default IP address associated with the VR. The reserved or default IP address may, for example, be the first IP address in a range of IP addresses associated with the subnet. The VNICs in the subnet may communicate (e.g., send and receive packets) with the VRs associated with the subnet using this default or reserved IP address. In such embodiments, the VR is the entry / exit point for the subnet. A VR associated with a subnet within the VCN may communicate with other VRs associated with other subnets within the VCN. The VR may also communicate with a gateway associated with the VCN. The VR functions for the subnet run on or are performed by one or more NVDs that perform VNIC functions for the VNICs in the subnet.

[0071] Routing tables, security rules, and DHCP options can be configured for a VCN. A routing table is a virtual routing table for a VCN and includes rules for routing traffic from subnets within the VCN to destinations outside the VCN by means of gateways or specially configured instances. A VCN's routing table can be customized to control how packets are forwarded / routed to and from the VCN. DHCP options refer to configuration information that is automatically provided to an instance when it boots up.

[0072] The security rules configured for a VCN represent the overlay firewall rules for the VCN. Security rules can include ingress and egress rules and specify the type of traffic (e.g., based on protocol and port) that is allowed in and out of the VCN instances. Customers can choose whether a given rule is stateful or stateless. For example, a customer can allow incoming SSH traffic from anywhere to a group of instances by setting up a stateful ingress rule with source CIDR 0.0.0.0 / 0 and destination TCP port 22. Security rules can be implemented using network security groups or security lists. A network security group consists of a set of security rules that apply only to the resources in that group. On the other hand, a security list includes rules that apply to all resources in any subnet that uses the security list. A default security list with default security rules can be provided for a VCN. The DHCP options configured for a VCN provide configuration information that is automatically provided to instances in the VCN when the instances are launched.

[0073] In some embodiments, configuration information for a VCN is determined and stored by a VCN control plane. For example, configuration information for a VCN may include information about address ranges associated with the VCN, subnets within the VCN and associated information, one or more VRs associated with the VCN, compute instances in the VCN and associated VNICs, NVDs that perform various virtualized network functions associated with the VCN (e.g., VNICs, VRs, gateways), state information for the VCN, and other VCN-related information. In some embodiments, a VCN distribution service publishes configuration information stored by a VCN control plane or a portion thereof to the NVD. The distributed information may be used to update information stored and used by the NVD (e.g., forwarding tables, routing tables, etc.) to forward packets to and from compute instances in the VCN.

[0074] In some embodiments, the creation of VCNs and subnets is handled by the VCN control plane (CP) and the launch of compute instances is handled by the compute control plane. The compute control plane is responsible for allocating physical resources to the compute instances and then calling the VCN control plane to create and attach VNICs to the compute instances. The VCN CP also sends VCN data maps to the VCN data plane, which is configured to perform packet forwarding and routing functions. In some embodiments, the VCN CP provides a distribution service that is responsible for providing updates to the VCN data plane. An example of a VCN control plane is also provided at Fig.18 , Fig.19 , Fig. 20 and Fig.21 Depicted in (see reference numerals 1816, 1916, 2016 and 2116) and described below.

[0075] Customers can create one or more VCNs using resources hosted by CSPI. Compute instances deployed on customer VCNs can communicate with different endpoints. These endpoints can include endpoints hosted by CSPI and endpoints external to CSPI.

[0076] Figure 1 , Figure 2 , Figure 3 , Figure 4 , Figure 5 , Fig.18 , Fig.19 , Fig. 20 and Fig.21 Various different architectures for implementing cloud-based services using CSPI are depicted in and are described below. Figure 1 is a high-level diagram illustrating a distributed environment 100 of an overlay or customer VCN hosted by CSPI in accordance with certain embodiments. Figure 1 The distributed environment depicted in includes multiple components in an overlay network. Figure 1 The distributed environment 100 depicted in FIG. 1 is merely an example and is not intended to unduly limit the scope of the claimed embodiments. Many variations, substitutions, and modifications are possible. For example, in some embodiments, Figure 1 The distributed environment depicted in Figure 1 More or fewer systems or components than those shown in the drawings may be used, two or more systems may be combined, or there may be different configurations or arrangements of the systems.

[0077] like Figure 1 As shown in the example depicted in , the distributed environment 100 includes a CSPI 101 that provides services and resources that customers can subscribe to and use to build their virtual cloud network (VCN). In some embodiments, CSPI 101 provides IaaS services to subscribing customers. Data centers within CSPI 101 can be organized into one or more regions. Figure 1 An example region "Region US" 102 is shown in FIG. A customer has configured a customer VCN 104 for region 102. The customer can deploy various computing instances on VCN 104, where the computing instances can include virtual machines or bare metal instances. Examples of instances include applications, databases, load balancers, etc.

[0078] exist Figure 1 In the embodiment depicted in , customer VCN 104 includes two subnets, namely, "Subnet-1" and "Subnet-2", each with its own CIDR IP address range. Figure 1, the coverage IP address range of subnet-1 is 10.0 / 16, and the address range of subnet-2 is 10.1 / 16. VCN virtual router 105 represents a logical gateway for the VCN that enables communication between subnets of VCN 104 and with other endpoints outside the VCN. VCN VR 105 is configured to route traffic between VNICs in VCN 104 and gateways associated with VCN 104. VCN VR 105 provides ports for each subnet of VCN 104. For example, VR 105 can provide a port with IP address 10.0.0.1 for subnet-1 and a port with IP address 10.1.0.1 for subnet-2.

[0079] Multiple computing instances can be deployed on each subnet, where the computing instances can be virtual machine instances and / or bare metal instances. The computing instances in a subnet can be hosted by one or more host machines within CSPI 101. The computing instances participate in the subnet via the VNICs associated with the computing instances. For example, Figure 1 As shown in , compute instance C1 becomes part of subnet-1 via a VNIC associated with the compute instance. Similarly, compute instance C2 becomes part of subnet-1 via a VNIC associated with C2. In a similar manner, multiple compute instances (which may be virtual machine instances or bare metal instances) may be part of subnet-1. Each compute instance is assigned a private overlay IP address and MAC address via its associated VNIC. For example, in Figure 1 , the overlay IP address of compute instance C1 is 10.0.0.2 and the MAC address is M1, while the private overlay IP address of compute instance C2 is 10.0.0.3 and the MAC address is M2. Each compute instance in subnet-1 (including compute instances C1 and C2) has a default route to VCN VR 105 using IP address 10.0.0.1, which is the IP address of the port of VCN VR 105 for subnet-1.

[0080] Multiple computing instances can be deployed on subnet-2, including virtual machine instances and / or bare metal instances. Figure 1 As shown in , compute instances D1 and D2 become part of subnet-2 via the VNICs associated with the respective compute instances. Figure 1In the embodiment depicted in , the overlay IP address of compute instance D1 is 10.1.0.2 and the MAC address is MM1, while the private overlay IP address of compute instance D2 is 10.1.0.3 and the MAC address is MM2. Each compute instance in subnet-2 (including compute instances D1 and D2) has a default route to VCN VR 105 using IP address 10.1.0.1, which is the IP address of the port of VCN VR 105 for subnet-2.

[0081] VCN A 104 may also include one or more load balancers. For example, a load balancer may be provided for a subnet, and the load balancer may be configured to load balance traffic across multiple compute instances on the subnet. A load balancer may also be provided to load balance traffic across subnets in the VCN.

[0082] A particular computing instance deployed on VCN 104 can communicate with a variety of different endpoints. These endpoints can include endpoints hosted by CSPI 200 and endpoints external to CSPI 200. Endpoints hosted by CSPI 101 can include: endpoints on the same subnet as a particular computing instance (e.g., communication between two computing instances in subnet-1); endpoints on different subnets but within the same VCN (e.g., communication between a computing instance in subnet-1 and a computing instance in subnet-2); endpoints in different VCNs in the same region (e.g., communication between a computing instance in subnet-1 and an endpoint in a VCN in the same region 106 or 110, communication between a computing instance in subnet-1 and an endpoint in a service network 110 in the same region); or endpoints in VCNs in different regions (e.g., communication between a computing instance in subnet-1 and an endpoint in a VCN in a different region 108). Compute instances in a subnet hosted by CSPI 101 can also communicate with endpoints that are not hosted by CSPI 101 (i.e., external to CSPI 101). These external endpoints include endpoints in a customer's on-premises network 116, endpoints within other remote cloud-hosted networks 118, public endpoints 114 accessible via a public network (such as the Internet), and other endpoints.

[0083] Use VNICs associated with source and destination computing instances to facilitate communication between computing instances on the same subnet. For example, computing instance C1 in subnet-1 may want to send a packet to computing instance C2 in subnet-1. For a packet originating from a source computing instance and whose destination is another computing instance in the same subnet, the packet is first processed by the VNIC associated with the source computing instance. The processing performed by the VNIC associated with the source computing instance may include determining the destination information of the packet from the packet header, identifying any policies (e.g., security lists) configured for the VNIC associated with the source computing instance, determining the next hop for the packet, performing any packet encapsulation / decapsulation functions as needed, and then forwarding / routing the packet to the next hop, with the purpose of facilitating communication of the packet to its intended destination. When the destination computing instance is in the same subnet as the source computing instance, the VNIC associated with the source computing instance is configured to identify the VNIC associated with the destination computing instance and forward the packet to the VNIC for processing. The VNIC associated with the destination computing instance is then executed and the packet is forwarded to the destination computing instance.

[0084] For packets to be transmitted from a compute instance in a subnet to an endpoint in a different subnet in the same VCN, communication is facilitated through the VNICs associated with the source and destination compute instances and the VCN VRs. Figure 1 If compute instance C1 in subnet-1 wants to send a packet to compute instance D1 in subnet-2, the packet is first processed by the VNIC associated with compute instance C1. The VNIC associated with compute instance C1 is configured to route the packet to VCN VR 105 using the default route or port 10.0.0.1 of the VCN VR. VCN VR 105 is configured to route the packet to subnet-2 using port 10.1.0.1. The VNIC associated with D1 then receives and processes the packet and the VNIC forwards the packet to compute instance D1.

[0085] For packets to be transmitted from a compute instance in VCN 104 to an endpoint outside of VCN 104, communication is facilitated by a VNIC associated with the source compute instance, a VCN VR 105, and a gateway associated with VCN 104. One or more types of gateways may be associated with VCN 104. A gateway is an interface between a VCN and another endpoint, where the other endpoint is outside of the VCN. A gateway is a layer 3 / IP layer concept and enables a VCN to communicate with an endpoint outside of the VCN. Thus, a gateway facilitates the flow of traffic between a VCN and other VCNs or networks. A variety of different types of gateways may be configured for a VCN to facilitate different types of communications with different types of endpoints. Depending on the gateway, communications may be conducted over a public network (e.g., the Internet) or over a private network. Various communication protocols may be used for these communications.

[0086] For example, compute instance C1 may want to communicate with an endpoint outside of VCN 104. The packet may first be processed by a VNIC associated with source compute instance C1. The VNIC processing determines that the destination of the packet is outside of C1's subnet-1. The VNIC associated with C1 may forward the packet to VCN VR 105 for VCN 104. VCN VR 105 then processes the packet and, as part of the processing, determines a specific gateway associated with VCN 104 as a next hop for the packet based on the packet's destination. VCN VR 105 may then forward the packet to the specific identified gateway. For example, if the destination is an endpoint within a customer's on-premises network, the packet may be forwarded by VCN VR 105 to a dynamic routing gateway (DRG) gateway 122 configured for VCN 104. The packet may then be forwarded from the gateway to the next hop to facilitate delivery of the packet to its final intended destination.

[0087] You can configure a VCN with different types of gateways. Figure 1 Examples of gateways that may be configured for a VCN are depicted in and described below. Examples of gateways associated with a VCN are also described in Fig.18 , Fig.19 , Fig. 20 and Fig.21 1834, 1836, 1838, 1934, 1936, 1938, 2034, 2036, 2038, 2134, 2136, and 2138) and are described below. Figure 1As shown in the embodiment depicted in , a dynamic routing gateway (DRG) 122 can be added to or associated with a customer VCN 104 and provides a path for private network traffic communication between the customer VCN 104 and another endpoint, where the other endpoint can be a customer's on-premises network 116, a VCN 108 in a different region of CSPI 101, or other remote cloud network 118 not hosted by CSPI 101. The customer's on-premises network 116 can be a customer network or a customer data center built using the customer's resources. Access to the customer's on-premises network 116 is generally very restricted. For customers who have both a customer's on-premises network 116 and one or more VCNs 104 deployed or hosted in the cloud by CSPI 101, the customer may want their on-premises network 116 and their cloud-based VCN 104 to be able to communicate with each other. This enables customers to build an extended hybrid environment that includes the customer's VCN 104 hosted by CSPI 101 and their on-premises network 116. DRG 122 enables this communication. To enable such communications, a communication channel 124 is set up, where one endpoint of the channel is located in the customer's on-premises network 116 and the other endpoint is located in the CSPI 101 and connected to the customer VCN 104. The communication channel 124 can pass through a public communication network (such as the Internet) or a private communication network. A variety of different communication protocols can be used, such as IPsec VPN technology over a public communication network (such as the Internet), Oracle's FastConnect technology that uses a private network instead of a public network, etc. The equipment or equipment that forms one endpoint of the communication channel 124 in the customer's on-premises network 116 is called customer premises equipment (CPE), such as Figure 1 On the CSPI 101 side, the endpoint may be a host machine executing DRG 122.

[0088] In some embodiments, a remote peer connection (RPC) can be added to the DRG, which allows a customer to peer one VCN with another VCN in a different region. Using such an RPC, a customer VCN 104 can connect to a VCN 108 in another region using a DRG 122. DRG 122 can also be used to communicate with other remote cloud networks 118 (such as Microsoft Azure Cloud, Amazon AWS Cloud, etc.) that are not hosted by CSPI 101.

[0089] like Figure 1As shown in , an Internet Gateway (IGW) 120 can be configured for a customer VCN 104, which enables computing instances on the VCN 104 to communicate with public endpoints 114 accessible via a public network, such as the Internet. The IGW 1120 is a gateway that connects the VCN to a public network, such as the Internet. The IGW 120 enables public subnets within a VCN, such as VCN 104, where resources in the public subnet have public overlay IP addresses, to directly access public endpoints 112 on a public network 114, such as the Internet. Using the IGW 120, a connection can be initiated from a subnet within the VCN 104 or from the Internet.

[0090] A network address translation (NAT) gateway 128 may be configured for a customer's VCN 104 and enables cloud resources in the customer's VCN that do not have dedicated public overlay IP addresses to access the Internet, and does so without exposing those resources to direct incoming Internet connections (e.g., L4-L7 connections). This enables private subnets within a VCN (such as Private Subnet-1 in VCN 104) to privately access public endpoints on the Internet. In a NAT gateway, connections can only be initiated from a private subnet to the public Internet, and connections cannot be initiated from the Internet to a private subnet.

[0091] In some embodiments, a service gateway (SGW) 126 may be configured for a customer VCN 104 and provide a path for private network traffic between the VCN 104 and service endpoints supported in the service network 110. In some embodiments, the service network 110 may be provided by a CSP and may provide a variety of services. An example of such a service network is Oracle's service network, which provides a variety of services available to customers. For example, a computing instance (e.g., a database system) in a private subnet of a customer VCN 104 may back up data to a service endpoint (e.g., an object storage) without requiring a public IP address or access to the Internet. In some embodiments, a VCN may have only one SGW, and connections may only be initiated from a subnet within the VCN, not from the service network 110. If a VCN is peered with another, resources in the other VCN generally cannot access the SGW. Resources in an on-premises network connected to a VCN using FastConnect or VPN Connect may also use the service gateway configured for that VCN.

[0092] In some embodiments, the SGW 126 uses the concept of a service classless inter-domain routing (CIDR) label, which is a string that represents all regional public IP address ranges for a service or group of services of interest. Customers use the service CIDR label when they configure the SGW and associated routing rules to control traffic to the service. If the public IP address of the service changes in the future, the customer can optionally use it when configuring security rules without having to adjust them.

[0093] A Local Peer Gateway (LPG) 132 is a gateway that can be added to a customer VCN 104 and enables the VCN 104 to peer with another VCN in the same region. Peering refers to the VCNs communicating using private IP addresses without the traffic traversing a public network such as the Internet or without routing the traffic through the customer's on-premises network 116. In a preferred embodiment, the VCN has a separate LPG for each peer it establishes. Local peering or VCN peering is a common practice for establishing network connectivity between different applications or infrastructure management functions.

[0094] Service providers (such as providers of services in service network 110) can provide access to services using different access models. According to the public access model, the service can be exposed as a public endpoint that can be publicly accessed by a computing instance in a customer VCN via a public network (such as the Internet), and / or can be privately accessed via SGW 126. According to a specific private access model, the service can be accessed as a private IP endpoint in a private subnet in the customer's VCN. This is called private endpoint (PE) access and enables service providers to expose their services as instances in the customer's private network. Private endpoint resources represent services within the customer's VCN. Each PE appears as a VNIC (called a PE-VNIC, with one or more private IPs) in a subnet selected by the customer in the customer's VCN. Therefore, PE provides a way to present services within a private customer VCN subnet using VNICs. Since the endpoint is exposed as a VNIC, all features associated with the VNIC (such as routing rules, security lists, etc.) can now be used for the PE VNIC.

[0095] Service providers can register their services to enable access through PE. Providers can associate policies with services, which limits the visibility of the service to customer tenancies. Providers can register multiple services under a single virtual IP address (VIP), especially for multi-tenant services. There can be multiple such private endpoints (in multiple VCNs) representing the same service.

[0096] The computing instances in the private subnet can then access the service using the private IP address or service DNS name of the PE VNIC. The computing instances in the customer VCN can access the service by sending traffic to the private IP address of the PE in the customer VCN. The private access gateway (PAGW) 130 is a gateway resource that can be attached to a service provider VCN (e.g., a VCN in the service network 110), which acts as an entry / exit point for all traffic from / to the private endpoints of the customer subnet. PAGW 130 enables the provider to expand the number of PE connections without utilizing its internal IP address resources. The provider only needs to configure one PAGW for any number of services registered in a single VCN. The provider can represent services as private endpoints in multiple VCNs for one or more customers. From the customer's perspective, the PE VNIC is not attached to the customer's instance, but appears to be attached to the service that the customer wants to interact with. Traffic to the private endpoint is routed to the service via PAGW 130. These are called customer-to-service private connections (C2S connections).

[0097] The PE concept can also be used to extend private access for services to the customer's on-premises network and data center by allowing traffic to flow over FastConnect / IPsec links and private endpoints in the customer's VCN. Private access to services can also be extended to the customer's peer VCNs by allowing traffic to flow between LPG 132 and the PEs in the customer's VCN.

[0098] A customer can control routing in a VCN at the subnet level, so a customer can specify which subnets in a customer's VCN (such as VCN 104) use each gateway. The routing table of a VCN is used to decide whether to allow traffic to leave the VCN through a particular gateway. For example, in a particular instance, a routing table for a public subnet within a customer VCN 104 can send non-local traffic through IGW 120. A routing table for a private subnet within the same customer VCN 104 can send traffic destined for a CSP service through SGW 126. All remaining traffic can be sent via NAT gateway 128. The routing table only controls traffic that leaves the VCN.

[0099] The security list associated with the VCN is used to control the traffic that enters the VCN via the gateway via an inbound connection. All resources in the subnet use the same routing table and security list. The security list can be used to control the specific type of traffic allowed in and out of the instances in the subnet of the VCN. The security list rules can include ingress (inbound) and egress (outbound) rules. For example, the ingress rule can specify the allowed source address range, and the egress rule can specify the allowed destination address range. The security rules can specify specific protocols (e.g., TCP, ICMP), specific ports (e.g., 22 for SSH, 3389 for Windows RDP), etc. In some embodiments, the operating system of the instance can enforce its own firewall rules that comply with the security list rules. The rules can be stateful (e.g., tracking connections and automatically allowing responses without explicit security list rules for response traffic) or stateless.

[0100] Access from a customer VCN (i.e., through resources or computing instances deployed on VCN 104) can be categorized as public access, private access, or dedicated access. Public access refers to an access model that uses a public IP address or NAT to access a public endpoint. Private access enables customer workloads (e.g., resources in a private subnet) in VCN 104 with private IP addresses to access services without traversing a public network such as the Internet. In some embodiments, CSPI 101 enables customer VCN workloads with private IP addresses to access services (public service endpoints) using a service gateway. Thus, the service gateway provides a private access model by establishing a virtual link between a customer's VCN and a public endpoint of a service that resides outside the customer's private network.

[0101] Additionally, CSPI can provide dedicated public access using technologies such as FastConnect public peering, where a customer on-premises instance can use a FastConnect connection to access one or more services in a customer VCN without traversing a public network such as the Internet. CSPI can also provide dedicated private access using FastConnect private peering, where a customer on-premises instance with a private IP address can use a FastConnect connection to access the customer's VCN workloads. FastConnect is a network connectivity alternative to using the public Internet to connect a customer's on-premises network to CSPI and its services. FastConnect provides a simple, resilient, and cost-effective way to create dedicated and private connections with higher bandwidth options and a more reliable and consistent networking experience compared to Internet-based connections.

[0102] Figure 1The descriptions above and accompanying thereto describe various virtualized components in an example virtual network.As described above, a virtual network is built on an underlying physical or substrate network. Figure 2 A simplified architecture diagram of physical components in a physical network within a CSPI 200 that provides an underlying virtual network according to certain embodiments is depicted. As shown, the CSPI 200 provides a distributed environment that includes components and resources (e.g., computing, storage, and networking resources) provided by a cloud service provider (CSP). These components and resources are used to provide cloud services (e.g., IaaS services) to subscribed customers (i.e., customers who have subscribed to one or more services provided by the CSP). Based on the services subscribed by the customer, a subset of the resources (e.g., computing, storage, and networking resources) of the CSPI 200 is provisioned to the customer. The customer can then build their own cloud-based (i.e., CSPI-hosted) customizable and private virtual networks using the physical computing, storage, and networking resources provided by the CSPI 200. As indicated above, these customer networks are referred to as virtual cloud networks (VCNs). Customers can deploy one or more customer resources, such as computing instances, on these customer VCNs. The computing instances can be in the form of virtual machines, bare metal instances, etc. CSPI 200 provides infrastructure and a set of complementary cloud services that enable customers to build and run a wide range of applications and services in a highly available hosting environment.

[0103] exist Figure 2 In the example embodiment depicted in , the physical components of CSPI 200 include one or more physical host machines or physical servers (e.g., 202, 206, 208), network virtualization devices (NVDs) (e.g., 210, 212), top-of-rack (TOR) switches (e.g., 214, 216), and physical networks (e.g., 218), as well as switches in physical network 218. The physical host machines or servers can host and execute various computing instances that participate in one or more subnets of the VCN. The computing instances can include virtual machine instances and bare metal instances. For example, Figure 1 The various computational examples depicted in Figure 2 The physical host machine depicted in Figure 1 hosts the virtual machine compute instances in a VCN. The virtual machine compute instances in a VCN can be executed by one host machine or by multiple different host machines. The physical host machine can also host virtual host machines, container-based hosts or functions, etc. Figure 1 The VNIC and VCN VR depicted in the figure can be represented by Figure 2 The NVD implementation depicted in FIG. Figure 1 The gateway depicted in can be Figure 2 The host machine and / or NVD described in .

[0104] A host machine or server may execute a hypervisor (also referred to as a virtual machine monitor or VMM) that creates and enables a virtualized environment on the host machine. A virtualized or virtualized environment facilitates cloud-based computing. One or more computing instances may be created, executed, and managed on the host machine by a hypervisor on the host machine. The hypervisor on the host machine enables the physical computing resources (e.g., computing, memory, and networking resources) of the host machine to be shared between the various computing instances executed by the host machine.

[0105] For example, Figure 2 As depicted in , host machines 202 and 208 execute hypervisors 260 and 266, respectively. These hypervisors may be implemented using software, firmware, or hardware, or a combination thereof. Typically, a hypervisor is a process or software layer that sits on top of the host machine's operating system (OS), which in turn executes on the host machine's hardware processors. The hypervisor provides a virtualized environment by enabling the host machine's physical computing resources (e.g., processing resources such as processors / cores, memory resources, networking resources) to be shared between various virtual machine computing instances executed by the host machine. For example, in Figure 2 In the embodiment of the present invention, the hypervisor 260 can be located on top of the OS of the host machine 202 and enable the computing resources (e.g., processing, memory, and networking resources) of the host machine 202 to be shared between computing instances (e.g., virtual machines) executed by the host machine 202. The virtual machine can have its own operating system (called a guest operating system), which can be the same as or different from the OS of the host machine. The operating system of a virtual machine executed by the host machine can be the same as or different from the operating system of another virtual machine executed by the same host machine. Therefore, the hypervisor enables multiple operating systems to be executed simultaneously while sharing the same computing resources of the host machine. Figure 2 The host machines depicted in FIG. 1 may have the same or different types of hypervisors.

[0106] Compute instances can be virtual machine instances or bare metal instances. Figure 2 , compute instance 268 on host machine 202 and compute instance 274 on host machine 208 are examples of virtual machine instances. Host machine 206 is an example of a bare metal instance provided to a customer.

[0107] In some cases, the entire host machine can be supplied to a single customer, and one or more computing instances (or virtual machines or bare metal instances) hosted by the host machine all belong to the same customer. In other cases, the host machine can be shared between multiple customers (i.e., multiple tenants). In this multi-tenancy scenario, the host machine can host virtual machine computing instances belonging to different customers. These computing instances can be members of different VCNs of different customers. In some embodiments, the bare metal computing instance is hosted by a bare metal server without a hypervisor. When supplying a bare metal computing instance, a single customer or tenant maintains control of the physical CPU, memory, and network interface of the host machine hosting the bare metal instance, and the host machine is not shared with other customers or tenants.

[0108] As previously described, each compute instance that is part of a VCN is associated with a VNIC that enables the compute instance to be a member of a subnet of the VCN. The VNIC associated with a compute instance facilitates the communication of packets or frames to and from the compute instance. The VNIC is associated with the compute instance when the compute instance is created. In some embodiments, for a compute instance executed by a host machine, the VNIC associated with the compute instance is executed by an NVD connected to the host machine. For example, in Figure 2 , host machine 202 executes virtual machine compute instance 268 associated with VNIC 276, and VNIC 276 is executed by NVD 210 connected to host machine 202. As another example, bare metal instance 272 hosted by host machine 206 is associated with VNIC 280 executed by NVD 212 connected to host machine 206. As yet another example, VNIC 284 is associated with compute instance 274 executed by host machine 208, and VNIC 284 is executed by NVD 212 connected to host machine 208.

[0109] For compute instances hosted by a host machine, the NVD connected to the host machine also executes the VCN VR corresponding to the VCN of which the compute instance is a member. Figure 2 In the embodiment depicted in , NVD 210 executes VCN VR 277 corresponding to the VCN of which compute instance 268 is a member. NVD 212 may also execute one or more VCN VRs 283 corresponding to the VCNs corresponding to the compute instances hosted by host machines 206 and 208.

[0110] The host machine may include one or more network interface cards (NICs) that enable the host machine to connect to other devices. The NIC on the host machine may provide one or more ports (or interfaces) that enable the host machine to communicatively connect to another device. For example, the host machine may connect to the NVD using one or more ports (or interfaces) provided on the host machine and on the NVD. The host machine may also connect to other devices (such as another host machine).

[0111] For example, in Figure 2 2, host machine 202 is connected to NVD 210 using link 220, which extends between port 234 provided by NIC 232 of host machine 202 and port 236 of NVD 210. Host machine 206 is connected to NVD 212 using link 224, which extends between port 246 provided by NIC 244 of host machine 206 and port 248 of NVD 212. Host machine 208 is connected to NVD 212 using link 226, which extends between port 252 provided by NIC 250 of host machine 208 and port 254 of NVD 212.

[0112] The NVD is in turn connected via communication links to top-of-rack (TOR) switches, which are connected to the physical network 218 (also referred to as a switch fabric). In some embodiments, the links between the host machine and the NVD and between the NVD and the TOR switches are Ethernet links. For example, in Figure 2 , NVDs 210 and 212 are connected to TOR switches 214 and 216 using links 228 and 230, respectively. In some embodiments, links 220, 224, 226, 228, and 230 are Ethernet links. A collection of host machines and NVDs connected to a TOR is sometimes referred to as a rack.

[0113] Physical network 218 provides a communication structure that enables TOR switches to communicate with each other. Physical network 218 can be a multi-level network. In some embodiments, physical network 218 is a multi-level Clos network of switches, where TOR switches 214 and 216 represent leaf nodes of the multi-level and multi-node physical switching network 218. Different Clos network configurations are possible, including but not limited to 2-level networks, 3-level networks, 4-level networks, 5-level networks, and general "n" level networks. Examples of Clos networks are described in Figure 5 and described below.

[0114] There may be various connection configurations between the host machine and the NVD, such as one-to-one configuration, many-to-one configuration, one-to-many configuration, etc. In a one-to-one configuration implementation, each host machine is connected to its own separate NVD. Figure 2 In , host machine 202 is connected to NVD 210 via NIC 232 of host machine 202. In a many-to-one configuration, multiple host machines are connected to one NVD. Figure 2 , host machines 206 and 208 are connected to the same NVD 212 via NICs 244 and 250, respectively.

[0115] In a one-to-many configuration, one host machine connects to multiple NVDs. Figure 3 An example within CSPI 300 is shown, where a host machine is connected to multiple NVDs. Figure 3 , a host machine 302 includes a network interface card (NIC) 304 that includes a plurality of ports 306 and 308. The host machine 300 is connected to a first NVD 310 via port 306 and link 320, and to a second NVD 312 via port 308 and link 322. Ports 306 and 308 may be Ethernet ports and links 320 and 322 between the host machine 302 and the NVDs 310 and 312 may be Ethernet links. The NVD 310 is in turn connected to a first TOR switch 314 and the NVD 312 is connected to a second TOR switch 316. The links between the NVDs 310 and 312 and the TOR switches 314 and 316 may be Ethernet links. The TOR switches 314 and 316 represent layer 0 switching devices in a multi-layer physical network 318.

[0116] Figure 3 The arrangement depicted in FIG. 3 provides two separate physical network paths to and from the physical switch network 318 to the host machine 302: a first path through the TOR switch 314 to the NVD 310 and then to the host machine 302, and a second path through the TOR switch 316 to the NVD 312 and then to the host machine 302. The separate paths provide enhanced availability (referred to as high availability) of the host machine 302. If a problem occurs in one of the paths (e.g., a link in one of the paths is broken) or a device (e.g., a particular NVD is not running), then the other path can be used for communications to / from the host machine 302.

[0117] exist Figure 3 In the configuration depicted in , the host machine is connected to two different NVDs using two different ports provided by the host machine's NIC. In other embodiments, the host machine may include multiple NICs that enable connectivity of the host machine to multiple NVDs.

[0118] Return to reference Figure 2 NVD is a physical device or component that performs one or more network and / or storage virtualization functions. NVD can be any device with one or more processing units (e.g., CPU, network processing unit (NPU), FPGA, packet processing pipeline, etc.), memory (including cache), and ports. Various virtualization functions can be performed by software / firmware executed by one or more processing units of NVD.

[0119] NVD can be implemented in a variety of different forms. For example, in some embodiments, NVD is implemented as an interface card called a smartNIC or an intelligent NIC with an onboard embedded processor. A smartNIC is a device that is independent of the NIC on the host machine. Figure 2 , NVDs 210 and 212 may be implemented as smartNICs connected to host machine 202 and host machines 206 and 208, respectively.

[0120] However, smartNIC is only one example of an NVD implementation. Various other implementations are possible. For example, in some other implementations, NVD or one or more functions performed by NVD may be incorporated into or performed by one or more host machines, one or more TOR switches, and other components of CSPI 200. For example, NVD may be implemented in a host machine, where the functions performed by NVD are performed by the host machine. As another example, NVD may be part of a TOR switch, or a TOR switch may be configured to perform the functions performed by NVD, which enables the TOR switch to perform various complex packet conversions for a public cloud. A TOR that performs the functions of NVD is sometimes referred to as an intelligent TOR. In other implementations that provide virtual machine (VM) instances to customers instead of bare metal (BM) instances, the functions performed by NVD may be implemented inside the hypervisor of the host machine. In some other implementations, some of the functions of NVD may be offloaded to a centralized service running on a group of host machines.

[0121] In some embodiments, such as when implemented as Figure 2 When the NVD is a smartNIC as shown in FIG. 1 , the NVD may include multiple physical ports that enable the NVD to connect to one or more host machines and one or more TOR switches. Ports on the NVD may be classified as host-facing ports (also referred to as "south ports") or network-facing or TOR-facing ports (also referred to as "north ports"). The host-facing ports of the NVD are the ports used to connect the NVD to the host machines. Figure 2Examples of host-facing ports include port 236 on NVD 210 and ports 248 and 254 on NVD 212. The network-facing ports of the NVD are the ports used to connect the NVD to the TOR switch. Figure 2 Examples of network-facing ports include port 256 on NVD 210 and port 258 on NVD 212. Figure 2 As shown in FIG. 1 , NVD 210 is connected to TOR switch 214 using link 228 extending from port 256 of NVD 210 to TOR switch 214. Similarly, NVD 212 is connected to TOR switch 216 using link 230 extending from port 258 of NVD 212 to TOR switch 216.

[0122] NVD receives packets and frames (e.g., packets and frames generated by a computing instance hosted by the host machine) from a host machine via a host-facing port, and after performing necessary packet processing, may forward the packets and frames to a TOR switch via a network-facing port of NVD. NVD may receive packets and frames from a TOR switch via a network-facing port of NVD, and after performing necessary packet processing, may forward the packets and frames to a host machine via a host-facing port of NVD.

[0123] In some embodiments, there may be multiple ports and associated links between the NVD and the TOR switch. These ports and links may be aggregated to form a link aggregate group (referred to as a LAG) of multiple ports or links. Link aggregation allows multiple physical links between two endpoints (e.g., between the NVD and the TOR switch) to be treated as a single logical link. All physical links in a given LAG may operate in full-duplex mode at the same speed. LAG helps increase the bandwidth and reliability of the connection between the two endpoints. If one of the physical links in the LAG fails, the traffic will be dynamically and transparently reassigned to one of the other physical links in the LAG. Aggregated physical links deliver higher bandwidth than each individual link. Multiple ports associated with the LAG are treated as a single logical port. Traffic may be load balanced between multiple physical links of the LAG. One or more LAGs may be configured between two endpoints. The two endpoints may be located between the NVD and the TOR switch, between the host machine and the NVD, and so on.

[0124] NVD implements or performs network virtualization functions. These functions are performed by software / firmware executed by NVD. Examples of network virtualization functions include, but are not limited to: packet encapsulation and decapsulation functions; functions for creating VCN networks; functions for implementing network policies, such as VCN security lists (firewall) functionality; functions for facilitating packet routing and forwarding to and from computing instances in the VCN; and the like. In some embodiments, upon receiving a packet, NVD is configured to execute a packet processing pipeline to process the packet and determine how to forward or route the packet. As part of this packet processing pipeline, NVD may execute one or more virtual functions associated with an overlay network, such as executing a VNIC associated with a cis in the VCN, executing a virtual router (VR) associated with the VCN, encapsulation and decapsulation of packets to facilitate forwarding or routing in a virtual network, execution of certain gateways (e.g., local peer gateways), implementation of security lists, network security groups, network address translation (NAT) functionality (e.g., translating public IPs to private IPs on a host-by-host basis), throttling functions, and other functions.

[0125] In some embodiments, the packet processing data path in NVD may include multiple packet pipelines, each pipeline consisting of a series of packet transformation stages. In some embodiments, after receiving a packet, the packet is parsed and classified into a single pipeline. The packet is then processed in a linear manner, one stage after another, until the packet is either discarded or sent out through the interface of NVD. These stages provide basic functional packet processing building blocks (e.g., verifying headers, enforcing throttling, inserting new layer 2 headers, enforcing L4 firewalls, VCN encapsulation / decapsulation, etc.) so that new pipelines can be built by combining existing stages, and new functionality can be added by creating new stages and plugging them into existing pipelines.

[0126] NVD can perform both control plane and data plane functions corresponding to the control plane and data plane of the VCN. An example of a VCN control plane is also provided in Fig.18 , Fig.19 , Fig. 20 and Fig.21 An example of a VCN data plane is depicted in FIG. 1 (see reference numerals 1816, 1916, 2016, and 2116) and described below. Fig.18 , Fig.19 , Fig. 20 and Fig.21(see reference numerals 1818, 1918, 2018, and 2118) and described below. Control plane functions include functions for configuring a network that controls how data is forwarded (e.g., setting routes and routing tables, configuring VNICs, etc.). In some embodiments, a VCN control plane is provided that centrally calculates all overlay-to-base mappings and publishes them to NVD and virtual network edge devices (such as various gateways, such as DRG, SGW, IGW, etc.). Firewall rules can also be published using the same mechanism. In some embodiments, NVD only obtains mappings associated with that NVD. Data plane functions include functions for actually routing / forwarding packets based on the configuration set using the control plane. The VCN data plane is implemented by encapsulating the customer's network packets before they traverse the base network. Encapsulation / decapsulation functionality is implemented on NVD. In some embodiments, NVD is configured to intercept all network packets entering and leaving the host machine and perform network virtualization functions.

[0127] As indicated above, NVD performs various virtualization functions, including VNIC and VCN VR. NVD can execute VNICs associated with computing instances hosted by one or more host machines connected to the VNIC. For example, Figure 2 , NVD 210 executes functionality of VNIC 276 associated with compute instance 268 hosted by host machine 202 connected to NVD 210. As another example, NVD 212 executes VNIC 280 associated with bare metal compute instance 272 hosted by host machine 206, and executes VNIC 284 associated with compute instance 274 hosted by host machine 208. The host machines may host compute instances belonging to different VCNs (of different customers), and the NVD connected to the host machines may execute the VNICs corresponding to the compute instances (i.e., execute VNIC-related functionality).

[0128] NVD also implements a VCN virtual router corresponding to the VCN of the compute instance. Figure 2 , NVD 210 executes VCN VR 277 corresponding to the VCN to which compute instance 268 belongs. NVD 212 executes one or more VCN VRs 283 corresponding to one or more VCNs to which compute instances hosted by host machines 206 and 208 belong. In some embodiments, the VCN VR corresponding to the VCN is executed by all NVDs connected to the host machine hosting at least one compute instance belonging to the VCN. If the host machine hosts compute instances belonging to different VCNs, then the NVDs connected to the host machine can execute VCN VRs corresponding to those different VCNs.

[0129] In addition to the VNICs and VCN VRs, NVD may also execute various software (e.g., daemons) and include one or more hardware components that facilitate the various network virtualization functions performed by NVD. For simplicity, these various components are grouped together as Figure 2 ” packet processing component” shown in . For example, NVD 210 includes packet processing component 286 and NVD 212 includes packet processing component 288. For example, a packet processing component for NVD may include a packet processor configured to interact with the port and hardware interface of NVD to monitor all packets received by NVD and transmitted using NVD and store network information. Network information may, for example, include network flow information and each flow information (e.g., each flow statistics) that identify different network flows handled by NVD. In some embodiments, network flow information may be stored on a per-VNIC basis. The packet processor may perform packet-by-packet manipulation and implement stateful NAT and L4 firewall (FW). As another example, the packet processing component may include a replication agent configured to copy information stored by NVD to one or more different replication target storages. As yet another example, the packet processing component may include a logging agent configured to perform the logging function of NVD. The packet processing component may also include software for monitoring the performance and health of NVD and possibly also monitoring the status and health of other components connected to NVD.

[0130] Figure 1 Components of an example virtual or overlay network are shown, including a VCN, subnets within the VCN, compute instances deployed on the subnets, VNICs associated with the compute instances, VRs for the VCN, and a set of gateways configured for the VCN. Figure 1 The overlay component depicted in can be represented by Figure 2 For example, a compute instance in a VCN can be executed or hosted by one or more of the physical components depicted in FIG. Figure 2 . For a compute instance hosted by a host machine, a VNIC associated with the compute instance is typically executed by an NVD connected to the host machine (i.e., VNIC functionality is provided by the NVD connected to the host machine). VCN VR functionality for a VCN is executed by all NVDs connected to the host machines that host or execute compute instances as part of the VCN. Gateways associated with a VCN may be executed by one or more different types of NVDs. For example, some gateways may be executed by smartNICs, while other gateways may be executed by one or more host machines or other implementations of NVDs.

[0131] As described above, a compute instance in a customer VCN can communicate with a variety of different endpoints, where the endpoint can be in the same subnet as the source compute instance, in a different subnet but in the same VCN as the source compute instance, or with an endpoint located outside the VCN of the source compute instance. These communications are facilitated using VNICs associated with the compute instances, VCN VRs, and gateways associated with the VCN.

[0132] For communication between two computing instances on the same subnet in a VCN, VNICs associated with source and destination computing instances are used to facilitate communication. The source and destination computing instances can be hosted by the same host machine or different host machines. Packets originating from the source computing instance can be forwarded from the host machine hosting the source computing instance to the NVD connected to the host machine. On the NVD, packets are processed using a packet processing pipeline, which can include the execution of a VNIC associated with the source computing instance. Since the destination endpoint for the packet is located in the same subnet, the execution of the VNIC associated with the source computing instance causes the packet to be forwarded to the NVD that executes the VNIC associated with the destination computing instance, which then processes the packet and forwards it to the destination computing instance. The VNICs associated with the source and destination computing instances can be executed on the same NVD (e.g., when both the source and destination computing instances are hosted by the same host machine) or on different NVDs (e.g., when the source and destination computing instances are hosted by different host machines connected to different NVDs). The VNIC can use the routing / forwarding table stored by the NVD to determine the next hop for the packet.

[0133] For packets to be transmitted from a computing instance in a subnet to an endpoint in a different subnet in the same VCN, packets originating from a source computing instance are transmitted from a host machine hosting the source computing instance to an NVD connected to the host machine. On the NVD, the packets are processed using a packet processing pipeline, which may include the execution of one or more VNICs and VRs associated with the VCN. For example, as part of the packet processing pipeline, the NVD executes or calls functionality corresponding to the VNIC associated with the source computing instance (also referred to as executing the VNIC). The functionality executed by the VNIC may include checking the VLAN tag on the packet. Since the destination of the packet is outside the subnet, the VCN VR functionality is next called and executed by the NVD. The VCN VR then routes the packet to the NVD that executes the VNIC associated with the destination computing instance. The VNIC associated with the destination computing instance then processes the packet and forwards the packet to the destination computing instance. The VNICs associated with the source and destination computing instances may be executed on the same NVD (e.g., when the source and destination computing instances are hosted by the same host machine) or on different NVDs (e.g., when the source and destination computing instances are hosted by different host machines connected to different NVDs).

[0134] If the destination for the packet is outside the VCN of the source compute instance, then the packet originating from the source compute instance is transmitted from the host machine hosting the source compute instance to the NVD connected to the host machine. The NVD executes the VNIC associated with the source compute instance. Since the destination endpoint of the packet is outside the VCN, the packet is then processed by the VCN VR for the VCN. The NVD invokes VCN VR functionality, which causes the packet to be forwarded to the NVD executing the appropriate gateway associated with the VCN. For example, if the destination is an endpoint within the customer's on-premises network, then the packet may be forwarded by the VCN VR to the NVD executing the DRG gateway configured for the VCN. The VCN VR may be executed on the same NVD as the NVD executing the VNIC associated with the source compute instance, or by a different NVD. The gateway may be executed by an NVD, which may be a smartNIC, a host machine, or other NVD implementation. The packet is then processed by the gateway and forwarded to the next hop, which facilitates delivery of the packet to its intended destination endpoint. For example, in Figure 2 , a packet originating from compute instance 268 may be transmitted from host machine 202 to NVD 210 via link 220 (using NIC 232). On NVD 210, VNIC 276 is invoked because it is the VNIC associated with source compute instance 268. VNIC 276 is configured to inspect the encapsulated information in the packet and determine a next hop for forwarding the packet with the intent of facilitating delivery of the packet to its intended destination endpoint, and then forward the packet to the determined next hop.

[0135] Compute instances deployed on a VCN can communicate with a variety of different endpoints. These endpoints may include endpoints hosted by CSPI 200 and endpoints external to CSPI 200. Endpoints hosted by CSPI 200 may include instances in the same VCN or other VCNs, which may be the customer's VCNs or VCNs that do not belong to the customer. Communications between endpoints hosted by CSPI 200 may be performed over physical network 218. Compute instances may also communicate with endpoints that are not hosted by CSPI 200 or that are external to CSPI 200. Examples of these endpoints include endpoints within a customer's on-premises network or data center, or public endpoints accessible over a public network such as the Internet. Communications with endpoints external to CSPI 200 may be performed over a public network (e.g., the Internet) using various communication protocols. Figure 2 not shown) or a private network ( Figure 2 Executed by (not shown).

[0136] Figure 2The architecture of CSPI 200 depicted in the drawings is merely an example and is not intended to be limiting. In alternative embodiments, variations, alternatives, and modifications are possible. For example, in some embodiments, CSPI 200 may have a Figure 2 The systems or components shown in FIG. 5 may include more or fewer systems or components, may combine two or more systems, or may have different configurations or arrangements of the systems. Figure 2 The systems, subsystems, and other components depicted in the can be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the corresponding system, using hardware, or a combination thereof. The software can be stored on a non-transitory storage medium (e.g., on a memory device).

[0137] Figure 4 Depicted is connectivity between a host machine and an NVD for providing I / O virtualization to support multi-tenancy in accordance with certain embodiments. Figure 4 As depicted in , a host machine 402 executes a hypervisor 404 that provides a virtualized environment. The host machine 402 executes two virtual machine instances, VM1 406 belonging to customer / tenant #1 and VM2 408 belonging to customer / tenant #2. The host machine 402 includes a physical NIC 410 connected to an NVD 412 via a link 414. Each compute instance is attached to a VNIC executed by the NVD 412. Figure 4 In the embodiment of FIG. 4 , VM1 406 is attached to VNIC-VM1 420 and VM2 408 is attached to VNIC-VM2 422 .

[0138] like Figure 4 As shown in FIG, NIC 410 includes two logical NICs, logical NIC A 416 and logical NIC B 418. Each virtual machine is attached to its own logical NIC and is configured to work with the logical NIC. For example, VM1 406 is attached to logical NIC A 416 and VM2 408 is attached to logical NIC B 418. Although the host machine 402 includes only one physical NIC 410 shared by multiple tenants, each tenant's virtual machines believe that they have their own host machine and NIC due to the logical NICs.

[0139] In some embodiments, each logical NIC is assigned its own VLAN ID. Thus, a particular VLAN ID is assigned to logical NIC A 416 for tenant #1, and a separate VLAN ID is assigned to logical NIC B 418 for tenant #2. When a packet is transmitted from VM1 406, a tag assigned to tenant #1 is attached to the packet by the hypervisor, and the packet is then transmitted from the host machine 402 to the NVD 412 via link 414. In a similar manner, when a packet is transmitted from VM2 408, a tag assigned to tenant #2 is attached to the packet by the hypervisor, and the packet is then transmitted from the host machine 402 to the NVD 412 via link 414. Thus, a packet 424 transmitted from the host machine 402 to the NVD 412 has an associated tag 426 that identifies the particular tenant and the associated VM. On the NVD, for a packet 424 received from the host machine 402, a tag 426 associated with the packet is used to determine whether the packet is processed by VNIC-VM1 420 or VNIC-VM2 422. The packet is then processed by the corresponding VNIC. Figure 4 The configuration described in enables each tenant's compute instances to believe that they have their own host machine and NIC. Figure 4 The setup described in provides I / O virtualization to support multi-tenancy.

[0140] Figure 5 A simplified block diagram of a physical network 500 is depicted in accordance with certain embodiments. Figure 5 The embodiments depicted in are structured as Clos networks. A Clos network is a specific type of network topology that is designed to provide connection redundancy while maintaining high bisection bandwidth and maximum resource utilization. A Clos network is a non-blocking, multi-tier or multi-level switching network, where the number of tiers or levels can be two, three, four, five, etc. Figure 5 The embodiment depicted in is a 3-layer network, including layers 1, 2, and 3. TOR switch 504 represents a layer 0 switch in a Clos network. One or more NVDs are connected to the TOR switch. Layer 0 switches are also called edge devices of the physical network. Layer 0 switches are connected to layer 1 switches, also called leaf switches. Figure 5In the embodiment depicted in , a group of "n" layer 0TOR switches are connected to a group of "n" layer 1 switches and form a cluster (pod). Each layer 0 switch in a cluster is interconnected with all layer 1 switches in the cluster, but there is no switch connectivity between clusters. In some embodiments, two clusters are referred to as blocks. Each block is served or connected to a group of "n" layer 2 switches (sometimes called backbone switches). There can be several blocks in the physical network topology. The layer 2 switches are in turn connected to "n" layer 3 switches (sometimes called super backbone switches). Communication of packets through the physical network 500 is typically performed using one or more layer 3 communication protocols. Typically, all layers of the physical network (except the TOR layer) are n-way redundant, thereby allowing high availability. Policies can be specified for clusters and blocks to control the visibility of switches to each other in the physical network, thereby enabling scaling of the physical network.

[0141] A Clos network is characterized by a fixed maximum number of hops from one level 0 switch to another level 0 switch (or from an NVD connected to a level 0 switch to another NVD connected to a level 0 switch). For example, in a 3-level Clos network, a packet takes at most seven hops to reach from one NVD to another NVD, where the source and destination NVDs are connected to the leaf level of the Clos network. Similarly, in a 4-level Clos network, a packet takes at most nine hops to reach from one NVD to another NVD, where the source and destination NVDs are connected to the leaf level of the Clos network. Therefore, the Clos network architecture maintains consistent latency throughout the network, which is important for communications within and between data centers. The Clos topology is horizontally scalable and cost-effective. The bandwidth / throughput capacity of the network can be easily increased by adding more switches at each level (e.g., more leaf switches and spine switches) and by increasing the number of links between switches at adjacent levels.

[0142] In some embodiments, each resource within the CSPI is assigned a unique identifier called a cloud identifier (CID). This identifier is included as part of the resource's information and can be used to manage the resource, for example, via the console or through an API. An example syntax for a CID is:

[0143] ocid1.<resource type>.<domain>.[region][.future use].<unique ID>

[0144] Among them, ocid1: a text string indicating the version of the CID; resource type: the type of resource (for example, instance, volume, VCN, subnet, user, group, etc.); domain: the domain where the resource is located. Example values ​​are "c1" for the commercial domain, "c2" for the government cloud domain, or "c3" for the federal government cloud domain. Each domain can have its own domain name; region: the region where the resource is located. If the region is not applicable to the resource, then this part may be empty; future use: reserved for future use. Unique ID: the unique part of the ID. The format may vary depending on the type of resource or service.

[0145] Dedicated Regional Customer Cloud (DRCC)

[0146] A Dedicated Regional Cloud (DRCC) at the Customer's Site corresponds to a cloud service provider's (CSP) infrastructure deployed in the customer's own data center. Leveraging a DRCC, enterprises can easily consolidate mission-critical database systems with applications previously deployed on expensive hardware on the CSP's highly available and secure infrastructure, creating operational efficiencies and modernization opportunities. Due to the inherent mismatch between traditional application architectures and cloud architectures, enterprises often find it costly and difficult to move to cloud infrastructure. These challenges are exacerbated for workloads that cannot be moved to the public cloud. Enterprises only have access to a small subset of cloud services deployed on-premises, and these services also have only a limited set of features and capabilities compared to those available in the public cloud.

[0147] The DRCC framework brings the full power of the public cloud to on-premises deployments, allowing enterprises to reduce infrastructure and operating costs, upgrade legacy applications on modern cloud services, and meet the most stringent regulatory, data residency and latency requirements - all of which are implemented through the CSP's infrastructure, which provides enhanced performance and the highest level of security. Customers get the choice and flexibility to run all of the CSP's cloud services in their data centers. Customers can choose from all public cloud services provided by the CSP, including, for example, VMware Cloud, Autonomous Database, Container Engine for Kubernetes, Bare Metal Servers, Exadata Cloud Services, and pay only for the services they consume. The DRCC framework is designed to keep data and customer operations completely isolated from the Internet - where control plane and data plane operations remain on-premises - to help customers meet their most stringent compliance and latency requirements. Through a fully managed experience and access to new capabilities the moment they become available in the public cloud, the DRCC framework provides cloud-scale security, elasticity and scale, as well as support for mission-critical workloads with tools to gradually modernize legacy workloads.

[0148] Some of the benefits brought by DRCC are as follows:-

[0149] Bring all public cloud services and autonomous databases on-premises to reduce the risk and cost of innovation

[0150] Provide a framework where customers pay only for the services they consume

[0151] Build a truly consistent development experience for all IaaS and PaaS applications by using the exact same tools, APIs, and SLAs available in public cloud infrastructure

[0152] Retain full control over all data to meet the most stringent data privacy and latency requirements

[0153] Deploy seamlessly between on-premises and public clouds without compromising functionality or development experience

[0154] Consolidate workloads onto a single cloud platform so customers can focus on business priorities

[0155] Reduce the cost of running on-premises workloads by leveraging CSP’s infrastructure that provides the highest levels of security

[0156] A CSP's infrastructure is typically hosted in regions and availability domains. A region is a localized geographic area, and an availability domain is one or more data centers located within a region. Therefore, a region consists of one or more availability domains. Most infrastructure resources are either region-specific (such as virtual cloud networks) or availability domain-specific (such as compute instances). Traffic between availability domains and between regions is encrypted. Availability domains are isolated from each other, fault-tolerant, and unlikely to fail at the same time. Because availability domains do not share infrastructure (such as power or cooling) or the internal availability domain network, a failure at one availability domain within a region is unlikely to affect the availability of other availability domains in the same region.

[0157] Availability domains within the same region are connected to each other through a low-latency, high-bandwidth network, making it possible to provide high-availability connectivity to the Internet and on-premises, and to build replicated systems in multiple availability domains for both high availability and disaster recovery. Regions are independent of other regions and can be separated by great distances—across countries or even continents. Generally speaking, one can deploy an application in the region where it is most frequently used, because using nearby resources is faster than using distant resources. However, one can also deploy applications in different regions to mitigate the risk of regional events such as earthquakes.

[0158] A fault domain is a grouping of hardware and infrastructure within an availability domain. Each availability domain contains multiple fault domains (for example, three fault domains). Fault domains provide anti-affinity, that is, fault domains allow you to distribute instances so that instances are not on the same physical hardware within a single availability domain. A hardware failure or compute hardware maintenance event that affects one fault domain does not affect instances in other fault domains. In addition, the physical hardware in a fault domain has independent and redundant power supplies, which prevents a failure in the power hardware within one fault domain from affecting other fault domains.

[0159] According to some embodiments, in the context of a DRCC, an availability domain is provided via a rack that includes multiple top-of-rack (TOR) switches and multiple host machines / servers. A TOR switch is a network switch used in a data center to connect servers and other network devices in a rack. The purpose of a TOR switch is to provide high-speed connectivity and efficient data transfer between the devices within the rack and the larger network infrastructure. A TOR switch typically has a high port density to accommodate multiple servers and devices within a single rack. It provides Ethernet connectivity to these devices, allowing them to communicate with each other and with the rest of the network. TOR switches often support high-speed Ethernet standards, such as 10 Gigabit Ethernet, 25 Gigabit Ethernet, 40 Gigabit Ethernet, or 100 Gigabit Ethernet. These fast data transfer rates ensure efficient communication between servers and the network. TOR switches provide minimal latency and provide low-latency switching, which is critical in data centers that require fast response times for applications and services running on servers. Next, refer to Figure 6 and Figure 7 Describe the different TOR configurations that can be employed within a rack in a data center.

[0160] Figure 6 Depicted are configurations of multiple TORs included in a rack according to at least one embodiment. Figure 6 The configuration 600 of multiple TORs depicted in FIG. 6 corresponds to a 3-TOR configuration. The rack includes three TORs (601, 602, and 603) and multiple host machines / servers. In one embodiment, a failure domain is created within an availability domain (i.e., a rack) by selecting a subset of host machines from the multiple host machines. Each host machine in the selected group of host machines (i.e., the subset of host machines) is communicatively coupled to one of the TORs included in the multiple TORs. For example, Figure 6 As shown in , a first subset of host machines from a plurality of host machines is depicted as 605A. Each host machine / server included in 605A is communicatively coupled to a first TOR (i.e., TOR1, 601). Therefore, the combination of the first subset of host machines (605A) and the first TOR (601) forms a first failure domain.

[0161] A second fault domain is created within the availability domain by selecting a second subset of host machines from the plurality of host machines. Each host machine in the second subset of host machines is communicatively coupled to another TOR included in the plurality of TORs (i.e., different from the TOR associated with the first fault domain). Note that the first subset of host machines is disjoint from the second subset of host machines. For example, Figure 6 As shown in , a second subset of host machines from the plurality of host machines is depicted as 605B. Each host machine / server included in 605B is communicatively coupled to a second TOR, i.e., TOR 2, 602.

[0162] In a similar manner, a third fault domain can be created within the availability domain by selecting a third subset of host machines from the plurality of host machines. Each host machine in the third subset of host machines is communicatively coupled to another TOR (i.e., different from the TORs associated with the first and second fault domains). Note that the third subset of host machines is disjoint from the first subset of host machines and the second subset of host machines. For example, Figure 6 As shown in , a third subset of host machines from the plurality of host machines is depicted as 605C. Each host machine / server included in 605C is communicatively coupled to a third TOR (ie, TOR 3, 603) to form a third failure domain.

[0163] exist Figure 6 , each of the first, second, and third subsets of servers (605A, 605B, and 605C) are depicted as including K servers. It should be appreciated that this in no way limits the scope of the present disclosure. A particular subset of servers may include a different number of servers than another subset of servers. In addition, the rack includes a plurality of network virtualization devices (NVDs). A first subset of servers / host machines is connected to a first TOR using a first subset of NVDs from the plurality of NVDs. In a similar manner, a second subset of servers / host machines is connected to a second TOR using a second subset of NVDs from the plurality of NVDs, and a third subset of servers / host machines is connected to a third TOR using a third subset of NVDs.

[0164] Additionally, for each failure domain (i.e., a combination of a subset of servers and TOR switches associated with the subset of servers), a set of addresses corresponding to hosts / servers included in a first subset of host machines / servers is associated with a TOR switch associated with the first subset of servers. In this manner, the control plane is configured to forward packets destined for a particular server included in the first subset of servers to a first TOR associated with the first subset of servers.

[0165] Therefore, if Figure 6The configuration of the rack depicted in FIG provides three different failure domains that provide a smaller blast radius (i.e., the percentage of capacity lost when a TOR switch fails) than when all servers within the rack are communicatively coupled to a single TOR switch. Specifically, Figure 6 The rack configuration depicted in the example results in a 33% blast radius, i.e., a 33% capacity loss when a single TOR fails. However, it should be noted that the probability of multiple TORs failing simultaneously within a rack is very low, i.e., negligible. In one embodiment, Figure 6 The rack configuration provides one or more fault domains that can be presented to the customer. After obtaining a request from the customer to allocate one or more host machines in the rack, the control plane can assign one or more host machines included in the one or more fault domains based on specific criteria. For example, if the customer requests high availability, the control plane can allocate host machines located in different fault domains.

[0166] Steering Figure 7 , depicts another configuration of multiple TORs included in a rack according to some embodiments. Specifically, Figure 7 The configuration 700 depicted in FIG. 7 is referred to herein as a dual TOR configuration. Figure 7 As shown in , the rack includes two TORs, i.e., TOR 1 701 and TOR 2 703. In addition, the rack includes multiple server / host machines. According to some embodiments, the multiple servers are grouped into disjoint subsets of servers. For example, the multiple servers can be grouped into a first subset 705A of servers and a second subset 705B of servers.

[0167] like Figure 7 As shown in , in a dual TOR configuration, each subset of servers is communicatively coupled to each TOR included in the rack. Specifically, the first subset of servers 705A and the second subset of servers are communicatively coupled to TOR 1 701 and TOR 2 703. Therefore, in this configuration, when a single TOR switch in the rack fails, no loss of capacity is incurred, and the server only needs to select another normally functioning TOR to transmit data. It is noted that the probability of two TORs in the rack failing at the same time is extremely low, i.e., almost negligible. The design of the NVD in which the servers are coupled to the TORs is unchanged (i.e., each NVD couples a single server to a single TOR). Figure 6 Compared to the TOR configuration in Figure 7 In the configuration of , it can be appreciated that the NVD is configured to connect to two TORs. Therefore, in this configuration, each NVD is configured to have multiple IP addresses. It can be appreciated that in Figure 7In a configuration of , the entire rack including multiple TORs (i.e., providing redundancy) is considered a fault domain. In some embodiments, a fault domain can span multiple racks. For example, consider a switch that provides services to multiple racks (e.g., four racks). In this case, the four racks can be considered a fault domain, where each rack can include multiple TORs to provide redundancy.

[0168] Figure 8 Depicted is a flow diagram illustrating steps performed in provisioning availability domains to customers in accordance with at least one embodiment. Figure 8 The processes described in the can be implemented in software (e.g., code, instructions, programs), hardware, or a combination thereof, executed by one or more processing units (e.g., processors, cores) of the corresponding system. The software can be stored on a non-transitory storage medium (e.g., a memory device). Figure 8 The methods presented in and described below are intended to be illustrative and non-limiting. Figure 8 Describe various processing steps that occur in a specific order or sequence, but this is not intended to be limiting. In some alternative embodiments, these steps can be performed in a different order, or some steps can also be performed in parallel.

[0169] The process begins at step 801, where the control plane provides an availability domain including a rack. The rack includes a plurality of TOR switches and a plurality of host machines or servers. In step 803, a first fault domain is created within the availability domain. The first fault domain includes a first TOR switch from a plurality of TOR switches and a first subset of host machines from a plurality of host machines. The first subset of host machines is communicatively coupled to the first TOR. Thereafter, the process moves to step 805, where a second fault domain is created within the availability domain. The second fault domain includes a second TOR switch from a plurality of TOR switches and a second subset of host machines from a plurality of host machines. Note that the second subset of host machines is disjoint from the first subset of host machines. The second subset of host machines is also communicatively coupled to the second TOR via NVD. In this manner, one or more fault domains may be presented to a customer. Upon obtaining a request from a customer requesting the allocation of one or more host machines in a rack, the control plane may assign one or more host machines included in the one or more fault domains based on certain criteria associated with the customer's requirements.

[0170] Now turn to Fig. 9 , depicts an exemplary architecture 900 of a DRCC framework that provides customers with the full capabilities of the public cloud. As a result, customers can reduce infrastructure and operating costs, upgrade legacy applications on modern cloud services, and meet the most stringent regulatory, data residency, and latency requirements.

[0171] According to some embodiments, Fig. 9 A data center 905 is depicted that includes a pair of TORs (i.e., TOR #1 922 and TOR #2 924), a network virtualization platform (e.g., NVD 926), and a computing host 928 (also referred to herein as a local computing host). It can be appreciated that the computing host 928 includes multiple virtual machines or bare metal instances. NVD 926 is referred to herein as a local NVD. The computing host 928 includes a host network interface card (i.e., a host NIC). For ease of illustration, Fig. 9 The computing host 928 is depicted as including two virtual machines, namely, VM1 and VM2, respectively. Note that each VM is communicatively coupled to a host NIC via one of the logical interfaces (e.g., the logical interfaces depicted as PF1 and PF2, respectively). Further, note that the local NVD 926 can be located on the same chassis as the host NIC included in the computing host 928.

[0172] The computing host 928 included in the data center can be coupled to another host machine 911, which is referred to as a remote host machine in this article. It can be appreciated that the remote host machine can be "any" host machine, such as: (i) another host within the DRCC and located behind another NVD, or (ii) another host in another DRCC (e.g., in a set of DRCCs for the same customer / organization) and located behind another NVD (e.g., in a set of DRCCs for the same customer / organization), or (iii) a host machine included in the customer's on-premises network. Note that in the case where the host machine is included in the customer's on-premises network, the host machine can be connected to the DRCC via a Fast-Connect or IPSecVPN tunnel and connected to the host machine in the DRCC using a dynamic routing gateway (DRG). For ease of explanation, in the following description, it is assumed that the remote host machine (e.g., host machine 911) is a host machine included in the DRCC and located behind another NVD (e.g., NVD 913 and served by a remote TOR 915). However, it is noted that the features described below are also applicable to other cases of remote host machines outlined above. Furthermore, it can be appreciated that in this case (and if Fig. 9 ), two host machines (ie, local host machine 928 and remote host machine 911) may be coupled via network structure 920. Additionally, for convenience, NVD 913 is referred to herein as a remote NVD.

[0173] According to some embodiments, the local NVD 926 has multiple physical ports. Fig. 9In one embodiment shown in FIG. 1 , the local NVD 926 has two physical ports: a first physical port 927A (referred to herein as a TOR-facing port) connected to the TORs 922 and 924, respectively, and a second physical port 927B (referred to herein as a host-facing port) connected to the computing host 928. Each physical port of the local NVD 926 can be divided into multiple logical ports. For example, Fig. 9 As shown in , the physical port 927B is divided into two logical ports on the side facing the host, while the physical port 927A is divided into two logical ports on the side facing the TOR.

[0174] Partitioning each physical port of the local NVD 926 provides the flexibility for each physical port of the NVD 926 to be represented by two logical ports, two MAC addresses, and two IP addresses. Fig. 9 In the , the overriding IP address and the overriding MAC address are indicated by an underlined symbol (for example, B1 , M1 ), while base IP and MAC addresses are represented by symbols without underscores (e.g., A0, M0). Fig. 9 As shown in FIG. 1 , the first physical port 927A of the local NVD 926 is associated with the first IP address (A1), the second IP address (A3), the first MAC address (M1), and the second MAC address (M3). The second physical port 927B of the local NVD 926 is associated with the first overlay IP address ( B1 ), the second coverage IP address ( C1 ), the first coverage MAC address ( M4 ) and the second overlay MAC address ( M6 ) is associated with.

[0175] It will be appreciated that the limit on the number of logical ports that can be obtained by dividing the physical port of NVD 926 (e.g., port 927A) depends on the width of the serializer / deserializer component (i.e., SerDes component) included in the NVD chipset. In one instance, each physical port of NVD 926 can be divided into four logical ports. It will be appreciated that a greater number of logical ports can be obtained for each physical port of NVD 926 by utilizing a gearbox component in NVD. Data center 905 (i.e., DRCC) brings the full power of the public cloud to customers. Specifically, the DRCC hosts applications and data that require strict data residency, control, and security, and provides a means for retaining data in a specific location to enable low-latency connectivity and data-intensive processing. Thus, customers can take advantage of all cloud services running directly in their own data center rather than in a cloud region hundreds or thousands of miles away. Therefore, the DRCC (which will be referenced later) occupies a smaller space. Fig.14 Describe) provides organizations with the opportunity to run workloads outside of the public cloud.

[0176] Steering Fig.10 , depicts a flowchart illustrating a process for providing a DRCC, according to some embodiments. Fig.10 The processes described in the can be implemented in software (e.g., code, instructions, programs), hardware, or a combination thereof, executed by one or more processing units (e.g., processors, cores) of the corresponding system. The software can be stored on a non-transitory storage medium (e.g., a memory device). Fig.10 The methods presented in and described below are intended to be illustrative and non-limiting. Fig.10 Various processing steps are depicted as occurring in a particular order or sequence, but this is not intended to be limiting. In some alternative embodiments, the steps may be performed in a different order, or some steps may be performed in parallel. In some implementations, Fig.10 The method illustrated in may be performed by a cloud service provider to provide DRCC to customers.

[0177] The method begins at step 1001, where a first physical port of a network virtualization device (NVD) included in a data center is communicatively coupled to a first top-of-rack (TOR) switch and a second TOR switch. Note that the first and second TOR switches may be included in a rack (e.g., Figure 6 and Figure 7 In step 1003, a second physical port of the NVD is communicatively coupled to a network interface card (NIC) associated with a host machine included in the data center. The second physical port provides a first logical port and a second logical port for communication between the NVD and the NIC.

[0178] Processing then moves to step 1005, where the NVD receives a packet from a host machine via the first logical port or the second logical port. In step 1007, the NVD determines a specific TOR for transmitting the packet from a group including the first TOR and the second TOR. According to some embodiments, the NVD may execute a packet forwarding mechanism (such as equal cost multi-path routing (ECMP) flow hashing) to select one of the two TORs. Additionally, in step 1009, the NVD transmits the packet to the specific TOR to facilitate transmission of the packet to a destination host machine (e.g., a host machine behind other NVDs in the same rack, or (one or more) other host machines behind other NVDs in other racks, or a host machine outside the data center (e.g., including in a customer's on-premises network)).

[0179] The following provides a detailed description of the following: (i) Reference Fig.11A, transmitting the packet from a computing host (e.g., computing host 928) included in data center 905 to remote host 911; and (ii) referring to Fig. 11B , transmitting packets from the remote host to the computing host in the data center, i.e., the return path. As previously mentioned, the overlay IP address and overlay MAC address are represented by underlined symbols (e.g., B1 , M1 ), while the base IP and MAC addresses are represented by symbols without underscores (e.g., A0, M0). For ease of explanation, we consider the case where a virtual machine (e.g., VM1) included in the computing host 928 sends a packet to the remote host 911. Fig.11A Describe the steps involved in the transmission of a packet.

[0180] Fig.11A A flow chart illustrating steps performed when transmitting a packet from a computing host included in a data center to a remote host is depicted. For ease of illustration, reference is made to Fig.11A and Fig. 11B The description provided relates to a host in a DRCC sending / receiving packets from another host in the DRCC. However, it can be appreciated that the features described herein are equally applicable to other situations of remote hosts, for example, remote host machines included in a customer's on-premises network. Fig.11A The processes described in the can be implemented in software (e.g., code, instructions, programs), hardware, or a combination thereof, executed by one or more processing units (e.g., processors, cores) of the corresponding system. The software can be stored on a non-transitory storage medium (e.g., a memory device). Fig.11A The methods presented in and described below are intended to be illustrative and non-limiting. Fig.11A Describe various processing steps that occur in a specific order or sequence, but this is not intended to be limiting. In some alternative embodiments, these steps can be performed in a different order, or some steps can also be performed in parallel.

[0181] In step 1101, having an overlay IP address B2 The VM1 transmission consists of B1 , M4 (i.e., the overlay IP and MAC address of one of the logical ports of the NVD) received a packet (intended for a B100remote host 911). Note that VM1 transmits the packet to the host NIC via logical interface PF1. In turn, the host NIC utilizes one of the two logical ports of NVD 926 to transmit the packet. According to some embodiments, note that each virtual machine of the computing host is associated with a logical interface on the host NIC. These interfaces are referred to herein as virtual functions, i.e., "VFs", such that each VF is associated with a VLAN-ID. Note that the packets transmitted by the VM can use either or both of the logical ports of the NVD. Whether the packet uses one or both ports, or one or the other, depends on the configuration of the NVD (e.g., whether the NVD is configured for active / active operation or active / backup operation) and the "state" of the interfaces of the NVD (e.g., whether both interfaces are up and running, or whether one interface is down / in backup mode).

[0182] In step 1103, after receiving the packet, NVD 926 performs a lookup operation in the VCN forwarding table. Specifically, NVD 926 obtains information about the base IP address of the NVD providing services to the remote host, that is, NVD 926 obtains information about the base IP address A100 of the remote NVD 913 providing services to the remote host.

[0183] In step 1105, the NVD modifies the header of the packet. Specifically, the NVD 926 encapsulates the packet, e.g., a VCN header, with information corresponding to the base IP address (A100) of the remote NVD 913 as the intended destination of the packet and its own software interface IP address A254 as the source of the packet. Thereafter, the NVD 926 attempts to transmit the packet to the remote host.

[0184] In step 1107, NVD 926 determines how to forward the packet to remote NVD 1113. According to one embodiment, NVD 926 recognizes that it has two routes to send the packet to the remote host, namely, one route through TOR#1 922 and another route through TOR#2 924. Specifically, NVD 926 recognizes the two routes via BGP communications performed with the two TORs. NVD 926 performs equal cost multi-path (ECMP) flow hashing to select one of the two TORs to forward the packet to remote NVD 913. In step 1109, the packet traverses the selected TOR (e.g., TOR#1) to traverse the network fabric to ultimately reach the remote TOR that provides services to the remote host machine. Additionally, in steps 1111 and 1113, the packet traverses the remote TOR 915 to reach the remote NVD 913. Remote NVD 913 decapsulates the packet to retrieve the destination address (i.e., B100) included in the packet and ultimately provides the packet to the remote host 911.

[0185] Now turn to Fig. 11B , depicts a diagram of the process of sending a message from a remote host (e.g., Fig. 9 remote host 911) to a computing host (e.g., Fig. 9 Flowchart of the steps performed by host 928 in when transmitting a packet. Fig. 11B The processes described in the can be implemented in software (e.g., code, instructions, programs), hardware, or a combination thereof, executed by one or more processing units (e.g., processors, cores) of the corresponding system. The software can be stored on a non-transitory storage medium (e.g., a memory device). Fig. 11B The methods presented in and described below are intended to be illustrative and non-limiting. Fig. 11B Describe various processing steps that occur in a specific order or sequence, but this is not intended to be limiting. In some alternative embodiments, these steps can be performed in a different order, or some steps can also be performed in parallel.

[0186] The process begins at step 1151, where the remote host 911 has an overlay IP address B100 The VM of sends a packet destined for VM1 (with overlay IP address B2) to NVD 913. It can be appreciated that Fig. 11B In the description of , it is assumed that both the remote host machine and the local host machine are included in the DRCC. However, it is noted that if the remote host machine is located outside the DRCC, then the packet transmitted by the remote host machine "enters" the DRCC using means such as FastConnect (or IPSec VPN), and then uses the DRG to reach the NVD of the host machine included in the DRCC. In step 1153, NVD 913 performs a lookup operation in the VCN forwarding table after receiving the packet. Specifically, NVD 913 determines that the packet has an overlay address. B2 VM1 is served by NVD with loopback IP address A254.

[0187] In step 1155, NVD 913 encapsulates the packet, e.g., a VCN header, with information corresponding to the loopback IP address (A254) of NVD 926 as the intended destination of the packet and its own base IP address (A100) as the source of the packet. Thereafter, in step 1157, NVD 913 forwards the packet to the remote TOR, which further delivers the packet to network fabric 920.

[0188] In step 1159, the network fabric 920 performs a hash operation (e.g., a modulo 2 operation) upon receiving the packet to select one of the routes (via TOR #1 or TOR #2) to forward the packet to the NVD 926. It will be appreciated that the hash operation performed by a switch (e.g., in the network fabric) may be different from the hash operation performed by the NVD.

[0189] In step 1161, NVD 926 receives the packet. In step 1163, NVD 926 may perform another hash operation to select two logical ports (eg, with overlay IP addresses). B1 and C1 1164. The NVD forwards the packet to one of the selected logical interfaces of the NVD (i.e., the ports of the NVD) to forward the packet to VM1, which is the intended destination of the packet. It will be appreciated that, as previously described, packets can arrive at any port of the NVD. For example, if the NVD is configured to be in an active / active mode of operation, then in one embodiment, approximately half of the flows will arrive at any port. However, if the NVD is configured to be in an active / backup mode of operation, then all flows will arrive at the active port (until the active port fails). In step 1165, the NVD forwards the packet to the VM on the selected logical interface of the NVD.

[0190] According to some embodiments, traffic originating from a local host machine and destined for a remote host machine (and vice versa) is statistically load balanced (e.g., via ECMP routing) across the dual TORs 922 and 924. An advantage of using dual TORs in a DRCC architecture is failover, i.e., traffic from a failed TOR can be switched to another (normally functioning) TOR. In addition, Fig. 9 The architecture also enables the use of less hardware, i.e., a single local NVD (e.g., smartNIC) is used instead of multiple smartNICs. Furthermore, it is noted that the above-described mechanism of utilizing dual TORs is by no means limited to use only in on-premises data centers as described above. Rather, the concept of utilizing dual TORs as described herein is equally applicable to any deployment - i.e., a commercial area or DRCC or any other type of network deployment.

[0191] Now turn to Fig.12 , depicts another exemplary architecture 1200 of the DRCC framework that brings the full power of the public cloud to customers. In this way, customers can reduce infrastructure and operating costs, upgrade legacy applications on modern cloud services, and meet the most stringent regulatory, data residency, and latency requirements. Fig.12A data center 1205 is depicted that includes a host machine 1228, which is coupled to a remote host machine 1211 via a network structure 1220. It can be appreciated that the remote host machine can be "any" host machine, such as: (i) another host within the DRCC and located behind another NVD, or (ii) another host in another DRCC and located behind another NVD, or (iii) a host included in a customer's on-premises network. Note that for hosts included in an on-premises network, it is possible to connect to the DRCC via Fast-Connect or IPSec VPN and connect to the host 928 using a dynamic routing gateway (DRG). For ease of explanation, in the following description, it is assumed that the remote host (e.g., host 1211) is a host included in the DRCC and located behind another NVD (e.g., NVD 1213 and served by a remote TOR 1215). However, it is noted that the features described below are equally applicable to other cases of remote hosts outlined above.

[0192] Data center 1205 includes a pair of TORs (i.e., TOR#1 1222 and TOR#2 1224), a network virtualization platform (e.g., NVD 1226), and a computing host machine 1228 (i.e., a host machine referred to herein as a local host machine). NVD 1226 is referred to herein as a local NVD. Computing host machine 1228 includes a host machine network interface card (i.e., a host NIC) and a plurality of virtual machines (VMs). For ease of illustration, Fig.12 The compute hosts 1228 are depicted as including two virtual machines, namely, VM1 and VM2, respectively. Note that each of the VMs is communicatively coupled to a host NIC via one of the logical interfaces (e.g., the logical interfaces depicted as PF1 and PF2, respectively). Also, note that the local NVD 1226 can be located on the same chassis as the host NIC included in the compute instances 1228.

[0193] According to some embodiments, the local NVD 1226 has multiple physical ports. Fig.12 In one embodiment shown in FIG. 1 , the local NVD 1226 has two physical ports: a first physical port 1227A (referred to herein as a TOR-facing port) connected to the TORs 1222 and 1224, respectively, and a second physical port 1227B (referred to herein as a host-facing port) connected to the computing host 1228. The physical port 1227A of the local NVD 1226 can be divided into a plurality of logical ports. For example, Fig.12As shown in FIG. 1 , the physical port 1227A is divided into two logical ports on the side facing the TOR, that is, each logical port is connected to a corresponding TOR included in the data center. The physical port 1227B is connected to the host NIC. Therefore, Fig. 9 Compared with the DRCC implementation, Fig.12 The DRCC implementation includes a single connection from NVD 1226 to the host NIC included in the compute instance 1228. Therefore, the host NIC is associated with a single pair of IP overlays and MAC addresses (i.e., B1 and M4 ). This overlay IP address of the host NIC can be reached via two base paths, i.e., via different TORs (i.e., TOR#1 and TOR#2), thereby providing TOR redundancy.

[0194] Fig.13 Depicted is a flow chart illustrating another process for providing a DRCC in accordance with some embodiments. Fig.13 The processes described in the can be implemented in software (e.g., code, instructions, programs), hardware, or a combination thereof, executed by one or more processing units (e.g., processors, cores) of the corresponding system. The software can be stored on a non-transitory storage medium (e.g., a memory device). Fig.13 The methods presented in and described below are intended to be illustrative and non-limiting. Fig.13 Various processing steps are depicted as occurring in a particular order or sequence, but this is not intended to be limiting. In some alternative embodiments, the steps may be performed in a different order, or some steps may be performed in parallel. In some implementations, Fig.13 The method shown in can be performed by a cloud service provider to provide DRCC to customers.

[0195] The process begins at step 1301, where a first physical port of a network virtualization device (NVD) included in a data center is communicatively coupled to a first top-of-rack (TOR) switch and a second TOR switch. Note that the first and second TOR switches may be included in a rack (e.g., Figure 6 and Figure 7 In step 1303, a second physical port of the NVD is communicatively coupled to a network interface card (NIC) associated with a host machine included in the data center.

[0196] The process then moves to step 1305, where the NVD receives a packet from the host machine via the second physical port. In step 1307, the NVD determines a particular TOR for transmitting the packet from the group including the first TOR and the second TOR. According to some embodiments, the NVD may perform equal cost multi-path routing (ECMP) flow hashing to select one of the two TORs. Additionally, in step 1309, the NVD transmits the packet to the particular TOR to facilitate transmission of the packet to the remote host machine, e.g., Fig.12 The remote host 1211.

[0197] A dedicated regional cloud (DRCC) at the customer corresponds to a cloud service provider's infrastructure provided in the customer's own data center. The DRCC framework brings the full power of the public cloud to on-premises deployments so that enterprises can reduce infrastructure and operating costs, upgrade legacy applications on modern cloud services, and meet the most stringent regulatory, data residency, and latency requirements - all of which are achieved through the CSP's infrastructure. Therefore, it is desirable to design a network architecture that occupies a small footprint and enables cloud services to be provided at an on-premises location of the customer's choice. The naive solution to enable the DRCC framework is to use a network design such as that used in commercial areas. However, the disadvantage of this approach is that customers typically do not have the power and space requirements to deploy such a network, and customers may not be able to take advantage of the full scale of such a network architecture. Therefore, a novel network architecture with a reduced footprint (e.g., fewer devices, racks, etc.) is required to provide cloud services to customers at a location of the customer's choice.

[0198] Fig.14 An exemplary network fabric architecture for a DRCC according to some embodiments is depicted. The network fabric architecture 1400 for a DRCC includes a combination of a computing fabric block (referred to herein as a CFAB, 1420A-1420B) and a network fabric block (referred to herein as a NFAB, 1415). The NFAB 1415 is communicatively coupled to each of the CFAB blocks (1420A-1420B) via a plurality of switch blocks (1405, 1410). The plurality of switch blocks are also referred to herein as layer 3 (T3) level switches. Each of the plurality of switch blocks includes a predetermined number of switches, for example, four switches. For example, the switch block 1405 includes four switches labeled 1405A-1405D, and the switch block 1410 includes four switches labeled 1410A-1410D.

[0199] According to some embodiments, a compute fabric block (e.g., CFAB block 1420A) is communicatively coupled to a plurality of switch blocks 1405, 1410. The compute fabric block 1420A includes a collection of one or more racks (e.g., Exadata racks 1424 or compute racks 1423). Each rack in the collection of one or more racks includes one or more servers configured to execute one or more workloads of a customer. It can be appreciated that each Exadata rack 1424 can be associated with a cluster network 1425 of virtual machines. The CFAB block 1420A also includes a first plurality of switches 1422 organized into a first plurality of levels (e.g., levels labeled CFAB T1 and CFAB T2). The first plurality of switches 1422 communicatively couples a collection of one or more racks (e.g., racks 1423, 1424) to the plurality of switch blocks (1405, 1410). Specifically, the first plurality of levels associated with the first plurality of switches 1422 in the compute fabric block 1420A include: (i) a first hierarchy 1 level switch (ie, CFAB T1), and (ii) a first hierarchy 2 level switch (ie, CFAB T2).

[0200] The first tier-1 level switch is communicatively coupled to a set of one or more chassis at a first end and is communicatively coupled to a first tier-2 level switch at a second end. In turn, the first tier-2 level switch (i.e., CFAB T2) connects the first tier-1 level switch (i.e., CFAB T1) to the plurality of switch blocks 1405, 1410. In one embodiment, the first tier-1 level switch (CFAB T1) in the computing fabric block includes eight switches, and the first tier-2 level switch (CFAB T2) in the computing fabric block includes four switches. Each of the first tier-1 level switches in the computing fabric block is connected to each of the first tier-2 level switches in the computing fabric block. In turn, each of the first tier-2 level switches (CFAB T2) in the computing fabric block is connected to at least one switch in each of the plurality of switch blocks 1405, 1410.

[0201] According to some embodiments, the NFAB block 1415 is communicatively coupled to the plurality of switch blocks 1405, 1410. The network fabric block 1415 includes: (i) one or more edge devices 1420, and (ii) a second plurality of switches 1418 organized into a second plurality of levels. The one or more edge devices 1420 include a first edge device that provides connectivity to a first external resource. For example, the first external resource may be a public communication network (e.g., the Internet), and the first edge device may be a gateway that provides connectivity to the public communication network. The one or more edge devices 1420 may include a gateway, a backbone edge device, a metropolitan edge device, and a route reflector. Thus, the first edge device (e.g., a gateway) enables a workload executed by a server included in a rack in a set of one or more racks included in the CFAB block 1420A to access a first external resource (e.g., the Internet).

[0202] A second plurality of switches 1418, organized into a second plurality of levels (labeled NFAB T1 and NFAB T2), communicatively couples one or more edge devices 1420 to the plurality of switch blocks 1405, 1410. The connections between the plurality of switch blocks and the second plurality of switches 1418 are Fig.14 The detailed connection of logical structure 1416 will be referred to later. Fig.15 According to some embodiments, the second plurality of levels associated with the second plurality of switches 1418 in the network fabric block 1415 include: (i)

[0203] The network fabric block includes a plurality of second level 1 switches (i.e., NFAB T1), and (ii) a second level 2 switch (i.e., NFABT2). Each of the second level 2 switches is communicatively coupled to each of the switches included in the second level 1 switches (i.e., NFABT1). In one embodiment, the second level 1 switches in the network fabric block include eight switches, and the second level 2 switches in the network fabric block include four switches.

[0204] According to some embodiments, the initial deployment of the DRCC network architecture includes deploying a NFAB block (1415), one CFAB block (1420A), and a T3 switch layer (i.e., multiple switch blocks 1405, 1410) that interconnects the NFAB to the CFAB. It can be appreciated that additional CFAB blocks (and additional T3 layer switch blocks) can be deployed on the fly (i.e., in real time) based on customer demand. It can be appreciated that the number of switches (e.g., eight switches) described above with reference to the first layer 1 level switches or the second layer 1 level switches in the network structure is for illustration purposes only. The number of switches included in the first layer 1 level or the second layer 1 level can be any other number of switches, such as four or sixteen or a variable number of switches. In a similar manner, the second layer 2 level switches in the network structure block described above include four switches. However, it can be appreciated that this is for illustration purposes only, and the actual number of switches in this level can be a variable number of switches, for example, half the number of switches in layer-1.

[0205] Fig.15 The diagram illustrates connections between the NFAB block and the plurality of switch blocks, as well as connections within the NFAB block (i.e., between the second plurality of switches organized into a second plurality of levels in the NFAB). The second plurality of levels associated with the second plurality of switches in the NFAB include: (i) second layer 1 level switches (i.e., NFAB layer 1, 1505) and (ii) second layer 2 level switches (i.e., NFAB layer 2, 1510). In one example, the second layer 1 level switches in the NFAB include eight switches (i.e., Fig.15 The second layer 2 level switches in the network structure block include four switches (in Fig.15 The labels are t2-r1 to t2-r4).

[0206] like Fig.15 As shown in , a first subset of switches included in a second layer 1 level switch (i.e., NFAB layer 1) is communicatively coupled to one or more edge devices at a first end. For example, as Fig.15 As shown in FIG. 1 , switches t1-r1, t1-r2, t1-r3, and t1-r4 are respectively coupled to route reflector 1520A and VPN gateway 1520. In addition, a second subset of switches (e.g., switches t1-r5, t1-r6, t1-r7, and t1-r8) included in the second hierarchy 1 level switches are communicatively coupled at a first end to a plurality of switch blocks (i.e., Fig.153). Note that the second subset of switches may also be coupled to a WDM metro switch (i.e., a switch for interconnecting racks located in different buildings). The first subset and the second subset of switches included in the second layer 1 level switches are coupled at a second end to a second layer 2 level switch included in the network fabric block.

[0207] Specifically, in order to provide connectivity between switches of the T1 layer 1505, a T2 layer switch (e.g., four switches) 1510 is adopted in the NFAB. As shown in the figure, each of the four switches included in the T2 layer switch of the NFAB is connected to each T1 layer switch. In this way, the service enclaves included in different CFAB blocks are communicatively coupled to the edge devices via the NFAB structure. Therefore, the workload executed by the server included in the rack of the computing structure block accesses the first external resource (e.g., the Internet) by establishing a connection with the first switch in the multiple switch blocks. Reference Fig.15 , this connection is further routed as follows: (i) from the first switch to a second switch included in the second subset of switches included in the second layer 1 level switches (e.g., switches t1-r5, t1-r6, t1-r7, and t1-r8), (ii) from the second switch to a third switch included in the second layer 2 level switches (e.g., one of the switches from the group of switches t1-r1, t1-r2, t1-r3, t1-r4), (iii) from the third switch to a fourth switch included in the first subset of switches in the second layer 1 level switches (e.g., switches t1-r1, t1-r2, t1-r3, and t1-r4), and (iv) from the fourth switch to the gateway. It can be appreciated that the above description of Fig.14 and Fig.15 The CFAB and JFAB blocks described support 400G connections and operate with a power budget of 100KW. The architecture includes a total of 3 network racks and an optional fourth rack to support backbone and metro connections.

[0208] Fig.16 An exemplary dedicated backbone network for a customer region is illustrated in accordance with some embodiments. Fig.16 Several geographic locations (e.g., countries) are depicted where a customer DRCC may be deployed. For example, Fig.16Three geographic locations are depicted, namely, geographic location A 1602, geographic location B 1604, and geographic location C 1606, where customer DRCCs are deployed. Within each geographic location, it is assumed that the customer has deployed DRCCs in two regions within the geographic location. For example, geographic location A has two regions, Region I 1602A and Region II 1602B, geographic location B has two regions, Region I 1604A and Region II 1604B, and geographic location C has two regions, Region I 1606A and Region II 1606B, in which customer DRCCs are deployed, respectively.

[0209] Each region includes a pair of dedicated backbone routers, namely, Router 1 and Router 2. Router 1 in each region is based on Fig.16 In a similar manner, the routers 2 in each region are connected in the order shown in Fig.16 The dedicated backbone ring network is connected in the order shown in to form another dedicated backbone ring network. It can be appreciated that the dedicated backbone ring network is dedicated to the customer, that is, no other traffic is allowed on the backbone network. The backbone network can support 10G or 100G encrypted connections and provide single link failure, that is, the interruption of a single backbone link included in the ring formed by router 1 or the ring formed by router 2 will not interfere with the traffic on the backbone network. It can be appreciated that Fig.16 The ring topology depicted in is for illustration purposes only. The backbone network topology may be mesh, ring, or any other topology based on certain criteria such as the number of DRCC regions desired by the customer and / or the customer's latency and bandwidth requirements.

[0210] Steering Fig.17 , depicts a flowchart illustrating a process of constructing a network structure according to some embodiments. Fig.17 The processes described in the can be implemented in software (e.g., code, instructions, programs), hardware, or a combination thereof, executed by one or more processing units (e.g., processors, cores) of the corresponding system. The software can be stored on a non-transitory storage medium (e.g., a memory device). Fig.17 The methods presented in and described below are intended to be illustrative and non-limiting. Fig.17 Various processing steps are depicted as occurring in a particular order or sequence, but this is not intended to be limiting. In some alternative embodiments, the steps may be performed in a different order, or some steps may be performed in parallel. In some embodiments, Fig.17 The method shown in can be performed by a cloud service provider to provide DRCC to customers.

[0211] The process begins at step 1701, where a plurality of switch blocks are provided. Fig.141405 and 1410 shown in correspondence. Then, the process moves to step 1703, where a first computing fabric block is provided, which is communicatively coupled to the plurality of switch blocks. The first computing fabric block includes: (i) a set of one or more racks, and (ii) a first plurality of switches organized into a first plurality of levels. Each rack in the set of one or more racks includes one or more servers configured to execute one or more workloads of a customer. The first plurality of switches communicatively couples the set of one or more racks to the plurality of switch blocks.

[0212] Thereafter, processing proceeds to step 1705, where a network fabric block is provided that is communicatively coupled to the plurality of switch blocks. The network fabric block includes: (i) one or more edge devices, including, and (ii) a second plurality of switches organized into a second plurality of levels. The first edge device provides connectivity to a first external resource. For example, the first external resource may be a public communications network (e.g., the Internet), and the first edge device is a gateway that provides connectivity to the public communications network. The first edge device enables a workload executed by a server included in a rack in a collection of one or more racks to access the first external resource. The second plurality of switches communicatively couples the one or more edge devices to the plurality of switch blocks.

[0213] Example Cloud Infrastructure Embodiment

[0214] As noted above, Infrastructure as a Service (IaaS) is a specific type of cloud computing. IaaS can be configured to provide virtualized computing resources over a public network (e.g., the Internet). In the IaaS model, a cloud computing provider can host infrastructure components (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer), etc.). In some cases, IaaS providers can also provide various services to accompany these infrastructure components (e.g., billing, monitoring, logging, security, load balancing, and clustering, etc.). Therefore, since these services can be policy-driven, IaaS users can be able to implement policies to drive load balancing to maintain the availability and performance of applications.

[0215] In some cases, IaaS customers can access resources and services over a wide area network (WAN) such as the Internet, and can use the cloud provider's services to install the remaining elements of the application stack. For example, a user can log in to the IaaS platform to create a virtual machine (VM), install an operating system (OS) on each VM, deploy middleware such as a database, create storage buckets for workloads and backups, and even install enterprise software into the VM. Customers can then use the provider's services to perform a variety of functions, including balancing network traffic, troubleshooting application problems, monitoring performance, managing disaster recovery, and more.

[0216] In most cases, the cloud computing model will require the involvement of a cloud provider. A cloud provider may, but need not, specialize in providing (e.g., provisioning, renting, selling) third-party IaaS services. An entity may also choose to deploy a private cloud and thereby become its own provider of infrastructure services.

[0217] In some examples, IaaS deployment is the process of placing a new application or a new version of an application onto a prepared application server, etc. It may also include the process of preparing the server (e.g., installing libraries, daemons, etc.). This is typically managed by the cloud provider, below the hypervisor layer (e.g., servers, storage, network hardware, and virtualization). Thus, the customer may be responsible for handling (OS), middleware, and / or application deployment (e.g., on a self-service virtual machine (e.g., that can be started on demand), etc.).

[0218] In some examples, IaaS provisioning can refer to obtaining computers or virtual hosts for use, and even installing required libraries or services on them. In most cases, deployment does not include provisioning, and provisioning may need to be performed first.

[0219] In some cases, there are two distinct problems with IaaS provisioning. First, there is the initial challenge of provisioning the initial set of infrastructure before anything is running. Second, once everything has been provisioned, there is the challenge of evolving the existing infrastructure (e.g., adding new services, changing services, removing services, etc.). In some cases, both of these challenges can be addressed by enabling configuration that defines the infrastructure in a declarative manner. In other words, the infrastructure (e.g., which components are required and how they interact) can be defined by one or more configuration files. Thus, the overall topology of the infrastructure (e.g., which resources depend on which resources, and how they work together) can be described in a declarative manner. In some cases, once the topology is defined, workflows can be generated to create and / or manage the different components described in the configuration files.

[0220] In some examples, the infrastructure may have many interconnected elements. For example, there may be one or more virtual private clouds (VPCs) (e.g., a potential on-demand pool of configurable and / or shared computing resources), also referred to as a core network. In some examples, one or more security group rules may also be provided to define how to set up security for the network and one or more virtual machines (VMs). Other infrastructure elements may also be provided, such as load balancers, databases, etc. The infrastructure may evolve gradually as more and more infrastructure elements are desired and / or added.

[0221] In some cases, continuous deployment techniques can be employed to enable deployment of infrastructure code across various virtual computing environments. In addition, the described techniques can enable infrastructure management within these environments. In some examples, a service team can write code that is expected to be deployed to one or more but typically many different production environments (e.g., across various different geographic locations, sometimes across the world). However, in some examples, the infrastructure on which the code will be deployed must first be set up. In some cases, provisioning can be done manually, resources can be provisioned using a provisioning tool, and / or code can be deployed using a deployment tool once the infrastructure is provisioned.

[0222] Fig.18 18 is a block diagram 1800 illustrating an example mode of an IaaS architecture according to at least one embodiment. A service operator 1802 can be communicatively coupled to a secure host lease 1804 that can include a virtual cloud network (VCN) 1806 and a secure host subnet 1808. In some examples, the service operator 1802 can use one or more client computing devices, which can be portable handheld devices (e.g., Cellular phone, computing tablets, personal digital assistants (PDAs), or wearable devices (e.g., Google head mounted display), running software such as Microsoft Windows ) and / or various mobile operating systems (such as iOS, Windows Phone, Android, BlackBerry 8, Palm OS, etc.), and supports Internet, email, short message service (SMS), Alternatively, the client computing device may be a general-purpose personal computer, including, for example, a computer running various versions of Microsoft Apple The client computing device may be a personal computer and / or laptop computer running various commercially available or UNIX-like operating systems, including but not limited to any of the various GNU / Linux operating systems (such as, for example, Google Chrome OS). Alternatively or additionally, the client computing device may be any other electronic device, such as a thin client computer, an Internet-enabled gaming system (e.g., with or without gesture input device), and / or a personal messaging device capable of communicating over a network that can access VCN 1806 and / or the Internet.

[0223] VCN 1806 may include a local peer gateway (LPG) 1810, which may be communicatively coupled to a secure shell (SSH) VCN 1812 via LPG 1810 contained in SSH VCN 1812. SSH VCN 1812 may include an SSH subnet 1814, and SSH VCN 1812 may be communicatively coupled to control plane VCN 1816 via LPG 1810 contained in control plane VCN 1816. Also, SSH VCN 1812 may be communicatively coupled to data plane VCN 1818 via LPG 1810. Control plane VCN 1816 and data plane VCN 1818 may be contained in a service lease 1819, which may be owned and / or operated by an IaaS provider.

[0224] The control plane VCN 1816 may include a control plane demilitarized zone (DMZ) layer 1820 that acts as a perimeter network (e.g., a portion of a corporate network between a corporate intranet and an external network). DMZ-based servers may have limited liability and help control security vulnerabilities. In addition, the DMZ layer 1820 may include (one or more) load balancer (LB) subnets 1822, a control plane application layer 1824 that may include (one or more) application subnets 1826, and a control plane data layer 1828 that may include (one or more) database (DB) subnets 1830 (e.g., (one or more) front-end DB subnets and / or (one or more) back-end DB subnets). The LB subnet(s) 1822 included in the control plane DMZ layer 1820 may be communicatively coupled to the application subnet(s) 1826 included in the control plane application layer 1824 and the Internet gateway 1834 that may be included in the control plane VCN 1816, and the application subnet(s) 1826 may be communicatively coupled to the DB subnet(s) 1830 included in the control plane data layer 1828 and the service gateway 1836 and the network address translation (NAT) gateway 1838. The control plane VCN 1816 may include the service gateway 1836 and the NAT gateway 1838.

[0225] The control plane VCN 1816 may include a data plane mirror application layer 1840, which may include application subnet(s) 1826. The application subnet(s) 1826 contained in the data plane mirror application layer 1840 may include a virtual network interface controller (VNIC) 1842 that may execute a compute instance 1844. The compute instance 1844 may communicatively couple the application subnet(s) 1826 of the data plane mirror application layer 1840 to the application subnet(s) 1826 that may be contained in the data plane application layer 1846.

[0226] The data plane VCN 1818 may include a data plane application layer 1846, a data plane DMZ layer 1848, and a data plane data layer 1850. The data plane DMZ layer 1848 may include (one or more) LB subnets 1822, which may be communicatively coupled to (one or more) application subnets 1826 of the data plane application layer 1846 and an internet gateway 1834 of the data plane VCN 1818. The (one or more) application subnets 1826 may be communicatively coupled to a service gateway 1836 of the data plane VCN 1818 and a NAT gateway 1838 of the data plane VCN 1818. The data plane data layer 1850 may also include (one or more) DB subnets 1830, which may be communicatively coupled to (one or more) application subnets 1826 of the data plane application layer 1846.

[0227] The internet gateway 1834 of the control plane VCN 1816 and the data plane VCN 1818 may be communicatively coupled to a metadata management service 1852, which may be communicatively coupled to a public Internet 1854. The public Internet 1854 may be communicatively coupled to a NAT gateway 1838 of the control plane VCN 1816 and the data plane VCN 1818. The service gateway 1836 of the control plane VCN 1816 and the data plane VCN 1818 may be communicatively coupled to a cloud service 1856.

[0228] In some examples, service gateway 1836 of control plane VCN 1816 or data plane VCN 1818 can make application programming interface (API) calls to cloud services 1856 without going through public internet 1854. API calls from service gateway 1836 to cloud services 1856 can be one-way: service gateway 1836 can make API calls to cloud services 1856, and cloud services 1856 can send requested data to service gateway 1836. However, cloud services 1856 may not initiate API calls to service gateway 1836.

[0229] In some examples, secure host lease 1804 can be directly connected to service lease 1819, which can otherwise be isolated. Secure host subnet 1808 can communicate with SSH subnet 1814 through LPG 1810, which can enable two-way communication on otherwise isolated systems. Connecting secure host subnet 1808 to SSH subnet 1814 can enable secure host subnet 1808 to access other entities within service lease 1819.

[0230] The control plane VCN 1816 may allow a user of the service lease 1819 to set up or otherwise provision desired resources. The desired resources provisioned in the control plane VCN 1816 may be deployed or otherwise used in the data plane VCN 1818. In some examples, the control plane VCN 1816 may be isolated from the data plane VCN 1818, and the data plane mirror application layer 1840 of the control plane VCN 1816 may communicate with the data plane application layer 1846 of the data plane VCN 1818 via the VNIC 1842, which may be included in the data plane mirror application layer 1840 and the data plane application layer 1846.

[0231] In some examples, a user or customer of the system may make a request, such as a create, read, update, or delete (CRUD) operation, through the public internet 1854 which may transmit the request to the metadata management service 1852. The metadata management service 1852 may transmit the request to the control plane VCN 1816 through the internet gateway 1834. The request may be received by the LB subnet(s) 1822 contained in the control plane DMZ layer 1820. The LB subnet(s) 1822 may determine that the request is valid, and in response to the determination, the LB subnet(s) 1822 may transmit the request to the application subnet(s) 1826 contained in the control plane application layer 1824. If the request is validated and a call to the public internet 1854 is required, the call to the public internet 1854 may be transmitted to the NAT gateway 1838 which may make the call to the public internet 1854. The storage that the request may desire to be stored may be stored in the DB subnet(s) 1830.

[0232] In some examples, data plane mirror application layer 1840 may facilitate direct communication between control plane VCN 1816 and data plane VCN 1818. For example, it may be desirable to apply changes, updates, or other appropriate modifications to the configuration to the resources contained in data plane VCN 1818. Control plane VCN 1816 may communicate directly with the resources contained in data plane VCN 1818 via VNIC 1842 and may thereby perform changes, updates, or other appropriate modifications to the configuration.

[0233] In some embodiments, control plane VCN 1816 and data plane VCN 1818 may be included in service lease 1819. In this case, a user or customer of the system may not own or operate control plane VCN 1816 or data plane VCN 1818. Alternatively, an IaaS provider may own or operate control plane VCN 1816 and data plane VCN 1818, both of which may be included in service lease 1819. This embodiment may enable isolation of networks that may prevent a user or customer from interacting with the resources of other users or other customers. Furthermore, this embodiment may allow a user or customer of the system to store a database privately without relying on the public Internet 1854, which may not have the desired level of threat prevention, for storage.

[0234] In other embodiments, the LB subnet(s) 1822 contained in the control plane VCN 1816 can be configured to receive signals from the service gateway 1836. In this embodiment, the control plane VCN 1816 and the data plane VCN 1818 can be configured to be called by customers of the IaaS provider without calling the public Internet 1854. Customers of the IaaS provider may desire this embodiment because the database(s) used by the customer can be controlled by the IaaS provider and can be stored on a service lease 1819, which may be isolated from the public Internet 1854.

[0235] Fig.19 1900 is a block diagram illustrating another example model of an IaaS architecture according to at least one embodiment. A service operator 1902 (e.g., Fig.18 The service operator 1802) can be communicatively coupled to the secure host lease 1904 (e.g., Fig.18 A secure host lease 1804 of the embodiment of the present invention may include a virtual cloud network (VCN) 1906 (e.g., Fig.18 VCN 1806) and secure host subnet 1908 (e.g., Fig.18 The VCN 1906 may include a local peer gateway (LPG) 1910 (e.g., Fig.18 The local peer gateway may be communicatively coupled to a secure shell (SSH) VCN 1912 (e.g., LPG 1810 of SSHVCN 1912) via LPG 1810 contained in SSHVCN 1912. Fig.18 SSH VCN 1812). SSH VCN 1912 may include SSH subnet 1914 (e.g., Fig.18 SSH subnet 1814), and SSH VCN 1912 can be communicatively coupled to control plane VCN 1916 via LPG 1910 contained in control plane VCN 1916 (e.g., Fig.18 The control plane VCN 1816). The control plane VCN 1916 can be included in the service lease 1919 (e.g., Fig.18 service lease 1819), and the data plane VCN 1918 (e.g., Fig.18 The data plane VCN 1818) may be contained in a customer lease 1921 which may be owned or operated by a user or customer of the system.

[0236] The control plane VCN 1916 may include a control plane DMZ layer 1920 (e.g., Fig.18 1820), which may include (one or more) LB subnets 1922 (e.g., Fig.18 (one or more) LB subnets 1822), may include (one or more) application subnets 1926 (e.g., Fig.18 (one or more) application subnets 1826) of the control plane application layer 1924 (e.g., Fig.18 The control plane application layer 1824 of FIG. 1824 may include (one or more) database (DB) subnets 1930 (e.g., similar to Fig.18 (one or more) DB subnets 1830) of the control plane data layer 1928 (e.g., Fig.18 The LB subnet(s) 1922 contained in the control plane DMZ layer 1920 may be communicatively coupled to the application subnet(s) 1926 contained in the control plane application layer 1924 and the internet gateway 1934 (e.g., Fig.18 1834), and the application subnet(s) 1926 may be communicatively coupled to the DB subnet(s) 1930 and the service gateway 1936 (e.g., Fig.18 ) and a network address translation (NAT) gateway 1938 (e.g., Fig.18NAT gateway 1838). The control plane VCN 1916 may include a service gateway 1936 and a NAT gateway 1938.

[0237] The control plane VCN 1916 may include a data plane mirror application layer 1940 (e.g., Fig.18 The data plane image application layer 1840). The application subnet(s) 1926 included in the data plane image application layer 1940 may include a computing instance 1944 (e.g., similar to Fig.18 The compute instance 1944 may facilitate the application subnet(s) 1926 of the data plane mirroring application layer 1940 and the application subnets 1926 that may be included in the data plane application layer 1946 (e.g., Fig.18 Communication between (one or more) application subnets 1926 in the data plane application layer 1846) via VNIC 1942 contained in the data plane mirror application layer 1940 and VNIC 1942 contained in the data plane application layer 1946.

[0238] The Internet gateway 1934 included in the control plane VCN 1916 can be communicatively coupled to the metadata management service 1952 (e.g., Fig.18 1852), which can be communicatively coupled to the public Internet 1954 (e.g., Fig.18 The public Internet 1854 of the control plane VCN 1916 can be communicatively coupled to the NAT gateway 1938 included in the control plane VCN 1916. The service gateway 1936 included in the control plane VCN 1416 can be communicatively coupled to the cloud service 1956 (e.g., Fig.18 Cloud service 1856).

[0239] In some examples, the data plane VCN 1918 may be contained in a customer lease 1921. In this case, the IaaS provider may provide a control plane VCN 1916 for each customer, and the IaaS provider may provision each customer with a unique compute instance 1944 contained in a service lease 1919. Each compute instance 1944 may allow communication between a control plane VCN 1916 contained in a service lease 1919 and a data plane VCN 1918 contained in a customer lease 1921. The compute instance 1944 may allow resources provisioned in a control plane VCN 1916 contained in a service lease 1919 to be deployed or otherwise used in a data plane VCN 1918 contained in a customer lease 1921.

[0240] In other examples, a customer of the IaaS provider may have a database that exists in the customer lease 1921. In this example, the control plane VCN 1916 may include a data plane mirror application layer 1940, which may include (one or more) application subnets 1926. The data plane mirror application layer 1940 may reside in the data plane VCN 1918, but the data plane mirror application layer 1940 may not be in the data plane VCN 1918. That is, the data plane mirror application layer 1940 may access the customer lease 1921, but the data plane mirror application layer 1940 may not exist in the data plane VCN 1918 or be owned or operated by the customer of the IaaS provider. The data plane mirror application layer 1940 may be configured to make calls to the data plane VCN 1918, but may not be configured to make calls to any entity contained in the control plane VCN 1916. A customer may desire to deploy or otherwise use resources provisioned in control plane VCN 1916 in data plane VCN 1918, and data plane mirror application layer 1940 may facilitate the customer's desired deployment or other use of resources.

[0241] In some embodiments, a customer of an IaaS provider may apply filters to the data plane VCN 1918. In this embodiment, the customer may determine what the data plane VCN 1918 may access, and the customer may restrict access to the public Internet 1954 from the data plane VCN 1918. The IaaS provider may not apply filters or otherwise control access to any external network or database by the data plane VCN 1918. The customer applying filters and controls to the data plane VCN 1918 contained in the customer lease 1921 may help isolate the data plane VCN 1918 from other customers and the public Internet 1954.

[0242] In some embodiments, cloud services 1956 may be called by service gateway 1936 to access services that may not exist on public Internet 1954, control plane VCN 1916, or data plane VCN 1918. The connection between cloud services 1956 and control plane VCN 1916 or data plane VCN 1918 may not be real-time or continuous. Cloud services 1956 may exist on different networks owned or operated by IaaS providers. Cloud services 1956 may be configured to receive calls from service gateway 1936 and may be configured not to receive calls from public Internet 1954. Some cloud services 1956 may be isolated from other cloud services 1956, and control plane VCN 1916 may be isolated from cloud services 1956 that may not be in the same region as control plane VCN 1916. For example, control plane VCN 1916 may be located in "Region 1", and cloud service "Deployment 13" may be located in Region 1 and in "Region 2". If a service gateway 1936 contained in a control plane VCN 1916 located in region 1 makes a call to deployment 13, the call may be transmitted to deployment 13 in region 1. In this example, control plane VCN 1916 or deployment 13 in region 1 may not be communicatively coupled or otherwise in communication with deployment 13 in region 2.

[0243] Fig. 20 2 is a block diagram 2000 illustrating another example model of an IaaS architecture according to at least one embodiment. A service operator 2002 (e.g., Fig.18 The service operator 1802) can be communicatively coupled to the secure host lease 2004 (e.g., Fig.18 A secure host lease 1804) may include a virtual cloud network (VCN) 2006 (e.g., Fig.18 VCN 1806) and secure host subnet 2008 (for example, Fig.18 VCN 2006 may include LPG 2010 (e.g., Fig.18 1810), which may be communicatively coupled to the SSH VCN 2012 via the LPG 2010 contained in the SSH VCN 2012 (e.g., Fig.18 SSH VCN 2012 may include SSH subnet 2014 (e.g., Fig.18 SSH subnet 1814), and SSH VCN 1812 can be communicatively coupled to control plane VCN 2016 via LPG 2010 included in control plane VCN 2016 (e.g., Fig.18The control plane VCN 1816 of the data plane VCN 2018 is coupled to the data plane VCN 2018 via the LPG 2010 included in the data plane VCN 2018 (e.g., Fig.18 The control plane VCN 2016 and the data plane VCN 2018 may be included in a service lease 2019 (e.g., Fig.18 of service lease 1819).

[0244] The control plane VCN 1816 may include a load balancer (LB) subnet 1822 (e.g., Fig.18 (one or more) LB subnets 1822) of the control plane DMZ layer 1820 (e.g., Fig.18 The control plane DMZ layer 1820 may include (one or more) application subnets 2026 (e.g., similar to Fig.18 (one or more) application subnets 1826) of the control plane application layer 2024 (e.g., Fig.18 The control plane application layer 1824 of the embodiment of the present invention may include a control plane data layer 2028 (e.g., Fig.18 The LB subnet(s) 2022 contained in the control plane DMZ layer 2020 may be communicatively coupled to the application subnet(s) 2026 contained in the control plane application layer 2024 and the Internet gateway 1834 (e.g., Fig.18 1834), and the application subnet(s) 2026 may be communicatively coupled to the DB subnet(s) 1830 and the service gateway 1836 (e.g., Fig.18 ) and a network address translation (NAT) gateway 1838 (e.g., Fig.18 NAT gateway 1838). The control plane VCN 2016 may include a service gateway 2036 and a NAT gateway 2038.

[0245] The data plane VCN 2018 may include a data plane application layer 2046 (e.g., Fig.18 Data plane application layer 1846), data plane DMZ layer 2048 (e.g., Fig.18 Data plane DMZ layer 1848), and data plane data layer 2050 (e.g., Fig.18The data plane DMZ layer 2048 may include (one or more) trusted application subnets 2060 and (one or more) untrusted application subnets 2062 that may be communicatively coupled to the data plane application layer 2046 and (one or more) LB subnets 2022 of the Internet gateway 2034 included in the data plane VCN 2018. The (one or more) trusted application subnets 2060 may be communicatively coupled to the service gateway 2036 included in the data plane VCN 2018, the NAT gateway 2038 included in the data plane VCN 2018, and (one or more) DB subnets 2030 included in the data plane data layer 2050. The (one or more) untrusted application subnets 2062 may be communicatively coupled to the service gateway 2036 included in the data plane VCN 2018 and (one or more) DB subnets 2030 included in the data plane data layer 2050. The data plane data layer 2050 may include (one or more) DB subnets 2030 that may be communicatively coupled to a service gateway 2036 included in the data plane VCN 2018.

[0246] The untrusted application subnet(s) 2062 may include one or more primary VNICs 2064(1)-(N) that may be communicatively coupled to tenant virtual machines (VMs) 2066(1)-(N). Each tenant VM 2066(1)-(N) may be communicatively coupled to a respective application subnet 2067(1)-(N) that may be contained in a respective container egress VCN 2068(1)-(N), which may be contained in a respective customer tenancy 2070(1)-(N). The respective secondary VNICs 2072(1)-(N) may facilitate communications between the untrusted application subnet(s) 2062 contained in the data plane VCN 2018 and the application subnets contained in the container egress VCNs 2068(1)-(N). Each container egress VCN 2068(1)-(N) may include a NAT gateway 2038 that may be communicatively coupled to the public Internet 2054 (e.g., Fig.18 of the public Internet 1854).

[0247] The Internet gateway 2034 included in the control plane VCN 2016 and included in the data plane VCN 2018 can be communicatively coupled to the metadata management service 2052 (e.g., Fig.18The metadata management system 1852 of the embodiment of the present invention can be communicatively coupled to the public Internet 2054. The public Internet 2054 can be communicatively coupled to the NAT gateway 2038 included in the control plane VCN 2016 and included in the data plane VCN 2018. The service gateway 2036 included in the control plane VCN 2016 and included in the data plane VCN 2018 can be communicatively coupled to the cloud service 2056.

[0248] In some embodiments, data plane VCN 2018 can be integrated with customer lease 2070. Such integration can be useful or desirable for customers of the IaaS provider in some cases, such as situations where support may be desired when executing code. A customer may provide code to run that may be destructive, may communicate with other customer resources, or may otherwise cause undesirable effects. In response to this, the IaaS provider can determine whether to run code given to the IaaS provider by the customer.

[0249] In some examples, a customer of an IaaS provider may grant temporary network access to the IaaS provider and request functionality attached to a data plane layer application 2046. Code running the functionality may be executed in a VM 2066(1)-(N), and the code may not be configured to run anywhere else on the data plane VCN 2018. Each VM 2066(1)-(N) may be connected to a customer tenancy 2070. The corresponding container 2071(1)-(N) contained in the VM 2066(1)-(N) may be configured to run the code. In this case, there may be double isolation (e.g., the container 2071(1)-(N) runs the code, where the container 2071(1)-(N) may be contained at least in a VM 2066(1)-(N) contained in (one or more) untrusted application subnets 2062), which may help prevent incorrect or otherwise undesirable code from damaging the IaaS provider's network or damaging the network of a different customer. Containers 2071(1)-(N) may be communicatively coupled to customer lease 2070 and may be configured to transmit or receive data from customer lease 2070. Containers 2071(1)-(N) may not be configured to transmit or receive data from any other entity in data plane VCN 2018. After running the code is complete, the IaaS provider may terminate or otherwise dispose of containers 2071(1)-(N).

[0250] In some embodiments, the trusted application subnet(s) 2060 may run code that may be owned or operated by the IaaS provider. In this embodiment, the trusted application subnet(s) 2060 may be communicatively coupled to the DB subnet(s) 2030 and configured to perform CRUD operations in the DB subnet(s) 2030. The untrusted application subnet(s) 2062 may be communicatively coupled to the DB subnet(s) 2030, but in this embodiment, the untrusted application subnet(s) may be configured to perform read operations in the DB subnet(s) 2030. The containers 2071(1)-(N) that may be included in each customer's VM 2066(1)-(N) and that may run code from the customer may not be communicatively coupled to the DB subnet(s) 2030.

[0251] In other embodiments, the control plane VCN 2016 and the data plane VCN 2018 may not be coupled in direct communication. In this embodiment, there may be no direct communication between the control plane VCN 2016 and the data plane VCN 2018. However, communication may occur indirectly through at least one method. The LPG 2010 may be established by an IaaS provider, which may facilitate communication between the control plane VCN 2016 and the data plane VCN 2018. In another example, the control plane VCN 2016 or the data plane VCN 2018 may call the cloud service 2056 via the service gateway 2036. For example, the call from the control plane VCN 2016 to the cloud service 2056 may include a request for a service that can communicate with the data plane VCN 2018.

[0252] Fig.21 21 is a block diagram 2100 illustrating another example model of an IaaS architecture according to at least one embodiment. A service operator 2102 (e.g., Fig.18 The service operator 1802) can be communicatively coupled to the secure host lease 2104 (e.g., Fig.18 A secure host lease 1804 of the embodiment of the present invention may include a virtual cloud network (VCN) 2106 (e.g., Fig.18 VCN 1806) and secure host subnet 2108 (e.g., Fig.18 VCN 2106 may include LPG 2110 (e.g., Fig.18 LPG 1810), which can be included in the SSH VCN 2112 (for example, Fig.18The LPG 2110 in the SSH VCN 1812 is communicatively coupled to the SSH VCN 2112. The SSH VCN 2112 may include an SSH subnet 2114 (e.g., Fig.18 SSH subnet 1814), and SSH VCN 2112 may be communicatively coupled to control plane VCN 2116 via LPG 2110 included in control plane VCN 2116 (e.g., Fig.18 The control plane VCN 1816 of the embodiment of the present invention is coupled to the data plane VCN 2118 via the LPG 2110 contained in the data plane VCN 2118 (e.g., Fig.18 The control plane VCN 2116 and the data plane VCN 2118 may be included in a service lease 2119 (e.g., Fig.18 of service lease 1819).

[0253] The control plane VCN 2116 may include (one or more) LB subnets 2122 (e.g., Fig.18 (one or more) LB subnets 1822) of the control plane DMZ layer 2120 (e.g., Fig.18 control plane DMZ layer 1820), may include (one or more) application subnets 2126 (e.g., Fig.18 (one or more) application subnets 1826) of the control plane application layer 2124 (e.g., Fig.18 control plane application layer 1824), may include (one or more) DB subnets 2130 (e.g., Fig. 20 (one or more) DB subnets 2030) of the control plane data layer 2128 (e.g., Fig.18 The LB subnet(s) 2122 contained in the control plane DMZ layer 2120 may be communicatively coupled to the application subnet(s) 2126 contained in the control plane application layer 2124 and the Internet gateway 2134 (e.g., Fig.18 1834), and the application subnet(s) 2126 may be communicatively coupled to the DB subnet(s) 2130 and the service gateway 2136 (e.g., Fig.18 ) and a network address translation (NAT) gateway 2138 (e.g., Fig.18 NAT gateway 1838). The control plane VCN 2116 may include a service gateway 2136 and a NAT gateway 2138.

[0254] The data plane VCN 2118 may include a data plane application layer 2146 (e.g., Fig.18 Data plane application layer 1846), data plane DMZ layer 2148 (e.g., Fig.18 Data plane DMZ layer 2148)), and data plane data layer 2150 (e.g., Fig.18 The data plane DMZ layer 2148 may include (one or more) trusted application subnets 2160 (e.g., Fig. 20 (one or more) trusted application subnets 2060) and (one or more) untrusted application subnets 2162 (e.g., Fig. 20 The trusted application subnet(s) 2160 may be communicatively coupled to the service gateway 2136 contained in the data plane VCN 2118, the NAT gateway 2138 contained in the data plane VCN 2118, and the DB subnet(s) 2130 contained in the data plane data layer 2150. The untrusted application subnet(s) 2162 may be communicatively coupled to the service gateway 2136 contained in the data plane VCN 2118 and the DB subnet(s) 2130 contained in the data plane data layer 2150. The data plane data layer 2150 may include the DB subnet(s) 2130 that may be communicatively coupled to the service gateway 2136 contained in the data plane VCN 2118.

[0255] The untrusted application subnet(s) 2162 may include primary VNICs 2164(1)-(N) that may be communicatively coupled to tenant virtual machines (VMs) 2166(1)-(N) residing within the untrusted application subnet(s) 2162. Each tenant VM 2166(1)-(N) may run code in a corresponding container 2167(1)-(N) and may be communicatively coupled to an application subnet 2126 that may be contained in a data plane application layer 2146 contained in a container egress VCN 2168. Corresponding secondary VNICs 2172(1)-(N) may facilitate communications between the untrusted application subnet(s) 2162 contained in the data plane VCN 2118 and the application subnets contained in the container egress VCN 2168. The container egress VCN may include a primary VNIC that may be communicatively coupled to a public Internet 2154 (e.g., Fig.18 NAT gateway 2138 of the public Internet 1854).

[0256] The Internet gateway 2134 included in the control plane VCN 2116 and included in the data plane VCN 2118 can be communicatively coupled to the metadata management service 2152 (e.g., Fig.18 2154) can be communicatively coupled to the public Internet 2154. The public Internet 2154 can be communicatively coupled to the NAT gateway 2138 included in the control plane VCN 2116 and included in the data plane VCN 2118. The service gateway 2136 included in the control plane VCN 2116 and included in the data plane VCN 2118 can be communicatively coupled to the cloud service 2156.

[0257] In some examples, Fig.21 The architecture illustrated in block diagram 2100 may be considered as Fig. 20 2000 , and may be desired by customers of the IaaS provider if the IaaS provider cannot communicate directly with the customer (e.g., a disconnected region). The customer may access in real time the corresponding container 2167(1)-(N) contained in each customer's VM 2166(1)-(N). The container 2167(1)-(N) may be configured to make calls to the corresponding secondary VNIC 2172(1)-(N) contained in the application subnet(s) 2126 of the data plane application layer 2146, which may be contained in the container egress VCN 2168. The secondary VNIC 2172(1)-(N) may transmit the call to the NAT gateway 2138, which may transmit the call to the public Internet 2154. In this example, containers 2167(1)-(N), which may be accessed by customers in real time, may be isolated from control plane VCN 2116 and may be isolated from other entities contained in data plane VCN 2118. Containers 2167(1)-(N) may also be isolated from resources from other customers.

[0258] In other examples, a customer can use containers 2167(1)-(N) to call cloud service 2156. In this example, a customer can run code in containers 2167(1)-(N) to request a service from cloud service 2156. Containers 2167(1)-(N) can transmit the request to secondary VNICs 2172(1)-(N), which can transmit the request to a NAT gateway, which can transmit the request to the public Internet 2154. The public Internet 2154 can transmit the request to LB subnet(s) 2122 contained in control plane VCN 2116 via Internet gateway 2134. In response to determining that the request is valid, the LB subnet(s) can transmit the request to application subnet(s) 2126, which can transmit the request to cloud service 2156 via service gateway 2136.

[0259] It should be appreciated that the IaaS architectures 1800, 1900, 2000, 2100 depicted in the various figures may have other components in addition to those depicted. In addition, the embodiments shown in the various figures are merely some examples of cloud infrastructure systems that may be combined with embodiments of the present disclosure. In some other embodiments, the IaaS system may have more or fewer components than shown in the various figures, may combine two or more components, or may have a different configuration or arrangement of components.

[0260] In certain embodiments, the IaaS system described herein may include application suites, middleware, and database service offerings delivered to customers in a self-service, subscription-based, elastically scalable, reliable, highly available, and secure manner. An example of such an IaaS system is Oracle Cloud Infrastructure (OCI) provided by the assignee.

[0261] Fig. 22 An example computer system 2200 in which various embodiments of the present disclosure may be implemented is illustrated. System 2200 may be used to implement any of the computer systems described above. As shown in the figure, computer system 2200 includes a processing unit 2204 that communicates with multiple peripheral subsystems via a bus subsystem 2202. These peripheral subsystems may include a processing acceleration unit 2206, an I / O subsystem 2208, a storage subsystem 2218, and a communication subsystem 2224. Storage subsystem 2218 includes a tangible computer-readable storage medium 2222 and a system memory 2210.

[0262] The bus subsystem 2202 provides a mechanism for allowing the various components and subsystems of the computer system 2200 to communicate with each other by intention. Although the bus subsystem 2202 is schematically shown as a single bus, the alternative embodiment of the bus subsystem can utilize multiple buses. The bus subsystem 2202 can be any of several types of bus structures, including a memory bus or a memory controller, a peripheral bus, and a local bus using any various bus architectures. For example, this architecture can include an industrial standard architecture (ISA) bus, a microchannel architecture (MCA) bus, an enhanced ISA (EISA) bus, a video electronics standard association (VESA) local bus, and a peripheral component interconnect (PCI) bus, which can be implemented as a Mezzanine bus manufactured by the IEEE P1386.1 standard.

[0263] The processing unit 2204, which may be implemented as one or more integrated circuits (e.g., conventional microprocessors or microcontrollers), controls the operation of the computer system 2200. One or more processors may be included in the processing unit 2204. These processors may include single-core or multi-core processors. In certain embodiments, the processing unit 2204 may be implemented as one or more independent processing units 2232 and / or 2234, wherein a single or multi-core processor is included in each processing unit. In other embodiments, the processing unit 2204 may also be implemented as a quad-core processing unit formed by integrating two dual-core processors into a single chip.

[0264] In various embodiments, the processing unit 2204 can execute various programs in response to the program code and can maintain multiple concurrently executed programs or processes. At any given time, some or all of the program code to be executed may reside in (one or more) processors 2204 and / or in the storage subsystem 2218. Through appropriate programming, (one or more) processors 2204 can provide the various functions described above. The computer system 2200 can additionally include a processing acceleration unit 2206, which can include a digital signal processor (DSP), a special processor, and the like.

[0265] I / O subsystem 2208 may include user interface input devices and user interface output devices. User interface input devices may include a keyboard, a pointing device such as a mouse or trackball, a touch pad or touch screen incorporated into a display, a scroll wheel, a click wheel, a dial, buttons, switches, a keyboard, an audio input device with a voice command recognition system, a microphone, and other types of input devices. User interface input devices may include, for example, motion sensing and / or gesture recognition devices such as Microsoft Motion sensors that enable users to control apps such as Microsoft 360 game controller. The user interface input device may also include an eye gesture recognition device, such as detecting eye activity from the user (e.g., a "wink" when taking a picture and / or making a menu selection) and translating eye gestures to the input device (e.g., Google ) in Google In addition, the user interface input device may include a method for enabling the user to communicate with the voice recognition system (e.g., Navigator) interactive voice recognition sensing device.

[0266] The user interface input device may also include, but is not limited to, a three-dimensional (3D) mouse, a joystick or pointing stick, a game panel and a drawing board, and audio / video equipment such as a speaker, a digital camera, a digital video camera, a portable media player, a webcam, an image scanner, a fingerprint scanner, a barcode reader 3D scanner, a 3D printer, a laser rangefinder, and a sight tracking device. In addition, the user interface input device may include, for example, a medical imaging input device such as a computer tomography, a magnetic resonance imaging, a positron emission tomography, a medical ultrasound device. The user interface input device may also include, for example, an audio input device such as a MIDI keyboard, a digital musical instrument, etc.

[0267] The user interface output devices may include a display subsystem, indicator lights, or non-visual displays such as audio output devices, etc. The display subsystem may be a cathode ray tube (CRT), a flat panel device such as using a liquid crystal display (LCD) or plasma display, a projection device, a touch screen, etc. In general, the use of the term "output device" is intended to include all possible types of devices and mechanisms for outputting information from the computer system 2200 to a user or other computers. For example, the user interface output devices may include, but are not limited to, various display devices that visually convey text, graphics, and audio / video information, such as monitors, printers, speakers, headphones, car navigation systems, plotters, voice output devices, and modems.

[0268] Computer system 2200 may include a storage subsystem 2218 containing software elements, shown as currently located in system memory 2210. System memory 2210 may store program instructions that may be loaded and executed on processing unit 2204, as well as data generated during execution of these programs.

[0269] Depending on the configuration and type of computer system 2200, system memory 2210 may be volatile (such as random access memory (RAM)) and / or non-volatile (such as read-only memory (ROM), flash memory, etc.). RAM typically contains data and / or program modules that are immediately accessible to and / or currently being operated and executed by processing unit 2204. In some implementations, system memory 2210 may include multiple different types of memory, such as static random access memory (SRAM) or dynamic random access memory (DRAM). In some implementations, a basic input / output system (BIOS), such as one that contains basic routines that help transfer information between elements of computer system 2200 during startup, may typically be stored in ROM. By way of example, but not limitation, system memory 2210 also illustrates application programs 2212 that may include client applications, web browsers, middle-tier applications, relational database management systems (RDBMS), etc., program data 2214, and an operating system 2216. By way of example, operating system 2216 may include various versions of Microsoft Apple and / or Linux operating systems, various commercially available or UNIX-like operating systems (including but not limited to various GNU / Linux operating systems, Google operating system, etc.) and / or such as iOS, Phone, OS, 17OS and OS operating system's mobile operating system.

[0270] The storage subsystem 2218 may also provide a tangible computer-readable storage medium for storing basic programming and data structures that provide the functionality of some embodiments. Software (programs, code modules, instructions) that provide the above-described functionality when executed by the processor may be stored in the storage subsystem 2218. These software modules or instructions may be executed by the processing unit 2204. The storage subsystem 2218 may also provide a repository for storing data used in accordance with the present disclosure.

[0271] The storage subsystem 2200 may also include a computer-readable storage medium reader 2220 that may be further connected to a computer-readable storage medium 2222. Together with the system memory 2210 and, optionally, in conjunction therewith, the computer-readable storage medium 2222 may comprehensively represent remote, local, fixed, and / or removable storage devices and storage media for temporarily and / or more permanently containing, storing, transmitting, and retrieving computer-readable information.

[0272] The computer-readable storage medium 2222 containing the code or portions of the code may also include any suitable media known or used in the art, including storage media and communication media, such as, but not limited to, volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage and / or transmission of information. This may include tangible computer-readable storage media, such as RAM, ROM, electrically erasable programmable ROM (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage, cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or other tangible computer-readable media. This may also include non-tangible computer-readable media, such as data signals, data transmissions, or any other media that can be used to transmit the desired information and can be accessed by the computing system 2200.

[0273] By way of example, computer-readable storage media 2222 may include a hard drive that reads from or writes to non-removable nonvolatile magnetic media, a magnetic disk drive that reads from or writes to removable nonvolatile magnetic disks, and a magnetic disk drive that reads from or writes to removable nonvolatile optical disks (such as CD ROMs, DVDs, and optical drives). Computer readable storage media 2222 may include, but are not limited to, The computer readable storage medium 2222 may include a solid-state drive (SSD) based on non-volatile memory (such as a flash memory-based SSD, an enterprise flash drive, a solid-state ROM, etc.), a volatile memory-based SSD (such as a solid-state RAM, a dynamic RAM, a static RAM), a DRAM-based SSD, a magnetoresistive RAM (MRAM) SSD, and a hybrid SSD using a combination of DRAM and flash memory-based SSDs. The disk drive and its associated computer-readable media may provide non-volatile storage of computer-readable instructions, data structures, program modules, and other data for the computer system 2200.

[0274] The communication subsystem 2224 provides an interface to other computer systems and networks. The communication subsystem 2224 is used as an interface for receiving data from other systems and sending data from the computer system 2200 to other systems. For example, the communication subsystem 2224 can enable the computer system 2200 to be connected to one or more devices via the Internet. In some embodiments, the communication subsystem 2224 may include a radio frequency (RF) transceiver component for accessing a wireless voice and / or data network (e.g., using cellular phone technology, advanced data network technology such as 3G, 4G or EDGE (Enhanced Data Rate for Global Evolution), Wi-Fi (IEEE 802.11 series standards), or other mobile communication technologies, or any combination thereof), a global positioning system (GPS) receiver component and / or other components. In some embodiments, as an addition or replacement of a wireless interface, the communication subsystem 2224 may provide a wired network connection (e.g., Ethernet).

[0275] In some embodiments, the communication subsystem 2224 may also receive input communications in the form of structured and / or unstructured data feeds 2226 , event streams 2228 , event updates 2230 , and the like on behalf of one or more users who may use the computer system 2200 .

[0276] For example, the communication subsystem 2224 may be configured to receive data feeds 2226 in real time from users of social networks and / or other communication services, such as feed, Updates, web feeds such as Rich Site Summary (RSS) feeds, and / or real-time updates from one or more third-party information sources.

[0277] Additionally, the communication subsystem 2224 may also be configured to receive data in the form of a continuous data stream, which may include event streams 2228 and / or event updates 2230, which may be continuous or unbounded in nature, real-time events without explicit termination. Examples of applications that generate continuous data may include, for example, sensor data applications, financial quote machines, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automotive traffic monitoring, and the like.

[0278] The communication subsystem 2224 can also be configured to output structured and / or unstructured data feeds 2226, event streams 2228, event updates 2230, etc. to one or more databases, which can communicate with one or more streaming data source computers coupled to the computer system 2200.

[0279] Computer system 2200 may be one of various types, including a handheld portable device (e.g., Cellular phone, computing tablets, PDAs), wearable devices (e.g. Glass head mounted display), PC, workstation, mainframe, kiosk, server rack, or any other data processing system.

[0280] Due to the ever-changing nature of computers and networks, the description of the computer system 2200 depicted in the figure is intended only as a specific example. Many other configurations with more or fewer components than the system depicted in the figure are possible. For example, customized hardware may also be used and / or specific elements may be implemented with hardware, firmware, software (including applets), or a combination thereof. In addition, connections to other computing devices such as network input / output devices may also be employed. Based on the disclosure and teachings provided herein, one of ordinary skill in the art will recognize other ways and / or methods of implementing various embodiments.

[0281] Although specific embodiments of the present disclosure have been described, various modifications, changes, alternative constructions and equivalent forms are also included in the scope of the present disclosure. The embodiments of the present disclosure are not limited to operating in certain specific data processing environments, but can be freely operated in multiple data processing environments. In addition, although the embodiments of the present disclosure have been described using a specific series of transactions and steps, it should be clear to those skilled in the art that the scope of the present disclosure is not limited to the described series of transactions and steps. The various features and aspects of the above-mentioned embodiments can be used alone or in combination.

[0282] In addition, although the embodiments of the present disclosure have been described using a specific combination of hardware and software, it should be appreciated that other combinations of hardware and software are also within the scope of the present disclosure. The embodiments of the present disclosure can be implemented using only hardware, or only software, or using a combination thereof. The various processes described herein can be implemented in any combination on the same processor or on different processors. Thus, in the case where a component or module is described as being configured to perform certain operations, such configurations can be accomplished by, for example, designing an electronic circuit to perform the operation, by programming a programmable electronic circuit (such as a microprocessor) to perform the operation, or any combination thereof. Processing can communicate using a variety of techniques, including but not limited to conventional techniques for inter-processing communication, and different processing pairs can use different techniques, or the same pair of processing can use different techniques at different times.

[0283] The description and drawings are therefore to be regarded as illustrative rather than restrictive. However, additions, subtractions, deletions and other modifications and changes may be made without departing from the broader spirit and scope set forth in the claims. Therefore, although specific disclosed embodiments have been described, these are not intended to be limiting. Various modifications and equivalent forms are within the scope of the following claims.

[0284] The use of the terms "a" and "an" and "the" and similar references in the context of describing the disclosed embodiments (especially in the context of the following claims) is to be interpreted as covering the singular and the plural, unless otherwise indicated herein or clearly contradicted by the context. Unless otherwise stated, the terms "include", "have", "include (including)" and "include (containing)" are to be interpreted as open terms (i.e., meaning "including but not limited to"). The term "connected" should be interpreted as being partially or completely included in, attached to or connected together, even if there is something in the middle. Unless otherwise indicated herein, the enumeration of the range of values ​​herein is intended only to be used as a shorthand method of individually referencing each individual value falling within the range, and each individual value is incorporated into the specification as if it were individually listed herein. Unless otherwise indicated herein or clearly contradicted by the context, all methods described herein can be performed in any suitable order. Unless otherwise stated, the use of any and all examples or exemplary language (e.g., "such as") provided herein is intended only to better illustrate the embodiments of the present disclosure and is not intended to limit the scope of the present disclosure. No language in the specification should be construed as indicating any non-claimed element as essential to the practice of the disclosure.

[0285] Disjunctive language, such as the phrase "at least one of X, Y, or Z," is intended to be understood within the context generally used to indicate that an item, term, or the like, can be X, Y, or Z, or any combination thereof (e.g., X, Y, and / or Z), unless expressly stated otherwise. Thus, such disjunctive language is generally not intended to, and should not, imply that certain embodiments require that at least one of X, at least one of Y, or at least one of Z each be present.

[0286] Preferred embodiments of the present disclosure are described herein, including the best mode known to the inventor for implementing the present disclosure. Variations of those preferred embodiments will become clear to those of ordinary skill in the art after reading the above description. The inventor expects that the technician can appropriately adopt such variations and the inventor expects to practice the present disclosure in a manner different from that specifically described herein. Thus, the present disclosure includes all modifications and equivalent forms of the subject matter recorded in the attached claims where permitted by applicable law. In addition, unless otherwise indicated herein or otherwise clearly contradictory to the context, the present disclosure includes any combination of the above-mentioned elements in all possible variations thereof.

[0287] All references cited herein, including publications, patent applications and patents, are incorporated herein by reference to the same extent, as if each reference is individually and specifically indicated to be incorporated by reference and set forth in full herein. In the aforementioned specification, various aspects of the present disclosure are described with reference to their specific embodiments, but it will be appreciated by those skilled in the art that the present disclosure is not limited thereto. Each feature and aspect disclosed above can be used individually or in combination. In addition, without departing from the broader spirit and scope of this specification, the embodiments can be used in any number of environments and applications other than those described herein. Thus, this specification and the accompanying drawings should be considered illustrative rather than restrictive.

Claims

1. A method comprising: Provides multiple switch blocks; providing a compute fabric block communicatively coupled to the plurality of switch blocks, the compute fabric block comprising: (i) a set of one or more racks, each rack in the set of one or more racks comprising one or more servers configured to execute one or more workloads of a customer, and (ii) a first plurality of switches organized into a first plurality of levels, the first plurality of switches communicatively coupling the set of one or more racks to the plurality of switch blocks; and A network fabric block is provided that is communicatively coupled to the plurality of switch blocks, the network fabric block comprising: (i) one or more edge devices, the one or more edge devices comprising a first edge device that provides connectivity to a first external resource, the first edge device enabling a workload executed by a server included in a rack in the set of one or more racks to access the first external resource, and (ii) a second plurality of switches organized into a second plurality of levels, the second plurality of switches communicatively coupling the one or more edge devices to the plurality of switch blocks.

2. The method of claim 1, wherein the first external resource is a public communication network, and wherein the first edge device is a gateway providing connectivity to the public communication network.

3. The method of claim 1 or 2, wherein the first plurality of levels associated with the first plurality of switches in the computing fabric block further comprises: (i) a first tier-1 level switch, and (ii) a first tier-2 level switch, wherein the first tier-1 level switch is communicatively coupled to the set of one or more chassis at a first end and is communicatively coupled to the first tier-2 level switch at a second end, and wherein the first tier-2 level switch connects the first tier-1 level switch to the plurality of switch blocks.

4. The method of claim 3, wherein the first hierarchy 1 level switches in the computing fabric block include eight switches and the first hierarchy 2 level switches in the computing fabric block include four switches.

5. The method of claim 3 or 4, wherein each of the first hierarchy 1 level switches in the computing fabric block is connected to each of the first hierarchy 2 level switches in the computing fabric block.

6. A method as claimed in claim 4 or 5, wherein each block of the plurality of switch blocks includes a predetermined number of switches, and wherein each switch of the first hierarchy 2 level switches in the computing structure block is connected to at least one switch in each block of the plurality of switch blocks.

7. The method of any of the preceding claims, wherein the second plurality of levels associated with the second plurality of switches in the network fabric block further comprises: (i) a second layer level 1 switch, and (ii) a second layer level 2 switch, wherein a first subset of switches included in the second hierarchy 1 level switches are communicatively coupled at a first end to the one or more edge devices, and a second subset of switches included in the second hierarchy 1 level switches are communicatively coupled at a first end to the plurality of switch blocks, and wherein A first subset and a second subset of switches included in a second hierarchy 1 level switch are coupled at a second end to a second hierarchy 2 level switch included in the network fabric block.

8. The method of claim 7, wherein the second hierarchy 1 level switches in the network fabric block include eight switches and the second hierarchy 2 level switches in the network fabric block include four switches.

9. The method of claim 6, 7 or 8, wherein the predetermined number of switches included in each of the plurality of switch blocks is four.

10. The method of any of the preceding claims, wherein the one or more edge devices include a gateway, a backbone edge device, a metro edge device, and a route reflector.

11. The method of any one of claims 7 to 10, wherein a workload executed by a server included in a rack of the computing fabric block accesses a first external resource by establishing a connection to a first switch of the plurality of switch blocks, the connection: (i) routing from the first switch to a second switch included in a second subset of switches in a second hierarchy 1 level switch, (ii) routing from the second switch to a third switch included in the second hierarchy 2 level switches, (iii) routing from the third switch to a fourth switch included in the first subset of switches in the second hierarchy 1 level switches, and (iv) Routing from the fourth switch to the gateway.

12. A non-transitory computer-readable medium storing specific computer-executable instructions that, when executed by a processor, cause a computer system to perform operations including: Provides multiple switch blocks; providing a compute fabric block communicatively coupled to the plurality of switch blocks, the compute fabric block comprising: (i) a set of one or more racks, each rack in the set of one or more racks comprising one or more servers configured to execute one or more workloads of a customer, and (ii) a first plurality of switches organized into a first plurality of levels, the first plurality of switches communicatively coupling the set of one or more racks to the plurality of switch blocks; and A network fabric block is provided that is communicatively coupled to the plurality of switch blocks, the network fabric block comprising: (i) one or more edge devices, the one or more edge devices comprising a first edge device that provides connectivity to a first external resource, the first edge device enabling a workload executed by a server included in a rack in the set of one or more racks to access the first external resource, and (ii) a second plurality of switches organized into a second plurality of levels, the second plurality of switches communicatively coupling the one or more edge devices to the plurality of switch blocks.

13. The non-transitory computer-readable medium of claim 12, wherein the first external resource is a public communication network, and wherein the first edge device is a gateway that provides connectivity to the public communication network.

14. The non-transitory computer readable medium of claim 12 or 13, wherein the first plurality of levels associated with the first plurality of switches in a compute fabric block further comprises: (i) a first tier-1 level switch, and (ii) a first tier-2 level switch, wherein the first tier-1 level switch is communicatively coupled to the set of one or more chassis at a first end and is communicatively coupled to the first tier-2 level switch at a second end, and wherein the first tier-2 level switch connects the first tier-1 level switch to the plurality of switch blocks.

15. The non-transitory computer readable medium of claim 12, 13, or 14, wherein the first hierarchy 1 level switches in the compute fabric block include eight switches and the first hierarchy 2 level switches in the compute fabric block include four switches.

16. The non-transitory computer readable medium of claim 14 or 15, wherein each of the first hierarchy 1 level switches in the computing fabric block is connected to each of the first hierarchy 2 level switches in the computing fabric block.

17. The non-transitory computer-readable medium of claim 16, wherein each of the plurality of switch blocks comprises a predetermined number of switches, and wherein each of the first hierarchy-2 level switches in the compute fabric block is connected to at least one switch in each of the plurality of switch blocks.

18. The non-transitory computer readable medium of any one of claims 12 to 17, wherein the second plurality of levels associated with the second plurality of switches in the network fabric block further comprises: (i) a second layer level 1 switch, and (ii) a second layer level 2 switch, wherein a first subset of switches included in the second hierarchy 1 level switches are communicatively coupled at a first end to the one or more edge devices, and a second subset of switches included in the second hierarchy 1 level switches are communicatively coupled at a first end to the plurality of switch blocks, and wherein A first subset and a second subset of switches included in a second hierarchy 1 level switch are coupled at a second end to a second hierarchy 2 level switch included in the network fabric block.

19. The non-transitory computer readable medium of claim 18, wherein a workload executed by a server included in a rack of the computing fabric block accesses a first external resource by establishing a connection to a first switch of the plurality of switch blocks, the connection: (i) routing from the first switch to a second switch included in a second subset of switches in a second hierarchy 1 level switch, (ii) routing from the second switch to a third switch included in the second hierarchy 2 level switches, (iii) routing from the third switch to a fourth switch included in the first subset of switches in the second hierarchy 1 level switches, and (iv) Routing from the fourth switch to the gateway.

20. A computing device comprising: processor; as well as a memory comprising instructions that, when executed by a processor, cause a computing device to at least: Provides multiple switch blocks; providing a compute fabric block communicatively coupled to the plurality of switch blocks, the compute fabric block comprising: (i) a set of one or more racks, each rack in the set of one or more racks comprising one or more servers configured to execute one or more workloads of a customer, and (ii) a first plurality of switches organized into a first plurality of levels, the first plurality of switches communicatively coupling the set of one or more racks to the plurality of switch blocks; and A network fabric block is provided that is communicatively coupled to the plurality of switch blocks, the network fabric block comprising: (i) one or more edge devices, the one or more edge devices comprising a first edge device that provides connectivity to a first external resource, the first edge device enabling a workload executed by a server included in a rack in the set of one or more racks to access the first external resource, and (ii) a second plurality of switches organized into a second plurality of levels, the second plurality of switches communicatively coupling the one or more edge devices to the plurality of switch blocks.

Images