Layer 2 networking information in a virtualized cloud environment

Abstract

Techniques are described for communication in a customer's L2 virtual network. In one example, the L2 virtual network includes a plurality of L2 compute instances hosted on a set of host machines and a plurality of L2 virtual network interfaces and L2 virtual switches hosted on a set of network virtualization devices. The L2 virtual network interfaces emulate L2 ports of the L2 virtual network. Information associated with the L2 virtual switches is collected and provided to the customer.

Description

Technical Field

[0001] Reference to Related Applications This international patent application claims priority to U.S. Patent Application No. 17 / 494,720, filed on October 5, 2021, titled "LAYER-2 NETWORKING INFORMATION IN A VIRTUALIZED CLOUD ENVIRONMENT", which claims the benefit of U.S. Provisional Application No. 63 / 132,377, filed on December 30, 2020, titled "LAYER-2 NETWORKING IN A VIRTUALIZED CLOUD ENVIRONMENT", the content of which is hereby incorporated by reference in its entirety for all purposes.

Background Art

[0002] Background Cloud computing provides on-demand availability of computing resources. Cloud computing can be based on data centers that are available to users via the Internet. Cloud computing can provide infrastructure as a service (IaaS). Virtual networks may be created for use by users. However, these virtual networks have limitations that restrict their functionality and value. Therefore, further improvements are desired.

Summary of the Invention

Means for Solving the Problems

[0003] Summary The present disclosure relates to a virtualized cloud environment. Techniques for providing layer 2 networking functions in a virtualized cloud environment are described. The layer 2 functions are provided in addition to and in relation to the layer 3 networking functions provided by the virtualized cloud environment.

[0004] Some embodiments of this disclosure relate to providing customers with Layer 2 virtual local area networks (VLANs) within a private network, such as a customer's virtual cloud network (VCN). In a Layer 2 VLAN, different compute instances are connected. The customer is given the perception of an emulated single switch connecting the compute instances. In fact, this emulated switch is implemented as an infinitely scalable distributed switch containing a set of local switches. More specifically, each compute instance runs on a host machine connected to a network virtualization device (NVD). For each compute instance on a host connected to the NVD, the NVD hosts a Layer 2 virtual network interface card (VNIC) and a local switch associated with the compute instance. A Layer 2 VNIC represents a port of the compute instance on the Layer 2 VLAN. A local switch connects the VNIC to other VNICs (e.g., other ports) associated with other compute instances on the Layer 2 VLAN. For example, various Layer 2 network services are supported, including providing information about Layer 2 VLANs, which may include Media Access Control (MAC) address forwarding tables and / or Layer 2 switch statistics.

[0005] This section describes various embodiments, including methods, systems, and non-temporary computer-readable storage media for storing programs, code, or instructions executable by one or more processors. [Brief explanation of the drawing]

[0006] [Figure 1] This is a high-level diagram of a distributed environment showing a virtual or overlay cloud network hosted by a cloud service provider infrastructure, according to a specific embodiment. [Figure 2] This is an architectural schematic diagram showing the physical elements of the physical network within the CSPI according to a specific embodiment. [Figure 3]This figure shows an exemplary configuration of a CSPI in which a host machine is connected to multiple network virtualization devices (NVDs) according to a particular embodiment. [Figure 4] This diagram shows the connection between a host machine and an NVD that provides I / O virtualization to support multi-tenancy functionality, according to a specific embodiment. [Figure 5] This is a schematic block diagram showing the physical network provided by CSPI according to a specific embodiment. [Figure 6] This is a schematic diagram of a computing network according to one embodiment. [Figure 7] This is a schematic diagram of the logic and hardware of a VLAN according to one embodiment. [Figure 8] This is a schematic logic diagram of multiple connected L2 VLANs according to one embodiment. [Figure 9] This is a logical schematic diagram of multiple connected L2 VLANs and subnet 900 according to one embodiment. [Figure 10] This is a schematic diagram of VLAN communication and VLAN learning according to one embodiment. [Figure 11] This is a schematic diagram of a VLAN according to one embodiment. [Figure 12] This flowchart shows process 1200 for communication within a VLAN according to one embodiment. [Figure 13] This figure shows an exemplary environment suitable for defining the configuration of an L2 virtual network and providing related L2 information according to one embodiment. [Figure 14] This figure shows exemplary L2 information in an L2 virtual network according to one embodiment. [Figure 15] This figure shows an example L2 forwarding table in a Layer 2 virtual network according to one embodiment. [Figure 16] This figure shows exemplary L2 metrics and statistics in a Layer 2 virtual network according to one embodiment. [Figure 17]This is a flowchart showing a process for providing L2 information according to one embodiment. [Figure 18] This is a flowchart showing the process for generating an L2 transfer table according to one embodiment. [Figure 19] This is a flowchart showing the process for generating L2 statistics according to one embodiment. [Figure 20] This is a block diagram showing one pattern for realizing a cloud infrastructure as a service system according to at least one embodiment. [Figure 21] This block diagram shows another pattern for realizing a cloud infrastructure as a service system, according to at least one embodiment. [Figure 22] This block diagram shows another pattern for realizing a cloud infrastructure as a service system, according to at least one embodiment. [Figure 23] This block diagram shows another pattern for realizing a cloud infrastructure as a service system, according to at least one embodiment. [Figure 24] A block diagram showing an exemplary computer system according to at least one embodiment. [Modes for carrying out the invention]

[0007] Detailed explanation In the following description, certain details are included for illustrative purposes to facilitate a full understanding of the particular embodiment. However, it will be apparent that various embodiments may be carried out without these specific details. The figures and descriptions are not intended to be limiting. The term “exemplary” is used here to mean “provided as an example, case, or illustration.” Any embodiment or design described herein as “exemplary” should not necessarily be construed as being preferable or advantageous over other embodiments or designs.

[0008] A. Exemplary virtual networking architecture The term "cloud service" generally refers to services that a cloud service provider (CSP) makes available to users or customers on demand (e.g., via a subscription model) using systems and infrastructure (cloud infrastructure). Usually, the servers and systems that make up the CSP's infrastructure are separate from the customer's own on-premises servers and systems. Thus, customers can utilize cloud services provided by a CSP without separately purchasing the hardware and software resources for the services. Cloud services are designed to provide customers who subscribe to them with easy and scalable access to applications and computing resources without the customer having to invest in the procurement of the infrastructure used to provide the services.

[0009] There are several cloud service providers that offer various types of cloud services. Cloud services include various different types or models such as SaaS (Software-as-a-Service), PaaS (Platform-as-a-Service), and IaaS (Infrastructure-as-a-Service).

[0010] Customers can subscribe to one or more cloud services provided by a CSP. The customer can be any entity such as an individual, an organization, or a business. When a customer subscribes or registers for the services provided by a CSP, a tenant or account is created for that customer. Thereafter, the customer can access one or more subscribed cloud resources associated with the account via this account.

[0011] As mentioned above, IaaS (Infrastructure as a Service) is a specific type of cloud computing service. In the IaaS model, the CSP provides the infrastructure (called Cloud Service Provider Infrastructure or CSPI) that customers can use to build their own customizable networks and deploy their customer resources. Thus, the customer's resources and network are hosted in a distributed environment by the infrastructure provided by the CSP. This differs from traditional computing where the customer's infrastructure hosts the customer's resources and network.

[0012] CSPI may include interconnected high-performance computing resources, including various host machines, memory resources, and network resources, forming a physical network also known as an underlay network or base network. CSPI resources may be distributed across one or more data centers geographically distributed across one or more geographical regions. Virtualization software can run on these physical resources to provide a virtualized distributed environment. Virtualization creates an overlay network (also known as a software-based network, software-defined network, or virtual network) on top of the physical network. The CSPI physical network provides the foundation for creating one or more overlay or virtual networks on top of the physical network. A virtual network or overlay network may include one or more virtual cloud networks (VCNs). Virtual networks are implemented using software virtualization technologies (e.g., hypervisors, functions performed by network virtualization devices (NVDs) (e.g., smart NICs), top-of-rack (TOR) switches, smart TORs implementing one or more functions performed by NVDs, and other mechanisms) to create a network abstraction layer that can run on top of the physical network. Virtual networks can take various forms, such as peer-to-peer networks and IP networks. A virtual network is typically either a Layer 3 IP network or a Layer 2 VLAN. Such virtual or overlay networks are often referred to as virtual Layer 3 networks or overlay Layer 3 networks. Examples of protocols developed for virtual networks include IP-in-IP (or GRE (Generic Routing Encapsulation)), virtual extensible LAN (VXLAN - IETF RFC7348), virtual private networks (VPNs) (e.g., MPLS Layer 3 virtual private network (RFC4364)), VMware NSX, and GENEVE (Generic Network Virtualization Encapsulation).

[0013] In the case of IaaS, the infrastructure provided by the CSP (CSPI) may be configured to deliver virtualized computing resources over a public network (e.g., the internet). In the IaaS model, the cloud computing service provider can host infrastructure elements (e.g., servers, storage, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer)). In some cases, the IaaS provider can provide various services associated with these infrastructure elements (e.g., billing, monitoring, logging, security, load balancing, and clustering). Because these services are policy-driven, IaaS users can maintain application availability and performance by implementing policies to drive load balancing. The CSPI provides infrastructure and a set of complementary cloud services. This allows customers to build and run a wide range of applications and services in a highly available hosted distributed environment. The CSPI provides high-performance computing resources and capabilities, as well as storage capacity, on a flexible virtual network that can be securely accessed from various network locations, such as the customer's on-premises network. When a customer subscribes to or registers for an IaaS service provided by a CSP, the tenancy created for that customer becomes a securely isolated partition from the CSP, allowing the customer to create, organize, and manage cloud resources.

[0014] Customers can build their own virtual networks using the computing, memory, and networking resources provided by CSPI. They can deploy one or more customer resources or workloads, such as compute instances, on these virtual networks. For example, a customer can build one or more customizable private virtual networks called Virtual Cloud Networks (VCNs) using the resources provided by CSPI. On a customer VCN, a customer can deploy one or more customer resources, such as compute instances. Compute instances may be virtual machines, bare-metal instances, etc. Thus, CSPI provides the infrastructure and a set of complementary cloud services that enable customers to build and run a variety of applications and services in a highly available virtual host environment. Customers do not manage or control the underlying physical resources provided by CSPI, but they control the operating system, memory, and deployed applications, and, in some cases, have limited control over certain networking components (e.g., firewalls).

[0015] A CSP can provide a console that enables customers and network administrators to configure, access, and manage resources deployed to the cloud using CSPI resources. In certain embodiments, the console provides a web-based user interface that can be used to utilize and manage CSPI. In some embodiments, the console is a web-based application provided by the CSP.

[0016] CSPI can support single-tenancy or multi-tenancy architectures. In a single-tenancy architecture, software (e.g., applications, databases) or hardware elements (e.g., host machines or servers) serve a single customer or tenant. In a multi-tenancy architecture, software or hardware elements serve multiple customers or tenants. Therefore, in a multi-tenancy architecture, CSPI resources are shared among multiple customers or tenants. In a multi-tenancy environment, CSPI implements precautions and safeguards to isolate each tenant's data and prevent it from being visible to other tenants.

[0017] In a physical network, a network endpoint (endpoint) refers to a computing device or system that is connected to the physical network and communicates bidirectionally with the connected network. A network endpoint in a physical network may be connected to a local area network (LAN), a wide area network (WAN), or other types of physical networks. Examples of traditional endpoints in a physical network include modems, hubs, bridges, switches, routers, and other networking devices, as well as physical computers (or host machines). Each physical device in a physical network has a fixed network address that can be used to communicate with that device. This fixed network address may be a Layer 2 address (e.g., a MAC address), a fixed Layer 3 address (e.g., an IP address), etc. In a virtualized environment or virtual network, endpoints can include various virtual endpoints, such as virtual machines hosted by elements of the physical network (e.g., hosted by a physical host machine). These endpoints in a virtual network are addressed by overlay addresses, such as overlay Layer 2 addresses (e.g., overlay MAC addresses) and overlay Layer 3 addresses (e.g., overlay IP addresses). Network overlays provide flexibility by allowing network administrators to move overlay addresses associated with network endpoints using software management (e.g., through software implementing the control plane of the virtual network). Therefore, unlike physical networks, in virtual networks, network management software can be used to move overlay addresses (e.g., overlay IP addresses) from one endpoint to another. Because virtual networks are built on top of physical networks, both the virtual network and the underlying physical network are involved in communication between elements of the virtual network.To facilitate such communication, each element of the CSPI is configured to learn and store mappings that map the overlay address of the virtual network to the actual physical address of the underlying network, or vice versa. These mappings are used to facilitate communication. To facilitate routing within the virtual network, customer traffic is encapsulated.

[0018] Therefore, physical addresses (e.g., physical IP addresses) are associated with elements of a physical network, while overlay addresses (e.g., overlay IP addresses) are associated with entities in a virtual network. Both physical and overlay IP addresses are real IP addresses. They are distinct from virtual IP addresses, which map to multiple real IP addresses. Virtual IP addresses provide a one-to-many mapping between virtual IP addresses and multiple real IP addresses.

[0019] The cloud infrastructure, or CSPI, is physically hosted in one or more data centers in one or more regions around the world. The CSPI may include elements of a physical network or underlying network and virtualization elements of a virtual network built on top of the physical network elements (e.g., virtual networks, compute instances, virtual machines). In certain embodiments, the CSPI is organized and hosted in realms, regions, and available domains. A region is typically a local geographical area containing one or more data centers. Regions are generally independent of each other and may be separated by vast distances, for example, across countries or continents. For example, one region may be in Australia, another in Japan, and yet another in India. CSPI resources are divided among these regions such that each region has an independent subset of CSPI resources. Each region can provide a set of core infrastructure services and resources, such as computing resources (e.g., bare metal servers, virtual machines, containers, and related infrastructure), storage resources (e.g., block volume storage, file storage, object storage, archive storage), networking resources (e.g., virtual cloud networks (VCNs), load balancing resources, connectivity to on-premises networks), database resources, edge networking resources (e.g., DNS), and access management and monitoring resources. Each region generally has multiple routes for connecting to other regions within the realm.

[0020] Generally, applications are deployed in the region where they are most frequently used (i.e., on infrastructure relevant to that region) because using nearby resources is faster than using distant resources. Applications may also be deployed in different regions for various reasons, such as redundancy to mitigate the risks of large-scale weather systems or region-wide events like earthquakes, or redundancy to meet various requirements for legal jurisdictions, tax domains, and other business or social standards.

[0021] Data centers within a region may be further organized and subdivided into availability domains (ADs). An availability domain may correspond to one or more data centers located in a region. A region may consist of one or more availability domains. In such a distributed environment, CSPI resources are region-specific, such as virtual cloud networks (VCNs), or availability domain-specific, such as compute instances.

[0022] ADs within a single region are configured to be fault-tolerant, isolated from one another, and configured to be highly unlikely to fail simultaneously. This is achieved by configuring ADs so that a failure in one AD within a region has little impact on the availability of other ADs within the same region, by not sharing critical infrastructure resources such as networking, physical cabling, cabling routes, and cabling entry points. Connecting ADs within the same region with a low-latency, high-bandwidth network provides highly available connectivity to other networks (e.g., the internet, customer on-premises networks), and a replication system for both high availability and disaster recovery can be built across multiple ADs. CloudSense utilizes multiple ADs to ensure high availability and protect against resource failures. As the infrastructure provided by the IaaS provider grows, more regions and ADs may be added along with additional capacity. Traffic between available domains is typically encrypted.

[0023] In certain embodiments, regions are grouped into realms. A realm is a logical collection of regions. Realms are isolated from each other and do not share any data. Regions within the same realm can communicate with each other, but regions in different realms cannot. A CSP customer's tenancy or account may reside in a single realm and span one or more regions belonging to that single realm. Typically, when a customer subscribes to an IaaS service, their tenancy or account is created in a customer-designated region within a realm (referred to as the "home" region). The customer can extend their tenancy to one or more other regions within a realm. A customer cannot access regions that do not reside in the realm where their tenancy resides.

[0024] An IaaS provider can offer multiple realms, each corresponding to a specific set of customers or users. For example, a commercial realm may be offered for commercial customers. Another example is that a realm may be offered for a specific country or for customers in that country. Yet another example is that a government realm may be offered for a government, for example. For example, a government realm may be created for a specific government and may have a higher security level than a commercial realm. For example, Oracle Cloud Infrastructure (OCI) currently offers a realm for the commercial domain and two realms for the government cloud domain (e.g., FedRAMP accreditation and IL5 accreditation).

[0025] In certain embodiments, an Active Directory (AD) can be subdivided into one or more fault domains. A fault domain is a grouping of infrastructure resources within the AD to provide anti-affinity. Fault domains can distribute compute instances so that they are not located on the same physical hardware within a single AD. This is known as anti-affinity. A fault domain refers to a collection of hardware elements (computers, switches, etc.) that share a single point of failure. The compute pool is logically divided into fault domains. Therefore, a hardware failure or compute hardware maintenance event affecting one fault domain does not affect instances in other fault domains. Depending on the embodiment, the number of fault domains in each AD may vary. For example, in certain embodiments, each AD may contain three fault domains. Fault domains function as logical data centers within the AD.

[0026] When a customer subscribes to an IaaS service, resources from CSPI are provisioned to the customer and associated with the customer's tenancy. The customer can use these provisioned resources to build private networks and deploy resources on these networks. Customer networks hosted on the cloud by CSPI are called Virtual Cloud Networks (VCNs). A customer can configure one or more Virtual Cloud Networks (VCNs) using the CSPI resources allocated to them. A VCN is a virtual or software-defined private network. Customer resources deployed in a customer's VCN can include compute instances (e.g., virtual machines, bare metal instances) and other resources. These compute instances may represent various customer workloads such as applications, load balancers, and databases. Compute instances deployed on a VCN can communicate with publicly accessible endpoints (public endpoints) over public networks such as the internet, with other instances within the same VCN or other VCNs (e.g., other VCNs of the customer, or VCNs not belonging to the customer), with the customer's on-premises data center or network, with Sendi endpoints, and with other types of endpoints.

[0027] A CSP can provide a variety of services using a CSPI. In some cases, a CSPI customer can act like a service provider themselves and provide services using CSPI resources. A service provider can expose service endpoints characterized by identifying information (e.g., IP address, DNS name, and port). A customer's resources (e.g., compute instances) can consume a particular service by accessing the service endpoint of that particular service exposed by the service. These service endpoints are generally publicly accessible endpoints that users can access via public communication networks such as the internet using the public IP address associated with the endpoint. Publicly accessible network endpoints are sometimes called public endpoints.

[0028] In certain embodiments, a service provider may expose a service through an endpoint of the service (sometimes called a service endpoint). Customers of the service can access the service using this service endpoint. In certain embodiments, the service endpoint provided for a service may be accessible to multiple customers who wish to consume that service. In other implementations, a dedicated service endpoint may be provided to a customer. Thus, only that customer can access the service using that dedicated service endpoint.

[0029] In certain embodiments, a VCN, once created, is associated with a private overlay classless inter-domain routing (CIDR) address space, which is a private overlay IP address range (e.g., 10.0 / 16) assigned to that VCN. A VCN includes associated subnets, route tables, and gateways. While a VCN resides within a single region, it can extend to one or more or all available domains within a region. A gateway is a virtual interface configured for a VCN, enabling traffic communication between the VCN and one or more endpoints outside the VCN. By configuring one or more different types of gateways for a VCN, communication between different types of endpoints can be enabled.

[0030] A VCN may be subdivided into one or more subnets, such as one or more subnets. Thus, a subnet is a constituent unit or partition that can be created within a VCN. A VCN can have one or more subnets. Each subnet within a VCN does not overlap with other subnets within that VCN and is associated with a contiguous range of overlay IP addresses (e.g., 10.0.0.0 / 24 and 10.0.1.0 / 24) that represent an address space subset of the VCN's address space.

[0031] Each compute instance is associated with a virtual network interface card (VNIC). This allows each compute instance to join a subnet in a VCN. A VNIC is a logical representation of a physical network interface card (NIC). Generally, a VNIC is the interface between an entity (e.g., compute instance, service) and a virtual network. A VNIC resides in a subnet and has one or more associated IP addresses and associated security rules or policies. A VNIC is equivalent to a Layer 2 port on a switch. A VNIC connects a compute instance to a subnet within a VCN. The VNIC associated with a compute instance enables the compute instance to be part of a subnet in a VCN and allows the compute instance to communicate (e.g., send and receive packets) with endpoints on the same subnet as the compute instance, endpoints in different subnets within the VCN, or endpoints outside the VCN. Therefore, the VNIC associated with a compute instance determines how the compute instance connects to internal and external endpoints within the VCN. The VNIC for a compute instance is created and associated with that compute instance when the compute instance is created and added to a subnet within the VCN. If a subnet consists of a set of compute instances, it includes the VNICs corresponding to the set of compute instances, and each VNIC is connected to a compute instance within the set of computer instances.

[0032] Each compute instance is assigned a private overlay IP address via the VNIC associated with it. This private overlay IP address is assigned to the VNIC associated with the compute instance when the compute instance is created and is used to route the compute instance's traffic. All VNICs within a given subnet use the same route table, security lists, and DHCP options. As mentioned above, each subnet within a VCN does not overlap with other subnets within that VCN and is associated with a contiguous range of overlay IP addresses (e.g., 10.0.0.0 / 24 and 10.0.1.0 / 24) that represents a subset of the address space of that VCN. For a VNIC on a particular subnet of a VCN, the overlay IP address assigned to the VNIC is an address from the contiguous range of overlay IP addresses assigned to the subnet.

[0033] In certain embodiments, a compute instance may be assigned additional overlay IP addresses, such as one or more public IP addresses in the case of a public subnet, in addition to its private overlay IP address, as needed. These multiple addresses may be assigned to the same VNIC or to multiple VNICs associated with the compute instance. However, each instance has a primary VNIC that is created at instance launch and associated with the overlay private IP address assigned to the instance. This primary VNIC cannot be deleted. Additional VNICs, called secondary VNICs, can be added to an existing instance in the same available domain as the primary VNIC. All VNICs are in the same available domain as the instance. Secondary VNICs may be in the same VCN subnet as the primary VNIC, or they may be in the same VCN or different VCN subnets.

[0034] Compute instances can optionally be assigned a public IP address if they are located in a public subnet. When creating a subnet, you can specify that subnet as either a public or private subnet. A private subnet means that resources within that subnet (e.g., compute instances) and associated VNICs cannot have public overlay IP addresses. A public subnet means that resources within that subnet and associated VNICs can have public IP addresses. Customers can specify subnets that exist across a single available domain or multiple available domains within a region or realm.

[0035] As described above, a VCN may be subdivided into one or more subnets. In certain embodiments, a virtual router (referred to as a VCN VR or simply a VR) configured for the VCN enables communication between subnets within the VCN. For subnets within a VCN, the VR represents the logical gateway for that subnet, enabling communication between the subnet (i.e., compute instances on that subnet) and endpoints on other subnets within the VCN and other endpoints outside the VCN. A VCN VR is a logical entity configured to route traffic between VNICs within the VCN and virtual gateways (gateways) associated with the VCN. Gateways are described further below with reference to Figure 1. A VCN VR is a Layer 3 / IP layer concept. In one embodiment, there is one VCN VR for one VCN. This VCN VR potentially has an unlimited number of ports addressed by IP addresses, with one port for each subnet of the VCN. Thus, the VCN VR has a different IP address for each subnet of the VCN to which the VCN VR is connected. The VR is also connected to various gateways configured for the VCN. In certain embodiments, specific overlay IP addresses from a subnet's overlay IP address range are held on ports of the VCN VR for that subnet. For example, consider a VCN having two subnets, each having the associated address ranges 10.0 / 16 and 10.1 / 16. For the first subnet of the VCN having the address range 10.0 / 16, addresses from this range are held on ports of the VCN VR for that subnet. In some cases, the first IP address from this range may be held on the VCN VR. For example, for a subnet having the overlay IP address range 10.0 / 16, the IP address 10.0.0.1 may be held on ports of the VCN VR for that subnet. For a second subnet within the same VCN having the address range 10.1 / 16, the VCN VR may have ports for the second subnet having the IP address 10.1.0.1.A VCN VR has a different IP address for each subnet within the VCN.

[0036] In some other embodiments, each subnet within a VCN may have its own associated VR, which is addressable by the subnet using a reserved or default IP address associated with the VR. The reserved or default IP address may be, for example, a first IP address from a range of IP addresses associated with that subnet. A VNIC within a subnet can use this default or reserved IP address to communicate with the VR associated with the subnet (e.g., send and receive packets). In such embodiments, the VR is the incoming / outgoing point for that subnet. A VR associated with a subnet within a VCN can communicate with other VRs associated with other subnets within the VCN. A VR can also communicate with gateways associated with the VCN. The VR functionality of a subnet is performed on, or by, one or more NVDs that perform the VNIC functionality of the VNICs within the subnet.

[0037] Route tables, security rules, and DHCP options may be configured for the VCN. The route table is the VCN's virtual route table and contains rules for routing traffic from subnets within the VCN to destinations outside the VCN, via a gateway or specially configured instance. The VCN's route table can be customized to control packet forwarding / routing to and from the VCN. DHCP options refer to configuration information automatically provided to an instance when it is launched.

[0038] Security rules configured for a VCN represent the VCN's overlay firewall rules. Security rules can include inbound and outbound rules and can specify the types of traffic allowed to enter and exit instances within the VCN (e.g., based on protocol and port). Customers can choose whether certain rules are stateful or stateless. For example, a customer can allow incoming SSH traffic to a pair of instances from any location by configuring a stateful inbound rule with source CIDR 0.0.0.0 / 0 and destination TCP port 22. Security rules may be implemented using network security groups or security lists. A network security group consists of a set of security rules that apply only to resources within that group. A security list, on the other hand, contains rules that apply to all resources within a subnet that uses that security list. A VCN may include default security rules and default security lists. DHCP options configured for a VCN provide configuration information that is automatically provided when instances within the VCN start up.

[0039] In certain embodiments, VCN configuration information is determined and stored by the VCN control plane. VCN configuration information may include, for example, address ranges associated with the VCN, subnets and associated information within the VCN, one or more VRs associated with the VCN, compute instances and associated VNICs within the VCN, NVDs that perform various virtualized network functions associated with the VCN (e.g., VNICs, VRs, gateways), VCN status information, and other VCN-related information. In certain embodiments, the VCN distribution service exposes the configuration information or a portion thereof stored by the VCN control plane to the NVD. Using the distributed information, packets can be forwarded to and from compute instances within the VCN by updating information stored and used by the NVD (e.g., forwarding tables, routing tables, etc.).

[0040] In certain embodiments, the creation of VCNs and subnets is handled by the VCN control plane (CP), and the startup of compute instances is handled by the compute control plane. The compute control plane is configured to allocate physical resources for compute instances and then call the VCN control plane to create VNICs and connect to the compute instances. The VCN CP also sends VCN data mappings to the VCN data plane, which is configured to perform packet forwarding and routing functions. In certain embodiments, the VCN CP provides a distribution service configured to provide updates to the VCN data plane. Examples of VCN control planes are shown in Figures 17, 18, 19, and 20 (see reference numbers 1716, 1816, 1916, and 2016) and are described below.

[0041] Customers can create one or more VCNs using resources hosted by CSPI. Compute instances deployed on a customer VCN can communicate with different endpoints. These endpoints can include endpoints hosted by CSPI and endpoints outside of CSPL.

[0042] Various different architectures for implementing cloud-based services using CSPI are shown in Figures 1, 2, 3, 4, 5, 17, 18, 19, and 21, and are described below. Figure 1 is a high-level diagram of a distributed environment 100 showing an overlay VCN or customer VCN hosted by CSPI according to a particular embodiment. The distributed environment shown in Figure 1 includes multiple elements within the overlay network. The distributed environment 100 shown in Figure 1 is merely an example and is not intended to unduly limit the scope of the claimed embodiments. Many variations, alternatives, and modifications are possible. For example, in some implementations, the distributed environment shown in Figure 1 may have more or fewer systems or elements than those shown in Figure 1, may combine two or more systems, or may have different system configurations or arrangements.

[0043] As shown in the example in Figure 1, the distributed environment 100 includes a CSPI 101 that provides services and resources that customers can subscribe to and use to build a virtual cloud network (VCN). In a particular embodiment, the CSPI 101 provides IaaS services to subscriber customers. The data centers within the CSPI 101 may be organized into one or more regions. Figure 1 shows an example of a region, the “US region” 102. The customer has configured a customer VCN 104 for region 102. The customer can deploy various compute instances on the VCN 104, which may include virtual machines or bare metal instances. Examples of instances include applications, databases, load balancers, etc.

[0044] In the embodiment shown in Figure 1, customer VCN 104 includes two subnets, namely "Subnet-1" and "Subnet-2", each subnet having its own CIDR IP address range. In Figure 1, the overlay IP address range for subnet-1 is 10.0 / 16, and the address range for subnet-2 is 10.1 / 16. The VCN virtual router 105 represents the logical gateway of the VCN, enabling communication between subnets of VCN 104 and communication with other endpoints outside the VCN. The VCN VR 105 is configured to route traffic between VNICs within VCN 104 and gateways associated with VCN 104. The VCN VR 105 provides ports to each subnet of VCN 104. For example, the VR 105 can provide a port with IP address 10.0.0.1 to subnet-1 and a port with IP address 10.1.0.1 to subnet-2.

[0045] Multiple compute instances can be deployed on each subnet. In this case, compute instances may be virtual machine instances and / or bare metal instances. Compute instances within a subnet may be hosted by one or more host machines within CSPI101. Compute instances join the subnet via the VNIC associated with them. For example, as shown in Figure 1, compute instance C1 is part of subnet-1 via the VNIC associated with it. Similarly, compute instance C2 is part of subnet-1 via the VNIC associated with C2. Similarly, multiple compute instances, which may be virtual machine instances or bare metal instances, may be part of subnet-1. Each compute instance is assigned a private overlay IP address and MAC address via the associated VNIC. For example, in Figure 1, compute instance C1 has the overlay IP address 10.0.0.2 and MAC address M1, and compute instance C2 has the private overlay IP address 10.0.0.3 and MAC address M2. Each compute instance in subnet-1, including compute instances C1 and C2, has a default route to VCN VR105 using IP address 10.0.0.1, which is the IP address of the port of VCN VR105 in subnet-1.

[0046] Multiple compute instances, including virtual machine instances and / or bare metal instances, can be deployed in subnet-2. For example, as shown in Figure 1, compute instances Dl and D2 are part of subnet-2 via the VNIC associated with each compute instance. In the embodiment shown in Figure 1, compute instance D1 has the overlay IP address 10.1.0.2 and MAC address MM1, and compute instance D2 has the private overlay IP address 10.1.0.3 and MAC address MM2. Each compute instance in subnet-2, including compute instances D1 and D2, has a default route to VCN VR105 using IP address 10.1.0.1, which is the IP address of the port of VCN VR105 in subnet-2.

[0047] Furthermore, VCN A104 may include one or more load balancers. For example, a load balancer may be provided for a subnet and configured to load balance traffic among multiple compute instances on the subnet. Alternatively, a load balancer may be provided to load balance traffic among subnets within the VCN.

[0048] A specific compute instance deployed on VCN104 can communicate with various different endpoints. These endpoints may include endpoints hosted by CSPI200 and endpoints outside of CSPI200. Endpoints hosted by CSPI101 may include endpoints on the same subnet as a particular compute instance (e.g., communication between two compute instances in subnet-1), endpoints on different subnets but within the same VCN (e.g., communication between a compute instance in subnet-1 and a compute instance in subnet-2), endpoints in different VCNs within the same region (e.g., communication between a compute instance in subnet-1 and an endpoint in a VCN in the same region 106 or 110, or between a compute instance in subnet-1 and an endpoint in service network 110 in the same region), or endpoints in VCNs in different regions (e.g., communication between a compute instance in subnet-1 and an endpoint in a VCN in a different region 108). In addition, compute instances in subnets hosted by CSPI101 can communicate with endpoints not hosted by CSPI101 (i.e., outside of CSPI101). These external endpoints include endpoints within the customer's on-premises network 116, endpoints within other remote cloud host networks 118, public endpoints 114 accessible via public networks such as the internet, and other endpoints.

[0049] Communication between compute instances on the same subnet is facilitated using VNICs associated with the source and destination compute instances. For example, compute instance C1 in subnet-1 may want to send a packet to compute instance C2, also in subnet-1. For a packet sent from the source compute instance, whose destination is another compute instance on the same subnet, this packet is first processed by the VNIC associated with the source compute instance. The processing performed by the VNIC associated with the source compute instance may include determining the packet's destination information from the packet header, identifying any policies (e.g., security lists) configured for the VNIC associated with the source compute instance, determining the packet's next hop, performing any packet encapsulation / decapsulation functions as needed, and forwarding / routing the packet to the next hop to facilitate communication to its intended destination. If the destination compute instance is on the same subnet as the source compute instance, the VNIC associated with the source compute instance is configured to identify the VNIC associated with the destination compute instance and forward the packet to that VNIC for processing. Next, the VNIC associated with the destination compute instance is executed and forwards the packets to the destination compute instance.

[0050] When a packet is transmitted from a compute instance within a subnet to an endpoint in a different subnet of the same VCN, the communication is facilitated by the VNICs associated with the source and destination compute instances, and the VCN VR. For example, if compute instance C1 in subnet-1 in Figure 1 wants to send a packet to compute instance D1 in subnet-2, the packet is first processed by the VNIC associated with compute instance C1. The VNIC associated with compute instance C1 is configured to route the packet to VCN VR105 using the VCN VR's default route or port 10.0.0.1. VCN VR105 is configured to route the packet to subnet-2 using port 10.1.0.1. The packet is then received and processed by the VNIC associated with D1, and the VNIC forwards the packet to compute instance D1.

[0051] To transmit packets from compute instances within VCN104 to endpoints outside VCN104, communication is facilitated by a VNIC associated with the source compute instance, VCN VR105, and a gateway associated with VCN104. One or more types of gateways can be associated with VCN104. A gateway is an interface between the VCN and another endpoint, which is outside the VCN. A gateway is a Layer 3 / IP layer concept that enables communication between the VCN and endpoints outside the VCN. Therefore, gateways facilitate traffic flow between the VCN and other VCNs or networks. Different types of gateways can be configured in the VCN to facilitate different types of communication with different types of endpoints. Through gateways, communication may take place over a public network (e.g., the internet) or a private network. Various communication protocols may be used for these communications.

[0052] For example, compute instance C1 may want to communicate with an endpoint outside of VCN104. The packet may first be processed by the VNIC associated with source compute instance C1. The VNIC processing determines that the packet's destination is outside subnet-1 of Cl. The VNIC associated with C1 can then forward the packet to VCN VR105 of VCN104. VCN VR105 then processes the packet and, as part of the processing, determines a specific gateway associated with VCN104 as the packet's next hop based on the packet's destination. VCN VR105 can then forward the packet to the specific gateway. For example, if the destination is an endpoint within the customer's operation-premise network, the packet may be forwarded by VCN VR105 to a dynamic routing gateway (DRG) 122 configured for VCN104. The packet is then forwarded from the gateway to the next hop, facilitating communication of the packet to its intended final destination.

[0053] Various different types of gateways may be configured for the VCN. Examples of gateways that may be configured for the VCN are shown in Figure 1 and described below. Examples of gateways associated with the VCN are also shown in Figures 17, 18, 19, and 20 (for example, gateways shown by reference numbers 1734, 1736, 1738, 1834, 1836, 1838, 1934, 1936, 1938, 2034, 2036, and 2038) and described below. As shown in the embodiment shown in Figure 1, a dynamic routing gateway (DRG) 122 may be added to or associated with the customer VCN 104. The DRG 122 provides a path for private network traffic communication between the customer VCN 104 and another endpoint. The other endpoint may be the customer on-premises network 116, VCN 108 in a different region of CSPI 101, or another remote cloud network 118 not hosted by CSPI 101. The customer on-premises network 116 may be a customer network or customer data center built using the customer's resources. Access to the customer on-premises network 116 is generally strictly restricted. For a customer that has both the customer on-premises network 116 and one or more VCNs 104 deployed or hosted in the cloud by CSPI 101, the customer may want the on-premises network 116 and the cloud-based VCNs 104 to be able to communicate with each other. This would allow the customer to build an enhanced hybrid environment that includes the customer's VCNs 104 hosted by CSPI 101 and the on-premises network 116. DRG 122 enables such communication. To enable such communication, a communication channel 124 is configured. In this case, one endpoint of the communication channel is on the customer on-premises network 116, and the other endpoint is on CSPI 101 and connected to the customer VCN 104. The communication channel 124 can be via a public communication network such as the internet, or a private communication network.Various different communication protocols can be used, such as IPsec VPN technology on public communication networks like the Internet, and Oracle®'s FastConnect technology which uses a private network instead of a public network. Devices or equipment within the customer on-premises network 116 that form one endpoint of communication channel 124 are called customer premises equipment (CPE), such as CPE126 shown in Figure 1. The endpoint on the CSPI101 side may be a host machine running DRG122.

[0054] In certain embodiments, a Remote Peering Connection (RPC) can be added to the DRG. This allows a customer to peer one VCN with another VCN in a different region. Using such an RPC, a customer VCN 104 can connect to a VCN 108 in a different region using the DRG 122. The DRG 122 may also be used to communicate with other remote cloud networks 118 not hosted by the CSPI 101, such as the Microsoft® Azure cloud or the Amazon® AWS cloud.

[0055] As shown in Figure 1, an Internet Gateway (IGW) 120 can be configured on the customer VCN 104 to enable compute instances on the customer VCN 104 to communicate with a public endpoint 114 accessible via a public network such as the Internet. The IGW 120 is a gateway for connecting the VCN to a public network such as the Internet. The IGW 120 enables public subnets within a VCN, such as VCN 104 (resources within public subnets have public overlay IP addresses), to directly access a public endpoint 112 on a public network such as the Internet 114. Connections can be initiated from subnets within VCN 104 or from the Internet using the IGW 120.

[0056] A Network Address Translation (NAT) gateway 128 can be configured in the customer VCN 104. The NAT gateway 128 enables cloud resources within the customer VCN that do not have dedicated public overlay IP addresses to access the internet without directly exposing them to incoming internet connectivity (e.g., L4-L7 connectivity). This allows private subnets within the VCN, such as private subnet-1 of VCN 104, to have private access to public endpoints on the internet. With the NAT gateway, private subnets can initiate connections to the public internet, but connections cannot be initiated from the internet to the private subnets.

[0057] In certain embodiments, a service gateway (SGW) 126 can be configured in a customer VCN 104. The SGW 126 provides a route for private network traffic between VCN 104 and service endpoints supported by a service network 110. In certain embodiments, the service network 110 may be provided by a CSP and can provide a variety of services. An example of such a service network is the Oracle® service network, which provides a variety of services that customers can use. For example, compute instances (e.g., database systems) in a private subnet of customer VCN 104 can back up data to service endpoints (e.g., object storage devices) without requiring a public IP address or access to the internet. In some embodiments, a VCN may have only one SGW, and connections can only be initiated from subnets within the VCN, and not from the service network 110. When peering a VCN with another VCN, resources in the other VCN typically cannot access the SGW. Resources in an on-premises network connected to a VCN via FastConnect or VPN Connect can also use the service gateway configured for that VCN.

[0058] In some implementations, SGW126 uses service-classless inter-domain routing (CIDR) labels. A CIDR label is a string representing all regionally exposed IP address ranges for a service or group of services of interest. Customers use service CIDR labels to control traffic to services when configuring SGW and associated routing rules. Customers can optionally use service CIDR labels when configuring security rules without having to adjust security rules if the public IP addresses of services change in the future.

[0059] The Local Peering Gateway (LPG) 132 is an addable gateway to the customer VCN 104 that enables the VCN 104 to peer with other VCNs within the same region. Peering means that VCNs communicate using private IP addresses without traffic traversing a public network such as the internet or routing traffic through the customer's on-premises network 116. In a preferred embodiment, the VCN has a separate LPG for each established peering. Local peering, or VCN peering, is a common practice used to establish network connectivity between different applications or infrastructure management functions.

[0060] Service providers, such as service providers on service network 110, can provide access to their services using different access models. According to the public access model, a service may be exposed as a public endpoint accessible publicly by compute instances within the customer VCN via a public network such as the internet, or it may be accessed privately via SGW126. According to a specific private access model, a service may be accessed as a private IP endpoint within a private subnet within the customer VCN. This is called private endpoint (PE) access and allows service providers to expose their services as instances within the customer's private network. A private endpoint resource represents a service within the customer VCN. Each PE appears as a VNIC (referred to as a PE-VNIC, having one or more private IPs) selected by the customer from a subnet within the customer VCN. Thus, a PE provides a way to provide services within the customer's private VCN subnet using a VNIC. Because the endpoint is exposed as a VNIC, the PE VNIC can utilize all the features associated with a VNIC, such as routing rules and security lists.

[0061] Service providers enable access via PE by registering their services. Providers can associate policies with services that restrict their visibility to customer tenants. Providers can register multiple services under a single virtual IP address (VIP), especially in the case of multi-tenant services. Multiple private endpoints may exist representing the same service (across multiple VCNs).

[0062] Subsequently, compute instances within the private subnet can access the service using the PE VNIC's private IP address or service DNS name. Compute instances within the customer VCN can access the service by sending traffic to the PE's private IP address within the customer VCN. The Private Access Gateway (PAGW) 130 is a gateway resource that can connect to a service provider VCN (e.g., a VCN within service network 110) and act as the receiving / transmitting point for all traffic to and from the customer subnet private endpoint. The PAGW 130 allows the provider to scale the number of PE connections without utilizing internal IP address resources. The provider only needs to configure one PAGW for any number of services registered in a single VCN. The provider can present a service as a private endpoint in multiple VCNs for one or more customers. From the customer's perspective, the PE VNIC appears to be connected to the service the customer wants to interact with, rather than to the customer's instances. Traffic directed to the private endpoint is routed to the service via the PAGW 130. These are called customer-to-service private connections (C2S connections).

[0063] Furthermore, by using the PE concept, private access to the service can be extended to the customer's on-premises network and data center by enabling traffic to flow through FastConnect / IPsec links and private endpoints within the customer's VCN. Private access to the service can also be extended to the customer's peering VCN by enabling traffic to flow between LPG132 and PEs within the customer's VCN.

[0064] Customers can control VCN routing at the subnet level, allowing them to specify which subnets use which gateways within their VCN, such as VCN104. The VCN's route table can be used to determine whether traffic can be routed outside the VCN through a particular gateway. For example, in a specific case, the route table for a public subnet within customer VCN104 might allow non-local traffic to be sent via IGW120. The route table for a private subnet within the same customer VCN104 might allow traffic to CSP services via SGW126. All remaining traffic could be sent via NAT gateway 128. The route table only controls traffic leaving the VCN.

[0065] Security lists associated with a VCN are used to control inbound connections and traffic entering the VCN via gateways. All resources within a subnet use the same mute table and security lists. Security lists may be used to control specific types of traffic entering and leaving instances within a VCN subnet. Security list rules may include inbound and outbound rules. For example, inbound rules may specify allowed source address ranges, and outbound rules may specify allowed destination address ranges. Security rules may specify specific protocols (e.g., TCP, ICMP), specific ports (e.g., port 22 for SSH, port 3389 for Windows® RDP), etc. In certain implementations, the instance's operating system may enforce its own firewall rules that match the security list rules. Rules may be stateful (e.g., connections are tracked and responses are automatically allowed without explicit security list rules for response traffic) or stateless.

[0066] Access from a customer VCN (i.e., resources or compute instances deployed on VCN104) can be classified as public access, private access, or dedicated access. Public access refers to an access model for accessing public endpoints using public IP addresses or NAT. Private access allows customer workloads within VCN104 with private IP addresses (e.g., resources in a private subnet) to access services without traversing a public network such as the internet. In certain embodiments, CSPI101 allows customer VCN workloads with private IP addresses to access the public service endpoint of a service using a service gateway. Thus, the service gateway provides a private access model by establishing a virtual link between the customer VCN and the public endpoint of a service that resides outside the customer's private network.

[0067] Furthermore, CSPI can provide dedicated public access using technologies such as FastConnect public peering. In this case, customer on-premises instances can access one or more services within the customer VCN using FastConnect connectivity without going through a public network such as the internet. CSPI can also provide dedicated private access using FastConnect private peering. In this case, customer on-premises instances with private IP addresses can access customer VCN workloads using FastConnect connectivity. FastConnect is a network connectivity used as an alternative to connecting customer on-premises networks to CSPI and its services using the public internet. FastConnect provides a simple, flexible, and economical way to create dedicated private connectivity with higher bandwidth options and a reliable, consistent networking experience compared to internet-based connectivity.

[0068] Figure 1 and the accompanying description above illustrate various virtualization elements in an exemplary virtual network. As mentioned above, the virtual network is built on an underlying physical network or infrastructure network. Figure 2 is a simplified architectural diagram showing the physical elements within the physical network within the CSPI200 that provide the foundation for the virtual network, according to a particular embodiment. As shown, the CSPI200 provides a distributed environment including elements and resources (e.g., compute resources, memory resources, and networking resources) provided by a Cloud Service Provider (CSP). These elements and resources are used to provide cloud services (e.g., IaaS services) to subscribers, i.e., customers who subscribe to one or more services provided by the CSP. Based on the services a customer subscribes to, the CSPI200 provides some resources (e.g., compute resources, memory resources, and networking resources) to the customer. The customer can then use the physical compute resources, memory resources, and networking resources provided by the CSPI200 to build their own cloud-based (i.e., CSPI-hosted) customizable private virtual network. As mentioned above, these customer networks are called virtual cloud networks (VCNs). Customers can deploy one or more customer resources, such as compute instances, to these customer VCNs. Compute instances may be virtual machines, bare metal instances, etc. CSPI200 provides infrastructure and a suite of complementary cloud services that enable customers to build and run a wide range of applications and services in a highly available host environment.

[0069] In the exemplary embodiment shown in Figure 2, the physical elements of CSPI200 include one or more physical host machines or physical servers (e.g., 202, 206, 208), network virtualization devices (NVDs) (e.g., 210, 212), top-of-rack (TOR) switches (e.g., 214, 216), a physical network (e.g., 218), and switches within physical network 218. The physical host machines or servers can host and run various compute instances participating in one or more subnets of the VCN. Compute instances may include virtual machine instances and bare metal instances. For example, the various compute instances shown in Figure 1 may be hosted by the physical host machines shown in Figure 2. Virtual machine compute instances in the VCN may run on one host machine or on several different host machines. The physical host machines can also host virtual host machines, container-based hosts or functions, etc. The VIC and VCN VR shown in Figure 1 may run on the FTVD shown in Figure 2. The gateway shown in Figure 1 may be run by the host machine and / or NVD shown in Figure 2.

[0070] A host machine or server can run a hypervisor (also known as a virtual machine monitor or VMM) that creates and enables a virtualized environment on the host machine. Virtualization or a virtualized environment facilitates cloud-based computing. One or more compute instances may be created, run, and managed on the host machine by a hypervisor on the host machine. The hypervisor on the host machine can share the host machine's physical compute resources (e.g., compute resources, memory resources, and networking resources) among various compute instances running on the host machine.

[0071] For example, as shown in Figure 2, host machines 202 and 208 run hypervisors 260 and 266, respectively. These hypervisors may be implemented using software, firmware, hardware, or a combination thereof. Typically, a hypervisor is a process or software layer residing in the host machine's operating system (OS), which runs on the host machine's hardware processor. A hypervisor provides a virtualization environment that allows the host machine's physical computing resources (e.g., processing resources such as processors / cores, memory resources, and networking resources) to be shared among various virtual machine computing instances running on the host machine. For example, in Figure 2, hypervisor 260 resides in the OS of host machine 202 and allows the host machine 202's computing resources (e.g., processing resources, memory resources, and networking resources) to be shared among computing instances (e.g., virtual machines) running on host machine 202. A virtual machine can have its own OS (called a guest OS). This guest OS may be the same as or different from the host machine's OS. The operating system (OS) of a virtual machine running on a host machine may be the same as, or different from, the operating systems of other virtual machines running on the same host machine. Therefore, a hypervisor can run multiple OSs in parallel while sharing the same computing resources of the host machine. The host machines shown in Figure 2 may have the same type of hypervisor or different types of hypervisors.

[0072] Compute instances may be virtual machine instances or bare metal instances. In Figure 2, compute instance 268 on host machine 202 and compute instance 274 on host machine 208 are examples of virtual machine instances. Host machine 206 is an example of a bare metal instance provided to a customer.

[0073] In certain examples, the entire host machine may be provided to a single customer, and one or more compute instances (either virtual machines or bare metal instances) hosted by that host machine may all belong to the same customer. In other examples, the host machine may be shared among multiple customers (i.e., multiple tenants). In such a multi-tenant scenario, the host machine can host virtual machine compute instances belonging to different customers. These compute instances may be members of different VCNs of different customers. In certain embodiments, bare metal compute instances are hosted by bare metal servers without a hypervisor. When bare metal compute instances are provided, a single customer or tenant maintains control of the physical CPU, memory, and network interfaces of the host machine hosting the bare metal instances, and the host machine is not shared with other customers or tenants.

[0074] As mentioned above, each compute instance that is part of a VCN is associated with a VNIC that enables the compute instance to be a member of the VCN's subnet. The VNIC associated with a compute instance facilitates the communication of packets or frames to and from the compute instance. The VNIC is associated with the compute instance when it is created. In certain embodiments, for a compute instance run by a host machine, the VNIC associated with that compute instance is run by an NVD connected to the host machine. For example, in Figure 2, host machine 202 runs virtual machine compute instance 268 associated with VNIC 276, and VNIC 276 is run by an NVD 210 connected to host machine 202. In another example, bare metal instance 272 hosted by host machine 206 is associated with VNIC 280, which is run by an NVD 212 connected to host machine 206. In yet another example, VNIC 284 is associated with compute instance 274 run by host machine 208, and VNIC 284 is run by an NVD 212 connected to host machine 208.

[0075] For compute instances hosted by a host machine, an NVD connected to that host machine executes a VCN VR corresponding to the VCN of which the compute instance is a member. For example, in the embodiment shown in Figure 2, NVD210 executes VCN VR277 corresponding to the VCN of which compute instance 268 is a member. Additionally, NVD212 can execute one or more VCN VR283 corresponding to the VCNs of compute instances hosted by host machines 206 and 208.

[0076] A host machine may include one or more network interface cards (NICs) for connecting it to other devices. The NICs on the host machine may provide one or more ports (or interfaces) for communicating with another device. For example, a host machine can be connected to an NVD using one or more ports (or interfaces) provided on the host machine and the NVD. Alternatively, a host machine can be connected to other devices, such as another host machine.

[0077] For example, in Figure 2, host machine 202 is connected to NVD210 using a link 220 that extends between port 234 provided by NIC 232 of host machine 202 and port 236 of NVD210. Host machine 206 is connected to NVD212 using a link 224 that extends between port 246 provided by NIC 244 of host machine 206 and port 248 of NVD212. Host machine 208 is connected to NVD212 using a link 226 that extends between port 252 provided by NIC 250 of host machine 208 and port 254 of NVD212.

[0078] Similarly, the NVDs are connected via communication links to top-of-rack (TOR) switches connected to a physical network 218 (also called a switch fabric). In certain embodiments, the links between the host machines and the NVDs, and between the NVDs and the TOR switches, are Ethernet® links. For example, in Figure 2, NVDs 210 and 212 are connected to TOR switches 214 and 216, respectively, via links 228 and 230. In certain embodiments, links 220, 224, 226, 228, and 230 are Ethernet® links. The collection of host machines and NVDs connected to the TOR is sometimes referred to as a rack.

[0079] The physical network 218 provides a communication fabric that enables communication between TOR switches. The physical network 218 may be a multi-layer network. In a particular implementation, the physical network 218 is a multi-layer Clos network of switches, and TOR switches 214 and 216 represent leaf-level nodes of the multi-layer and multi-node physical switching network 218. Different Clos network configurations are possible, including but not limited to 2-layer, 3-layer, 4-layer, 5-layer networks, and generally "n"-layer networks. An example of a Clos network is shown in Figure 5 and described below.

[0080] Various connection configurations are possible between the host machine and N VDs, including one-to-one, many-to-one, and one-to-many configurations. In an example of a one-to-one configuration, each host machine is connected to its own separate NVD. For example, in Figure 2, host machine 202 is connected to NVD210 via host machine 202's NIC232. In a many-to-one configuration, multiple host machines are connected to a single NVD. For example, in Figure 2, host machines 206 and 208 are connected to the same NVD212 via NIC244 and 250, respectively.

[0081] In a one-to-many configuration, one host machine is connected to multiple NVDs. Figure 3 shows an example within CSPI300 where a host machine is connected to multiple NVDs. As shown in Figure 3, the host machine 302 has a network interface card (NIC) 304 that includes multiple ports 306 and 30S. The host machine 300 is connected to the first NVD 310 via port 306 and link 320, and to the second NVD 312 via port 308 and link 322. Ports 306 and 308 may be Ethernet® ports, and links 320 and 322 between the host machine 302 and the NVDs 310 and 312 may be Ethernet® links. The NVD 310 is connected to the first TOR switch 314, and the NVD 312 is connected to the second TOR switch 316. The links between the NVDs 310 and 312 and the TOR switches 314 and 316 may be Ethernet® links. TOR switches 314 and 316 represent layer-0 switching devices within a multilayer physical network 318.

[0082] The configuration shown in Figure 3 provides two separate physical network paths from the physical switch network 318 to the host machine 302: a first path from TOR switch 314 through NVD 310 to the host machine 302, and a second path from TOR switch 316 through NVD 312 to the host machine 302. These separate paths provide enhanced availability (referred to as high availability) for the host machine 302. If one path experiences a problem (e.g., one link in the path fails) or if there is a problem with a device (e.g., a particular NVD is not functioning), the other path can be used for communication with the host machine 302.

[0083] In the configuration shown in Figure 3, the host machine is connected to two different NVDs using two different ports provided by the host machine's NIC. In other embodiments, the host machine may include multiple NICs that enable connections between the host machine and multiple NVDs.

[0084] Referring again to Figure 2, the NVD is a physical device or element that performs one or more network virtualization functions and / or memory virtualization functions. The NVD may be any device having one or more processing units (e.g., a CPU, a network processing unit (NPU), an FPGA, a packet processing pipeline), memory including a cache, and ports. Various virtualization functions may be performed by software / firmware executed by one or more processing units of the NVD.

[0085] NVDs may be implemented in various different forms. For example, in certain embodiments, an NVD may be implemented as an interface card called a smart NIC or intelligent NIC with an integrated processor. A smart NIC is a separate device from the NIC on the host machine. In Figure 2, NVD210 may be implemented as a smart NIC connected to host machine 202, and NVD212 may be implemented as smart NICs connected to host machines 206 and 208.

[0086] However, the smart NIC is just one example of an NVD implementation. Various other implementations are possible. For example, in some other implementations, the NVD or one or more functions performed by the NVD may be incorporated into or performed by one or more host machines, one or more TOR switches, and other elements of the CSPI200. For example, the NVD may be integrated into the host machine. In this case, the functions performed by the NVD are performed by the host machine. As another example, the NVD may be part of a TOR switch, or the TOR switch may be configured to perform functions performed by the NVD, enabling the TOR switch to perform various complex packet translations used in public clouds. A TOR that performs the functions of the NVD is sometimes called a smart TOR. In yet another implementation that provides customers with virtual machine (VM) instances rather than bare metal (BM) instances, the functions provided by the NVD may be implemented within the hypervisor of the host machine. In some other implementations, some of the functions of the NVD may be offloaded to a centralized service running on a set of host machines.

[0087] As shown in Figure 2, in certain embodiments, such as when implemented as a smart NIC, the NVD may have multiple physical ports that enable it to connect to one or more host machines and one or more TOR switches. The ports on the NVD can be classified as host-facing ports (also called "south ports") or network-facing or TOR-facing ports (also called "north ports"). Host-facing ports on the NVD are the ports used to connect the NVD to a host machine. Examples of host-facing ports in Figure 2 include port 236 on the NVD210 and ports 248 and 254 on the NVD212. Network-facing ports on the NVD are the ports used to connect the NVD to a TOR switch. Examples of network-facing ports in Figure 2 include port 256 on the NVD210 and port 258 on the NVD212. As shown in Figure 2, the NVD210 is connected to the TOR switch 214 via a link 228 extending from port 256 on the NVD210 to the TOR switch 214. Similarly, the NVD212 is connected to the TOR switch 216 via a link 230 that extends from port 258 of the NVD212 to the TOR switch 216.

[0088] The NVD can receive packets and frames from the host machine (for example, packets and frames generated by compute instances hosted by the host machine) via its host-facing port, perform the necessary packet processing, and then forward the packets and frames to the TOR switch via its network-facing port. The NVD can also receive packets and frames from the TOR switch via its network-facing port, perform the necessary packet processing, and then forward the packets and frames to the host machine via its host-facing port.

[0089] In certain embodiments, multiple ports and associated links may be provided between the NVD and the TOR switch. By aggregating these ports and links, a link aggregator group (LAG) of multiple ports or links can be formed. Link aggregation allows multiple physical links between two endpoints (e.g., between the NVD and the TOR switch) to be treated as a single logical link. All physical links within a given LAG can operate in full-duplex mode at the same speed. LAGs help to increase the bandwidth and reliability of the connection between the two endpoints. If one of the physical links in the LAG fails, traffic is dynamically and transparently reassigned to another physical link within the LAG. Aggregated physical links provide higher bandwidth than individual links. Multiple ports associated with a LAG are treated as a single logical port. Traffic can be load-balanced across the multiple physical links of the LAG. One or more LAGs can be configured between two endpoints. The two endpoints may be, for example, between the NVD and the TOR switch, or between a host machine and the NVD.

[0090] NVD implements or performs network virtualization functions. These functions are performed by software / firmware run by NVD. Examples of network virtualization functions include, but are not limited to, packet encapsulation and decapsulation functions, functions for creating VCN networks, functions for implementing network policies such as VCN security list (firewall) functions, and functions for facilitating the routing and forwarding of packets between compute instances within the VCN. In certain embodiments, upon receiving a packet, NVD is configured to run a packet processing pipeline that processes the packet and determines how to forward or route it. As part of this packet processing pipeline, NVD provides the execution of VNICs related to cis within the VCN, the execution of virtual routers (VRs) related to the VCN, packet encapsulation and decapsulation to facilitate forwarding or routing within the virtual network, the execution of specific gateways (e.g., local peering gateways), the implementation of security lists, network security groups, network address translation (NAT) functions (e.g., translation from public IP to private IP on a per-host basis), throttling functions, and other functions.

[0091] In some embodiments, the packet processing data path in the NVD may include multiple packet pipelines. Each packet pipeline consists of a set of packet translation stages. In some implementations, upon receiving a packet, it is parsed and classified into a single pipeline. The packet is then processed linearly, stage by stage, until it is discarded or sent out through the NVD's interface. These stages provide the basic functional packet processing building blocks (e.g., header validation, throttling, insertion of new Layer 2 headers, L4 firewall execution, VCN encapsulation / decapsulation), and as a result, new pipelines can be constructed by assembling existing stages, and new functionality can be added by creating new stages and inserting them into existing pipelines.

[0092] The NVD can perform both control plane and data plane functions corresponding to the VCN's control plane and data plane. Examples of the VCN control plane are shown in Figures 17, 18, 19, and 20 (see reference numbers 1716, 1816, 1916, and 2016) and are described below. Examples of the VCN data plane are shown in Figures 17, 18, 19, and 20 (see reference numbers 1718, 1818, 1918, and 2018) and are described below. Control plane functions include functions used to configure the network to control how data is forwarded (e.g., setting routes and route tables, configuring VNICs). In certain embodiments, a VCN control plane is provided that centrally calculates the mapping of all overlays to the substrate and exposes it to the NVD and virtual network edge devices (e.g., various gateways such as DRG, SGW, IGW). Firewall rules can also be exposed using the same mechanism. In certain embodiments, the NVD retrieves only the mappings relevant to that NVD. The data plane function includes the ability to perform the actual routing / forwarding of packets based on the configuration set using the control plane. The VCN data plane is implemented by encapsulating customer network packets before they pass through the backbone network. The encapsulation / decapsulation function is implemented in the NVD. In certain embodiments, the NVD is configured to intercept all network packets entering and leaving the host machine and to perform network virtualization functions.

[0093] As described above, NVD performs various virtualization functions, including VNICs and VCN VRs. An NVD can run VNICs associated with compute instances hosted by one or more host machines connected to a VNIC. For example, as shown in Figure 2, NVD210 runs the functions of VNIC276 associated with compute instance 268 hosted by host machine 202 connected to NVD210. As another example, NVD212 runs VNIC280 associated with bare-metal compute instance 272 hosted by host machine 206 and VNIC284 associated with compute instance 274 hosted by host machine 208. A host machine can host compute instances belonging to different VCNs belonging to different customers. An NVD connected to a host machine can run VNICs corresponding to compute instances (i.e., perform functions associated with VNICs).

[0094] Furthermore, the NVD runs a VCN virtual router corresponding to the VCN of the compute instance. For example, in the embodiment shown in Figure 2, NVD210 runs VCN VR277 corresponding to the VCN to which compute instance 268 belongs. NVD212 runs one or more VCN VR283 corresponding to one or more VCNs to which compute instances hosted on host machines 206 and 208 belong. In a particular embodiment, a VCN VR corresponding to a VCN is run by all NVDs connected to a host machine that hosts at least one compute instance belonging to that VCN. If a host machine hosts compute instances belonging to a different VCN, the NVDs connected to that host machine can run VCN VRs corresponding to different VCNs.

[0095] In addition to VNICs and VCN VRs, an NVD may include one or more hardware elements that run various software (e.g., daemons) and facilitate various network virtualization functions performed by the NVD. For simplicity, these various elements are grouped as “packet processing elements” as shown in Figure 2. For example, NVD210 includes packet processing element 286, and NVD212 includes packet processing element 288. For example, a packet processing element of an NVD may include a packet processor configured to monitor all packets received and communicated using the NVD and to store network information by interacting with the NVD’s ports and hardware interfaces. Network information may include, for example, network flow information to identify different network flows processed by the NVD and information about each flow (e.g., statistics for each flow). In certain embodiments, network flow information may be stored on a per-VNIC basis. As another example, a packet processing element may include a replication agent configured to replicate the information stored by the NVD to one or more different replication target stores. As yet another example, the packet processing element may include a logging agent configured to perform the NVD's logging function. The packet processing element may also include software to monitor the performance and health of the NVD, and optionally the status and health of other elements connected to the NVD.

[0096] Figure 1 shows the elements of an exemplary virtual or overlay network, including a VCN, subnets within the VCN, compute instances deployed on the subnets, VNICs associated with the compute instances, a VR for the VCN, and a set of gateways configured for the VCN. The overlay elements shown in Figure 1 may be run or hosted by one or more of the physical elements shown in Figure 2. For example, compute instances within a VCN may be run or hosted by one or more host machines shown in Figure 2. In the case of compute instances hosted by host machines, the VNICs associated with those compute instances are typically run by NVDs connected to that host machine (i.e., VNIC functionality is provided by NVDs connected to that host machine). The VCN VR functionality of the VCN is run by all NVDs connected to the host machines that host or run the compute instances that are part of that VCN. Gateways associated with the VCN may be run by one or more different types of NVDs. For example, some gateways may be run by smart NICs, and others may be run by one or more host machines or other implementations of NVDs.

[0097] As described above, compute instances within a customer VCN can communicate with a variety of different endpoints. These endpoints may be on the same subnet as the source compute instance, on a different subnet but still within the same VCN, or may include endpoints outside the source compute instance's VCN. These communications are facilitated using the VNIC associated with the compute instance, the VCN VR, and the gateway associated with the VCN.

[0098] Communication between two compute instances on the same subnet within a VCN is facilitated using VNICs associated with the source and destination compute instances. The source and destination compute instances may be hosted on the same host machine or on different host machines. Packets originating from the source compute instance may be forwarded from the host machine hosting the source compute instance to an NVD connected to that host machine. In the NVD, packets are processed using a packet processing pipeline, which may include the execution of the VNIC associated with the source compute instance. Because the destination endpoint of the packets is on the same subnet, the execution of the VNIC associated with the source compute instance forwards the packets to the NVD running the VNIC associated with the destination compute instance, where the NVD processes the packets and forwards them to the destination compute instance. The VNICs associated with the source and destination compute instances may run on the same NVD (for example, if both the source and destination compute instances are hosted on the same host machine) or on different NVDs (for example, if the source and destination compute instances are hosted on different host machines connected to different NVDs). The VNIC can use the routing / forwarding table stored by the NVD to determine the next hop of a packet.

[0099] When a packet is communicated from a compute instance within a subnet to an endpoint in a different subnet within the same VCN, the packet originating from the source compute instance is communicated from the host machine hosting the source compute instance to the NVD connected to that host machine. In the NVD, the packet is processed using a packet processing pipeline and a VR associated with the VCN, which may include the execution of one or more VNICs. For example, as part of the packet processing pipeline, the NVD executes or invokes a function corresponding to the VNIC associated with the source compute instance (also called executing the VNIC). The function executed by the VNIC may include examining the VLAN tag on the packet. Because the packet's destination is outside the subnet, a VCN VR function is invoked and executed by the NVD. The VCN VR then routes the packet to the NVD executing the VNIC associated with the destination compute instance. The VNIC associated with the destination compute instance then processes the packet and forwards it to the destination compute instance. The VNICs associated with the source compute instance and the destination compute instance may run on the same NVD (for example, if both the source compute instance and the destination compute instance are hosted by the same host machine), or they may run on different NVDs (for example, if the source compute instance and the destination compute instance are hosted by different host machines connected to different NVDs).

[0100] If the packet's destination is outside the VCN of the source compute instance, the packet originating from the source compute instance is communicated from the host machine hosting the source compute instance to the NVD connected to that host machine. The NVD runs the VNIC associated with the source compute instance. Because the packet's destination endpoint is outside the VCN, the packet is processed by the VCN VR of that VCN. The NVD invokes VCN VR functionality, which may result in the packet being forwarded to an NVD running the appropriate gateway associated with the VCN. For example, if the destination is an endpoint within the customer's on-premises network, the packet may be forwarded by the VCN VR to an NVD running the DRG gateway configured for the VCN. The VCN VR may run on the same NVD as the NVD running the VNIC associated with the source compute instance, or it may run on a different NVD. The gateway may run on an NVD that is a smart NIC, a host machine, or another NVD implementation. The packet is then processed by the gateway and forwarded to the next hop to facilitate communication of the packet to its intended destination endpoint. For example, in the embodiment shown in Figure 2, a packet originating from compute instance 268 may be communicated from host machine 202 to NVD210 via link 220 (using NIC 232). VNIC 276 on NVD210 is invoked because it is the VNIC associated with source compute instance 268. VNIC 276 is configured to examine the encapsulated information in the packet, determine the next hop for forwarding the packet to facilitate communication of the packet to its intended destination endpoint, and forward the packet to the determined next hop.

[0101] Compute instances deployed on a VCN can communicate with various different endpoints. These endpoints may include endpoints hosted by CSPI200 and endpoints outside of CSPI200. Endpoints hosted by CSPI200 may include instances within the same VCN or other VCNs (which may be customer VCNs or VCNs not belonging to a customer). Communication between endpoints hosted by CSPI200 may be performed over the physical network 218. Compute instances can also communicate with endpoints not hosted by CSPI200 or located outside of CSPI200. Examples of these endpoints include endpoints within the customer's on-premises network or data center, or public endpoints accessible over a public network such as the Internet. Communication with endpoints outside of CSPI200 may be performed over a public network (e.g., the Internet) (not shown in Figure 2) or a private network (not shown in Figure 2) using various communication protocols.

[0102] The architecture of the CSPI200 shown in Figure 2 is merely an example and is not intended to be limiting. Alternative embodiments are possible, and variations, substitutions, and modifications are possible. For example, in some implementations, the CSPI200 may have more or fewer systems or elements than those shown in Figure 2, may combine two or more systems, or may have different system configurations or arrangements. The systems, subsystems, and other elements shown in Figure 2 may be implemented as software (e.g., code, instructions, programs), hardware, or a combination thereof, executed by one or more processing units (e.g., processors, cores) of each system. The software may be stored in a non-temporary storage medium (e.g., a memory device).

[0103] Figure 4 shows a connection between a host machine and an NVD to provide I / O virtualization to support multi-tenancy functionality, according to a particular embodiment. As shown in Figure 4, the host machine 402 runs a hypervisor 404 that provides the virtualization environment. The host machine 402 runs two virtual machine instances, namely VM1 406 belonging to customer / tenant #1 and VM2 408 belonging to customer / tenant #2. The host machine 402 includes a physical NIC 410 connected to the NVD 412 via link 414. Each compute instance is connected to a VNIC run by the NVD 412. In the embodiment of Figure 4, VM1 406 is connected to VNIC-VM1 420 and VM2 408 is connected to VNIC-VM2 422.

[0104] As shown in Figure 4, NIC410 includes two logical NICs, namely logical NIC A 416 and logical NIC B 418. Each virtual machine is connected to its own logical NIC and configured to operate with its own logical NIC. For example, VM1 406 is connected to logical NIC A 416, and VM2 408 is connected to logical NIC B 418. Although the host machine 402 consists of only one physical NIC 410 shared by multiple tenants, the logical NICs allow each tenant's virtual machine to believe that it owns its own host machine and NIC.

[0105] In a particular embodiment, each logical NIC is assigned its own VLAN ID. Thus, logical NIC A 416 for tenant #1 is assigned a specific VLAN ID, and logical NIC B 418 for tenant #2 is assigned a different VLAN ID. When a packet is communicated from VM1 406, the hypervisor attaches the tag assigned to tenant #1 to the packet and then communicates the packet from host machine 402 to NVD412 via link 414. Similarly, when a packet is communicated from VM2 408, the hypervisor attaches the tag assigned to tenant #2 to the packet and then communicates the packet from host machine 402 to NVD412 via link 414. Thus, the packet 424 communicated from host machine 402 to NVD412 has an associated tag 426 that identifies a specific tenant and associated VM. When packet 424 is received from host machine 402 on NVD, the tag 426 associated with the packet is used to determine whether the packet should be processed by VNIC-VM1 420 or VNIC-VM2 422. The packet is then processed by the corresponding VNIC. The configuration shown in Figure 4 allows each tenant's compute instance to believe that it owns its own host machine and NIC. The configuration shown in Figure 4 provides I / O virtualization to support multi-tenancy functionality.

[0106] Figure 5 is a schematic block diagram showing a physical network 500 according to a particular embodiment. The embodiment shown in Figure 5 is constructed as a Clos network. A Clos network is a specific type of network topology designed to provide connectivity redundancy while maintaining high bimodal bandwidth and maximum resource utilization. A Clos network is a type of non-blocking, multi-stage or multi-layer switching network, where the number of stages or layers may be 2, 3, 4, 5, etc. The embodiment shown in Figure 5 is a 3-layer network including layers 1, 2, and 3. A TOR switch 504 represents a layer-0 switch in the Clos network. One or more NVDs are connected to the TOR switch. Layer-0 switches are also called edge devices in the physical network. Layer-0 switches are connected to layer-1 switches, also called leaf switches. In the embodiment shown in Figure 5, "n" layer-0 TOR switches are connected to "n" layer-1 switches to form pods. Each layer-0 switch in a pod is interconnected to all layer-1 switches in the pod, but switches between pods are not connected. In a particular implementation, two pods are referred to as a block. Each block is serviced by or connected to n Layer-2 switches (also called spine switches). The physical network topology may contain multiple blocks. Similarly, the Layer-2 switches are connected to n Layer-3 switches (also called superspine switches). Packet communication over the physical network 500 is typically performed using one or more Layer 3 communication protocols. Typically, all layers of the physical network except the TOR layer are n-way redundant, thus achieving high availability. The physical network can be extended by specifying policies for pods and blocks to control the mutual visibility of switches in the physical network.

[0107] A key feature of Clos networks is that the maximum hop count required to reach one Layer-0 switch from one Layer-0 switch to another (or from an NVD connected to a Layer-0 switch to another NVD connected to a Layer-0 switch) remains constant. For example, in a Layer 3 Clos network, a packet requires a maximum of 7 hops to reach one NVD from another. In this case, the source NVD and target NVD are connected to the leaf layer of the Clos network. Similarly, in a Layer 4 Clos network, a packet requires a maximum of 9 hops to reach one NVD from another. In this case, the source NVD and target NVD are connected to the leaf layer of the Clos network. Therefore, the Clos network architecture maintains a constant overall network latency, which is crucial for communication within and between data centers. Clos topologies are horizontally scalable and cost-effective. Network bandwidth / throughput capacity can be easily increased by adding more switches to each layer (e.g., more leaf and spine switches) and increasing the number of links between switches in adjacent layers.

[0108] In certain embodiments, each resource within the CSPI is assigned a unique identifier called a Cloud Identifier (CID). This identifier is included as part of the resource's information. This identifier can be used to manage the resource, for example, via a console or API. An example syntax for a CID is as follows:

[0109] ocid1.<RESOURCE TYPE> . <realm>[REGION] [FUTURE USE]<UNIQUE ID> In the formula, "ocid1" is a string that indicates the CID version.

[0110] "RESOURCE TYPE" represents the type of resource (e.g., instance, volume, VCN, subnet, user, group).

[0111] "REALM" represents the region where the resources reside. Exemplary values ​​include "c1" representing a commercial region, "c2" representing a government cloud region, or "c3" representing a federal government cloud region. Each region can have its own domain name.

[0112] "REGION" represents the region to which the resource belongs. If no region applies to the resource, this section may be left blank.

[0113] "FUTURE USE" indicates that it is reserved for future use. The "UNIQUE ID" is the unique identifier portion. This format may vary depending on the type of resource or service.

[0114] B. Exemplary Layer 2 VLAN Architecture This section describes technologies for providing Layer 2 networking capabilities in a virtualized cloud environment. Layer 2 capabilities are provided in addition to, and in relation to, the Layer 3 networking capabilities provided by the virtualized cloud environment. In certain embodiments, virtual Layer 2 and Layer 3 capabilities are provided by Oracle Cloud Infrastructure (OCI), provided by Oracle Corporation.

[0115] Following the introduction of Layer 2 networking functionality, this section describes Layer 2 implementation for VLANs. Subsequently, it describes Layer 2 VLAN services, including providing information about Layer 2 VLANs, which may include Media Access Control (MAC) address forwarding tables and / or Layer 2 switch statistics.

[0116] Preface The number of enterprise customers migrating their on-premises applications to cloud environments provided by cloud service providers (CSPs) continues to grow rapidly. However, many of these customers quickly realize that the migration journey to the cloud can be extremely challenging, requiring their existing applications to be rebuilt and redesigned to function in the cloud environment. This is because applications written for on-premises environments often rely on the characteristics of the physical network in terms of monitoring, availability, and scalability. Therefore, these on-premises applications need to be rebuilt and redesigned before they can function in the cloud environment.

[0117] There are several reasons why on-premises applications cannot easily migrate to a cloud environment. One of the main reasons is that current cloud virtual networks operate at Layer 3 of the OSI model, for example, the IP layer, and do not provide the Layer 2 functionality required by applications. Layer 3-based routing or forwarding involves determining where a packet should be sent (for example, to which customer instance) based on information contained in the Layer 3 header of the packet, for example, based on the destination IP address contained in the Layer 3 header of the packet. To facilitate this, the location of IP addresses within the virtualized cloud network is determined via a centralized control and orchestration system or controller. These may include, for example, IP addresses associated with customer entities or resources within the virtualized cloud environment.

[0118] Many customers are running applications in their on-premises environments that have stringent requirements for Layer 2 networking capabilities that are not currently addressed by current cloud offerings and IaaS service providers. For example, traffic is routed using Layer 3 protocols with Layer 3 headers in current cloud offerings, and the Layer 2 capabilities required by the applications are not supported. These Layer 2 capabilities may include features such as Address Resolution Protocol (ARP) processing, Media Access Control (MAC) address learning, and Layer 2 broadcast capabilities, Layer 2 (MAC-based) forwarding, and Layer 2 networking constructs. By providing virtualized Layer 2 networking capabilities in a virtualized cloud network as described in this disclosure, customers can now seamlessly migrate their legacy applications to the cloud environment without requiring any substantial restructuring or redesign. For example, the virtualized Layer 2 networking capabilities described herein enable such applications (e.g., VMware vSphere, vCenter, vSAN, and NSX-T components) to communicate at Layer 2 as they would in an on-premises environment. These applications can run the same versions and configurations in the public cloud, allowing customers to use legacy on-premises applications, including existing knowledge, tools, and processes associated with the legacy applications. Customers can also access native cloud services from their applications (for example, using VMware Software-Defined Data Centers (SDDCs)).

[0119] Another example is several legacy on-premises applications that require Layer 2 broadcast support for failover (e.g., enterprise clustering software applications, network virtual appliances). Illustrative applications include Fortinet FortiGate, IBM® QRadar, Palo Alto firewalls, Cisco ASA, Juniper SRX, and Oracle RAC (Real Application Clustering). As described in this disclosure, by providing virtualized Layer 2 networking in a virtualized public cloud, these appliances can now operate in a virtualized public cloud environment without modification. Virtualized Layer 2 networking capabilities comparable to on-premises are provided, as described herein. The virtualized Layer 2 networking capabilities described in this disclosure support traditional Layer 2 networking, including customer-defined VLANs, as well as support for unicast, broadcast, and multicast Layer 2 traffic capabilities. Layer 2-based packet routing and forwarding include using Layer 2 protocols and routing or forwarding packets, for example, based on the destination MAC address contained in the Layer 2 header, using information contained in the packet's Layer 2 header. Protocols used by enterprise applications (e.g., clustering software applications), such as ARP, Gratuitous Address Resolution Protocol (GARP), and Reverse Address Resolution Protocol (RARP), can now also function in cloud environments.

[0120] There are several reasons why traditional virtualized cloud infrastructure supports virtualized Layer 3 networking and not Layer 2 networking. Layer 2 networks typically do not scale in the same way as Layer 3 networks. Layer 2 network control protocols do not have the level of sophistication desired for scaling. For example, Layer 3 networks do not have to worry about packet looping, which Layer 2 networks must deal with. IP packets (i.e., Layer 3 packets) have the concept of Time To Live (TTL), while Layer 2 packets do not. IP addresses contained within Layer 3 packets have topological concepts such as subnets and CIDR ranges, while Layer 2 addresses (e.g., MAC addresses) do not. Layer 3 IP networks have built-in tools that facilitate troubleshooting, such as packet internet exploration and routing, for finding routing information. Such tools are not available for Layer 2. Layer 3 networks support multipath functionality, which is not available for Layer 2. Due to the lack of sophisticated control protocols for exchanging information between entities in a network (e.g., Border Gateway Protocol (BGP) and Open Shortest Path First (OSPF)), Layer 2 networks must rely on broadcast and multicast to learn about the network, which can negatively impact network performance. As the network changes, the learning process for Layer 2 must be repeated, which is not necessary for Layer 3. For these and other reasons, it is more desirable for cloud IaaS service providers to provide infrastructure that operates at Layer 3 rather than Layer 2.

[0121] However, Layer 2 functionality is required by many on-premises applications despite its numerous drawbacks. For example, consider a virtualized cloud configuration where a customer (customer 1) has two instances, instance A with IP1 and instance B with IP2, in a virtual network "V" where instances can be compute instances (e.g., bare metal, virtual machines, or containers) or service instances such as load balancers, NFS mount points, or other service instances. Virtual network V is a separate address space isolated from other virtual networks and the underlying physical network. This isolation can be achieved using various techniques, including packet encapsulation or NAT. For this reason, the IP addresses of instances in the customer's virtual network are different from the addresses in the physical network where they are hosted. A centralized SDN (Software Defined Networking) control plane is provided that knows the physical IPs and the virtual interfaces of all virtual IP addresses. When a packet is sent from instance A to a destination IP2 in virtual network V, the virtual network SDN stack needs to know where IP2 is located. It must know this in advance so that it can send the packet to the IP in the physical network where the virtual IP address IP2 for V is hosted. The location of a virtual IP address can be modified within the cloud, thus changing the relationship between the physical IP and the virtual IP address. Every time a virtual IP address is moved (for example, moving the IP address associated with a virtual machine to another virtual machine, or migrating a virtual machine to a new physical host), an API call must be made to the SDN control plane to inform the controller that the IP has been moved, so that it can update all participants in the SDN stack, including the packet processor (data plane). However, there is a class of applications that does not make such API calls.Examples include various on-premises applications and applications provided by various virtualization software vendors such as VMware. The value of facilitating virtual Layer 2 networking in a virtualized cloud environment lies in enabling support for applications that are not programmed to make such API calls, or applications that rely on other Layer 2 networking features, such as support for non-IP Layer 3 and MAC learning.

[0122] A virtual Layer 2 network creates a broadcast domain, and learning is performed by the members of the broadcast domain. In a virtual Layer 2 domain, any IP can exist on any MAC on any host within that Layer 2 domain, and the system learns using standard Layer 2 networking protocols. The system virtualizes these networking primitives, and it does not need to be explicitly told by a central controller where the MAC and IP reside within its virtual Layer 2 network. This allows applications requiring low-latency failover, applications that need to support broadcast or multicast protocols to multiple nodes, and legacy applications that do not know how to make API calls to the SDN control plane or API endpoints to determine where IP and MAC addresses are valid to run. Therefore, providing Layer 2 networking capabilities in a virtualized cloud environment is required to support functionality that is not available at the IP Layer 3 level.

[0123] Another technical advantage of providing virtual Layer 2 in a virtualized cloud environment is that it enables support for a variety of different Layer 3 protocols (such as IPv4 and IPv6), including non-IP protocols. For example, it can support various non-IP protocols such as IPX and AppleTalk. Existing cloud IaaS providers do not provide Layer 2 functionality in their virtualized cloud networks, and therefore cannot support these non-IP protocols. By providing Layer 2 networking functionality as described in this disclosure, it is possible to provide support for applications that require and depend on the availability of Layer 3 protocols and Layer 2 level functionality.

[0124] Using the technologies described in this disclosure, both Layer 3 and Layer 2 functionality is provided in a virtualized cloud infrastructure. As previously stated, Layer 3-based networking provides certain efficiencies not provided by Layer 2 networking, particularly efficiencies that are well-suited for scaling. By providing Layer 2 functionality in addition to Layer 3 functionality, it becomes possible to leverage such efficiencies provided by Layer 3 (for example, to provide a more scalable solution) while providing Layer 2 functionality in a more scalable manner. For example, virtualized Layer 3 avoids the need to use broadcasts for learning purposes. By providing Layer 3 for its efficiency, and simultaneously providing virtualized Layer 2 to enable applications that require it, applications that cannot function without Layer 2 functionality, and to support non-IP protocols, etc., customers are provided with complete flexibility in a virtualized cloud environment.

[0125] Customers themselves have hybrid environments where Layer 2 and Layer 3 environments coexist, and virtualized cloud environments can now support both of these environments. Customers can have Layer 3 networks such as subnets and / or Layer 2 networks such as VLANs, and these two environments can interact with each other within the virtualized cloud environment.

[0126] Virtualized cloud environments also need to support multi-tenancy. Multi-tenancy makes provisioning both Layer 3 and Layer 2 functionalities within the same virtualized cloud environment technically difficult and complex. For example, a Layer 2 broadcast domain must be managed across many different customers within the cloud provider's infrastructure. The embodiments described in this disclosure overcome these technical challenges.

[0127] For virtualization providers (e.g., VMware), a virtualized Layer 2 network that emulates a physical Layer 2 network allows workloads to run without modification. Applications provided by such a virtualization provider can then run on the virtualized Layer 2 network provided by the cloud infrastructure. For example, such an application might include a set of instances that need to run on a Layer 2 network. If a customer wants to lift and shift such applications from their on-premises environment to a virtualized cloud environment, they cannot simply import the applications and run them in the cloud because these applications rely on an underlying Layer 2 network that is not provided by the current virtualized cloud provider (for example, Layer 2 networking capabilities are used to perform virtual machine migration or move where MAC and IP addresses are valid). For these reasons, such applications cannot run natively in a virtualized cloud environment. Using the techniques described here, a cloud provider can provide a virtualized Layer 2 network in addition to a virtualized Layer 3 network. Here, such an application stack can run without modification in the cloud environment and can perform nested virtualization within the cloud environment. Customers can now run and manage their own Layer 2 applications within the cloud. Application providers do not need to make any changes to their own software to facilitate this. Such legacy applications or workloads (e.g., legacy load balancers, legacy applications, KVM, OpenStack, clustering software) can now run unchanged in a virtualized cloud environment.

[0128] By providing virtualized Layer 2 functionality as described here, various Layer 3 protocols, including non-IP protocols, can now be supported by virtualized cloud environments. Taking Ethernet as an example, it can support various different EtherTypes (fields in the Layer 2 header that indicate what type of Layer 3 packet is being sent; what protocols should be expected at Layer 3) including various non-IP protocols. EtherTypes are two-octet fields within an Ethernet® frame. They are used by the data link layer at the receiving end to determine which protocols are encapsulated in the frame's payload and how the payload will be processed. EtherTypes are also used as the basis for 802.1Q VLAN tagging, encapsulating packets from a VLAN for transmission that is multiplexed with other VLAN traffic over an Ethernet trunk. Examples of EtherTypes include IPv4, IPv6, Address Resolution Protocol (ARP), AppleTalk, and IPX. Cloud networks that support Layer 2 protocols can support any protocol at the Layer 3 layer. Similarly, when cloud infrastructure provides support for Layer 3 protocols, it can support various Layer 4 protocols such as TCP, UDP, and ICMP. A network can be agnostic to Layer 4 protocols when virtualization is provided at Layer 3. Similarly, a network can be agnostic to Layer 3 protocols when virtualization is provided at Layer 2. This technology can be extended to support any Layer 2 network type, including FDDI and InfiniBand.

[0129] Therefore, many applications written for physical networks, particularly those operating with clusters of computer nodes sharing a broadcast domain, utilize Layer 2 features that are not supported in L3 virtual networks. The following six examples highlight the complexities that can arise from the lack of Layer 2 networking capabilities: (1) MAC and IP assignment without prior API calls. Network appliances and hypervisors (such as VMware) were not built for cloud virtual networks. They assume that they can use MACs as long as the MAC is unique, and can obtain dynamic addresses from a DHCP server or use any IP assigned to the cluster. Often there is no mechanism that they can be configured to notify the control plane about the assignment of these Layer 2 and Layer 3 addresses. If the MAC and IP are unknown, the Layer 3 virtual network does not know where to send the traffic. (2) Low-latency reallocation of MAC and IP for high availability and live migration. Many on-premises applications use ARP to reallocate IPs and MACs for high availability - when an instance in a cluster or HA pair stops responding, a newly active instance sends a Gratuitous ARP (GARP) to reallocate the service IP to its MAC, or sends a Reverse ARP (RARP) to reallocate the service MAC to its interface. This is also important when live migrating instances on a hypervisor: the new host must send a RARP when the guest moves so that guest traffic is sent to the new host. The reallocation must not only be done without API calls, but also with very low latency (sub-milliseconds). This cannot be achieved with HTTPS calls to REST endpoints. (3) Interface multiplexing by MAC address. When a hypervisor hosts multiple virtual machines on a single host, all of which are on the same network, guest interfaces are distinguished by their MAC addresses. This requires support for multiple MAC addresses on the same virtual interface. (4) VLAN support. A single physical virtual machine host may need to be on multiple broadcast domains, as indicated by the use of VLAN tags. For example, VMware ESX uses VLANs for traffic isolation (for example, a guest virtual machine may communicate on one VLAN, storage on another VLAN, and the host virtual machine on yet another VLAN). (5) Use of broadcast and multicast traffic. ARP requires L2 broadcasting, and there are examples of on-premises applications that use broadcast and multicast traffic for cluster and HA applications. (6) Support for non-IP traffic. Since L3 networks require IPv4 or IPv6 headers to communicate, the use of L3 protocols other than IP would not work. L2 virtualization means that networks within a VLAN can be L3 protocol independent—the L3 header could be IPv4, IPv6, IPX, or something else—or even nonexistent.

[0130] Example of Layer 2 VLAN implementation As disclosed herein, a Layer 2 (L2) network can be created within a cloud network. This virtual L2 network includes one or more Layer 2 virtual networks, such as virtualized L2 VLANs, which are referred to here as VLANs. Each VLAN may contain multiple compute instances, each of which may be associated with at least one L2 virtual network interface (e.g., an L2 VNIC) and an L2 virtual switch. In some embodiments, each pair of L2 virtual network interfaces and L2 virtual switches is hosted on an NVD. The NVD may host multiple such pairs, each pair being associated with a different compute instance. A collection of L2 virtual switches represents a single L2 switch emulated by a VLAN. An L2 virtual network interface represents a collection of L2 ports on a single L2 switch emulated by an L2 switch. VLANs can connect to other VLANs, Layer 3 (L3) networks, on-premises networks, and / or other networks via a VLAN Switching and Routing Service (VSRS), also referred here to as a Reality Virtual Router (RVR) or L2 VSRS. An example of this architecture is described below.

[0131] Referring here to Figure 6, a schematic diagram of one embodiment of a computing network is shown. VCN602 resides in CSPI601. VCN602 includes multiple gateways that connect VCN602 to other networks. These gateways include DRG604, which can connect VCN602 to an on-premises network, such as an on-premises data center 606. The gateways may further include gateway 600, which may include an LPG for connecting VCN602 to another VCN, and / or an IGW and / or NAT gateway for connecting VCN602 to the Internet. The gateways of VCN602 may further include a service gateway 610, which can connect VCN602 to a service network 612. The service network 612 may include one or more databases and / or stores, such as an autonomous database 614 and / or an object store 616. The service network may include a conceptual network that includes an aggregation of IP ranges, which may be, for example, a public IP range. In some embodiments, these IP ranges may cover some or all of the public services provided by the CSPI601 provider. These services can be accessed, for example, via an internet gateway or a NAT gateway. In some embodiments, the service network provides a way for services within the service network to be accessed from the local area through a dedicated gateway (service gateway) for that purpose. In some embodiments, the backend of these services can be implemented, for example, in their own private network. In some embodiments, the service network 612 may include further additional databases.

[0132] VCN602 can contain multiple virtual networks. Each of these networks can contain one or more compute instances, and one or more compute instances can communicate within their respective networks, between networks, or outside of VCN602. One of the virtual networks in VCN602 is L3 subnet 620. L3 subnet 620 is a unit or subdivision of configuration created within VCN602. Subnet 620 can contain a virtual Layer 3 network in the virtualized cloud environment of VCN602, and VCN602 is hosted on the underlying physical network of CPSI601. Figure 6 shows a single subnet 620, but VCN602 can have one or more subnets. Each subnet within VCN602 can be associated with a contiguous range of overlay IP addresses (e.g., 10.0.0.0 / 24 and 10.0.1.0 / 24) that does not overlap with other subnets within that VCN and represents a subset of the address space within the VCN's address space. In some embodiments, this IP address space can be isolated from the address space associated with CPSI601.

[0133] Subnet 620 contains one or more compute instances, specifically a first compute instance 622-A and a second compute instance 622-B. Compute instances 622-A and 622-B can communicate with each other within subnet 620, or with other instances, devices, and / or networks outside subnet 620. Communication outside subnet 620 is enabled by a virtual router (VR) 624. VR624 enables communication between subnet 620 and other networks in VCN602. For subnet 620, VR624 represents a logical gateway that enables subnet 620 (e.g., compute instances 622-A and 622-B) to communicate with endpoints on other networks within VCN602, and with other endpoints outside VCN602.

[0134] VCN602 may further include additional networks, specifically one or more L2 VLANs (referred to here as VLANs), which are examples of virtual L2 networks. Each of these VLANs may include a virtual Layer 2 network, localized to the cloud environment of VCN602 and / or hosted by the underlying physical network of CPSI601. In the embodiment of Figure 6, VCN602 includes VLAN A630 and VLAN B640. Each VLAN 630, 640 within VCN602 may be associated with a contiguous range of overlay IP addresses (e.g., 10.0.0.0 / 24 and 10.0.1.0 / 24) that do not overlap with other networks within that VCN, such as other subnets or VLANs within that VCN, and represent a subset of the address space within the VCN's address space. In some embodiments, this IP address space of the VLANs may be isolated from the address space associated with CPSI601. Each of VLANs 630 and 640 may contain one or more compute instances. Specifically, VLAN A630 may contain, for example, a first compute instance 632-A and a second compute instance 632-B. In some embodiments, VLAN A630 may contain additional compute instances. VLAN B640 may contain, for example, a first compute instance 642-A and a second compute instance 642-B. Each of compute instances 632-A, 632-B, 642-A, and 642-B may have an IP address and a MAC address. These addresses may be assigned or generated in any desired manner. In some embodiments, these addresses may be within the VLAN's CIDR, and in some embodiments, these addresses may be arbitrary addresses. In embodiments where a compute instance of a VLAN communicates with an endpoint outside the VLAN, one or both of these addresses may be from the VLAN CIDR; however, if all communication is within the VLAN, these addresses are not limited to addresses within the VLAN CIDR.In contrast to networks where addresses are assigned by the control plane, the IP and / or MAC addresses of compute instances within a VLAN may be assigned by the users / customers of that VLAN, and these IP and / or MAC addresses may then be discovered and / or learned by compute instances within the VLAN according to the learning process discussed below.

[0135] Each VLAN can include a VLAN Switching and Routing Service (VSRS); specifically, VLAN A630 includes VSRS A634, and VLAN B640 includes VSRS B644. Each VSRS634, 644 participates in Layer 2 switching and local learning within the VLAN, and also performs all necessary Layer 3 network functions, including ARP, NDP, and routing. VSRS performs ARP (which is a Layer 2 protocol) because it must map IP to MAC addresses.

[0136] In these cloud-based VLANs, each virtual interface or virtual gateway can be associated with one or more media access control (MAC) addresses, which may be virtual MAC addresses. Within the VLAN, one or more compute instances 632-A, 632-B, 642-A, 642-B, and / or one or more service instances, which may be bare metal, VMs, or containers, can communicate directly with each other via a virtual switch. External communication with other VLANs or L3 networks is enabled via VSRS634,644. VSRS634,644 is a distributed service that provides Layer 3 functionality, such as IP routing, to the VLAN network. In some embodiments, VSRS634,644 is a horizontally scalable, highly available routing service located at the intersection of IP and L2 networks, and capable of participating in IP routing and L2 learning within a cloud-based L2 domain.

[0137] VSRS634,644 can be distributed across multiple nodes within the infrastructure, and the VSRS634,644 functionality can be scalable, specifically horizontally scalable. In some embodiments, each node implementing the VSRS634,644 functionality shares and replicates router and / or switch functionality with one another. Furthermore, these nodes can present themselves as a single VSRS634,644 to all instances within VLAN630,640. VSRS634,644 can be implemented on any virtualization device within CSPI601, specifically within a virtual network. Therefore, in some embodiments, VSRS634,644 can be implemented on any virtual network virtualization device, including NICs, SmartNICs, switches, Smart switches, or general-purpose computing hosts.

[0138] VSRS634,644 can be a service residing on one or more hardware nodes, such as one or more x86 servers or one or more networking devices, specifically one or more SmartNICs, that support a cloud network. In some embodiments, VSRS634,644 can be implemented on a server fleet. Thus, VSRS634,644 can be a service distributed across a fleet of nodes, which may be a centrally managed fleet or distributed to the edge, that participates in and shares L2 and L3 learning along with evaluating routing and security policies. In some embodiments, each VSRS instance can update other VSRS instances with new mapping information learned by the VSRS instance. For example, if a VSRS instance learns IP, interface, and / or MAC mappings for one or more CIs in its VLAN, the VSRS instance can provide its updated information to other VSRS instances in the VCN. Through this cross-update, a VSRS instance associated with a first VLAN can know the mappings, including IP, interface, and / or MAC mappings, for CIs in other VLANs, and in some embodiments, it can know the mappings, including IP, interface, and / or MAC mappings, for CIs in other VLANs within VCN602. When VSRS resides on a server fleet and / or is distributed across a fleet of nodes, these updates can be greatly accelerated.

[0139] In some embodiments, VSRS634, 644 may also host one or more higher-level services necessary for networking, including but not limited to DHCP relay; DHCP (hosting); DHCPv6; IPv6 neighbor discovery protocols such as the IPv6 neighbor discovery protocol; DNS; hosting DNSv6; SLAAC for IPv6; NTP; metadata services; and block store mount points. In some embodiments, VSRS may support one or more network address translation (NAT) functions for translating between multiple network address spaces. In some embodiments, VSRS may incorporate anti-spoofing, anti-MAC spoofing, ARP cache poisoning countermeasures for IPv4, IPv6 router advertisement (RA) guard, DHCP guard, packet filtering with access control lists (ACLs); and / or reverse route forwarding checks. VSRS may implement functions including, for example, ARP, GARP, packet filtering (ACLs), DHCP relay, and / or IP routing protocols. VSRS634 and 644 can, for example, learn MAC addresses, invalidate expired MAC addresses, handle MAC address migration, look up MAC address information, handle MAC information flooding, handle storm control, prevent loops, perform Layer 2 multicast via protocols such as IGMP in the cloud, collect statistics including logs, use SNMP for statistics and monitoring, and / or collect and use statistics about broadcast, total traffic, bits, spanning tree packets, etc.

[0140] Within a virtual network, VSRS634,644 may appear as different instantiations. In some embodiments, each of these instantiations of VSRS may be associated with VLAN630,640, and in some embodiments, each VLAN630,640 may have an instantiation of VSRS634,644. In some embodiments, each instantiation of VSRS634,644 may have one or more unique tables corresponding to the VLAN630,640 to which the VSRS634,644 instantiation is associated. Each instantiation of VSRS634,644 may generate and / or curate unique tables associated with that instantiation of VSRS634,644. Therefore, while a single service may provide VSRS634,644 functionality for one or more cloud networks, individual instances of VSRS634,644 within a cloud network may have their own Layer 2 and Layer 3 forwarding tables, while multiple such customer networks may have overlapping Layer 2 and Layer 3 forwarding tables.

[0141] In some embodiments, VSRS634,644 may support competing VLANs and IP spaces across multiple tenants. This may include having multiple tenants on the same VSRS634,644. In some embodiments, some or all of these tenants may choose to use some or all of the same IP address space, the same MAC space, and the same VLAN space. This can provide users with extreme flexibility when choosing addresses. In some embodiments, this multi-tenancy is supported by providing each tenant with a separate virtual network, which is a private network within the cloud network. Each virtual network is given a unique identifier. Similarly, in some embodiments, each host may have a unique identifier, and / or each virtual interface or virtual gateway may have a unique identifier. In some embodiments, these unique identifiers, specifically the unique identifiers of the virtual network for each tenant, may be encoded in each communication. By providing each virtual network with a unique identifier and including it in the communication, a single instantiation of VSRS634,644 can accommodate multiple tenants with overlapping addresses and / or namespaces.

[0142] VSRS634 and 644 can perform switching and / or routing functions to facilitate and / or enable the creation of and / or communication with L2 networks within VLANs 630 and 640. These VLANs 630 and 640 may be found within a cloud computing environment, more specifically within a virtual network in that cloud computing environment.

[0143] For example, each of VLANs 630 and 640 includes multiple compute instances 632-A, 632-B, 642-A, and 642-B. VSRS634 and 644 enable communication between compute instances in one VLAN 630 or 640 and compute instances in another VLAN 630 or 640, or subnet 620. In some embodiments, VSRS634 and 644 enable communication between compute instances within one VLAN 630 or 640 and another VCN, another network outside the VCN including the internet, an on-premises data center, etc. In such embodiments, for example, compute instance 632-A can send communications to an endpoint outside the VLAN, in this example VLAN A630. A compute instance (632-A) can send communications to VSRS A634, which can then direct those communications to routers 624, 644, or gateways 604, 608, 610 that are communicatively coupled to the desired endpoint. Routers 624, 644, or gateways 604, 608, 610 that are communicatively coupled to the desired endpoint can receive communications from the compute instance (632-A) and direct those communications to the desired endpoint.

[0144] Referring here to Figure 7, a schematic diagram of the logical and hardware aspects of VLAN 700 is shown. As can be understood, VLAN 700 includes multiple endpoints, specifically multiple compute instances and VSRSs. Multiple compute instances (CIs) are instantiated on one or more host machines. In some embodiments, this can be a one-to-one relationship, where each CI is instantiated on its own host machine, and / or in some embodiments, this can be a many-to-one relationship, where multiple CIs are instantiated on a single common host machine. In various embodiments, CIs can be Layer 2 CIs by being configured to communicate with each other using an L2 protocol. Figure 7 illustrates a scenario where several CIs are instantiated on their own host machines and several CIs share a common host machine. As shown in Figure 7, instance 1 (CI1) 704-A is instantiated on host machine 1 702-A, instance 2 (CI2) 704-B is instantiated on host machine 2 702-B, and instance 3 (CI3) 704-C and instance 4 (CI4) 704-D are instantiated on a common host machine 702-C.

[0145] Each of the CI704-A, 704-B, 704-C, and 704-D is coupled to communicate with other CI704-A, 704-B, 704-C, 704-D, and VSRS714 within VLAN 700. Specifically, each of the CI704-A, 704-B, 704-C, and 704-D is connected to other CI704-A, 704-B, 704-C, 704-D, and VSRS714 within VLAN 700 via L2 VNICs and switches. Each CI704-A, 704-B, 704-C, and 704-D is associated with its own L2 VNIC and switch. The switch may be a local, L2 virtual switch that is specifically associated with and deployed for the L2 VNIC. Specifically, CI1 704-A is associated with L2 VNIC1 708-A and switch 1 710-A; CI2 704-B is associated with L2 VNIC2 708-B and switch 710-B; CI3 704-C is associated with L2 VNIC3 708-C and switch 3 710-C; and CI4 704-D is associated with L2 VNIC4 708-D and switch 4 710-D.

[0146] In some embodiments, each L2 VNIC 708 and its associated switch 710 may be instantiated on an NVD 706. This instantiation can be one-to-one, with a single L2 VNIC 708 and its associated switch 710 being instantiated on a specific NVD 706, or it can be many-to-one, with multiple L2 VNIC 708s and their associated switches 710 being instantiated on a single common NVD 706. Specifically, L2 VNIC1 708-A and switch 1 710-A are instantiated on NVD1 706-A, L2 VNIC2 708-B and switch 2 710-B are instantiated on NVD2, and both L2 VNIC3 708-C and switch 3 710-C, as well as L2 VNIC4 708-D and switch 710-D, are instantiated on a common NVD, i.e., NVD 706-C.

[0147] In some embodiments, the VSRS714 can support competing VLANs and IP spaces across multiple tenants. This may include having multiple tenants on the same VSRS714. In some embodiments, some or all of these tenants may choose to use some or all of the same IP address space, the same MAC space, and the same VLAN space. This may provide users with extreme flexibility when choosing addresses. In some embodiments, this multi-tenancy is supported by providing each tenant with a separate virtual network, which is a private network within the cloud network. Each virtual network (e.g., each VLAN or VCN) is given a unique identifier, such as a VCN identifier which may be a VLAN identifier. This unique identifier may be selected, for example, by the control plane, specifically by the CSPI control plane. In some embodiments, this unique VLAN identifier may include one or more bits which may be included and / or used in packet encapsulation. Similarly, in some embodiments, each host may have a unique identifier, and / or each virtual interface or virtual gateway may have a unique identifier. In some embodiments, these unique identifiers, specifically the unique identifiers of the virtual network for a tenant, can be encoded in each communication. By providing a unique identifier for each virtual network and including it in the communication, a single instantiation of VSRS can accommodate multiple tenants with overlapping addresses and / or namespaces. In some embodiments, VSRS714 can determine which tenant a packet belongs to based on the VCN identifier and / or VLAN identifier associated with the communication, specifically in the VCN header of the communication. In the embodiments disclosed herein, communications entering and leaving a VLAN may have a VCN header that can include a VLAN identifier.Based on the VCN header containing the VLAN identifier, the VSRS714 can determine the tenancy; in other words, the receiving VSRS can determine which VLAN and / or tenant to send communications to. In addition, each compute instance belonging to a VLAN (e.g., an L2 compute instance) is given a unique interface identifier that identifies the L2 VNIC associated with that compute instance. The interface identifier may be included in traffic from and to the compute instance (e.g., by being included in the frame header) and can be used by the NVD to identify the L2 VNIC associated with the compute instance. In other words, the interface identifier can uniquely identify a compute instance and / or its associated L2 VNIC.

[0148] As shown in Figure 7, switches 710-A, 710-B, 710-C, and 710-D together can form an L2 distributed switch 712, also referred to here as the distributed switch 712. From the customer's perspective, each switch 710-A, 710-B, 710-C, and 710-D within the L2 distributed switch 712 is a single switch connecting to all CIs within the VLAN. However, the L2 distributed switch 712, which emulates the user experience of a single switch, is infinitely scalable and includes a set of local switches (for example, switches 710-A, 710-B, 710-C, and 710-D in the exemplary example in Figure 7). As shown in Figure 7, each CI runs on a host machine connected to the NVD. For each CI on a host connected to the NVD, the NVD hosts a Layer 2 VNIC and a local switch associated with the compute instance (for example, an L2 virtual switch that is local to the NVD, associated with the Layer 2 VNIC, and is a member or component of an L2 distributed switch 712). The Layer 2 VNIC represents the port of the compute instance on the Layer 2 VLAN. The local switch connects the VNIC to other VNICs (for example, other ports) associated with other compute instances on the Layer 2 VLAN.

[0149] Each of the CI704-A, 704-B, 704-C, and 704-D can communicate with other CI704-A, 704-B, 704-C, and 704-D within VLAN 700, or with a VSRS714. One of the CI704-A, 704-B, 704-C, and 704-D transmits a frame to another CI704-A, 704-B, 704-C, or VSRS714 by sending the frame to the MAC address and interface identifier of the receiving CI or VSRS714. The MAC address and interface identifier may be included in the frame header. Here, as described above, the interface identifier may indicate the L2 VNIC of the receiving CI or VSRS714.

[0150] In one embodiment, CI1 704-A may be a source CI, L2 VNIC708-A may be a source L2 VNIC, and switch 710-A may be a source L2 virtual switch. In this embodiment, CI3 704-C may be a destination CI, and L2 VNIC3 708-C may be a destination L2 VNIC. The source CI can send a frame along with the source MAC address and destination MAC address. This frame may be intercepted by NVD706-A, which instantiates the source VNIC and source switch.

[0151] L2 VNICs 708-A, 708-B, 708-C, and 708-D can each learn to map MAC addresses to L2 VNIC interface identifiers for VLAN 700. This mapping can be learned based on frames and / or communications received from within VLAN 700. Based on this previously determined mapping, the source VNIC can determine the interface identifier of the destination interface associated with the destination CI within the VLAN and encapsulate the frame. In some embodiments, this encapsulation may include Geneve encapsulation, specifically L2 Geneve encapsulation which includes an L2 (Ethernet®) header of the frame being encapsulated. The encapsulated frame can identify the destination MAC, destination interface identifier, source MAC, and source interface identifier.

[0152] The source VNIC can pass the encapsulated frame to the source switch, which can then direct the frame to the destination VNIC. Upon receiving the frame, the destination VNIC can deencapsulate it and then provide the frame to the destination CI.

[0153] Referring now to Figure 8, a logical schematic diagram of multiple connected L2 VLANs 800 is shown. In the particular embodiment shown in Figure 8, both VLANs reside in the same VCN. As can be understood, the multiple connected L2 VLANs 800 can include a first VLAN, VLAN A802-A, and a second VLAN, VLAN B802-B. Each of these VLANs 802-A and 802-B can contain one or more CIs, each of which can have an associated L2 VNIC and an associated L2 virtual switch. Furthermore, each of these VLANs 802-A and 802-B can contain a VSRS.

[0154] Specifically, VLAN A802-A can include instance 1 804-A connected to L2 VNIC1 806-A and switch 1 808-A, instance 2 804-B connected to L2 VNIC2 806-B and switch 808-B, and instance 3 804-C connected to L2 VNIC3 806-C and switch 3 808-C. VLAN B 802-B can include instance 4 804-D connected to L2 VNIC4 806-D and switch 4 808-D, instance 5 804-E connected to L2 VNIC5 806-E and switch 808-E, and instance 6 804-F connected to L2 VNIC6 806-F and switch 3 808-F. VLAN A802-A may further include VSRS A810-A, and VLAN B802-B may include VSRS B810-B. Each of CI804-A, 804-B, and 804-C of VLAN A802-A can be communicatively coupled to VSRS A810-A, and each of CI804-D, 804-E, and 804-F of VLAN B802-B can be communicatively coupled to VSRS B810-B.

[0155] VLAN A802-A can be communicably coupled to VLAN B802-B via their respective VSRS810-A and 810-B. Each VSRS can similarly be coupled to gateway 812, which can provide access to CI804-A, 804-B, 804-C, 804-D, 804-E, and 804-F within each VLAN802-A and 802-B to other networks outside the VCN where VLAN802-A and 802-B are located. In some embodiments, these networks may include, for example, one or more on-premises networks, another VCN, a service network, or a public network such as the Internet.

[0156] Each of the CIs 804-A, 804-B, and 804-C in VLAN A802-A can communicate with CIs 804-D, 804-E, and 804-F in VLAN B802-B via the VSRS 810-A and 810-B of each VLAN 802-A and 802-B. For example, one of the CIs 804-A, 804-B, 804-C, 804-D, 804-E, and 804-F in one of VLANs 802-A and 802-B can send a frame to CIs 804-A, 804-B, 804-C, 804-D, 804-E, and 804-F in the other VLAN 802-A and 802-B. This frame can leave the source VLAN via the VSRS of the source VLAN, enter the destination VLAN, and be routed to the destination CI via the destination VSRS.

[0157] In one embodiment, CI1 804-A may be a source CI, VNIC806-A may be a source VNIC, and switch 808-A may be a source switch. In this embodiment, CI 5 804-E may be a destination CI, and L2 VNIC5 806-E may be a destination VNIC. VSRS A810-A may be a source VSRS identified as an SVSRS, and VSRS B810-B may be a destination VSRS identified as a DVSRS.

[0158] The source CI can send a frame along with its MAC address. This frame may be intercepted by the NVD that instantiates the source VNIC and source switch. The source VNIC encapsulates the frame. In some embodiments, this encapsulation may include Geneve encapsulation, specifically L2 Geneve encapsulation. The encapsulated frame can identify the destination address of the destination CI. In some embodiments, this destination address may also include the destination address of the destination VSRS. The destination address of the destination CI may include the destination IP address, the destination MAC address of the destination CI, and / or the destination interface identifier of the destination VNIC associated with the destination CI. The destination address of the destination VSRS may include the IP address of the destination VSRS, the interface identifier of the destination VNIC associated with the destination VSRS, and / or the MAC address of the destination VSRS.

[0159] The source VSRS can receive the frame from the source switch, look up the VNIC mapping from the frame's destination address (which may be a destination IP address), and forward the packet to the destination VSRS. The destination VSRS can receive the frame. Based on the destination address contained in the frame, the destination VSRS can forward the frame to the destination VNIC. The destination VNIC can receive the frame, decapsulate it, and then deliver the frame to the destination CI.

[0160] Referring now to Figure 9, a logical schematic diagram of multiple connected L2 VLANs and subnets 900 is shown. In the particular embodiment shown in Figure 9, both the VLANs and subnets reside in the same VCN. This is shown because the virtual routers and VSRSs of both the VLANs and subnets are directly connected rather than connected via a gateway.

[0161] To be understood, this can include a first VLAN, VLAN A902-A, a second VLAN, VLAN B902-B, and subnet 930. Each of these VLANs 902-A and 902-B can contain one or more CIs, each of which can have an associated L2 VNIC and an associated L2 switch. Furthermore, each of these VLANs 902-A and 902-B can contain a VSRS. Similarly, subnet 930, which can be an L3 subnet, can contain one or more CIs, each of which can have an associated L3 VNIC, and L3 subnet 930 can contain a virtual router 916.

[0162] Specifically, VLAN A902-A can include instance 1 904-A connected to L2 VNIC1 906-A and switch 1 908-A, instance 2 904-B connected to L2 VNIC2 906-B and switch 2 908-B, and instance 3 904-C connected to L2 VNIC3 906-C and switch 3 908-C. VLAN B 902-B can include instance 4 904-D connected to L2 VNIC4 906-D and switch 4 908-D, instance 5 904-E connected to L2 VNIC5 906-E and switch 5 908-E, and instance 6 904-F connected to L2 VNIC6 906-F and switch 6 908-F. VLAN A902-A may further include VSRS A910-A, and VLAN B902-B may include VSRS B910-B. Each of CI904-A, 904-B, and 904-C of VLAN A902-A can be communicatively coupled to VSRS A910-A, and each of CI904-D, 904-E, and 904-F of VLAN B902-B can be communicatively coupled to VSRS B910-B. L3 subnet 930 may include one or more CIs, specifically instance 7 904-G which is communicatively coupled to L3 VNIC 7 906-G. L3 subnet 930 may include virtual router 916.

[0163] VLAN A902-A can be communicably coupled to VLAN B902-B via their respective VSRS instances 910-A and 910-B. L3 subnet 930 can be communicably coupled to VLAN A902-A and VLAN B902-B via virtual router 916. Virtual router 916 and each of VSRS instances 910-A and 910-B can similarly be coupled to gateway 912, which can grant access to CIs 904-A, 904-B, 904-C, 904-D, 904-E, 904-F, and 904-G in each VLAN 902-A, 902-B and subnet 930 to other networks outside the VCN where VLAN 902-A, 902-B and subnet 930 are located. In some embodiments, these networks may include, for example, one or more on-premises networks, another VCN, a service network, a public network such as the Internet, etc.

[0164] Each VSRS instance 910-A, 910-B can provide an outgoing path for frames leaving their associated VLANs 902-A, 902-B, and an inbound path for frames entering their associated VLANs 902-A, 902-B. From VSRS instances 910-A, 910-B of VLANs 902-A, 902-B, frames can be sent to any desired endpoint, including L2 endpoints such as L2 CIs in another VLAN on the same VCN or a different VCN or network, and / or L3 endpoints such as L3 CIs in a subnet on the same VCN or a different VCN or network.

[0165] In one embodiment, CI1 904-A may be a source CI, VNIC906-A may be a source VNIC, and switch 908-A may be a source switch. In this embodiment, CI7 904-G may be a destination CI, and VNIC7 906-G may be a destination VNIC. VSRS A910-A may be a source VSRS identified as an SVSRS, and virtual router (VR) 916 may be a destination VR.

[0166] The source CI can send a frame along with its MAC address. This frame may be intercepted by the NVD that instantiates the source VNIC and source switch. The source VNIC encapsulates the frame. In some embodiments, this encapsulation may include Geneve encapsulation, specifically L2 Geneve encapsulation. The encapsulated frame can identify the destination address of the destination CI. In some embodiments, this destination address may also include the destination address of the VSRS of the source CI's VLAN. The destination address of the destination CI may include the destination IP address, the destination MAC address of the destination CI, and / or the destination interface identifier of the destination VNIC of the destination CI.

[0167] The source VSRS can receive the frame from the source switch, look up the VNIC mapping from the frame's destination address (which may be a destination IP address), and forward the frame to the destination VR. The destination VR can receive the frame. Based on the destination address contained in the frame, the destination VR can forward the frame to the destination VNIC. The destination VNIC can receive the frame, decapsulate it, and then deliver the frame to the destination CI.

[0168] Learning by L2 VNICs and / or L2 virtual switches within a virtual L2 network Referring now to Figure 10, a schematic diagram of one embodiment of intra-VLAN communication and learning within VLAN 1000 is shown. The learning here is specific to how the L2 VNIC, the VSRS of the source CI's VLAN, and / or L2 virtual switch learn the association between MAC addresses and the L2 VNIC / / VSRS VNIC (more specifically, between MAC addresses associated with an L2 compute instance or VSRS and interface identifiers associated with the L2 VNICs of those L2 compute instances associated with the VSRS VNIC). Generally, the learning is based on incoming traffic. This learning is different from the learning process (e.g., the ARP process) that an L2 compute instance may implement to learn a destination MAC address in terms of interface-to-MAC address learning. The two learning processes (e.g., L2 VNIC / L2 virtual switch and L2 compute instance) are shown as being implemented jointly in Figure 12.

[0169] As can be understood, VLAN1000 includes compute instance 1 1000-A, which is communicatively coupled to NVD1 1001-A, which instantiates L2 VNIC1 1002-A and L2 switch 1 1004-A. VLAN1000 also includes compute instance 2 1000-B, which is communicatively coupled to NVD2 1001-B, which instantiates L2 VNIC2 1002-B and L2 switch 2 1004-A. VLAN1000 also runs on a server fleet and includes VSRS1015, which includes VSRS VNIC1002-C and VSRS switch 1004-C. All switches 1004-A, 1004-B, and 1004-C together form L2 distributed switch 1050. VSRS1015 is communicatively coupled to endpoint 1008, which may include a gateway, specifically an L2 / L3 router in the form of another VSRS, or an L3 router in the form of a virtual router.

[0170] The control plane 1010 of the VCN hosting VLAN 1000 maintains information identifying each L2 VNIC on VLAN 1000 and the network configuration of each L2 VNIC. For example, this information may include, for each L2 VNIC, the interface identifier associated with the L2 VNIC and / or the physical IP address of the NVD hosting the L2 VNIC. The control plane 1010 updates the interfaces in VLAN 1000 with this information (e.g., periodically or on demand). Thus, each L2 VNIC in VLAN 1000, 1002-A, 1002-B, 1002-C, receives information from the control plane 1010, identifies the interface in the VLAN, and populates this information into a table. The table populated by the L2 VNICs can be stored locally on the NVD hosting the L2 VNICs. If VNIC1002-A, 1002-B, and 1002-C already contain the current table, VNIC1002-A, 1002-B, and 1002-C can determine any discrepancies between their current table and the information / table received from control plane 1010. In some embodiments, VNIC1002-A, 1002-B, and 1002-C can update their table to match the information received from control plane 1010.

[0171] As shown in Figure 10, frames are transmitted via L2 switches 1004-A, 1004-B, and 1004-C and received by receiving VNICs 1002-A, 1002-B, and 1002-C. When frames are received by VNICs 1002-A, 1002-B, and 1002-C, the VNIC learns the mapping of the source interface (source VNIC) and source MAC address of that frame. Based on the table of information received from the control plane 1010, the VNIC can map the source MAC address (from the received frame) to the interface identifier of the source VNIC and the IP address of that VNIC and / or the IP address of the NVD hosting that VNIC (if the interface identifier and IP address are available from the table). Thus, L2 VNICs 1002-A, 1002-B, and 1002-C learn the mapping of interface identifiers to MAC addresses based on the received communications and / or frames. Each VNIC 1002-A, 1002-B, and 1002-C may have its L2 forwarding (FWD) tables 1006-A, 1006-B, and 1006-C, along with this learned mapping information. In some embodiments, the L2 forwarding table includes a MAC address and associates it with at least one of an interface identifier or a physical IP address. In such embodiments, the MAC address may be an address assigned to an L2 compute instance and may correspond to a port emulated by the L2 VNIC associated with the L2 compute instance. The interface identifier can uniquely identify the L2 VNIC and / or L2 compute instance. The virtual IP address may be that of the L2 VNIC, and the physical IP address may be that of the NVD hosting the L2 VNIC. L2 forwarding updated by the L2 VNIC may be stored locally on the NVD hosting the L2 VNIC and may be used by the L2 virtual switch associated with the L2 VNIC to direct frames.In some embodiments, VNICs within a common VLAN can share all or part of their mapping tables with one another.

[0172] In light of the network architecture described above, the traffic flow is described below. For clarity, the traffic flow is described in relation to compute instance 2 1000-B, L2 VNIC2 10002-B, L2 switch 2 1004-B, and NVD2 1001-B. This description applies equivalently to the traffic flow between other compute instances.

[0173] As described above, VLANs are implemented within a VCN as overlay L2 networks on top of L3 physical networks. An L2 compute instance of a VLAN can send or receive L2 frames that include an overlay MAC address (also called a virtual MAC address) as the source MAC address and destination MAC address. An L2 frame can also encapsulate a packet that includes an overlay IP address (also called a virtual IP address) as the source IP address and destination IP address. In some embodiments, the overlay IP address of a compute instance may belong to the CIDR range of the VLAN. Other overlay IP addresses may belong to the CIDR range (in which case the L2 frame flows within the VLAN) or outside the CIDR range (in which case the L2 frame is destined for or received from another network). An L2 frame may also include a VLAN tag, which can be used to uniquely identify the VLAN and distinguish it from multiple L2 VNICs on the same NVD. L2 frames can be received by an NVD in encapsulated packets via a tunnel from the host machine of a compute instance, from another NVD, or from a server fleet hosting a VSRS. In these different cases, the encapsulated packet may be an L3 packet transmitted over the physical network, with source and destination IP addresses being physical IP addresses. Different types of encapsulation are possible, including Geneve encapsulation. An NVD can decapsulate the received packet to extract the L2 frame. Similarly, to transmit an L2 frame, an NVD can encapsulate it in an L3 packet and transmit it over the physical board.

[0174] For inbound traffic within the VLAN from instance 2 1000-B, NVD2 1001-B receives a frame from the host machine of instance 2 1000-B via the Ethernet link. The frame contains an interface identifier that identifies L2 VNIC2 1000-B. The frame contains the overlay MAC address of instance 2 1000-B (e.g., M.2) as the source MAC address and the overlay MAC address of instance 1 1000-A (e.g., M.1) as the destination MAC address. Given the interface identifier, NVD2 1001-B passes the frame to L2 VNIC2 1002-B for further processing. L2 VNIC2 1002-B forwards the frame to L2 switch 2 1004-B. Based on L2 forwarding table 1006-B, L2 switch 2 1004-B determines whether the destination MAC address is known (for example, by matching it with an entry in L2 forwarding table 1006-B).

[0175] If known, L2 switch 2 1004-B determines that L2 VNIC1 1002-A is the associated tunnel endpoint and forwards the frame to L2 VNIC1 1002-A. Forwarding may involve encapsulation and decapsulation of the frame in the packet (e.g., Geneve encapsulation and decapsulation), and the packet may contain the frame, the physical IP address of NVD1 1001-A as the destination address (e.g., IP.1), and the physical IP address of NVD 2 1001-B as the source address (e.g., IP.2).

[0176] If unknown, L2 switch 2 1004-B broadcasts the frame to various VNICs in the VLAN (e.g., including L2 VNIC 1 1002-A and any other L2 VNICs in the VLAN), and the broadcasted frame is processed (e.g., encapsulated, transmitted, decapsulated) among the relevant NVDs. In some embodiments, this broadcast is performed in the physical network, or more specifically, emulated in the physical network, and the frame can be encapsulated separately to each L2 VNIC, including VSRS in the VLAN. Thus, the broadcast is emulated in the physical network via a series of replicated unicast packets. Each L2 VNIC then receives the frame and learns the association between the interface identifier of L2 VNIC 2 1002-B and the source MAC address (e.g., M.2) and the source physical IP address (e.g., IP.2).

[0177] For incoming traffic within a VLAN from compute instance 1 1000-A to compute instance 2 1000-B, NVD2 1001-B receives the packet from NVD1. The packet has IP.1 as the source address and a frame, and the frame contains M.2 as the destination MAC address and M.1 as the source MAC address. The frame also contains the network identifier of L2 VNIC1 1002-A. Upon decapsulation, VNIC2 receives the frame and learns that this interface identifier is associated with M.1 and / or IP.1, and if this information was previously unknown, stores this learned information in the L2 forwarding table 1006-B on switch 2 for subsequent outgoing traffic. Alternatively, upon decapsulation, L2 VNIC2 1002-B receives the frame and learns that this interface identifier is associated with M.1 and / or IP.1, and if this information is known, refreshes the validity period.

[0178] For outgoing traffic sent from instance 2 1000-B in VLAN 1000 to an instance in another VLAN, a similar flow to the outgoing traffic described above may exist, except that a VSRS VNIC and VSRS switch are used. In particular, the destination MAC address is not within the L2 broadcast of VLAN 1000 (it is in another L2 VLAN). Therefore, the overlay destination IP address of the destination instance (e.g., IP.A) is used for this outgoing traffic. For example, L2 VNIC 2 1002-B determines that IP.A is outside the CIDR range of VLAN 1000. Therefore, L2 VNIC 2 1002-B sets the destination MAC address to the default gateway MAC address (e.g., M.DG). Based on M.DG, L2 switch 2 1004-B sends the outgoing traffic to the VSRS VNIC (e.g., via a tunnel with appropriate end-to-end encapsulation). The VSRS VNIC forwards the outgoing traffic to the VSRS switch. Next, the VSRS switch performs routing functionality, where, based on the overlay destination IP address (e.g., IP.A), the VSRS switch on VLAN 1000 sends the outgoing traffic to the VSRS switch on the other VLAN (e.g., via a virtual router between these two VLANs, with appropriate end-to-end encapsulation). The VSRS switch on the other VLAN then performs switching functionality by determining that IP.A is within the CIDR range of this VLAN, and performs an ARP cache lookup based on IP.A to determine the destination MAC address associated with IP.A. If no match is found in the ARP cache, an ARP request is sent to a different L2 VNIC on the other VLAN to determine the destination MAC address. Otherwise, the VSRS switch sends the outgoing traffic to the relevant VNIC (e.g., via a tunnel, with appropriate encapsulation).

[0179] For incoming traffic from an instance in another VLAN to an instance in VLAN 1000, the traffic flow is the same as above, except that it is in the reverse direction. For outgoing traffic from an instance in VLAN 1000 to the L3 network, the traffic flow is the same as above, except that the VSRS switch in VLAN 1000 directly routes the packet to the destination VNIC in the virtual L3 network via the virtual router (e.g., the packet does not need to be routed through another VSRS switch). For incoming traffic from the virtual L3 network to an instance in VLAN 1000, the traffic flow is the same as above, except that the packet is received by the VSRS switch in VLAN 1000A, which transmits the packet as a frame within the VLAN. For traffic between VLAN 1000 and other networks (outgoing or incoming), the VSRS switch is used similarly, with its routing function used for outgoing traffic to send packets through the appropriate gateway (e.g., IGW, NGW, DRG, SGW, LPG), and its switching function used for incoming traffic to transmit frames within VLAN 1000.

[0180] Referring to Figure 11, a schematic diagram of an embodiment of VLAN 1100 (for example, a cloud-based virtual L2 network) is shown, specifically a diagram of the VLAN implementation.

[0181] As described above, a VLAN can contain "n" compute instances 1102-A, 1102-B, 1102-N, each running on a host machine. As previously stated, there can be one-to-one associations between compute instances and host machines, or many-to-one associations between multiple compute instances and a single host machine. Each compute instance 1102-A, 1102-B, 1102-N can be an L2 compute instance, in which case it is associated with at least one virtual interface (e.g., L2 VNIC) 1104-A, 1104-B, 1104-N and switches 1106-A, 1106-B, 1106-N. Switches 1106-A, 1106-B, 1106-N are L2 virtual switches and together form an L2 distributed switch type 1107.

[0182] Pairs of L2 VNICs 1104-A, 1104-B, 1104-N and switches 1106-A, 1106-B, 1106-N associated with compute instances 1102-A, 1102-B, 1102-N on the host machine are pairs of software modules on NVDs 1108-A, 1108-B, 1108-N connected to the host machine. Each L2 VNIC 1104-A, 1104-B, 1104-N represents an L2 port of a single customer-recognized switch (referred to here as the v-switch). Generally, host machine "i" runs compute instance "i" and is connected to NVD "i". Then NVD "i" runs L2 VNIC "i" and switch "i". L2 VNIC "i" represents L2 port "i" of the v-switch. "i" is a positive integer between 1 and "n". Here, a one-to-one association is described, but other types of associations are also possible. For example, a single NVD can be connected to multiple hosts, each running one or more compute instances belonging to a VLAN. In this case, the NVD hosts multiple pairs of L2 VNICs and switches, each corresponding to one of the compute instances.

[0183] A VLAN can include an instance of VSRS1110. VSRS1110 performs switching and routing functions and includes instances of VSRS VNIC1112 and VSRS switch 1114. VSRS VNIC1112 represents a port on the v-switch that connects the v-switch to other networks via a virtual router. As shown, VSRS1110 can be instantiated on server fleet 1116.

[0184] The control plane 1118 can track information identifying the L2 VNICs 1104-A, 1104-B, and 1104-N and their placement in the VLAN. The control plane 1110 can further provide this information to the L2 interfaces 1104-A, 1104-B, and 1104-N within the VLAN.

[0185] As shown in Figure 11, the VLAN can be a cloud-based virtual L2 network that can be built on top of the physical network 1120. In some embodiments, this physical network 1120 may include NVD1108-A, 1108-B, and 1108-N.

[0186] Generally, a first L2 compute instance in a VLAN (e.g., compute instance 1 1102-A) can communicate with a second compute instance in the VLAN (e.g., compute instance 2 1102-B) using L2 protocols. For example, a frame can be sent between these two L2 compute instances across the VLAN. Nevertheless, the frame can be encapsulated, tunneled, routed, and / or otherwise processed so that it can be sent over the underlying physical network 1120.

[0187] For example, compute instance 1 1102-A sends a frame destined for compute instance 2 1102-B. Depending on the network connections between host machine 1 and NVD1, between NVD1 and physical network 1120, between physical network 1120 and NVD2, and between NVD2 and host machine 2 (e.g., TCP / IP connection, Ethernet® connection, tunneling connection, etc.), different types of processing may be applied to the frame. For example, the frame is received and encapsulated by NVD1 until it reaches compute instance 2, and so on. This processing is assumed to be possible so that the frame can be transmitted between lower-layer physical resources, and for brevity and clarity, its explanation is omitted from the explanation of VLANs and related L2 operations.

[0188] Virtual L2 network communication Multiple forms of communication can occur within or using a virtual L2 network. These may include intra-VLAN communication. In such embodiments, a source compute instance (CI) can send communication to a destination compute instance located in the same VLAN as the source compute instance (CI). Communication can also be sent to an endpoint outside the VLAN of the source CI. This may include, for example, communication between a source CI in a first VLAN and a destination CI in a second VLAN, communication between a source CI in a first VLAN and a destination CI in an L3 subnet, and / or communication from a source CI in a first VLAN to a destination CI outside the VCN containing the source CI's VLAN. This communication may further include, for example, receiving communication at the destination CI from a source CI outside the destination CI's VLAN. This source CI may be in another VLAN, an L3 subnet, or outside the VCN containing the source CI's VLAN.

[0189] Each CI within a VLAN can play an active role in the traffic flow. This includes learning interface identifiers versus MAC addresses (also referred to here as interface versus MAC address), mapping instances within the VLAN to maintain the L2 forwarding table within the VLAN, and sending and / or receiving communications (e.g., frames in the case of L2 communications). VSRS can play an active role in communications within the VLAN and in communications with source or destination CIs outside the VLAN. VSRS can maintain its presence within the L2 and L3 networks, enabling outgoing and incoming communications.

[0190] Referring now to Figure 12, a flowchart illustrating one embodiment of process 1200 for intra-VLAN communication is shown. In some embodiments, process 1200 may be executed by a compute instance within a common VLAN. Specifically, this process may be executed when a source CI sends communication to a destination CI within a VLAN, but does not know the IP-to-MAC address mapping of that destination CI. This can occur, for example, when a source CI sends a packet to a destination CI that has an IP address in the VLAN, but the source CI does not know the MAC address corresponding to that IP address. In this case, an ARP process can be executed to learn the destination MAC address and the IP-to-MAC address mapping.

[0191] If the source CI knows the IP-to-MAC address mapping, the source CI can send the packet directly to the destination CI without the need for an ARP process to be performed. In some embodiments, this packet may be intercepted by a source VNIC whose source VNIC is an L2 VNIC in intra-VLAN communication. If the source VNIC knows the interface-to-MAC address mapping for the destination MAC address, the source VNIC can encapsulate the packet, for example, with L2 encapsulation, and forward the corresponding frame to the destination MAC address to the destination VNIC whose destination VNIC is an L2 VNIC in intra-VLAN communication.

[0192] If the source VNIC does not know the interface-to-MAC address mapping for a MAC address, the source VNIC can perform an interface-to-MAC address learning process. This may involve the source VNIC sending a frame to all interfaces in the VLAN. In some embodiments, this frame may be sent to all interfaces in the VLAN via broadcast. In some embodiments, this broadcast may be implemented in the form of a serial unicast in the physical network. This frame may include the destination MAC and IP addresses, the interface identifier, and the MAC and IP addresses of the source VNIC. Each VNIC in the VLAN can receive this frame and learn the interface-to-MAC address mapping of the source VNIC.

[0193] Each receiving VNIC can further decapsulate a frame and forward the decapsulated frame (e.g., the corresponding packet) to its associated CI. Each CI may include a network interface from which it can evaluate the forwarded packet. If the network interface determines that the CI receiving the forwarded packet does not match the destination MAC and / or IP address, the packet is dropped. If the network interface determines that the CI receiving the forwarded frame matches the destination MAC and / or IP address, the packet is received by the CI. In some embodiments, a CI having a MAC and / or IP address that matches the destination MAC and / or IP address of a packet can send a response to the source CI, thereby allowing the source VNIC to learn the interface-to-MAC address mapping of the destination CI, and thereby allowing the source CI to learn the IP-to-MAC address mapping of the destination CI.

[0194] If the source CI does not know the IP-to-MAC address mapping, or if the source CI's IP-to-MAC address mapping to the destination CI is outdated, process 1200 can be executed. Thus, once the IP-to-MAC address mapping is known, the source CI can send the packet. If the IP-to-MAC address mapping is not known, process 1200 can be executed. If the interface-to-MAC address mapping is not known, the interface-to-MAC address learning process outlined above can be executed. If the interface-to-MAC address mapping is known, the source VNIC can send the corresponding frame to the destination VNIC. Process 1200 begins at block 1202, where the source CI determines that the destination CI's IP-to-MAC address mapping is unknown to the source CI. In some embodiments, this may include the source CI determining the destination IP address for the packet and determining that the destination IP address is not associated with any MAC address stored in the source CI's mapping table. Alternatively, the source CI may determine that the IP-to-MAC address mapping for its destination CI is outdated. In some embodiments, a mapping can be outdated if it has not been updated and / or validated within a certain time limit. If the source CI determines that the destination CI's IP-to-MAC address mapping is unknown and / or outdated, the source CI initiates an ARP request for the destination IP address and sends an ARP request for Ethernet broadcast.

[0195] In block 1204, the source VNIC, also called the source interface, receives an ARP request from the source CI. The source interface identifies all interfaces on the VLAN and sends ARP requests to all interfaces on the VLAN broadcast domain. As previously mentioned, the control plane knows all interfaces on the VLAN and provides this information to the interfaces with the VLAN, so the source interface also knows all interfaces within the VLAN and can send ARP requests to each of them. To do this, the source interface duplicates the ARP requests and encapsulates one of the ARP requests for each interface on the VLAN. Each encapsulated ARP request includes the source CI interface identifier, the source CI MAC address and IP address, the target IP address, and the destination CI interface identifier. The source CI interface duplicates the Ethernet broadcast by sending the duplicated and encapsulated ARP requests (e.g., ARP messages) as serial unicast, so that one ARP request is sent to each interface in the VLAN.

[0196] In block 1206, each interface in the VLAN broadcast domain receives and decapsulates an ARP message. Each interface in the VLAN broadcast domain that receives an ARP message learns the interface-to-MAC address mapping of the source VNIC of the source CI (e.g., the interface identifier of the source interface to the MAC address of the source CI) because the message identifies the source CI's MAC address and IP address, as well as the source CI interface identifier. As part of learning the interface-to-MAC address mapping for the source CI, each interface can update its mapping table (e.g., its L2 forwarding table) and provide the updated mapping to its associated switch and / or CI. Each receiving interface, except for VSRS, can forward the decapsulated packet to its associated CI. The CI recipient of the forwarded decapsulated packet, specifically the network interface of that CI, can determine whether the target IP address matches the IP address of the CI. If the IP address of the CI associated with that interface does not match the destination CI IP address, in some embodiments, the packet is dropped by that CI and no further action is taken. In the case of VSRS, VSRS can determine whether the target IP address matches the VSRS's IP address. If the VSRS's IP address does not match the target IP address specified in the received packet, in some embodiments, the packet is dropped by the VSRS and no further action is taken.

[0197] If the destination CI IP address specified in the received packet is determined to match the IP address of the CI (destination CI) associated with the receiving interface, the destination CI sends a response, which may be a unicast ARP response, to the source interface, as shown in block 1208. This response includes the destination CI MAC address and destination CI IP address, and the source CI IP address and MAC address. This response is received by the destination interface, as shown in block 1210, and it encapsulates the unicast ARP response. In some embodiments, this encapsulation may include Geneve encapsulation. The destination interface can forward the encapsulated ARP response to the source interface via the destination switch. This response includes the destination CI MAC address and IP address, as well as the destination CI interface identifier, and the source CI MAC address and IP address, as well as the source CI interface identifier.

[0198] In block 1212, the source interface receives the ARP response and decapsulates it. The source interface can then learn an interface-to-MAC address mapping for the destination CI based on the information contained in the encapsulated and / or encapsulated frame. In some embodiments, the source interface can forward the ARP response to the source CI.

[0199] In block 1214, the source CI receives an ARP response. In some embodiments, the source CI can update its mapping table based on the information contained in the ARP response, specifically, it can update the mapping table to reflect the IP-to-MAC address mapping based on the destination CI's MAC address and IP address. The source CI can then send a packet to the destination CI based on this MAC address. This packet may include the source CI's MAC address and interface identifier as the source MAC address and source interface, and the destination CI's MAC address and interface identifier as the destination MAC address and destination interface.

[0200] In block 1216, the source interface can receive packets from the source CI. The source interface can encapsulate packets, and in some embodiments, this encapsulation uses Geneve encapsulation. The source interface can forward the corresponding frames to the destination CI, specifically to the destination interface. The encapsulated frames may include the MAC address and interface identifier of the source CI as the source MAC address and source interface identifier, and the MAC address and interface identifier of the destination CI as the destination MAC address and destination interface.

[0201] In block 1218, the destination interface receives a frame from the source interface. The destination interface can decapsulate the frame and then forward the corresponding packet to the destination CI. In block 1220, the destination CI receives a packet from the destination interface.

[0202] Layer 2 Networking Information An L2 physical network includes a single switch. Information associated with the traffic flow of an L2 physical network can be determined by querying the switch. For example, a switch may maintain a single L2 forwarding table, and this table can be queried. In another example, a switch may maintain certain metrics for its ports, and these metrics can be queried. By comparison, an L2 virtual network (referred to here as VLANs for brevity) includes a distributed L2 switch (referred to here as a v-switch, indicating that this corresponds to the customer's perception of a single virtual switch), and a distributed L2 switch includes multiple L2 switches, each L2 switch associated with a compute instance and hosted by a certain NVD. Each of such L2 switches can maintain its own L2 information. Such information can be collected from multiple L2 switches to generate collective L2 information about the L2 virtual network, which is then presented to the customer as if it were a single central switch with an unlimited or arbitrary number of ports and VLANs (as desired by the customer). For example, each L2 switch is associated with an L2 forwarding table. Different L2 forwarding tables are collected and presented to the customer as a global L2 forwarding table. Similarly, metrics associated with each L2 switch can be collected and statistics about the L2 virtual network can be presented to the customer. Furthermore, the L2 virtual network allows the customer the flexibility to configure the ports of that L2 virtual network. These ports are implemented as L2 virtual network interfaces (e.g., VNICs). The L2 information collected regarding the L2 virtual switch (e.g., L2 forwarding tables and / or statistics) typically includes information about the L2 virtual network interfaces. Therefore, before presenting the L2 information, fragments of information are updated based on the L2 virtual network interface-to-port correspondence, and the customer receives L2 information formatted according to their port configuration.

[0203] In one example of an L2 forwarding table, in an L2 physical network, a switch maintains an L2 forwarding table that shows the association between ports and MAC addresses. For example, a user can query a switch to receive the L2 forwarding table for troubleshooting purposes. In comparison, in embodiments of this disclosure, a v-switch of a VLAN is a collection of local switches hosted on one or more NVDs. Each local switch is associated with its own L2 forwarding table that maps MAC addresses to tunnel endpoints (e.g., interface identifiers of the VNIC) and / or NVDs (e.g., IP addresses of the NVD). The customer is also presented with a single view of the v-switch. Thus, the customer may be presented with a global L2 forwarding table, synthesized from the local L2 forwarding tables and formatted using the customer's definition of that VLAN, including the port configuration of that VLAN. This global L2 forwarding table can be presented in response to customer queries or in response to event-based triggers, each of which is described below.

[0204] In one example of L2 statistics, in an L2 physical network, a customer can query a switch to receive statistics on frame transmission. In comparison, in embodiments of this disclosure, a VLAN v-switch is a collection of local switches hosted on one or more NVDs, each switch associated with a VNIC hosted on the same NVD. A VNIC represents an L2 port of the v-switch. In response to a customer query, statistics on frame transmission for the VLAN can be collected from various NVDs and presented to the customer as v-switch statistics.

[0205] Figure 13 illustrates an exemplary environment suitable for defining the configuration of an L2 virtual network and providing relevant L2 information according to one embodiment. In the embodiment, the environment includes a computer system 1310 that communicates with a customer device 1320 over one or more networks (not shown). The computer system 1310 may include a set of hardware computing resources that host VCN 1312. A control plane hosted by one or more of the hardware computing resources can receive and process input from the customer device 1320 to deploy an L2 virtual network (shown as L2VLAN 1314 in Figure 13) within VCN 1312.

[0206] In one example, input from customer device 1320 may include various types of information. This information can be specified via console or API calls and may include L2 VLAN configuration 1322. Although not shown in Figure 13, additional input can be received from the customer device via console or API calls, and this additional input may include queries for L2 information (referred to here as L2 information queries). L2 information queries may include queries for L2 forwarding tables (for example, referred to here as L2 forwarding table queries) and / or queries for L2 statistics (referred to here as L2 statistics queries). L2 forwarding table queries can request entries in the L2 forwarding table for a specific set of ports or for all ports in an L2 VLAN (for example, the entire VLAN). Similarly, L2 statistics queries can request statistics specific to a specific set of ports or for all ports in an L2 VLAN (for example, the entire VLAN).

[0207] The L2 VLAN configuration 1322 can, for example, specify the number, type, and configuration of L2 compute instances that should be included in L2 VLAN 1314. In addition, the L2 VLAN configuration 1322 can specify the customer-specified name of the port on the customer-aware v-switch, the MAC address of the compute instance (which may be an L2 compute instance), and the association between the port and the MAC address (or, more generally, the compute instance). For example, the customer may specify that L2 VLAN 1314 should contain two L2 compute instances, the first L2 compute instance having MAC address M.1 and associated with a first port named P1, and the second L2 compute instance having MAC address M.2 and associated with a second port named P2. The control plane receives the various information and then deploys and manages the different resources of L2 VLAN 1314.

[0208] A system (for example, a subsystem of computer system 1310, such as a control plane and / or telemetry system) can collect L2 information associated with different L2 virtual switches that form a v-switch presented to the customer, and can convert this collected information into L2 information 1324 associated with a port. The L2 information 1324 can be sent to the customer device 1320 (for example, periodically in response to an L2 information query, or based on another trigger event).

[0209] In one embodiment, the L2 information 1324 includes an L2 forwarding table having entries corresponding to customer-defined ports (or subsets of ports). In such an embodiment, ports are implemented as L2 VNICs. An L2 VNIC is associated with an L2 virtual switch. The pair of L2 VNIC and L2 virtual switch is then associated with a compute instance. In this case, the system determines the L2 forwarding table of the L2 virtual switch and associates the entries in the L2 forwarding table with ports (for example, this table shows MAC addresses and L2 VNICs associated with another compute instance, but the information about the L2 VNIC is replaced with information about the port). Similar entries may be collected from different L2 virtual switches and updated to include port information instead of L2 VNIC information. These entries are included in the L2 forwarding table.

[0210] In an embodiment, L2 information 1324 includes L2 statistics with entries corresponding to ports (or subsets of ports) defined by the customer. In such an embodiment, ports are also realized as L2 VNICs. An L2 VNIC is associated with an L2 virtual switch. The pair of L2 VNIC and L2 virtual switch is associated with a compute instance. In this case, the system determines the L2 metrics associated with the pair (e.g., the rate of frames received by the L2 VNIC on an incoming frame flow, as shown in Figure 10, and the rate of frames transmitted by the L2 virtual switch on an outgoing frame flow, also as shown in Figure 10). The system also updates these metrics so that they are associated with ports. Similar metrics may be collected from different L2 VNIC-L2 virtual switch pairs and updated with their associations to the corresponding ports. These metrics, and their associations with ports, may be included in the L2 statistics. Furthermore, the system can perform statistical analysis (e.g., averaging, percentile calculation, etc.) on the collected metrics to present L2 statistics for a set of ports or an entire L2 VLAN.

[0211] Figure 14 shows exemplary L2 information in a Layer 2 virtual network according to several embodiments. The Layer 2 virtual network is referred to here as a VLAN. The top of Figure 14 shows a VLAN implementation diagram 1410. The bottom of Figure 14 shows a customer presentation of the VLAN 1420.

[0212] As mentioned above, a VLAN can contain "n" compute instances, each of which runs on a host machine. Figure 14 illustrates a one-to-one association between a compute instance and a host machine, but a many-to-one association is possible, where one host machine can run multiple compute instances. Each compute instance is associated with at least one virtual interface (e.g., an L2 VNIC) and a switch (e.g., an L2 virtual switch). The VNIC and switch pair associated with a compute instance on a host machine can be a pair of software modules on an NVD connected to the host machine. Each L2 VNIC represents an L2 port on the customer's v-switch. In the example in Figure 14, host machine "i" runs compute instance "i" and is connected to NVD "i". Next, NVD "i" runs VNIC "i" and switch "i". VNIC "i" represents L2 port "i" on the v-switch, where i is a positive integer from 1 to n. Again, a one-to-one association is described, but other types of associations are also possible. For example, a single NVD can be connected to multiple hosts, each running one or more compute instances belonging to a VLAN. In this case, the NVD would host multiple pairs of VNICs and switches, each corresponding to one of the compute instances.

[0213] Customer input can be received via console or API and transmitted from there to VCN systems including VLANs (e.g., the VCN control plane and / or telemetry systems). Input can be received via API calls and / or console and may specify VLAN configurations, including at least port configurations (e.g., port naming conventions, MAC addresses to be associated with ports, etc., shown as port configuration 1422 in Figure 14). Such configuration information can be used to respond to L2 information queries to customers and / or to provide L2 information related to port configurations.

[0214] In one example, as described above, the NVD's L2 VNIC learns interface-to-MAC address mappings based on incoming traffic. Such mappings can be sent to the system along with VLAN identifiers. The system can receive similar mappings from different NVDs hosting different L2 VNICs and generate mappings between interface identifiers, MAC addresses, (e.g., the NVD's) physical IP addresses, and VLAN identifiers.

[0215] For example, VNIC1 learns that M.2 (the overlay MAC address of compute instance 2) is associated with ID.2 (the interface identifier of L2 VNIC2) and IP.2 (the physical address of NVD2), and that Mn (the overlay MAC address of compute instance n) is associated with ID.n (the interface identifier of L2 VNIC n) and IP.n (the physical address of NVD n). Similarly, VNIC2 learns that M.1 (the overlay MAC address of compute instance 1) is associated with ID.1 (the interface identifier of L2 VNIC1) and IP.1 (the physical address of NVD1). These associations are reported to the system as part of the mapping, and the system can generate mappings such as {Customer 1; M.1 → ID.1, IP.1; VLAN A}, {Customer 1, M.2 → ID.2, IP.2; VLAN A}, ..., {Customer 1, Mn → ID.n, IP.n; VLAN A}.

[0216] Furthermore, the system can generate and maintain a mapping between port configuration 1422 and the distribution of VLAN resources on the physical network. For example, this mapping indicates that port "i" is configured for compute instance "i", corresponds to L2 VNIC "i", which is then associated with L2 virtual switch "i", and this pair is hosted by NVD "i". This mapping can be represented as {customer1;CI.1→P.1;P.1→M.1;M.1→ID.1,SW.1,IP.1;VLAN A}, where "CI.1" identifies compute instance 1 (using a naming convention defined by the customer), P.1 identifies port 1 (using a naming convention defined by the customer), M.1 identifies the MAC address associated with compute instance 1, ID.1 identifies L2 VNIC 1, SW.1 identifies L2 virtual switch 1, IP.1 identifies the IP address of NVD 1, and VLAN A identifies the VLAN.

[0217] The system can receive information from an NVD L2 associated with the L2 virtual switch hosted by that NVD (shown as individual L2 switch information 1412 in Figure 14). This L2 switch is part of a pair associated with a compute instance. The other part of the pair is an L2 VNIC emulating a port. Individual L2 information 1414 may include the L2 forwarding table of the L2 virtual switch and L2 metrics of the traffic flow to the compute instance (e.g., the flow of incoming frames through the L2 VNIC or outgoing frames through the L2 virtual switch). The system can collect different individual L2 information 1414 from different NVDs, each associated with a different L2 virtual switch.

[0218] Based on the collected individual L2 switch information 1414, the system can generate L2 information associated with the v-switch (shown as global L2 information 1414 in Figure 14). For example, global L2 information 1414 may be a combination and / or aggregation of individual L2 switch information 1412 collected from different NVDs. For instance, global L2 information 1414 is an aggregation of entries from individual L2 forwarding tables, which represents the global L2 forwarding table for an L2 VLAN. In another example, global L2 information 1414 may include individual L2 metrics and / or L2 statistics derived from individual L2 metrics.

[0219] Based on the mapping between the port configuration 1422 and the distribution of VLAN resources on the physical network, the system can generate L2 information from global L2 information that is associated with a set of ports or a full set of ports (e.g., the entire L2 VLAN) in an L2 VLAN. This L2 information is shown in Figure 14 as L2v switch information 1424. In one example, L2v switch information 1424 includes entries from global L2 information 1414, and these entries are associated with ports (e.g., using a customer-defined naming convention) rather than L2 virtual switches and / or L2VNICs. L2v switch information 1424 can be sent to the customer's device. For example, L2v switch information 1424 includes entries from a global L2 forwarding table, and these entries are mapped to ports. In another example, L2v switch information 1424 includes individual L2 metrics and / or L2 statistics mapped to ports.

[0220] Figure 15 shows an exemplary L2 forwarding table in a Layer 2 virtual network according to several embodiments. The Layer 2 virtual network is referred to here as a VLAN. The upper part of Figure 15 shows a VLAN implementation diagram 1410. The lower part of Figure 15 shows a VLAN customer presentation diagram 1420. The VLAN configuration is identical or similar to that in Figure 14. The commonalities between these two diagrams will not be repeated here.

[0221] A system such as the control plane receives individual L2 forwarding tables 1512 from the NVD. Each of the individual L2 forwarding tables 1512 is associated with an L2 virtual switch of the VLAN. The individual forwarding tables of the L2 virtual switch associate MAC addresses with the interface identifier of the L2 VNIC and / or the physical IP address of the NVD hosting the L2 VNIC, based on incoming traffic learning of the L2 VNIC associated with the L2 virtual switch. In such embodiments, the MAC address may be an address assigned to an L2 compute instance and may correspond to a port emulated by the L2 VNIC associated with the L2 compute instance.

[0222] From the individual L2 forwarding tables 1512, the system generates a global L2 forwarding table 1514. For example, the global L2 forwarding table 1514 is an aggregation of different entries from the individual L2 forwarding tables 1512, with redundant entries removed. Thus, the global L2 forwarding table 1514 associates MAC addresses with the interface identifier of the L2 VNIC and / or the physical IP address of the NVD hosting the L2 VNIC, based on incoming traffic learning across different L2 VNICs.

[0223] From the global L2 forwarding table 1514, based on the mapping between port configurations and the distribution of VLAN resources on the physical network, the system can generate the L2v switch forwarding table 1522. For example, entries in the global L2 forwarding table 1514 are updated to be associated with ports (e.g., using customer-defined port naming conventions) rather than L2 VNICs, L2 virtual switches, and / or NVDs. Thus, the L2v switch forwarding table 1522 associates MAC addresses with port identifiers (e.g., M.1 is the MAC address of compute instance 1, and compute instance 1 is behind port 1).

[0224] In one embodiment, in response to a customer L2 transfer table query, the system can generate a global L2 transfer table 1514 from the collected L2 transfer information, and then generate an L2v switch transfer table 1522. Alternatively, the global L2 transfer table 1514 is generated prior to the customer L2 transfer table query, while the L2v switch transfer table 1522 is generated in response to such a query.

[0225] In another example, instead of storing the individual L2 transfer tables 1512 that were collected, or in addition to that, the system could determine which NVD hosts the L2 virtual switch and request such tables from the NVD in response to customer queries.

[0226] Furthermore, partly due to the learning capabilities of the L2 VNICs and the flexibility of the network, conflicts may exist between multiple parts of the collected individual L2 forwarding tables 1512. For example, NVD1 may report to the system that compute instance 2 is behind M.2 when L2 VNIC1 receives and processes a frame from compute instance 2 to compute instance 1. Subsequently, due to a network event, M.2 may be reassigned to compute instance 3 associated with L2 VNIC3 (computation instance 3 and L2 VNIC3 are not shown in Figure 15). L2 VNIC1 may not receive subsequent frames from compute instance 3. Instead, L2 VNIC n may receive subsequent frames by receiving and processing frames from compute instance 3 to compute instance n. Thus, NVD n may report to the system that compute instance 3 is behind M.2. Since NVD1 previously reported that compute instance 2 was behind M.2, the newer report from NVDn will conflict with that. In such a situation, the control plane may discard older, conflicting L2 forwarding information (e.g., compute instance 2 is behind M.2) and use only the most recent L2 forwarding information in the global L2 forwarding table (e.g., compute instance 2 is behind M.2).

[0227] The above example of reassigning an overlay MAC address from one compute instance to another is an example of a network event that can trigger the control plane to generate and send an updated global L2 forwarding table to a customer. For example, a customer may subscribe to receive such updates. This subscription may be received as customer input via an API call or console and may be specific to a certain type of network event or may be inclusive of any type of network event that results in an update to the global L2 forwarding table. In the example above, the overlay MAC address remained the same but was reassigned to a different compute instance. Another example of a network event would involve a change to the overlay MAC address of the same compute instance. Further examples of network events would involve the addition of an overlay MAC address and / or compute instance. Yet another example, a network event could include the relocation of a compute instance from one host to another. Other types of events are also possible, such as time-based events. For example, a customer could demonstrate periodicity for reporting L2v switch FWD1522. The system can then periodically collect individual L2 forwarding tables 1512, update the global L2 forwarding table 1514, and then update the v-switch forwarding table 1522 for periodic reporting to customers.

[0228] Compared to customer queries, for event-based triggers, the system can send configuration information to the NVD, which in turn requests the NVD to automatically report the local individual L2 forwarding tables when an event (e.g., network-based or time-based event) is detected. When the NVD detects an event, it sends the local individual L2 forwarding tables of the applicable L2 virtual switches to the system, which then updates the global L2 forwarding table 1514 and the L2v switch forwarding table 1522 and notifies the customer.

[0229] Figure 16 shows exemplary L2 metrics and statistics in a Layer 2 virtual network according to several embodiments. The bottom of Figure 16 shows a customer presentation of VLANs 1420. The VLAN configuration is identical or similar to that in Figure 14. The commonalities between these two figures will not be repeated here.

[0230] A system, such as a control plane or telemetry system, receives individual L2 metrics 1612 from the NVD. Each individual L2 metric 1612 is associated with a VLAN's L2 virtual switch, L2 VNIC, or L2 VNIC-L2 virtual switch pair. Each individual L2 metric includes metrics relating to incoming traffic flow to the L2 VNIC or outgoing traffic flow from the L2 virtual switch (e.g., frame rate indicated by customer input or any other type of metric). The system associates such metrics with the MAC address, the L2 VNIC interface identifier, the L2 virtual switch identifier, and / or the physical IP address of the NVD hosting the pair.

[0231] From individual L2 metrics 1612, the system generates global L2 statistics 1614. For example, global L2 statistics 1614 is a statistical combination of the individual L2 metrics 1612. Global L2 metrics 1614 are associated with the MAC address, the interface identifier of the L2 VNIC, the identifier of the L2 virtual switch, and / or the physical IP address of the NVD hosting the pair.

[0232] From the individual L2 statistics 1612, based on the mapping between port configuration and the distribution of VLAN resources on the physical network, the system can generate individual port metrics (shown as individual port metrics 1622 in Figure 16). Each of the individual port metrics 1612 corresponds to one of the individual L2 metrics 1616, and the value is mapped to the port, rather than the corresponding L2 VNIC, L2 virtual switch, and / or pair. For example, an individual port metric for a port might include metrics related to incoming traffic flow to the L2 VNIC or outgoing traffic flow from the L2 virtual switch associated with that port (e.g., frame rate indicated by customer input or any other type of metric). The system associates such metrics with MAC addresses and user-defined port naming conventions.

[0233] From the global L2 statistics 1614, based on the mapping between port configurations and the distribution of VLAN resources on the physical network, the system can generate L2v switch statistics 1624. For example, entries in the global L2 statistics 1614 are updated to be associated with ports (e.g., using customer-defined port naming conventions) rather than L2 VNICs, L2 virtual switches, and / or NVDs. Thus, the L2v switch statistics 1622 associate the statistic values ​​with MAC addresses and port identifiers.

[0234] In the embodiment, customer input includes different dimensions relating to statistics. Each dimension indicates the type of metric to be collected and / or the type of statistics to be generated. In the first dimension, the customer can specify whether the number of frames (e.g., FPS), frame size, and / or the amount of bits (e.g., BPS) should be reported. In the second dimension, the customer can indicate whether the statistics should be reported for the entire VLAN, per port, or for a specific set of ports. In the third dimension, the customer can request that the statistics be specific to transmitted frames, dropped frames, frame errors, or all frames.

[0235] An NVD hosting a VLAN's L2 VNIC (for example, NVD1 hosting L2 VNIC1) periodically reports metrics to the system. These metrics can be annotated with metadata that identifies the VLAN and L2 VNIC. Thus, the system collects metrics from different NVDs, and these metrics are annotated.

[0236] Next, the system generates statistics requested by the customer based on annotated metrics, port configuration, and mapping. For example, if a customer requests the number of frames dropped per hour and per port for a VLAN, the system will present this number of frames as dropped on port 1, assuming a correspondence between L2 VNIC1 and port 1, rather than showing the number of frames dropped on L2 VNIC1 in the most recent hour.

[0237] Figure 17 is a flowchart of process 1700 for providing L2 information according to several embodiments. In some embodiments, process 1700 can be performed by a system of the VCN, such as a control plane that manages the deployment of L2 virtual networks on the underlying physical network, or a telemetry system that monitors data related to the deployment and use of resources in the L2 virtual network. Process 1700 will be described in relation to L2 information. Such L2 information may include L2 forwarding tables, L2 metrics, and L2 statistics, or a combination thereof.

[0238] Process 1700 begins in block 1702, when the system receives customer input, which indicates the customer configuration of a Layer 2 virtual network. In some embodiments, customer input is received from a customer device via API calls and / or a console and indicates an L2 virtual network configuration (e.g., an L2 VLAN configuration as described in relation to Figure 13). The L2 virtual network configuration may include port configurations, such as the number and naming convention of ports on a v-switch, as well as their association with compute instances defined by the customer.

[0239] In block 1704, the system determines a mapping between the customer configuration and the distribution of Layer 2 virtual network resources on the physical network. In some embodiments, the resources include L2 VNICs and L2 virtual switches in addition to compute instances. Here, as described above, the L2 VNIC-L2 virtual switch pair is hosted by the NVD, while the compute instances are hosted by a host machine communicably coupled to the NVD. Each L2 VNIC emulates a port of a v-switch known to the customer. For each port, the system generates a mapping indicating the correspondence to the L2 VNIC. This mapping can also associate each port with a MAC address, a compute instance, an L2 virtual switch, and the NVD hosting the L2 VNIC-L2 virtual switch pair.

[0240] In block 1706, the system determines events for which L2 information is collected. In some embodiments, events include customer L2 information queries. In other embodiments, events include network events that may result in changes to previously collected L2 information, such as changes to the network topology or configuration of an L2 virtual network. In yet another embodiment, events include time events, such as predetermined periodic time intervals for collecting L2 information.

[0241] In block 1708, the system determines the first Layer 2 information associated with the first Layer 2 virtual switch of the customer's Layer 2 virtual network. In some embodiments, a pull mechanism is used. For example, the first Layer 2 virtual switch is hosted by a first NVD. When the system detects an event, the system requests Layer 2 information from the NVD. In response, the NVD sends the first L2 information, along with instructions that this information is associated with the first Layer 2 virtual switch, the L2 virtual network interface paired with the first Layer 2 virtual switch, and / or the L2 virtual network interface-L2 virtual switch pair. In other embodiments, a push mechanism is used. For example, the NVD detects an event and sends the first L2 information to the system along with association instructions.

[0242] In block 1710, the system determines the second Layer 2 information associated with the second L2 virtual switch. In some embodiments, the operation of block 1710 may be similar to the operation of block 1708, thereby the second Layer 2 information may be received from the same NVD (if such NVD hosts both the first and second L2 virtual switches) or different NVDs (if the first and second L2 virtual switches are hosted by different NVDs). For brevity, only two L2 virtual switches are described in relation to process 1700. However, if the L2 virtual network includes additional L2 virtual switches, the L2 information associated with each of such switches may be determined in a similar manner.

[0243] In block 1712, the system generates third Layer 2 information based on the first and second Layer 2 information. In some embodiments, the third L2 information is a combination or aggregation of the first and second L2 information, as well as any other L2 information determined for any other L2 virtual switches in the L2 virtual network. In some embodiments, the first and / or second L2 information is updated based on mapping so that the third L2 information indicates port information rather than L2 VNIC information. For example, the system determines that the first Layer 2 virtual network interface is associated with the first Layer 2 virtual switch, and the second Layer 2 virtual network interface is associated with the second Layer 2 virtual switch. Based on the mapping, the system determines that the first Layer 2 virtual network corresponds to the first port, and the second Layer 2 virtual network interface corresponds to the second port. The system then indicates that the third Layer 2 information is associated with the first and second ports, rather than the first and second L2 virtual network interfaces. This indication can be implemented in different ways. In some embodiments, the entries and / or metadata (e.g., annotations) of the first L2 information identify the first L2 virtual network interface (and / or the first L2 virtual switch). The system then updates the entries and / or metadata to identify the first ports instead. A similar update can be made for the second L2 information. The updated first and second L2 information (e.g., their entries) can be combined or aggregated to generate the third L2 information. Generally, the updates remove information specific to the distribution of resources in the L2 virtual network and replace it with port-specific information so that when the third L2 information is presented to the customer, the customer can understand this information based on how they configured the L2 virtual network.

[0244] In block 1714, the system sends third Layer 2 information to the customer's device. In some embodiments, the third L2 information is sent in response to an event determined in block 1702. For example, if an L2 information query is received, the third L2 information is sent as a result in response to such a query. In this example, the L2 information query may indicate a port, a set of ports, or all ports of an L2 virtual network. The result includes a portion of the third L2 information corresponding to the indicated port, or, where applicable, the entire third L2 information. For example, the L2 information query may indicate a first port. In this example, the system determines, based on the mapping, that the first port corresponds to a first Layer 2 virtual network interface, and that the first Layer 2 virtual network interface is associated with a first Layer 2 virtual switch. The system can then generate a query result associated with the first port based on the first Layer 2 information. For example, only the entry for the first Layer 2 information is updated and sent as the port in the query result. Alternatively, if the third Layer 2 information is already updated and available, the system can filter it to send only the portion related to the first port in the query results. In another example, network events are detected to collect L2 information. In this example, the entire third L2 information can be sent to the customer's device. Alternatively, if the network event affects only a portion of the third L2 information, only that portion will be sent (for example, if the network event corresponds only to a change to the first Layer 2 virtual switch and / or the first L2 virtual network interface; in this case, only the updated first L2 information will be sent). In yet another example, time events are detected to collect L2 information. In this example, the entire third L2 information can be sent to the customer's device. Alternatively, only the changes to the third L2 information can be sent.

[0245] Figure 18 is a flowchart of process 1800 for generating an L2 forwarding table according to several embodiments. Process 1800 may be implemented as a subprocess of process 1700, for example, when L2 information includes an L2 forwarding table. For brevity, only one L2 virtual switch is described in relation to process 1800. However, if the L2 virtual network includes additional L2 virtual switches, the operation of process 1800 may be similarly repeated to process the L2 forwarding table for each of such switches.

[0246] Process 1800 begins in block 1802, in which the system determines the first Layer 2 forwarding table of the first Layer 2 virtual switch. In some embodiments, this determination can be based on event detection and can use a pull mechanism or a push mechanism, as described in relation to process 1700. Here, the NVD hosting the first L2 virtual switch can send the first L2 forwarding table to the system, along with instructions to associate this table with the first L2 virtual switch, the first L2 virtual network interface paired with the first L2 virtual switch, and / or the pair.

[0247] In block 1804, the system determines that the first Layer 2 virtual switch is associated with the first Layer 2 virtual network interface. In some embodiments, this determination is based on the distribution of L2 virtual network resource deployments on the physical network.

[0248] In block 1806, the system determines that the first Layer 2 virtual network interface corresponds to the first port. In some embodiments, this determination is based on a mapping between ports (e.g., based on customer port configuration) and the distribution of L2 virtual network resource deployments on the physical network. For example, this mapping indicates that the first port corresponds to the first L2 virtual network interface and / or the first L2 virtual network switch.

[0249] In block 1808, the system generates a second Layer 2 forwarding table associated with the first port. In some embodiments, the entries and / or metadata of the first L2 forwarding table are updated to associate the first L2 forwarding table with the first port rather than with a first L2 virtual switch and / or first L2 virtual network interface. Furthermore, the entries may indicate a mapping of MAC addresses to L2 virtual network interfaces, L2 virtual switches, and / or NVDs (for example, so as learned by the first L2 virtual network interface on incoming traffic). These entries may be updated to remap MAC addresses to ports corresponding to L2 virtual network interfaces. The second L2 forwarding table includes the updated entries, along with an indication that this table is associated with the first port.

[0250] Although not shown in Figure 18, a second L2 forwarding table can be sent to the customer's device based on an event (e.g., an L2 forwarding table query, a network event, or a time event). In some embodiments, if an L2 forwarding table query is received and it indicates only a first port, the second L2 forwarding table is included in the query results. In comparison, if an L2 forwarding table query is received and it indicates multiple ports, the corresponding L2 forwarding tables (presumably their association with ports) are determined and sent in the query results in the combined L2 forwarding table. Alternatively, an L2v switch table may already be generated, and entries within this table may be filtered and sent in the query results according to the indicated ports.

[0251] In block 1810, the system determines a change to the first Layer 2 forwarding table. In some embodiments, certain events can trigger a change to an entry (e.g., a mapping) in the first Layer 2 forwarding table. As described above, this event could be the reassociation of a first MAC address from a second Layer 2 virtual network interface to a third Layer 2 virtual network interface, or the association of a second MAC address with a second Layer 2 virtual network interface. Such an event can trigger an update to the first Layer 2 forwarding table (based on incoming learning of the first Layer 2 virtual network interface), thereby remapping MAC addresses in this table where applicable.

[0252] In block 1812, the system updates a second Layer 2 forwarding table based on the changes. In some embodiments, a previously generated second Layer 2 forwarding table is also updated to reflect the MAC address remapping. Upon detecting changes to the second Layer 2 forwarding table, the system determines that this table is associated with a first port and can associate the changes with the port. Thus, when a customer query requesting an L2 forwarding table update is received, this association is used to return the updated second Layer 2 forwarding table. As an addition or alternative, updates made to the second L2 forwarding table may be pushed to the customer's device.

[0253] Figure 19 is a flowchart of process 1900 for generating L2 statistics according to several embodiments. Process 1900 may be implemented as a subprocess of process 1700 when, for example, the L2 information includes L2 metrics and / or L2 statistics. Process 1900 may be used in conjunction with process 1800 when, for example, the L2 information also includes L2 forwarding tables. For brevity, L2 metrics and statistics associated with a single L2 virtual switch are described in relation to process 1900. However, if the L2 virtual network includes additional L2 virtual switches, the operation of process 1900 may be similarly repeated to process the L2 metrics and statistics associated with each of such switches.

[0254] Process 1900 begins in block 1902, in which the system determines a first L2 metric associated with a first Layer 2 virtual switch. In some embodiments, this determination can be based on event detection and can use a pull or push mechanism, as described in relation to process 1700. Here, the NVD hosting the first L2 virtual switch can send the first L2 metrics to the system, along with instructions (e.g., annotations in metadata) to associate these metrics with the first L2 virtual switch, the first L2 virtual network interface paired with the first L2 virtual switch, and / or the pair. The type of L2 metrics to be collected may be indicated in the customer input. For example, the customer may specify that metrics regarding the number of frames, frame size, and / or bit amount should be collected. Thus, the system collects such L2 metrics from different NVDs, each hosting one or more L2 virtual switches.

[0255] In block 1904, the system determines that the first Layer 2 virtual switch is associated with the first Layer 2 virtual network interface. In some embodiments, this determination is based on the distribution of L2 virtual network resource deployments on the physical network.

[0256] In block 1906, the system determines that the first Layer 2 virtual network interface corresponds to the first port. In some embodiments, this determination is based on a mapping between ports (e.g., based on customer port configuration) and the distribution of L2 virtual network resource deployments on the physical network. For example, this mapping indicates that the first port corresponds to the first L2 virtual network interface and / or the first L2 virtual network switch. Furthermore, customer input can identify the target from which L2 metrics should be collected. For example, the customer can specify a port, a set of ports, or a Layer 2 virtual network as the target. Thus, if the input indicates the first port, the system can then determine that the first L2 metric should be collected and that such metric should be associated with the first port.

[0257] In block 1908, the system associates a first L2 metric with a first port. In some embodiments, the instruction (e.g., an annotation in metadata) is updated to identify the first port. In addition, the identification of the first L2 virtual switch and / or the first L2 virtual network interface may be removed from the annotation.

[0258] In block 1910, the system generates L2 statistics based on a first L2 metric. In some embodiments, the L2 statistics are specific to a first port and are generated from a first L2 metric associated with that port. In other embodiments, the L2 statistics are associated with more than one port, such as a set of ports, an L2 virtual network, or an entire v-switch. In this case, the L2 metric can be determined and associated with applicable ports as described above, and the L2 statistics are generated from such L2 metrics. The type of statistics (e.g., number of frames, frame size, and / or mean, sum, or other statistical analysis of the amount of bits) and / or the target from which the statistics will be generated (e.g., a port, a set of ports, or a Layer 2 virtual network) can also be indicated by customer input.

[0259] In block 1912, the system associates L2 statistics with a set of ports and / or an L2 virtual network. In some embodiments, this association depends on the target from which the L2 statistics were generated. The association may include annotations (e.g., in metadata) that identify the target (e.g., a port, a set of ports, or a Layer 2 virtual network).

[0260] Although not shown in Figure 19, L2 metrics and / or L2 statistics, as well as their association with targets, can be sent to the customer's device. In some embodiments, if an L2 statistics query is received and it indicates only a first port, only the first L2 metrics and L2 statistics determined from such metrics can be included in the query results. In other embodiments, in response to an event (for example, an L2 statistic indicating an average frame rate exceeding a predefined threshold, or a first L2 metric indicating that the frame rate associated with the first port exceeds a predefined threshold), the relevant L2 statistics and / or metrics are sent to the customer's device.

[0261] C. Exemplary Infrastructure as a Service Architecture As mentioned above, IaaS (Infrastructure as a Service) is a specific type of cloud computing. IaaS may be configured to provide virtualized computing resources over a public network (e.g., the internet). In the IaaS model, a cloud computing provider can host infrastructure elements (e.g., servers, storage, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer)). In some cases, the IaaS provider can provide various services associated with the infrastructure elements (e.g., billing, monitoring, logging, security, load balancing, and clustering). Therefore, since these services can be policy-driven, IaaS users can implement policies to drive load balancing in order to maintain application availability and performance.

[0262] In some cases, IaaS customers can access resources and services over a wide area network (WAN), such as the internet, and install the rest of their application stack using the cloud provider's services. For example, a user can log into an IaaS platform, create virtual machines (VMs), install an operating system (OS) on each VM, deploy middleware such as databases, create storage buckets for workloads and backups, and install enterprise software on the VMs. Customers can use the provider's services to perform a variety of functions, including balancing network traffic, troubleshooting applications, monitoring performance, and managing disaster recovery.

[0263] In most cases, the cloud computing model requires the participation of a cloud provider. This cloud provider may or may not be a third-party service specializing in IaaS provision (e.g., offering, renting, or selling). Alternatively, a company can become a provider of private clouds and infrastructure services.

[0264] In some cases, IaaS deployment is the process of deploying a new application or a new version of an application to a pre-configured application server. IaaS deployment may also include the process of preparing the server (e.g., installing libraries, daemons, etc.). IaaS deployment is often managed by the cloud provider under the hypervisor layer (e.g., servers, storage, network hardware, and virtualization). Therefore, customers can deploy the OS, middleware, and / or applications (e.g., self-service virtual machines, which can be spun up on demand).

[0265] In some cases, IaaS provisioning may include acquiring the computers or virtual hosts to be used and installing the necessary libraries or services on those computers or virtual hosts. In most cases, deployment does not include provisioning, and provisioning must be performed first.

[0266] In some cases, IaaS provisioning presents two distinct challenges. First, there's the challenge of provisioning an initial set of infrastructure before doing anything. Second, there's the challenge of evolving existing infrastructure after everything has been provisioned (e.g., adding new services, modifying services, removing services). In some cases, these two challenges can be addressed by enabling the declarative definition of infrastructure configuration. In other words, the infrastructure (e.g., what elements are needed and how these elements interact) may be defined by one or more configuration files. Thus, the overall topology of the infrastructure (e.g., which resources depend on which and how they work together) can be described declaratively. In some examples, once the topology is defined, workflows can be generated to create and / or manage the different elements described in the configuration files.

[0267] In some examples, infrastructure can include many interconnected elements. For example, there may be one or more virtual private clouds (VPCs), also known as core networks (e.g., configurable compute resources and / or potential on-demand pools of shared compute resources). In some examples, there may be one or more security group rules provisioned to define how network security is configured, and one or more virtual machines (VMs). Other infrastructure elements such as load balancers and databases may also be provisioned. As more and more infrastructure elements are desired and / or added, infrastructure can evolve incrementally.

[0268] In some examples, sequential deployment techniques may be employed to enable the deployment of infrastructure code across various virtual computing environments. Furthermore, the techniques described can enable infrastructure management within these environments. In some examples, a service team may write code that is intended to be deployed to one or more different production environments, typically many different geographical locations, sometimes across the globe. However, in some examples, the infrastructure for deploying the code must first be configured. In some examples, provisioning can be done manually, resources can be provisioned using provisioning tools, and / or the code can be deployed using deployment tools after the infrastructure has been provisioned.

[0269] Figure 20 is a block diagram 2000 illustrating an exemplary pattern of an IaaS architecture according to at least one embodiment. The service operator 2002 may be communicably coupled to a secure host tenancy 2004, which may include a virtual cloud network (VCN) 2006 and a secure host subnet 2008. In some examples, the service operator 2002 may use one or more client computing devices. One or more client computing devices may be handheld mobile devices (e.g., iPhone®, mobile phones, iPad®, tablets, personal digital assistants (PDAs) or wearable devices (Google® Glass® head-mounted displays)) with Internet, email, short message service (SMS), BlackBerry®, or other communication protocols enabled, and which can run software such as Microsoft Windows Mobile® and / or various mobile operating systems such as iOS, Windows Phone, Android®, BlackBerry 8, and Palm OS. The client computing devices may also be general-purpose personal computers, including, exemplarily, personal computers and / or laptop computers, which run various versions of the Microsoft Windows® operating system, Apple Macintosh® operating system, and / or Linux® operating system. Alternatively, the client computing devices may be workstation computers running various commercially available UNIX® or UNIX-like operating systems, including, but not limited to, various GNU / Linux operating systems and, for example, Google Chrome® OS.Alternatively or additionally, the client computing device may be another electronic device communicable via a network that can access the VCN 2006 and / or the Internet, such as a sync client computer, an Internet-enabled gaming system (e.g., a Microsoft Xbox (registered trademark) game console with or without a Kinect (registered trademark) gesture input device), and / or a personal messaging device.

[0270] The VCN 2006 can include a local peering gateway (LPG) 2010 communicably coupled to a secure shell (SSH) VCN 2012 via the LPG 2010 included in the SSH VCN 2012. The SSH VCN 2012 can include an SSH subnet 2014, and the SSH VCN 2012 can be communicably coupled to a control plane VCN 2016 via the LPG 2010 included in the control plane VCN 2016. Also, the SSH VCN 2012 can be communicably coupled to a data plane VCN 2018 via the LPG 2010. The control plane VCN 2016 and the data plane VCN 2018 may be included in a service tenancy 2019 that can be owned and / or operated by an IaaS provider.

[0271] The control plane VCN2016 can include a control plane DMZ (demilitarized zone) layer 2020, which functions as a perimeter network (e.g., the portion of the corporate network between the corporate intranet and the external network). DMZ-based servers have a certain level of reliability and can contain security breaches. Furthermore, the DMZ layer 2020 can include a control plane application layer 2024, which can include one or more load balancer (LB) subnets 2022 and application subnets 2026, and a control plane data layer 2028, which can include database (DB) subnets 2030 (e.g., a front-end DB subnet and / or a back-end DB subnet). The LB subnet 2022 included in the control plane DMZ layer 2020 may be communicatively coupled to the application subnet 2026 included in the control plane application layer 2024 and to an internet gateway 2034 which may be included in the control plane VCN 2016. The application subnet 2026 may be communicatively coupled to the DB subnet 2030 included in the control plane data layer 2028, to a service gateway 2036, and to a network address translation (NAT) gateway 2038. The control plane VCN 2016 may include the service gateway 2036 and the NAT gateway 2038.

[0272] The control plane VCN2016 may include a data plane mirror application layer 2040, which may include an application subnet 2026. The application subnet 2026 included in the data plane mirror application layer 2040 may include a virtual network interface controller (VNIC) 2042 on which compute instance 2044 can run. Compute instance 2044 can communicatively connect the application subnet 2026 of the data plane mirror application layer 2040 to an application subnet 2026 that may be included in the data plane application layer 2046.

[0273] The data plane VCN2018 can include a data plane application layer 2046, a data plane DMZ layer 2048, and a data plane data layer 2050. The data plane DMZ layer 2048 can include an LB subnet 2022 that can be communicatively coupled to the application subnet 2026 of the data plane application layer 2046 and the Internet gateway 2034 of the data plane VCN2018. The application subnet 2026 may be communicatively coupled to the service gateway 2036 of the data plane VCN2018 and the NAT gateway 2038 of the data plane VCN2018. Also, the data plane data layer 2050 can include a DB subnet 2030 that can be communicatively coupled to the application subnet 2026 of the data plane application layer 2046.

[0274] The Internet gateway 2034 of the control plane VCN2016 and the Internet gateway 2034 of the data plane VCN2018 may be communicatively coupled to a metadata management service 2052 that can be communicatively coupled to the public Internet 2054. The public Internet 2054 may be communicatively coupled to the NAT gateway 2038 of the control plane VCN2016 and the NAT gateway 2038 of the data plane VCN2018. The service gateway 2036 of the control plane VCN2016 and the service gateway 2036 of the data plane VCN2018 may be communicatively coupled to a cloud service 2056.

[0275] In some cases, a service gateway 2036 in the control plane VCN2016 or data plane VCN2018 can make application programming interface (API) calls to a cloud service 2056 without going through the public internet 2054. API calls from service gateway 2036 to cloud service 2056 can be one-way. Service gateway 2036 can make API calls to cloud service 2056, and cloud service 2056 can send request data to service gateway 2036. However, cloud service 2056 may not initiate an API call to service gateway 2036.

[0276] In some examples, Secure Host Tenancy 2004 may be directly connected to Service Tenancy 2019, which may be isolated. Secure Host Subnet 2008 can communicate with SSH Subnet 2014 via LPG2010, which enables bidirectional communication with the isolated system. By connecting Secure Host Subnet 2008 to SSH Subnet 2014, Secure Host Subnet 2008 can access other entities within Service Tenancy 2019.

[0277] The control plane VCN2016 allows users of service tenancy 2019 to configure or provision desired resources. Desired resources provisioned in the control plane VCN2016 may be deployed or used in the data plane VCN2018. In some examples, the control plane VCN2016 may be isolated from the data plane VCN2018, and the data plane mirror application layer 2040 of the control plane VCN2016 can communicate with the data plane application layer 2046 of the data plane VCN2018 via VNIC2042, which may be included in the data plane mirror application layer 2040 and the data plane application layer 2046.

[0278] In some examples, a system user or customer may make requests, such as create, read, update, or delete (CRUD) operations, via the public internet 2054, which can communicate the requests to the metadata management service 2052. The metadata management service 2052 can communicate the requests to the control plane VCN 2016 via the internet gateway 2034. The requests may also be received by an LB subnet 2022, which is included in the control plane DMZ layer 2020. The LB subnet 2022 may determine that the request is valid, and in response to this determination, it may send the request to an application subnet 2026, which is included in the control plane application layer 2024. If the request is validated and requires a call to the public internet 2054, the call to the public internet 2054 can be sent to a NAT gateway 2038, which can make calls to the public internet 2054. Memory for storing the requests may be stored in the DB subnet 2030.

[0279] In some cases, the data plane mirror application layer 2040 can facilitate direct communication between the control plane VCN2016 and the data plane VCN2018. For example, it may be desirable that changes, updates, or other appropriate modifications to the configuration be applied to the resources contained in the data plane VCN2018. Since the control plane VCN2016 can communicate directly with the resources contained in the data plane VCN2018 via VNIC2042, it can perform changes, updates, or other appropriate modifications to the configuration.

[0280] In some embodiments, the control plane VCN2016 and data plane VCN2018 may be included in the service tenancy 2019. In this case, the system user or customer does not have to own or operate either the control plane VCN2016 or the data plane VCN2018. Instead, the IaaS provider may own or operate the control plane VCN2016 and the data plane VCN2018, and both of these may be included in the service tenancy 2019. This embodiment can prevent a user or customer from interacting with other users' resources or other customers' resources by enabling network isolation. This embodiment can also enable a system user or customer to store databases privately without having to rely on the public internet, which may not have the desired level of security for storage.

[0281] In another embodiment, the LB subnet 2022 included in the control plane VCN2016 may be configured to receive signals from the service gateway 2036. In this embodiment, the control plane VCN2016 and the data plane VCN2018 may be configured to be invoked by the IaaS provider's customer without calling the public internet 2054. The IaaS provider's customer may prefer this embodiment because the database used by the customer may be stored in a service tenancy 2019 that is controlled by the IaaS provider and can be isolated from the public internet 2054.

[0282] Figure 21 is a block diagram 2100 showing another exemplary parameter of an IaaS architecture according to at least one embodiment. A service operator 2102 (e.g., service operator 2002 in Figure 20) may be communicably coupled to a secure host tenancy 2104 (e.g., secure host tenancy 2004 in Figure 20), which may include a virtual cloud network (VCN) 2106 (e.g., VCN2006 in Figure 20) and a secure host subnet 2108 (e.g., secure host subnet 2008 in Figure 20). VCN 2106 may include a local peering gateway (LPG) 2110 (e.g., LPG2010 in Figure 20), which may be communicably coupled to a secure shell (SSH) VCN 2112 (e.g., SSH VCN2012 in Figure 20) via an LPG2010 contained in an SSH VCN 2112. SSH VCN2112 may include SSH subnet 2114 (e.g., SSH subnet 2014 in Figure 20), and SSH VCN2112 may be communicably coupled to control plane VCN2116 (e.g., control plane VCN2016 in Figure 20) via LPG2110 included in control plane VCN2116. Control plane VCN2116 may be included in service tenancy 2119 (e.g., service tenancy 2019 in Figure 20), and data plane VCN2118 (e.g., data plane VCN2018 in Figure 20) may be included in customer tenancy 2121, which may be owned or operated by a user or customer of the system.

[0283] The control plane VCN2116 may include a control plane DMZ tier 2120 (e.g., control plane DMZ tier 2020 in Figure 20) which may include an LB subnet 2122 (e.g., LB subnet 2022 in Figure 20), a control plane application tier 2124 (e.g., control plane application tier 2024 in Figure 20) which may include an application subnet 2126 (e.g., application subnet 2026 in Figure 20), and a control plane data tier 2128 (e.g., control plane data tier 2028 in Figure 20) which may include a database (DB) subnet 2130 (e.g., similar to DB subnet 2030 in Figure 20). The LB subnet 2122 included in the control plane DMZ tier 2120 may be coupled to communicate with the application subnet 2126 included in the control plane application tier 2124 and with an internet gateway 2134 (e.g., internet gateway 2034 in Figure 20) which may be included in the control plane VCN2116. The application subnet 2126 may be communicatively coupled to the DB subnet 2130, service gateway 2136 (e.g., the service gateway in Figure 20), and network address translation (NAT) gateway 2138 (e.g., the NAT gateway 2038 in Figure 20), which are included in the control plane data layer 2128. The control plane VCN 2116 may include the service gateway 2136 and the NAT gateway 2138.

[0284] The control plane VCN2116 may include a data plane mirror application layer 2140 (e.g., data plane mirror application layer 2040 in Figure 20) which may include an application subnet 2126. The application subnet 2126 included in the data plane mirror application layer 2140 may include a virtual network interface controller (VNIC) 2142 (e.g., VNIC2042) which can run a compute instance 2144 (e.g., similar to compute instance 2044 in Figure 20). The compute instance 2144 can facilitate communication between the application subnet 2126 of the data plane mirror application layer 2140 and the application subnet 2126 that may be included in the data plane application layer 2146 (e.g., data plane application layer 2046 in Figure 20) via the VNIC2142 included in the data plane mirror application layer 2140 and the VNIC2142 included in the data plane application layer 2146.

[0285] The Internet gateway 2134 included in the control plane VCN2116 may be communicably coupled to a metadata management service 2152 (e.g., metadata management service 2052 in Figure 20), which can be communicably coupled to the public internet 2154 (e.g., public internet 2054 in Figure 20). The public internet 2154 may be communicably coupled to a NAT gateway 2138 included in the control plane VCN2116. The service gateway 2136 included in the control plane VCN2116 may be communicably coupled to a cloud service 2156 (e.g., cloud service 2056 in Figure 20).

[0286] In some examples, the data plane VCN2118 may be included in the customer tenancy 2121. In this case, the IaaS provider can provide a control plane VCN2116 for each customer, and the IaaS provider can configure a unique compute instance 2144 for each customer, included in the service tenancy 2119. Each compute instance 2144 can allow communication between the control plane VCN2116 included in the service tenancy 2119 and the data plane VCN2118 included in the customer tenancy 2121. The compute instance 2144 can allow resources provisioned in the control plane VCN2116 included in the service tenancy 2119 to be deployed or used in the data plane VCN2118 included in the customer tenancy 2121.

[0287] In another example, an IaaS provider's customer may have a database residing in customer tenancy 2121. In this example, control plane VCN 2116 may include a data plane minor application tier 2140 that can include application subnet 2126. A data plane mirror application tier 2140 may reside in data plane VCN 2118, but does not have to reside in data plane VCN 2118. That is, a data plane mirror application tier 2140 can access customer tenancy 2121, but does not have to reside in data plane VCN 2118 and does not have to be owned or operated by an IaaS provider's customer. A data plane mirror application tier 2140 may be configured to make calls to data plane VCN 2118, but does not have to be configured to make calls to any entity contained in control plane VCN 2116. The customer may wish to deploy or use resources within the data plane VCN2118 provisioned in the control plane VCN2116, and the data plane mirror application layer 2140 can facilitate the customer's desired deployment or other use of resources.

[0288] In some embodiments, a customer of the IaaS provider can apply filters to the data plane VCN2118. In this embodiment, the customer can determine what the data plane VCN2118 can access and can restrict access from the data plane VCN2118 to the public internet 2154. The IaaS provider may not be able to apply filters or control access from the data plane VCN2118 to any external network or database. Applying filters and controls to the data plane VCN2118 included in the customer tenancy 2121 can help isolate the data plane VCN2118 from other customers and the public internet 2154.

[0289] In some embodiments, the cloud service 2156 can be invoked by the service gateway 2136 to access services that may not reside on the public internet 2154, on the control plane VCN 2116, or on the data plane VCN 2118. The connection between the cloud service 2156 and the control plane VCN 2116 or data plane VCN 2118 does not have to be live or continuous. The cloud service 2156 may reside on another network owned or operated by the IaaS provider. The cloud service 2156 may be configured to receive calls from the service gateway 2136 and not to receive calls from the public internet 2154. Some cloud services 2156 may be isolated from other cloud services 2156, and the control plane VCN 2116 may be isolated from cloud services 2156 that may not be located in the same region as the control plane VCN 2116. For example, the control plane VCN 2116 may be located in "Region 1", and the cloud service "Deployment 20" may be located in "Region 1" and "Region 2". If a call to deployment 20 is made by a service gateway 2136 included in the control plane VCN2116 located in region 1, this call may be sent to deployment 20 in region 1. In this example, the control plane VCN2116 or deployment 20 in region 1 does not need to be communicatively coupled with deployment 20 in region 2.

[0290] Figure 22 is a block diagram 2200 showing another exemplary pattern of an IaaS architecture according to at least one embodiment. A service operator 2202 (e.g., service operator 2002 in Figure 20) may be communicatively coupled to a secure host tenancy 2204 (e.g., secure host tenancy 2004 in Figure 20), which may include a virtual cloud network (VCN) 2206 (e.g., VCN2006 in Figure 20) and a secure host subnet 2208 (e.g., secure host subnet 2008 in Figure 20). VCN2206 may include an LPG2210 (e.g., LPG2010 in Figure 20), which may be communicatively coupled to an SSH VCN2212 (e.g., SSH VCN2012 in Figure 20) via an LPG2210 contained in the SSH VCN2212. SSH VCN2212 may contain SSH subnet 2214 (e.g., SSH subnet 2014 in Figure 20), and SSH VCN2212 may be communicatively coupled to control plane VCN2216 (e.g., control plane VCN2016 in Figure 20) via LPG2210 contained in control plane VCN2216, and may be communicatively coupled to data plane VCN2218 (e.g., data plane 2018 in Figure 20) via LPG2210 contained in data plane VCN2218. Control plane VCN2216 and data plane VCN2218 may be contained in service tenant 2219 (e.g., service tenant 2019 in Figure 20).

[0291] The control plane VCN2216 may include a control plane DMZ layer 2220 (e.g., control plane DMZ layer 2020 in Figure 20) which may include a load balancer (LB) subnet 2222 (e.g., LB subnet 2022 in Figure 20), a control plane application layer 2224 (e.g., control plane application layer 2024 in Figure 20) which may include an application subnet 2226 (e.g., similar to application subnet 2026 in Figure 20), and a control plane data layer 2228 (e.g., control plane data layer 2028 in Figure 20) which may include a DB subnet 2230. The LB subnet 2222 included in the control plane DMZ layer 2220 may be coupled in a communicative manner to the application subnet 2226 included in the control plane application layer 2224 and to an internet gateway 2234 (e.g., internet gateway 2034 in Figure 20) which may be included in the control plane VCN2216. The application subnet 2226 may be communicatively coupled to the DB subnet 2230, which is included in the control plane data layer 2228, and to the service gateway 2236 (e.g., the service gateway in Figure 20) and the network address translation (NAT) gateway 2238 (e.g., the NAT gateway 2038 in Figure 20). The control plane VCN 2216 may include the service gateway 2236 and the NAT gateway 2238.

[0292] The data plane VCN2218 may include a data plane application layer 2246 (e.g., data plane application layer 2046 in Figure 20), a data plane DMZ layer 2248 (e.g., data plane DMZ layer 2048 in Figure 20), and a data plane data layer 2250 (e.g., data plane data layer 2050 in Figure 20). The data plane DMZ layer 2248 may include an LB subnet 2222 which can be communicatively coupled to the trusted application subnet 2260 and untrusted application subnet 2262 of the data plane application layer 2246 and Internet gateway 2234 included in the data plane VCN2218. The trusted application subnet 2260 may be communicatively coupled to the service gateway 2236 included in the data plane VCN2218, the NAT gateway 2238 included in the data plane VCN2218, and the DB subnet 2230 included in the data plane data layer 2250. The untrusted application subnet 2262 may be communicatively coupled to the service gateway 2236 included in the data plane VCN 2218 and to the DB subnet 2230 included in the data plane data layer 2250. The data plane data layer 2250 may include the DB subnet 2230, which can be communicatively coupled to the service gateway 2236 included in the data plane VCN 2218.

[0293] The untrusted application subnet 2262 can include one or more primary VNICs 2264(1) to (N) communicatively coupled to tenant virtual machines (VMs) 2266(1) to (N). Each tenant VM 2266(1) to (N) may be communicatively coupled to respective application subnets 2267(1) to (N) that may be included in respective container transmission VCNs 2268(1) to (N) that may be included in respective customer tenancies 2270(1) to (N). Each secondary VNIC 2272(1) to (N) can facilitate communication between the untrusted application subnet 2262 included in the data plane VCN 2218 and the application subnets included in the container transmission VCNs 2268(1) to (N). Each container transmission VCN 2268(1) to (N) can include a NAT gateway 2238 communicatively coupled to the public internet 2254 (e.g., the public internet 2054 of FIG. 20).

[0294] The internet gateway 2234 included in the control plane VCN 2216 and the internet gateway 2234 included in the data plane VCN 2218 may be communicatively coupled to a metadata management service 2252 (e.g., the metadata management system 2052 of FIG. 20) communicatively coupled to the public internet 2254. The public internet 2254 may be communicatively coupled to the NAT gateway 2238 included in the control plane VCN 2216 and the NAT gateway 2238 included in the data plane VCN 2218. The service gateway 2236 included in the control plane VCN 2216 and the service gateway 2236 included in the data plane VCN 2218 may be communicatively coupled to cloud services 2256.

[0295] In some embodiments, the data plane VCN2218 may be integrated with the customer tenancy 2270. This integration may be useful or desirable for the IaaS provider's customer in some cases, such as when they may want support when executing code. The customer may provide code that, when executed, could be destructive, could communicate with other customer resources, or could cause undesirable effects. Thus, the IaaS provider can determine whether or not to execute the code that the customer has provided to the IaaS provider.

[0296] In some examples, an IaaS provider's customer may grant the IaaS provider temporary network access and request functionality to be added to the data plane application layer 2246. The code for executing the functionality may run in VMs 2266(1)-(N), but cannot be configured to run elsewhere on the data plane VCN 2218. Each VM 2266(1)-(N) may be connected to one customer tenancy 2270. Each container 2271(1)-(N) contained within VMs 2266(1)-(N) may be configured to run the code. In this case, a double isolation may exist (for example, containers 2271(1)-(N) run the code, and containers 2271(1)-(N) may be contained within VMs 2266(1)-(N) that are in at least an untrusted application subnet 2262), which can help prevent erroneous or unwanted code from damaging the IaaS provider's network or the networks of different customers. Containers 2271(1)-(N) may be communicatively coupled to customer tenancy 2270 and may be configured to send or receive data from customer tenancy 2270. Containers 2271(1)-(N) do not have to be configured to send or receive data from any other entities in the data plane VCN2218. Once code execution is complete, the IaaS provider may kill or discard containers 2271(I)-(N).

[0297] In some embodiments, a trusted application subnet 2260 may execute code that may be owned or operated by the IaaS provider. In this embodiment, the trusted application subnet 2260 may be communicatively coupled to the DB subnet 2230 and configured to perform CRUD operations in the DB subnet 2230. An untrusted application subnet 2262 may be communicatively coupled to the DB subnet 2230, but in this embodiment, the untrusted application subnet may be configured to perform read operations within the DB subnet 2230. Containers 2271(1)~(N) contained in each customer's VM2266(1)~(N) and capable of executing code from the customer do not need to be communicatively coupled to the DB subnet 2230.

[0298] In other embodiments, the control plane VCN2216 and the data plane VCN2218 do not have to be directly coupled in a communicative manner. In this embodiment, direct communication between the control plane VCN2216 and the data plane VCN2218 may not exist. However, indirect communication by at least one method may exist. An LPG2210 that facilitates communication between the control plane VCN2216 and the data plane VCN2218 may be established by the IaaS provider. In another example, the control plane VCN2216 or the data plane VCN2218 can make a call to the cloud service 2256 via the service gateway 2236. For example, a call from the control plane VCN2216 to the cloud service 2256 may include a request for a service that can communicate with the data plane VCN2218.

[0299] Figure 23 is a block diagram 2300 showing another exemplary parameter of an IaaS architecture according to at least one embodiment. A service operator 2302 (e.g., service operator 2002 in Figure 20) may be communicatively coupled to a secure host tenancy 2304 (e.g., secure host tenancy 2004 in Figure 20), which may include a virtual cloud network (VCN) 2306 (e.g., VCN2006 in Figure 20) and a secure host subnet 2308 (e.g., secure host subnet 2008 in Figure 20). VCN 2306 may include an LPG 2310 (e.g., LPG2010 in Figure 20), which may be communicatively coupled to an SSH VCN 2312 (e.g., SSH VCN 2012 in Figure 20) via an LPG 2310 contained in an SSH VCN 2312. SSH VCN2312 may include SSH subnet-2314 (e.g., SSH subnet 2014 in Figure 20), and SSH VCN2312 may be communicatively coupled to control plane VCN2316 (e.g., control plane VCN2016 in Figure 20) via LPG2310 included in control plane VCN2316, and may be communicatively coupled to data plane VCN2318 (e.g., data plane 2018 in Figure 20) via LPG2310 included in data plane VCN2318. Control plane VCN2316 and data plane VCN2318 may be included in service tenancy 2319 (e.g., service tenancy 2019 in Figure 20).

[0300] The control plane VCN2316 may include a control plane DMZ layer 2320 (e.g., control plane DMZ layer 2020 in Figure 20) which may include an LB subnet 2322 (e.g., LB subnet 2022 in Figure 20), a control plane application layer 2324 (e.g., control plane application layer 2024 in Figure 20) which may include an application subnet 2326 (e.g., application subnet 2026 in Figure 20), and a control plane data tier 2328 (e.g., control plane data tier 2028 in Figure 20) which may include a DB subnet 2330 (e.g., DB subnet 2230 in Figure 22). The LB subnet 2322 included in the control plane DMZ layer 2320 may be coupled to communicate with the application subnet 2326 included in the control plane application layer 2324 and with an internet gateway 2334 (e.g., internet gateway 2034 in Figure 20) which may be included in the control plane VCN2316. The application subnet 2326 may be communicatively coupled to the DB subnet 2330, which is included in the control plane data layer 2328, and to the service gateway 2336 (e.g., the service gateway in Figure 20) and the network address translation (NAT) gateway 2338 (e.g., the NAT gateway 2038 in Figure 20). The control plane VCN 2316 may include the service gateway 2336 and the NAT gateway 2338.

[0301] The data plane VCN2318 may include a data plane application layer 2346 (e.g., data plane application layer 2046 in Figure 20), a data plane DMZ layer 2348 (e.g., data plane DMZ layer 2048 in Figure 20), and a data plane data layer 2350 (e.g., data plane data layer 2050 in Figure 20). The data plane DMZ layer 2348 may include a trusted application subnet 2360 (e.g., trusted application subnet 2260 in Figure 22) and an untrusted application subnet 2362 (e.g., untrusted application subnet 2262 in Figure 22) of the data plane application layer 2346, and an LB subnet 2322 that can be communicatively coupled to an internet gateway 2334 included in the data plane VCN2318. A trusted application subnet 2360 may be communicatively coupled to a service gateway 2336 included in data plane VCN 2318, a NAT gateway 2338 included in data plane VCN 2318, and a DB subnet 2330 included in data plane data layer 2350. An untrusted application subnet 2362 may be communicatively coupled to a service gateway 2336 included in data plane VCN 2318, and a DB subnet 2330 included in data plane data layer 2350. The data plane data layer 2350 may include a DB subnet 2330 that can be communicatively coupled to a service gateway 2336 included in data plane VCN 2318.

[0302] An untrusted application subnet 2362 may include primary YNICs 2364(1)-(N) that can be communicatively coupled to tenant virtual machines (VMs) 2366(1)-(N) residing in the untrusted application subnet 2362. Each tenant VM 2366(1)-(N) may execute code in its respective container 2367(1)-(N) and may be communicatively coupled to an application subnet 2326 that may be included in a data plane application layer 2346 that may be included in a container-transmitting VCN 2368. Each secondary VNIC 2372(1)-(N) can facilitate communication between the untrusted application subnet 2362 included in a data plane VCN 2318 and the application subnet included in a container-transmitting VCN 2368. The container-transmitting VCN may include a NAT gateway 2338 that can be communicatively coupled to the public internet 2354 (e.g., public internet 2054 in Figure 20).

[0303] The Internet gateway 2334 included in the control plane VCN2316 and the Internet gateway 2334 included in the data plane VCN2318 may be communicatively coupled to a metadata management service 2352 (e.g., metadata management system 2052 in Figure 20) which can be communicatively coupled to the public internet 2354. The public internet 2354 may be communicatively coupled to the Internet gateway 2334 included in the control plane VCN2316 and the NAT gateway 2338 included in the data plane VCN2318. The Internet gateway 2334 included in the control plane VCN2316 and the service gateway 2336 included in the data plane VCN2318 may be communicatively coupled to a cloud service 2356.

[0304] In some examples, the pattern shown by the architecture in block diagram 2300 of Figure 23 can be considered an exception to the pattern shown by the architecture in block diagram 2200 of Figure 22, and may be desirable for the IaaS provider's customers when the IaaS provider cannot communicate directly with the customers (e.g., in a disconnected area). Customers can access in real time each container 2367(1)~(N) contained within each customer's VM2366(1)~(N). Containers 2367(1)~(N) may be configured to call each secondary VNIC 2372(1)~(N) contained within the application subnet 2326 of the data plane application layer 2346, which may be contained within the container sending VCN 2368. Secondary VNICs 2372(1)~(N) can send calls to a NAT gateway 2338 which can send calls to the public internet 2354. In this example, the containers 2367(1) to (N) that customers can access in real time may be isolated from the control plane VCN2316 and from other entities included in the data plane VCN2318. Furthermore, the containers 2367(1) to (N) may be isolated from other customers' resources.

[0305] In another example, a customer can use containers 2367(1)-(N) to invoke cloud service 2356. In this example, the customer can execute code in containers 2367(1)-(N) to request a service from cloud service 2356. Containers 2367(1)-(N) can then send this request to secondary VNICs 2372(1)-(N), which can send the request to a NAT gateway that can send the request to the public internet 2354. The public internet 2354 can then send this request to LB subnet 2322, which is included in control plane VCN 2316, via internet gateway 2334. In response to determining that the request is valid, the LB subnet can send this request to application subnet 2326, which can then send this request to cloud service 2356 via service gateway 2336.

[0306] The illustrated IaaS architectures 2000, 2100, 2200, and 2300 may include elements other than those shown. Furthermore, the illustrated embodiments are only examples of some cloud infrastructure systems that may incorporate embodiments of this disclosure. In some other embodiments, the IaaS system may have more or fewer elements than those shown, may combine two or more elements, or may have different configurations or arrangements of elements.

[0307] In certain embodiments, the IaaS systems described herein may include a suite of applications, middleware, and database services delivered to customers in a self-service, subscription-based, flexibly scalable, reliable, highly available, and secure manner. An example of such an IaaS system is Oracle® Cloud Infrastructure (OCI), provided by the applicant.

[0308] Figure 24 shows an exemplary computer system 2400 in which various embodiments may be implemented. System 2400 may be used to implement any of the computer systems described above. As shown, computer system 2400 includes a processing unit 2404 that communicates with a number of peripheral subsystems via a bus subsystem 2402. These peripheral subsystems may include a processing acceleration unit 2406, an I / O subsystem 2408, a storage subsystem 2418, and a communication subsystem 2424. The storage subsystem 2418 includes a tangible computer-readable storage medium 2422 and system memory 2410.

[0309] The bus subsystem 2402 provides a mechanism for various components and subsystems of the computer system 2400 to communicate with each other as intended. Although the bus subsystem 2402 is schematically shown as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses. The bus subsystem 2402 may be one of several types of bus structures, including a memory bus or memory controller, a peripheral bus, and a local bus using any of the various bus architectures. For example, such architectures may include industry standard architecture (ISA) buses, microchannel architecture (MCA), bus extension ISA (EISA) buses, video electronics standards (VESA) local buses, and peripheral interconnect (PCI) buses, which can be implemented as mezzanine buses manufactured in accordance with the IEEE P1386.1 standard.

[0310] A processing unit 2404, which can be implemented as one or more integrated circuits (e.g., conventional microprocessors or microcontrollers), controls the operation of the computer system 2400. The processing unit 2404 may include one or more processors. These processors may include single-core or multi-core processors. In some embodiments, the processing unit 2404 may be implemented as one or more independent processing units 2432 and / or 2434, each containing a single-core or multi-core processor. In other embodiments, the processing unit 2404 may be implemented as a quad-core processing unit formed by integrating two dual-core processors onto a single chip.

[0311] In various embodiments, the processing unit 2404 can execute various programs in response to program code and can maintain multiple programs or processes running simultaneously. At any given time, some or all of the program code being executed can reside in the processor 2404 and / or the storage subsystem 2418. The processor 2404 can provide the various functionalities described above through appropriate programming. The computer system 2400 may further include a processing acceleration unit 2406 which may include a digital signal processor (DSP), a dedicated processor and / or the same.

[0312] The I / O subsystem 2408 may include user interface input devices and user interface output devices. User interface input devices may include pointing devices such as keyboards, mice or trackballs, touchpads or touchscreens integrated into displays, scroll wheels, click wheels, dials, buttons, switches, keypads, voice input devices with voice command recognition systems, microphones, and other types of input devices. User interface input devices may also include motion detection and / or gesture recognition devices, such as the Microsoft Kinect® motion sensor. The Microsoft Kinect® motion sensor can control and interact with input devices such as Microsoft Xbox® 360 game controllers via a natural user interface (NUI) that utilizes gestures and voice commands. User interface input devices may also include eye gesture recognition devices, such as the Google Glass® blink detector. The Google Glass® blink detector detects the user's eye activity (e.g., blinking when taking a picture and / or selecting a menu) and converts the eye activity into input to an input device (e.g., Google Glass®). Furthermore, the user interface input device may include a voice recognition detection device that enables interaction between the user and a voice recognition system (e.g., Siri® Navigator) via voice commands.

[0313] Furthermore, user interface input devices include, but are not limited to, three-dimensional (3D) mice, joysticks or pointing sticks, gamepads, graphic tablets, audio / visual devices such as speakers, digital cameras, digital video cameras, portable media players, webcams, image scanners, fingerprint scanners, barcode readers, 3D scanners, 3D printers, laser rangefinders, and eye-tracking devices. In addition, user interface input devices may include medical image input devices such as computed tomography scanners, magnetic resonance imaging scanners, ultrasound imaging scanners, and medical ultrasound devices. Furthermore, user interface input devices may include audio input devices such as MIDI keyboards and electronic musical instruments.

[0314] The user interface output device may include non-visual displays such as display subsystems, indicator lights, or audio output devices. The display subsystem may be, for example, a flat-panel display using a cathode ray tube (CRT), liquid crystal display (LCD), or plasma display, a projection device, or a touchscreen. Generally, when the term “output device” is used, it is intended to include all possible types of devices and mechanisms for outputting information from the computer system 2400 to a user or another computer. For example, the user interface output device includes, but is not limited to, various display devices for visually conveying text, images, and audio / video information, such as monitors, printers, speakers, headphones, car navigation systems, plotters, audio output devices, and modems.

[0315] The computer system 2400 may include a storage subsystem 2418. The storage subsystem 2418 comprises software elements, which, in the illustration, are located in the system memory 2410. The system memory 2410 can store program instructions that can be loaded and executed by the processing unit 2404, and data generated by the execution of these programs.

[0316] Depending on the configuration and type of the computer system 2400, the system memory 2410 may be volatile memory (e.g., random access memory: RAM) and / or non-volatile memory (e.g., read-only memory: ROM, flash memory). Generally, RAM contains data and / or program modules that the processing unit 2404 can access immediately, and / or data and / or program modules currently being operated and executed by the processing unit 2404. In some implementations, the system memory 2410 may include several different types of memory, such as static random access memory (SRAM) or dynamic random access memory (DRAM). In some implementations, a basic input / output system (BIOS), which includes basic routines that help transfer information between elements within the computer system 2400 during startup and other times, may generally be stored in ROM. As an example and not limited thereto, system memory 2410 also shows application programs 2412, program data 2414, and operating system 2416, which may include client applications, web browsers, middle-tier applications, relational database management systems (RDBMS), etc.For example, Operating System 2416 may include various versions of Microsoft Windows®, Apple Macintosh®, and / or Linux® operating systems, various commercially available UNIX® or UNIX-like operating systems (including, but not limited to, various GNU / Linux operating systems, Google Chrome® OS, etc.), and / or mobile operating systems such as iOS, Windows® Phone, Android® OS, BlackBerry® 10 OS, and Palm® OS.

[0317] Furthermore, the storage subsystem 2418 can provide a tangible, computer-readable storage medium for storing basic programming and data structures that provide the functionality of several embodiments. Software (programs, code modules, instructions) that provides the above functionality when executed by the processor may be stored in the storage subsystem 2418. These software modules or instructions may be executed by the processing unit 2404. The storage subsystem 2418 can also provide a repository for storing data used in accordance with this disclosure.

[0318] Furthermore, the storage subsystem 2400 may include a computer-readable storage medium reader 2420 that can be further connected to the computer-readable storage medium 2422. The computer-readable storage medium 2422, together with the system memory 2410, or in combination with the system memory 2410 as needed, can comprehensively represent remote storage devices, local storage devices, fixed storage devices, and / or removable storage devices, in addition to storage media for temporarily and / or permanently storing, storing, transmitting, and retrieving computer-readable information.

[0319] Furthermore, the computer-readable storage medium 2422 containing code or a portion of code may include any suitable medium known or used in the art. Such medium includes, but is not limited to, volatile and non-volatile, removable and non-removable media, which are implemented in any way or technique for storing and / or transmitting information. This may include tangible computer-readable storage media such as RAM, ROM, electronically erasable programmable ROM (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile disk (DVD), or other optical storage devices, magnetic cassettes, magnetic tapes, magnetic disk storage devices or other magnetic storage devices, or other tangible computer-readable media. It may also include intangible computer-readable media such as data signals, data transmissions, or other media that can be used to transmit desired information and are accessible by the computer system 2400.

[0320] For example, the computer-readable storage medium 2422 may include a hard disk drive that reads from or writes to a non-removable non-volatile magnetic medium, a magnetic disk drive that reads from or writes to a removable non-volatile magnetic disk, and an optical disk drive that reads from or writes to a removable non-volatile optical disk such as a CD-ROM, DVD, or Blu-ray® disc or other optical medium. The computer-readable storage medium 2422 may include, but is not limited to, a zip® drive, a flash memory card, a universal serial bus (USB) flash drive, a secure digital (SD) card, a DVD disc, a digital videotape, and the like. Furthermore, the computer-readable storage medium 2422 may include flash memory-based SSDs, enterprise flash drives, solid-state drives (SSDs) based on non-volatile memory such as solid-state ROM, SSDs based on volatile memory such as solid-state RAM, dynamic RAM, and static RAM, DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs that use a combination of DRAM and flash memory-based SSDs. Disk drives and their associated computer-readable media can provide computer system 2400 with non-volatile storage devices for computer-readable instructions, data structures, program modules, and other data.

[0321] The communication subsystem 2424 provides interfaces with other computer systems and networks. It acts as an interface for receiving data from other systems and transmitting data from computer system 2400 to other systems. For example, the communication subsystem 2424 may enable computer system 2400 to connect to one or more devices via the Internet. In some embodiments, the communication subsystem 2424 may include radio frequency (RF) transceiver components for accessing wireless voice and / or data networks (using, for example, cellular technologies such as 3G, 4G, or EDGE (enhanced data rates for global evolution), advanced data network technologies), WiFi (IEEE 802.11 family standards or other mobile communication technologies or any combination thereof), global positioning system (GPS) receiver components, and / or other components. In some embodiments, the communication subsystem 2424 may provide wired network connectivity (e.g., Ethernet®) in addition to, or instead of, the wireless interface.

[0322] In addition, in some embodiments, the communication subsystem 2424 can receive input communications on behalf of one or more users who may be using the computer system 2400, in the form of structured and / or unstructured data feeds 2426, event streams 2428, event updates 2430, etc.

[0323] For example, the communications subsystem 2424 may be configured to receive data feeds 2426 in real time from users of social networks and / or other communications services, such as Twitter® feeds, Facebook® updates, and Rich Site Summary (RSS) feeds, and / or to receive real-time updates from one or more third-party sources.

[0324] Furthermore, the communication subsystem 2424 may be configured to receive data in the form of a continuous data stream, which may include an event stream 2428 and / or event update 2430 of real-time events, which may be continuous or have no boundaries in an essentially definite-end state. Examples of applications that generate continuous data may include, for example, sensor data applications, financial tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, and automotive traffic monitoring.

[0325] The communication subsystem 2424 may also be configured to output structured and / or unstructured data feeds 2426, event streams 2428, event updates 2430, etc., to one or more databases that can communicate with one or more streaming data source computers coupled to the computer system 2400.

[0326] The computer system 2400 may be one of a variety of types, including handheld portable devices (e.g., iPhone® mobile phones, iPad® computing tablets, PDAs), wearable devices (e.g., Google Glass® head-mounted displays), PCs, workstations, mainframes, kiosks, server racks, or other data processing systems.

[0327] In the preceding description, specific details are provided for illustrative purposes to enable a full understanding of the embodiments of this disclosure. However, it will be apparent that various embodiments can be carried out without these specific details. The following description provides examples only and is not intended to limit the scope, applicability, or configuration of this disclosure. Rather, the following description of embodiments provides a possible description for carrying out the embodiments to those skilled in the art. It should be understood that various modifications can be made to the function and arrangement of elements without departing from the spirit and scope of this disclosure as set forth in the appended claims. The drawings and descriptions are not intended to be restrictive. Circuits, systems, networks, processes, and other components may be shown as components in the form of block diagrams so as not to obscure the embodiments with unnecessary details. In other embodiments, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary details so as not to obscure the embodiments. The teachings of this disclosure can also be applied to various types of applications, such as mobile applications, non-mobile applications, desktop applications, web applications, and enterprise applications. Furthermore, the teachings in this disclosure are not limited to a specific operating environment (e.g., operating system, device, platform, etc.) but can be applied to multiple different operating environments.

[0328] It should also be noted that each embodiment is described as a process shown as a flowchart, flow diagram, data flow diagram, structure diagram, or block diagram. While flowcharts describe operations as sequential processes, many operations can be performed in parallel or simultaneously. Furthermore, the order of operations may be rearranged. A process terminates when its operations are completed, but it may include additional steps not shown in the diagram. A process can correspond to a method, function, procedure, subroutine, subprogram, etc. If a process corresponds to a function, its termination can correspond to the return of the calling function or the main function.

[0329] The terms “example” and “exemplary” are used herein to mean “serving as an example, case, or illustration.” Any embodiment or design described herein as “exemplary” or “example” should not necessarily be construed as being preferable or advantageous to other embodiments or designs.

[0330] The terms “machine-readable storage medium” or “computer-readable storage medium” include, but are not limited to, portable or non-portable storage devices, optical storage devices, and various other media that can store, store, or transport instructions and / or data. Machine-readable storage medium or computer-readable storage medium may also include non-transient media that can store data and do not contain carrier waves and / or transient electronic signals that propagate over wireless or wired connections. Examples of non-transient media may include, but are not limited to, magnetic disks or tapes, optical storage media such as compact discs (CDs) or digital multipurpose discs (DVDs), flash memory, memory, or memory devices. Computer program products may include code and / or machine-executable instructions that can represent any combination of procedures, functions, subprograms, programs, routines, subroutines, modules, software packages, classes, instructions, data structures, or program statements. Code segments may be coupled to other code segments or hardware circuits by transferring and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, and data may be transmitted, transferred, or sent via any suitable means, such as memory sharing, message transfer, token transfer, or network transmission.

[0331] Furthermore, embodiments may be implemented in hardware, software, firmware, middleware, microcode, hardware description language, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segment that performs the required work may be stored in a machine-readable medium. The processor can then perform the required work. Systems shown in some drawings may be provided in various configurations. In some examples, the system may be configured as a distributed system in which one or more elements of the system are distributed across one or more networks within a cloud computing system. Where an element is described as "configured" to perform a particular operation, such a configuration may be achieved, for example, by designing electronics or other hardware to perform the operation, by programming or controlling electronics (e.g., a microprocessor or other suitable electronics) to perform the operation, or by any combination thereof.

[0332] While specific embodiments of the Disclosure have been described, various changes, modifications, alternative configurations, and equivalents are also included within the scope of the Disclosure. The embodiments of the Disclosure are not limited to operating within a specific data processing environment, but can freely operate within multiple data processing environments. Furthermore, while the embodiments of the Disclosure have been described using a set of specific procedures and steps, it will be apparent to those skilled in the art that the scope of the Disclosure is not limited to the procedures and steps described. The various features and aspects of the embodiments described above can be used individually or in combination.

[0333] Furthermore, while embodiments of the Disclosure have been described using specific combinations of hardware and software, it should be recognized that other combinations of hardware and software are also included within the scope of the Disclosure. Embodiments of the Disclosure can be implemented using hardware alone, software alone, or a combination thereof. The various processes described in the Disclosure can be run on the same processor or on different processors in any combination. Thus, where it is described that a component or module is configured to perform a particular process, that configuration can be implemented, for example, by designing an electronic circuit to perform that process, by programming a programmable electronic circuit (such as a microprocessor) to perform that process, or by a combination thereof. Processes can communicate using a variety of technologies, including but not limited to prior art for communication between processes. Different pairs of processes may use different technologies, or the same pair of processes may use different technologies at different times.

[0334] Therefore, the specification and drawings should be considered illustrative rather than restrictive. However, it will be apparent that additions, reductions, deletions, and other modifications and alterations may be made without departing from the broad spirit and scope defined by the claims. Thus, although specific embodiments of this disclosure have been described, these embodiments are not intended to be limiting. Various modifications and their equivalents are included in the appended claims.

[0335] Examples of embodiments of this disclosure may be described in view of the following sections. Paragraph 1. A method comprising determining first Layer 2 information associated with a first Layer 2 virtual switch of a customer's Layer 2 virtual network, wherein the Layer 2 virtual network is hosted by a physical network and comprises a first compute instance, a second compute instance, a first Layer 2 virtual network interface, a second Layer 2 virtual network interface, a first Layer 2 virtual switch, and a second Layer 2 virtual switch, wherein the first compute instance and the second compute instance are hosted by the same host machine or different host machines of the physical network, and the first Layer 2 virtual network interface and The method further comprises determining second Layer 2 information associated with the second Layer 2 virtual switch, generating third Layer 2 information based on the first Layer 2 information and the second Layer 2 information, and transmitting the third Layer 2 information to the customer's device.

[0336] The method according to paragraph 1, further comprising receiving the above customer input, the input indicating the customer configuration of the Layer 2 virtual network, the method further comprising determining a mapping between the above customer configuration and the distribution of the Layer 2 virtual network resources on the physical network, the third Layer 2 information further generated based on the above mapping.

[0337] The method according to paragraph 2, wherein the above customer configuration indicates a first port and a second port of the above Layer 2 virtual network, the mapping indicates that the first port corresponds to the first Layer 2 virtual network interface and the second port corresponds to the second Layer 2 virtual network interface, and the method further includes determining that the first Layer 2 virtual network interface is associated with the first Layer 2 virtual switch, determining that the second Layer 2 virtual network interface is associated with the second Layer 2 virtual switch, and indicating that the third Layer 2 information is associated with the first port and the second port.

[0338] Section 4. The method according to Section 2, wherein the above customer configuration indicates a first port, the method further comprises receiving an information query for the above customer, the information query indicates the first port, the method further comprises determining, based on the above mapping, that the first port corresponds to the first Layer 2 virtual network interface, that the first Layer 2 virtual network interface is associated with the first Layer 2 virtual switch, and generating a query result associated with the first port based on the first Layer 2 information, the third Layer 2 information includes the query result.

[0339] The method according to paragraph 2, wherein the above customer configuration indicates a first port, the method further includes receiving the above first Layer 2 information associated with the above first Layer 2 virtual switch from the above NVD, determining that the above first Layer 2 virtual switch is associated with the above first Layer 2 virtual network interface, determining, based on the above mapping, that the above first Layer 2 virtual network interface corresponds to the above first port, and including in the above third Layer 2 information at least a portion of the above first Layer 2 information associated with the above first port.

[0340] Section 6. The method described in Section 5, wherein the above-mentioned Layer 2 information is received from the NVD at least in part on the above-mentioned customer's Layer 2 information query or a change in the distribution of resources of the above-mentioned Layer 2 virtual network on the above-mentioned physical network.

[0341] Paragraph 7. The method according to any one of paragraphs 1 to 6, wherein the third Layer 2 information includes at least one of the third Layer 2 forwarding table of the Layer 2 virtual network or statistics relating to the Layer 2 virtual network.

[0342] The method according to any one of paragraphs 1 to 7, wherein the first Layer 2 information includes a first Layer 2 forwarding table of the first Layer 2 virtual switch, the second Layer 2 information includes a second Layer 2 forwarding table of the second Layer 2 virtual switch, and the third Layer 2 information includes a third Layer 2 forwarding table of the Layer 2 virtual network generated based on the first Layer 2 forwarding table and the second Layer 2 forwarding table.

[0343] The method according to paragraph 8, further comprising: receiving the customer input, the input indicating a first port of the Layer 2 virtual network; the method further comprising determining that the first port corresponds to a first Layer 2 virtual network interface; determining that the first Layer 2 virtual network interface is associated with a first Layer 2 virtual switch; determining, based on the second Layer 2 forwarding table, that the Media Access Control (MAC) address of the first Layer 2 virtual network interface is associated with a first compute instance; and including in the third Layer 2 forwarding table an instruction that the MAC address corresponds to the first port and is associated with the first compute instance.

[0344] Paragraph 10. The method according to Paragraph 8, further comprising receiving the above customer input, the input indicating a first port of the above Layer 2 virtual network; the method further comprising receiving the above customer's Layer 2 forwarding table query, the Layer 2 forwarding table query indicating the first port; the method further comprising determining that the first port corresponds to the above first Layer 2 virtual network interface; determining that the first Layer 2 virtual network interface is associated with the above first Layer 2 virtual switch; and generating query results associated with the first port based on the above first Layer 2 information, the third Layer 2 information including the above query results.

[0345] The method according to paragraph 8, further comprising: receiving the customer input, the input indicating a first port of the Layer 2 virtual network; the method further comprising: determining a change to the first Layer 2 forwarding table associated with the first Layer 2 virtual network interface; determining that the first Layer 2 virtual network interface corresponds to the first port; associating the change with the first port; and including the change in the third Layer 2 forwarding table, associated with the first port.

[0346] Paragraph 12. The method according to Paragraph 11, wherein the above change includes at least one of reassociating a first media access control (MAC) address from the second Layer 2 virtual network interface to another Layer 2 virtual network interface, or associating a second MAC address with the second Layer 2 virtual network interface.

[0347] Paragraph 13. The method according to any one of paragraphs 1 to 12, wherein the first Layer 2 information includes a first metric relating to the first frame flow of the first Layer 2 virtual switch, the second Layer 2 information includes a second metric relating to the second frame flow of the second Layer 2 virtual switch, and the third Layer 2 information includes statistics relating to the third frame flow of the Layer 2 virtual network, generated based on the first and second metrics.

[0348] The method described in paragraph 13 further comprises receiving the above customer input, the input indicating the type and target of the above statistics, the type including at least one of the number of frames, the frame size, or the amount of bits, and the target including at least one of a port, a set of ports, or the above Layer 2 virtual network.

[0349] The method according to paragraph 13, further comprising receiving from the NVD the first metric and metadata associated with the first metric, wherein the metadata identifies at least one of the Layer 2 virtual network or the first Layer 2 virtual network interface, and the statistics are further generated based at least in part on the metadata.

[0350] The method according to paragraph 15, further comprising: receiving the customer input, the input indicating a first port of the Layer 2 virtual network; the method further comprising determining, based on the metadata, that the first metric is associated with the first Layer 2 virtual network interface; determining that the first Layer 2 virtual network interface corresponds to the first port; and associating the first metric with the first port.

[0351] Section 17. A system comprising one or more processors and one or more computer-readable storage media for storing instructions, wherein, when executed by the one or more processors, the system is configured to determine first Layer 2 information associated with a first Layer 2 virtual switch of a customer's Layer 2 virtual network, the Layer 2 virtual network being hosted by a physical network and comprising a first compute instance, a second compute instance, a first Layer 2 virtual network interface, a second Layer 2 virtual network interface, a first Layer 2 virtual switch, and a second Layer 2 virtual switch, wherein the first compute instance and the second compute instance are hosted by the same host machine or different host machines of the physical network, and the first Layer 2 virtual network is hosted by the same host machine or different host machines of the physical network. A system wherein the Layer 2 virtual network interface and the first Layer 2 virtual switch are hosted by a network virtualization device (NVD) of the physical network and associated with the first compute instance, and the second Layer 2 virtual network interface and the second Layer 2 virtual switch are hosted by the NVD of the physical network or a different NVD and associated with the second compute instance, and the instruction, when executed by one or more processors, configures the system to determine second Layer 2 information associated with the second Layer 2 virtual switch, to generate third Layer 2 information based on the first Layer 2 information and the second Layer 2 information, and to transmit the third Layer 2 information to the customer's device.

[0352] Paragraph 18. The system as described in Paragraph 17, wherein the first Layer 2 information includes a first Layer 2 forwarding table of the first Layer 2 virtual switch, the second Layer 2 information includes a second Layer 2 forwarding table of the second Layer 2 virtual switch, and the third Layer 2 information includes a third Layer 2 forwarding table of the Layer 2 virtual network, which is generated based on the first Layer 2 forwarding table and the second Layer 2 forwarding table.

[0353] Clause 19. One or more non-temporary computer-readable storage media, which, when executed on a system, store instructions causing the system to perform an operation, the operation includes determining first Layer 2 information associated with a first Layer 2 virtual switch of a customer's Layer 2 virtual network, the Layer 2 virtual network being hosted by a physical network and comprising a first compute instance, a second compute instance, a first Layer 2 virtual network interface, a second Layer 2 virtual network interface, a first Layer 2 virtual switch, and a second Layer 2 virtual switch, wherein the first compute instance and the second compute instance are hosted by the same host machine or different host machines of the physical network, and the first Layer 2 virtual network is hosted by the same host machine or different host machines of the physical network. One or more non-transient computer-readable storage media, wherein the Layer 2 virtual network interface and the first Layer 2 virtual switch are hosted by a network virtualization device (NVD) of the physical network and associated with the first compute instance, and the second Layer 2 virtual network interface and the second Layer 2 virtual switch are hosted by the NVD of the physical network or a different NVD and associated with the second compute instance, and the operation further includes determining second Layer 2 information associated with the second Layer 2 virtual switch, generating third Layer 2 information based on the first Layer 2 information and the second Layer 2 information, and transmitting the third Layer 2 information to the customer's device.

[0354] Paragraph 20. One or more non-temporary computer-readable storage media as described in Paragraph 19, wherein the first Layer 2 information includes a first metric relating to the first frame flow of the first Layer 2 virtual switch, the second Layer 2 information includes a second metric relating to the second frame flow of the second Layer 2 virtual switch, and the third Layer 2 information includes statistics relating to the third frame flow of the Layer 2 virtual network, generated based on the first and second metrics.

[0355] The indefinite articles “a” / “an”, the definite article “the”, and similar references used in the context describing this disclosure (particularly in the context of the claims) should be interpreted as including both singular and plural unless otherwise specifically stated herein or unless the meaning is clearly indicated otherwise. The terms “comprising”, “having”, “including”, and “containing” should be interpreted as non-restrictive terms (i.e., “including but not limited to”) unless otherwise specifically stated. The term “connected” should be interpreted as some or all of it being contained, attached, or joined together, even if something is intervening. In this specification, enumerations of value ranges are intended simply as a shorthand way of referring to each individual value that falls within that range, and unless otherwise specifically stated herein, each individual value is incorporated herein as it is described separately herein. Unless otherwise specifically stated herein or unless the meaning is clearly indicated otherwise, all methods described herein may be performed in any appropriate order. In this specification, the use of any and all examples or exemplary language (e.g., "like") is intended to clarify embodiments of the Disclosure and, unless otherwise specified, does not limit the scope of the Disclosure. Terms in the specification should not be construed as indicating any non-claimed elements essential to the implementation of the Disclosure.

[0356] Disjunctive language, such as the phrase "at least one of X, Y, or Z," is intended to be understood in context as commonly used to indicate that an item, term, etc., may be X, Y, Z, or any combination thereof (e.g., X, Y, and / or Z), unless otherwise specified. Therefore, such disjunctive language is not generally intended, nor implies, that a particular embodiment requires the presence of at least one X, at least one Y, or at least one Z.

[0357] Preferred embodiments of the Disclosure are described herein, including the best known mode for carrying out the Disclosure. Variations of these preferred embodiments will become apparent to those skilled in the art by reading the foregoing description. Those skilled in the art may, as appropriate, adopt such modifications, and the Disclosure may be carried out in ways other than those specifically described herein. Accordingly, the Disclosure includes all variations and equivalents of the subject matter described in the claims appended herein, as permitted by applicable law. Furthermore, any combination of the above elements in all possible variations is incorporated herein unless otherwise indicated herein.

[0358] All references cited herein, including publications, patent applications, and patents, shall be incorporated by reference to the same extent as if they were included herein, provided that each reference is individually and clearly indicated as being incorporated by reference.

[0359] While aspects of the disclosure described above are explained with reference to specific embodiments, those skilled in the art will recognize that the disclosure is not limited thereto. The various features and aspects of the disclosure described above may be used individually or in combination. Furthermore, embodiments may be used in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of this specification. Accordingly, the specification and drawings should be considered illustrative rather than restrictive.< / realm>

Claims

1. A method implemented by a computer, This includes determining the first Layer 2 information associated with the first Layer 2 virtual switch of the customer's Layer 2 virtual network, The Layer 2 virtual network is hosted by a physical network and comprises a first compute instance, a second compute instance, a first Layer 2 virtual network interface, a second Layer 2 virtual network interface, a first Layer 2 virtual switch, and a second Layer 2 virtual switch. The first and second computing instances are hosted on the same or different host machines of the physical network. The first Layer 2 virtual network interface and the first Layer 2 virtual switch are hosted by a network virtualization device (NVD) of the physical network and are associated with the first compute instance. The second Layer 2 virtual network interface and the second Layer 2 virtual switch are hosted by the NVD of the physical network or a different NVD and associated with the second compute instance, and the method further, Determining the second Layer 2 information associated with the second Layer 2 virtual switch, To generate a third layer 2 information based on the first layer 2 information and the second layer 2 information, This includes transmitting the aforementioned third layer 2 information to the customer's device, The above method further, The method includes receiving the customer input, the input indicating the customer configuration of the Layer 2 virtual network, and the method further includes: A method comprising determining a mapping between the customer configuration and the distribution of resources of the Layer 2 virtual network on the physical network, wherein the third Layer 2 information is further generated based on the mapping.

2. The customer configuration indicates a first port and a second port of the Layer 2 virtual network, the mapping indicates that the first port corresponds to the first Layer 2 virtual network interface and the second port corresponds to the second Layer 2 virtual network interface, and the method further, The first Layer 2 virtual network interface is determined to be associated with the first Layer 2 virtual switch, The second Layer 2 virtual network interface is determined to be associated with the second Layer 2 virtual switch, The method according to claim 1, further comprising indicating that the third layer 2 information is associated with the first port and the second port.

3. The aforementioned customer configuration indicates a first port, and the method further, The method includes receiving an information query from the customer, wherein the information query indicates the first port, and the method further includes: Based on the mapping, it is determined that the first port corresponds to the first Layer 2 virtual network interface, The first Layer 2 virtual network interface is determined to be associated with the first Layer 2 virtual switch, The method according to claim 1, comprising generating a query result associated with the first port based on the first layer 2 information, wherein the third layer 2 information includes the query result.

4. The aforementioned customer configuration indicates a first port, and the method further, The NVD receives the first Layer 2 information associated with the first Layer 2 virtual switch, The first Layer 2 virtual switch is determined to be associated with the first Layer 2 virtual network interface, Based on the mapping, it is determined that the first Layer 2 virtual network interface corresponds to the first port, The method according to claim 1, further comprising including in the third layer 2 information at least a portion of the first layer 2 information associated with the first port.

5. The method according to claim 4, wherein the first Layer 2 information is received from the NVD at least in part on the customer's Layer 2 information query or a change in the distribution of resources of the Layer 2 virtual network on the physical network.

6. A method implemented by a computer, This includes determining the first Layer 2 information associated with the first Layer 2 virtual switch of the customer's Layer 2 virtual network, The Layer 2 virtual network is hosted by a physical network and comprises a first compute instance, a second compute instance, a first Layer 2 virtual network interface, a second Layer 2 virtual network interface, a first Layer 2 virtual switch, and a second Layer 2 virtual switch. The first and second computing instances are hosted on the same or different host machines of the physical network. The first Layer 2 virtual network interface and the first Layer 2 virtual switch are hosted by a network virtualization device (NVD) of the physical network and are associated with the first compute instance. The second Layer 2 virtual network interface and the second Layer 2 virtual switch are hosted by the NVD of the physical network or a different NVD and associated with the second compute instance, and the method further, Determining the second Layer 2 information associated with the second Layer 2 virtual switch, To generate a third layer 2 information based on the first layer 2 information and the second layer 2 information, This includes transmitting the aforementioned third layer 2 information to the customer's device, A method wherein the third layer 2 information includes at least one of the third layer 2 forwarding table of the layer 2 virtual network or statistics relating to the layer 2 virtual network.

7. A method implemented by a computer, This includes determining the first Layer 2 information associated with the first Layer 2 virtual switch of the customer's Layer 2 virtual network, The Layer 2 virtual network is hosted by a physical network and comprises a first compute instance, a second compute instance, a first Layer 2 virtual network interface, a second Layer 2 virtual network interface, a first Layer 2 virtual switch, and a second Layer 2 virtual switch. The first and second computing instances are hosted on the same or different host machines of the physical network. The first Layer 2 virtual network interface and the first Layer 2 virtual switch are hosted by a network virtualization device (NVD) of the physical network and are associated with the first compute instance. The second Layer 2 virtual network interface and the second Layer 2 virtual switch are hosted by the NVD of the physical network or a different NVD and associated with the second compute instance, and the method further, Determining the second Layer 2 information associated with the second Layer 2 virtual switch, To generate a third layer 2 information based on the first layer 2 information and the second layer 2 information, This includes transmitting the aforementioned third layer 2 information to the customer's device, A method wherein the first Layer 2 information includes a first Layer 2 forwarding table of the first Layer 2 virtual switch, the second Layer 2 information includes a second Layer 2 forwarding table of the second Layer 2 virtual switch, and the third Layer 2 information includes a third Layer 2 forwarding table of the Layer 2 virtual network, which is generated based on the first Layer 2 forwarding table and the second Layer 2 forwarding table.

8. The above method further, The method further includes receiving the customer input, the input indicating a first port of the Layer 2 virtual network, and the method further includes It is determined that the first port corresponds to the first Layer 2 virtual network interface, The first Layer 2 virtual network interface is determined to be associated with the first Layer 2 virtual switch, Based on the second Layer 2 forwarding table, it is determined that the media access control (MAC) address of the first Layer 2 virtual network interface is associated with the first compute instance, The method according to claim 7, further comprising including in the third Layer 2 forwarding table an instruction that the MAC address corresponds to the first port and is associated with the first compute instance.

9. The above method further, The method further includes receiving the customer input, the input indicating a first port of the Layer 2 virtual network, and the method further includes The method includes receiving a Layer 2 transfer table query from the customer, wherein the Layer 2 transfer table query indicates the first port, and the method further includes: It is determined that the first port corresponds to the first Layer 2 virtual network interface, The first Layer 2 virtual network interface is determined to be associated with the first Layer 2 virtual switch, The method according to claim 7, comprising generating a query result associated with the first port based on the first layer 2 information, wherein the third layer 2 information includes the query result.

10. The above method further, The method further includes receiving the customer input, the input indicating a first port of the Layer 2 virtual network, and the method further includes Determining changes to the first Layer 2 forwarding table associated with the first Layer 2 virtual network interface, The first Layer 2 virtual network interface is determined to correspond to the first port, Associating the aforementioned change with the first port, The method according to claim 7, further comprising including the change in association with the first port in the third Layer 2 transfer table.

11. The method according to claim 10, wherein the modification includes at least one of reassociating a first media access control (MAC) address from the second Layer 2 virtual network interface to another Layer 2 virtual network interface, or associating a second MAC address with the second Layer 2 virtual network interface.

12. A method implemented by a computer, This includes determining the first Layer 2 information associated with the first Layer 2 virtual switch of the customer's Layer 2 virtual network, The Layer 2 virtual network is hosted by a physical network and comprises a first compute instance, a second compute instance, a first Layer 2 virtual network interface, a second Layer 2 virtual network interface, a first Layer 2 virtual switch, and a second Layer 2 virtual switch. The first and second computing instances are hosted on the same or different host machines of the physical network. The first Layer 2 virtual network interface and the first Layer 2 virtual switch are hosted by a network virtualization device (NVD) of the physical network and are associated with the first compute instance. The second Layer 2 virtual network interface and the second Layer 2 virtual switch are hosted by the NVD of the physical network or a different NVD and associated with the second compute instance, and the method further, Determining the second Layer 2 information associated with the second Layer 2 virtual switch, To generate a third layer 2 information based on the first layer 2 information and the second layer 2 information, This includes transmitting the aforementioned third layer 2 information to the customer's device, A method wherein the first Layer 2 information includes a first metric relating to a first frame flow of the first Layer 2 virtual switch, the second Layer 2 information includes a second metric relating to a second frame flow of the second Layer 2 virtual switch, and the third Layer 2 information includes statistics relating to a third frame flow of the Layer 2 virtual network, generated based on the first and second metrics.

13. The above method further, The method according to claim 12, comprising receiving input from the customer, wherein the input indicates the type and target of the statistics, the type includes at least one of the number of frames, the frame size, or the amount of bits, and the target includes at least one of a port, a set of ports, or the Layer 2 virtual network.

14. The above method further, The method according to claim 12, comprising receiving from the NVD a first metric and metadata associated with the first metric, wherein the metadata identifies at least one of the Layer 2 virtual network or the first Layer 2 virtual network interface, and the statistics are further generated based at least in part on the metadata.

15. The above method further, The method further includes receiving the customer input, the input indicating a first port of the Layer 2 virtual network, and the method further includes Based on the metadata, it is determined that the first metric is associated with the first Layer 2 virtual network interface, The first Layer 2 virtual network interface is determined to correspond to the first port, The method according to claim 14, further comprising associating the first metric with the first port.

16. A system, One or more processors, The system comprises one or more computer-readable storage media for storing instructions, and when an instruction is executed by the one or more processors, the system Configure to determine the first Layer 2 information associated with the first Layer 2 virtual switch of the customer's Layer 2 virtual network, The Layer 2 virtual network is hosted by a physical network and comprises a first compute instance, a second compute instance, a first Layer 2 virtual network interface, a second Layer 2 virtual network interface, a first Layer 2 virtual switch, and a second Layer 2 virtual switch. The first and second computing instances are hosted on the same or different host machines of the physical network. The first Layer 2 virtual network interface and the first Layer 2 virtual switch are hosted by a network virtualization device (NVD) of the physical network and are associated with the first compute instance. The second Layer 2 virtual network interface and the second Layer 2 virtual switch are hosted by the NVD of the physical network or a different NVD, associated with the second compute instance, and the instructions, when executed by one or more processors, further enable the system, The system is configured to determine the second Layer 2 information associated with the second Layer 2 virtual switch, The system is configured to generate a third layer 2 information based on the first layer 2 information and the second layer 2 information. The third layer 2 information is configured to be transmitted to the customer's device. A system in which the first Layer 2 information includes a first Layer 2 forwarding table of the first Layer 2 virtual switch, the second Layer 2 information includes a second Layer 2 forwarding table of the second Layer 2 virtual switch, and the third Layer 2 information includes a third Layer 2 forwarding table of the Layer 2 virtual network, which is generated based on the first Layer 2 forwarding table and the second Layer 2 forwarding table.

17. A system, One or more processors, The system comprises one or more computer-readable storage media for storing instructions, and when an instruction is executed by the one or more processors, the system Configure to determine the first Layer 2 information associated with the first Layer 2 virtual switch of the customer's Layer 2 virtual network, The Layer 2 virtual network is hosted by a physical network and comprises a first compute instance, a second compute instance, a first Layer 2 virtual network interface, a second Layer 2 virtual network interface, a first Layer 2 virtual switch, and a second Layer 2 virtual switch. The first and second computing instances are hosted on the same or different host machines of the physical network. The first Layer 2 virtual network interface and the first Layer 2 virtual switch are hosted by a network virtualization device (NVD) of the physical network and are associated with the first compute instance. The second Layer 2 virtual network interface and the second Layer 2 virtual switch are hosted by the NVD of the physical network or a different NVD, associated with the second compute instance, and the instructions, when executed by one or more processors, further enable the system, The system is configured to determine the second Layer 2 information associated with the second Layer 2 virtual switch, The system is configured to generate a third layer 2 information based on the first layer 2 information and the second layer 2 information. The third layer 2 information is configured to be transmitted to the customer's device. The system is configured to receive the aforementioned customer input, the input indicating the customer configuration of the Layer 2 virtual network, and the instruction, when executed by one or more processors, further, A system configured to determine a mapping between the customer configuration and the distribution of resources in the Layer 2 virtual network on the physical network, wherein the third Layer 2 information is further generated based on the mapping.

18. A system, One or more processors, The system comprises one or more computer-readable storage media for storing instructions, and when an instruction is executed by the one or more processors, the system Configure to determine the first Layer 2 information associated with the first Layer 2 virtual switch of the customer's Layer 2 virtual network, The Layer 2 virtual network is hosted by a physical network and comprises a first compute instance, a second compute instance, a first Layer 2 virtual network interface, a second Layer 2 virtual network interface, a first Layer 2 virtual switch, and a second Layer 2 virtual switch. The first and second computing instances are hosted on the same or different host machines of the physical network. The first Layer 2 virtual network interface and the first Layer 2 virtual switch are hosted by a network virtualization device (NVD) of the physical network and are associated with the first compute instance. The second Layer 2 virtual network interface and the second Layer 2 virtual switch are hosted by the NVD of the physical network or a different NVD, associated with the second compute instance, and the instructions, when executed by one or more processors, further enable the system, The system is configured to determine the second Layer 2 information associated with the second Layer 2 virtual switch, The system is configured to generate a third layer 2 information based on the first layer 2 information and the second layer 2 information. The third layer 2 information is configured to be transmitted to the customer's device. The system wherein the third Layer 2 information includes at least one of the third Layer 2 forwarding table of the Layer 2 virtual network or statistics relating to the Layer 2 virtual network.

19. A program for causing a processor to perform the method described in any one of claims 1 to 15.

Images