Layer 2 Networking Span Ports in Virtualized Cloud Environments

JP2024503600A5Active Publication Date: 2024-11-22ORACLE INT CORP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
JP2023540148
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2021-10-05
Filing Date
2021-11-24
Publication Date
2024-11-22
Estimated Expiration
2041-11-24

AI Technical Summary

Technical Problem

Current cloud virtual networks primarily operate at Layer 3, lacking essential Layer 2 networking functionalities required by many on-premises applications, hindering seamless migration and operation of legacy applications in cloud environments.

Method used

Implementing virtualized Layer 2 networking capabilities within cloud environments, including VLANs and span port functionality, to provide emulated Layer 2 networks that support MAC learning, broadcast, and multicast, enabling seamless migration of legacy applications.

Benefits of technology

Enables seamless migration of legacy applications by providing Layer 2 networking capabilities, allowing them to operate unchanged in cloud environments, supporting non-IP protocols, and facilitating low-latency failover, while maintaining scalability and efficiency of Layer 3 networking.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 00000000_0000_ABST
    Figure 00000000_0000_ABST
Patent Text Reader

Abstract

Techniques for communication in an L2 virtual network are described. In one example, the L2 virtual network includes a plurality of L2 compute instances hosted on a set of host machines and a plurality of L2 virtual network interfaces and L2 virtual switches hosted on a set of network virtualization devices. The L2 virtual network interfaces emulate L2 ports of the L2 virtual network. Span port information applicable to the L2 ports is sent to the network virtualization devices hosting the L2 virtual network interfaces.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] REFERENCE TO RELATED APPLICATIONS This international patent application claims priority to U.S. patent application 17 / 494,719, entitled "LAYER-2 NETWORKING SPAN PORT IN A VIRTUALIZED CLOUD ENVIRONMENT," filed on October 5, 2021, which claims the benefit of U.S. provisional application 63 / 132,377, entitled "LAYER-2 NETWORKING IN A VIRTUALIZED CLOUD ENVIRONMENT," filed on December 30, 2020, the contents of which are incorporated herein by reference in their entirety for all purposes. [Background technology]

[0002] background Cloud computing provides on-demand availability of computing resources. Cloud computing can be based on data centers available to users via the Internet. Cloud computing can provide Infrastructure as a Service (IaaS). Virtual networks may be created for use by users. However, these virtual networks have limitations that limit their functionality and value. Thus, further improvements are desired. Summary of the Invention

[0003] overview The present disclosure relates to virtualized cloud environments. Techniques are described for providing Layer 2 networking functionality in a virtualized cloud environment. The Layer 2 functionality is provided in addition to, and in conjunction with, Layer 3 networking functionality provided by the virtualized cloud environment.

[0004] Some embodiments of the present disclosure relate to providing customers with layer-2 virtual local area networks (VLANs) within a private network, such as a customer's virtual cloud network (VCN). In the layer-2 VLAN, different compute instances are connected. The customer is given the perception of an emulated single switch that connects the compute instances. In fact, this emulated switch is realized as an infinitely scalable distributed switch that includes a collection of local switches. More specifically, each compute instance runs on a host machine connected to a network virtualization device (NVD). For each compute instance on a host connected to the NVD, the NVD hosts a layer-2 virtual network interface card (VNIC) associated with the compute instance and a local switch. The layer-2 VNIC represents a port of the compute instance on a layer-2 VLAN. The local switch connects the VNIC to other VNICs (e.g., other ports) associated with other compute instances in the layer-2 VLAN. Various layer-2 network services are supported, including, for example, span port functionality.

[0005] Various embodiments are described herein, including methods, systems, non-transitory computer-readable storage media that store programs, code, or instructions executable by one or more processors, and the like. [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 illustrates a high-level diagram of a distributed environment showing a virtual or overlay cloud network hosted by a cloud service provider infrastructure according to certain embodiments. [Diagram 2] 1 is an architectural schematic diagram illustrating physical elements of a physical network in a CSPI according to a particular embodiment. [Diagram 3] FIG. 2 illustrates an exemplary arrangement of CSPI in which a host machine is connected to multiple network virtualization devices (NVDs), according to certain embodiments. [Figure 4] A diagram illustrating connections between a host machine and a NVD that provides I / O virtualization to support multi-tenancy, according to certain embodiments. [Diagram 5] 1 is a schematic block diagram illustrating a physical network provided by CSPI in accordance with certain embodiments. [Figure 6] 1 is a schematic diagram of a computing network according to an embodiment. [Figure 7] 2 is a logical and hardware schematic diagram of a VLAN according to an embodiment. [Figure 8] FIG. 2 is a logical schematic diagram of multiple connected L2 VLANs according to an embodiment. [Figure 9] 9 is a logical schematic diagram of multiple connected L2 VLANs and subnets 900 according to one embodiment. [Figure 10] FIG. 2 is a schematic diagram of intra-VLAN communication and learning according to an embodiment. [Figure 11] 1 is a schematic diagram of a VLAN according to an embodiment. [Figure 12] 12 is a flow chart illustrating a process 1200 for intra-VLAN communication, according to an embodiment. [Figure 13] FIG. 2 illustrates an exemplary environment suitable for defining span ports for an L2 virtual network, according to one embodiment. [Figure 14] FIG. 2 illustrates an exemplary span port technique in a VLAN, according to an embodiment. [Figure 15] 1 is a flowchart illustrating a process for distributing span port information in a Layer 2 virtual network according to an embodiment. [Figure 16] 11 is a flowchart illustrating a process for generating span port information in accordance with an embodiment. [Figure 17] 11 is a flowchart illustrating a process for using span port information in accordance with an embodiment. [Figure 18]1 is a flowchart illustrating a process for determining whether span port information is applicable, according to an embodiment. [Figure 19] FIG. 1 is a block diagram illustrating one pattern for implementing a cloud infrastructure as a service system in accordance with at least one embodiment. [Figure 20] FIG. 2 is a block diagram illustrating another pattern for implementing a cloud infrastructure as a service system in accordance with at least one embodiment. [Figure 21] FIG. 2 is a block diagram illustrating another pattern for implementing a cloud infrastructure as a service system in accordance with at least one embodiment. [Figure 22] FIG. 2 is a block diagram illustrating another pattern for implementing a cloud infrastructure as a service system in accordance with at least one embodiment. [Figure 23] FIG. 1 is a block diagram illustrating an example computer system in accordance with at least one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Detailed Description In the following description, for purposes of explanation, specific details are set forth in order to facilitate a thorough understanding of particular embodiments. It will be apparent, however, that various embodiments may be practiced without these specific details. The figures and description are not intended to be limiting. The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment or design described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments or designs.

[0008] A. Exemplary Virtual Networking Architecture The term cloud services generally refers to services that are made available on demand (e.g., via a subscription model) to users or customers by a cloud service provider (CSP) using systems and infrastructure (cloud infrastructure). Typically, the servers and systems that make up the CSP's infrastructure are separate from the customer's own on-premise servers and systems. Thus, customers can utilize cloud services provided by the CSP without separately purchasing hardware and software resources for the services. Cloud services are designed to provide subscribing customers with easy and scalable access to applications and computing resources without the customer having to invest in procuring the infrastructure used to deliver the services.

[0009] There are several cloud service providers that offer different types of cloud services, including a variety of different types or models, such as Software-as-a-Service (SaaS), Platform-as-a-Service (PaaS), and Infrastructure-as-a-Service (IaaS).

[0010] A customer can subscribe to one or more cloud services offered by a CSP. A customer can be any entity, such as an individual, an organization, or a business. When a customer subscribes or registers for a service offered by a CSP, a tenant or account is created for the customer. The customer can then access one or more subscribed cloud resources associated with the account through this account.

[0011] As mentioned above, Infrastructure as a Service (IaaS) is one particular type of cloud computing service. In the IaaS model, the CSP provides infrastructure (called cloud service provider infrastructure or CSPI) that customers can use to build their own customizable network and deploy customer resources. Thus, the customer's resources and network are hosted in a distributed environment by the infrastructure provided by the CSP. This is different from traditional computing, where the customer's infrastructure hosts the customer's resources and network.

[0012] CSPI may include interconnected high performance computing resources including various host machines, memory resources, and network resources that form a physical network, also called a substrate network or underlay network. CSPI resources may be distributed across one or more data centers that are geographically distributed across one or more geographic regions. Virtualization software can be run by these physical resources to provide a virtualized distributed environment. Virtualization creates an overlay network (also called a software-based network, software-defined network, or virtual network) on the physical network. The CSPI physical network provides the basis for creating one or more overlay or virtual networks on top of the physical network. The virtual or overlay network can include one or more virtual cloud networks (VCNs). The virtual network is implemented using software virtualization technologies (e.g., hypervisors, functions performed by network virtualization devices (NVDs) (e.g., smart NICs), top-of-rack (TOR) switches, smart TORs that implement one or more functions performed by NVDs, and other mechanisms) to create a network abstraction layer that can run on top of the physical network. The virtual network can take various forms such as peer-to-peer networks, IP networks, etc. A virtual network is typically either a Layer 3 IP network or a Layer 2 VLAN. Such a virtual network or overlay network is often called a virtual Layer 3 network or an overlay Layer 3 network. Examples of protocols developed for virtual networks include IP-in-IP (or Generic Routing Encapsulation (GRE)), Virtual Extensible LAN (VXLAN-IETF RFC7348), Virtual Private Networks (VPNs) (e.g., MPLS Layer 3 Virtual Private Networks (RFC4364)), VMware NSX, Generic Network Virtualization Encapsulation (GENEVE), etc.

[0013] In the case of IaaS, the infrastructure provided by the CSP (CSPI) may be configured to provide virtualized computing resources over a public network (e.g., the Internet). In the IaaS model, a cloud computing service provider may host the infrastructure elements (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer), etc.). In some cases, the IaaS provider may provide various services (e.g., billing, monitoring, logging, security, load balancing and clustering, etc.) that accompany those infrastructure elements. As these services are policy-driven, IaaS users can maintain application availability and performance by implementing policies to drive load balancing. CSPI provides infrastructure and a set of complementary cloud services. This allows customers to build and run a wide range of applications and services in a highly available, hosted, distributed environment. CSPI provides high-performance computing resources and power as well as storage capacity on a flexible virtual network that can be securely accessed from various network locations such as customer on-premise networks. When a customer subscribes or registers for an IaaS service offered by a CSP, the tenancy created for that customer is a secure, isolated partition from the CSP where the customer can create, organize, and manage their cloud resources.

[0014] Customers can build their own virtual networks using the compute, memory, and networking resources provided by CSPI. On these virtual networks, they can deploy one or more customer resources or workloads, such as compute instances. For example, customers can build one or more customizable private virtual networks, called Virtual Cloud Networks (VCNs), using resources provided by CSPI. Customers can deploy one or more customer resources, e.g., compute instances, on the customer VCN. The compute instances may be virtual machines, bare metal instances, etc. Thus, CSPI provides an infrastructure and a set of complementary cloud services that enable customers to build and run various applications and services in a highly available virtual hosted environment. Customers do not manage or control the underlying physical resources provided by CSPI, but they control the operating systems, storage, and deployed applications, and in some cases have limited control over some networking components (e.g., firewalls).

[0015] The CSP may provide a console that allows customers and network administrators to configure, access, and manage resources deployed in the cloud using CSPI resources. In certain embodiments, the console provides a web-based user interface that can be used to utilize and manage the CSPI. In some embodiments, the console is a web-based application provided by the CSP.

[0016] CSPI can support single-tenancy or multi-tenancy architectures. In a single-tenancy architecture, software (e.g., applications, databases) or hardware elements (e.g., host machines or servers) serve a single customer or tenant. In a multi-tenancy architecture, software or hardware elements serve multiple customers or tenants. Thus, in a multi-tenancy architecture, CSPI resources are shared among multiple customers or tenants. In a multi-tenancy environment, CSPI has precautions and safeguards in place to ensure that each tenant's data is isolated and not visible to other tenants.

[0017] In a physical network, a network endpoint (endpoint) refers to a computing device or system that is connected to the physical network and communicates bidirectionally with the connected network. A network endpoint of a physical network may be connected to a local area network (LAN), a wide area network (WAN), or other types of physical networks. Examples of traditional endpoints of a physical network include modems, hubs, bridges, switches, routers, and other networking devices, physical computers (or host machines), and the like. Each physical device of a physical network has a fixed network address that can be used to communicate with the device. This fixed network address may be a layer 2 address (e.g., a MAC address), a fixed layer 3 address (e.g., an IP address), and the like. In a virtualization environment or virtual network, the endpoints may include various virtual endpoints, such as virtual machines hosted by elements of the physical network (e.g., hosted by a physical host machine). These endpoints of the virtual network are addressed by overlay addresses, such as overlay layer 2 addresses (e.g., an overlay MAC address) and overlay layer 3 addresses (e.g., an overlay IP address). Network overlays achieve flexibility by allowing network administrators to move overlay addresses associated with network endpoints using software management (e.g., via software implementing the virtual network's control plane). Thus, unlike physical networks, in a virtual network, overlay addresses (e.g., overlay IP addresses) can be moved from one endpoint to another using network management software. Because virtual networks are built on physical networks, both the virtual network and the underlying physical network are involved in communications between elements of the virtual network.To facilitate such communications, each element of the CSPI is configured to learn and store mappings that map overlay addresses of the virtual network to real physical addresses of the substrate network, or real physical addresses of the substrate network to overlay addresses of the virtual network. These mappings are used to facilitate communications. To facilitate virtual network routing, customer traffic is encapsulated.

[0018] Thus, physical addresses (e.g., physical IP addresses) are associated with elements of a physical network, and overlay addresses (e.g., overlay IP addresses) are associated with entities of a virtual network. Both physical and overlay IP addresses are real IP addresses. They are distinct from virtual IP addresses, which are mapped to multiple real IP addresses. A virtual IP address provides a one-to-many mapping between a virtual IP address and multiple real IP addresses.

[0019] A cloud infrastructure or CSPI is physically hosted in one or more data centers in one or more regions around the world. The CSPI may include elements of a physical or substrate network and virtualized elements of a virtual network (e.g., virtual networks, compute instances, virtual machines) built on the physical network elements. In certain embodiments, the CSPI is organized and hosted in realms, regions, and availability domains. A region is a localized geographic area that typically includes one or more data centers. Regions are generally independent of each other and may be separated by vast distances, for example across countries or continents. For example, a first region may be in Australia, another region may be in Japan, and yet another region may be in India. CSPI resources are divided among these regions such that each region has an independent subset of CSPI resources. Each region can provide a set of core infrastructure services and resources, such as compute resources (e.g., bare metal servers, virtual machines, containers and related infrastructure), storage resources (e.g., block volume storage, file storage, object storage, archive storage), networking resources (e.g., virtual cloud networks (VCNs), load balancing resources, connectivity to on-premise networks), database resources, edge networking resources (e.g., DNS), access management and monitoring resources, etc. Each region typically has multiple routes connecting it to other regions in the realm.

[0020] Typically, applications are deployed in the region where they will be used the most (i.e., on infrastructure associated with that region) because using nearby resources is faster than using resources that are farther away. Applications may also be deployed in different regions for a variety of reasons, such as redundancy to reduce risk from region-wide events such as large weather systems or earthquakes, redundancy to meet various requirements for legal jurisdictions, tax domains, and other business or societal standards.

[0021] Datacenters within a region may be further organized and subdivided into availability domains (ADs). An availability domain may correspond to one or more datacenters located in a region. A region may consist of one or more availability domains. In such a distributed environment, CSPI resources are specific to a region, such as a virtual cloud network (VCN), or specific to an availability domain, such as a compute instance.

[0022] ADs in one region are isolated from each other to be fault tolerant and configured to have a very low chance of failing at the same time. This is achieved by configuring the ADs to not share critical infrastructure resources such as networking, physical cables, cable paths, and cable entrances so that a failure of one AD in one region will have little impact on the availability of other ADs in the same region. ADs in the same region can be connected to each other with a low-latency, high-bandwidth network to provide high availability connections to other networks (e.g., the Internet, customer on-premise networks) and to build replicated systems for both high availability and disaster recovery in multiple ADs. Crowdsense utilizes multiple ADs to ensure high availability as well as protect against resource failures. As the infrastructure provided by the IaaS provider grows, more regions and ADs may be added along with additional capacity. Traffic between available domains is typically encrypted.

[0023] In certain embodiments, regions are grouped into realms. A realm is a logical collection of regions. Realms are isolated from each other and do not share any data. Regions in the same realm can communicate with each other, but regions in different realms cannot. A CSP's customer's tenancy or account exists in a single realm and can span one or more regions that belong to that single realm. Typically, when a customer subscribes to an IaaS service, a tenancy or account for that customer is created in a customer-specified region (called a "home" region) within a realm. The customer can extend the customer's tenancy to one or more other regions within the realm. The customer cannot access regions that do not exist within the realm in which the customer's tenancy resides.

[0024] An IaaS provider may offer multiple realms, each corresponding to a particular set of customers or users. For example, a commercial realm may be offered for commercial customers. As another example, a realm may be offered for a particular country or for customers in that country. As yet another example, a government realm may be offered, for example, for a government. For example, a government realm may be created for a particular government and may have a higher security level than a commercial realm. For example, Oracle® Cloud Infrastructure (OCI) currently offers a realm for the commercial domain and two realms for the government cloud domain (e.g., FedRAMP authorized and IL5 authorized).

[0025] In certain embodiments, an AD can be subdivided into one or more fault domains. A fault domain is a grouping of infrastructure resources within an AD to provide anti-affinity. A fault domain can distribute compute instances so that compute instances are not placed on the same physical hardware within an AD. This is known as anti-affinity. A fault domain refers to a collection of hardware elements (computers, switches, etc.) that share a single point of failure. A compute pool is logically divided into fault domains. Thus, a hardware failure or compute hardware maintenance event that affects one fault domain does not affect instances in other fault domains. In some embodiments, the number of fault domains in each AD may be different. For example, in certain embodiments, each AD includes three fault domains. Fault domains act as logical data centers within an AD.

[0026] When a customer subscribes to an IaaS service, resources from CSPI are provisioned to the customer and associated with the customer's tenancy. Customers can use these provisioned resources to build private networks and deploy resources on these networks. A customer network hosted on the cloud by CSPI is called a Virtual Cloud Network (VCN). Customers can configure one or more Virtual Cloud Networks (VCNs) using CSPI resources allocated for the customer. A VCN is a virtual or software-defined private network. Customer resources deployed on a customer's VCN can include compute instances (e.g., virtual machines, bare metal instances) and other resources. These compute instances may represent various customer workloads such as applications, load balancers, databases, etc. Compute instances deployed on a VCN can communicate with publicly accessible endpoints (public endpoints) over a public network such as the Internet, can communicate with other instances in the same VCN or other VCNs (e.g., other VCNs of the customer, or VCNs that do not belong to the customer), can communicate with customer on-premise data centers or networks, can communicate with sendee endpoints, and can communicate with other types of endpoints.

[0027] CSPs can offer a variety of services using CSPI. In some cases, customers of CSPI themselves can act as service providers and offer services using CSPI resources. Service providers can expose service endpoints that are characterized by identifying information (e.g., IP addresses, DNS names and ports). Customer resources (e.g., compute instances) can consume a particular service by accessing the service endpoint for that particular service exposed by the service. These service endpoints are generally publicly accessible endpoints over a public communications network, such as the Internet, with users using the public IP address associated with the endpoint. Publicly accessible network endpoints are sometimes referred to as public endpoints.

[0028] In certain embodiments, a service provider can expose a service through a service endpoint (sometimes referred to as a service endpoint). A customer of the service can access the service using this service endpoint. In certain embodiments, a service endpoint provided for a service can be accessed by multiple customers wishing to consume the service. In other implementations, a dedicated service endpoint may be provided to a customer. Thus, only that customer can access the service using that dedicated sendee endpoint.

[0029] In certain embodiments, when a VCN is created, it is associated with a private overlay Classless Inter-Domain Routing (CIDR) address space that is a private overlay IP address range (e.g., 10.0 / 16) that is assigned to the VCN. A VCN includes associated subnets, route tables, and gateways. A VCN exists within a single region, but can extend to one or more or all available domains in the region. A gateway is a virtual interface that is configured for a VCN and enables traffic communication between the VCN and one or more endpoints outside the VCN. One or more different types of gateways for a VCN can be configured to enable communication between different types of endpoints.

[0030] A VCN may be subdivided into one or more subnetworks, such as one or more subnets. A subnet is thus a unit or division that may be created within a VCN. A VCN may have one or more subnets. Each subnet in a VCN is associated with a contiguous range of overlay IP addresses (e.g., 10.0.0.0 / 24 and 10.0.1.0 / 24) that does not overlap with other subnets in the VCN and that represents an address space subset of the VCN's address space.

[0031] Each compute instance is associated with a virtual network interface card (VNIC), which allows each compute instance to participate in a subnet of a VCN. A VNIC is a logical representation of a physical network interface card (NIC). In general, a VNIC is an interface between an entity (e.g., compute instance, service) and a virtual network. A VNIC resides in a subnet and has one or more associated IP addresses and associated security rules or policies. A VNIC corresponds to a layer 2 port on a switch. A VNIC is connected to a compute instance and a subnet in a VCN. A VNIC associated with a compute instance allows the compute instance to be part of a subnet of a VCN, and allows the compute instance to communicate (e.g., send and receive packets) with endpoints on the same subnet as the compute instance, endpoints in a different subnet in the VCN, or endpoints outside the VCN. Thus, a VNIC associated with a compute instance determines how the compute instance connects with endpoints inside and outside the VCN. A VNIC for a compute instance is created and associated with the compute instance when the compute instance is created and added to a subnet in a VCN. When a subnet consists of a set of compute instances, it includes VNICs corresponding to the set of compute instances, where each VNIC is connected to a compute instance in the set of compute instances.

[0032] Each compute instance is assigned a private overlay IP address via the VNIC associated with the compute instance. This private overlay IP address is assigned to the VNIC associated with the compute instance when the compute instance is created and is used to route traffic for the compute instance. All VNICs in a particular subnet use the same route table, security lists, and DHCP options. As described above, each subnet in a VCN is associated with a contiguous range of overlay IP addresses (e.g., 10.0.0.0 / 24 and 10.0.1.0 / 24) that do not overlap with other subnets in the VCN and that represent an address space subset of the VCN's address space. For a VNIC on a particular subnet of a VCN, the overlay IP address assigned to the VNIC is an address from the contiguous range of overlay IP addresses assigned to the subnet.

[0033] In certain embodiments, a compute instance may be assigned additional overlay IP addresses, e.g., one or more public IP addresses in the case of a public subnet, in addition to the private overlay IP address, if desired. These multiple addresses are assigned to the same VNIC or multiple VNICs associated with the compute instance. However, each instance has a primary VNIC associated with an overlay private IP address that is created at instance launch and assigned to the instance. This primary VNIC cannot be deleted. Additional VNICs, called secondary VNICs, can be added to an existing instance in the same availability domain as the primary VNIC. All VNICs are in the same availability domain as the instance. The secondary VNICs may be in the same subnetwork of the primary VNIC, or in different subnetworks of the same VCN or different VCN.

[0034] Compute instances can optionally be assigned a public IP address if they are in a public subnet. When creating a subnet, you can specify the subnet as either a public or private subnet. A private subnet means that resources in the subnet (e.g., compute instances) and associated VNICs cannot have public overlay IP addresses. A public subnet means that resources in the subnet and associated VNICs can have public IP addresses. Customers can specify subnets that exist across a single available domain or multiple available domains within a region or realm.

[0035] As mentioned above, a VCN may be subdivided into one or more subnets. In certain embodiments, a virtual router (referred to as a VCN VR or simply VR) configured for a VCN allows communication between subnets of the VCN. For a subnet in a VCN, the VR represents a logical gateway for that subnet that allows the subnet (i.e., the compute instances on that subnet) to communicate with endpoints on other subnets inside the VCN and with other endpoints outside the VCN. A VCN VR is a logical entity configured to route traffic between VNICs in a VCN and a virtual gateway (gateway) associated with the VCN. Gateways are further described below with respect to FIG. 1. A VCN VR is a layer 3 / IP layer concept. In one embodiment, there is one VCN VR for a VCN. This VCN VR potentially has an unlimited number of ports addressed by IP addresses, one port for each subnet of the VCN. In this way, the VCN VR has a different IP address for each subnet of the VCN to which the VCN VR is connected. The VRs are also connected to various gateways configured for the VCN. In certain embodiments, a particular overlay IP address from a subnet's overlay IP address range is reserved in a port of the subnet's VCN VR. For example, consider a VCN with two subnets, each with associated address ranges 10.0 / 16 and 10.1 / 16. For a first subnet of the VCN with address range 10.0 / 16, addresses from this range are reserved in a port of the subnet's VCN VR. In some cases, a first IP address from this range may be reserved in a VCN VR. For example, for a subnet with overlay IP address range 10.0 / 16, IP address 10.0.0.1 may be reserved in a port of the subnet's VCN VR. For a second subnet in the same VCN with address range 10.1 / 16, the VCN VR may have a port of the second subnet with IP address 10.1.0.1.A VCN VR has a different IP address for each subnet in the VCN.

[0036] In some other embodiments, each subnet in a VCN may have a VR associated with it that is addressable by the subnet using a reserved or default IP address associated with the VR. The reserved or default IP address may be, for example, the first IP address from a range of IP addresses associated with the subnet. VNICs in a subnet can use this default or reserved IP address to communicate (e.g., send and receive packets) with the VR associated with the subnet. In such embodiments, the VR is the ingress / egress point for that subnet. VRs associated with subnets in a VCN can communicate with other VRs associated with other subnets in the VCN. VRs can also communicate with gateways associated with the VCN. The VR functions of a subnet are performed on or by one or more NVDs that perform the VNIC functions of VNICs in the subnet.

[0037] Route tables, security rules, and DHCP options may be configured for a VCN. A route table is a virtual route table for a VCN and contains rules for routing traffic from subnets inside the VCN to destinations outside the VCN through gateways or specially configured instances. You can customize a VCN's route table to control the forwarding / routing of packets into and out of the VCN. DHCP options refer to configuration information that is automatically provided to an instance when the instance is launched.

[0038] Security rules configured for a VCN represent the overlay firewall rules for the VCN. Security rules can include inbound and outbound rules and can specify the types of traffic allowed to flow in and out of instances in the VCN (e.g., based on protocol and port). Customers can choose to make certain rules stateful or stateless. For example, a customer can allow incoming SSH traffic from anywhere to a set of instances by configuring a stateful inbound rule with source CIDR 0.0.0.0 / 0 and destination TCP port 22. Security rules may be implemented using network security groups or security lists. A network security group consists of a set of security rules that apply only to resources in that group. A security list, on the other hand, contains rules that apply to all resources in a subnet that uses that security list. A VCN may include default security rules and a default security list. DHCP options configured for a VCN provide configuration information that is automatically provided when instances in the VCN launch.

[0039] In certain embodiments, configuration information for a VCN is determined and stored by a VCN control plane. The configuration information for a VCN may include, for example, address ranges associated with the VCN, subnets and associated information within the VCN, one or more VRs associated with the VCN, compute instances and associated VNICs within the VCN, NVDs performing various virtualized network functions (e.g., VNICs, VRs, gateways) associated with the VCN, state information for the VCN, and other VCN-related information. In certain embodiments, the VCN distribution service publishes the configuration information stored by the VCN control plane, or a portion thereof, to the NVD. The distributed information can be used to forward packets to and from compute instances in the VCN by updating information stored and used by the NVD (e.g., forwarding tables, routing tables, etc.).

[0040] In a particular embodiment, the creation of VCNs and subnets is handled by a VCN control plane (CP), and the launch of compute instances is handled by the compute control plane. The compute control plane is configured to allocate physical resources for the compute instances and then invoke the VCN control plane to create and connect VNICs to the compute instances. The VCN CP also sends VCN data mappings to a VCN data plane configured to perform packet forwarding and routing functions. In a particular embodiment, the VCN CP provides a distribution service configured to provide updates to the VCN data plane. Examples of the VCN control plane are shown in Figures 17, 18, 19, and 20 (see reference numbers 1716, 1816, 1916, and 2016) and described below.

[0041] Customers can create one or more VCNs with resources hosted by CSPI. Compute instances deployed on the Customer VCNs can communicate with different endpoints. These endpoints can include endpoints hosted by CSPI and endpoints external to CSPL.

[0042] Various different architectures for implementing cloud-based services using CSPI are shown in Figures 1, 2, 3, 4, 5, 17, 18, 19, and 21 and described below. Figure 1 is a high-level diagram of a distributed environment 100 showing an overlay VCN or customer VCN hosted by CSPI, according to certain embodiments. The distributed environment shown in Figure 1 includes multiple elements in an overlay network. The distributed environment 100 shown in Figure 1 is merely an example and is not intended to unduly limit the scope of the claimed embodiments. Many variations, alternatives, and modifications are possible. For example, in some implementations, the distributed environment shown in Figure 1 may have more or fewer systems or elements than those shown in Figure 1, may combine two or more systems, or may have a different system configuration or arrangement.

[0043] As shown in the example of FIG. 1, distributed environment 100 includes CSPI 101, which provides services and resources that customers can subscribe to and use to build a virtual cloud network (VCN). In a particular embodiment, CSPI 101 provides IaaS services to subscribing customers. Data centers in CSPI 101 may be organized into one or more regions. FIG. 1 shows an example of a region, “US Region” 102. A customer configures a customer VCN 104 for region 102. A customer can deploy various compute instances on VCN 104, which may include virtual machines or bare metal instances. Example instances include applications, databases, load balancers, etc.

[0044] In the embodiment shown in FIG. 1, customer VCN 104 includes two subnets, namely "Subnet-1" and "Subnet-2," each with its own CIDR IP address range. In FIG. 1, the overlay IP address range of Subnet-1 is 10.0 / 16, and the address range of Subnet-2 is 10.1 / 16. VCN virtual router 105 represents the logical gateway of the VCN that enables communication between the subnets of VCN 104 and with other endpoints outside the VCN. VCN VR 105 is configured to route traffic between VNICs in VCN 104 and the gateways associated with VCN 104. VCN VR 105 provides ports to each subnet of VCN 104. For example, VR 105 can provide a port with IP address 10.0.0.1 to Subnet-1 and a port with IP address 10.1.0.1 to Subnet-2.

[0045] Multiple compute instances can be deployed on each subnet. In this case, the compute instances can be virtual machine instances and / or bare metal instances. The compute instances in a subnet can be hosted by one or more host machines in CSPI101. The compute instances join the subnet via a VNIC associated with the compute instance. For example, as shown in FIG. 1, compute instance C1 is part of subnet-1 via a VNIC associated with the compute instance. Similarly, compute instance C2 is part of subnet-1 via a VNIC associated with C2. Similarly, multiple compute instances, which can be virtual machine instances or bare metal instances, can be part of subnet-1. Each compute instance is assigned a private overlay IP address and a MAC address via an associated VNIC. For example, in FIG. 1, compute instance C1 has an overlay IP address of 10.0.0.2 and a MAC address of M1, and compute instance C2 has a private overlay IP address of 10.0.0.3 and a MAC address of M2. Each compute instance in Subnet-1, including compute instances C1 and C2, has a default route to VCN VR105 using IP address 10.0.0.1, which is the IP address of a port in VCN VR105 in Subnet-1.

[0046] Subnet-2 may have multiple compute instances deployed thereon, including virtual machine instances and / or bare metal instances. For example, as shown in FIG. 1, compute instances D1 and D2 are part of subnet-2 via VNICs associated with the respective compute instances. In the embodiment shown in FIG. 1, compute instance D1 has an overlay IP address 10.1.0.2 and a MAC address MM1, and compute instance D2 has a private overlay IP address 10.1.0.3 and a MAC address MM2. Each compute instance in subnet-2, including compute instances D1 and D2, has a default route to VCN VR105 using IP address 10.1.0.1, which is the IP address of a port in VCN VR105 in subnet-2.

[0047] VCN A 104 may also include one or more load balancers. For example, a load balancer may be provided for a subnet and configured to load balance traffic among multiple compute instances on the subnet. A load balancer may also be provided to load balance traffic among subnets within the VCN.

[0048] A particular compute instance deployed on VCN 104 can communicate with a variety of different endpoints. These endpoints may include endpoints hosted by CSPI 200 and endpoints external to CSPI 200. Endpoints hosted by CSPI 101 may include endpoints on the same subnet as the particular compute instance (e.g., communication between two compute instances in subnet-1), endpoints in a different subnet but in the same VCN (e.g., communication between a compute instance in subnet-1 and a compute instance in subnet-2), endpoints in a different VCN in the same region (e.g., communication between a compute instance in subnet-1 and an endpoint in a VCN in the same region 106 or 110, communication between a compute instance in subnet-1 and an endpoint in a service network 110 in the same region), or endpoints in a VCN in a different region (e.g., communication between a compute instance in subnet-1 and an endpoint in a VCN in a different region 108). Also, compute instances in a subnet hosted by CSPI 101 can communicate with endpoints not hosted by CSPI 101 (i.e., external to CSPI 101). These external endpoints include endpoints in customer on-premise networks 116, endpoints in other remote cloud host networks 118, public endpoints 114 accessible via a public network such as the Internet, and other endpoints.

[0049] Communication between compute instances on the same subnet is facilitated using VNICs associated with the source and destination compute instances. For example, compute instance C1 in subnet-1 may want to send a packet to compute instance C2 in subnet-1. For a packet sent from the source compute instance whose destination is another compute instance in the same subnet, the packet is first processed by the VNIC associated with the source compute instance. The processing performed by the VNIC associated with the source compute instance may include determining the packet's destination information from the packet header, identifying any policies (e.g., security lists) configured for the VNIC associated with the source compute instance, determining the packet's next hop, performing any packet encapsulation / descapsulation functions as necessary, and forwarding / routing the packet to the next hop to facilitate communication to the packet's intended destination. If the destination compute instance is in the same subnet as the source compute instance, the VNIC associated with the source compute instance is configured to identify the VNIC associated with the destination compute instance and forward the packet for processing to that VNIC. The VNIC associated with the destination compute instance is then executed and forwards the packet to the destination compute instance.

[0050] When communicating a packet from a compute instance in a subnet to an endpoint in a different subnet of the same VCN, the communication is facilitated by the VNICs associated with the source and destination compute instances and the VCN VRs. For example, if compute instance C1 in subnet-1 of FIG. 1 wants to send a packet to compute instance D1 in subnet-2, the packet is first processed by the VNIC associated with compute instance C1. The VNIC associated with compute instance C1 is configured to route the packet to VCN VR105 using the VCN VR's default route or port 10.0.0.1. VCN VR105 is configured to route the packet to subnet-2 using port 10.1.0.1. The packet is then received and processed by the VNIC associated with D1, which forwards the packet to compute instance D1.

[0051] To communicate a packet from a compute instance in VCN 104 to an endpoint outside VCN 104, the communication is facilitated by a VNIC associated with the source compute instance, VCN VR 105, and a gateway associated with VCN 104. One or more types of gateways may be associated with VCN 104. A gateway is an interface between a VCN and another endpoint, the other endpoint being outside the VCN. A gateway is a layer 3 / IP layer concept that enables a VCN to communicate with an endpoint outside the VCN. Thus, a gateway facilitates traffic flow between a VCN and other VCNs or networks. A variety of different types of gateways may be configured in a VCN to facilitate different types of communications with different types of endpoints. With a gateway, the communications may occur over a public network (e.g., the Internet) or a private network. These communications may use a variety of communication protocols.

[0052] For example, compute instance C1 may wish to communicate with an endpoint outside of VCN104. The packet may first be processed by a VNIC associated with the source compute instance C1. The VNIC processing determines that the packet's destination is outside of Cl's subnet-1. The VNIC associated with C1 may forward the packet to VCN VR105 of VCN104. VCN VR105 then processes the packet and, as part of the processing, determines a particular gateway associated with VCN104 as the packet's next hop based on the packet's destination. VCN VR105 may then forward the packet to the particular gateway. For example, if the destination is an endpoint within the customer's operating premises network, the packet may be forwarded by VCN VR105 to a dynamic routing gateway (DRG) 122 configured for VCN104. The packet may then be forwarded from the gateway to the next hop to facilitate communication of the packet to its intended final destination.

[0053] Various different types of gateways may be configured for a VCN. Examples of gateways that may be configured for a VCN are shown in FIG. 1 and described below. Examples of gateways associated with a VCN are also shown in FIGS. 17, 18, 19, and 20 (e.g., gateways indicated by reference numbers 1734, 1736, 1738, 1834, 1836, 1838, 1934, 1936, 1938, 2034, 2036, and 2038) and described below. As shown in the embodiment shown in FIG. 1, a dynamic routing gateway (DRG) 122 may be added to or associated with a customer VCN 104. The DRG 122 provides a path for private network traffic communication between the customer VCN 104 and another endpoint. The other endpoint may be a customer on-premise network 116, a VCN 108 in a different region of the CSPI 101, or another remote cloud network 118 not hosted by the CSPI 101. The customer on-premise network 116 may be a customer network or customer data center built with the customer's resources. Access to the customer on-premise network 116 is typically highly restricted. For a customer that has both the customer on-premise network 116 and one or more VCNs 104 deployed or hosted in the cloud by the CSPI 101, the customer may want the on-premise network 116 and the cloud-based VCNs 104 to be able to communicate with each other. This allows the customer to build an extended hybrid environment that includes the customer's VCNs 104 and the on-premise network 116 hosted by the CSPI 101. The DRG 122 enables such communication. To enable such communication, a communication channel 124 is set up. In this case, one endpoint of the communication channel is in the customer on-premise network 116 and the other endpoint is in the CSPI 101 and connected to the customer VCN 104. The communication channel 124 may be via a public communication network, such as the Internet, or a private communication network.A variety of different communication protocols may be used, such as IPsec VPN technology over a public communication network such as the Internet, Oracle's FastConnect technology using a private network instead of a public network, etc. The device or equipment in the customer on-premise network 116 that forms one endpoint of the communication channel 124 is called customer premises equipment (CPE), such as CPE 126 shown in Figure 1. The endpoint on the CSPI 101 side may be a host machine running DRG 122.

[0054] In certain embodiments, a remote peering connection (RPC) can be added to a DRG, which allows a customer to peer one VCN with another VCN in another region. Using such an RPC, a customer VCN 104 can connect to a VCN 108 in another region using a DRG 122. The DRG 122 may also be used to communicate with other remote cloud networks 118 not hosted by CSPI 101, such as the Microsoft® Azure cloud, the Amazon® AWS cloud, etc.

[0055] As shown in Figure 1, an Internet Gateway (IGW) 120 can be configured in a customer VCN 104 to allow compute instances on the customer VCN 104 to communicate with public endpoints 114 accessible over a public network, such as the Internet. The IGW 120 is a gateway for connecting a VCN to a public network, such as the Internet. The IGW 120 allows public subnets in a VCN, such as VCN 104 (resources in the public subnet have public overlay IP addresses) to directly access public endpoints 112 on a public network 114, such as the Internet. The IGW 120 can be used to initiate connections from subnets in the VCN 104 or from the Internet.

[0056] A network address translation (NAT) gateway 128 can be configured in customer VCN 104. NAT gateway 128 allows cloud resources without dedicated public overlay IP addresses in the customer VCN to access the Internet without exposing them to direct incoming Internet connections (e.g., L4-L7 connections). This allows private subnets in a VCN, such as private subnet-1 in VCN 104, to privately access public endpoints on the Internet. With a NAT gateway, connections can be initiated from the private subnet to the public Internet, but connections cannot be initiated from the Internet to the private subnet.

[0057] In certain embodiments, a service gateway (SGW) 126 can be configured in the customer VCN 104. The SGW 126 provides a path for private network traffic between the VCN 104 and service endpoints supported by the service network 110. In certain embodiments, the service network 110 may be provided by a CSP and can provide a variety of services. An example of such a service network is the Oracle® Service Network, which provides a variety of services that customers can use. For example, a compute instance (e.g., a database system) in a private subnet of the customer VCN 104 can back up data to a service endpoint (e.g., an object store) without needing a public IP address or access to the Internet. In some embodiments, a VCN can have only one SGW, and connections can be initiated only from subnets in the VCN, not from the service network 110. When peering a VCN to another VCN, resources in the other VCN typically cannot access the SGW. Resources in an on-premises network connected to a VCN with FastConnect or VPN Connect can also use a service gateway configured in that VCN.

[0058] In some implementations, the SGW 126 uses service Classless Inter-Domain Routing (CIDR) labels. A CIDR label is a string that represents all the regional public IP address ranges for a service or group of services of interest. A customer uses the service CIDR label to control traffic to a service when configuring the SGW and associated routing rules. A customer can use the service CIDR label as an option when configuring security rules without having to adjust the security rules if the service's public IP address changes in the future.

[0059] A local peering gateway (LPG) 132 is a gateway that can be added to a customer VCN 104 to allow the VCN 104 to peer with another VCN in the same region. Peering means that the VCNs communicate using private IP addresses without the traffic going over a public network such as the Internet or routing the traffic through the customer on-premises network 116. In a preferred embodiment, a VCN has a separate LPG for each peering it establishes. Local peering or VCN peering is a common practice used to establish network connections between different applications or infrastructure management functions.

[0060] A service provider, such as a provider of services in service network 110, can provide access to a service using different access models. According to a public access model, the service may be exposed as a public endpoint publicly accessible by a compute instance in the customer VCN over a public network such as the Internet, or may be accessed privately through SGW 126. According to a specific private access model, the service may be accessed as a private IP endpoint in a private subnet in the customer VCN. This is called private endpoint (PE) access and allows a service provider to expose its service as an instance in the customer's private network. A private endpoint resource represents a service in a customer VCN. Each PE appears as a VNIC (called a PE-VNIC and has one or more private IPs) that the customer selects from a subnet in the customer VCN. Thus, the PE provides a way to provide a service in the customer's private VCN subnet using a VNIC. Because the endpoints are exposed as VNICs, the PE VNIC can utilize all the functionality associated with a VNIC, such as routing rules and security lists.

[0061] A service provider enables access through a PE by registering a service. The provider can associate policies with the service that regulate the visibility of the service to customer tenants. A provider can register multiple services under a single virtual IP address (VIP), especially for multi-tenant services. There can also be multiple private endpoints representing the same service (in multiple VCNs).

[0062] Compute instances in the private subnet can then access the service using the private IP address or service DNS name of the PE VNIC. Compute instances in the customer VCN can access the service by sending traffic to the private IP address of the PE in the customer VCN. A private access gateway (PAGW) 130 is a gateway resource that can connect to a service provider VCN (e.g., a VCN in the service network 110) that serves as the ingress / egress point for all traffic from / to the customer subnet private endpoints. The PAGW 130 allows the provider to scale the number of PE connections without utilizing internal IP address resources. The provider only needs to configure one PAGW for any number of services registered in a single VCN. The provider can present the services as private endpoints in multiple VCNs of one or more customers. From the customer's perspective, the PE VNIC appears not to be connected to the customer's instance, but to the service the customer wants to interact with. Traffic destined for the private endpoint is routed to the service through the PAGW 130. These are called customer-to-service private connections (C2S connections).

[0063] The PE concept can also be used to extend private access of services to customer on-premises networks and data centers by allowing traffic to flow through FastConnect / IPsec links and private endpoints in the customer VCN, and to extend private access of services to customer peering VCNs by allowing traffic to flow between LPG132 and PEs in the customer VCN.

[0064] Customers can control VCN routing at the subnet level, so they can specify which subnets in a customer VCN, such as VCN 104, use each gateway. The VCN's route tables can be used to determine whether traffic can be routed outside of the VCN through a particular gateway. For example, in a particular case, the route table for a public subnet in customer VCN 104 can send non-local traffic through IGW 120. The route table for a private subnet in the same customer VCN 104 can send traffic to CSP services through SGW 126. All remaining traffic may be sent through NAT gateway 128. The route tables control only traffic that leaves the VCN.

[0065] Security lists associated with a VCN are used to control traffic coming into the VCN through inbound connections and gateways. All resources in a subnet use the same mute table and security lists. Security lists may be used to control certain types of traffic entering and leaving instances in a subnet of a VCN. Security list rules may include inbound (inbound) rules and outbound (outbound) rules. For example, inbound rules may specify allowed source address ranges and outbound rules may specify allowed destination address ranges. Security rules may specify a particular protocol (e.g., TCP, ICMP), a particular port (e.g., port 22 for SSH, port 3389 for Windows RDP), etc. In certain implementations, the operating system of the instance may enforce its own firewall rules that match the security list rules. Rules may be stateful (e.g., connections are tracked and responses are automatically allowed without an explicit security list rule for the response traffic) or stateless.

[0066] Access from a customer VCN (i.e., resources or compute instances deployed on VCN 104) can be categorized as public access, private access, or dedicated access. Public access refers to an access model for accessing public endpoints using public IP addresses or NAT. Private access allows customer workloads in VCN 104 with private IP addresses (e.g., resources in a private subnet) to access a service without going through a public network such as the Internet. In certain embodiments, CSPI 101 allows customer VCN workloads with private IP addresses to access public service endpoints of a service using a service gateway. Thus, the service gateway provides a private access model by establishing a virtual link between the customer VCN and the public endpoints of the service that reside outside the customer private network.

[0067] Additionally, CSPI can provide dedicated public access using technologies such as FastConnect public peering, where customer on-premises instances can access one or more services in the customer VCN using a FastConnect connection without traversing a public network such as the Internet. CSPI can also provide dedicated private access using FastConnect private peering, where customer on-premises instances with private IP addresses can access customer VCN workloads using a FastConnect connection. FastConnect is a network connection used instead of connecting customer on-premises networks to CSPI and its services using the public Internet. FastConnect provides an easy, flexible, and economical way to create dedicated private connections with higher bandwidth options and a reliable and consistent networking experience compared to Internet-based connections.

[0068] FIG. 1 and the accompanying description above describe various virtualized elements in an exemplary virtual network. As mentioned above, a virtual network is built on an underlying physical or substrate network. FIG. 2 is a simplified architecture diagram illustrating physical elements in a physical network in CSPI 200 that provides the foundation for a virtual network, according to a particular embodiment. As shown, CSPI 200 provides a distributed environment that includes elements and resources (e.g., compute resources, memory resources, and networking resources) provided by a cloud service provider (CSP). These elements and resources are used to provide cloud services (e.g., IaaS services) to subscribing customers, i.e., customers who subscribe to one or more services provided by the CSP. Based on the services to which the customers subscribe, CSPI 200 provides some resources (e.g., compute resources, memory resources, and networking resources) to the customers. The customers can then build their own cloud-based (i.e., CSPI-hosted) customizable private virtual networks using the physical compute resources, memory resources, and networking resources provided by CSPI 200. As mentioned above, these customer networks are called virtual cloud networks (VCNs). Customers may deploy one or more customer resources, such as compute instances, into these Customer VCNs. The compute instances may be virtual machines, bare metal instances, etc. CSPI200 provides infrastructure and a set of complementary cloud services that enable customers to build and run a wide range of applications and services in a highly available hosted environment.

[0069] In the exemplary embodiment shown in FIG. 2, the physical elements of CSPI 200 include one or more physical host machines or servers (e.g., 202, 206, 208), network virtualization devices (NVDs) (e.g., 210, 212), top-of-rack (TOR) switches (e.g., 214, 216), a physical network (e.g., 218), and switches within physical network 218. The physical host machines or servers can host and execute various compute instances participating in one or more subnets of a VCN. The compute instances may include virtual machine instances and bare metal instances. For example, the various compute instances shown in FIG. 1 may be hosted by the physical host machines shown in FIG. 2. The virtual machine compute instances in a VCN may be executed by one host machine or may be executed by multiple different host machines. Also, the physical host machines may host virtual host machines, container-based hosts or functions, etc. The VICs and VCN VRs shown in FIG. 1 may be executed by the FTVDs shown in FIG. 2. The gateway shown in FIG. 1 may be implemented by the host machine and / or NVD shown in FIG.

[0070] A host machine or server may run a hypervisor (also called a virtual machine monitor or VMM) that creates and enables a virtualized environment on the host machine. Virtualization or a virtualized environment facilitates cloud-based computing. One or more computing instances may be created, executed, and managed on the host machine by the hypervisor on the host machine. The hypervisor on the host machine may allow the physical computing resources (e.g., computing resources, memory resources, and networking resources) of the host machine to be shared among various computing instances running on the host machine.

[0071] For example, as shown in FIG. 2, host machines 202 and 208 run hypervisors 260 and 266, respectively. These hypervisors may be implemented using software, firmware, hardware, or a combination thereof. Typically, a hypervisor is a process or software layer that resides in the host machine's operating system (OS), which runs on the host machine's hardware processors. The hypervisor provides a virtualization environment that allows the host machine's physical computing resources (e.g., processing resources such as processors / cores, memory resources, networking resources, etc.) to be shared among various virtual machine computing instances executed by the host machine. For example, in FIG. 2, hypervisor 260 resides in the OS of host machine 202 and allows the host machine's computing resources (e.g., processing resources, memory resources, and networking resources) to be shared among computing instances (e.g., virtual machines) executed by host machine 202. A virtual machine can have its own OS (called a guest OS). This guest OS may be the same as or different from the host machine's OS. The OS of a virtual machine executed by a host machine may be the same as or different from the OS of other virtual machines executed by the same host machine. Thus, the hypervisor can run multiple OSs in parallel while sharing the same computing resources of the host machine. The host machines shown in FIG. 2 may have the same type of hypervisor or different types of hypervisors.

[0072] A compute instance may be a virtual machine instance or a bare metal instance. In Figure 2, compute instance 268 on host machine 202 and compute instance 274 on host machine 208 are examples of virtual machine instances. Host machine 206 is an example of a bare metal instance provided to a customer.

[0073] In certain examples, an entire host machine may be provided to a single customer, and one or more compute instances (either virtual machines or bare metal instances) hosted by that host machine may all belong to the same customer. In other examples, a host machine may be shared among multiple customers (i.e., multiple tenants). In such a multi-tenant scenario, a host machine may host virtual machine compute instances that belong to different customers. These compute instances may be members of different VCNs for different customers. In certain embodiments, bare metal compute instances are hosted by bare metal servers that do not have a hypervisor. When bare metal compute instances are provided, a single customer or tenant maintains control of the physical CPU, memory, and network interfaces of the host machine that hosts the bare metal instance, and the host machine is not shared with other customers or tenants.

[0074] As previously mentioned, each compute instance that is part of a VCN is associated with a VNIC that enables the compute instance to be a member of a subnet of the VCN. The VNIC associated with a compute instance facilitates communication of packets or frames to and from the compute instance. The VNIC is associated with the compute instance when the compute instance is created. In certain embodiments, for a compute instance executed by a host machine, the VNIC associated with the compute instance is executed by an NVD connected to the host machine. For example, in FIG. 2, host machine 202 executes virtual machine compute instance 268 associated with VNIC 276, which is executed by NVD 210 connected to host machine 202. As another example, bare metal instance 272 hosted by host machine 206 is associated with VNIC 280, which is executed by NVD 212 connected to host machine 206. As yet another example, VNIC 284 is associated with compute instance 274 executed by host machine 208, which is executed by NVD 212 connected to host machine 208.

[0075] For a compute instance hosted by a host machine, the NVD connected to that host machine executes a VCN VR corresponding to the VCN of which the compute instance is a member. For example, in the embodiment shown in Figure 2, NVD 210 executes VCN VR 277 corresponding to the VCN of which compute instance 268 is a member. NVD 212 may also execute one or more VCN VRs 283 corresponding to the VCNs corresponding to the compute instances hosted by host machines 206 and 208.

[0076] A host machine may include one or more network interface cards (NICs) for connecting the host machine to other devices. The NICs on a host machine may provide one or more ports (or interfaces) for communicatively connecting the host machine to another device. For example, the host machine may be connected to the NVD using one or more ports (or interfaces) provided on the host machine and the NVD. The host machine may also be connected to other devices, such as other host machines.

[0077] 2, host machine 202 is connected to NVD 210 using link 220 extending between port 234 provided by NIC 232 of host machine 202 and port 236 of NVD 210. Host machine 206 is connected to NVD 212 using link 224 extending between port 246 provided by NIC 244 of host machine 206 and port 248 of NVD 212. Host machine 208 is connected to NVD 212 using link 226 extending between port 252 provided by NIC 250 of host machine 208 and port 254 of NVD 212.

[0078] Similarly, the NVDs are connected via communication links to top-of-rack (TOR) switches that are connected to a physical network 218 (also called a switch fabric). In certain embodiments, the links between the host machines and the NVDs and between the NVDs and the TOR switches are Ethernet links. For example, in FIG. 2, NVDs 210 and 212 are connected to TOR switches 214 and 216, respectively, via links 228 and 230. In certain embodiments, links 220, 224, 226, 228, and 230 are Ethernet links. The collection of host machines and NVDs connected to a TOR may be referred to as a rack.

[0079] The physical network 218 provides a communications fabric that allows the TOR switches to communicate with each other. The physical network 218 may be a multi-tier network. In a particular implementation, the physical network 218 is a multi-tier Clos network of switches, with the TOR switches 214 and 216 representing leaf-level nodes of the multi-tier and multi-node physical switching network 218. Different Clos network configurations are possible, including, but not limited to, two-tier networks, three-tier networks, four-tier networks, five-tier networks, and generally "n"-tier networks. An example of a Clos network is shown in FIG. 5 and described below.

[0080] A variety of different connection configurations are possible between the host machine and the N VDs, including one-to-one, many-to-one, and one-to-many configurations. In a one-to-one implementation, each host machine is connected to its own separate NVD. For example, in FIG. 2, host machine 202 is connected to NVD 210 via NIC 232 of host machine 202. In a many-to-one configuration, multiple host machines are connected to one NVD. For example, in FIG. 2, host machines 206 and 208 are connected to the same NVD 212 via NICs 244 and 250, respectively.

[0081] In a one-to-many configuration, one host machine is connected to multiple NVDs. Figure 3 shows an example in CSPI 300 where a host machine is connected to multiple NVDs. As shown in Figure 3, a host machine 302 includes a network interface card (NIC) 304 including multiple ports 306 and 30S. The host machine 300 is connected to a first NVD 310 via port 306 and link 320, and to a second NVD 312 via port 308 and link 322. The ports 306 and 308 may be Ethernet ports, and the links 320 and 322 between the host machine 302 and the NVDs 310 and 312 may be Ethernet links. The NVD 310 is connected to a first TOR switch 314, and the NVD 312 is connected to a second TOR switch 316. The links between the NVDs 310 and 312 and the TOR switches 314 and 316 may be Ethernet links. TOR switches 314 and 316 represent layer-0 switching devices within a multi-tier physical network 318 .

[0082] 3 provides two separate physical network paths from the physical switch network 318 to the host machine 302: a first path from the TOR switch 314 to the host machine 302 via the NVD 310, and a second path from the TOR switch 316 to the host machine 302 via the NVD 312. The separate paths provide enhanced availability (called high availability) of the host machine 302. If there is a problem with one of the paths (e.g., a link on one of the paths fails) or if there is a problem with a device (e.g., a particular NVD is not functioning), the other path can be used for communication to and from the host machine 302.

[0083] In the configuration shown in Figure 3, the host machine is connected to two different NVDs using two different ports provided by the host machine's NIC. In other embodiments, the host machine may include multiple NICs allowing the host machine to connect to multiple NVDs.

[0084] Referring again to FIG. 2, an NVD is a physical device or element that performs one or more network virtualization functions and / or storage virtualization functions. An NVD may be any device that has one or more processing units (e.g., CPU, network processing unit (NPU), FPGA, packet processing pipeline), memory including cache, and ports. Various virtualization functions may be performed by software / firmware executed by one or more processing units of the NVD.

[0085] The NVD may be implemented in a variety of different ways. For example, in a particular embodiment, the NVD is implemented as an interface card with an embedded processor, called a smart NIC or intelligent NIC. The smart NIC is a separate device from the NIC on the host machine. In FIG. 2, the NVD 210 may be implemented as a smart NIC connected to the host machine 202, and the NVD 212 may be implemented as a smart NIC connected to the host machines 206 and 208.

[0086] However, smart NICs are only one example of an NVD implementation. Various other implementations are possible. For example, in some other implementations, the NVD or one or more functions performed by the NVD may be incorporated into or performed by one or more host machines, one or more TOR switches, and other elements of CSPI200. For example, the NVD may be integrated into a host machine. In this case, the functions performed by the NVD are performed by the host machine. As another example, the NVD may be part of a TOR switch, or the TOR switch may be configured to perform the functions performed by the NVD that allow the TOR switch to perform various complex packet transformations used for public clouds. A TOR that performs the functions of the NVD may be referred to as a smart TOR. In still other implementations that provide customers with virtual machine (VM) instances rather than bare metal (BM) instances, the functions provided by the NVD may be implemented inside the hypervisor of the host machine. In some other implementations, some of the functions of the NVD may be offloaded to a centralized service running on a set of host machines.

[0087] In certain embodiments, such as when implemented as a smart NIC as shown in FIG. 2, an NVD may comprise multiple physical ports that allow the NVD to be connected to one or more host machines and one or more TOR switches. Ports on an NVD can be classified as host-facing ports (also called "south ports") or network-facing or TOR-facing ports (also called "north ports"). A host-facing port of an NVD is a port used to connect the NVD to a host machine. Examples of host-facing ports in FIG. 2 include port 236 of NVD 210, and ports 248 and 254 of NVD 212. A network-facing port of an NVD is a port used to connect the NVD to a TOR switch. Examples of network-facing ports in FIG. 2 include port 256 of NVD 210, and port 258 of NVD 212. As shown in FIG. 2, the NVD 210 is connected to the TOR switch 214 via a link 228 that extends from port 256 of NVD 210 to the TOR switch 214. Similarly, the NVD 212 is connected to the TOR switch 216 via a link 230 that extends from a port 258 of the NVD 212 to the TOR switch 216 .

[0088] The NVD can receive packets and frames (e.g., packets and frames generated by compute instances hosted by the host machine) from the host machine via the host-facing port, perform any necessary packet processing, and then forward the packets and frames to the TOR switch via the NVD's network-facing port. The NVD can receive packets and frames from the TOR switch via the NVD's network-facing port, and perform any necessary packet processing, and then forward the packets and frames to the host machine via the NVD's host-facing port.

[0089] In certain embodiments, multiple ports and associated links may be provided between the NVD and the TOR switch. These ports and links may be aggregated to form a link aggregator group (referred to as a LAG) of multiple ports or links. Link aggregation allows multiple physical links between two endpoints (e.g., between the NVD and the TOR switch) to be treated as a single logical link. All physical links in a given LAG may operate at the same speed and in full-duplex mode. LAGs help increase the bandwidth and reliability of the connection between the two endpoints. If one of the physical links in the LAG fails, the traffic is dynamically and transparently reassigned to another physical link in the LAG. The aggregated physical link provides higher bandwidth than individual links. Multiple ports associated with a LAG are treated as a single logical port. Traffic may be load balanced across the multiple physical links of the LAG. One or more LAGs may be configured between two endpoints. The two endpoints may be, for example, between the NVD and the TOR switch, or between a host machine and the NVD.

[0090] The NVD implements or performs network virtualization functions. These functions are performed by software / firmware executed by the NVD. Examples of network virtualization functions include, but are not limited to, packet encapsulation and decapsulation functions, functions for creating VCN networks, functions for implementing network policies such as VCN security list (firewall) functions, functions for facilitating routing and forwarding of packets to and from compute instances in the VCN, etc. In a particular embodiment, upon receiving a packet, the NVD is configured to execute a packet processing pipeline that processes the packet and determines how to forward or route the packet. As part of this packet processing pipeline, the NVD provides functions such as running VNICs associated with cis in the VCN, running virtual routers (VRs) associated with the VCN, encapsulating and decapsulating packets to facilitate forwarding or routing within the virtual network, running specific gateways (e.g., local peering gateways), implementing security lists, network security groups, network address translation (NAT) functions (e.g., public IP to private IP translation per host), throttling functions, and other functions.

[0091] In some embodiments, the packet processing data path in the NVD may include multiple packet pipelines. Each packet pipeline is composed of a series of packet transformation stages. In some implementations, upon receiving a packet, the packet is parsed and sorted into a single pipeline. The packet is then processed stage by stage in a linear fashion until it is either discarded or sent out through an interface of the NVD. These stages provide packet processing building blocks of basic functions (e.g., validate headers, perform throttling, insert new layer 2 headers, perform L4 firewalling, VCN encapsulation / decapsulation), such that new pipelines can be constructed by assembling existing stages, and new functionality can be added by creating and inserting new stages into existing pipelines.

[0092] The NVD can perform both control plane and data plane functions corresponding to the control and data planes of the VCN. Examples of the VCN control plane are shown in Figures 17, 18, 19, and 20 (see reference numbers 1716, 1816, 1916, and 2016) and described below. Examples of the VCN data plane are shown in Figures 17, 18, 19, and 20 (see reference numbers 1718, 1818, 1918, and 2018) and described below. The control plane functions include functions used to configure the network (e.g., setting routes and route tables, configuring VNICs) to control how data is forwarded. In certain embodiments, a VCN control plane is provided that centrally computes and exposes all overlay-to-substrate mappings to the NVD and virtual network edge devices (e.g., various gateways such as DRGs, SGWs, IGWs, etc.). The same mechanism can also be used to expose firewall rules. In certain embodiments, the NVD retrieves only mappings that are relevant to the NVD. The data plane functions include those that perform the actual routing / forwarding of packets based on the configurations set using the control plane. The VCN data plane is implemented by encapsulating customer network packets before they traverse the backbone network. The encapsulation / decapsulation functions are implemented in the NVD. In certain embodiments, the NVD is configured to intercept all network packets entering and leaving the host machine and perform network virtualization functions.

[0093] As described above, the NVD performs various virtualization functions including VNICs and VCN VRs. The NVD may perform VNICs associated with compute instances hosted by one or more host machines connected to the VNICs. For example, as shown in FIG. 2, the NVD 210 performs functions of a VNIC 276 associated with a compute instance 268 hosted by a host machine 202 connected to the NVD 210. As another example, the NVD 212 performs VNICs 280 associated with a bare metal compute instance 272 hosted by a host machine 206 and VNICs 284 associated with a compute instance 274 hosted by a host machine 208. The host machines may host compute instances that belong to different VCNs that belong to different customers. The NVDs connected to the host machines may perform VNICs (i.e., perform functions related to the VNICs) corresponding to the compute instances.

[0094] NVDs also run VCN virtual routers corresponding to the VCNs of compute instances. For example, in the embodiment shown in FIG. 2, NVD 210 runs VCN VR 277 corresponding to the VCN to which compute instance 268 belongs. NVD 212 runs one or more VCN VRs 283 corresponding to one or more VCNs to which compute instances hosted on host machines 206 and 208 belong. In certain embodiments, a VCN VR corresponding to a VCN is run by all NVDs connected to a host machine that hosts at least one compute instance that belongs to that VCN. If a host machine hosts compute instances that belong to different VCNs, the NVDs connected to that host machine may run VCN VRs corresponding to the different VCNs.

[0095] In addition to VNICs and VCN VRs, the NVD may include one or more hardware elements that run various software (e.g., daemons) and facilitate various network virtualization functions performed by the NVD. For simplicity, these various elements are grouped as "packet processing elements" shown in FIG. 2. For example, NVD 210 includes packet processing element 286, and NVD 212 includes packet processing element 288. For example, the packet processing element of the NVD may include a packet processor configured to monitor all packets received and communicated using the NVD by interacting with the ports and hardware working interfaces of the NVD and store network information. The network information may include, for example, network flow information to identify different network flows processed by the NVD and information for each flow (e.g., statistics for each flow). In certain embodiments, the network flow information may be stored on a per-VNIC basis. As another example, the packet processing element may include a replication agent configured to replicate information stored by the NVD to one or more different replication target stores. As yet another example, the packet processing element may include a logging agent configured to perform logging functions for the NVD, and may include software for monitoring the performance and health of the NVD, and possibly the status and health of other elements connected to the NVD.

[0096] FIG. 1 illustrates elements of an exemplary virtual or overlay network including a VCN, a subnet in the VCN, compute instances deployed on the subnet, VNICs associated with the compute instances, VRs of the VCN, and a set of gateways configured for the VCN. The overlay elements illustrated in FIG. 1 may be executed or hosted by one or more of the physical elements illustrated in FIG. 2. For example, the compute instances in the VCN may be executed or hosted by one or more host machines illustrated in FIG. 2. For a compute instance hosted by a host machine, the VNICs associated with the compute instance are typically executed by an NVD connected to the host machine (i.e., the VNIC functionality is provided by an NVD connected to the host machine). The VCN VR functionality of the VCN is executed by all NVDs connected to the host machines that host or execute the compute instances that are part of the VCN. The gateways associated with the VCN may be executed by one or more different types of NVDs. For example, some gateways may be executed by smart NICs and other gateways may be executed by one or more host machines or other implementations of NVDs.

[0097] As described above, a compute instance in a customer VCN can communicate with a variety of different endpoints. These endpoints may be in the same subnet as the source compute instance, in a different subnet but in the same VCN as the source compute instance, or may include endpoints outside the VCN of the source compute instance. These communications are facilitated using VNICs associated with the compute instance, VCN VRs, and gateways associated with the VCN.

[0098] Communication between two compute instances on the same subnet within a VCN is facilitated using VNICs associated with the source compute instance and the destination compute instance. The source compute instance and the destination compute instance may be hosted by the same host machine or by different host machines. A packet originating from a source compute instance may be forwarded from the host machine hosting the source compute instance to an NVD connected to that host machine. In the NVD, the packet is processed using a packet processing pipeline, which may include execution of a VNIC associated with the source compute instance. Because the destination endpoint of the packet is in the same subnet, execution of the VNIC associated with the source compute instance forwards the packet to an NVD that executes a VNIC associated with the destination compute instance, which processes the packet and forwards it to the destination compute instance. The VNICs associated with the source compute instance and the destination compute instance may be executed on the same NVD (e.g., if both the source compute instance and the destination compute instance are hosted by the same host machine) or may be executed on different NVDs (e.g., if the source compute instance and the destination compute instance are hosted by different host machines connected to different NVDs). The VNIC can use the routing / forwarding tables stored by the NVD to determine the next hop for a packet.

[0099] When communicating a packet from a compute instance in a subnet to an endpoint in a different subnet in the same VCN, the packet originating from the source compute instance is communicated from the host machine hosting the source compute instance to the NVD connected to that host machine. In the NVD, the packet is processed using a packet processing pipeline that may include running one or more VNICs and VRs associated with the VCN. For example, the NVD executes or invokes a function corresponding to a VNIC associated with the source compute instance (also referred to as executing a VNIC) as part of the packet processing pipeline. The function executed by the VNIC may include examining a VLAN tag on the packet. Because the destination of the packet is outside the subnet, a VCN VR function is invoked and executed by the NVD. The VCN VR then routes the packet to the NVD that executes the VNIC associated with the destination compute instance. The VNIC associated with the destination compute instance then processes the packet and forwards the packet to the destination compute instance. The VNICs associated with the source compute instance and the destination compute instance may run on the same NVD (e.g., if both the source compute instance and the destination compute instance are hosted by the same host machine) or may run on different NVDs (e.g., if the source compute instance and the destination compute instance are hosted by different host machines connected to different NVDs).

[0100] If the packet's destination is outside the VCN of the source compute instance, the packet originating from the source compute instance is communicated from the host machine hosting the source compute instance to the NVD connected to that host machine. The NVD runs the VNIC associated with the source compute instance. Because the packet's destination endpoint is outside the VCN, the packet is processed by the VCN VR for that VCN. The NVD invokes a VCN VR function, which may result in the packet being forwarded to an NVD running an appropriate gateway associated with the VCN. For example, if the destination is an endpoint in a customer on-premises network, the packet may be forwarded by the VCN VR to an NVD running a DRG gateway configured for the VCN. The VCN VR may run on the same NVD as the NVD running the VNIC associated with the source compute instance, or by a different NVD. The gateway may be run by an NVD that is a smart NIC, a host machine, or another NVD implementation. The packet is then processed by the gateway and forwarded to a next hop to facilitate communication of the packet to the intended destination endpoint. 2, a packet originating from compute instance 268 may be communicated from host machine 202 to NVD 210 over link 220 (using NIC 232). VNIC 276 on NVD 210 is referred to as being the VNIC associated with the source compute instance 268. VNIC 276 is configured to inspect information encapsulated within the packet, determine a next hop for forwarding the packet in order to facilitate communication of the packet to an intended destination endpoint, and forward the packet to the determined next hop.

[0101] Compute instances deployed on a VCN can communicate with a variety of different endpoints. These endpoints may include endpoints hosted by CSPI200 and endpoints external to CSPI200. Endpoints hosted by CSPI200 may include instances in the same VCN or other VCNs (which may be customer VCNs or VCNs not belonging to the customer). Communications between endpoints hosted by CSPI200 may be performed over physical network 218. Compute instances may also communicate with endpoints not hosted by or external to CSPI200. Examples of these endpoints include endpoints in a customer on-premise network or datacenter, or public endpoints accessible over a public network such as the Internet. Communications with endpoints external to CSPI200 may be performed over a public network (e.g., the Internet) (not shown in FIG. 2) or a private network (not shown in FIG. 2) using a variety of communication protocols.

[0102] The architecture of CSPI 200 shown in FIG. 2 is merely an example and is not intended to be limiting. Variations, substitutions, and modifications are possible in alternative embodiments. For example, in some implementations, CSPI 200 may have more or fewer systems or elements than those shown in FIG. 2, may combine two or more systems, or may have a different system configuration or arrangement. The systems, subsystems, and other elements shown in FIG. 2 may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective systems, hardware, or a combination thereof. The software may be stored in a non-transitory storage medium (e.g., a memory device).

[0103] FIG. 4 illustrates a connection between a host machine and an NVD to provide I / O virtualization to support multitenancy, according to certain embodiments. As shown in FIG. 4, a host machine 402 runs a hypervisor 404 that provides a virtualization environment. The host machine 402 runs two virtual machine instances: VM1 406 that belongs to customer / tenant #1 and VM2 408 that belongs to customer / tenant #2. The host machine 402 includes a physical NIC 410 that is connected to an NVD 412 via link 414. Each of the compute instances is connected to a VNIC that is executed by the NVD 412. In the embodiment of FIG. 4, VM1 406 is connected to VNIC-VM1 420, and VM2 408 is connected to VNIC-VM2 422.

[0104] 4, NIC 410 includes two logical NICs: logical NIC A 416 and logical NIC B 418. Each virtual machine is connected to and configured to operate with its own logical NIC. For example, VM1 406 is connected to logical NIC A 416, and VM2 408 is connected to logical NIC B 418. The logical NICs allow each tenant's virtual machine to believe it owns its own host machine and NIC, even though host machine 402 consists of only one physical NIC 410 shared by multiple tenants.

[0105] In a particular embodiment, each logical NIC is assigned its own VLAN ID. Thus, logical NIC A 416 for tenant #1 is assigned a particular VLAN ID, and logical NIC B 418 for tenant #2 is assigned another VLAN ID. When a packet is communicated from VM1 406, the hypervisor attaches a tag assigned to tenant #1 to the packet before communicating the packet from host machine 402 to NVD 412 over link 414. Similarly, when a packet is communicated from VM2 408, the hypervisor attaches a tag assigned to tenant #2 to the packet before communicating the packet from host machine 402 to NVD 412 over link 414. Thus, a packet 424 communicated from host machine 402 to NVD 412 has an associated tag 426 that identifies the particular tenant and associated VM. When a packet 424 is received on the NVD from host machine 402, a tag 426 associated with the packet is used to determine whether the packet should be processed by VNIC-VM1 420 or VNIC-VM2 422. The packet is then processed by the corresponding VNIC. The configuration shown in Figure 4 allows each tenant's compute instance to believe that it owns its own host machine and NIC. The configuration shown in Figure 4 provides I / O virtualization to support multi-tenancy.

[0106] FIG. 5 is a schematic block diagram illustrating a physical network 500 according to a particular embodiment. The embodiment illustrated in FIG. 5 is constructed as a Clos network. A Clos network is a particular type of network topology designed to provide connection redundancy while maintaining high bisection bandwidth and maximum resource utilization. A Clos network is a kind of non-blocking, multi-stage or multi-layer switching network, where the number of stages or layers may be 2, 3, 4, 5, etc. The embodiment illustrated in FIG. 5 is a three-layer network including layers 1, 2, and 3. A TOR switch 504 represents a layer-0 switch of the Clos network. One or more NVDs are connected to the TOR switch. The layer-0 switch is also called an edge device of the physical network. The layer-0 switch is connected to a layer-1 switch, also called a leaf switch. In the embodiment illustrated in FIG. 5, "n" layer-0 TOR switches are connected to "n" layer-1 switches to form a pod. Each layer-0 switch in a pod is interconnected to all layer-1 switches in the pod, but the switches between pods are not connected. In a particular implementation, the two pods are referred to as blocks. Each block is served by or connected to "n" layer-2 switches (also referred to as spine switches). The physical network topology may include multiple blocks. In turn, the layer-2 switches are connected to "n" layer-3 switches (also referred to as super-spine switches). Communication of packets through the physical network 500 is typically performed using one or more layer-3 communication protocols. Typically, all layers of the physical network, except the TOR layer, are n-way redundant, thus enabling high availability. The physical network can be scaled by specifying policies on the pods and blocks to control the mutual visibility of the switches of the physical network.

[0107] A feature of Clos networks is that the maximum hop count from one tier-0 switch to another tier-0 switch (or from an NVD connected to a tier-0 switch to another NVD connected to a tier-0 switch) is constant. For example, in a three-tier Clos network, a maximum of seven hops are required for a packet to reach from one NVD to another NVD. In this case, the source NVD and the target NVD are connected to the leaf tier of the Clos network. Similarly, in a four-tier Clos network, a maximum of nine hops are required for a packet to reach from one NVD to another NVD. In this case, the source NVD and the target NVD are connected to the leaf tier of the Clos network. Thus, the Clos network architecture keeps the overall network latency constant, which is important for intra-datacenter and inter-datacenter communications. Clos topologies are horizontally scalable and cost-effective. The bandwidth / throughput capacity of the network can be easily increased by adding more switches (e.g., more leaf switches and spine switches) at each tier and by increasing the number of links between switches at adjacent tiers.

[0108] In certain embodiments, each resource in the CSPI is assigned a unique identifier called a Cloud Identifier (CID). This identifier is included as part of the resource's information. This identifier can be used to manage the resource, for example, via a console or API. An exemplary syntax for a CID is as follows:

[0109] ocid1.<RESOURCE TYPE> . <realm>.[REGION].[FUTURE USE].<UNIQUE ID> In the formula: "ocid1" is a string indicating the version of the CID.

[0110] "RESOURCE TYPE" represents the type of resource (e.g., instance, volume, VCN, subnet, user, group).

[0111] "REALM" represents the realm in which the resource resides. Example values ​​include "c1" representing a commercial realm, "c2" representing a government cloud realm, or "c3" representing a federal cloud realm. Each realm can have its own domain name.

[0112] "REGION" represents the region that the resource belongs to. If a region does not apply to the resource, this part may be blank.

[0113] "FUTURE USE" indicates that the item is reserved for future use. "UNIQUE ID" is the unique ID portion. This format may vary depending on the type of resource or service.

[0114] B. Example Layer 2 VLAN Architecture This section describes techniques for providing Layer 2 networking functionality in a virtualized cloud environment. The Layer 2 functionality is provided in addition to, and in conjunction with, Layer 3 networking functionality provided by the virtualized cloud environment. In certain embodiments, the virtual Layer 2 and Layer 3 functionality is provided by Oracle Cloud Infrastructure (OCI), provided by Oracle Corporation.

[0115] After introducing Layer 2 network features, this section describes the Layer 2 realization of VLANs, followed by a description of Layer 2 VLAN services, including span port capabilities.

[0116] Preface The number of enterprise customers migrating their on-premise applications to cloud environments offered by cloud service providers (CSPs) continues to grow rapidly. However, many of these customers quickly realize that the journey to a cloud environment can be very challenging and that their existing applications need to be re-architected and re-designed to be able to run in the cloud environment. This is because applications written for on-premise environments often depend on the characteristics of the physical network for monitoring, availability, and scale. Thus, these on-premise applications need to be re-architected and re-designed before they can run in a cloud environment.

[0117] There are several reasons why on-premise applications cannot be easily migrated to cloud environments. One of the main reasons is that current cloud virtual networks operate at Layer 3 of the OSI model, e.g., the IP layer, and do not provide the Layer 2 functionality required by applications. Layer 3 based routing or forwarding involves determining where a packet should be sent (e.g., to which customer instance) based on information contained in the packet's Layer 3 header, e.g., the destination IP address contained in the packet's Layer 3 header. To facilitate this, the location of IP addresses within the virtualized cloud network are determined via a centralized control and orchestration system or controller. These may include, for example, IP addresses associated with customer entities or resources within the virtualized cloud environment.

[0118] Many customers run applications in their on-premise environments that have stringent requirements for layer 2 networking capabilities that are not currently addressed by current cloud offerings and IaaS service providers. For example, traffic is currently cloud offered and routed using layer 3 protocols that use layer 3 headers and do not support layer 2 features required by the applications. These layer 2 features may include functions such as Address Resolution Protocol (ARP) processing, Media Access Control (MAC) address learning, and layer 2 broadcast capabilities, layer 2 (MAC-based) forwarding, layer 2 networking constructs, etc. By providing virtualized layer 2 networking capabilities in a virtualized cloud network as described in this disclosure, customers can now seamlessly migrate their legacy applications to cloud environments without requiring any substantial re-architecting or re-engineering. For example, the virtualized layer 2 networking capabilities described herein enable such applications (e.g., VMware vSphere, vCenter, vSAN and NSX-T components) to communicate at layer 2 as they do in on-premise environments. These applications can run the same versions and configurations in the public cloud, allowing customers to use legacy on-premise applications, including the existing knowledge, tools and processes associated with those applications. Customers can also access native cloud services from their applications (e.g., using VMware Software Defined Data Center (SDDC)).

[0119] As another example, there are several legacy on-premise applications (e.g., enterprise clustering software applications, network virtual appliances) that require Layer 2 broadcast support for failover. Exemplary applications include Fortinet FortiGate, IBM® QRadar, Palo Alto firewalls, Cisco ASA, Juniper SRX, and Oracle RAC (Real Application Clustering). By providing virtualized Layer 2 networking in the virtualized public cloud as described in this disclosure, these appliances can now operate unmodified in the virtualized public cloud environment. As described herein, virtualized Layer 2 networking capabilities comparable to on-premise are provided. The virtualized Layer 2 networking capabilities described in this disclosure support traditional Layer 2 networking. This includes support for customer-defined VLANs, as well as unicast, broadcast, and multicast Layer 2 traffic capabilities. Layer 2-based packet routing and forwarding includes using Layer 2 protocols and using information contained in the Layer 2 header of the packet to route or forward the packet, for example, based on the destination MAC address contained in the Layer 2 header. Protocols used by enterprise applications (eg, clustering software applications), such as ARP, Gratuitous Address Resolution Protocol (GARP), and Reverse Address Resolution Protocol (RARP), can now also function in cloud environments.

[0120] There are several reasons why traditional virtualized cloud infrastructure supports virtualized Layer 3 networking and not Layer 2 networking. Layer 2 networks typically do not scale as well as Layer 3 networks. Layer 2 network control protocols do not have the level of sophistication desired for scaling. For example, Layer 3 networks do not have to worry about packet looping that Layer 2 networks must contend with. IP packets (i.e., Layer 3 packets) have a concept of time to live (TTL), whereas Layer 2 packets do not. IP addresses contained inside Layer 3 packets have a concept of topology, such as subnets, CIDR ranges, etc., whereas Layer 2 addresses (e.g., MAC addresses) do not. Layer 3 IP networks have built-in tools to facilitate troubleshooting, such as packet internet probing, route discovery, to find routing information. Such tools are not available for Layer 2. Layer 3 networks support multipathing capabilities that are not available for Layer 2. Especially due to the lack of advanced control protocols for exchanging information between entities in the network (e.g., Border Gateway Protocol (BGP) and Open Shortest Path First (OSPF)), Layer 2 networks must rely on broadcast and multicast to learn about the network, which can adversely affect network performance. As the network changes, the learning process for Layer 2 must be repeated, which is not necessary for Layer 3. For these and other reasons, it is more desirable for cloud IaaS service providers to offer infrastructure that operates at Layer 3 rather than Layer 2.

[0121] However, layer 2 functionality is required by many on-premise applications despite its multiple shortcomings. For example, assume a virtualised cloud configuration where a customer (customer 1) has two instances, instance A with IP1 and instance B with IP2, in a virtual network "V" where the instances can be compute instances (e.g. bare metal, virtual machine or container) or service instances such as load balancers, nfs mount points or other service instances. Virtual network V is a separate address space isolated from other virtual networks and the underlying physical network. For example, this isolation can be achieved using various techniques including packet encapsulation or NAT. Because of this, the IP address of an instance in the customer's virtual network is different from the address in the physical network in which it is hosted. A centralised SDN (Software Defined Networking) control plane is provided that knows the physical IPs and virtual interfaces of all virtual IP addresses. When a packet is sent from instance A to a destination of IP2 in virtual network V, the virtual network SDN stack needs to know where IP2 is located. It must know this in advance so that it can send the packet to the IP in the physical network where the virtual IP address IP2 for V is hosted. The location of a virtual IP address can be modified in the cloud, thus changing the relationship between the physical IP and the virtual IP address. Every time a virtual IP address is moved (e.g., an IP address associated with a virtual machine is moved to another virtual machine or a virtual machine is migrated to a new physical host), an API call must be made to the SDN control plane to inform the controller that the IP is being moved so that it can update all participants in the SDN stack, including the packet processor (data plane). However, there is a class of applications that do not make such API calls.Examples include various on-premise applications, applications offered by various virtualization software vendors such as VMware, etc. The value of facilitating virtual Layer 2 networking in a virtualized cloud environment allows for support of applications that are not programmed to make such API calls, or that rely on other Layer 2 networking capabilities, such as support for non-IP Layer 3 and MAC learning.

[0122] A virtual Layer 2 network creates a broadcast domain and learning is performed by the members of the broadcast domain. In a virtual Layer 2 domain, any IP can exist on any MAC on any host in this Layer 2 domain, and the system learns using standard Layer 2 networking protocols, and the system virtualizes these networking primitives without having to be explicitly told by a central controller as to where the MAC and IP reside in the virtual Layer 2 network. This allows applications that require low latency failover, applications that need to support broadcast or multicast protocols to multiple nodes, as well as legacy applications that do not know how to make API calls to an SDN control plane or API endpoints to determine where IP and MAC addresses are valid to run. Thus, providing Layer 2 networking capabilities in a virtualized cloud environment is required to support functionality that is not available at the IP Layer 3 level.

[0123] Another technical advantage of providing a virtual Layer 2 in a virtualized cloud environment is that it allows a variety of different Layer 3 protocols (IPV4, IPV6, etc.) to be supported, including non-IP protocols. For example, a variety of non-IP protocols such as IPX, AppleTalk, etc. can be supported. Existing cloud IaaS providers cannot support these non-IP protocols because they do not provide Layer 2 functionality in their virtualized cloud networks. By providing Layer 2 networking functionality as described in this disclosure, support can be provided for protocols at Layer 3 and applications that require and depend on the availability of Layer 2 level functionality.

[0124] Using the techniques described in this disclosure, both Layer 3 and Layer 2 functionality are provided in a virtualized cloud infrastructure. As previously mentioned, Layer 3-based networking provides certain efficiencies not provided by Layer 2 networking, particularly efficiencies that are well suited for scaling. Providing Layer 2 functionality in addition to Layer 3 functionality allows for leveraging such efficiencies provided by Layer 3 (e.g., to provide a more scalable solution) while providing Layer 2 functionality in a more scalable manner. For example, virtualized Layer 3 avoids having to use broadcast for learning purposes. By providing Layer 3 for its efficiency and at the same time providing virtualized Layer 2 to enable applications that require it and cannot function without Layer 2 functionality, and to support non-IP protocols, etc., complete flexibility in a virtualized cloud environment is provided to customers.

[0125] The customers themselves have hybrid environments where a Layer 2 environment exists alongside a Layer 3 environment, and the virtualized cloud environment can now support both of these environments. Customers can have Layer 3 networks, such as subnets, and / or Layer 2 networks, such as VLANs, and these two environments can communicate with each other within the virtualized cloud environment.

[0126] Virtualized cloud environments also need to support multi-tenancy, which makes it technically difficult and complex to provision both Layer 3 and Layer 2 functions in the same virtualized cloud environment. For example, Layer 2 broadcast domains must be managed across many different customers within a cloud provider's infrastructure. The embodiments described in this disclosure overcome these technical problems.

[0127] For virtualization providers (e.g., VMware), a virtualized Layer 2 network that emulates a physical Layer 2 network allows workloads to run unmodified. Applications provided by such virtualization providers can then run on the virtualized Layer 2 network provided by the cloud infrastructure. For example, such applications may include a set of instances that need to run on a Layer 2 network. If a customer wants to lift and shift such applications from their on-premise environment to a virtualized cloud environment, they cannot simply take the application and run it in the cloud, because these applications rely on underlying Layer 2 networks that are not provided by current virtualized cloud providers (e.g., Layer 2 network functions are used to perform virtual machine migrations or moves where MAC and IP addresses are valid). For these reasons, such applications cannot run natively in a virtualized cloud environment. Using the techniques described herein, in addition to providing a virtualized Layer 3 network, the cloud provider also provides a virtualized Layer 2 network. Now, such application stacks can run unmodified in the cloud environment, and nested virtualization can be performed in the cloud environment. Customers can now run their own Layer 2 applications in the cloud and manage them. Application providers do not need to make changes to their software to facilitate this. Such legacy applications or workloads (e.g., legacy load balancers, legacy applications, KVM, Openstack, clustering software) can now run unmodified in a virtualized cloud environment.

[0128] By providing virtualized Layer 2 functionality as described herein, a variety of Layer 3 protocols, including non-IP protocols, can now be supported by the virtualized cloud environment. Taking Ethernet as an example, a variety of different EtherTypes (fields in the Layer 2 header that tell what type of Layer 3 packet is being sent; what protocols to expect at Layer 3) can be supported, including a variety of non-IP protocols. EtherTypes are two-octet fields in an Ethernet frame. They are used to indicate which protocol is encapsulated in the payload of the frame and are used at the receiving end by the Data Link Layer to determine how the payload is processed. EtherTypes are also used as the basis for 802.1Q VLAN tagging, encapsulating packets from a VLAN for transmission on an Ethernet trunk where they are multiplexed with other VLAN traffic. Examples of EtherTypes include IPV4, IPV6, Address Resolution Protocol (ARP), AppleTalk, IPX, etc. A cloud network that supports Layer 2 protocols can support any protocol at the Layer 3 layer. Similarly, when the cloud infrastructure provides support for layer 3 protocols, the cloud infrastructure can support a variety of protocols at layer 4, such as TCP, UDP, ICMP, etc. The network can be layer 4 protocol agnostic when virtualization is provided at layer 3. Similarly, the network can be layer 3 protocol agnostic when virtualization is provided at layer 2. This technique can be extended to support any layer 2 network type, including FDDI, InfiniBand, etc.

[0129] Thus, many applications written for physical networks, especially those that operate with clusters of computing nodes that share a broadcast domain, use layer 2 features that are not supported in L3 virtual networks. The following six examples highlight the complications that can result from not providing layer 2 networking capabilities: (1) MAC and IP allocation without a preceding API call. Network appliances and hypervisors (such as VMware) were not built for cloud virtual networks. They assume they can use any MAC as long as it is unique, and they can get a dynamic address from a DHCP server or use any IP assigned to the cluster. There is often no mechanism that they can be configured to inform the control plane about these Layer 2 and Layer 3 address allocations. If they don't know where the MAC and IP are, the Layer 3 virtual network doesn't know where to send traffic. (2) Low latency reallocation of MAC and IP for high availability and live migration. Many on-premise applications use ARP to reallocate IPs and MACs for high availability - when an instance in a cluster or HA pair stops responding, the newly active instance sends a Gratuitous ARP (GARP) to reallocate a service IP to its MAC or a Reverse ARP (RARP) to reallocate a service MAC to its interface. This is also important when live migrating an instance on a hypervisor: the new host must send a RARP when a guest moves so that guest traffic is sent to the new host. The allocation must not only be done without an API call, but also with very low latency (sub-milliseconds). This cannot be achieved with HTTPS calls to a REST endpoint. (3) Interface multiplexing by MAC address. When a hypervisor hosts multiple virtual machines on a single host, all of which are on the same network, the guest interfaces are differentiated by their MAC. This requires support for multiple MACs on the same virtual interface. (4) VLAN support. A single physical virtual machine host may need to be on multiple broadcast domains, as indicated by the use of VLAN tags. For example, VMware ESX uses VLANs for traffic separation (e.g., guest virtual machines may communicate on one VLAN, storage on another, and host virtual machines on yet another). (5) Use of broadcast and multicast traffic. ARP requires L2 broadcasts, and there are examples of on-premise applications that use broadcast and multicast traffic for cluster and HA applications. (6) Support for non-IP traffic. Since L3 networks require IPv4 or IPv6 headers to communicate, the use of L3 protocols other than IP will not work. L2 virtualization means that networks within a VLAN can be L3 protocol agnostic - the L3 headers may be IPv4, IPv6, IPX, or something else - or even may not exist at all.

[0130] Example of Layer 2 VLAN implementation As disclosed herein, a Layer 2 (L2) network can be created in a cloud network. This virtual L2 network includes one or several Layer 2 virtual networks, such as virtualized L2 VLANs, referred to herein as VLANs. Each VLAN can include multiple compute instances, each of which can be associated with at least one L2 virtual network interface (e.g., L2 VNIC) and L2 virtual switch. In some embodiments, each pair of L2 virtual network interface and L2 virtual switch is hosted on an NVD. The NVD can host multiple such pairs, each pair associated with a different compute instance. A collection of L2 virtual switches represents a single emulated L2 switch for a VLAN. A L2 virtual network interface represents a collection of L2 ports on the single emulated L2 switch. A VLAN can be connected to other VLANs, Layer 3 (L3) networks, on-premise networks, and / or other networks via a VLAN Switching and Routing Service (VSRS), also referred to herein as a Real Virtual Router (RVR) or L2 VSRS. An example of this architecture is described below.

[0131] Referring now to FIG. 6, a schematic diagram of one embodiment of a computing network is shown. A VCN 602 resides in a CSPI 601. The VCN 602 includes multiple gateways that connect the VCN 602 to other networks. These gateways include a DRG 604 that can connect the VCN 602 to an on-premise network, such as, for example, an on-premise data centre 606. The gateways may further include a gateway 600 that may include, for example, an LPG to connect the VCN 602 to another VCN, and / or an IGW and / or a NAT gateway to connect the VCN 602 to the Internet. The gateways of the VCN 602 may further include a service gateway 610 that can connect the VCN 602 to a service network 612. The service network 612 may include one or several databases and / or stores, including, for example, an autonomous database 614 and / or an object store 616. The service network may include a conceptual network that includes an aggregation of IP ranges, which may be, for example, public IP ranges. In some embodiments, these IP ranges may cover some or all of the public services offered by the CSPI 601 providers. These services can be accessed, for example, through an Internet gateway or a NAT gateway. In some embodiments, the service network provides a way for services in the service network to be accessed from a local area through a gateway dedicated for that purpose (the service gateway). In some embodiments, the backends of these services can be realized, for example, in their own private networks. In some embodiments, the service network 612 can include further additional databases.

[0132] VCN 602 can include multiple virtual networks. These networks can each include one or several compute instances, and the one or several compute instances can communicate within their respective networks, between networks, or outside of VCN 602. One of the virtual networks of VCN 602 is L3 subnet 620. L3 subnet 620 is a unit or subdivision of configuration created within VCN 602. Subnet 620 can include a virtual layer 3 network in the virtualized cloud environment of VCN 602, which is hosted on the underlying physical network of CPSI 601. Although FIG. 6 shows a single subnet 620, VCN 602 can have one or more subnets. Each subnet in VCN 602 can be associated with a contiguous range of overlay IP addresses (e.g., 10.0.0.0 / 24 and 10.0.1.0 / 24) that does not overlap with other subnets in that VCN and represents an address space subset within the VCN's address space. In some embodiments, this IP address space may be segregated from the address space associated with CPSI 601.

[0133] Subnet 620 includes one or more compute instances, specifically a first compute instance 622-A and a second compute instance 622-B. Compute instances 622-A, 622-B can communicate with each other within subnet 620 or with other instances, devices, and / or networks outside of subnet 620. Communications outside of subnet 620 are enabled by virtual router (VR) 624. VR 624 enables communication between subnet 620 and other networks in VCN 602. In the case of subnet 620, VR 624 represents a logical gateway that enables subnet 620 (e.g., compute instances 622-A, 622-B) to communicate with endpoints on other networks in VCN 602, as well as with other endpoints outside of VCN 602.

[0134] VCN 602 may further include additional networks, specifically one or several L2 VLANs (referred to herein as VLANs), which are examples of virtual L2 networks. Each of these one or several VLANs may include a virtual layer 2 network that is localized in the cloud environment of VCN 602 and / or hosted by the underlying physical network of CPSI 601. In the embodiment of FIG. 6, VCN 602 includes VLAN A 630 and VLAN B 640. Each VLAN 630, 640 in VCN 602 may be associated with a contiguous range of overlay IP addresses (e.g., 10.0.0.0 / 24 and 10.0.1.0 / 24) that does not overlap with other networks in the VCN, such as other subnets or VLANs in the VCN, and that represents a subset of address space within the VCN's address space. In some embodiments, this IP address space of the VLAN may be isolated from the address space associated with CPSI 601. Each of the VLANs 630, 640 may include one or more compute instances, specifically, VLAN A 630 may include, for example, a first compute instance 632-A and a second compute instance 632-B. In some embodiments, VLAN A 630 may include additional compute instances. VLAN B 640 may include, for example, a first compute instance 642-A and a second compute instance 642-B. Each of the compute instances 632-A, 632-B, 642-A, 642-B may have an IP address and a MAC address. These addresses may be assigned or generated in any desired manner. In some embodiments, these addresses may be within the CIDR of the compute instance's VLAN, and in some embodiments, these addresses may be any address. In embodiments where the compute instances of the VLAN communicate with endpoints outside the VLAN, one or both of these addresses are from the VLAN CIDR, but if all communication is intra-VLAN, these addresses are not limited to addresses within the VLAN CIDR.In contrast to networks where addresses are assigned by the control plane, the IP and / or MAC addresses of compute instances in a VLAN may be assigned by users / customers of that VLAN, and these IP and / or MAC addresses may then be discovered and / or learned by the compute instances in the VLAN according to the process for learning discussed below.

[0135] Each VLAN may include a VLAN Switching and Routing Service (VSRS); specifically, VLAN A 630 includes VSRS A 634, and VLAN B 640 includes VSRS B 644. Each VSRS 634, 644 participates in Layer 2 switching and local learning within the VLAN, and also performs all necessary Layer 3 network functions including ARP, NDP, and routing. The VSRS performs ARP (which is a Layer 2 protocol) since it must map IP to MAC.

[0136] In these cloud-based VLANs, each virtual interface or virtual gateway can be associated with one or more media access control (MAC) addresses, which can be virtual MAC addresses. Within a VLAN, one or more compute instances 632-A, 632-B, 642-A, 642-B, which can be bare metal, VMs, or containers, and / or one or more service instances can communicate directly with each other through virtual switches. Communication outside of the VLAN, such as with other VLANs or L3 networks, is enabled through the VSRS 634, 644. The VSRS 634, 644 is a distributed service that provides Layer 3 functions, such as IP routing, for the VLAN network. In some embodiments, the VSRS 634, 644 is a horizontally scalable, highly available routing service that sits at the intersection of IP and L2 networks and can participate in IP routing and L2 learning within the cloud-based L2 domain.

[0137] The VSRS 634, 644 may be distributed across multiple nodes in the infrastructure, and the VSRS 634, 644 functionality may be scalable, specifically horizontally scalable. In some embodiments, each of the nodes implementing the functionality of the VSRS 634, 644 share and replicate the functionality of the router and / or switch with each other. Furthermore, these nodes may present themselves as a single VSRS 634, 644 to all of the instances in the VLAN 630, 640. The VSRS 634, 644 may be implemented on any virtualization device in the CSPI 601, specifically in a virtual network. Thus, in some embodiments, the VSRS 634, 644 may be implemented on any of the virtual network virtualization devices including NICs, SmartNICs, switches, Smart switches, or general-purpose compute hosts.

[0138] The VSRS 634, 644 can be a service that resides on one or several hardware nodes, such as one or several servers, such as one or several x86 servers, or one or several networking devices, such as one or several NICs, specifically one or several SmartNICs, that support a cloud network. In some embodiments, the VSRS 634, 644 can be realized on a server fleet. Thus, the VSRS 634, 644 can be a service that is distributed across a fleet of nodes, which may be a centrally managed fleet or distributed to the edge, of a virtual networking practitioner that participates in and shares L2 and L3 learning along with evaluating routing and security policies. In some embodiments, each of the VSRS instances can update other VSRS instances with new mapping information as this information is learned by the VSRS instance. For example, when a VSRS instance learns IP, interface, and / or MAC mappings for one or several CIs in its VLAN, the VSRS instance can provide the updated information to other VSRS instances in the VCN. Through this cross-update, the VSRS instance associated with the first VLAN can learn mappings including IP, interface, and / or MAC mappings for CIs in other VLANs, and in some embodiments, can learn mappings including IP, interface, and / or MAC mappings for CIs in other VLANs within VCN 602. When the VSRS resides on a server fleet and / or is distributed across a fleet of nodes, these updates can be greatly expedited.

[0139] In some embodiments, the VSRS 634, 644 may also host one or several higher level services required for networking, including, but not limited to, DHCP relay; DHCP (hosting); DHCPv6; neighbor discovery protocols such as IPv6 Neighbor Discovery Protocol; DNS; hosting DNSv6; SLAAC for IPv6; NTP; metadata services; and block store mount points. In some embodiments, the VSRS may support one or more network address translation (NAT) functions for translating between multiple network address spaces. In some embodiments, the VSRS may incorporate anti-spoofing, anti-MAC spoofing, ARP cache poisoning protection for IPv4, IPv6 Router Advertisement (RA) guard, DHCP guard, packet filtering using Access Control Lists (ACLs); and / or reverse path forwarding checks. The VSRS may implement functions including, for example, ARP, GARP, packet filters (ACLs), DHCP relay, and / or IP routing protocols. The VSRS 634, 644 can, for example, learn MAC addresses, invalidate expired MAC addresses, handle MAC address movement, look up MAC address information, handle flooding of MAC information, handle storm control, loop prevention, Layer 2 multicast via protocols such as IGMP in the cloud, statistics collection including logging, statistics using SNMP, monitoring, and / or collect and use statistics about broadcasts, total traffic, bits, spanning tree packets, etc.

[0140] Within the virtual network, the VSRS 634, 644 may appear as different instantiations. In some embodiments, each of these instantiations of the VSRS may be associated with a VLAN 630, 640, and in some embodiments, each VLAN 630, 640 may have an instantiation of the VSRS 634, 644. In some embodiments, each instantiation of the VSRS 634, 644 may have one or several unique tables corresponding to the VLAN 630, 640 with which the instantiation of the VSRS 634, 644 is associated. Each instantiation of the VSRS 634, 644 may generate and / or curate the unique tables associated with that instantiation of the VSRS 634, 644. Thus, while a single service may provide VSRS 634, 644 functionality for one or several cloud networks, individual instantiations of the VSRS 634, 644 within a cloud network may have unique Layer 2 and Layer 3 forwarding tables, while multiple such customer networks may have overlapping Layer 2 and Layer 3 forwarding tables.

[0141] In some embodiments, the VSRS 634, 644 may support competing VLANs and IP spaces across multiple tenants. This may include having multiple tenants on the same VSRS 634, 644. In some embodiments, some or all of these tenants may choose to use some or all of the same IP address space, the same MAC space, and the same VLAN space. This may provide extreme flexibility to the user in choosing addresses. In some embodiments, this multi-tenancy is supported by providing each tenant with a separate virtual network, which is a private network within the cloud network. Each virtual network is given a unique identifier. Similarly, in some embodiments, each host may have a unique identifier and / or each virtual interface or virtual gateway may have a unique identifier. In some embodiments, these unique identifiers, specifically the unique identifier of the virtual network for the tenant, may be encoded in each communication. By providing each virtual network with a unique identifier and including this in the communication, a single instantiation of the VSRS 634, 644 may serve multiple tenants with overlapping addresses and / or namespaces.

[0142] The VSRS 634, 644 may perform these switching and / or routing functions to facilitate and / or enable the creation and / or communication with L2 networks within the VLANs 630, 640, which may be found within a cloud computing environment, and more specifically, within a virtual network in the cloud computing environment.

[0143] For example, each of the VLANs 630, 640 includes multiple compute instances 632-A, 632-B, 642-A, 642-B. The VSRS 634, 644 enables communication between compute instances in one VLAN 630, 640 and compute instances in another VLAN 630, 640 or subnet 620. In some embodiments, the VSRS 634, 644 enables communication between compute instances in one VLAN 630, 640 and another VCN, another network outside the VCN including the Internet, an on-premise data center, etc. In such embodiments, a compute instance, such as, for example, compute instance 632-A, can send a communication to an endpoint outside the VLAN, in this example VLAN A 630. The computation instance (632-A) can send a communication to VSRS A 634, which can direct the communication to a router 624, 644 or a gateway 604, 608, 610 communicatively coupled to the desired endpoint. The router 624, 644 or a gateway 604, 608, 610 communicatively coupled to the desired endpoint can receive the communication from the computation instance (632-A) and direct the communication to the desired endpoint.

[0144] Referring now to FIG. 7, a logical and hardware schematic diagram of a VLAN 700 is shown. As will be appreciated, the VLAN 700 includes multiple endpoints, specifically including multiple compute instances and VSRS. Multiple compute instances (CIs) are instantiated on one or several host machines. In some embodiments, this can be a one-to-one relationship such that each CI is instantiated on a unique host machine, and / or in some embodiments, this can be a many-to-one relationship such that multiple CIs are instantiated on a single common host machine. In various embodiments, the CIs can be layer 2 CIs by being configured to communicate with each other using an L2 protocol. FIG. 7 illustrates a scenario in which some CIs are instantiated on unique host machines and some CIs share a common host machine. As can be seen in FIG. 7, instance 1 (CI1) 704-A is instantiated on host machine 1 702-A, instance 2 (CI2) 704-B is instantiated on host machine 2 702-B, and instance 3 (CI3) 704-C and instance 4 (CI4) 704-D are instantiated on a common host machine 702-C.

[0145] Each of the CIs 704-A, 704-B, 704-C, and 704-D is communicatively coupled to the other CIs 704-A, 704-B, 704-C, and 704-D in the VLAN 700, and the VSRS 714. Specifically, each of the CIs 704-A, 704-B, 704-C, and 704-D is connected to the other CIs 704-A, 704-B, 704-C, and 704-D in the VLAN 700, and the VSRS 714 via an L2 VNIC and a switch. Each CI 704-A, 704-B, 704-C, and 704-D is associated with a unique L2 VNIC and switch. The switch may be an L2 virtual switch that is local and uniquely associated with and deployed for the L2 VNIC. Specifically, CI1 704-A is associated with L2 VNIC1 708-A and Switch 1 710-A, CI2 704-B is associated with L2 VNIC2 708-B and Switch 710-B, CI3 704-C is associated with L2 VNIC3 708-C and Switch 3 710-C, and CI4 704-D is associated with L2 VNIC4 708-D and Switch 4 710-D.

[0146] In some embodiments, each L2 VNIC 708 and its associated switch 710 may be instantiated on the NVD 706. This instantiation can be in a one-to-one relationship such that a single L2 VNIC 708 and its associated switch 710 are instantiated on a unique NVD 706, or this instantiation can be in a many-to-one relationship such that multiple L2 VNICs 708 and their associated switches 710 are instantiated on a single common NVD 706. Specifically, L2 VNIC1 708-A and Switch1 710-A are instantiated on NVD1 706-A, L2 VNIC2 708-B and Switch2 710-B are instantiated on NVD2, and both L2 VNIC3 708-C and Switch3 710-C and L2 VNIC4 708-D and Switch 710-D are instantiated on a common NVD, i.e., NVD 706-C.

[0147] In some embodiments, the VSRS 714 can support competing VLANs and IP spaces across multiple tenants. This can include having multiple tenants on the same VSRS 714. In some embodiments, some or all of these tenants can choose to use some or all of the same IP address space, the same MAC space, and the same VLAN space. This can provide extreme flexibility to the user in choosing addresses. In some embodiments, this multi-tenancy is supported by providing each tenant with a separate virtual network, which is a private network within the cloud network. Each virtual network (e.g., each VLAN or VCN) is given a unique identifier, such as a VCN identifier, which may be a VLAN identifier. This unique identifier may be selected, for example, by the control plane, specifically the CSPI control plane. In some embodiments, this unique VLAN identifier may include one or several bits that may be included and / or used in packet encapsulation. Similarly, in some embodiments, each host may have a unique identifier, and / or each virtual interface or virtual gateway may have a unique identifier. In some embodiments, these unique identifiers, specifically the unique identifier of the virtual network for the tenant, may be encoded in each communication. By providing a unique identifier for each virtual network and including it in the communication, a single instantiation of the VSRS may serve multiple tenants with overlapping addresses and / or namespaces. In some embodiments, the VSRS 714 may determine which tenant a packet belongs to based on a VCN identifier and / or a VLAN identifier associated with the communication, specifically in the VCN header of the communication. In embodiments disclosed herein, communications to and from a VLAN may have a VCN header that may include a VLAN identifier.Based on the VCN header containing the VLAN identifier, the VSRS 714 can determine the tenancy; in other words, the receiving VSRS can determine which VLAN and / or which tenant to send the communication to. Additionally, each compute instance (e.g., an L2 compute instance) that belongs to a VLAN is given a unique interface identifier that identifies the L2 VNIC associated with that compute instance. The interface identifier can be included in traffic to and / or from the computer instance (e.g., by being included in the header of the frame) and used by the NVD to identify the L2 VNIC associated with the compute instance. In other words, the interface identifier can uniquely indicate the compute instance and / or its associated L2 VNIC.

[0148] As shown in FIG. 7, switches 710-A, 710-B, 710-C, and 710-D together can form an L2 distributed switch 712, also referred to herein as a distributed switch 712. From a customer's perspective, each switch 710-A, 710-B, 710-C, and 710-D in the L2 distributed switch 712 is a single switch that connects to all of the CIs in a VLAN. However, the L2 distributed switch 712 that emulates the user experience of a single switch is infinitely scalable and includes a collection of local switches (e.g., switches 710-A, 710-B, 710-C, and 710-D in the illustrative example of FIG. 7). As shown in FIG. 7, each CI runs on a host machine connected to a NVD. For each CI on a host connected to the NVD, the NVD hosts a layer 2 VNIC associated with the compute instance and a local switch (e.g., an L2 virtual switch that is local to the NVD, associated with the layer 2 VNIC, and is a member or component of the L2 distributed switch 712). The layer 2 VNIC represents a port of the compute instance on a layer 2 VLAN. The local switch connects the VNIC to other VNICs (e.g., other ports) associated with other compute instances in the layer 2 VLAN.

[0149] Each of the CIs 704-A, 704-B, 704-C, 704-D can communicate with others of the CIs 704-A, 704-B, 704-C, 704-D in the VLAN 700 or with the VSRS 714. One of the CIs 704-A, 704-B, 704-C, 704-D transmits a frame to another of the CIs 704-A, 704-B, 704-C, or the VSRS 714 by sending the frame to the MAC address and interface identifier of the receiving CI or VSRS 714 of the CIs 704-A, 704-B, 704-C, 704-D. The MAC address and interface identifier can be included in the header of the frame. As described herein above, the interface identifier can indicate the L2 VNIC of the receiving CI or VSRS 714 of the CIs 704-A, 704-B, 704-C, 704-D.

[0150] In one embodiment, CI1 704-A may be the source CI, L2 VNIC 708-A may be the source L2 VNIC, and switch 710-A may be the source L2 virtual switch. In this embodiment, CI3 704-C may be the destination CI, and L2 VNIC3 708-C may be the destination L2 VNIC. The source CI may transmit a frame with a source MAC address and a destination MAC address. The frame may be intercepted by NVD 706-A, which instantiates the source VNIC and the source switch.

[0151] The L2 VNICs 708-A, 708-B, 708-C, 708-D can each learn a mapping of MAC addresses to the interface identifiers of the L2 VNICs for the VLAN 700. This mapping can be learned based on frames and / or communications received from within the VLAN 700. Based on this previously determined mapping, the source VNIC can determine the interface identifier of the destination interface associated with the destination CI in the VLAN and can encapsulate the frame. In some embodiments, this encapsulation can include GENEVE encapsulation, specifically, L2 GENEVE encapsulation including an L2 (Ethernet) header for the encapsulated frame. The encapsulated frame can identify the destination MAC, the destination interface identifier, the source MAC, and the source interface identifier.

[0152] The source VNIC can pass the encapsulated frame to the source switch, which can direct the frame to the destination VNIC. Upon receiving the frame, the destination VNIC can decapsulate the frame and then provide it to the destination CI.

[0153] Now referring to FIG. 8, a logical schematic diagram of multiple connected L2 VLANs 800 is shown. In the particular embodiment shown in FIG. 8, both VLANs are located in the same VCN. As can be seen, the multiple connected L2 VLANs 800 can include a first VLAN, VLAN A 802-A, and a second VLAN, VLAN B 802-B. Each of these VLANs 802-A, 802-B can include one or several CIs, each of which can have an associated L2 VNIC and an associated L2 virtual switch. Furthermore, each of these VLANs 802-A, 802-B can include a VSRS.

[0154] Specifically, VLAN A 802-A may include instance 1 804-A connected to L2 VNIC1 806-A and switch 1 808-A, instance 2 804-B connected to L2 VNIC2 806-B and switch 808-B, and instance 3 804-C connected to L2 VNIC3 806-C and switch 3 808-C. VLAN B 802-B may include instance 4 804-D connected to L2 VNIC4 806-D and switch 4 808-D, instance 5 804-E connected to L2 VNIC5 806-E and switch 808-E, and instance 6 804-F connected to L2 VNIC6 806-F and switch 3 808-F. VLAN A 802-A may further include VSRS A 810-A, and VLAN B 802-B may include VSRS B 810-B. Each of CIs 804-A, 804-B, 804-C of VLAN A 802-A may be communicatively coupled to VSRS A 810-A, and each of CIs 804-D, 804-E, 804-F of VLAN B 802-B may be communicatively coupled to VSRS B 810-B.

[0155] VLAN A 802-A can be communicatively coupled to VLAN B 802-B via respective VSRSs 810-A, 810-B. Each VSRS can in turn be coupled to a gateway 812, which can provide access for CIs 804-A, 804-B, 804-C, 804-D, 804-E, 804-F in each VLAN 802-A, 802-B to other networks outside the VCN in which the VLANs 802-A, 802-B are located. In some embodiments, these networks can include, for example, one or several on-premise networks, another VCN, a service network, a public network such as the Internet, and the like.

[0156] Each of CIs 804-A, 804-B, 804-C in VLAN A 802-A can communicate with CIs 804-D, 804-E, 804-F in VLAN B 802-B via VSRSs 810-A, 810-B of the respective VLANs 802-A, 802-B. For example, one of CIs 804-A, 804-B, 804-C, 804-D, 804-E, 804-F in one of VLANs 802-A, 802-B can transmit a frame to CIs 804-A, 804-B, 804-C, 804-D, 804-E, 804-F in the other of VLANs 802-A, 802-B. The frame can exit the source VLAN via the VSRS of the source VLAN, enter the destination VLAN, and be routed to the destination CI via the destination VSRS.

[0157] In one embodiment, CI1 804-A may be the source CI, VNIC 806-A may be the source VNIC, and switch 808-A may be the source switch. In this embodiment, CI 5 804-E may be the destination CI, and L2 VNIC5 806-E may be the destination VNIC. VSRS A 810-A may be the source VSRS identified as an SVSRS, and VSRS B 810-B may be the destination VSRS identified as a DVSRS.

[0158] The source CI may send a frame with a MAC address. The frame may be intercepted by an NVD that instantiates the source VNIC and the source switch. The source VNIC encapsulates the frame. In some embodiments, the encapsulation may include GENEVE encapsulation, specifically L2 GENEVE encapsulation. The encapsulated frame may identify a destination address of the destination CI. In some embodiments, the destination address may also include a destination address of the destination VSRS. The destination address of the destination CI may include a destination IP address, a destination MAC of the destination CI, and / or a destination interface identifier of a destination VNIC associated with the destination CI. The destination address of the destination VSRS may include an IP address of the destination VSRS, an interface identifier of a destination VNIC associated with the destination VSRS, and / or a MAC address of the destination VSRS.

[0159] The source VSRS can receive the frame from the source switch, can look up a VNIC mapping from the frame's destination address, which may be a destination IP address, and can forward the packet to the destination VSRS. The destination VSRS can receive the frame. Based on the destination address included in the frame, the destination VSRS can forward the frame to the destination VNIC. The destination VNIC can receive the frame, decapsulate it, and then provide the frame to the destination CI.

[0160] 9, there is shown a logical schematic diagram of multiple connected L2 VLANs and subnets 900. In the particular embodiment shown in FIG 9, both the VLANs and the subnets are located in the same VCN. This is shown because the virtual routers and VSRSs for both the VLANs and the subnets are directly connected, rather than connected through a gateway.

[0161] As can be seen, this can include a first VLAN, VLAN A 902-A, a second VLAN, VLAN B 902-B, and a subnet 930. Each of these VLANs 902-A, 902-B can include one or several CIs, each of which can have an associated L2 VNIC and an associated L2 switch. Furthermore, each of these VLANs 902-A, 902-B can include a VSRS. Similarly, the subnet 930, which can be an L3 subnet, can include one or several CIs, each of which can have an associated L3 VNIC, and the L3 subnet 930 can include a virtual router 916.

[0162] Specifically, VLAN A 902-A may include instance 1 904-A connected to L2 VNIC1 906-A and switch 1 908-A, instance 2 904-B connected to L2 VNIC2 906-B and switch 2 908-B, and instance 3 904-C connected to L2 VNIC3 906-C and switch 3 908-C. VLAN B 902-B may include instance 4 904-D connected to L2 VNIC4 906-D and switch 4 908-D, instance 5 904-E connected to L2 VNIC5 906-E and switch 5 908-E, and instance 6 904-F connected to L2 VNIC6 906-F and switch 6 908-F. VLAN A 902-A may further include VSRS A 910-A, and VLAN B 902-B may include VSRS B 910-B. Each of CIs 904-A, 904-B, 904-C of VLAN A 902-A may be communicatively coupled to VSRS A 910-A, and each of CIs 904-D, 904-E, 904-F of VLAN B 902-B may be communicatively coupled to VSRS B 910-B. L3 subnet 930 may include one or several CIs, and specifically may include instance 7 904-G that is communicatively coupled to L3 VNIC 7 906-G. L3 subnet 930 may include a virtual router 916.

[0163] VLAN A 902-A can be communicatively coupled to VLAN B 902-B via respective VSRSs 910-A, 910-B. L3 subnet 930 can be communicatively coupled to VLAN A 902-A and VLAN B 902-B via a virtual router 916. Each of the virtual routers 916 and VSRS instances 910-A, 910-B can in turn be coupled to a gateway 912, which can provide access to CIs 904-A, 904-B, 904-C, 904-D, 904-E, 904-F, 904-G in each VLAN 902-A, 902-B and subnet 930 to other networks outside of the VCN in which the VLANs 902-A, 902-B and subnet 930 are located. In some embodiments, these networks can include, for example, one or several on-premise networks, another VCN, a service network, a public network such as the Internet, and the like.

[0164] Each VSRS instance 910-A, 910-B can provide an outgoing path for frames exiting the associated VLAN 902-A, 902-B and an incoming path for frames entering the associated VLAN 902-A, 902-B. From the VSRS instance 910-A, 910-B of a VLAN 902-A, 902-B, frames can be sent to any desired endpoint, including an L2 endpoint, such as an L2 CI in another VLAN, either on the same VCN or on a different VCN or network, and / or an L3 endpoint, such as an L3 CI in a subnet on the same VCN or a different VCN or network.

[0165] In one embodiment, CI1 904-A may be the source CI, VNIC 906-A may be the source VNIC, and switch 908-A may be the source switch. In this embodiment, CI7 904-G may be the destination CI, and VNIC7 906-G may be the destination VNIC. VSRS A 910-A may be the source VSRS identified as an SVSRS, and virtual router (VR) 916 may be the destination VR.

[0166] The source CI may send a frame with a MAC address. The frame may be intercepted by an NVD that instantiates the source VNIC and the source switch. The source VNIC encapsulates the frame. In some embodiments, the encapsulation may include GENEVE encapsulation, specifically L2 GENEVE encapsulation. The encapsulated frame may identify a destination address of the destination CI. In some embodiments, the destination address may also include a destination address of a VSRS of the source CI's VLAN. The destination address of the destination CI may include a destination IP address, a destination MAC of the destination CI, and / or a destination interface identifier of the destination VNIC of the destination CI.

[0167] The source VSRS can receive the frame from the source switch, can look up a VNIC mapping from the frame's destination address, which may be a destination IP address, and can forward the frame to the destination VR. The destination VR can receive the frame. Based on the destination address included in the frame, the destination VR can forward the frame to the destination VNIC. The destination VNIC can receive the frame, decapsulate it, and then provide the frame to the destination CI.

[0168] Learning by L2 VNICs and / or L2 virtual switches within a virtual L2 network Referring now to FIG. 10, a schematic diagram of one embodiment of intra-VLAN communication and learning within a VLAN 1000 is shown. The learning here is specific to how the L2 VNIC, the VSRS of the source CI's VLAN, and / or the L2 virtual switch learns the association between MAC addresses and L2 VNICs / VSRS VNICs (more specifically, between MAC addresses associated with an L2 compute instance or VSRS and interface identifiers associated with the L2 VNICs of these L2 compute instances associated with the VSRS VNICs). In general, the learning is based on incoming traffic. This learning is different from the learning process (e.g., ARP process) that an L2 compute instance may implement to learn destination MAC addresses, in terms of the interface-to-MAC address learning aspect. The two learning processes (e.g., of the L2 VNIC / L2 virtual switch and the L2 compute instance) are shown as being implemented jointly in FIG. 12.

[0169] As can be seen, VLAN 1000 includes compute instance 1 1000-A that is communicatively coupled to NVD1 1001-A that instantiates L2 VNIC1 1002-A and L2 switch 1 1004-A. VLAN 1000 also includes compute instance 2 1000-B that is communicatively coupled to NVD2 1001-B that instantiates L2 VNIC2 1002-B and L2 switch 2 1004-A. VLAN 1000 also includes VSRS 1015 that runs on the server fleet and includes VSRS VNIC 1002-C and VSRS switch 1004-C. All of switches 1004-A, 1004-B, 1004-C together form L2 distributed switch 1050. The VSRS 1015 is communicatively coupled to the endpoint 1008, which may include a gateway, and in particular may include an L2 / L3 router, for example in the form of another VSRS, or an L3 router, for example in the form of a virtual router.

[0170] The control plane 1010 of the VCN hosting the VLAN 1000 maintains information identifying each L2 VNIC on the VLAN 1000 and the network placement of each L2 VNIC. For example, this information may include, for an L2 VNIC, an interface identifier associated with the L2 VNIC and / or a physical IP address of the NVD hosting the L2 VNIC. The control plane 1010 updates the interfaces in the VLAN 1000 with this information (e.g., periodically or on-demand). Thus, each L2 VNIC 1002-A, 1002-B, 1002-C in the VLAN 1000 receives information from the control plane 1010, identifies the interfaces in the VLAN, and populates a table with this information. The table populated by the L2 VNIC may be stored locally in the NVD hosting the L2 VNIC. If VNICs 1002-A, 1002-B, 1002-C already contain a current table, VNICs 1002-A, 1002-B, 1002-C can determine any discrepancies between VNICs 1002-A, 1002-B, 1002-C's current table and the information / table received from control plane 1010. VNICs 1002-A, 1002-B, 1002-C can, in some embodiments, update their tables to match the information received from control plane 1010.

[0171] As seen in FIG. 10, a frame is transmitted through L2 switches 1004-A, 1004-B, 1004-C and received by a receiving VNIC 1002-A, 1002-B, 1002-C. When a frame is received by a VNIC 1002-A, 1002-B, 1002-C, that VNIC learns the mapping of the source interface (source VNIC) and source MAC address of that frame. Based on its table of information received from the control plane 1010, the VNIC can map the source MAC address (from the received frame) to the interface identifier of the source VNIC and the IP address of that VNIC and / or the IP address of the NVD hosting that VNIC (if the interface identifier and IP address are available from the table). Thus, the L2 VNICs 1002-A, 1002-B, 1002-C learn the mapping of interface identifiers to MAC addresses based on the received communications and / or frames. Each VNIC 1002-A, 1002-B, 1002-C may have its L2 forwarding (FWD) table 1006-A, 1006-B, 1006-C with this learned mapping information. In some embodiments, the L2 forwarding table includes a MAC address and associates it with at least one of an interface identifier or a physical IP address. In such embodiments, the MAC address may be an address assigned to the L2 compute instance and correspond to a port emulated by the L2 VNIC associated with the L2 compute instance. The interface identifier may uniquely identify the L2 VNIC and / or the L2 compute instance. The virtual IP address may be that of the L2 VNIC. And, the physical IP address may be that of the NVD hosting the L2 VNIC. The L2 forwarding updated by the L2 VNIC may be stored locally on the NVD hosting the L2 VNIC and may be used by the L2 virtual switch associated with the L2 VNIC to direct frames.In some embodiments, VNICs in a common VLAN may share all or parts of their mapping tables with each other.

[0172] In light of the above network architecture, traffic flow will now be described. For clarity of explanation, traffic flow will be described with respect to compute instance 2 1000-B, L2 VNIC2 10002-B, L2 switch 2 1004-B, and NVD2 1001-B. This description applies equally to traffic flow between other compute instances.

[0173] As described herein above, a VLAN is realized in a VCN as an overlay L2 network on top of a L3 physical network. An L2 compute instance of a VLAN can transmit or receive an L2 frame that includes an overlay MAC address (also called a virtual MAC address) as the source and destination MAC address. The L2 frame can also encapsulate a packet that includes an overlay IP address (also called a virtual IP address) as the source and destination IP address. The overlay IP address of a compute instance can belong to the CIDR range of the VLAN in some embodiments. Other overlay IP addresses can belong to the CIDR range (in which case the L2 frame flows within the VLAN) or outside the CIDR range (in which case the L2 frame is destined for or received from another network). The L2 frame can also include a VLAN tag that can be used to uniquely identify the VLAN and distinguish between multiple L2 VNICs on the same NVD. An L2 frame may be received by the NVD in an encapsulated packet from the host machine of the compute instance, from another NVD, or from a server fleet hosting a VSRS, via a tunnel. In these different cases, the encapsulated packet may be an L3 packet sent on the physical network, with the source and destination IP addresses being physical IP addresses. Different types of encapsulation are possible, including GENEVE encapsulation. The NVD may decapsulate the received packet to extract the L2 frame. Similarly, to send an L2 frame, the NVD may encapsulate it in an L3 packet and send it on the physical substrate.

[0174] For intra-VLAN outgoing traffic from Instance 2 1000-B, NVD2 1001-B receives a frame from the host machine of Instance 2 1000-B over the Ethernet link. The frame includes an interface identifier that identifies L2 VNIC2 1000-B. The frame includes the overlay MAC address of Instance 2 1000-B (e.g., M.2) as the source MAC address and the overlay MAC address of Instance 1 1000-A (e.g., M.1) as the destination MAC address. Given the interface identifier, NVD2 1001-B passes the frame to L2 VNIC2 1002-B for further processing. L2 VNIC2 1002-B forwards the frame to L2 Switch 2 1004-B. Based on L2 forwarding table 1006-B, L2 Switch 2 1004-B determines if the destination MAC address is known (eg, matches it with an entry in L2 forwarding table 1006-B).

[0175] If so, L2 Switch 2 1004-B determines that L2 VNIC1 1002-A is the associated tunnel endpoint and forwards the frame to L2 VNIC1 1002-A. The forwarding may include encapsulating and decapsulating the frame in a packet (e.g., GENEVE encapsulation and decapsulation), where the packet includes the frame, the physical IP address of NVD1 1001-A (e.g., IP.1) as the destination address, and the physical IP address of NVD 2 1001-B (e.g., IP.2) as the source address.

[0176] If unknown, L2 Switch2 1004-B broadcasts the frame to the various VNICs of the VLAN (e.g., including L2 VNIC1 1002-A and any other L2 VNICs of the VLAN), and the broadcasted frame is processed (e.g., encapsulated, forwarded, decapsulated) between the associated NVDs. In some embodiments, this broadcast is performed in the physical network, or more specifically, emulated in the physical network, and may include VSRSs in the VLAN to encapsulate the frame separately to each L2 VNIC. Thus, the broadcast is emulated in the physical network via a series of replicated unicast packets. Each L2 VNIC then receives the frame and learns the association between the interface identifier and source MAC address (e.g., M.2) of L2 VNIC2 1002-B and the source physical IP address (e.g., IP.2).

[0177] For intra-VLAN incoming traffic from compute instance 1 1000-A to compute instance 2 1000-B, NVD2 1001-B receives the packet from NVD1. The packet has IP.1 as the source address and the frame includes M.2 as the destination MAC address and M.1 as the source MAC address. The frame also includes the network identifier of L2 VNIC1 1002-A. Upon decapsulation, VNIC2 receives the frame, learns that this interface identifier is associated with M.1 and / or IP.1, and stores this learned information in L2 forwarding table 1006-B to switch 2 for future outgoing traffic if it was previously unknown. Alternatively, upon decapsulation, L2 VNIC2 1002-B receives the frame, learns that this interface identifier is associated with M.1 and / or IP.1, and refreshes the validity time if this information was previously known.

[0178] For outgoing traffic sent from instance 2 1000-B in VLAN1000 to an instance in another VLAN, a similar flow may exist as for the outgoing traffic described above, except that a VSRS VNIC and a VSRS switch are used. In particular, the destination MAC address is not within the L2 broadcast of VLAN1000 (it is in the other L2 VLAN). Therefore, the overlay destination IP address of the destination instance (e.g., IP.A) is used for this outgoing traffic. For example, L2 VNIC2 1002-B determines that IP.A is outside the CIDR range of VLAN1000. Therefore, L2 VNIC2 1002-B sets the destination MAC address to the default gateway MAC address (e.g., M.DG). Based on M.DG, L2 Switch2 1004-B sends the outgoing traffic to the VSRS VNIC (e.g., via a tunnel, with appropriate end-to-end encapsulation). The VSRS VNIC forwards the outgoing traffic to the VSRS switch. The VSRS switch then performs a routing function, where based on the overlay destination IP address (e.g., IP.A), the VSRS switch in VLAN1000 sends the outgoing traffic to the VSRS switch in the other VLAN (e.g., via a virtual router between these two VLANs, with appropriate end-to-end encapsulation). The VSRS switch in the other VLAN then performs a switching function by determining that IP.A is within the CIDR range of this VLAN, and performs a lookup of its ARP cache based on IP.A to determine the destination MAC address associated with IP.A. If there is no match in the ARP cache, an ARP request is sent to a different L2 VNIC in the other VLAN to determine the destination MAC address. Otherwise, the VSRS switch sends the outgoing traffic to the associated VNIC (e.g., via a tunnel, with appropriate encapsulation).

[0179] For inbound traffic from an instance in another VLAN to an instance in VLAN1000, the traffic flow is similar to above, except in the opposite direction. For outbound traffic from an instance in VLAN1000 to the L3 network, the traffic flow is similar to above, except the VSRS switch in VLAN1000 routes the packet directly to the destination VNIC in the virtual L3 network via the virtual router (e.g., the packet does not have to be routed through another VSRS switch). For inbound traffic from the virtual L3 network to an instance in VLAN1000, the traffic flow is similar to above, except the packet is received by the VSRS switch in VLAN1000A, which sends the packet as a frame within the VLAN. For traffic between VLAN1000 and other networks (outbound or inbound), the VSRS switch is used similarly, with its routing function used outbound to send the packet through the appropriate gateway (e.g., IGW, NGW, DRG, SGW, LPG) and its switching function used inbound to send the frame within VLAN1000.

[0180] Referring now to FIG. 11, a schematic diagram of an embodiment of a VLAN 1100 (eg, a cloud-based virtual L2 network) is shown, and in particular, an implementation of a VLAN is shown.

[0181] As described herein above, a VLAN can include "n" compute instances 1102-A, 1102-B, 1102-N, each of which runs on a host machine. As previously described, there can be a one-to-one association between compute instances and host machines, or a many-to-one association between multiple compute instances and a single host machine. Each compute instance 1102-A, 1102-B, 1102-N can be an L2 compute instance, in which case it is associated with at least one virtual interface (e.g., L2 VNIC) 1104-A, 1104-B, 1104-N and a switch 1106-A, 1106-B, 1106-N. The switches 1106-A, 1106-B, 1106-N are L2 virtual switches and together form an L2 distributed switch type 1107.

[0182] A pair of L2 VNICs 1104-A, 1104-B, 1104-N and switches 1106-A, 1106-B, 1106-N associated with a compute instance 1102-A, 1102-B, 1102-N on a host machine is a pair of software modules on a NVD 1108-A, 1108-B, 1108-N connected to the host machine. Each L2 VNIC 1104-A, 1104-B, 1104-N represents an L2 port of a customer-perceived single switch (herein referred to as a v-switch). In general, a host machine "i" runs a compute instance "i" and is connected to an NVD "i". The NVD "i" then runs an L2 VNIC "i" and a switch "i". The L2 VNIC "i" represents an L2 port "i" of the v-switch, where "i" is a positive integer between 1 and "n". Again, a one-to-one association is described, but other types of associations are possible. For example, a single NVD can be connected to multiple hosts, each running one or more compute instances that belong to a VLAN. If so, the NVD hosts multiple pairs of L2 VNICs and switches, each corresponding to one of the compute instances.

[0183] A VLAN may include an instance of VSRS 1110. VSRS 1110 performs switching and routing functions and includes instances of VSRS VNIC 1112 and VSRS Switch 1114. VSRS VNIC 1112 represents a port on the vSwitch, which connects the vSwitch to other networks through a virtual router. As shown, VSRS 1110 may be instantiated on a server fleet 1116.

[0184] The control plane 1118 can track information identifying the L2 VNICs 1104-A, 1104-B, 1104-N and their placement in a VLAN. The control plane 1110 can further provide this information to the L2 interfaces 1104-A, 1104-B, 1104-N within the VLAN.

[0185] 11, a VLAN may be a cloud-based virtual L2 network that may be built on top of a physical network 1120. In some embodiments, this physical network 1120 may include NVDs 1108-A, 1108-B, 1108-N.

[0186] In general, a first L2 compute instance of a VLAN (e.g., compute instance 1 1102-A) can communicate with a second compute instance of the VLAN (e.g., compute instance 2 1102-B) using an L2 protocol. For example, a frame can be transmitted between these two L2 compute instances over the VLAN. Nevertheless, the frame can be encapsulated, tunneled, routed, and / or otherwise processed so that the frame can be transmitted over the underlying physical network 1120.

[0187] For example, compute instance 1 1102-A sends a frame destined for compute instance 2 1102-B. Depending on the network connection between host machine 1 and NVD1, the network connection between NVD1 and physical network 1120, the network connection between physical network 1120 and NVD2, and the network connection between NVD2 and host machine 2 (e.g., a TCP / IP connection, an Ethernet connection, a tunneling connection, etc.), different types of processing may be applied to the frame. For example, the frame is received and encapsulated by NVD1, and so on, until the frame reaches compute instance 2. This processing is assumed so that the frame can be transmitted between underlying physical resources, and for brevity and clarity, its description is omitted from the description of VLANs and related L2 operations.

[0188] Virtual L2 network communication Multiple forms of communication may occur within or with the virtual L2 network. These may include intra-VLAN communication. In such an embodiment, a source compute instance may send a communication to a destination compute instance that is in the same VLAN as the source compute instance (CI). The communication may also be sent to an endpoint outside the source CI's VLAN. This may include, for example, communication between a source CI in a first VLAN and a destination CI in a second VLAN, communication between a source CI in a first VLAN and a destination CI in an L3 subnet, and / or communication from a source CI in a first VLAN to a destination CI outside the VCN that includes the source CI's VLAN. This communication may further include, for example, receiving a communication at the destination CI from a source CI outside the destination CI's VLAN. This source CI may be in another VLAN, in an L3 subnet, or outside the VCN that includes the source CI's VLAN.

[0189] Each CI in a VLAN can play an active role in traffic flow, including learning interface identifiers to MAC addresses, also referred to herein as interface-to-MAC addresses, mapping instances in the VLAN to maintain an L2 forwarding table in the VLAN, and sending and / or receiving communications (e.g., frames in the case of L2 communications). A VSRS can play an active role in communications within a VLAN and with source or destination CIs outside the VLAN. A VSRS can maintain a presence in the L2 and L3 networks to enable outgoing and incoming communications.

[0190] 12, a flow chart is shown illustrating one embodiment of a process 1200 for intra-VLAN communication. In some embodiments, the process 1200 may be performed by a compute instance in a common VLAN. This process may be performed specifically when a source CI sends a communication to a destination CI in a VLAN, but does not know the IP-to-MAC address mapping of the destination CI. This may occur, for example, when a source CI sends a packet to a destination CI that has an IP address in a VLAN, but the source CI does not know the MAC address for that IP address. In this case, an ARP process may be performed to learn the destination MAC address and the IP-to-MAC address mapping.

[0191] If the source CI knows the IP-to-MAC address mapping, the source CI can directly send the packet to the destination CI, and no ARP process needs to be performed. In some embodiments, the packet can be intercepted by the source VNIC, which is an L2 VNIC in intra-VLAN communication. If the source VNIC knows the interface-to-MAC address mapping for the destination MAC address, the source VNIC can encapsulate the packet, for example, with L2 encapsulation, and forward the corresponding frame toward the destination MAC address to the destination VNIC, which is an L2 VNIC in intra-VLAN communication.

[0192] If the source VNIC does not know the interface-to-MAC address mapping for the MAC address, the source VNIC can perform aspects of an interface-to-MAC address learning process. This can include the source VNIC sending a frame to all interfaces in the VLAN. In some embodiments, the frame can be sent to all of the interfaces in the VLAN via broadcast. In some embodiments, the broadcast can be realized in the form of a serial unicast in the physical network. The frame can include destination MAC and IP addresses, an interface identifier, and the MAC and IP addresses of the source VNIC. Each of the VNICs in the VLAN can receive the frame and can learn the interface-to-MAC address mapping of the source VNIC.

[0193] Each of the receiving VNICs can further decapsulate the frame and forward the decapsulated frame (e.g., corresponding packet) to their associated CI. Each CI can include a network interface that can evaluate the forwarded packet. If the network interface determines that the CI that received the forwarded packet does not match the destination MAC and / or IP address, the packet is dropped. If the network interface determines that the CI that received the forwarded frame matches the destination MAC and / or IP address, the packet is received by the CI. In some embodiments, the CI with a MAC and / or IP address that matches the destination MAC and / or IP address of the packet can send a response to the source CI, thereby allowing the source VNIC to learn the interface-to-MAC address mapping of the destination CI, and thereby allowing the source CI to learn the IP-to-MAC address mapping of the destination CI.

[0194] If the source CI does not know the IP-to-MAC address mapping or if the source CI's IP-to-MAC address mapping for the destination CI is out of date, the process 1200 can be executed. Thus, once the IP-to-MAC address mapping is known, the source CI can transmit the packet. If the IP-to-MAC address mapping is not known, the process 1200 can be executed. If the interface-to-MAC address mapping is not known, the interface-to-MAC address learning process outlined above can be executed. If the interface-to-MAC address mapping is known, the source VNIC can transmit the corresponding frame to the destination VNIC. The process 1200 begins at block 1202, where the source CI determines that the destination CI's IP-to-MAC address mapping is unknown to the source CI. In some embodiments, this may include the source CI determining a destination IP address for the packet and determining that the destination IP address is not associated with a MAC address stored in the source CI's mapping table. Alternatively, the source CI may determine that the IP-to-MAC address mapping for the destination CI is out of date. A mapping may be stale, in some embodiments, if the mapping has not been updated and / or verified within a certain time limit. Upon determining that the IP-to-MAC address mapping of the destination CI is unknown and / or stale to the source CI, the source CI initiates an ARP request for the destination IP address and sends an ARP request for an Ethernet broadcast.

[0195] At block 1204, the source VNIC, also referred to herein as the source interface, receives the ARP request from the source CI. The source interface identifies all interfaces on the VLAN and sends the ARP request to all interfaces on the VLAN broadcast domain. As previously described, since the control plane knows all of the interfaces on the VLAN and provides that information to the interfaces with the VLAN, the source interface similarly knows all of the interfaces in the VLAN and can send an ARP request to each of the interfaces in the VLAN. To do this, the source interface duplicates the ARP request and encapsulates one of the ARP requests for each of the interfaces on the VLAN. Each encapsulated ARP request includes the source CI interface identifier, the source CI MAC address and IP address, the target IP address, and the destination CI interface identifier. The source CI interface duplicates the Ethernet broadcast by sending the duplicated and encapsulated ARP requests (e.g., ARP messages) as serial unicasts, one ARP request is sent to each interface in the VLAN.

[0196] At block 1206, each interface in the VLAN broadcast domain receives and decapsulates the ARP message. Each of the interfaces in the VLAN broadcast domain that receives the ARP message learns the interface-to-MAC address mapping (e.g., the interface identifier of the source interface to the MAC address of the source CI) of the source VNIC of the source CI since the message identifies the source CI MAC address and IP address and the source CI interface identifier. As part of learning the interface-to-MAC address mapping for the source CI, each of the interfaces can update its mapping table (e.g., its L2 forwarding table) and provide the updated mapping to its associated switch and / or CI. Each receiving interface can forward the decapsulated packet to their associated CI, except for the VSRS. The CI recipient of the forwarded decapsulated packet, specifically the network interface of the CI, can determine whether the target IP address matches the IP address of the CI. If the IP address of the CI associated with the interface does not match the destination CI IP address, in some embodiments, the packet is dropped by the CI and no further action is taken. In the case of a VSRS, the VSRS can determine whether the target IP address matches the IP address of the VSRS. If the IP address of the VSRS does not match the target IP address specified in the received packet, in some embodiments, the packet is dropped by the VSRS and no further action is taken.

[0197] If it is determined that the destination CI IP address specified in the received packet matches the IP address of the CI (destination CI) associated with the receiving interface, the destination CI sends a response, which may be a unicast ARP response, to the source interface, as shown in block 1208. The response includes the destination CI MAC address and destination CI IP address, and the source CI IP address and MAC address. The response is received by the destination interface, as shown in block 1210, which encapsulates the unicast ARP response. In some embodiments, the encapsulation may include GENEVE encapsulation. The destination interface may forward the encapsulated ARP response to the source interface through the destination switch. The response includes the destination CI MAC address and IP address and destination CI interface identifier, and the source CI MAC address and IP address and source CI interface identifier.

[0198] In block 1212, the source interface receives the ARP response and decapsulates it. The source interface may further learn an interface-to-MAC address mapping for the destination CI based on the encapsulation and / or information contained in the encapsulated frame. The source interface may forward the ARP response to the source CI in some embodiments.

[0199] In block 1214, the source CI receives the ARP response. In some embodiments, the source CI can update a mapping table based on the information included in the ARP response, specifically, the mapping table can be updated to reflect an IP-to-MAC address mapping based on the MAC address and IP address of the destination CI. The source CI can then send a packet to the destination CI based on this MAC address. The packet can include the MAC address and interface identifier of the source CI as the source MAC address and source interface, and the MAC address and interface identifier of the destination CI as the destination MAC address and destination interface.

[0200] In block 1216, the source interface may receive a packet from the source CI. The source interface may encapsulate the packet, which in some embodiments uses GENEVE encapsulation. The source interface may forward the corresponding frame to the destination CI, specifically to the destination interface. The encapsulated frame may include the MAC address and interface identifier of the source CI as the source MAC address and source interface identifier, and the MAC address and interface identifier of the destination CI as the destination MAC address and destination interface.

[0201] In block 1218, the destination interface receives the frame from the source interface. The destination interface can decapsulate the frame and then forward the corresponding packet to the destination CI. In block 1220, the destination CI receives the packet from the destination interface.

[0202] Span Port In an L2 physical network, a network administrator can configure a switch to copy a frame received on one port and send the copy through another port. Such a function can be called a span port function. Another similar function for targeted copying and transmission can also be provided by a network test access point (TAP). Generally, a TAP can be a device that is connected to an L2 physical network and located between two other devices in the L2 physical network. Using an internal splitter, the TAP generates and sends a copy of the traffic for monitoring while the original traffic continues unimpeded through the L2 physical network.

[0203] In comparison, customers of an L2 virtual network can similarly utilize span port and / or TAP capabilities, despite the distributed nature of the L2 virtual network's L2 switches. In addition, the architecture of an L2 virtual network provides additional flexibility that is not otherwise available in an L2 physical network. For example, customers can specify filtering criteria for triggering a copy, processing criteria (e.g., adding a specific encapsulation to the copy), and destinations (which may be within the L2 virtual network or within another network).

[0204] For clarity of explanation, the embodiments of the present disclosure are described in relation to span ports. However, the embodiments apply to the TAP function as well. In particular, a customer can specify filtering and processing criteria for copying frames. Information about such criteria is generated by the control plane and used by the NVD. Based on traffic to an L2 VNIC hosted by the NVD or from an L2 virtual switch, the NVD can apply those criteria to generate and send a copy of the traffic without disturbing the original traffic. This function can be realized by the L2 virtual switch (as its program code) for outgoing traffic, and by the L2 VNIC (as its program code) and / or another software module hosted by the NVD (in which case the function corresponds to the TAP function) for incoming traffic.

[0205] Thus, the terms "span port" and "TAP" are used interchangeably herein, and descriptions related to "span port" refer to the ability to copy traffic and send the copies without disrupting the original traffic, which may be realized as a true span port function (e.g., as part of a distributed L2 switch), as a pseudo span port function (e.g., as part of an L2 VNIC associated with an L2 virtual switch and emulating a port on the switch), or as a true TAP function (e.g., some other module hosted by the NVD).

[0206] 13 illustrates an exemplary environment suitable for defining span ports for an L2 virtual network. In an embodiment, the environment includes a computer system 1310 that communicates with customer devices 1320 over one or more networks (not shown). The computer system 1310 can include a set of hardware computing resources that host a VCN 1312. A control plane hosted by one or more of the hardware computing resources can receive and process input from the customer devices 1320 and deploy an L2 virtual network (shown in FIG. 13 as an L2 VLAN 1314) within the VCN 1312.

[0207] In one example, input from customer device 1320 can include various types of information, which can be specified via a console or API calls and can include L2 VLAN configuration 1322 and span port configuration 1324, among other customer specified configurations.

[0208] L2 VLAN configuration 1322 may indicate, for example, the number, type, and configuration of L2 compute instances to be included in L2 VLAN 1314. Additionally, L2 VLAN configuration 1322 may indicate customer-specified names of ports on the customer-aware vSwitch, MAC addresses of the L2 compute instances, and associations between ports and MAC addresses (or, more generally, L2 compute instances). For example, a customer may specify that L2 VLAN 1314 should include two L2 compute instances, a first L2 compute instance having a MAC address M.1 and associated with a first port named P1, and a second L2 compute instance having a MAC address M.2 and associated with a second port named P2.

[0209] The span port configuration 1324 may, for example, indicate one or more criteria for copying frames and possibly further processing the copied frames. These criteria generally include filtering criteria that may be applied to determine whether a copy needs to be made and processing criteria that indicate the type of processing to apply to the copy. In one example, a customer may specify filtering criteria with respect to any header information including, for example, source MAC address, destination MAC address, source IP address, destination IP address, and / or protocol. The filtering criteria may additionally or alternatively indicate source port, destination port, source of the traffic (e.g., compute instance within the L2 VLAN or source outside the L2 VLAN), and / or destination of the traffic (e.g., compute instance within the L2 VLAN or source outside the L2 VLAN). With respect to the processing criteria, a customer may specify, among other things, the number of copies to make, any encapsulation to add to the copies (e.g., add generic routing encapsulation (GRE) or virtual extensible LAN (VXLAN) encapsulation to the frame copies). Adding encapsulation allows the destination to distinguish the received copy as being spanned traffic from other received traffic that is normal or non-spanned traffic. With respect to the destination, the customer may specify a MAC address, an IP address, a port, a compute instance in an L2 VLAN, a compute instance on another L2 VLAN, a compute instance on a virtual L3 network, a host outside of an L2 VLAN (e.g., a host in the customer's on-premise network), or any other type of destination for receiving the copy. Although a single destination is described, the customer may specify multiple destinations of the same or different types. Additionally, the customer may specify whether filtering and processing (e.g., copying, and, if applicable, encapsulation) should be performed egress (e.g., at the source port or source L2 VNIC) or ingress (e.g., at the destination port or destination L2 VNIC).

[0210] For example, a customer may specify that traffic received on port 1 (corresponding to L2 VNIC1) should be copied and sent to port n (corresponding to L2 VNICn). In a more detailed definition, a customer may specify that traffic with IP address X.1, MAC address A.1, TCP port 80, and should be copied and sent to port n. In a further illustration, rather than or in addition to sending a copy to port n, a customer may specify that the copied traffic be encapsulated with a particular encapsulation scheme and sent to a host on the customer's on-premises network.

[0211] The control plane receives various information, then deploys and manages different resources of L2 VLAN 1314, generates and distributes relevant span port information to these resources. For example, L2 VLAN 1314 includes requested L2 compute instances configured according to L2 VLAN configuration 1322 and hosted on host machines, and L2 VNIC-L2 virtual switch pairs hosted on the NVD. To generate span port information, the control plane translates the customer definition from a reference using span port configuration 1324 to a reference using the actual topology of L2 VLAN 1314 based on a mapping between the customer-specified L2 VLAN configuration 1322 and the actual topology of L2 VLAN 1314. For example, each L2 VNIC emulates a port, and the control plane associates the L2 VNIC (e.g., its interface identifier, its IP address, and / or the IP address of the NVD hosting the L2 VNIC) with the name of the port and a specified MAC address. Rather than using port names, the span port information indicates copying the frames and possibly further processing by identifying the associated L2 VNICs (e.g., their interface identifiers, their IP addresses, and / or the IP address of the NVD hosting the L2 VNICs). The NVD hosting the L2 VNICs with which the span port information is associated receives and applies this span port information, enabling the NVD to perform traffic flow enforcement.

[0212] Figure 14 illustrates an exemplary span port technique in a VLAN, according to an embodiment. A Layer 2 virtual network is referred to herein as a VLAN. The top part of Figure 14 illustrates an implementation diagram 1410 of a VLAN. The bottom part of Figure 14 illustrates a customer presentation 1420 of a VLAN.

[0213] As described above herein, a VLAN can include "n" compute instances, each of which runs on a host machine. While FIG. 14 illustrates a one-to-one association between compute instances and host machines, a many-to-one association is possible, where one host machine can run multiple compute instances. Each compute instance is associated with at least one virtual interface (e.g., L2 VNIC) and switch (e.g., L2 virtual switch). A VNIC and switch pair associated with a compute instance on a host machine can be a pair of software modules on an NVD connected to the host machine. Each L2 VNIC represents an L2 port of a customer's v-switch. In the illustrative example of FIG. 14, host machine "i" runs compute instance "i" and is connected to NVD "i". NVD "i" then runs VNIC "i" and switch "i". VNIC "i" represents L2 port "i" of the v-switch, where i is a positive integer from 1 to n. Again, a one-to-one association is described, but other types of associations are possible. For example, a single NVD can be connected to multiple hosts, each of which runs one or more compute instances that belong to a VLAN, in which case the NVD hosts multiple VNIC and switch pairs, each corresponding to one of the compute instances.

[0214] Customer input can be received by a system (e.g., the VCN's control plane) for the VCN, including the VLANs. Input can be received via API calls and / or the console, and can specify the VLAN configuration and the span port configuration for the VLAN. The span port configuration can include filtering and processing criteria, can indicate one or more destinations to receive frame copies, and can be specific to a port, a set of ports, or the entire vSwitch.

[0215] In one example, as described herein above, the L2 VNICs of the NVD learn interface-to-MAC address mappings based on incoming traffic. Such mappings can be sent to the system along with the VLAN's identifier. The system can receive similar mappings from different NVDs hosting different L2 VNICs and generate mappings between interface identifiers, MAC addresses, physical IP addresses (e.g., of the NVD), and VLAN identifiers.

[0216] For example, VNIC1 learns that M.2 (overlay MAC address of compute instance 2) is associated with ID.2 (interface identifier of L2 VNIC2) and IP.2 (physical address of NVD2), and that Mn (overlay MAC address of compute instance n) is associated with ID.n (interface identifier of L2 VNIC n) and IP.n (physical address of NVD n). Similarly, VNIC2 learns that M.1 (overlay MAC address of compute instance 1) is associated with ID.1 (interface identifier of L2 VNIC1) and IP.1 (physical address of NVD1). These associations are reported to the system as part of the mappings, and the system can generate the following mappings: {customer1; M.1 → ID.1, IP.1; VLAN A}, {customer1, M.2 → ID.2, IP.2; VLAN A}, ..., {customer1, Mn → ID.n, IP.n; VLAN A}.

[0217] Additionally, the system can generate and maintain a mapping between port VLAN configurations and the distribution of VLAN resources on the physical network. For example, this mapping indicates that port "i" is configured for compute instance "i" and corresponds to L2 VNIC "i", which is then associated with L2 virtual switch "i", and this pair is hosted by NVD "i". This mapping may be represented as {Customer 1; CI.1 → P.1; P.1 → M.1; M.1 → ID.1, SW.1, IP.1; VLAN A}, where "CI.1" identifies compute instance 1 (using a customer-defined nomenclature), P.1 identifies port 1 (using a customer-defined nomenclature), M.1 identifies the MAC address associated with compute instance 1, ID.1 identifies L2 VNIC 1, SW.1 identifies L2 virtual switch 1, IP.1 identifies the IP address of NVD 1, and VLAN A identifies the VLAN.

[0218] Based on the mapping between the VLAN configuration and the distribution of resources of the VLAN on the physical network, the system can generate updated span port information from the customer-specified span port information, including updated criteria that can indicate an updated destination. Typically, the customer-specified filtering criteria use customer-aware parameters (e.g., indicate ports using a nomenclature defined by the customer in the VLAN configuration). These parameters can be updated to use L2 VLAN topology parameters (e.g., identifiers of the corresponding L2 VNICs). The processing criteria typically do not use customer-aware parameters and do not need to be updated. The destination indicated by the customer-specified span port may also be updated (e.g., if the customer identifies a port to which the copy may be sent, this port destination is updated to indicate the corresponding L2 VNIC; in comparison, if the customer identifies an L2 compute instance as the destination, no update may be required).

[0219] To illustrate, consider the following three examples that refer to the NVD: In the first example, a customer specifies that incoming frames received on port 1 are to be copied and sent to port n. In this case, the control plane generates span port information indicating that frames with M.1 as the destination MAC address are to be copied and sent to L2 VNIC n. NVD1 receives and stores this span information for L2 VNIC1. Upon receiving a frame originating from M.2 (e.g., compute instance 2) and destined for M.1, L2 VNIC1 determines that the filtering criteria are met and makes a copy, replacing M.1 with Mn. This copy is forwarded to L2 Switch 1, which then sends it to L2 VNIC n.

[0220] In a second example, the customer specifies that outgoing frames received on port 1 from port 2 are to be copied and sent to port n. In this case, the control plane generates span information indicating that frames with M.1 as the destination MAC address and M.2 as the source MAC address are to be copied and sent to VNIC n. NVD2 receives and stores this span information for VNIC2 on behalf of NVD1. Upon receiving a frame on L2 compute instance 2 with M.1 and M.2 as the destination and source MAC addresses, L2 VNIC2 determines that the filtering criteria are met and makes a copy, replacing M.1 with Mn. This copy is forwarded to L2 Switch 2, which then sends it to L2 VNIC n.

[0221] In a third example, a customer specifies that frames received on port 1 from port 2 are to be copied, GRE encapsulated, and sent to a host with IP address X.1 on its on-premise network. In this case, the control plane generates span information indicating that frames with M.1 as the destination MAC address and M.2 as the source MAC address are to be copied and sent to the VSRS VNIC. NVD1 for incoming and / or NVD2 for outgoing receive and store this span information for L2 VNIC1. For purposes of illustration, this example assumes inbound filtering and processing, but applies equally to outgoing. Upon receiving a frame originating from M.2 (e.g., compute instance 2) and destined for M.1, L2 VNIC1 determines that the filtering criteria are met, makes a copy, and replaces M.1 with the default gateway MAC address (e.g., M.DG). Additionally, L2 VNIC1 adds GRE encapsulation and the host's IP address to the header. This copy is forwarded to the VSRS VNIC for routing to the on-premise network.

[0222] 15, a flow chart is shown illustrating a process 1500 for distributing span port information in a Layer 2 virtual network. In some embodiments, the process 1500 may be performed by a control plane or another VCN system that manages the deployment of Layer 2 virtual networks on an underlying physical network.

[0223] Process 1500 begins at block 1502, where the control plane receives customer input indicating Layer 2 VLAN configuration and span port configuration. In some embodiments, the customer input is received from a customer device via an API call and / or a console.

[0224] At block 1504, the control plane generates span port information based on the customer input. In some embodiments, the control plane deploys the L2 VLANs according to the L2 VLAN configurations indicated in the customer input. Based on this deployment, the control plane determines and tracks the topology of the L2 VLAN over time by determining and tracking the placement of L2 compute instances on host machines and the placement of L2 VNICs and L2 virtual switches on the NVD, where the L2 VLAN includes L2 compute instances, L2 VNICs, and L2 virtual switches. The topology information can indicate, for each L2 compute instance, the associated L2 VNIC (and corresponding interface identifier), the associated MAC address (which may be indicated in the customer input), the associated IP address of the L2 VNIC (which may also be indicated in the customer input), and the associated IP address of the NVD hosting the L2 VNIC, among other attributes of the topology of the L2 VLAN. Additionally, the control plane can translate customer-specified port names into topology information for the corresponding L2 VNICs that emulate these ports, and then generate the span port information by updating the customer-specified span port configurations to use the topology information rather than the customer definitions.

[0225] At block 1506, the control plane determines that the span port information is applicable to a subset of the L2 VNICs. In some embodiments, the customer may indicate in the above input that the span port configuration is applicable to some but not all ports of the customer-perceived v-switch. Furthermore, the span port configuration may vary among these ports. In this case, the control plane may determine the L2 VNICs that correspond to the indicated ports and may generate associated span port information for each such L2 VNIC. Additionally, the control plane may maintain a mapping of the particular span port information that applies to each L2 VNIC.

[0226] At block 1508, the control plane determines that the span port information should be sent to a set of NVDs. In some embodiments, the control plane uses the mapping of block 1506 to determine a subset of L2 VNICs that are subject to span ports. Based on the topology information, the control plane determines the NVDs that host such L2 VNICs. Thus, the control plane determines the specific span port information that each NVD should receive. For example, if a first NVD hosts two L2 VNICs, each L2 VNIC associated with its own specific span port information, the first NVD will receive a first span port information specific to the first L2 VNIC and a second span port information specific to the second L2 VNIC. Similarly, if a second NVD hosts one L2 VNIC, the L2 VNIC associated with its own specific span port information, the second NVD will receive the span port information. Furthermore, if the third NVD hosts one or more L2 VNICs that are not associated with any span port information, the third NVD will not receive any span port information.

[0227] The control plane sends the span port information to a set of NVDs in block 1510. In one example, each span port information is sent to an associated NVD during deployment of the L2 VLAN and / or in a push or pull mechanism after deployment.

[0228] 16, a flow diagram is shown illustrating a process 1600 for generating span port information, according to an embodiment. In some embodiments, the process 1600 can be performed by a control plane or another VCN system that manages the deployment of layer 2 virtual networks on an underlying physical network.

[0229] Process 1600 begins at block 1602, where a control plane stores a mapping between a customer-specified Layer 2 virtual network configuration and the network topology of the Layer 2 virtual network. In some embodiments, the control plane receives customer input specifying a VLAN configuration. The control plane also tracks the deployment of resources of the L2 virtual network on the physical network and receives information from the NVD regarding incoming traffic learning. Given the deployment and incoming traffic learning, the control plane generates the mapping.

[0230] At block 1604, the control plane receives customer input indicating first filtering and processing criteria for the span ports. In some embodiments, the customer input may be received from a customer device via an API call and / or a console. The first filtering criteria may use customer-defined parameters (e.g., copy frames received on a specific port with a customer-defined nomenclature). In comparison, the processing criteria may use network-defined parameters (e.g., GRE encapsulate the copy). Additionally, the customer input may indicate destinations to which the frame copies may be sent. The destinations may be specified using customer-defined (e.g., send the frame copies to a specific port) or network-defined (e.g., send the frame copies to an L2 compute instance).

[0231] At block 1606, the control plane generates second filtering criteria based on the first filtering criteria and the mapping. In some embodiments, the control plane converts customer-defined parameters into parameters specific to the topology of the L2 virtual network. For example, the first filtering criteria includes an identifier of a port, but the identifier of the corresponding L2 VNIC is included in the second filtering criteria instead. Similarly, if a destination is specified using a customer definition, the customer definition can be updated to a network definition based on the mapping, and the network definition can include span port information instead.

[0232] At block 1608, the control plane generates span port information including the second criterion and the processing criterion. In some embodiments, the span port configuration specified by the customer is updated, thereby replacing the first filtering criterion with the second filtering criterion. Additionally, when the destination is updated, the updated destination is included in the span port information.

[0233] 17, a flow chart illustrating a process 1700 for using span port information. In some embodiments, the process 1700 may be performed by a NVD hosting an L2 VNIC-L2 virtual switch pair for which span ports may be applicable.

[0234] Process 1700 begins at block 1702, where the NVD hosts L2 VNICs and L2 virtual switches that belong to a customer's L2 virtual network (e.g., a VLAN). In some embodiments, the L2 VNICs and L2 virtual switches are associated with an L2 compute instance of the L2 virtual network. The L2 compute instance may be hosted on a host machine that is communicatively connected to the NVD.

[0235] At block 1704, the NVD receives and stores span port information associated with the L2 VNICs and / or L2 virtual switches. In some embodiments, the NVD receives the span port information from the control plane. The span port information may include one or more filtering criteria and / or one or more processing criteria and may indicate one or more destinations for receiving frame copies.

[0236] At block 1706, the NVD receives the frame with the header information. In some embodiments, the frame may be received from an L2 compute instance via a host machine (in which case the frame is an outgoing frame sent from the L2 compute instance). In some embodiments, the frame may be received from another L2 compute instance via another host machine (in which case the frame is an incoming frame sent to the L2 compute instance).

[0237] In block 1708, the NVD determines that the frame is associated with the L2 VNIC based on the header information. In various embodiments, the header information includes an interface identifier and / or a MAC address associated with the L2 VNIC. The NVD can use the interface identifier and / or the MAC address to determine that the frame should be processed by the L2 VNIC.

[0238] At block 1710, the NVD determines that the span port information is applicable. In some embodiments, the span port information indicates one or more filtering criteria. The NVD parses the header information for comparison to the one or more filtering criteria. If a match is found based on the comparison, the NVD determines that the frame should be copied. Based on one or more processing criteria, the NVD determines how many copies to generate and any additional processing (e.g., any encapsulation) to apply to them. Additionally, the NVD determines one or more destinations to receive the frame copies based on the span port information.

[0239] At block 1712, the NVD generates a copy of the frame and further processes the copy, if applicable. In some embodiments, the NVD does not modify the payload of the frame. Instead, the NVD updates the header information and uses the updated header information to form the frame copy, along with the original payload. The updated header information can modify the destination address, protocol identifier, and / or include encapsulation information.

[0240] In block 1712, the NVD transmits the frame and sends a copy of the frame to the destination. In some embodiments, the NVD does not modify the original frame. For example, if the frame is an incoming frame to an L2 compute instance, the L2 VNIC processes the frame for passing to the L2 compute instance. If the frame is an outgoing frame from an L2 compute instance, the L2 VNIC passes the frame to the L2 virtual switch for further processing and transmission to the destination L2 VNIC. In comparison, sending a copy typically involves an outgoing path. Thus, the L2 VNIC passes the frame copy to the L2 virtual switch for further processing and transmission to the destination L2 VNIC.

[0241] 18, a flowchart illustrating a process 1800 for determining whether span port information is applicable is shown. In some embodiments, the process 1800 may be performed by a NVD hosting an L2 VNIC for which span ports may be applicable. Compared to the process 1700, the process 1800 may include adding the span port information (or a portion thereof) to the frame such that enforcement of the span port information may be performed at various nodes between the source compute instance and the destination compute instance.

[0242] Process 1800 begins at block 1802, where the NVD receives a frame having header information. In some embodiments, the frame may be received from an L2 compute instance via a host machine (in which case the frame is an outgoing frame sent from the L2 compute instance). In some embodiments, the frame may be received from another L2 compute instance via another host machine (in which case the frame is an incoming frame sent to the L2 compute instance).

[0243] In block 1804, the NVD determines whether the frame may be copied and whether the copy should be further processed. If the frame is copied, block 1806 follows block 1804, otherwise, block 1810 follows block 1804. In some embodiments, this determination involves header information and span port information. In particular, the NVD analyzes the header information for comparison with the filtering criteria. If a match is found based on the comparison, the NVD determines that the frame should be copied. Based on the processing criteria, the NVD determines the type of processing to apply to the copy. While a filtering criteria match may be found, the applicability of the span port information may depend on whether the customer requested the copy on an inbound flow and / or an outbound flow. This flow directionality may be indicated in the span port information. Thus, before or after determining the filtering criteria match, the NVD may determine a flow match (e.g., if a frame is received and the flow directionality is inbound, or if a frame is sent and the flow directionality is outbound). If a flow match is also found, the NVD determines that the frame can be copied.

[0244] At block 1806, the NVD generates a copy for the frame. In some embodiments, the NVD generates a copy by copying the payload and modifying the header information according to filtering and processing criteria. Alternatively, the NVD generates a copy by copying the payload and the header information according to filtering and processing criteria, and adding encapsulating header information.

[0245] The NVD sends the copy at block 1808. In some embodiments, the copy is sent to a destination indicated by the span port information.

[0246] In block 1810, the NVD transmits the frame. In some embodiments, the frame is transmitted to a destination indicated by the header information. Frame transmission may be performed before, in parallel with, or after any of blocks 1804, 1806, 1808, or 1810.

[0247] C. Exemplary Infrastructure-as-a-Service Architecture As mentioned above, Infrastructure as a Service (IaaS) is one particular type of cloud computing. IaaS may be configured to provide virtualized computing resources over a public network (e.g., the Internet). In an IaaS model, a cloud computing provider may host infrastructure elements (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer), etc.). In some cases, an IaaS provider may provide various services (e.g., billing, monitoring, logging, security, load balancing and clustering, etc.) that accompany the infrastructure elements. Thus, since these services may be policy-driven, an IaaS user may implement policies to drive load balancing to maintain application availability and performance.

[0248] In some examples, an IaaS customer can access resources and services over a wide area network (WAN), such as the Internet, and can use the cloud provider's services to install the remaining elements of the application stack. For example, a user can log into an IaaS platform to create virtual machines (VMs), install an operating system (OS) on each VM, deploy middleware such as databases, create storage buckets for workloads and backups, and install enterprise software on the VMs. The customer can use the provider's services to perform a variety of functions, including balancing network traffic, troubleshooting applications, monitoring performance, managing disaster recovery, etc.

[0249] In most cases, the cloud computing model requires the participation of a cloud provider, which may be, but does not have to be, a third-party service dedicated to providing (e.g., offering, renting, selling) IaaS. Enterprises can also deploy private clouds and become providers of infrastructure services.

[0250] In some examples, IaaS deployment is the process of deploying a new application or a new version of an application onto provisioned application servers, etc. IaaS deployment may include the process of provisioning a server (e.g., installing libraries, daemons, etc.). IaaS deployment is often managed below the hypervisor layer (e.g., servers, storage, network hardware, and virtualization) by the cloud provider. Thus, the customer may perform OS, middleware, and / or application deployment (e.g., self-service virtual machines (e.g., that can be spun up on demand), etc.

[0251] In some instances, IaaS provisioning may include obtaining a computer or virtual host to be used and installing the necessary libraries or services on the computer or virtual host. In most cases, deployment does not include provisioning, which must be performed first.

[0252] In some cases, there are two different challenges in IaaS provisioning. First, there is the challenge of provisioning an initial set of infrastructure before anything can be done. Second, there is the challenge of evolving the existing infrastructure (e.g., adding new services, modifying services, removing services) after everything has been provisioned. In some cases, these two challenges can be addressed by allowing the configuration of the infrastructure to be defined declaratively. In other words, the infrastructure (e.g., which elements are required and how these elements interact) may be defined by one or more configuration files. Thus, the overall topology of the infrastructure (e.g., which resources depend on which and how they work together) can be described declaratively. In some instances, once the topology is defined, workflows can be generated to create and / or manage the different elements described in the configuration files.

[0253] In some examples, the infrastructure can include many interconnected elements. For example, there may be one or more virtual private clouds (VPCs) (e.g., potentially on-demand pools of configurable and / or shared computing resources), also known as a core network. In some examples, there may be one or more security group rules and one or more virtual machines (VMs) that are provisioned to define how the security of the network is configured. Other infrastructure elements such as load balancers, databases, etc. may also be provisioned. The infrastructure can evolve incrementally as more and more infrastructure elements are desired and / or added.

[0254] In some examples, continuous deployment techniques may be employed to enable deployment of infrastructure code across various virtual computing environments. The described techniques may also enable infrastructure management within these environments. In some examples, a service team may write code that is desired to be deployed to one or more, typically many, different production environments (e.g., across a variety of different geographic locations, sometimes spanning the globe). However, in some examples, the infrastructure for deploying the code must first be set up. In some examples, provisioning may be done manually, resources may be provisioned using a provisioning tool, and / or the code may be deployed using a deployment tool after the infrastructure has been provisioned.

[0255] 19 is a block diagram 1900 illustrating an example pattern of an IaaS architecture in accordance with at least one embodiment. A service operator 1902 may be communicatively coupled to a secure host tenancy 1904, which may include a virtual cloud network (VCN) 1906 and a secure host subnet 1908. In some examples, the service operator 1902 may use one or more client computing devices. The one or more client computing devices may run software such as Microsoft Windows Mobile® and / or various mobile operating systems such as iOS, Windows Phone, Android®, BlackBerry 8, and Palm OS, and may be Internet, email, short message service (SMS), BlackBerry®, or other communication protocol enabled handheld mobile devices (e.g., iPhone®, cell phone, iPad®, tablet, personal digital assistant (PDA) or wearable devices (Google® Glass® head mounted display). The client computing devices may be general purpose personal computers including, by way of example, personal computers and / or laptop computers running various versions of Microsoft Windows® operating systems, Apple Macintosh® operating systems, and / or Linux® operating systems. Alternatively, the client computing devices may be workstation computers running various commercially available UNIX® or UNIX-like operating systems including, but not limited to, various GNU / Linux operating systems, e.g., Google Chrome® OS.Alternatively or additionally, the client computing devices may be other electronic devices capable of communicating over the VCN 1906 and / or an Internet-accessible network, such as thin-client computers, Internet-enabled gaming systems (e.g., Microsoft Xbox® game consoles with or without Kinect® gesture input devices), and / or personal messaging devices.

[0256] The VCN 1906 may include a local peering gateway (LPG) 1910 that may be communicatively coupled to a secure shell (SSH) VCN 1912 via an LPG 1910 that is included in the SSH VCN 1912. The SSH VCN 1912 may include an SSH subnet 1914, which may be communicatively coupled to a control plane VCN 1916 via an LPG 1910 that is included in the control plane VCN 1916. The SSH VCN 1912 may also be communicatively coupled to a data plane VCN 1918 via the LPG 1910. The control plane VCN 1916 and the data plane VCN 1918 may be included in a service tenancy 1919, which may be owned and / or operated by the IaaS provider.

[0257] The control plane VCN 1916 may include a control plane demilitarized zone (DMZ) tier 1920 that serves as a perimeter network (e.g., a portion of an enterprise network between an enterprise intranet and an external network). DMZ-based servers may have a certain level of reliability and contain security breaches. Additionally, the DMZ tier 1920 may include one or more load balancer (LB) subnets 1922, a control plane app tier 1924 that may include app subnets 1926, and a control plane data tier 1928 that may include database (DB) subnets 1930 (e.g., a front-end DB subnet and / or a back-end DB subnet). LB subnet 1922 included in control plane DMZ tier 1920 may be communicatively coupled to app subnet 1926 included in control plane app tier 1924 and an Internet gateway 1934 that may be included in control plane VCN 1916, which may be communicatively coupled to DB subnet 1930, service gateway 1936, and network address translation (NAT) gateway 1938 included in control plane data tier 1928. Control plane VCN 1916 may include service gateway 1936 and NAT gateway 1938.

[0258] The control plane VCN 1916 can include a data plane mirror app layer 1940, which can include an app subnet 1926. The app subnet 1926 included in the data plane mirror app layer 1940 can include a virtual network interface controller (VNIC) 1942 on which a compute instance 1944 can run. The compute instance 1944 can communicatively couple the app subnet 1926 of the data plane mirror app layer 1940 to the app subnet 1926, which can be included in the data plane app layer 1946.

[0259] The data plane VCN 1918 may include a data plane app layer 1946, a data plane DMZ layer 1948, and a data plane data layer 1950. The data plane DMZ layer 1948 may include a LB subnet 1922 that may be communicatively coupled to an app subnet 1926 of the data plane app layer 1946 and an Internet gateway 1934 of the data plane VCN 1918. The app subnet 1926 may be communicatively coupled to a service gateway 1936 of the data plane VCN 1918 and a NAT gateway 1938 of the data plane VCN 1918. The data plane data layer 1950 may also include a DB subnet 1930 that may be communicatively coupled to the app subnet 1926 of the data plane app layer 1946.

[0260] The internet gateway 1934 of the control plane VCN 1916 and the internet gateway 1934 of the data plane VCN 1918 may be communicatively coupled to a metadata management service 1952, which may be communicatively coupled to the public internet 1954. The public internet 1954 may be communicatively coupled to a NAT gateway 1938 of the control plane VCN 1916 and the NAT gateway 1938 of the data plane VCN 1918. The service gateway 1936 of the control plane VCN 1916 and the service gateway 1936 of the data plane VCN 1918 may be communicatively coupled to cloud services 1956.

[0261] In some examples, a service gateway 1936 of the control plane VCN 1916 or the data plane VCN 1918 can make application programming interface (API) calls to cloud services 1956 without traversing the public Internet 1954. The API calls from the service gateway 1936 to the cloud services 1956 can be one-way. The service gateway 1936 can make API calls to the cloud services 1956, and the cloud services 1956 can send request data to the service gateway 1936. However, the cloud services 1956 may not initiate the API calls to the service gateway 1936.

[0262] In some examples, secure host tenancy 1904 may be directly connected to service tenancy 1919, which may be an orphan. Secure host subnet 1908 can communicate with SSH subnet 1914 via LPG 1910, which allows bidirectional communication with the orphan system. By connecting secure host subnet 1908 to SSH subnet 1914, secure host subnet 1908 can access other entities in service tenancy 1919.

[0263] The control plane VCN 1916 allows users of the service tenancy 1919 to configure or provision desired resources. The desired resources provisioned in the control plane VCN 1916 may be deployed or used in the data plane VCN 1918. In some examples, the control plane VCN 1916 may be isolated from the data plane VCN 1918, and the data plane mirror app layer 1940 of the control plane VCN 1916 can communicate with the data plane app layer 1946 of the data plane VCN 1918 via a VNIC 1942, which may be included in the data plane mirror app layer 1940 and the data plane app layer 1946.

[0264] In some examples, a user or customer of the system may make a request, such as, for example, a create, read, update, or delete (CRUD) operation, via the public internet 1954, which may communicate the request to a metadata management service 1952. The metadata management service 1952 may communicate the request to the control plane VCN 1916 via an internet gateway 1934. The request may be received by a LB subnet 1922 included in the control plane DMZ layer 1920. The LB subnet 1922 may determine that the request is valid, and in response to this determination, the LB subnet 1922 may send the request to an app subnet 1926 included in the control plane app layer 1924. If the request is validated and requires a call to the public internet 1954, the call to the public internet 1954 may be sent to a NAT gateway 1938, which may make the call to the public internet 1954. A memory for storing the request may be stored in the DB subnet 1930.

[0265] In some examples, the data plane mirror app layer 1940 can facilitate direct communication between the control plane VCN 1916 and the data plane VCN 1918. For example, it may be desirable for a change, update, or other suitable modification to a configuration to be applied to resources included in the data plane VCN 1918. Because the control plane VCN 1916 can communicate directly with the resources included in the data plane VCN 1918 via VNIC 1942, the change, update, or other suitable modification to the configuration can be performed.

[0266] In some embodiments, the control plane VCN 1916 and the data plane VCN 1918 may be included in the service tenancy 1919. In this case, a user or customer of the system may not own or operate either the control plane VCN 1916 or the data plane VCN 1918. Instead, an IaaS provider may own or operate the control plane VCN 1916 and the data plane VCN 1918, both of which may be included in the service tenancy 1919. This embodiment may prevent a user or customer from interacting with other users' resources or other customers' resources by enabling network isolation. This embodiment may also enable a user or customer of the system to store databases privately without having to rely on the public internet 1954, which may not have the desired level of security for storage.

[0267] In another embodiment, the LB subnet 1922 included in the control plane VCN 1916 may be configured to receive signals from the service gateway 1936. In this embodiment, the control plane VCN 1916 and the data plane VCN 1918 may be configured to be called by the IaaS provider's customers without calling the public Internet 1954. The IaaS provider's customers may desire this embodiment because databases used by the customers may be stored in the service tenancy 1919, which may be controlled by the IaaS provider and isolated from the public Internet 1954.

[0268] FIG. 20 is a block diagram 2000 illustrating another example parameter of an IaaS architecture in accordance with at least one embodiment. A service operator 2002 (e.g., service operator 1902 of FIG. 19 ) may be communicatively coupled to a secure host tenancy 2004 (e.g., secure host tenancy 1904 of FIG. 19 ), which may include a virtual cloud network (VCN) 2006 (e.g., VCN 1906 of FIG. 19 ) and a secure host subnet 2008 (e.g., secure host subnet 1908 of FIG. 19 ). The VCN 2006 may include a local peering gateway (LPG) 2010 (e.g., LPG 1910 of FIG. 19 ), which may be communicatively coupled to a secure shell (SSH) VCN 2012 (e.g., SSH VCN 1912 of FIG. 19 ) via an LPG 1910 included in the SSH VCN 2012. SSH VCN 2012 can include an SSH subnet 2014 (e.g., SSH subnet 1914 in FIG. 19), which can be communicatively coupled to a control plane VCN 2016 (e.g., control plane VCN 1916 in FIG. 19) via an LPG 2010 included in the control plane VCN 2016. The control plane VCN 2016 can be included in a service tenancy 2019 (e.g., service tenancy 1919 in FIG. 19), and the data plane VCN 2018 (e.g., data plane VCN 1918 in FIG. 19) can be included in a customer tenancy 2021, which can be owned or operated by a user or customer of the system.

[0269] The control plane VCN 2016 may include a control plane DMZ tier 2020 (e.g., control plane DMZ tier 1920 of FIG. 19 ) that may include a LB subnet 2022 (e.g., LB subnet 1922 of FIG. 19 ), a control plane app tier 2024 (e.g., control plane app tier 1924 of FIG. 19 ) that may include an app subnet 2026 (e.g., app subnet 1926 of FIG. 19 ), and a control plane data tier 2028 (e.g., control plane data tier 1928 of FIG. 19 ) that may include a database (DB) subnet 2030 (e.g., similar to DB subnet 1930 of FIG. 19 ). The LB subnet 2022 included in the control plane DMZ tier 2020 may be communicatively coupled to the app subnet 2026 included in the control plane app tier 2024 and to an Internet gateway 2034 (e.g., Internet gateway 1934 of FIG. 19 ), which may be included in the control plane VCN 2016. The app subnet 2026 may be communicatively coupled to a DB subnet 2030, a service gateway 2036 (e.g., the service gateway of FIG. 19 ), and a network address translation (NAT) gateway 2038 (e.g., the NAT gateway 1938 of FIG. 19 ) included in the control plane data layer 2028. The control plane VCN 2016 may include the service gateway 2036 and the NAT gateway 2038.

[0270] The control plane VCN 2016 can include a data plane mirror app layer 2040 (e.g., data plane mirror app layer 1940 of FIG. 19 ), which can include an app subnet 2026. The app subnet 2026 included in the data plane mirror app layer 2040 can include a virtual network interface controller (VNIC) 2042 (e.g., VNIC 1942) that can run a compute instance 2044 (e.g., similar to compute instance 1944 of FIG. 19 ). The compute instance 2044 can facilitate communication between the app subnet 2026 of the data plane mirror app layer 2040 and the app subnet 2026 that can be included in the data plane app layer 2046 (e.g., data plane app layer 1946 of FIG. 19 ) via the VNIC 2042 included in the data plane mirror app layer 2040 and the VNIC 2042 included in the data plane app layer 2046.

[0271] The Internet gateway 2034 included in the control plane VCN 2016 may be communicatively coupled to a metadata management service 2052 (e.g., metadata management service 1952 of FIG. 19), which may be communicatively coupled to a public Internet 2054 (e.g., public Internet 1954 of FIG. 19). The public Internet 2054 may be communicatively coupled to a NAT gateway 2038 included in the control plane VCN 2016. The service gateway 2036 included in the control plane VCN 2016 may be communicatively coupled to cloud services 2056 (e.g., cloud services 1956 of FIG. 19).

[0272] In some examples, the data plane VCN 2018 may be included in the customer tenancy 2021. In this case, the IaaS provider may provide a control plane VCN 2016 for each customer, and the IaaS provider may configure a unique compute instance 2044 for each customer, which is included in the service tenancy 2019. Each compute instance 2044 may allow communication between the control plane VCN 2016 included in the service tenancy 2019 and the data plane VCN 2018 included in the customer tenancy 2021. The compute instance 2044 may allow resources provisioned in the control plane VCN 2016 included in the service tenancy 2019 to be deployed or used in the data plane VCN 2018 included in the customer tenancy 2021.

[0273] In another example, a customer of the IaaS provider may have a database that resides in the customer tenancy 2021. In this example, the control plane VCN 2016 may include a data plane minor app tier 2040, which may include an app subnet 2026. The data plane mirror app tier 2040 may reside in the data plane VCN 2018, but the data plane mirror app tier 2040 may not reside in the data plane VCN 2018. That is, the data plane mirror app tier 2040 may have access to the customer tenancy 2021, but the data plane mirror app tier 2040 may not reside in the data plane VCN 2018 and may not be owned or operated by the IaaS provider customer. The data plane mirror app tier 2040 may be configured to make calls to the data plane VCN 2018, but may not be configured to make calls to any entities included in the control plane VCN 2016. A customer may desire to deploy or use resources in the data plane VCN 2018 provisioned to the control plane VCN 2016, and the data plane mirror application tier 2040 may facilitate the desired deployment or other use of the customer's resources.

[0274] In some embodiments, the IaaS provider's customer can apply filters to the data plane VCN 2018. In this embodiment, the customer can determine what the data plane VCN 2018 can access, and the customer can restrict access from the data plane VCN 2018 to the public internet 2054. The IaaS provider may not be able to apply filters or control access from the data plane VCN 2018 to any external networks or databases. Applying filters and controls to the data plane VCN 2018 that the customer includes in the customer tenancy 2021 can help isolate the data plane VCN 2018 from other customers and the public internet 2054.

[0275] In some embodiments, cloud services 2056 can be called by service gateway 2036 to access services that may not be on the public internet 2054, on the control plane VCN 2016, or on the data plane VCN 2018. The connection between cloud services 2056 and the control plane VCN 2016 or data plane VCN 2018 may not be live or continuous. Cloud services 2056 may be on another network owned or operated by the IaaS provider. Cloud services 2056 may be configured to receive calls from service gateway 2036 and may be configured not to receive calls from the public internet 2054. Some cloud services 2056 may be isolated from other cloud services 2056, and control plane VCN 2016 may be isolated from cloud services 2056 that may not be located in the same region as control plane VCN 2016. For example, control plane VCN 2016 may be located in “Region 1” and cloud service “Deployment 19” may be located in “Region 1” and “Region 2”. If a call to a deployment 19 is made by a service gateway 2036 included in a control plane VCN 2016 located in Region 1, the call may be sent to the deployment 19 in Region 1. In this example, the control plane VCN 2016 or the deployment 19 in Region 1 may not be communicatively coupled with the deployment 19 in Region 2.

[0276] FIG. 21 is a block diagram 2100 illustrating another example pattern of an IaaS architecture, according to at least one embodiment. A service operator 2102 (e.g., service operator 1902 of FIG. 19 ) may be communicatively coupled to a secure host tenancy 2104 (e.g., secure host tenancy 1904 of FIG. 19 ), which may include a virtual cloud network (VCN) 2106 (e.g., VCN 1906 of FIG. 19 ) and a secure host subnet 2108 (e.g., secure host subnet 1908 of FIG. 19 ). The VCN 2106 may include an LPG 2110 (e.g., LPG 1910 of FIG. 19 ), which may be communicatively coupled to an SSH VCN 2112 (e.g., SSH VCN 1912 of FIG. 19 ) via an LPG 2110 included in the SSH VCN 2112. SSH VCN 2112 can include SSH subnet 2114 (e.g., SSH subnet 1914 in FIG. 19 ), which may be communicatively coupled to control plane VCN 2116 (e.g., control plane VCN 1916 in FIG. 19 ) via LPG 2110 included in control plane VCN 2116, and may be communicatively coupled to data plane VCN 2118 (e.g., data plane 1918 in FIG. 19 ) via LPG 2110 included in data plane VCN 2118. Control plane VCN 2116 and data plane VCN 2118 may be included in service tenancy 2119 (e.g., service tenant 1919 in FIG. 19 ).

[0277] The control plane VCN 2116 may include a control plane DMZ layer 2120 (e.g., control plane DMZ layer 1920 of FIG. 19 ) that may include a load balancer (LB) subnet 2122 (e.g., LB subnet 1922 of FIG. 19 ), a control plane app layer 2124 (e.g., control plane app layer 1924 of FIG. 19 ) that may include an app subnet 2126 (e.g., similar to app subnet 1926 of FIG. 19 ), and a control plane data layer 2128 (e.g., control plane data layer 1928 of FIG. 19 ) that may include a DB subnet 2130. The LB subnet 2122 included in the control plane DMZ layer 2120 may be communicatively coupled to the app subnet 2126 included in the control plane app layer 2124 and to an Internet gateway 2134 (e.g., Internet gateway 1934 of FIG. 19 ), which may be included in the control plane VCN 2116. The app subnet 2126 may be communicatively coupled to a DB subnet 2130 included in the control plane data layer 2128, and to a service gateway 2136 (e.g., the service gateway of FIG. 19) and a network address translation (NAT) gateway 2138 (e.g., the NAT gateway 1938 of FIG. 19). The control plane VCN 2116 may include the service gateway 2136 and the NAT gateway 2138.

[0278] The data plane VCN 2118 may include a data plane app layer 2146 (e.g., data plane app layer 1946 of FIG. 19 ), a data plane DMZ layer 2148 (e.g., data plane DMZ layer 1948 of FIG. 19 ), and a data plane data layer 2150 (e.g., data plane data layer 1950 of FIG. 19 ). The data plane DMZ layer 2148 may include a LB subnetwork 2122 that may be communicatively coupled to the data plane app layer 2146 included in the data plane VCN 2118 and a trusted app subnetwork 2160 and an untrusted app subnetwork 2162 of the Internet gateway 2134. The trusted app subnetwork 2160 may be communicatively coupled to a service gateway 2136 included in the data plane VCN 2118, a NAT gateway 2138 included in the data plane VCN 2118, and a DB subnetwork 2130 included in the data plane data layer 2150. The untrusted app subnet 2162 may be communicatively coupled to a service gateway 2136 included in the data plane VCN 2118 and to a DB subnet 2130 included in the data plane data layer 2150. The data plane data layer 2150 may include a DB subnet 2130, which may be communicatively coupled to a service gateway 2136 included in the data plane VCN 2118.

[0279] The untrusted app subnet 2162 may include one or more primary VNICs 2164(1)-(N), which may be communicatively coupled to tenant virtual machines (VMs) 2166(1)-(N). Each tenant VM 2166(1)-(N) may be communicatively coupled to a respective app subnet 2167(1)-(N), which may be included in a respective container egress VCN 2168(1)-(N), which may be included in a respective customer tenancy 2170(1)-(N). Each secondary VNIC 2172(1)-(N) may facilitate communication between the untrusted app subnet 2162 included in the data plane VCN 2118 and the app subnet included in the container egress VCN 2168(1)-(N). Each container egress VCN 2168(1)-(N) may include a NAT gateway 2138, which may be communicatively coupled to the public Internet 2154 (e.g., public Internet 1954 of FIG. 19 ).

[0280] The Internet gateway 2134 included in the control plane VCN 2116 and the Internet gateway 2134 included in the data plane VCN 2118 may be communicatively coupled to a metadata management service 2152 (e.g., metadata management system 1952 of FIG. 19 ), which may be communicatively coupled to the public Internet 2154. The public Internet 2154 may be communicatively coupled to a NAT gateway 2138 included in the control plane VCN 2116 and the NAT gateway 2138 included in the data plane VCN 2118. The service gateway 2136 included in the control plane VCN 2116 and the service gateway 2136 included in the data plane VCN 2118 may be communicatively coupled to cloud services 2156.

[0281] In some embodiments, data plane VCN 2118 may be consolidated into customer tenancies 2170. This consolidation may be useful or desirable for an IaaS provider's customers in some cases, such as when they may want support when executing code. Customers may provide code that, when executed, may be disruptive, may communicate with other customer resources, or may cause undesirable effects. Thus, the IaaS provider may determine whether or not to run code that a customer has provided to the IaaS provider.

[0282] In some examples, an IaaS provider's customer can grant temporary network access to the IaaS provider and can request a function to be added to the data plane app layer 2146. The code to perform the function may be executed in the VMs 2166(1)-(N) but cannot be configured to run elsewhere on the data plane VCN 2118. Each VM 2166(1)-(N) may be connected to one customer tenancy 2170. Each container 2171(1)-(N) included in the VMs 2166(1)-(N) may be configured to execute code. In this case, there may be double isolation (e.g., the containers 2171(1)-(N) execute code, and the containers 2171(1)-(N) may be included in at least the VMs 2166(1)-(N) included in the untrusted app subnet 2162), which can help prevent erroneous or unwanted code from damaging the IaaS provider's network or from damaging a different customer's network. Containers 2171(1)-(N) may be communicatively coupled to customer tenancy 2170 and may be configured to send or receive data from customer tenancy 2170. Containers 2171(1)-(N) may not be configured to send or receive data from any other entity in data plane VCN 2118. Once code execution is complete, the IaaS provider may kill or discard containers 2171(I)-(N).

[0283] In some embodiments, trusted app subnet 2160 can execute code that may be owned or operated by the IaaS provider. In this embodiment, trusted app subnet 2160 can be communicatively coupled to DB subnet 2130 and configured to perform CRUD operations in DB subnet 2130. Untrusted app subnet 2162 can be communicatively coupled to DB subnet 2130, but in this embodiment, untrusted app subnet 2162 can be configured to perform read operations in DB subnet 2130. Containers 2171(1)-(N) included in each customer's VMs 2166(1)-(N) that can execute code from the customer may not be communicatively coupled to DB subnet 2130.

[0284] In other embodiments, the control plane VCN 2116 and the data plane VCN 2118 may not be directly communicatively coupled. In this embodiment, there may be no direct communication between the control plane VCN 2116 and the data plane VCN 2118. However, there may be indirect communication by at least one method. An LPG 2110 may be established by an IaaS provider that can facilitate communication between the control plane VCN 2116 and the data plane VCN 2118. In another example, the control plane VCN 2116 or the data plane VCN 2118 can make a call to a cloud service 2156 through a service gateway 2136. For example, a call from the control plane VCN 2116 to the cloud service 2156 can include a request for a service that can communicate with the data plane VCN 2118.

[0285] 22 is a block diagram 2200 illustrating another example parameter of an IaaS architecture in accordance with at least one embodiment. A service operator 2202 (e.g., service operator 1902 of FIG. 19) may be communicatively coupled to a secure host tenancy 2204 (e.g., secure host tenancy 1904 of FIG. 19), which may include a virtual cloud network (VCN) 2206 (e.g., VCN 1906 of FIG. 19) and a secure host subnet-2208 (e.g., secure host subnet 1908 of FIG. 19). VCN 2206 may include an LPG 2210 (e.g., LPG 1910 of FIG. 19), which may be communicatively coupled to an SSH VCN 2212 (e.g., SSH VCN 1912 of FIG. 19) via an LPG 2210 included in SSH VCN 2212. SSH VCN 2212 can include SSH subnet-2214 (e.g., SSH subnet 1914 in FIG. 19), which may be communicatively coupled to control plane VCN 2216 (e.g., control plane VCN 1916 in FIG. 19) via LPG 2210 included in control plane VCN 2216, and may be communicatively coupled to data plane VCN 2218 (e.g., data plane 1918 in FIG. 19) via LPG 2210 included in data plane VCN 2218. Control plane VCN 2216 and data plane VCN 2218 may be included in service tenancy 2219 (e.g., service tenancy 1919 in FIG. 19).

[0286] The control plane VCN 2216 may include a control plane DMZ layer 2220 (e.g., control plane DMZ layer 1920 of FIG. 19) that may include a LB subnet 2222 (e.g., LB subnet 1922 of FIG. 19), a control plane app layer 2224 (e.g., control plane app layer 1924 of FIG. 19) that may include an app subnet 2226 (e.g., app subnet 1926 of FIG. 19), a control plane data tier 2228 (e.g., control plane data tier 1928 of FIG. 19) that may include a DB subnet 2230 (e.g., DB subnet 2130 of FIG. 21). The LB subnet 2222 included in the control plane DMZ layer 2220 may be communicatively coupled to the app subnet 2226 included in the control plane app layer 2224 and an Internet gateway 2234 (e.g., Internet gateway 1934 of FIG. 19) that may be included in the control plane VCN 2216. The app subnet 2226 may be communicatively coupled to a DB subnet 2230 included in the control plane data layer 2228, and to a service gateway 2236 (e.g., the service gateway of FIG. 19) and a network address translation (NAT) gateway 2238 (e.g., the NAT gateway 1938 of FIG. 19). The control plane VCN 2216 may include the service gateway 2236 and the NAT gateway 2238.

[0287] The data plane VCN 2218 can include a data plane app layer 2246 (e.g., data plane app layer 1946 of FIG. 19 ), a data plane DMZ layer 2248 (e.g., data plane DMZ layer 1948 of FIG. 19 ), and a data plane data layer 2250 (e.g., data plane data layer 1950 of FIG. 19 ). The data plane DMZ layer 2248 can include a trusted app subnet 2260 (e.g., trusted app subnet 2160 of FIG. 21 ) and an untrusted app subnet 2262 (e.g., untrusted app subnet 2162 of FIG. 21 ) of the data plane app layer 2246 and a LB subnet 2222 that can be communicatively coupled to an Internet gateway 2234 included in the data plane VCN 2218. The trusted app subnet 2260 may be communicatively coupled to a service gateway 2236 included in the data plane VCN 2218, a NAT gateway 2238 included in the data plane VCN 2218, and a DB subnet 2230 included in the data plane data layer 2250. The untrusted app subnet 2262 may be communicatively coupled to a service gateway 2236 included in the data plane VCN 2218, and a DB subnet 2230 included in the data plane data layer 2250. The data plane data layer 2250 may include a DB subnet 2230 that may be communicatively coupled to a service gateway 2236 included in the data plane VCN 2218.

[0288] The untrusted app subnet 2262 may include primary YNICs 2264(1)-(N), which may be communicatively coupled to tenant virtual machines (VMs) 2266(1)-(N) residing in the untrusted app subnet 2262. Each tenant VM 2266(1)-(N) may execute code in a respective container 2267(1)-(N), which may be communicatively coupled to an app subnet 2226, which may be included in a data plane app layer 2246, which may be included in a container egress VCN 2268. Each secondary VNIC 2272(1)-(N) may facilitate communication between the untrusted app subnet 2262, which may be included in the data plane VCN 2218, and the app subnet, which may be included in the container egress VCN 2268. The container egress VCN may include a NAT gateway 2238, which may be communicatively coupled to the public Internet 2254 (e.g., public Internet 1954 of FIG. 19 ).

[0289] The Internet gateway 2234 included in the control plane VCN 2216 and the Internet gateway 2234 included in the data plane VCN 2218 may be communicatively coupled to a metadata management service 2252 (e.g., metadata management system 1952 of FIG. 19 ), which may be communicatively coupled to the public Internet 2254. The public Internet 2254 may be communicatively coupled to the Internet gateway 2234 included in the control plane VCN 2216 and the NAT gateway 2238 included in the data plane VCN 2218. The Internet gateway 2234 included in the control plane VCN 2216 and the service gateway 2236 included in the data plane VCN 2218 may be communicatively coupled to cloud services 2256.

[0290] In some examples, the pattern illustrated by the architecture of block diagram 2200 of FIG. 22 may be considered an exception to the pattern illustrated by the architecture of block diagram 2100 of FIG. 21 and may be desirable for a customer of an IaaS provider when the IaaS provider cannot directly communicate with the customer (e.g., in a non-connected region). The customer may have real-time access to each container 2267(1)-(N) included in each customer's VM 2266(1)-(N). The containers 2267(1)-(N) may be configured to call each secondary VNIC 2272(1)-(N) included in the app subnet 2226 of the data plane app tier 2246 that may be included in the container egress VCN 2268. The secondary VNIC 2272(1)-(N) may send the call to the NAT gateway 2238 that may send the call to the public Internet 2254. In this example, containers 2267(1)-(N) that a customer can access in real time may be isolated from control plane VCN 2216 and may be isolated from other entities included in data plane VCN 2218. Containers 2267(1)-(N) may also be isolated from resources of other customers.

[0291] In another example, a customer can use containers 2267(1)-(N) to invoke cloud service 2256. In this example, the customer can execute code in containers 2267(1)-(N) that requests a service from cloud service 2256. Containers 2267(1)-(N) can send the request to secondary VNICs 2272(1)-(N), which can send the request to a NAT gateway that can send the request to public Internet 2254. Public Internet 2254 can send the request to LB subnet 2222 included in control plane VCN 2216 via Internet gateway 2234. In response to determining that the request is valid, LB subnet can send the request to app subnet 2226, which can send the request to cloud service 2256 via service gateway 2236.

[0292] It should be noted that the illustrated IaaS architectures 1900, 2000, 2100, and 2200 may include elements other than those illustrated, and the illustrated embodiments are only some examples of cloud infrastructure systems that may incorporate embodiments of the present disclosure. In other embodiments, the IaaS system may have more or fewer elements than those illustrated, may combine two or more elements, or may have a different configuration or arrangement of elements.

[0293] In certain embodiments, the IaaS system described herein may include a suite of application, middleware, and database services that are delivered to customers in a self-service, subscription-based, elastically scalable, reliable, highly available, and secure manner. One example of such an IaaS system is the Oracle® Cloud Infrastructure (OCI) offered by the applicant.

[0294] 23 illustrates an exemplary computer system 2300 upon which various embodiments may be implemented. The system 2300 may be used to implement any of the computer systems described above. As illustrated, the computer system 2300 includes a processing unit 2304 that communicates with a number of peripheral subsystems via a bus subsystem 2302. These peripheral subsystems may include a processing acceleration unit 2306, an I / O subsystem 2308, a storage subsystem 2318, and a communication subsystem 2324. The storage subsystem 2318 includes a tangible computer readable storage medium 2322 and a system memory 2310.

[0295] Bus subsystem 2302 provides a mechanism for allowing the various components and subsystems of computer system 2300 to communicate with each other as intended. Although bus subsystem 2302 is shown diagrammatically as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses. Bus subsystem 2302 may be any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. For example, such architectures may include a Peripheral Component Interconnect (PCI) bus, which may be implemented as an Industry Standard Architecture (ISA) bus, a MicroChannel Architecture (MCA), a Bus Extended ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a mezzanine bus manufactured in accordance with the IEEE P1386.1 standard, among others.

[0296] The processing unit 2304, which may be implemented as one or more integrated circuits (e.g., conventional microprocessors or microcontrollers), controls the operation of the computer system 2300. The processing unit 2304 may include one or more processors. These processors may include single-core or multi-core processors. In some embodiments, the processing unit 2304 may be implemented as one or more independent processing units 2332 and / or 2334 with single-core or multi-core processors included in each processing unit. In other embodiments, the processing unit 2304 may be implemented as a quad-core processing unit formed by integrating two dual-core processors into a single chip.

[0297] In various embodiments, the processing unit 2304 can execute various programs in response to program code and can maintain multiple programs or processes executing simultaneously. At any given time, some or all of the program code being executed may reside in the processor 2304 and / or the storage subsystem 2318. The processor 2304 can provide the various functionality discussed above through appropriate programming. The computer system 2300 may further include a processing acceleration unit 2306, which may include a digital signal processor (DSP), a special purpose processor, and / or the like.

[0298] The I / O subsystem 2308 can include user interface input devices and user interface output devices. The user interface input devices can include a pointing device such as a keyboard, a mouse or a trackball, a touchpad or touch screen integrated into a display, a scroll wheel, a click wheel, a dial, a button, a switch, a keypad, a voice input device with a voice command recognition system, a microphone, and other types of input devices. The user interface input devices can also include a motion sensing and / or gesture recognizer, such as a Microsoft Kinect® motion sensor. The Microsoft Kinect® motion sensor can control and interact with input devices, such as a Microsoft Xbox® 360 game controller, through a natural user interface (NUI) that utilizes gestures and voice commands. The user interface input devices can also include an eye gesture recognizer, such as a Google Glass® blink detector. The Google Glass® blink detector detects the user's eye activity (e.g., "blinking" when taking a picture and / or selecting a menu) and translates the eye activity into input that is entered into an input device (e.g., Google Glass®). Additionally, the user interface input devices may include a voice recognition detection device that allows a user to interact with a voice recognition system (eg, the Siri® navigator) via voice commands.

[0299] User interface input devices may also include, but are not limited to, three-dimensional (3D) mice, joysticks or pointing sticks, game pads, graphics tablets, audio / visual devices such as speakers, digital cameras, digital video cameras, portable media players, webcams, image scanners, fingerprint scanners, barcode readers, 3D scanners, 3D printers, laser range finders, and eye-tracking devices. User interface input devices may also include medical imaging input devices, such as, for example, computed tomography, magnetic resonance imaging, ultrasound emission tomography, or medical ultrasound. User interface input devices may also include audio input devices, such as, for example, MIDI keyboards and electronic musical instruments.

[0300] User interface output devices may include non-visual displays such as a display subsystem, indicator lights, or audio output devices. The display subsystem may be, for example, a flat panel device using a cathode ray tube (CRT), liquid crystal display (LCD) or plasma display, a projection device, or a touch screen. In general, when the term "output device" is used, it is intended to include all possible types of devices and mechanisms for outputting information from computer system 2300 to a user or to another computer. For example, user interface output devices include, but are not limited to, various display devices that visually convey text, images, and audio / video information, such as monitors, printers, speakers, headphones, car navigation systems, plotters, voice output devices, and modems.

[0301] Computer system 2300 may include a storage subsystem 2318. The storage subsystem 2318 comprises software elements, which are illustratively located in a system memory 2310. The system memory 2310 may store program instructions loadable and executable by the processing unit 2304, and data generated by the execution of these programs.

[0302] Depending on the configuration and type of computer system 2300, the system memory 2310 may be volatile memory (e.g., random access memory (RAM)) and / or non-volatile memory (e.g., read-only memory (ROM), flash memory). Typically, RAM contains data and / or program modules that are immediately accessible to the processing unit 2304 and / or that are currently being operated on and executed by the processing unit 2304. In some implementations, the system memory 2310 may include a number of different types of memory, such as static random access memory (SRAM) or dynamic random access memory (DRAM). In some implementations, a basic input / output system (BIOS), containing the basic routines that help to transfer information between elements within the computer system 2300, such as during start-up, may typically be stored in ROM. By way of example and not limitation, system memory 2310 also illustrates application programs 2312, program data 2314, and an operating system 2316, which may include client applications, a web browser, a mid-tier application, a relational database management system (RDBMS), and the like.As an example, operating system 2316 may include various versions of Microsoft Windows®, Apple Macintosh®, and / or Linux® operating systems, various commercially available UNIX® or UNIX-like operating systems (including, but not limited to, various GNU / Linux operating systems, Google Chrome® OS, etc.), and / or mobile operating systems such as iOS, Windows® Phone, Android® OS, BlackBerry® 23 OS, and Palm® OS operating systems.

[0303] The storage subsystem 2318 may also provide a tangible computer readable storage medium for storing the basic programming and data constructs that provide the functionality of some embodiments. Software (programs, code modules, instructions) that, when executed by a processor, provide the above-described functionality may be stored in the storage subsystem 2318. These software modules or instructions may be executed by the processing unit 2304. The storage subsystem 2318 may also provide a repository for storing data used in accordance with the present disclosure.

[0304] The storage subsystem 2300 may also include a computer readable storage medium reader 2320 that may be further connected to a computer readable storage medium 2322. The computer readable storage medium 2322 may comprehensively represent remote, local, fixed and / or removable storage devices, in addition to storage media for temporarily and / or permanently containing, storing, transmitting and retrieving computer readable information together with, or in combination with, the system memory 2310 as appropriate.

[0305] Additionally, the computer readable storage medium 2322 containing the code or portions of code may include any suitable medium known or used in the art. Such media includes storage media and communication media, such as, but not limited to, volatile and non-volatile, removable and non-removable media implemented in any manner or technology for storing and / or transmitting information. This may include tangible computer readable storage media, such as RAM, ROM, electronically erasable programmable ROM (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage, or other tangible computer readable media. This may also include intangible computer readable media, such as data signals, data transmissions, or other media usable to transmit the desired information and accessible by computer system 2300.

[0306] By way of example, computer readable storage medium 2322 may include hard disk drives that read from or write to non-removable, non-volatile magnetic media, magnetic disk drives that read from or write to removable, non-volatile magnetic disks, and optical disk drives that read from or write to removable, non-volatile optical disks, such as CD ROMs, DVDs and Blu-ray disks or other optical media. Computer readable storage medium 2322 may include, but is not limited to, Zip drives, flash memory cards, universal serial bus (USB) flash drives, secure digital (SD) cards, DVD disks, digital video tapes, and the like. The computer readable storage media 2322 may also include flash memory based SSDs, enterprise flash drives, solid-state drives (SSDs) based on non-volatile memory such as solid state ROM, SSDs based on volatile memory such as solid state RAM, dynamic RAM, static RAM, DRAM based SSDs, magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs using a combination of DRAM and flash memory based SSDs. Disk drives and their associated computer readable media can provide non-volatile storage of computer readable instructions, data structures, program modules and other data for the computer system 2300.

[0307] The communications subsystem 2324 provides an interface with other computer systems and networks. The communications subsystem 2324 serves as an interface for receiving data from other systems and transmitting data from the computer system 2300 to other systems. For example, the communications subsystem 2324 may enable the computer system 2300 to connect to one or more devices via the Internet. In some embodiments, the communications subsystem 2324 may include radio frequency (RF) transceiver components for accessing wireless voice and / or data networks (e.g., using cellular technology, advanced data network technologies such as 3G, 4G, or EDGE (enhanced data rates for global evolution)), WiFi (IEEE 802.11 family of standards or other mobile communications technologies or any combination thereof), global positioning system (GPS) receiver components, and / or other components. In some embodiments, the communications subsystem 2324 may provide a wired network connection (e.g., Ethernet) in addition to or instead of a wireless interface.

[0308] Additionally, in some embodiments, the communications subsystem 2324 may receive incoming communications in the form of structured and / or unstructured data feeds 2326, event streams 2328, event updates 2330, etc. on behalf of one or more users that may be using the computer system 2300.

[0309] As an example, the communications subsystem 2324 may be configured to receive data feeds 2326, such as web feeds, such as Twitter® feeds, Facebook® updates, Rich Site Summary (RSS) feeds, in real time from users of social networks and / or other communications services, and / or receive real-time updates from one or more third party sources.

[0310] The communications subsystem 2324 may also be configured to receive data in the form of a continuous data stream, which may include an event stream 2328 of real-time events that may be continuous or may be essentially unbounded with no clear ends and / or event updates 2330. Examples of applications that generate continuous data may include, for example, sensor data applications, financial tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, and the like.

[0311] The communications subsystem 2324 may also be configured to output structured and / or unstructured data feeds 2326, event streams 2328, event updates 2330, etc. to one or more databases that may be in communication with one or more streaming data source computers coupled to the computer system 2300.

[0312] The computer system 2300 may be one of a variety of types, including a handheld mobile device (e.g., an iPhone® mobile phone, an iPad® computing tablet, a PDA), a wearable device (e.g., a Google Glass® head mounted display), a PC, a workstation, a mainframe, a kiosk, a server rack or other data processing system.

[0313] In the foregoing description, for purposes of explanation, specific details are set forth to allow a thorough understanding of the embodiments of the present disclosure. However, it will be apparent that various embodiments can be practiced without these specific details. The following description provides examples only and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the following description of the embodiments provides a person skilled in the art with a possible description for practicing the embodiments. It should be understood that various changes can be made in the function and arrangement of elements without departing from the spirit and scope of the present disclosure as defined in the appended claims. The drawings and descriptions are not intended to be limiting. Circuits, systems, networks, processes, and other components may be shown as components in block diagram form so as not to obscure the embodiments with unnecessary detail. In other embodiments, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail so as not to obscure the embodiments. The teachings of the present disclosure may also be applied to various types of applications, such as mobile applications, non-mobile applications, desktop applications, web applications, enterprise applications, and the like. Additionally, the teachings of the present disclosure are not limited to a particular operating environment (eg, operating system, device, platform, etc.) but may be applied to a number of different operating environments.

[0314] Also, note that each embodiment is described as a process that is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although the flowcharts describe operations as a sequential process, many operations may be performed in parallel or simultaneously. Moreover, the order of operations may be rearranged. A process terminates when the operation is completed, but may include additional steps not shown in the diagram. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination may correspond to a return of the calling function or the main function.

[0315] The words "example" and "exemplary" are used herein to mean "serving as an example, instance, or illustration." Any embodiment or design described herein as "exemplary" or "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments or designs.

[0316] The term "machine-readable storage medium" or "computer-readable storage medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, storing, or transporting instructions and / or data. Machine-readable storage media or computer-readable storage media may also include non-transitory media that can store data and do not include carrier waves and / or transitory electronic signals propagating over wireless or wired connections. Examples of non-transitory media may include, but are not limited to, magnetic disks or tapes, optical storage media such as compact disks (CDs) or digital versatile disks (DVDs), flash memory, memory or memory devices. A computer program product may include code and / or machine-executable instructions, which may represent any combination of procedures, functions, subprograms, programs, routines, subroutines, modules, software packages, classes, instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by transferring and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be conveyed, forwarded, or transmitted via any suitable means, such as memory sharing, message passing, token passing, network transmission, etc.

[0317] Further, the embodiments may be implemented by hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segments (e.g., computer program product) performing the necessary operations may be stored in a machine-readable medium. A processor may perform the necessary operations. The system illustrated in some figures may be provided in various configurations. In some examples, the system may be configured as a distributed system in which one or more elements of the system are distributed across one or more networks in a cloud computing system. When an element is described as being "configured" to perform a particular operation, such configuration may be achieved, for example, by designing electronic circuitry or other hardware to perform the operation, by programming or controlling electronic circuitry (e.g., a microprocessor or other suitable electronic circuitry) to perform the operation, or any combination thereof.

[0318] Although specific embodiments of the present disclosure have been described, various changes, modifications, alternative configurations, and equivalents are also encompassed within the scope of the present disclosure. The embodiments of the present disclosure are not limited to operating in a specific data processing environment, but can freely operate in multiple data processing environments. Furthermore, while the embodiments of the present disclosure have been described using a specific sequence of actions and steps, it will be apparent to those skilled in the art that the scope of the present disclosure is not limited to the sequence of actions and steps described. Various features and aspects of the above-described embodiments can be used individually or jointly.

[0319] Furthermore, while embodiments of the present disclosure have been described using a particular combination of hardware and software, it should be appreciated that other combinations of hardware and software are within the scope of the present disclosure. Embodiments of the present disclosure may be implemented using only hardware, only software, or a combination thereof. Various processes described in this disclosure may be executed on the same processor or any combination of different processors. Thus, when a component or module is described as being configured to perform a particular process, the configuration may be achieved, for example, by designing an electronic circuit to perform the process, by programming a programmable electronic circuit (such as a microprocessor) to perform the process, or a combination thereof. Processes may communicate using a variety of techniques, including but not limited to conventional techniques for communicating between processes. Different pairs of processes may use different techniques, or the same pair of processes may use different techniques at different times.

[0320] Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense. It will be apparent, however, that additions, subtractions, deletions and other modifications and changes may be made without departing from the broad spirit and scope defined by the appended claims. Thus, although specific embodiments of the disclosure have been described, these embodiments are not intended to be limiting. Various modifications and equivalents thereof are intended to be encompassed within the scope of the appended claims.

[0321] The indefinite articles "a" / "an", the definite article "the" and similar references used in the context of describing this disclosure (particularly in the context of the claims) should be construed to include both the singular and the plural unless otherwise stated herein or the context clearly indicates otherwise. The terms "comprising", "having", "including", and "containing" should be construed as open-ended terms (i.e., meaning "including but not limited to") unless otherwise stated. The term "connected" should be construed as being partly or entirely contained within, attached to, or joined together, even if there is something intervening. The recitation of ranges of values ​​herein is intended merely as a shorthand method of referring to each individual value contained within the range, and each individual value is incorporated herein as if it were individually set forth herein, unless otherwise stated herein. All methods described herein can be performed in any suitable order unless otherwise stated herein or the context clearly indicates otherwise. The use of any and all examples or exemplary language (e.g., "such as") herein is intended to better clarify the embodiments of the disclosure and does not limit the scope of the disclosure, unless otherwise specified. No language in the specification should be construed as indicating any non-claimed element essential to the practice of the disclosure.

[0322] Disjunctive language, such as the phrase "at least one of X, Y, or Z," is intended to be understood in context as being generally used to indicate that an item, term, etc. may be either X, Y, or Z, or any combination thereof (e.g., X, Y, and / or Z), unless otherwise noted. Thus, such disjunctive language is not generally intended to, and does not imply, that a particular embodiment requires that at least one of X, at least one of Y, or at least one of Z be present.

[0323] Example embodiments of the present disclosure can be described in terms of the following clauses: Item 1. A method, comprising: generating, based on customer input, information indicating one or more criteria associated with copying a frame to be transmitted in a Layer 2 virtual network and a destination to receive the frame copy, the Layer 2 virtual network being hosted on a physical network and comprising a plurality of Layer 2 compute instances, a plurality of Layer 2 virtual network interfaces, and a plurality of Layer 2 virtual switches, the method further comprising determining that the information should be sent to a network virtualization device (NVD) of the physical network, the NVD hosting a Layer 2 virtual network interface of the plurality of Layer 2 virtual network interfaces and a Layer 2 virtual switch of the plurality of Layer 2 virtual switches, the Layer 2 virtual network interface and the Layer 2 virtual switch being associated with a Layer 2 compute instance of the plurality of Layer 2 compute instances, the Layer 2 compute instance being hosted on a host machine of the physical network, the host machine and the NVD being communicatively connected, the method further comprising sending the information to the NVD.

[0324] Item 2. The method of item 1, wherein determining that the customer's input indicates that the one or more criteria should be applied to outgoing traffic from the certain Layer 2 compute instance and that the information should be sent to the NVD includes determining that the NVD hosts the certain Layer 2 virtual switch associated with the certain Layer 2 compute instance.

[0325] Clause 3. The method of any one of clauses 1 to 2, wherein determining that the customer's input indicates that the one or more criteria should be applied to incoming traffic to the certain layer 2 compute instance and that the information should be sent to the NVD includes determining that the NVD hosts the certain layer 2 virtual switch associated with the certain layer 2 compute instance.

[0326] Clause 4. The method according to any one of clauses 1 to 3, wherein the information, the NVD, the Layer 2 virtual network interface, and the Layer 2 virtual switch are first information, a first NVD, a first Layer 2 virtual network interface, and a first Layer 2 virtual switch, respectively; the method further includes generating second information based on the input of the customer, the second information including one or more criteria that are the same or different and identify the same or different destinations; the method further includes determining that the second information should be sent to a second NVD of the physical network, the second NVD hosts a second Layer 2 virtual network interface of the plurality of Layer 2 virtual network interfaces and a second Layer 2 virtual switch of the plurality of Layer 2 virtual switches, the second Layer 2 virtual network interface and the second Layer 2 virtual switch are associated with a second Layer 2 compute instance of the plurality of Layer 2 compute instances; and the method further includes sending the second information to the second NVD.

[0327] Item 5. The method of any one of items 1 to 4, wherein the method further includes determining a customer-specified configuration of the Layer 2 virtual network and determining a mapping between the customer-specified configuration and a topology of the Layer 2 virtual network, the information being generated further based on the mapping.

[0328] Clause 6. The method of clause 5, wherein the customer's input indicates a media access control (MAC) address of the layer 2 computing instance, the method further including: determining, based on the mapping, that the layer 2 virtual network interface is associated with the MAC address; and including an identifier of the layer 2 virtual network interface in the information.

[0329] Clause 7. The method of clause 6, wherein determining that the information should be sent to the NVD includes determining that the NVD hosts the certain layer 2 virtual network interface.

[0330] Clause 8. The method of clause 5, wherein the customer's input indicates a port of the one of the layer 2 compute instances, the method further including: determining, based on the mapping, that the one of the layer 2 virtual network interfaces emulates the port; and including an identifier of the one of the layer 2 virtual network interfaces in the information.

[0331] Clause 9. The method of clause 5, wherein the input of the customer indicates a first filtering criterion and a processing criterion, the first filtering criterion is updated based on the mapping to generate a second filtering criterion, and the information includes the second filtering criterion and the processing criterion.

[0332] Clause 10. The method of clause 9, wherein the first filtering criteria includes a source Media Access Control (MAC) address, a destination MAC address, a source Internet Protocol (IP) address, a destination IP address, a protocol identifier, a source port, a destination port, a traffic source, or a traffic destination.

[0333] Item 11. The method of item 9, wherein the processing criteria include the number of copies to be generated or encapsulation to be added to the copies to be generated.

[0334] Item 12. A network virtualization device, comprising: one or more processors; and one or more computer-readable storage media storing instructions, which, when executed by the one or more processors, configure the network virtualization device to host a layer 2 virtual network interface and a layer 2 virtual switch belonging to a layer 2 virtual network of a customer, the layer 2 virtual network interface and the layer 2 virtual switch being associated with a layer 2 compute instance belonging to the layer 2 virtual network, the layer 2 compute instance being hosted on a host machine of a physical network comprising the network virtualization device, the host machine and the network virtualization device being communicatively connected, and the layer 2 virtual network being configured to host a layer 2 virtual network interface and a layer 2 virtual switch belonging to the layer 2 virtual network, the layer 2 compute instance being hosted on a host machine of a physical network comprising the network virtualization device, the host machine and the network virtualization device being communicatively connected, and the layer 2 virtual network being configured to host a layer 2 virtual network interface and a layer 2 virtual switch belonging to the layer 2 virtual network, the layer 2 virtual network interface and the layer 2 virtual switch being associated with a layer 2 compute instance ... virtual network interface and the layer 2 virtual switch being associated with a layer 2 virtual network instance belonging to the layer 2 virtual network, the layer 2 virtual network instance being hosted on a host machine of a physical network comprising the network virtualization device, the host machine and the network virtualization device being communicatively connected, and the layer 2 virtual network being configured to host a layer 2 virtual network interface and a layer 2 virtual switch belonging to the layer 2 virtual network, the layer 2 virtual network interface and the layer and a network virtualization device configured to: receive frames destined for or originating from the Layer 2 compute instance; determine, based on header information of the frame, that the one or more criteria are met; generate a copy of the frame; and transmit the copy to the destination.

[0335] Clause 13. The network virtualization device of clause 12, wherein the instructions, when executed by one or more processors, further configure the network virtualization device to receive the information from a control plane of the physical network, the information being received based on the customer input indicating that the one or more criteria should be applied to incoming traffic to the Layer 2 compute instance or outgoing traffic from the Layer 2 compute instance and based on the network virtualization device hosting the Layer 2 virtual switch associated with the Layer 2 compute instance.

[0336] Clause 14. The network virtualization device of clause 13, wherein the customer input indicates that an incoming frame received on a first port of the Layer 2 compute instance should be copied and transmitted on a second port, and the instructions, when executed by one or more processors, further configure the network virtualization device to determine a first destination media access control (MAC) address of the frame, the first destination MAC address being included in the information and associated with the first port based on a topology of the Layer 2 virtual network, and the instructions, when executed by one or more processors, further configure the network virtualization device to include in the copy a second destination MAC address, the second destination MAC address being included in the information and associated with the second port based on the topology.

[0337] Clause 15. The network virtualization device of clause 13 or 14, wherein the customer input indicates that an outgoing frame received on a first port of the Layer 2 compute instance should be copied and transmitted on a second port, and the instructions, when executed by one or more processors, further configure the network virtualization device to determine a source media access control (MAC) address of the frame, the source MAC address being included in the information and associated with the first port based on a topology of the Layer 2 virtual network, and the instructions, when executed by one or more processors, further configure the network virtualization device to include in the copy the destination MAC address, the destination MAC address being included in the information and associated with the second port based on the topology.

[0338] Clause 16. The network virtualization device of any one of clauses 13 to 15, wherein the customer input indicates that a frame received on a first port of the Layer 2 compute instance should be copied and transmitted on a second port using an encapsulation protocol, and the instructions, when executed by one or more processors, further configure the network virtualization device to determine a medium access control (MAC) address of the frame, the MAC address being a source MAC address or a first destination MAC address, which is included in the information and associated with the first port based on a topology of the Layer 2 virtual network, and the instructions, when executed by one or more processors, further configure the network virtualization device to include in the copy a second destination MAC address, which is included in the information and associated with the second port based on the topology, and the instructions, when executed by one or more processors, further configure the network virtualization device to encapsulate the copy based on the encapsulation protocol.

[0339] Clause 17. A system, comprising: one or more processors; and one or more computer-readable storage media storing instructions, the instructions, when executed by the one or more processors, further configure the system to generate, based on customer input, one or more criteria associated with copying a frame to be transmitted in a layer 2 virtual network and information indicating a destination to receive the frame copy, the layer 2 virtual network being hosted on a physical network and comprising a plurality of layer 2 compute instances, a plurality of layer 2 virtual network interfaces, and a plurality of layer 2 virtual switches, the instructions, when executed by the one or more processors, further configure the system to generate information indicating one or more criteria associated with copying a frame to be transmitted in a layer 2 virtual network and ... D), the NVD hosts a Layer 2 virtual network interface of the plurality of Layer 2 virtual network interfaces and a Layer 2 virtual switch of the plurality of Layer 2 virtual switches, the Layer 2 virtual network interface and the Layer 2 virtual switch are associated with a Layer 2 compute instance of the plurality of Layer 2 compute instances, the Layer 2 compute instance is hosted on a host machine of the physical network, the host machine and the NVD are communicatively coupled, and the instructions, when executed by the one or more processors, further configure the system to send the information to the NVD.

[0340] Clause 18. The system described in clause 17, further comprising the NVD, the NVD configured to store the information, configured to receive frames addressed to or originating from the layer 2 computational instance, configured to determine that the one or more criteria are met based on header information of the frame, configured to generate a copy of the frame, and configured to send the copy to the destination.

[0341] Clause 19. The system of any of clauses 17 or 18, wherein the instructions, when executed by one or more processors, further configure the system to determine a customer-specified configuration of the Layer 2 virtual network and to determine a mapping between the customer-specified configuration and a topology of the Layer 2 virtual network, the information being further generated based on the mapping.

[0342] Clause 20. The system described in clause 19, wherein the input of the customer indicates a first filtering criterion and a processing criterion, the first filtering criterion is updated based on the mapping to generate a second filtering criterion, and the information includes the second filtering criterion and the processing criterion.

[0343] Preferred embodiments of the present disclosure are described herein, including the best mode known for carrying out the present disclosure. Variations of these preferred embodiments will become apparent to those skilled in the art upon reading the foregoing description. Those skilled in the art can adopt such variations as appropriate, and the present disclosure may be carried out in ways other than as specifically described herein. Accordingly, this disclosure includes all modifications and equivalents of the subject matter recited in the claims appended hereto as permitted by applicable law. Moreover, any combination of the above-described elements in all possible variations thereof is encompassed by this disclosure unless otherwise indicated herein.

[0344] All references cited in this specification, including publications, patent applications, and patents, are incorporated by reference to the same extent as if each individual reference was individually and specifically indicated to be incorporated by reference and was set forth in its entirety herein.

[0345] In the foregoing specification, aspects of the disclosure have been described with reference to specific embodiments thereof, but those skilled in the art will recognize that the disclosure is not limited thereto. Various features and aspects of the above-mentioned disclosure may be used individually or jointly. Moreover, the embodiments may be utilized in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of the specification. Thus, the specification and drawings should be regarded as illustrative rather than restrictive.< / realm>

Claims

1. 1. A method comprising: generating, based on customer input, information indicative of one or more criteria associated with copying frames transmitted in a Layer 2 virtual network and a destination to receive the frame copies, the Layer 2 virtual network being hosted on a physical network and comprising a plurality of Layer 2 compute instances, a plurality of Layer 2 virtual network interfaces, and a plurality of Layer 2 virtual switches, the method further comprising: determining that the information should be sent to a network virtualization device (NVD) of the physical network; the NVD hosts a Layer 2 virtual network interface of the plurality of Layer 2 virtual network interfaces and a Layer 2 virtual switch of the plurality of Layer 2 virtual switches; the one layer 2 virtual network interface and the one layer 2 virtual switch are associated with a one layer 2 compute instance of the plurality of layer 2 compute instances; The certain layer 2 computing instance is hosted on a host machine of the physical network, the host machine and the NVD are communicatively coupled, and the method further comprises: transmitting the information to the NVD.

2. 2. The method of claim 1, wherein determining that the customer's input indicates that the one or more criteria should be applied to outgoing traffic from the one Layer 2 compute instance and that the information should be sent to the NVD includes determining that the NVD hosts the one Layer 2 virtual switch associated with the one Layer 2 compute instance.

3. 3. The method of claim 1 or 2, wherein determining that the customer input indicates that the one or more criteria should be applied to incoming traffic to the one Layer 2 compute instance and that the information should be sent to the NVD includes determining that the NVD hosts the one Layer 2 virtual switch associated with the one Layer 2 compute instance.

4. The information, the NVD, the certain Layer 2 virtual network interface, and the certain Layer 2 virtual switch are first information, a first NVD, a first Layer 2 virtual network interface, and a first Layer 2 virtual switch, respectively, and the method further includes: generating second information based on the customer's input, the second information including the same or different one or more criteria and identifying the same or different destination, the method further comprising: determining that the second information should be sent to a second NVD of the physical network, the second NVD hosting a second Layer 2 virtual network interface of the plurality of Layer 2 virtual network interfaces and a second Layer 2 virtual switch of the plurality of Layer 2 virtual switches, the second Layer 2 virtual network interface and the second Layer 2 virtual switch associated with a second Layer 2 compute instance of the plurality of Layer 2 compute instances, the method further comprising: The method of any one of claims 1 to 3, further comprising transmitting the second information to the second NVD.

5. The method further comprises: determining a customer-specified configuration of the Layer 2 virtual network; 5. The method of claim 1, further comprising determining a mapping between the customer-specified configuration and a topology of the Layer 2 virtual network, the information being generated further based on the mapping.

6. The customer input indicates a Media Access Control (MAC) address of the one Layer 2 computing instance, and the method further comprises: determining, based on the mapping, that the one Layer 2 virtual network interface is associated with the MAC address; and including an identifier of the one Layer 2 virtual network interface in the information.

7. The method of claim 6 , wherein determining that the information should be sent to the NVD comprises determining that the NVD hosts the one Layer 2 virtual network interface.

8. The customer input indicates a port of the one layer 2 compute instance, and the method further comprises: determining, based on the mapping, that the one Layer 2 virtual network interface emulates the port; and including an identifier of the one Layer 2 virtual network interface in the information.

9. 6. The method of claim 5, wherein the input of the customer indicates a first filtering criterion and a processing criterion, the first filtering criterion is updated based on the mapping to generate a second filtering criterion, and the information includes the second filtering criterion and the processing criterion.

10. 10. The method of claim 9, wherein the first filtering criteria includes a source media access control (MAC) address, a destination MAC address, a source Internet Protocol (IP) address, a destination IP address, a protocol identifier, a source port, a destination port, a traffic source, or a traffic destination.

11. The method of claim 9 , wherein the processing criteria includes a number of copies to be generated or encapsulation to add to the copies to be generated.

12. 1. A network virtualization device, comprising: one or more processors; and one or more computer-readable storage media storing instructions that, when executed by the one or more processors, cause the network virtualization device to: configured to host layer 2 virtual network interfaces and layer 2 virtual switches belonging to a customer's layer 2 virtual network; the layer 2 virtual network interface and the layer 2 virtual switch are associated with a layer 2 compute instance that belongs to the layer 2 virtual network; the layer 2 computing instance is hosted on a host machine of a physical network that includes the network virtualization device, the host machine and the network virtualization device being communicatively coupled; the layer 2 virtual network is hosted on the physical network and comprises a plurality of layer 2 compute instances, a plurality of layer 2 virtual network interfaces, and a plurality of layer 2 virtual switches, the instructions, when executed by the one or more processors, further causing the network virtualization device to: storing information indicative of one or more criteria associated with copying the frame and a destination to receive the frame copy; configured to receive frames addressed to or originating from the layer-2 computing instance; determining that the one or more criteria are satisfied based on header information of the frame; generating a copy of said frame; A network virtualization device configured to transmit the copy to the destination.

13. The instructions, when executed by the one or more processors, further cause the network virtualization device to:

13. The network virtualization device of claim 12, configured to receive the information from a control plane of the physical network, the information being received based on the customer input indicating that the one or more criteria should be applied to incoming traffic to the Layer 2 compute instance or outgoing traffic from the Layer 2 compute instance and based on the network virtualization device hosting the Layer 2 virtual switch associated with the Layer 2 compute instance.

14. The customer input indicates that an incoming frame received on a first port of the Layer 2 compute instance should be copied and transmitted on a second port, and the instructions, when executed by the one or more processors, further cause the network virtualization device to: and configuring the network virtualization device to determine a first destination media access control (MAC) address of the frame, the first destination MAC address being included in the information and associated with the first port based on a topology of the Layer 2 virtual network, the instructions when executed by the one or more processors further configuring the network virtualization device to:

14. The network virtualization device of claim 13, further configured to include in the copy a second destination MAC address, the second destination MAC address being included in the information and associated with the second port based on the topology.

15. The customer input indicates that outgoing frames received on a first port of the Layer 2 compute instance should be copied and transmitted on a second port, and the instructions, when executed by the one or more processors, further cause the network virtualization device to: and configuring the network virtualization device to determine a source media access control (MAC) address of the frame, the source MAC address being included in the information and associated with the first port based on a topology of the Layer 2 virtual network, the instructions when executed by the one or more processors further configuring the network virtualization device to:

15. The network virtualization device of claim 13 or 14, configured to include in the copy the destination MAC address, the destination MAC address being included in the information and associated with the second port based on the topology.

16. The customer input indicates that a frame received on a first port of the Layer 2 compute instance should be copied and transmitted on a second port using an encapsulation protocol, and the instructions, when executed by the one or more processors, further cause the network virtualization device to: and configuring the network virtualization device to determine a medium access control (MAC) address of the frame, the MAC address of the frame being a source MAC address or a first destination MAC address, included in the information and associated with the first port based on a topology of the Layer 2 virtual network, the instructions, when executed by the one or more processors, further comprising: and configuring the copy to include a second destination MAC address, the second destination MAC address being included in the information and associated with the second port based on the topology, and the instructions when executed by the one or more processors further cause the network virtualization device to: The network virtualization device according to any one of claims 13 to 15, configured to encapsulate the copy based on the encapsulation protocol.

17. 1. A system comprising: one or more processors; and one or more computer-readable storage media storing instructions that, when executed by the one or more processors, cause the system to: and generating, based on customer input, one or more criteria associated with copying a frame transmitted in a Layer 2 virtual network and information indicative of a destination to receive the frame copies, the Layer 2 virtual network being hosted on a physical network and comprising a plurality of Layer 2 compute instances, a plurality of Layer 2 virtual network interfaces, and a plurality of Layer 2 virtual switches, the instructions, when executed by the one or more processors, further configuring the system to: determining that the information should be sent to a network virtualization device (NVD) of the physical network; the NVD hosts a Layer 2 virtual network interface of the plurality of Layer 2 virtual network interfaces and a Layer 2 virtual switch of the plurality of Layer 2 virtual switches; the one layer 2 virtual network interface and the one layer 2 virtual switch are associated with a one layer 2 compute instance of the plurality of layer 2 compute instances; The certain layer 2 computing instance is hosted on a host machine of the physical network, the host machine and the NVD are communicatively coupled, and the instructions, when executed by the one or more processors, further configure the system to: The system is configured to transmit the information to the NVD.

18. The NVD further comprises: configured to store said information; configured to receive frames addressed to or originating from the layer 2 computing instance; configured to determine, based on header information of the frame, that the one or more criteria are met; configured to generate a copy of the frame; The system of claim 17 configured to transmit the copy to the destination.

19. The instructions, when executed by the one or more processors, further cause the system to: configured to determine a customer-specified configuration of the Layer 2 virtual network; 19. The system of claim 17 or 18, configured to determine a mapping between the customer specified configuration and a topology of the Layer 2 virtual network, the information being generated further based on the mapping.

20. 20. The system of claim 19, wherein the customer input indicates a first filtering criterion and a processing criterion, the first filtering criterion is updated based on the mapping to generate a second filtering criterion, and the information includes the second filtering criterion and the processing criterion.