Layer 2 networking information in virtualized cloud environments

Abstract

Techniques for communicating in a customer's L2 virtual network are described. In an example, the L2 virtual network includes a plurality of L2 compute instances hosted on a set of host machines and a plurality of L2 virtual network interfaces and L2 virtual switches hosted on a set of network virtualization devices. The L2 virtual network interfaces emulate L2 ports of the L2 virtual network. Information associated with the L2 virtual switches is collected and provided to the customer.

Description

[0001] Cross-references to related applications

[0002] This international patent application claims priority to U.S. Patent Application No. 17 / 494,722, filed October 5, 2021, entitled “LAYER-2 NETWORKING INFORMATION IN A VIRTUALIZED CLOUD ENVIRONMENT,” which claims the benefit of U.S. Provisional Patent Application No. 63 / 132,377, filed December 30, 2020, entitled “LAYER-2 NETWORKING IN AVIRTUALIZED CLOUD ENVIRONMENT,” the contents of which are incorporated herein by reference in their entirety for all purposes. Background Technology

[0003] Cloud computing provides on-demand availability of computing resources. It can be based on data centers accessible to users via the internet. Cloud computing can provide Infrastructure as a Service (IaaS). Virtual networks can be created for user use. However, these virtual networks have limitations that restrict their functionality and value. Therefore, further improvements are expected. Summary of the Invention

[0004] This disclosure relates to virtualized cloud environments. Techniques for providing Layer 2 networking functionality in virtualized cloud environments are described. Layer 2 functionality is provided in conjunction with and complements Layer 3 networking functionality provided by the virtualized cloud environment.

[0005] Some embodiments of this disclosure relate to providing a Layer 2 Virtual Local Area Network (VLAN) to a customer in a private network, such as a customer's Virtual Cloud Network (VCN). Different compute instances are connected within Layer 2 VLANs. The customer perceives it as a single, emulated switch connecting the compute instances. In reality, this emulated switch is implemented as an infinitely scalable, distributed switch comprising a collection of local switches. More specifically, each compute instance runs on a host machine connected to a Network Virtualization Device (NVD). For each compute instance on a host connected to the NVD, the NVD hosts a Layer 2 Virtual Network Interface Card (VNIC) and a local switch associated with the compute instance. The Layer 2 VNIC represents a port of the compute instance on the Layer 2 VLAN. The local switch connects the VNIC to other VNICs (e.g., other ports) associated with other compute instances in the Layer 2 VLAN. Various Layer 2 network services are supported, including, for example, providing information about the Layer 2 VLANs, where the information may include a Media Access Control (MAC) address forwarding table and / or Layer 2 switch statistics.

[0006] This document describes various embodiments, including methods, systems, and non-transitory computer-readable storage media that store programs, code, or instructions executable by one or more processors. Attached Figure Description

[0007] Figure 1 This is a high-level diagram of a distributed environment, illustrating a virtual or overlay cloud network hosted by a cloud service provider infrastructure according to certain embodiments.

[0008] Figure 2 A simplified architecture diagram of the physical components in the physical network within the CSPI according to certain embodiments is depicted.

[0009] Figure 3 An example arrangement within CSPI according to certain embodiments is shown, in which a host machine is connected to multiple network virtualization devices (NVDs).

[0010] Figure 4 The connectivity between the host machine and the NVD, according to certain embodiments, is described for providing I / O virtualization to support multiple tenancy.

[0011] Figure 5 A simplified block diagram of a physical network provided by CSPI according to certain embodiments is depicted.

[0012] Figure 6 This is a schematic diagram of a computing network according to certain embodiments.

[0013] Figure 7 This is a logical and hardware diagram of VLANs according to certain embodiments.

[0014] Figure 8 This is a logical diagram of multiple connected L2 VLANs according to certain embodiments.

[0015] Figure 9 This is a logical diagram of multiple connected L2 VLANs and subnet 900 according to certain embodiments.

[0016] Figure 10 This is a schematic diagram of intra-VLAN communication and learning within a VLAN according to certain embodiments.

[0017] Figure 11 This is a schematic diagram of a VLAN according to certain embodiments.

[0018] Figure 12 This is a flowchart illustrating a process 1200 for intra-VLAN communication according to certain embodiments.

[0019] Figure 13The illustration shows an example environment suitable for defining the configuration of an L2 virtual network and providing related L2 information according to certain embodiments.

[0020] Figure 14 The illustration shows example L2 information in a layer 2 virtual network according to certain embodiments.

[0021] Figure 15 The illustration shows an example L2 forwarding table in a layer 2 virtual network according to certain embodiments.

[0022] Figure 16 The illustration shows example L2 metrics and statistics in a layer 2 virtual network according to certain embodiments.

[0023] Figure 17 This is a flowchart illustrating a process for providing L2 information according to certain embodiments.

[0024] Figure 18 This is a flowchart illustrating a process for generating an L2 forwarding table according to certain embodiments.

[0025] Figure 19 This is a flowchart illustrating a process for generating L2 statistics according to certain embodiments.

[0026] Figure 20 This is a block diagram illustrating a pattern for implementing cloud infrastructure as a service system according to at least one embodiment.

[0027] Figure 21 This is a block diagram illustrating another pattern for implementing cloud infrastructure as a service system according to at least one embodiment.

[0028] Figure 22 This is a block diagram illustrating another pattern for implementing cloud infrastructure as a service system according to at least one embodiment.

[0029] Figure 23 This is a block diagram illustrating another pattern for implementing cloud infrastructure as a service system according to at least one embodiment.

[0030] Figure 24 This is a block diagram illustrating an example computer system according to at least one embodiment. Detailed Implementation

[0031] In the following description, specific details are set forth for purposes of explanation in order to provide a thorough understanding of certain embodiments. However, it will be apparent that various embodiments may be practiced without these specific details. The accompanying drawings and description are not intended to be limiting. The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment or design described herein as "exemplary" is not necessarily to be construed as preferred or superior to other embodiments or designs.

[0032] A- Example Virtual Networking Architecture

[0033] The term cloud service generally refers to services provided by a cloud service provider (CSP) to users or customers on demand (e.g., via a subscription model) using systems and infrastructure (cloud infrastructure) provided by the CSP. Typically, the servers and systems that make up the CSP's infrastructure are separate from the customer's own on-premises servers and systems. Therefore, customers can utilize cloud services provided by the CSP without having to purchase separate hardware and software resources for the service. Cloud services are designed to provide subscribers with simple, scalable access to applications and computing resources without requiring customers to invest in the infrastructure used to provide the service.

[0034] Several cloud service providers offer various types of cloud services. There are various types or models of cloud services, including Software as a Service (SaaS), Platform as a Service (PaaS), Infrastructure as a Service (IaaS), etc.

[0035] Customers can subscribe to one or more cloud services provided by a CSP. Customers can be any entity, such as individuals, organizations, or businesses. When a customer subscribes to or registers for a service provided by a CSP, a lease or account is created for that customer. The customer can then access one or more cloud resources associated with that account through their subscription.

[0036] As mentioned above, Infrastructure as a Service (IaaS) is a specific type of cloud computing service. In the IaaS model, a CSP provides infrastructure (called Cloud Service Provider Infrastructure or CSPI) that customers can use to build their own customizable networks and deploy customer resources. Therefore, the customer's resources and network are hosted in a distributed environment by the infrastructure provided by the CSP. This differs from traditional computing, where the customer's resources and network are hosted by the infrastructure provided by the customer.

[0037] CSPI can include interconnected high-performance computing resources forming a physical network, including various host machines, memory resources, and network resources. This physical network is also known as a substrate network or underlying network. Resources in CSPI can be distributed across one or more data centers, which may be geographically dispersed across one or more geographic regions. Virtualization software can be executed by these physical resources to provide a virtualized distributed environment. Virtualization creates an overlay network (also known as a software-based network, software-defined network, or virtual network) on the physical network. The CSPI physical network provides the underlying foundation for creating one or more overlay or virtual networks on top of the physical network. Virtual or overlay networks can include one or more Virtual Cloud Networks (VCNs). A virtual network is a layer of network abstraction that can run on top of a physical network, implemented using software virtualization technologies (e.g., hypervisors, functions performed by network virtualization devices (NVDs) (e.g., smartNICs), top-of-rack (TOR) switches, smart TORs that implement one or more functions performed by NVDs, and other mechanisms). Virtual networks can take many forms, including peer-to-peer networks, IP networks, etc. Virtual networks are typically either Layer 3 IP networks or Layer 2 VLANs. This method of virtual or overlay networking is often referred to as virtual or overlay layer 3 networking. Examples of protocols developed for virtual networks include IP-in-IP (or Generic Routing Encapsulation (GRE)), Virtual Scalable Local Area Network (VXLAN—IETF RFC 7348), Virtual Private Network (VPN) (e.g., MPLS layer 3 Virtual Private Network (RFC 4364)), VMware's NSX, GENEVE (Generic Network Virtualization Encapsulation), etc.

[0038] For IaaS, the infrastructure provided by a CSP (Center for Service Providers) can be configured to offer virtualized computing resources over a public network (e.g., the Internet). In the IaaS model, cloud service providers can host infrastructure components (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer), etc.). In some cases, IaaS providers can also provision various services to accompany those infrastructure components (e.g., billing, monitoring, logging, security, load balancing, and clustering, etc.). Therefore, since these services can be policy-driven, IaaS users can implement policies to drive load balancing to maintain application availability and performance. CSPI provides a collection of infrastructure and complementary cloud services that enable customers to build and run a wide range of applications and services in a highly available, hosted, distributed environment. CSPI provides high-performance computing resources and capabilities, as well as storage capacity, in a flexible virtual network securely accessible from various networked locations, such as the customer's on-premises network. When a customer subscribes to or enrolls in an IaaS service provided by a CSP, the lease created for that customer is a secure and isolated partition within the CSP, where the customer can create, organize, and manage their cloud resources.

[0039] Customers can build their own virtual networks using the compute, storage, and networking resources provided by CSPI. One or more customer resources or workloads, such as compute instances, can be deployed on these virtual networks. For example, a customer can use resources provided by CSPI to build one or more customizable and private virtual networks, called Virtual Cloud Networks (VCNs). A customer can deploy one or more customer resources, such as compute instances, on a customer VCN. Compute instances can take the form of virtual machines, bare metal instances, etc. Therefore, CSPI provides a collection of infrastructure and complementary cloud services that enable customers to build and run a wide range of applications and services in a highly available, virtually managed environment. Customers do not manage or control the underlying physical resources provided by CSPI, but they can control the operating system, storage devices, and deployed applications; and may have limited control over selected networking components (e.g., firewalls).

[0040] The CSP can provide a console that enables customers and network administrators to configure, access, and manage resources deployed in the cloud using CSPI resources. In some embodiments, the console provides a web-based user interface that can be used to access and manage the CSPI. In other embodiments, the console is a web-based application provided by the CSP.

[0041] CSPI can support both single-tenant and multi-tenant architectures. In a single-tenant architecture, software (e.g., applications, databases) or hardware components (e.g., host machines or servers) serve a single customer or tenant. In a multi-tenant architecture, the software or hardware components serve multiple customers or tenants. Therefore, in a multi-tenant architecture, CSPI resources are shared among multiple customers or tenants. In a multi-tenant scenario, precautions and safeguards are implemented in CSPI to ensure that each tenant's data is isolated and invisible to other tenants.

[0042] In a physical network, a network endpoint (“endpoint”) is a computing device or system that connects to and communicates with the physical network. Network endpoints in a physical network can connect to a Local Area Network (LAN), a Wide Area Network (WAN), or other types of physical networks. Examples of traditional endpoints in a physical network include modems, hubs, bridges, switches, routers and other network devices, physical computers (or host machines), etc. Each physical device in a physical network has a fixed network address that can be used to communicate with the device. This fixed network address can be a Layer 2 address (e.g., a MAC address), a fixed Layer 3 address (e.g., an IP address), etc. In a virtualized environment or virtual network, endpoints can include various virtual endpoints, such as virtual machines hosted by components of the physical network (e.g., hosted by physical host machines). These endpoints in a virtual network are addressed by overlay addresses, such as overlay Layer 2 addresses (e.g., overlay MAC addresses) and overlay Layer 3 addresses (e.g., overlay IP addresses). Network overlays provide flexibility by allowing network administrators to move around the overlay addresses associated with network endpoints using software management (e.g., via software implementing a control plane for virtual networks). Therefore, unlike physical networks, in virtual networks, overlay addresses (e.g., overlay IP addresses) can be moved from one endpoint to another using network management software. Since virtual networks are built on top of physical networks, communication between components within a virtual network involves both the virtual network and the underlying physical network. To facilitate this communication, CSPI components are configured to learn and store mappings that map overlay addresses in the virtual network to actual physical addresses in the baseboard network, and vice versa. These mappings are then used to facilitate communication. Client traffic is encapsulated to facilitate routing within the virtual network.

[0043] Therefore, physical addresses (e.g., physical IP addresses) are associated with components in a physical network, and overlay addresses (e.g., overlay IP addresses) are associated with entities in a virtual network. Both physical IP addresses and overlay IP addresses are types of real IP addresses. These are separate from virtual IP addresses, which map to multiple real IP addresses. Virtual IP addresses provide a one-to-many mapping between virtual IP addresses and multiple real IP addresses.

[0044] Cloud infrastructure, or CSPI, is physically hosted in one or more data centers in one or more regions of the world. CSPI can include components in a physical or substrate network and virtualized components (e.g., virtual networks, compute instances, virtual machines, etc.) in a virtual network built on top of the physical network components. In some embodiments, CSPI is organized and hosted in domains, regions, and availability domains. A region is typically a localized geographical area containing one or more data centers. Regions are generally independent of each other and can be geographically distant, for example, spanning countries or even continents. For example, one region might be in Australia, another in Japan, another in India, and so on. CSPI resources are partitioned between regions so that each region has its own independent subset of CSPI resources. Each region can provide a set of core infrastructure services and resources, such as compute resources (e.g., bare metal servers, virtual machines, containers, and related infrastructure); storage resources (e.g., block volume storage devices, file storage devices, object storage devices, archive storage devices); network resources (e.g., virtual cloud networks (VCNs), load balancing resources, connectivity to on-premises networks), database resources; edge networking resources (e.g., DNS); and access management and monitoring resources, etc. Each region typically has multiple paths connecting it to other regions within the domain.

[0045] Generally, applications are deployed in the areas where they are used most frequently (i.e., on infrastructure associated with that area) because using nearby resources is faster than using distant ones. Applications may also be deployed in different areas for various reasons, such as redundancy to mitigate the risk of events within a region (such as large weather systems or earthquakes), or to meet different requirements of legal jurisdictions, tax zones, and other business or social standards.

[0046] Data centers within a region can be further organized and subdivided into Availability Domains (ADs). An Availability Domain can correspond to one or more data centers located within a region. A region can consist of one or more Availability Domains. In this distributed environment, CSPI resources are either region-specific, such as Virtual Cloud Networks (VCNs), or Availability Domain-specific, such as compute instances.

[0047] Availability Zones (ADs) within a region are isolated from each other, fault-tolerant, and configured to make simultaneous failures highly unlikely. This is achieved by ensuring that ADs do not share critical infrastructure resources (such as networking, physical cabling, cable paths, cable entry points, etc.), making a failure at one AD within a region unlikely to affect the availability of other ADs in the same region. ADs within the same region can be interconnected via low-latency, high-bandwidth networks, enabling high-availability connectivity to other networks (e.g., the internet, customer on-premises networks, etc.) and allowing for replication systems across multiple ADs to achieve high availability and disaster recovery. Cloud services use multiple ADs to ensure high availability and prevent resource failures. As the infrastructure provided by an IaaS provider grows, more regions and ADs, along with additional capacity, can be added. Traffic between availability domains is typically encrypted.

[0048] In some embodiments, regions are grouped into domains. A domain is a logical collection of regions. Domains are isolated from each other and do not share any data. Regions within the same domain can communicate with each other, but regions in different domains cannot. A customer's lease or account with the CSP resides in a single domain and can be distributed across one or more regions belonging to that domain. Typically, when a customer subscribes to an IaaS service, a lease or account is created for that customer in a region within the domain that the customer designates (called the "primary" region). A customer can extend their lease to one or more other regions within the domain. A customer cannot access regions that are not within the domain where their lease is located.

[0049] IaaS providers can offer multiple domains, each catering to a specific set of customers or users. For example, a business domain can be offered for business customers. As another example, a domain can be offered for customers within a specific country. As yet another example, a government domain can be offered for governments, etc. For instance, a government domain can cater to a specific government and may have a higher level of security than the business domain. For example, Oracle Cloud Infrastructure (OCI) currently offers domains for the business region and two domains for the government cloud region (e.g., FedRAMP licensed and IL5 licensed).

[0050] In some embodiments, an Active Directory (AD) can be subdivided into one or more fault domains. A fault domain is a grouping of infrastructure resources within an AD to provide anti-affinity. Fault domains allow for the distribution of compute instances such that these instances do not reside on the same physical hardware within a single AD. This is known as anti-affinity. A fault domain refers to a collection of hardware components (computers, switches, etc.) that share a single point of failure. Compute pools are logically divided into fault domains. Therefore, a hardware failure or compute hardware maintenance event affecting one fault domain does not affect instances in other fault domains. Depending on the embodiment, the number of fault domains for each AD can vary. For example, in some embodiments, each AD contains three fault domains. Fault domains act as logical data centers within the AD.

[0051] When a customer subscribes to IaaS services, resources from CSPI are provisioned to the customer and associated with the customer's lease. Customers can use these provisioned resources to build private networks and deploy resources on those networks. Customer networks hosted in the cloud by CSPI are called Virtual Cloud Networks (VCNs). Customers can use the CSPI resources allocated to them to set up one or more VCNs. A VCN is a virtual or software-defined private network. Customer resources deployed in a customer's VCN can include compute instances (e.g., virtual machines, bare metal instances) and other resources. These compute instances can represent various customer workloads, such as applications, load balancers, databases, etc. Compute instances deployed on a VCN can communicate with publicly accessible endpoints (“public endpoints”) via public networks (such as the Internet), with other instances in the same VCN or other VCNs (e.g., other VCNs belonging to the customer or VCNs not belonging to the customer), with the customer's on-premises data center or network, and with service endpoints and other types of endpoints.

[0052] CSPs can use CSPIs to provide various services. In some cases, CSPI clients can function as service providers themselves and use CSPI resources to provide services. Service providers can expose service endpoints, which are characterized by identifying information such as IP addresses, DNS names, and ports. Client resources (e.g., compute instances) can access a specific service by accessing the service endpoints exposed by the service for that specific service. These service endpoints are generally publicly accessible to users via public communication networks (such as the Internet) using the public IP addresses associated with the endpoints. Publicly accessible network endpoints are sometimes also called public endpoints.

[0053] In some embodiments, a service provider may expose the service via an endpoint (sometimes referred to as a service endpoint) for the service. Clients of the service can then use this service endpoint to access the service. In some implementations, the service endpoint provided for the service can be accessed by multiple clients intending to consume the service. In other implementations, a dedicated service endpoint can be provided to a client, such that only that client can use that dedicated service endpoint to access the service.

[0054] In some embodiments, when a VCN is created, it is associated with a Private Overlay Classless Inter-Domain Routing (CIDR) address space, which refers to a set of private overlay IP addresses (e.g., 10.0 / 16) assigned to the VCN. A VCN includes associated subnets, routing tables, and gateways. A VCN resides within a single region but can span one or more of the availability domains within that region. A gateway is a virtual interface configured for the VCN and enables communication between the VCN and one or more endpoints outside the VCN. One or more different types of gateways can be configured for a VCN to enable communication to and from different types of endpoints.

[0055] A VCN can be subdivided into one or more subnets, such as one or more subnets. Therefore, a subnet is a configurable unit or subdivision that can be created within a VCN. A VCN can have one or more subnets. Each subnet within a VCN is associated with a contiguous range of overlay IP addresses (e.g., 10.0.0.0 / 24 and 10.0.1.0 / 24), which do not overlap with other subnets within that VCN and represent a subset of the VCN's address space.

[0056] Each compute instance is associated with a Virtual Network Interface Card (VNIC), which enables the compute instance to participate in subnets within a VCN. A VNIC is the logical representation of a physical network interface card (NIC). Generally, a VNIC is the interface between an entity (e.g., a compute instance, a service) and the virtual network. A VNIC exists within a subnet and has one or more associated IP addresses, along with associated security rules or policies. A VNIC is equivalent to a Layer 2 port on a switch. A VNIC is attached to both the compute instance and the subnet within the VCN. The VNIC associated with a compute instance makes the compute instance part of the VCN's subnet and enables the compute instance to communicate (e.g., send and receive packets) with endpoints on the same subnet as the compute instance, with endpoints in different subnets within the VCN, or with endpoints outside the VCN. Therefore, the VNIC associated with a compute instance determines how the compute instance connects to endpoints inside and outside the VCN. When a compute instance is created and added to a subnet within a VCN, a VNIC for the compute instance is created and associated with that compute instance. For a subnet that includes a set of compute instances, the subnet contains a VNIC corresponding to that set of compute instances, with each VNIC attached to a compute instance within that set of compute instances.

[0057] A private overlay IP address is assigned to each compute instance via the VNIC associated with it. This private overlay network IP address is assigned to the VNIC associated with the compute instance when the compute instance is created and is used to route traffic to and from the compute instance. All VNICs within a given subnet use the same routing table, security list, and DHCP options. As described above, each subnet within a VCN is associated with a contiguous range of overlay IP addresses (e.g., 10.0.0.0 / 24 and 10.0.1.0 / 24) that do not overlap with other subnets within that VCN and represent a subset of the address space within the VCN's address space. For a VNIC on a specific subnet of a VCN, the private overlay IP address assigned to that VNIC is an address from the contiguous range of overlay IP addresses allocated to the subnet.

[0058] In some embodiments, in addition to a private overlay IP address, a compute instance may optionally be assigned additional overlay IP addresses, such as one or more public IP addresses, for example, if in a public subnet. These multiple addresses may be assigned either on the same VNIC or on multiple VNICs associated with the compute instance. However, each instance has a primary VNIC, which is created during instance startup and associated with the overlay private IP address assigned to that instance—this primary VNIC cannot be deleted. Additional VNICs, called secondary VNICs, can be added to existing instances in the same availability domain as the primary VNIC. All VNICs are in the same availability domain as the instance. Secondary VNICs may reside in a subnet within the same VCN as the primary VNIC, or in a different subnet within the same VCN or different VCNs.

[0059] If a compute instance is in a public subnet, it can optionally be assigned a public IP address. When creating a subnet, it can be specified as either a public or private subnet. A private subnet means that resources within the subnet (such as compute instances) and associated VNICs cannot have public overriding IP addresses. A public subnet means that resources within the subnet and associated VNICs can have public IP addresses. Customers can specify that a subnet exists within a single availability domain or across multiple availability domains in regions or domains.

[0060] As described above, a VCN can be subdivided into one or more subnets. In some embodiments, a virtual router (VR) configured for the VCN (referred to as a VCN VR or simply VR) enables communication between the subnets of the VCN. For a subnet within the VCN, the VR represents a logical gateway for that subnet, enabling that subnet (i.e., the compute instances on that subnet) to communicate with endpoints on other subnets within the VCN and other endpoints outside the VCN. The VCN VR is a logical entity configured to route traffic between the VNIC within the VCN and the virtual gateway (“gateway”) associated with the VCN. The following section discusses… Figure 1Further description of the gateway. A VCN VR is a Layer 3 / IP layer concept. In one embodiment, there exists a VCN VR for a VCN, where the VCN VR has a potentially unlimited number of IP-addressable ports, one port for each subnet of the VCN. In this way, the VCN VR has a different IP address for each subnet within the VCN to which the VCN VR is attached. The VR also connects to various gateways configured for the VCN. In some embodiments, a specific overlay IP address within the overlay IP address range for a subnet is reserved for the port of the VCN VR for that subnet. For example, consider a VCN with two subnets, associated with address ranges 10.0 / 16 and 10.1 / 16. For the first subnet in the VCN with an address range of 10.0 / 16, addresses within this range are reserved for the port of the VCN VR for that subnet. In some cases, the first IP address within the range can be reserved for the VCN VR. For example, for a subnet covering the IP address range of 10.0 / 16, the IP address 10.0.0.1 could be reserved for the port of the VCN VR for that subnet. For a second subnet within the same VCN with an address range of 10.1 / 16, the VCN VR can have a port for the second subnet with the IP address 10.1.0.1. The VCN VR has a different IP address for each subnet within the VCN.

[0061] In some other embodiments, each subnet within a VCN may have its own associated VR, which can be addressed by the subnet using a reserved or default IP address associated with the VR. For example, the reserved or default IP address may be the first IP address in a range of IP addresses associated with the subnet. The VNICs within the subnet can use this default or reserved IP address to communicate with the VR associated with the subnet (e.g., send and receive packets). In this embodiment, the VR is the ingress / egress point of the subnet. The VR associated with a subnet within the VCN can communicate with other VRs associated with other subnets within the VCN. The VR can also communicate with the gateway associated with the VCN. The VR functionality of a subnet runs on or is performed by one or more NVDs that perform VNIC functionality for the VNICs within the subnet.

[0062] You can configure routing tables, security rules, and DHCP options for a VCN. A routing table is a virtual routing table used by the VCN and contains rules that route traffic from subnets within the VCN to destinations outside the VCN via gateways or specially configured instances. You can customize the VCN's routing table to control how packets are forwarded / routed to and from the VCN. DHCP options refer to configuration information automatically provided to the instance when it starts up.

[0063] Security rules configured for a VCN represent overlay firewall rules used by the VCN. Security rules can include ingress and egress rules, specifying the types of traffic allowed to enter and exit the VCN instance (e.g., based on protocol and port). Clients can choose whether a given rule is stateful or stateless. For example, a client can allow a set of incoming SSH traffic from anywhere to the instance by setting a stateful ingress rule with source CIDR 0.0.0.0 / 0 and destination TCP port 22. Security rules can be implemented using network security groups or security lists. A network security group consists of a set of security rules that apply only to resources within that group. A security list, on the other hand, includes rules applicable to all resources in any subnet using that security list. A default security list with default security rules can be provided to the VCN. DHCP options configured for the VCN provide configuration information that is automatically provided to instances within the VCN at instance startup.

[0064] In some embodiments, configuration information for a VCN is determined and stored by the VCN control plane. For example, configuration information for a VCN may include information about: the address range associated with the VCN, subnets and associated information within the VCN, one or more VRs associated with the VCN, compute instances and associated VNICs within the VCN, NVDs (e.g., VNICs, VRs, gateways) performing various virtualization network functions associated with the VCN, status information for the VCN, and other VCN-related information. In some embodiments, a VCN distribution service publishes configuration information stored by the VCN control plane or a portion thereof to the NVD. The distributed information can be used to update information stored and used by the NVD (e.g., forwarding tables, routing tables, etc.) to forward packets to or from compute instances within the VCN.

[0065] In some embodiments, the creation of VCNs and subnets is handled by the VCN control plane (CP), and the initiation of compute instances is handled by the compute control plane. The compute control plane is responsible for allocating physical resources to the compute instances and then invoking the VCN control plane to create VNICs and attach them to the compute instances. The VCN CP also maps VCN data to the VCN data plane, which is configured to perform packet forwarding and routing functions. In some embodiments, the VCN CP provides a distribution service responsible for providing updates to the VCN data plane. Examples of VCN control planes are also available in... Figure 17 , Figure 18 , Figure 19 and Figure 20 The description is as follows (see references 1716, 1816, 1916 and 2016).

[0066] Customers can create one or more VCNs using resources hosted by CSPI. Compute instances deployed on a customer's VCN can communicate with different endpoints. These endpoints can include endpoints hosted by CSPI and endpoints external to CSPI.

[0067] Figure 1 , Figure 2 , Figure 3 , Figure 4 , Figure 5 , Figure 17 , Figure 18 , Figure 19 and Figure 21 The document describes various architectures for implementing cloud-based services using CSPI, and these are described below. Figure 1 This is a high-level diagram of a distributed environment 100, illustrating an overlay or client VCN hosted by CSPI according to certain embodiments. Figure 1 The distributed environment described includes multiple components in the overlay network. Figure 1 The distributed environment 100 depicted herein is merely an example and is not intended to unduly limit the scope of the claimed embodiments. Many variations, substitutions, and modifications are possible. For example, in some implementations, Figure 1 The distributed environment described in the text can have more than Figure 1 The more or fewer systems or components shown can be combined into two or more systems, or can have different system configurations or arrangements.

[0068] like Figure 1 As illustrated in the example, distributed environment 100 includes a CSPI 101 that provides services and resources that customers can subscribe to and use to build their Virtual Cloud Network (VCN). In some embodiments, CSPI 101 provides IaaS services to subscribing customers. Data centers within CSPI 101 can be organized into one or more regions. Figure 1 The example shown is a region "Region US" 102. The customer has already configured customer VCN 104 for Region 102. The customer can deploy various compute instances on VCN 104, which can include virtual machines or bare metal instances. Examples of instances include applications, databases, load balancers, etc.

[0069] exist Figure 1 In the embodiment depicted, customer VCN 104 includes two subnets, namely "Subnet-1" and "Subnet-2", each with its own CIDR IP address range. Figure 1In this configuration, subnet-1 covers the IP address range of 10.0 / 16, and subnet-2 covers the address range of 10.1 / 16. VCN Virtual Router 105 represents the logical gateway for the VCN, enabling communication between subnets of VCN 104 and with other endpoints outside the VCN. VCN VR 105 is configured to route traffic between the VNICs within VCN 104 and the gateway associated with VCN 104. VCN VR 105 provides a port for each subnet of VCN 104. For example, VR 105 could provide a port with IP address 10.0.0.1 for subnet-1 and a port with IP address 10.1.0.1 for subnet-2.

[0070] Multiple compute instances can be deployed on each subnet, where compute instances can be virtual machine instances and / or bare metal instances. Compute instances within a subnet can be hosted by one or more host machines within CSPI 101. Compute instances participate in the subnet via a VNIC associated with the compute instance. For example, as... Figure 1 As shown, compute instance C1 becomes part of subnet-1 via the VNIC associated with it. Similarly, compute instance C2 becomes part of subnet-1 via the VNIC associated with it. In a similar manner, multiple compute instances (which can be virtual machine instances or bare metal instances) can be part of subnet-1. Each compute instance is assigned a private overlay IP address and MAC address via its associated VNIC. For example, in... Figure 1 In this subnet, compute instance C1 has an overlay IP address of 10.0.0.2 and a MAC address of M1, while compute instance C2 has a private overlay IP address of 10.0.0.3 and a MAC address of M2. Each compute instance in subnet-1 (including compute instances C1 and C2) has a default route to VCN VR 105 using IP address 10.0.0.1, which is the IP address of the port of VCN VR 105 used in subnet-1.

[0071] Multiple compute instances, including virtual machine instances and / or bare metal instances, can be deployed on subnet-2. For example, such as Figure 1 As shown, compute instances D1 and D2 become part of subnet-2 via the VNIC associated with the respective compute instance. Figure 1 In the illustrated embodiment, compute instance D1 has an overlay IP address of 10.1.0.2 and a MAC address of MM1, while compute instance D2 has a private overlay IP address of 10.1.0.3 and a MAC address of MM2. Each compute instance in subnet-2 (including compute instances D1 and D2) has a default route to VCN VR 105 using IP address 10.1.0.1, which is the IP address of the port of VCN VR 105 in subnet-2.

[0072] VCN A 104 may also include one or more load balancers. For example, a load balancer can be provided for a subnet, and the load balancer can be configured to load balance traffic across multiple compute instances on the subnet. Load balancers can also be provided to load balance traffic across subnets within a VCN.

[0073] A specific compute instance deployed on VCN 104 can communicate with a variety of different endpoints. These endpoints can include endpoints hosted by CSPI 200 and endpoints outside of CSPI 200. Endpoints hosted by CSPI 101 can include: endpoints located on the same subnet as the specific compute instance (e.g., communication between two compute instances in subnet-1); endpoints located on different subnets but within the same VCN (e.g., communication between a compute instance in subnet-1 and a compute instance in subnet-2); endpoints in different VCNs within the same region (e.g., communication between a compute instance in subnet-1 and an endpoint in a VCN in the same region 106 or 110, or communication between a compute instance in subnet-1 and an endpoint in service point 110 within the same region); or endpoints in VCNs in different regions (e.g., communication between a compute instance in subnet-1 and an endpoint in a VCN in a different region 108). Compute instances in a subnet hosted by CSPI 101 can also communicate with endpoints not hosted by CSPI 101 (i.e., outside of CSPI 101). These external endpoints include endpoints in the customer’s on-premises network 116, endpoints in other remote cloud-hosted networks 118, public endpoints 114 accessible via public networks such as the Internet, and other endpoints.

[0074] VNICs associated with source and destination compute instances facilitate communication between compute instances on the same subnet. For example, compute instance C1 in subnet-1 might want to send a packet to compute instance C2 in subnet-1. For a packet originating from the source compute instance and destined for another compute instance in the same subnet, the packet is first processed by the VNIC associated with the source compute instance. The processing performed by the VNIC associated with the source compute instance may include determining the packet's destination information from the packet header, identifying any policies (e.g., security lists) configured for the VNIC associated with the source compute instance, determining the next hop for the packet, performing any packet encapsulation / decapsulation functions as needed, and then forwarding / routing the packet to the next hop to facilitate communication to its intended destination. When the destination compute instance is in the same subnet as the source compute instance, the VNIC associated with the source compute instance is configured to identify the VNIC associated with the destination compute instance and forward the packet to that VNIC for processing. The VNIC associated with the destination compute instance then performs the processing and forwards the packet to the destination compute instance.

[0075] For packets to be transferred from compute instances in a subnet to endpoints in different subnets within the same VCN, communication is facilitated by the VNIC associated with the source and destination compute instances, as well as the VCN VR. For example, if Figure 1 If compute instance C1 in subnet-1 wants to send a packet to compute instance D1 in subnet-2, the packet is first processed by the VNIC associated with compute instance C1. The VNIC associated with compute instance C1 is configured to route the packet to VCN VR 105 using the default route or port 10.0.0.1 of the VCN VR. VCN VR 105 is configured to route the packet to subnet-2 using port 10.1.0.1. Then, the VNIC associated with D1 receives and processes the packet and forwards it to compute instance D1.

[0076] For packets to be transferred from a compute instance in VCN 104 to an endpoint outside VCN 104, communication is facilitated by the VNIC associated with the source compute instance, VCN VR 105, and the gateway associated with VCN 104. One or more types of gateways can be associated with VCN 104. A gateway is an interface between a VCN and another endpoint located outside the VCN. A gateway is a Layer 3 / IP concept and enables a VCN to communicate with endpoints outside the VCN. Therefore, a gateway facilitates traffic flow between a VCN and other VCNs or networks. Various different types of gateways can be configured for a VCN to facilitate different types of communication with different types of endpoints. Depending on the gateway, communication can occur over a public network (e.g., the Internet) or over a private network. Various communication protocols can be used for these communications.

[0077] For example, compute instance C1 might want to communicate with an endpoint outside of VCN 104. The packet can first be processed by the VNIC associated with the source compute instance C1. The VNIC processing determines that the packet's destination is outside C1's subnet-1. The VNIC associated with C1 can then forward the packet to VCN VR 105 for VCN 104. VCN VR 105 then processes the packet and, as part of the processing, determines a specific gateway associated with VCN 104 as the next hop for the packet based on its destination. VCN VR 105 can then forward the packet to the specific identified gateway. For example, if the destination is an endpoint within a customer's on-premises network, the packet can be forwarded by VCN VR 105 to the Dynamic Routing Gateway (DRG) gateway 122 configured for VCN 104. The packet can then be forwarded from the gateway to the next hop to facilitate delivery to its final intended destination.

[0078] Various types of gateways can be configured for a VCN. Examples of gateways that can be configured for a VCN are available in [link to example]. Figure 1 It is depicted in [the text] and described below. Examples of gateways associated with VCNs are also [described in the text]. Figure 17 , Figure 18 , Figure 19 and Figure 20 The gateways are depicted (e.g., referenced by reference numerals 1734, 1736, 1738, 1834, 1836, 1838, 1934, 1936, 1938, 2034, 2036, and 2038) and are described below. Figure 1As depicted in the embodiments, a Dynamic Routing Gateway (DRG) 122 can be added to or associated with a customer VCN 104 and provides a path for private network traffic communication between the customer VCN 104 and another endpoint, which can be the customer's on-premises network 116, a VCN 108 in a different region of CSPI 101, or another remote cloud network 118 not hosted by CSPI 101. The customer's on-premises network 116 can be a customer network or customer data center built using the customer's resources. Access to the customer's on-premises network 116 is generally very restricted. For customers who have both a customer's on-premises network 116 and one or more VCNs 104 deployed or hosted in the cloud by CSPI 101, the customer may want their on-premises network 116 and their cloud-based VCN 104 to be able to communicate with each other. This allows customers to build extended hybrid environments that include the customer's VCN 104 hosted by CSPI 101 and their on-premises network 116. The DRG 122 enables this communication. To enable this type of communication, a communication channel 124 is set up, with one endpoint located in the customer's on-premises network 116 and the other endpoint located in CSPI 101 and connected to the customer's VCN 104. Communication channel 124 can be over a public communication network (such as the Internet) or a private communication network. Various communication protocols can be used, such as IPsec VPN technology over a public communication network (such as the Internet), Oracle's FastConnect technology using a private network instead of a public network, etc. The device or equipment forming one endpoint of communication channel 124 in the customer's on-premises network 116 is referred to as a customer field equipment (CPE), such as... Figure 1 The CPE126 is depicted in the diagram. On the CSPI 101 side, the endpoint can be a host machine executing DRG 122.

[0079] In some embodiments, a remote peering connection (RPC) can be added to the DRG, which allows a customer to peer one VCN to another VCN in a different region. Using such an RPC, customer VCN 104 can use DRG 122 to connect to VCN 108 in another region. DRG 122 can also be used to communicate with other remote cloud networks 118 (such as Microsoft Azure, Amazon AWS, etc.) not hosted by CSPI 101.

[0080] like Figure 1As shown, an Internet Gateway (IGW) 120 can be configured for customer VCN 104, enabling compute instances on VCN 104 to communicate with a public endpoint 114 accessible via a public network, such as the Internet. IGW 1120 is a gateway connecting the VCN to a public network such as the Internet. IGW 120 enables public subnets within the VCN (such as VCN 104), where resources within the public subnet have publicly overriding IP addresses, to directly access a public endpoint 112 on the public network 114 (such as the Internet). Using IGW 120, connections can be initiated from subnets within VCN 104 or from the Internet.

[0081] A Network Address Translation (NAT) gateway 128 can be configured for a customer's VCN 104, enabling cloud resources within the customer's VCN that do not have dedicated public overlay IP addresses to access the Internet, and doing so to direct incoming Internet connections (e.g., L4-L7 connections) without exposing those resources. This allows private subnets within the VCN (such as private subnet-1 in VCN 104) to privately access public endpoints on the Internet. In a NAT gateway, connections can only originate from private subnets to the public Internet, and not from the Internet to private subnets.

[0082] In some embodiments, a Service Gateway (SGW) 126 can be configured for customer VCN 104 to provide a path for private network traffic between VCN 104 and service endpoints supported in service network 110. In some embodiments, service network 110 may be provided by a CSP and may offer a variety of services. An example of such a service network is Oracle's service network, which provides a variety of services available to customers. For example, compute instances (e.g., database systems) in a private subnet of customer VCN 104 can back up data to service endpoints (e.g., object storage devices) without requiring a public IP address or access to the Internet. In some embodiments, a VCN may have only one SGW, and connections can only originate from subnets within the VCN, not from service network 110. If a VCN is peered to another, resources in the other VCN typically cannot access the SGW. Resources in the on-premises network of a VCN connected using FastConnect or VPN Connect can also use the service gateway configured for that VCN.

[0083] In some implementations, the SGW 126 uses the concept of a Service Classless Inter-Domain Routing (CIDR) label, which is a string representing all regional public IP address ranges for the service or group of services of interest. Customers use the Service CIDR label when configuring the SGW and associated routing rules to control traffic to the service. If the public IP address of the service changes in the future, customers can optionally use it when configuring security rules without having to adjust them.

[0084] Local peering gateway (LPG) 132 is a gateway that can be added to customer VCN 104 and enable VCN 104 to peer with another VCN in the same region. Peering means that VCNs communicate using private IP addresses, and traffic does not need to traverse public networks (such as the Internet) or be routed through the customer's on-premises network 116. In a preferred embodiment, a VCN has a separate LPG for each peer it establishes. Local peering or VCN peering is a common practice for establishing network connectivity between different applications or infrastructure management functions.

[0085] Service providers (such as those providing services in service network 110) can offer access to services using different access models. Under the public access model, a service can be exposed as a public endpoint that is publicly accessible by compute instances within a customer's VCN via a public network (such as the Internet), and / or can be privately accessed via SGW 126. Under a specific private access model, a service can be accessed as a private IP endpoint within a private subnet of the customer's VCN. This is called Private Endpoint (PE) access and enables service providers to expose their services as instances within the customer's private network. A private endpoint resource represents a service within the customer's VCN. Each PE is represented as a VNIC (called a PE-VNIC, with one or more private IPs) within the customer's VCN in a subnet selected by the customer. Thus, the PE provides a way to present services within a private customer VCN subnet using a VNIC. Since the endpoint is exposed as a VNIC, all characteristics associated with the VNIC (such as routing rules, security lists, etc.) are now available for the PE VNIC.

[0086] Service providers can register their services to enable access via PE. Providers can associate policies with services, which restricts the visibility of the service to customer leases. Providers can register multiple services under a single Virtual IP address (VIP), especially for multi-tenant services. Multiple such private endpoints (across multiple VCNs) can represent the same service.

[0087] Compute instances in a private subnet can then access the service using the private IP address or service DNS name of the PE VNIC. Compute instances in a customer VCN can access the service by sending traffic to the private IP address of the PE in the customer VCN. A Private Access Gateway (PAGW) 130 is a gateway resource that can be attached to a service provider VCN (e.g., a VCN in service network 110), which acts as the ingress / egress point for all traffic from / to the private endpoints of the customer subnet. PAGW 130 enables providers to scale the number of PE connections without utilizing their internal IP address resources. A provider only needs to configure one PAGW for any number of services registered in a single VCN. A provider can represent a service as a private endpoint in multiple VCNs of one or more customers. From the customer's perspective, the PE VNIC is not an instance attached to the customer, but rather appears to be attached to the service that the customer wants to interact with. Traffic to the private endpoint is routed to the service via PAGW 130. These are referred to as customer-to-service private connections (C2S connections).

[0088] The PE concept can also be used to extend private access to services to the customer's on-premises network and data center by allowing traffic to flow through FastConnect / IPsec links and private endpoints in the customer's VCN. Private access to services can also be extended to the customer's peering VCN by allowing traffic to flow between the LPG 132 and the PE in the customer's VCN.

[0089] Customers can control routing within a VCN at the subnet level, allowing them to specify which subnets within a customer's VCN (such as VCN104) use each gateway. The VCN's routing table is used to determine whether traffic is allowed to leave the VCN via a specific gateway. For example, in a given instance, a routing table for a public subnet within customer VCN 104 might send non-local traffic via IGW 120. A routing table for a private subnet within the same customer VCN 104 might send traffic destined for a CSP service via SGW 126. All remaining traffic can be sent via NAT gateway 128. The routing table only controls traffic flowing out of the VCN.

[0090] Security lists associated with a VCN are used to control traffic entering the VCN via a gateway through inbound connections. All resources within a subnet use the same routing tables and security lists. Security lists can be used to control specific types of traffic allowed to enter or leave instances within a subnet of the VCN. Security list rules can include inbound and outbound rules. For example, inbound rules can specify allowed source address ranges, while outbound rules can specify allowed destination address ranges. Security rules can specify specific protocols (e.g., TCP, ICMP), specific ports (e.g., port 22 for SSH, port 3389 for Windows RDP), etc. In some implementations, the instance's operating system can enforce its own firewall rules that conform to the security list rules. Rules can be stateful (e.g., tracking connections and automatically allowing responses without explicit security list rules for response traffic) or stateless.

[0091] Access from a customer's VCN (i.e., through resources or compute instances deployed on VCN 104) can be categorized as public access, private access, or dedicated access. Public access refers to an access model that uses a public IP address or NAT to access a public endpoint. Private access enables customer workloads with private IP addresses within VCN 104 (e.g., resources in a private subnet) to access services without traversing a public network such as the Internet. In some embodiments, CSPI 101 enables customer VCN workloads with private IP addresses to access services (public service endpoints) using a service gateway. Thus, the service gateway provides a private access model by establishing a virtual link between the customer's VCN and the public endpoint of the service residing outside the customer's private network.

[0092] Furthermore, CSPI can provide dedicated public access using technologies such as FastConnect public peering, where on-premises instances can access one or more services within a customer's VCN using FastConnect connections without traversing public networks such as the internet. CSPI can also provide dedicated private access using FastConnect private peering, where on-premises instances with private IP addresses can access a customer's VCN workloads using FastConnect connections. FastConnect is a network connectivity alternative to using the public internet to connect a customer's on-premises network to CSPI and its services. Compared to internet-based connections, FastConnect offers a simple, flexible, and cost-effective way to create dedicated and private connections with higher bandwidth options and a more reliable and consistent network experience.

[0093] Figure 1The accompanying description above describes the various virtualized components in the example virtual network. As mentioned above, the virtual network is built on the underlying physical or substrate network. Figure 2 A simplified architecture diagram of the physical components within the physical network of the CSPI 200, which provides the underlying layer for virtual networks according to certain embodiments, is depicted. As shown, the CSPI 200 provides a distributed environment including components and resources (e.g., compute, memory, and network resources) provided by a cloud service provider (CSP). These components and resources are used to provide cloud services (e.g., IaaS services) to subscribed customers (i.e., customers who have subscribed to one or more services provided by the CSP). Based on the services subscribed to by the customer, a subset of the resources of the CSPI 200 (e.g., compute, memory, and network resources) is provisioned to the customer. The customer can then use the physical compute, memory, and networking resources provided by the CSPI 200 to build their own cloud-based (i.e., CSPI-hosted) customizable and private virtual networks. As indicated above, these customer networks are referred to as Virtual Cloud Networks (VCNs). Customers can deploy one or more customer resources, such as compute instances, on these customer VCNs. Compute instances can take the form of virtual machines, bare metal instances, etc. The CSPI 200 provides a collection of infrastructure and complementary cloud services, enabling customers to build and run a wide range of applications and services in a highly available managed environment.

[0094] exist Figure 2 In the example embodiment depicted, the physical components of CSPI 200 include one or more physical host machines or physical servers (e.g., 202, 206, 208), network virtualization devices (NVDs) (e.g., 210, 212), top-of-rack (TOR) switches (e.g., 214, 216), and a physical network (e.g., 218), as well as switches within physical network 218. The physical host machines or servers can host and execute various compute instances participating in one or more subnets of the VCN. Compute instances can include virtual machine instances and bare metal instances. For example, Figure 1 The various computational examples described in the text can be derived from... Figure 2 The physical host machine described in the diagram is used for hosting virtual machine compute instances in a VCN. Virtual machine compute instances in a VCN can be executed by one host machine or multiple different host machines. A physical host machine can also host virtual host machines, container-based hosts, or functions, etc. Figure 1 The VNIC and VCN VR described in the text can be generated by Figure 2 The NVD execution described in the text. Figure 1 The gateway described in the text can be... Figure 2 The host machine and / or NVD execution described in the document.

[0095] A host machine or server can run a hypervisor (also known as a virtual machine monitor or VMM) that creates and enables virtualized environments on the host machine. Virtualization or virtualized environments facilitate cloud-based computing. One or more compute instances can be created, executed, and managed on the host machine by a hypervisor on that host machine. The hypervisor on the host machine enables the host machine's physical computing resources (e.g., compute, memory, and network resources) to be shared among various compute instances executed by the host machine.

[0096] For example, such as Figure 2 As depicted, host machines 202 and 208 execute hypervisors 260 and 266, respectively. These hypervisors can be implemented using software, firmware, or hardware, or a combination thereof. Typically, a hypervisor is a process or software layer residing above the host machine's operating system (OS), which in turn executes on the host machine's hardware processor. Hypervisors provide a virtualized environment by enabling the host machine's physical computing resources (e.g., processing resources such as processors / cores, memory resources, network resources) to be shared among various virtual machine computing instances executed by the host machine. For example, in... Figure 2 In this configuration, the hypervisor 260 can reside on top of the operating system of the host machine 202 and enable the computing resources (e.g., processing, memory, and network resources) of the host machine 202 to be shared among computing instances (e.g., virtual machines) executed by the host machine 202. A virtual machine can have its own operating system (called a guest operating system), which may be the same as or different from the host machine's operating system. The operating system of a virtual machine executed by the host machine can be the same as or different from the operating system of another virtual machine executed by the same host machine. Therefore, the hypervisor enables multiple operating systems to be executed simultaneously, while sharing the same computing resources of the host machine. Figure 2 The host machines described in the text may have the same or different types of management programs.

[0097] A compute instance can be a virtual machine instance or a bare metal instance. Figure 2 In the diagram, compute instance 268 on host machine 202 and compute instance 274 on host machine 208 are examples of virtual machine instances. Host machine 206 is an example of a bare metal instance provided to a customer.

[0098] In some cases, an entire host machine can be provisioned to a single customer, and one or more compute instances (or virtual machines or bare metal instances) hosted by that host machine all belong to the same customer. In other cases, the host machine can be shared among multiple customers (i.e., multiple tenants). In this multi-tenancy scenario, the host machine can host virtual machine compute instances belonging to different customers. These compute instances can be members of different VCNs for different customers. In some embodiments, bare metal compute instances are hosted by bare metal servers without a hypervisor. When provisioning bare metal compute instances, a single customer or tenant maintains control over the physical CPU, memory, and network interfaces of the host machine hosting the bare metal instance, and the host machine is not shared with other customers or tenants.

[0099] As previously mentioned, each compute instance as part of a VCN is associated with a VNIC that enables that compute instance to become a member of a subnet of the VCN. The VNIC associated with a compute instance facilitates communication to and from the compute instance. The VNIC is associated with the compute instance when the compute instance is created. In some embodiments, for compute instances executed by a host machine, the VNIC associated with that compute instance is executed by an NVD connected to the host machine. For example, in Figure 2 In this example, host machine 202 executes a virtual machine compute instance 268 associated with VNIC 276, and VNIC 276 is executed by NVD 210 connected to host machine 202. As another example, a bare metal instance 272 hosted by host machine 206 is associated with VNIC 280 executed by NVD 212 connected to host machine 206. As yet another example, VNIC 284 is associated with compute instance 274 executed by host machine 208, and VNIC 284 is executed by NVD 212 connected to host machine 208.

[0100] For compute instances hosted by a host machine, NVDs connected to that host machine also execute VCN VR corresponding to the VCN where the compute instance is a member. For example, in Figure 2 In the embodiment depicted, NVD 210 executes VCN VR 277 corresponding to a VCN that is a member of compute instance 268. NVD 212 may also execute one or more VCN VR 283 corresponding to a VCN that corresponds to a compute instance hosted by host machines 206 and 208.

[0101] The host machine may include one or more network interface cards (NICs) that enable the host machine to connect to other devices. The NIC on the host machine can provide one or more ports (or interfaces) that allow the host machine to communicate with another device. For example, the host machine can use one or more ports (or interfaces) provided on the host machine and the NVD to connect to the NVD. The host machine can also connect to other devices (such as another host machine).

[0102] For example, in Figure 2 In this configuration, host machine 202 is connected to NVD 210 via link 220, which extends between port 234 provided by NIC 232 of host machine 202 and port 236 of NVD 210. Host machine 206 is connected to NVD 212 via link 224, which extends between port 246 provided by NIC 244 of host machine 206 and port 248 of NVD 212. Host machine 208 is connected to NVD 212 via link 226, which extends between port 252 provided by NIC 250 of host machine 208 and port 254 of NVD 212.

[0103] The NVD is then connected to top-of-rack (TOR) switches via communication links, which are connected to physical network 218 (also known as a switch architecture). In some embodiments, the links between the host machine and the NVD, and between the NVD and the TOR switches, are Ethernet links. For example, in Figure 2 In this configuration, NVDs 210 and 212 are connected to TOR switches 214 and 216 via links 228 and 230, respectively. In some embodiments, links 220, 224, 226, 228, and 230 are Ethernet links. The collection of host machines and NVDs connected to the TOR is sometimes referred to as a rack.

[0104] Physical network 218 provides a communication architecture that enables TOR switches to communicate with each other. Physical network 218 can be a multi-layer network. In some implementations, physical network 218 is a multi-layer Clos network of switches, where TOR switches 214 and 216 represent leaf-level nodes of the multi-layer and multi-node physical switching network 218. Different Clos network configurations are possible, including but not limited to Layer 2 networks, Layer 3 networks, Layer 4 networks, Layer 5 networks, and general "n"-layer networks. Examples of Clos networks are provided in... Figure 5 It is depicted in the middle and described below.

[0105] Various connection configurations can exist between the host machine and the NVD, such as one-to-one, many-to-one, and one-to-many configurations. In a one-to-one configuration, each host machine connects to its own individual NVD. For example, in... Figure 2 In this configuration, host machine 202 connects to NVD 210 via its NIC 232. In a many-to-one configuration, multiple host machines connect to a single NVD. For example, in... Figure 2 In this configuration, host machines 206 and 208 are connected to the same NVD 212 via NIC244 and 250, respectively.

[0106] In a one-to-many configuration, a host machine connects to multiple NVDs. Figure 3 An example within the CSPI 300 is shown, where a host machine is connected to multiple NVDs. (Example follows) Figure 3 As shown, host machine 302 includes a network interface card (NIC) 304, which includes multiple ports 306 and 308. Host machine 300 is connected to a first NVD 310 via port 306 and link 320, and to a second NVD 312 via port 308 and link 322. Ports 306 and 308 may be Ethernet ports, and links 320 and 322 between host machine 302 and NVDs 310 and 312 may be Ethernet links. NVD 310 is further connected to a first TOR switch 314, and NVD 312 is connected to a second TOR switch 316. Links between NVDs 310 and 312 and TOR switches 314 and 316 may be Ethernet links. TOR switches 314 and 316 represent Layer 0 switching devices in a multi-layer physical network 318.

[0107] Figure 3 The arrangement depicted provides two separate physical network paths from physical switch network 318 to host machine 302: the first path passes through TOR switch 314 to NVD 310 and then to host machine 302, and the second path passes through TOR switch 316 to NVD 312 and then to host machine 302. These separate paths provide enhanced availability (referred to as high availability) for host machine 302. If one of the paths (e.g., a link in one of the paths breaks) or a device (e.g., a particular NVD is not running) experiences a problem, the other path can be used for communication with host machine 302.

[0108] exist Figure 3 In the configuration depicted, the host machine connects to two different NVDs using two different ports provided by the host machine's NIC. In other embodiments, the host machine may include multiple NICs that enable the host machine to connect to multiple NVDs.

[0109] Go back to reference Figure 2An NVD is a physical device or component that performs one or more network and / or storage virtualization functions. An NVD can be any device with one or more processing units (e.g., CPU, Network Processing Unit (NPU), FPGA, packet processing pipeline, etc.), memory (including cache), and ports. Various virtualization functions can be executed by software / firmware performed by one or more processing units of the NVD.

[0110] NVDs can be implemented in various different forms. For example, in some embodiments, an NVD is implemented as an interface card called a smartNIC or a smart NIC with an onboard embedded processor. A smartNIC is a device that is independent of the NIC on the host machine. Figure 2 In this context, NVD 210 and 212 can be implemented as smartNICs connected to host machine 202 and host machines 206 and 208, respectively.

[0111] However, smartNIC is just one example of an NVD implementation. Various other implementations are possible. For example, in some other implementations, the NVD, or one or more functions performed by the NVD, may be integrated into or performed by one or more host machines, one or more TOR switches, and other components of the CSPI 200. For instance, the NVD may be implemented within a host machine, where the functions performed by the NVD are performed by the host machine. As another example, the NVD may be part of a TOR switch, or the TOR switch may be configured to perform functions performed by the NVD, enabling the TOR switch to perform various complex packet transformations for public clouds. A TOR performing the functions of the NVD is sometimes referred to as a smart TOR. In other implementations that serve virtual machine (VM) instances rather than bare metal (BM) instances to customers, the functions performed by the NVD may be implemented within the hypervisor of the host machine. In some other implementations, some of the functions of the NVD may be offloaded to a centralized service running on a set of host machines.

[0112] In some embodiments, such as when implemented as Figure 2 As shown in the smartNIC diagram, the NVD can include multiple physical ports that enable it to connect to one or more host machines and one or more TOR switches. Ports on the NVD can be categorized as host-facing ports (also known as "south ports") or network-facing or TOR-facing ports (also known as "north ports"). The host-facing ports of the NVD are those used to connect the NVD to the host machine. Figure 2 Examples of host-facing ports include port 236 on the NVD 210 and ports 248 and 254 on the NVD 212. Network-facing ports on the NVD are used to connect the NVD to a TOR switch. Figure 2 Examples of network-facing ports include port 256 on the NVD 210 and port 258 on the NVD 212. Figure 2 As shown, NVD 210 is connected to TOR switch 214 via link 228, which extends from port 256 of NVD 210 to TOR switch 214. Similarly, NVD 212 is connected to TOR switch 216 via link 230, which extends from port 258 of NVD 212 to TOR switch 216.

[0113] The NVD receives packets and frames from the host machine (e.g., packets and frames generated by compute instances hosted by the host machine) via its host-facing port, and after performing the necessary packet processing, can forward the packets and frames to the TOR switch via its network-facing port. The NVD can also receive packets and frames from the TOR switch via its network-facing port, and after performing the necessary packet processing, can forward the packets and frames to the host machine via its host-facing port.

[0114] In some embodiments, there can be multiple ports and associated links between the NVD and TOR switches. These ports and links can be aggregated to form a link aggregation group (called a LAG) of multiple ports or links. Link aggregation allows multiple physical links between two endpoints (e.g., between the NVD and TOR switches) to be treated as a single logical link. All physical links in a given LAG can operate at the same speed in full-duplex mode. LAGs help increase the bandwidth and reliability of the connection between two endpoints. If one of the physical links in the LAG fails, traffic is dynamically and transparently reassigned to one of the other physical links in the LAG. Aggregated physical links deliver higher bandwidth than each individual link. Multiple ports associated with an LAG are treated as a single logical port. Traffic can be load-balanced across the multiple physical links of the LAG. One or more LAGs can be configured between two endpoints. These endpoints can be located between the NVD and TOR switches, between a host machine and the NVD, etc.

[0115] The NVD implements or performs network virtualization functions. These functions are performed by software / firmware executed by the NVD. Examples of network virtualization functions include, but are not limited to: packet encapsulation and decapsulation functions; functions for creating VCN networks; functions for implementing network policies, such as VCN security list (firewall) functionality; functions for facilitating packet routing and forwarding to and from compute instances within the VCN; and so on. In some embodiments, upon receiving a packet, the NVD is configured to perform a packet processing pipeline to process the packet and determine how to forward or route it. As part of this packet processing pipeline, the NVD may perform one or more virtual functions associated with the overlay network, such as performing VNICs associated with the CIS in the VCN, performing virtual routers (VRs) associated with the VCN, packet encapsulation and decapsulation to facilitate forwarding or routing in the virtual network, execution of certain gateways (e.g., local peer gateways), implementation of security lists, network security groups, Network Address Translation (NAT) functionality (e.g., per-host translation of public IPs to private IPs), throttling functions, and other functions.

[0116] In some embodiments, the packet processing data path in the NVD may include multiple packet pipelines, each consisting of a series of packet transformation stages. In some implementations, upon receiving a packet, the packet is parsed and classified into a single pipeline. The packet is then processed linearly, stage by stage, until the packet is either dropped or sent out through the NVD's interface. These stages provide basic functional packet processing building blocks (e.g., header verification, throttling, insertion of new Layer 2 headers, L4 firewall enforcement, VCN encapsulation / decapsulation, etc.) so that new pipelines can be built by combining existing stages, and new functionality can be added by creating new stages and inserting them into existing pipelines.

[0117] NVD can perform control plane and data plane functions corresponding to those of VCN's control plane and data plane. An example of the VCN control plane is also available in... Figure 17 , Figure 18 , Figure 19 and Figure 20 Depicted in (see references 1716, 1816, 1916, and 2016) and described below. An example of the VCN data plane is... Figure 17 , Figure 18 , Figure 19 and Figure 20The data plane is depicted (see reference numerals 1718, 1818, 1918, and 2018) and described below. Control plane functions include features for configuring how control data is forwarded on the network (e.g., setting routes and routing tables, configuring VNICs, etc.). In some embodiments, a VCN control plane is provided that centrally computes all overlay mappings to the baseboard and publishes them to the NVD and virtual network edge devices (such as various gateways, such as DRGs, SGWs, IGWs, etc.). Firewall rules can also be published using the same mechanism. In some embodiments, the NVD only receives mappings associated with that NVD. Data plane functions include the ability to actually route / forward packets based on the configuration set using the control plane. The VCN data plane is implemented by encapsulating client network packets before they traverse the baseboard network. Encapsulation / decapsulation functionality is implemented on the NVD. In some embodiments, the NVD is configured to intercept all network packets entering and leaving the host machine and perform network virtualization functions.

[0118] As indicated above, NVD performs various virtualization functions, including VNIC and VCN VR. NVD can execute VNICs associated with compute instances hosted on one or more host machines connected to the VNIC. For example, as... Figure 2 As depicted, NVD 210 performs the functionality of VNIC 276 associated with compute instance 268 hosted by host machine 202 connected to NVD 210. As another example, NVD 212 performs VNIC 280 associated with bare-metal compute instance 272 hosted by host machine 206, and VNIC 284 associated with compute instance 274 hosted by host machine 208. Host machines can host compute instances belonging to different VCNs (belonging to different customers), and NVDs connected to host machines can perform VNICs corresponding to compute instances (i.e., perform VNIC-related functionality).

[0119] NVD also executes a VCN virtual router corresponding to the VCN of the compute instance. For example, in Figure 2 In the embodiments depicted, NVD 210 executes VCN VR 277 corresponding to the VCN to which compute instance 268 belongs. NVD 212 executes one or more VCN VR 283 corresponding to one or more VCNs to which compute instances hosted by host machines 206 and 208 belong. In some embodiments, the VCN VR corresponding to a VCN is executed by all NVDs connected to host machines hosting at least one compute instance belonging to that VCN. If a host machine hosts compute instances belonging to different VCNs, then NVDs connected to that host machine can execute VCN VR corresponding to those different VCNs.

[0120] In addition to VNIC and VCN VR, NVD can execute various software (e.g., daemons) and include one or more hardware components that facilitate various network virtualization functions performed by NVD. For simplicity, these various components are grouped together as... Figure 2 The term "packet processing component" is shown in the diagram. For example, NVD 210 includes packet processing component 286 and NVD 212 includes packet processing component 288. For instance, a packet processing component for an NVD may include a packet processor configured to interact with the NVD's ports and hardware interfaces to monitor all packets received by and transmitted using the NVD and to store network information. This network information may include, for example, network flow information identifying different network flows handled by the NVD and per-flow information (e.g., per-flow statistics). In some embodiments, network flow information may be stored on a per-VNIC basis. The packet processor may perform per-packet manipulation and implement stateful NAT and L4 firewall (FW). As another example, a packet processing component may include a replication agent configured to replicate information stored by the NVD to one or more different replication target repositories. As yet another example, a packet processing component may include a logging agent configured to perform logging functions of the NVD. The packet processing component may also include software for monitoring the performance and health of the NVD and may also monitor the status and health of other components connected to the NVD.

[0121] Figure 1 The components of an example virtual or overlay network are shown, including a VCN, subnets within the VCN, compute instances deployed on the subnets, VNICs associated with the compute instances, VRs for the VCN, and a collection of gateways configured for the VCN. Figure 1 The overlay component described in the text can be made by Figure 2 One or more executions or hosts are described in the physical components. For example, a compute instance in a VCN can be executed or managed by... Figure 2 The VNIC described herein is executed or hosted by one or more host machines. For a compute instance hosted by a host machine, the VNIC associated with that compute instance is typically executed by an NVD connected to that host machine (i.e., VNIC functionality is provided by an NVD connected to that host machine). The VCN VR functionality for a VCN is executed by all NVDs connected to the host machine hosting or executing a compute instance as part of that VCN. The gateway associated with a VCN can be executed by one or more different types of NVDs. For example, some gateways can be executed by smartNICs, while others can be executed by one or more host machines or other implementations of NVDs.

[0122] As described above, compute instances in a client VCN can communicate with various endpoints, which can be within the same subnet as the source compute instance, in a different subnet but within the same VCN as the source compute instance, or communicate with endpoints located outside the VCN of the source compute instance. These communications are facilitated using VNICs, VCN VRs, and gateways associated with the VCNs.

[0123] For communication between two compute instances on the same subnet within a VCN, a VNIC associated with both the source and destination compute instances facilitates the communication. The source and destination compute instances can be hosted by the same host machine or different host machines. Packets originating from the source compute instance can be forwarded from the host machine hosting the source compute instance to an NVD connected to that host machine. On the NVD, packets are processed using a packet processing pipeline, which may include the execution of the VNIC associated with the source compute instance. Because the destination endpoint for the packet is within the same subnet, the execution of the VNIC associated with the source compute instance results in the packet being forwarded to the NVD executing the VNIC associated with the destination compute instance, whereupon the NVD processes the packet and forwards it to the destination compute instance. The VNIC associated with the source and destination compute instances can execute on the same NVD (e.g., when the source and destination compute instances are hosted by the same host machine) or on different NVDs (e.g., when the source and destination compute instances are hosted by different host machines connected to different NVDs). The VNIC can use a routing / forwarding table stored by the NVD to determine the next hop for the packet.

[0124] For packets destined for endpoints in different subnets within the same VCN, the packet originating from the source compute instance is forwarded from the host machine hosting the source compute instance to the NVD connected to that host machine. On the NVD, the packet is processed using a packet processing pipeline, which may include the execution of one or more VNICs and the VR associated with the VCN. For example, as part of the packet processing pipeline, the NVD executes or invokes functionality (also known as executing the VNIC) associated with the source compute instance. Functionality executed by the VNIC may include viewing the VLAN tag on the packet. Since the packet's destination is outside the subnet, the VCN VR functionality is then invoked and executed by the NVD. The VCN VR then routes the packet to the NVD executing the VNIC associated with the destination compute instance. The VNIC associated with the destination compute instance then processes the packet and forwards it to the destination compute instance. The VNICs associated with the source and destination compute instances may execute on the same NVD (e.g., when the source and destination compute instances are hosted by the same host machine) or on different NVDs (e.g., when the source and destination compute instances are hosted by different host machines connected to different NVDs).

[0125] If the destination of the packet is outside the VCN of the source compute instance, the packet originating from the source compute instance is forwarded from the host machine hosting the source compute instance to the NVD connected to that host machine. The NVD executes the VNIC associated with the source compute instance. Since the destination endpoint of the packet is outside the VCN, the packet is subsequently processed by the VCN VR used by that VCN. The NVD invokes the VCN VR functionality, which results in the packet being forwarded to the NVD executing the appropriate gateway associated with the VCN. For example, if the destination is an endpoint within a customer's on-premises network, the packet may be forwarded by the VCN VR to the NVD executing the DRG gateway configured for the VCN. The VCN VR may execute on the same NVD as the NVD executing the VNIC associated with the source compute instance, or it may be executed by a different NVD. The gateway may be executed by the NVD, which may be a smartNIC, a host machine, or another NVD implementation. The packet is then processed by the gateway and forwarded to the next hop, which facilitates the forwarding of the packet to its intended destination endpoint. For example, in Figure 2 In the embodiment depicted, packets originating from compute instance 268 can be transmitted from host machine 202 to NVD 210 via link 220 (using NIC 232). On NVD 210, VNIC 276 is invoked because it is the VNIC associated with the source compute instance 268. VNIC 276 is configured to examine the information encapsulated in the packet and determine the next hop for forwarding the packet, with the aim of facilitating the transmission of the packet to its intended destination endpoint, and then forwarding the packet to the determined next hop.

[0126] Compute instances deployed on a VCN can communicate with a variety of endpoints. These endpoints can include endpoints hosted by CSPI 200 and endpoints outside of CSPI 200. Endpoints hosted by CSPI 200 can include instances within the same VCN or other VCNs, which can be the customer's VCN or a VCN not belonging to the customer. Communication between endpoints hosted by CSPI 200 can be performed over physical network 218. Compute instances can also communicate with endpoints not hosted by CSPI 200 or outside of CSPI 200. Examples of these endpoints include endpoints within the customer's on-premises network or data center, or public endpoints accessible via public networks such as the Internet. Communication with endpoints outside of CSPI 200 can use various communication protocols over public networks (e.g., the Internet). Figure 2 (not shown in the image) or a dedicated network ( Figure 2 (Not shown in the image) to execute.

[0127] Figure 2The architecture of the CSPI 200 depicted herein is merely an example and is not intended to be limiting. Variations, alternatives, and modifications are possible in alternative embodiments. For example, in some implementations, the CSPI 200 may have a more advanced architecture than... Figure 2 The systems or components shown may include more or fewer systems or components, and may combine two or more systems, or may have different system configurations or arrangements. Figure 2 The systems, subsystems, and other components described herein may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective system, using hardware, or a combination thereof. The software may be stored on a non-transitory storage medium (e.g., a memory device).

[0128] Figure 4 The diagram depicts a connection between a host machine and an NVD, according to certain embodiments, for providing I / O virtualization to support multi-tenancy. For example... Figure 4 As depicted, host machine 402 executes a hypervisor 404 that provides a virtualized environment. Host machine 402 executes two virtual machine instances, VM1 406 belonging to customer / tenant #1 and VM2 408 belonging to customer / tenant #2. Host machine 402 includes a physical NIC 410 connected to NVD 412 via link 414. Each compute instance is attached to a VNIC executed by NVD 412. Figure 4 In one embodiment, VM1 406 is attached to VNIC-VM1 420 and VM2 408 is attached to VNIC-VM2 422.

[0129] like Figure 4 As shown, NIC 410 includes two logical NICs, logical NIC A 416 and logical NIC B 418. Each virtual machine is attached to its own logical NIC and configured to work with its own logical NIC. For example, VM1 406 is attached to logical NIC A 416 and VM2 408 is attached to logical NIC B 418. Although host machine 402 includes only one physical NIC 410 shared by multiple tenants, due to the logical NICs, each tenant's virtual machines believe they have their own host machine and network interface card.

[0130] In some embodiments, each logical NIC is assigned its own VLAN ID. Therefore, a specific VLAN ID is assigned to logical NIC A 416 for tenant #1, and a separate VLAN ID is assigned to logical NIC B 418 for tenant #2. When a packet is transmitted from VM1 406, the hypervisor appends a tag assigned to tenant #1 to the packet, and the packet is then transmitted from host machine 402 to NVD 412 via link 414. Similarly, when a packet is transmitted from VM2 408, the hypervisor appends a tag assigned to tenant #2 to the packet, and the packet is then transmitted from host machine 402 to NVD 412 via link 414. Thus, a packet 424 transmitted from host machine 402 to NVD 412 has an associated tag 426 identifying the specific tenant and the associated VM. On the NVD, for a packet 424 received from the host machine 402, the tag 426 associated with the packet is used to determine whether the packet is processed by VNIC-VM1 420 or VNIC-VM2 422. The packet is then processed by the corresponding VNIC. Figure 4 The configuration described in [the document] enables each tenant's compute instance to believe that they own their own host machine and NIC. Figure 4 The setup described in [the document] provides I / O virtualization to support multi-tenancy.

[0131] Figure 5 A simplified block diagram of a physical network 500 according to certain embodiments is depicted. Figure 5 The embodiments depicted are structured as Clos networks. Clos networks are a specific type of network topology designed to provide connectivity redundancy while maintaining high bandwidth and maximum resource utilization. Clos networks are non-blocking, multi-stage or multi-layer switching networks, where the number of stages or layers can be two, three, four, five, etc. Figure 5 The embodiment depicted is a Layer 3 network, including Layer 1, Layer 2, and Layer 3. TOR switch 504 represents a Layer 0 switch in a Clos network. One or more NVDs are connected to the TOR switch. Layer 0 switches are also referred to as edge devices of the physical network. Layer 0 switches are connected to Layer 1 switches, also known as leaf switches. Figure 5In the embodiments depicted, a set of "n" Layer 0 TOR switches is connected to a set of "n" Layer 1 switches, forming a pod. Each Layer 0 switch in the pod is interconnected to all Layer 1 switches in that pod, but there is no switch connectivity between pods. In some implementations, two pods are referred to as blocks. Each block is served by or connected to a set of "n" Layer 2 switches (sometimes called backbone switches). There can be several blocks in the physical network topology. The Layer 2 switches are then connected to "n" Layer 3 switches (sometimes called super backbone switches). Packet communication over the physical network 500 is typically performed using one or more Layer 3 communication protocols. Typically, all layers of the physical network (except the TOR layer) are n-way redundant, thus allowing high availability. Policies can be specified for pods and blocks to control the visibility of switches to each other in the physical network, thereby enabling the scaling of the physical network.

[0132] A key characteristic of Clos networks is that the maximum number of hops from one Layer 0 switch to another (or from an NVD connected to a Layer 0 switch to another NVD connected to a Layer 0 switch) is fixed. For example, in a Layer 3 Clos network, a packet takes a maximum of seven hops to reach another NVD, where the source and destination NVDs are connected to the leaf layers of the Clos network. Similarly, in a Layer 4 Clos network, a packet takes a maximum of nine hops to reach another NVD, where the source and destination NVDs are connected to the leaf layers of the Clos network. Therefore, the Clos network architecture maintains consistent latency throughout the network, which is important for communication within and between data centers. Clos topologies are horizontally scalable and cost-effective. Network bandwidth / throughput capacity can be easily increased by adding more switches at each layer (e.g., more leaf switches and backbone switches) and by increasing the number of links between switches in adjacent layers.

[0133] In some embodiments, each resource within the CSPI is assigned a unique identifier called a Cloud Identifier (CID). This identifier is included as part of the resource's information and can be used to manage the resource, for example, via a console or API. An example syntax for a CID is:

[0134] ocid1.<RESOURCE TYPE> . <realm>[REGION][FUTURE USE].<UNIQUE ID>

[0135] in,

[0136] ocid1: A text string indicating the version of the CID;

[0137] resource type: The type of resource (e.g., instance, volume, VCN, subnet, user, group, etc.);

[0138] realm: The realm where the resource resides. Example values ​​are "c1" for the commercial realm, "c2" for the government cloud realm, or "c3" for the federal government cloud realm, etc. Each realm can have its own domain name;

[0139] region: The region where the resource is located. This section may be empty if the region is not applicable to the resource.

[0140] future use: to be reserved for future use.

[0141] Unique ID: The unique part of the ID. The format may vary depending on the type of resource or service.

[0142] B- Example Layer 2 VLAN Architecture

[0143] This section describes techniques for providing Layer 2 networking functionality in a virtualized cloud environment. Layer 2 functionality is provided in conjunction with and complements Layer 3 networking functionality provided by the virtualized cloud environment. In some embodiments, virtual Layer 2 and Layer 3 functionality are provided by Oracle Cloud Infrastructure (OCI) provided by Oracle Corporation.

[0144] After introducing Layer 2 network functionality, this section describes a Layer 2 implementation of VLANs. Subsequently, a description of providing Layer 2 VLAN services is provided, including information about Layer 2 VLANs, which may include Media Access Control (MAC) address forwarding tables and / or Layer 2 switch statistics.

[0145] introduce

[0146] The number of enterprise customers transitioning their on-premises applications to cloud environments provided by cloud service providers (CSPs) continues to grow rapidly. However, many of these customers quickly realize that the transition to a cloud environment can be very bumpy, requiring them to rebuild and redesign their existing applications to make them work in the cloud environment. This is because applications written for on-premises environments often depend on the physical network's monitoring, availability, and scalability features. Therefore, these on-premises applications need to be rebuilt and redesigned before they can work in a cloud environment.

[0147] There are several reasons why on-premises applications cannot easily transition to cloud environments. One major reason is that current cloud virtual networks operate at Layer 3 of the OSI model, such as at the IP layer, and do not provide the Layer 2 capabilities required by applications. Layer 3-based routing or forwarding involves determining where a packet should be sent (e.g., to which customer instance) based on information contained in the Layer 3 header of the packet (e.g., the destination IP address contained in the Layer 3 header of the packet). To facilitate this, the location of IP addresses in the virtualized cloud network is determined through a centralized control and orchestration system or controller. These can include, for example, IP addresses associated with customer entities or resources within the virtualized cloud environment.

[0148] Many customers run applications in their on-premises environments that have stringent requirements for Layer 2 networking capabilities, issues that current cloud offerings and IaaS providers do not currently address. For example, traffic in current cloud offerings is routed using Layer 3 protocols with Layer 3 headers and does not support the Layer 2 features required by applications. These Layer 2 features may include Address Resolution Protocol (ARP) processing, Media Access Control (MAC) address learning and Layer 2 broadcast capabilities, Layer 2 (MAC-based) forwarding, Layer 2 networking constructs, and others. By providing virtualized Layer 2 networking functionality within a virtualized cloud network, as described in this disclosure, customers can now seamlessly migrate their legacy applications to cloud environments without any substantial rebuilding or redesign. For example, the virtualized Layer 2 networking capabilities described herein enable such applications (e.g., VMware vSphere, vCenter, vSAN, and NSX-T components) to communicate at Layer 2 as they would in an on-premises environment. These applications can run the same versions and configurations in the public cloud, allowing customers to utilize their legacy on-premises applications, including existing knowledge, tools, and processes associated with those applications. Customers are also able to access native cloud services from their applications (e.g., using VMware software-defined data centers (SDDC)).

[0149] As another example, several legacy on-premises applications (e.g., enterprise clustering software applications, network virtual appliances) require Layer 2 broadcast support for failover. Example applications include Fortinet FortiGate, IBM QRadar, Palo Alto firewall, Cisco ASA, Juniper SRX, and Oracle RAC (True Application Clustering). By providing virtualized Layer 2 networking in a virtualized public cloud as described in this disclosure, these applications can now run unchanged in a virtualized public cloud environment. Virtualized Layer 2 networking functionality equivalent to on-premises deployment is provided as described herein. The virtualized Layer 2 networking functionality described in this disclosure supports traditional Layer 2 networking. This includes support for customer-defined VLANs and unicast, broadcast, and multicast Layer 2 traffic functionality. Layer 2-based packet routing and forwarding involves routing or forwarding packets using Layer 2 protocols and information contained in the Layer 2 header of the packet, such as the destination MAC address contained in the Layer 2 header. Protocols used by enterprise applications (such as clustering software applications), such as ARP, gratuitous address resolution protocol (GARP), and reverse address resolution protocol (RARP), can now also work in cloud environments.

[0150] Traditional virtualized cloud infrastructures support Layer 3 networking but not Layer 2 networking for several reasons. Layer 2 networks typically cannot scale as quickly as Layer 3 networks. Layer 2 network control protocols lack the complexity expected of scaling. For example, Layer 3 networks do not have to worry about packet looping, a problem that Layer 2 networks must solve. IP packets (i.e., Layer 3 packets) have a Time-to-Live (TTL) concept, while Layer 2 packets do not. IP addresses contained in Layer 3 packets have topological concepts such as subnets and CIDR ranges, while Layer 2 addresses (e.g., MAC addresses) do not. Layer 3 IP networks have built-in tools that facilitate troubleshooting, such as ping and traceroute for finding path information. Such tools are unavailable for Layer 2. Layer 3 networks support multipathing, which is unavailable in Layer 2 networks. Due to the lack of sophisticated control protocols specifically designed for exchanging information between entities in the network (e.g., Border Gateway Protocol (BGP) and Open Shortest Path First (OSPF)), Layer 2 networks must rely on broadcast and multicast to learn the network sequentially, which negatively impacts network performance. As networks change, the learning process used for Layer 2 must be repeated, while it is not required for Layer 3. For these and other reasons, cloud IaaS service providers prefer to provide infrastructure that operates at Layer 3 rather than Layer 2.

[0151] However, despite several drawbacks, many on-premises applications still require Layer 2 functionality. For example, consider a virtualized cloud configuration where a customer (customer 1) has two instances in a virtual network "V," instance A with IP1 and instance B with IP2. These instances can be compute instances (e.g., bare metal, virtual machines, or containers) or service instances (such as load balancers, NFS mount points, or other service instances). Virtual network V is a unique address space isolated from other virtual networks and the underlying physical network. This isolation can be achieved using various techniques, including packet encapsulation or NAT. For this reason, the IP address of the instance in the virtual network used by the customer is different from the address in the physical network hosting it. A centralized SDN (Software-Defined Networking) control plane is provided, which knows the physical IP and virtual interface of all virtual IP addresses. When a packet is sent from instance A to its destination IP2 in virtual network V, the virtual network SDN stack needs to know the location of IP2. It must know this in advance so that it can send the packet to the IP in the physical network hosting the virtual IP address IP2 used for V. The location of virtual IP addresses can be modified in the cloud, thereby altering the relationship between physical IP addresses and virtual IP addresses. Whenever a virtual IP address needs to be moved (e.g., moving an IP address associated with a virtual machine to another virtual machine or migrating a virtual machine to a new physical host), an API call must be made to the SDN control plane to inform the controller that the IP is moving so it can update all participants in the SDN stack, including the packet processor (data plane). However, some application classes do not make such API calls. Examples include various on-premises applications and applications provided by various virtualization software vendors (such as VMware). The value of facilitating virtual Layer 2 networking in a virtualized cloud environment lies in enabling applications not programmed to make such API calls or applications that rely on other Layer 2 networking features, such as supporting non-IP Layer 3 and MAC learning.

[0152] Virtual Layer 2 networks create broadcast domains, where learning is performed by members of the broadcast domain. Within a virtual Layer 2 domain, any IP address can be on any MAC address of any host within that domain, and the system will learn to use standard Layer 2 networking protocols. These networking primitives are virtualized without the central controller explicitly informing the MAC and IP addresses of their locations within the virtual Layer 2 network. This enables applications requiring low-latency failover, applications supporting broadcast or multicast protocols to multiple nodes, and legacy applications that don't know how to make API calls to the SDN control plane or API endpoints to determine IP and MAC address locations. Therefore, Layer 2 networking capabilities are needed in virtualized cloud environments to support functionality not available at the IP Layer 3 level.

[0153] Another technical advantage of providing virtual Layer 2 in a virtualized cloud environment is that it enables support for a variety of different Layer 3 protocols (such as IPv4 and IPv6), including non-IP protocols. For example, various non-IP protocols, such as IPX and AppleTalk, can be supported. Because existing cloud IaaS providers do not provide Layer 2 functionality in their virtualized cloud networks, they cannot support these non-IP protocols. By providing the Layer 2 networking functionality described in this disclosure, support can be provided for protocols at Layer 3, as well as for applications that require and rely on the availability of Layer 2 level functionality.

[0154] Using the techniques described in this disclosure, both Layer 3 and Layer 2 functionality are provided in a virtualized cloud infrastructure. As previously mentioned, Layer 3-based networking offers certain efficiencies, particularly suited for scalability, which Layer 2 networking does not provide. Providing Layer 2 functionality in addition to Layer 3 functionality allows full utilization of such efficiencies offered by Layer 3 (e.g., providing a more scalable solution) while providing Layer 2 functionality in a more scalable manner. For example, virtualized Layer 3 avoids the use of broadcasting for learning purposes. By providing Layer 3 for increased efficiency, while providing virtualized Layer 2 to enable applications that require it and those that cannot function without Layer 2 functionality, and supporting non-IP protocols, etc., customers are provided with complete flexibility in a virtualized cloud environment.

[0155] Customers typically have hybrid environments where Layer 2 and Layer 3 environments coexist, and the virtualized cloud environment can now support both. Customers can have Layer 3 networks such as subnets and / or Layer 2 networks such as VLANs, and these two environments can communicate with each other within the virtualized cloud environment.

[0156] Virtualized cloud environments also need to support multi-tenancy. Multi-tenancy makes it technically difficult and complex to provision both Layer 3 and Layer 2 functionality within the same virtualized cloud environment. For example, a Layer 2 broadcast domain must be managed across many different customers in the cloud provider's infrastructure. The embodiments described in this disclosure overcome these technical problems.

[0157] For virtualization providers (e.g., VMware), virtualized Layer 2 networks that emulate physical Layer 2 networks allow workloads to run without modification. Applications provided by such providers can then run on virtualized Layer 2 networks provided by cloud infrastructure. For example, such applications may include a collection of instances that require a Layer 2 network to run. When customers want to elevate such applications from their on-premises environments to virtualized cloud environments, they cannot simply acquire the application and run it in the cloud because those applications rely on the underlying Layer 2 network (e.g., Layer 2 network features are used to perform virtual machine migrations or move the location of MAC and IP addresses), which is not currently provided by virtualized cloud providers. For these reasons, such applications cannot run natively in virtualized cloud environments. Using the techniques described herein, cloud providers provide virtualized Layer 2 networks in addition to providing virtualized Layer 3 networks. Now, such application stacks can run in cloud environments without modification, and nested virtualizations can run in cloud environments. Customers can now run and manage their own Layer 2 applications in the cloud. Application providers do not need to make any changes to their software to facilitate this. Such legacy applications or workloads (e.g., legacy load balancers, legacy applications, KVM, OpenStack, clustering software) can now run unchanged in a virtualized cloud environment.

[0158] By providing the Layer 2 functionality of virtualization as described in this article, virtualized cloud environments can now support a wide range of Layer 3 protocols, including non-IP protocols. Taking Ethernet as an example, various different EtherTypes (a field in the Layer 2 header that tells the type of Layer 3 packet being sent; it tells what protocol is expected at Layer 3) can be supported, including a variety of non-IP protocols. EtherType is a two-octet field in an Ethernet frame. It is used to indicate which protocol is encapsulated in the frame's payload and is used by the data link layer at the receiving end to determine how to process the payload. EtherType also serves as the basis for 802.1Q VLAN tagging, encapsulating packets from VLANs for multiplexing with other VLAN traffic over Ethernet relay. Examples of EtherTypes include IPv4, IPv6, Address Resolution Protocol (ARP), AppleTalk, IPX, etc. Cloud networks that support Layer 2 protocols can support any protocol at Layer 3. Similarly, when cloud infrastructure provides support for Layer 3 protocols, it can support a variety of protocols at Layer 4, such as TCP, UDP, ICMP, etc. When virtualization is provided at Layer 3, the network can be unaffected by Layer 4 protocols. Similarly, when virtualization is provided at Layer 2, the network can be unaffected by Layer 3 protocols. This technology can be extended to support any Layer 2 network type, including FDDI, Infiniband, etc.

[0159] Therefore, many applications written for physical networks (especially those working with clusters of computer nodes sharing a broadcast domain) utilize Layer 2 features not supported by L3 virtual networks. The following six examples highlight the complexities that can result from not providing Layer 2 networking capabilities:

[0160] (1) Assigning MAC and IP addresses without prior API calls. Network appliances and management programs (such as VMware) are not built for cloud virtual networks. They assume they can use MAC addresses as long as they are unique, and either obtain dynamic addresses from a DHCP server or use any IP address assigned to the cluster. Often there is no mechanism to configure them to inform the control plane about the assignment of these Layer 2 and Layer 3 addresses. If the location of the MAC and IP addresses is unknown, then the Layer 3 virtual network does not know where to send traffic.

[0161] (2) Low-latency reallocation of MAC and IP addresses for high availability and live migration. Many on-premises applications use ARP to reassign IP and MAC addresses for high availability—when an instance in a cluster or HA pair stops responding, the new active instance sends a gratuitous ARP (GARP) to reassign the service IP to its MAC address or a reverse ARP (RARP) to reassign the service MAC address to its interface. This is also important when live migrating instances on a hypervisor: the new host must send RARP after a guest migration to redirect guest traffic to the new host. The assignment not only requires no API calls but also extremely low latency (sub-milliseconds). This cannot be accomplished via HTTPS calls to REST endpoints.

[0162] (3) Interface multiplexing via MAC addresses. When a hypervisor hosts multiple virtual machines on a single host, all of these virtual machines are on the same network, and guest interfaces are distinguished by their MAC addresses. This requires support for multiple MAC addresses on the same virtual interface.

[0163] (4) VLAN support. A single physical virtual machine host will need to be located on multiple broadcast domains, as indicated by VLAN tags. For example, VMware ESX uses VLANs for traffic separation (e.g., a guest virtual machine can communicate on one VLAN, store on another VLAN, and host the virtual machine on yet another VLAN).

[0164] (5) Use of broadcast and multicast traffic. ARP requires L2 broadcast, and there are examples of on-premises applications using broadcast and multicast traffic for cluster and HA applications.

[0165] (6) Support for non-IP traffic. Since L3 networks require IPv4 or IPv6 headers for communication, using any L3 protocol other than IP will not work. L2 virtualization means that the network within a VLAN can be independent of L3 protocols—the L3 header can be IPv4, IPv6, IPX or anything else—or even not exist at all.

[0166] Layer 2 VLAN Implementation

[0167] As disclosed herein, a Layer 2 (L2) network can be created within a cloud network. This virtual L2 network includes one or more Layer 2 virtual networks, such as virtualized L2 VLANs, referred to herein as VLANs. Each VLAN may include multiple compute instances, each of which may be associated with at least one L2 virtual network interface (e.g., an L2 VNIC) and an L2 virtual switch. In some embodiments, each pair of L2 virtual network interfaces and L2 virtual switches is hosted on an NVD. An NVD may host multiple such pairs, each associated with a different compute instance. A set of L2 virtual switches represents a simulated single L2 switch for a VLAN. An L2 virtual network interface represents a set of L2 ports on a simulated single L2 switch. VLANs may be connected to other VLANs, Layer 3 (L3) networks, on-premises networks, and / or other networks via VLAN Switching and Routing Service (VSRS) (also referred to herein as a Real Virtual Router (RVR) or L2 VSRS). An example of this architecture is described below.

[0168] Now for reference Figure 6 The diagram illustrates one embodiment of a computing network. VCN 602 resides in CSPI 601. VCN 602 includes multiple gateways connecting VCN 602 to other networks. These gateways include DRG 604, which can connect VCN 602 to, for example, an on-premises network (such as on-premises data center 606). Gateways may also include gateway 600, which may include, for example, an LPG for connecting VCN 602 to another VCN, and / or an IGW and / or NAT gateway for connecting VCN 602 to the Internet. Gateways of VCN 602 may also include service gateway 610, which can connect VCN 602 to service network 612. Service network 612 may include one or more databases and / or repositories, including, for example, autonomous database 614 and / or object repository 616. Service network may include a conceptual network that includes an aggregation of IP ranges, such as public IP ranges. In some embodiments, these IP ranges may cover some or all of the public services provided by the CSPI 601 provider. For example, these services can be accessed through an Internet gateway or a NAT gateway. In some embodiments, the service network provides a way for services within the service network to be accessed from a local area via a dedicated gateway (service gateway) for that purpose. In some embodiments, the backends of these services may be implemented, for example, in their own private network. In some embodiments, service network 612 may include additional databases.

[0169] VCN 602 can include multiple virtual networks. Each of these networks can include one or more compute instances that can communicate within their respective networks, between networks, or outside of VCN 602. One of the virtual networks of VCN 602 is L3 subnet 620. L3 subnet 620 is a unit subdivision of a configuration created within VCN 602. Subnet 620 can include a virtual Layer 3 network within the virtualized cloud environment of VCN 602, which is hosted on the underlying physical network of CPSI 601. Although Figure 6 A single subnet 620 is depicted, but VCN 602 may have one or more subnets. Each subnet within VCN 602 may be associated with a contiguous range of IP addresses (e.g., 10.0.0.0 / 24 and 10.0.1.0 / 24) that does not overlap with other subnets within that VCN and represents a subset of the address space within that VCN's address space. In some embodiments, this IP address space may be isolated from the address space associated with CPSI 601.

[0170] Subnet 620 includes one or more computing instances, and specifically includes a first computing instance 622-A and a second computing instance 622-B. Computing instances 622-A and 622-B can communicate with each other within subnet 620, or they can communicate with other instances, devices, and / or networks outside subnet 620. Communication outside subnet 620 is enabled by a virtual router (VR) 624. VR 624 enables communication between subnet 620 and other networks of VCN 602. For subnet 620, VR 624 represents a logical gateway that enables subnet 620 (e.g., computing instances 622-A and 622-B) to communicate with endpoints on other networks within VCN 602 and with other endpoints outside VCN 602.

[0171] The VCN 602 may also include additional networks, and specifically may include one or more L2 VLANs (referred to herein as VLANs), which are examples of virtual L2 networks. These one or more VLANs may each comprise a virtual Layer 2 network located in the cloud environment of the VCN 602 and / or hosted by the underlying physical network of the CPSI 601. Figure 6 In this embodiment, VCN 602 includes VLAN A 630 and VLAN B 640. Each VLAN 630, 640 within VCN 602 may be associated with a contiguous range of IP addresses (e.g., 10.0.0.0 / 24 and 10.0.1.0 / 24) that do not overlap with other networks within the VCN (such as other subnets or VLANs within the VCN) and represent a subset of the address space within the VCN's address space. In some embodiments, this IP address space of the VLAN may be isolated from the address space associated with CPSI 601. Each of VLANs 630, 640 may include one or more compute instances, and specifically, VLAN A 630 may include, for example, a first compute instance 632-A and a second compute instance 632-B. In some embodiments, VLAN A 630 may include additional compute instances. VLAN B 640 may include, for example, a first compute instance 642-A and a second compute instance 642-B. Each of compute instances 632-A, 632-B, 642-A, and 642-B can have both an IP address and a MAC address. These addresses can be assigned or generated in any desired manner. In some embodiments, these addresses can be within the CIDR of the compute instance's VLAN, and in some embodiments, these addresses can be any addresses. In embodiments where the compute instance in the VLAN communicates with endpoints outside the VLAN, one or both of these addresses come from the VLAN CIDR, while when all communication is within the VLAN, these addresses are not limited to addresses within the VLAN CIDR. Unlike networks where addresses are assigned by the control plane, the IP and / or MAC addresses of compute instances within a VLAN can be assigned by the users / clients of that VLAN, and these IP and / or MAC addresses can then be discovered and / or learned by the compute instances within the VLAN according to the learning process discussed below.

[0172] Each VLAN may include a VLAN Switching and Routing Service (VSRS), and specifically, VLAN A 630 includes VSRS A 634 and VLAN B 640 includes VSRS B 644. Each VSRS 634, 644 participates in Layer 2 switching and local learning within the VLAN and also performs all necessary Layer 3 network functions, including ARP, NDP, and routing. VSRS performs ARP (a Layer 2 protocol) because VSRS must map IPs to MAC addresses.

[0173] Within these cloud-based VLANs, each virtual interface or virtual gateway can be associated with one or more Media Access Control (MAC) addresses, which can be virtual MAC addresses. Within a VLAN, one or more compute instances 632-A, 632-B, 642-A, 642-B (e.g., bare metal, VMs, or containers, and / or one or more service instances) can communicate directly with each other via virtual switches. Communication outside the VLAN, such as with other VLANs or with L3 networks, is enabled via VSRS 634, 644. VSRS 634, 644 are distributed services that provide Layer 3 functionality, such as IP routing, to the VLAN network. In some embodiments, VSRS 634, 644 are horizontally scalable, highly available routing services that can reside at the intersection of IP and L2 networks and participate in IP routing and L2 learning within the cloud-based L2 domain.

[0174] VSRS 634 and 644 can be distributed across multiple nodes within an infrastructure, and their functionality can be scalable, particularly horizontally scalable. In some embodiments, each node implementing the functionality of VSRS 634 and 644 shares and replicates the functionality of the router and / or switch with each other. Furthermore, these nodes can present themselves as a single VSRS 634 and 644 to all instances in VLANs 630 and 640. VSRS 634 and 644 can be implemented on any virtualization device within the CSPI 601, specifically within a virtual network. Therefore, in some embodiments, VSRS 634 and 644 can be implemented on any virtual network virtualization device, including NICs, SmartNICs, switches, intelligent switches, or general-purpose computing hosts.

[0175] VSRS 634 and 644 can be services residing on one or more hardware nodes supporting a cloud network. These hardware nodes may be, for example, one or more servers, such as one or more x86 servers, or one or more networking devices supporting the cloud network, such as one or more NICs, and specifically one or more SmartNICs. In some embodiments, VSRS 634 and 644 can be implemented on a server cluster. Therefore, VSRS 634 and 644 can be services distributed across a cluster of nodes, which can be a centrally managed cluster or distributed at the edge of virtual networking executors that participate in and share L2 and L3 learning and evaluate routing and security policies. In some embodiments, each VSRS instance can update other VSRS instances with new mapping information, as this new mapping information is learned by the VSRS instance. For example, when a VSRS instance learns the IP, interface, and / or MAC mappings of one or more CIs in its VLAN, the VSRS instance can provide this updated information to other VSRS instances within the VCN. Through this cross-update, the VSRS instance associated with the first VLAN can know the mappings for CIs in other VLANs (in some embodiments, for CIs in other VLANs within VCN 602), including IP, interface, and / or MAC mappings. These updates can be significantly accelerated when the VSRS resides on a server cluster and / or is distributed across nodes.

[0176] In some embodiments, VSRS 634, 644 may also host one or more higher-level services necessary for networking, including but not limited to: DHCP relay; DHCP (managed); DHCPv6; neighbor discovery protocols (such as IPv6 neighbor discovery protocol); DNS; managed DNSv6; SLAAC for IPv6; NTP; metadata services; and blockstore mount points. In some embodiments, VSRS may support one or more Network Address Translation (NAT) functions for translation between network address spaces. In some embodiments, VSRS may incorporate anti-spoofing, anti-MAC spoofing, ARP cache poisoning protection for IPv4, IPv6 Route Advertisement (RA) protection, DHCP protection, packet filtering using Access Control Lists (ACLs); and / or reverse path forwarding checks. Functions that VSRS can implement include, for example, ARP, GARP, packet filtering (ACLs), DHCP relay, and / or IP routing protocols. For example, VSRS 634 and 644 can learn MAC addresses, invalidate expired MAC addresses, handle MAC address migration, vet MAC address information, handle MAC information flooding, handle storm control, prevent loops, multicast via Layer 2 protocols such as IGMP in the cloud, collect statistics including logs, collect statistics using SNMP, monitor, and / or collect and use statistics for broadcast, total traffic, bits, cross-tree groups, etc.

[0177] Within a virtual network, VSRS 634 and 644 can manifest as distinct instantiations. In some embodiments, each of these VSRS instantiations can be associated with VLAN 630 and 640, and in some embodiments, each VLAN 630 and 640 can have an instantiation of VSRS 634 and 644. In some embodiments, each instantiation of VSRS 634 and 644 can have one or more unique tables corresponding to the VLAN 630 and 640 associated with each instantiation of VSRS 634 and 644. Each instantiation of VSRS 634 and 644 can generate and / or orchestrate a unique table associated with that instantiation of VSRS 634 and 644. Therefore, while a single service can provide VSRS 634 and 644 functionality for one or more cloud networks, individual instantiations of VSRS 634 and 644 within a cloud network can have unique Layer 2 and Layer 3 forwarding tables, while multiple such customer networks can have overlapping Layer 2 and Layer 3 forwarding tables.

[0178] In some embodiments, VSRS 634, 644 can support conflicting VLANs and IP spaces across multiple tenants. This can include multiple tenants on the same VSRS 634, 644. In some embodiments, some or all of these tenants may choose to use some or all of the following: the same IP address space, the same MAC space, and the same VLAN space. This provides users with great flexibility in address selection. In some embodiments, this multi-tenancy is supported by providing each tenant with a different virtual network, which is a private network within a cloud network. Each virtual network is assigned a unique identifier. Similarly, in some embodiments, each host may have a unique identifier, and / or each virtual interface or virtual gateway may have a unique identifier. In some embodiments, these unique identifiers, and specifically the unique identifiers of the tenant's virtual network, may be encoded in each communication. By providing a unique identifier for each virtual network and including it in communications, a single instantiation of VSRS 634, 644 can serve multiple tenants with overlapping addresses and / or namespaces.

[0179] VSRS 634 and 644 can perform these switching and / or routing functions to facilitate and / or enable the creation and / or communication with L2 networks within VLANs 630 and 640. These VLANs 630 and 640 can be found within a cloud computing environment, and more specifically, within a virtual network within that cloud computing environment.

[0180] For example, each of VLANs 630 and 640 includes multiple compute instances 632-A, 632-B, 642-A, and 642-B. VSRS 634 and 644 enable communication between compute instances in one VLAN 630 or 640 and compute instances in another VLAN 630 or 640 or subnet 620. In some embodiments, VSRS 634 and 644 enable communication between compute instances in one VLAN 630 or 640 and another VCN, or another network outside that VCN (including the Internet, on-premises data centers, etc.). In such embodiments, for example, a compute instance (such as compute instance 632-A) may send communication to an endpoint outside the VLAN (in this example, an endpoint outside VLAN A 630). Compute instance (632-A) may send communication to VSRS A 634, which may direct the communication to routers 624 and 644 or gateways 604, 608, and 610 coupled to the desired endpoint. Routers 624, 644 or gateways 604, 608, 610 that are coupled to the desired endpoint can receive communication from the computing instance (632-A) and can direct the communication to the desired endpoint.

[0181] Now for reference Figure 7 The diagram illustrates the logical and hardware schematics of VLAN 700. As can be seen, VLAN 700 includes multiple endpoints, specifically multiple compute instances and VSRS. Multiple compute instances (CIs) are instantiated on one or more host machines. In some embodiments, this can be a one-to-one relationship, such that each CI is instantiated on a unique host machine, and / or in some embodiments, this can be a many-to-one relationship, such that multiple CIs are instantiated on a single shared host machine. In various embodiments, CIs can be Layer 2 CIs by being configured to communicate with each other using the L2 protocol. Figure 7 This describes a scenario where some CIs are instantiated on a single host machine, and some CIs share a common host machine. For example... Figure 7 As seen in the diagram, instance 1 (CI1) 704-A is instantiated on host machine 1 702-A, instance 2 (CI2) 704-B is instantiated on host machine 2 702-B, and instance 3 (CI3) 704-AC and instance 4 (CI4) 704-D are instantiated on the shared host machine 702-C.

[0182] Each of CI 704-A, 704-B, 704-C, and 704-D is communicatively coupled to the other CI 704-A, 704-B, 704-C, and 704-D in VLAN 700 and to VSRS 714. Specifically, each of CI 704-A, 704-B, 704-C, and 704-D is connected to the other CI 704-A, 704-B, 704-C, and 704-D in VLAN 700 via an L2 VNIC and a switch, and is connected to VSRS 714. Each CI 704-A, 704-B, 704-C, and 704-D is associated with a unique L2 VNIC and switch. The switch can be a local L2 virtual switch and is uniquely associated with and deployed for the L2 VNIC. Specifically, CI1 704-A is associated with L2 VNIC 1 708-A and switch 1 710-A, CI2 704-B is associated with L2 VNIC 2708-B and switch 710-B, CI3 704-C is associated with L2 VNIC 3 708-C and switch 3 710-C, and CI4 704-D is associated with L2 VNIC 4 708-D and switch 4 710-D.

[0183] In some embodiments, each L2 VNIC 708 and its associated switch 710 can be instantiated on NVD 706. This instantiation can be a one-to-one relationship, whereby a single L2 VNIC 708 and its associated switch 710 are instantiated on a unique NVD 706, or it can be a many-to-one relationship, where multiple L2 VNIC 708s and their associated switches 710s are instantiated on a single shared NVD 706. Specifically, L2 VNIC 1 708-A and switch 1 710-A are instantiated on NVD 1 706-A, L2 VNIC 2 708-B and switch 2 710-B are instantiated on NVD 2, and L2 VNIC 3 708-C and switch 3 710-C, as well as L2 VNIC 4 708-D and switch 710-D, are all instantiated on a shared NVD (i.e., NVD 706-C).

[0184] In some embodiments, the VSRS 714 can support conflicting VLANs and IP spaces across multiple tenants. This can include multiple tenants on the same VSRS 714. In some embodiments, some or all of these tenants may choose to use some or all of the following: the same IP address space, the same MAC space, and the same VLAN space. This provides users with great flexibility in address selection. In some embodiments, this multi-tenancy is supported by providing each tenant with a different virtual network, which is a private network within the cloud network. Each virtual network (e.g., each VLAN or VCN) is assigned a unique identifier, such as a VCN identifier that can be a VLAN identifier. This unique identifier may be selected by, for example, the control plane, and specifically by the CSPI control plane. In some embodiments, this unique VLAN identifier may include one or more bits that may be included and / or used in packet encapsulation. Similarly, in some embodiments, each host may have a unique identifier, and / or each virtual interface or virtual gateway may have a unique identifier. In some embodiments, these unique identifiers, and specifically the unique identifiers of the tenant's virtual network, may be encoded in each communication. By providing a unique identifier for each virtual network and including it in communications, a single instantiation of VSRS can serve multiple tenants with overlapping addresses and / or namespaces. In some embodiments, VSRS 714 can determine which tenant a packet belongs to based on the VCN identifier and / or VLAN identifier associated with the communication, specifically within the VCN header of the communication. In the embodiments disclosed herein, communications leaving or entering a VLAN may have a VCN header that may include a VLAN identifier. Based on the VCN header containing the VLAN identifier, VSRS 714 can determine the lease, or in other words, the receiving VSRS can determine which VLAN and / or which tenant to send the communication to. Furthermore, each compute instance belonging to a VLAN (e.g., an L2 compute instance) is assigned a unique interface identifier that identifies the L2 VNIC associated with the compute instance. The interface identifier can be included in traffic to and / or to the compute instance (e.g., by including it in the header of a frame) and can be used by the NVD to identify the L2 VNIC associated with the compute instance. In other words, the interface identifier can uniquely indicate the compute instance and / or its associated L2 VNIC.

[0185] like Figure 7 As indicated in the document, switches 710-A, 710-B, 710-C, and 710-D can be combined to form an L2 distributed switch 712, also referred to herein as a distributed switch 712. From a customer's perspective, each switch 710-A, 710-B, 710-C, and 710-D in the L2 distributed switch 712 is a single switch connected to all CIs in the VLAN. However, the L2 distributed switch 712, simulating the user experience of a single switch, is infinitely scalable and includes a collection of local switches (e.g., in...). Figure 7 In the illustrative examples, switches 710-A, 710-B, 710-C, and 710-D are shown. Figure 7 As shown, each CI executes on a host machine connected to the NVD. For each CI on a host connected to the NVD, the NVD hosts a Layer 2 VNIC and a local switch associated with the compute instance (e.g., an L2 virtual switch, local to the NVD, associated with a Layer 2 VNIC, and a member or component of an L2 distributed switch 712). A Layer 2 VNIC represents a port of the compute instance on a Layer 2 VLAN. The local switch connects the VNIC to other VNICs (e.g., other ports) associated with other compute instances on other Layer 2 VLANs.

[0186] Each of CI 704-A, 704-B, 704-C, and 704-D can communicate with the other CI 704-A, 704-B, 704-C, and 704-D in VLAN 700, or with VSRS 714. One of CI 704-A, 704-B, 704-C, and 704-D sends a frame to another of CI 704-A, 704-B, 704-C, and 704-D or to VSRS 714 by sending the MAC address and interface identifier of the receiver in CI 704-A, 704-B, 704-C, or 704-D, or to VSRS 714. The MAC address and interface identifier can be included in the frame header. As explained above, the interface identifier can indicate the receiver of CI 704-A, 704-B, 704-C, 704-D or the L2 VNIC of VSRS 714.

[0187] In one embodiment, CI1 704-A can be the source CI, L2 VNIC 708-A can be the source L2 VNIC, and switch 710-A can be the source L2 virtual switch. In this embodiment, CI3 704-C can be the destination CI, and L2VNIC 3 708-C can be the destination L2 VNIC. The source CI can send a frame with the source MAC address and the destination MAC address. This frame can be intercepted by NVD 706-A, thereby instantiating the source VNIC and the source switch.

[0188] For VLAN 700, L2 VNICs 708-A, 708-B, 708-C, and 708-D can each learn a mapping from MAC addresses to L2 VNIC interface identifiers. This mapping can be learned based on frames and / or communications received within VLAN 700. Based on this previously determined mapping, the source VNIC can determine the interface identifier of the destination interface associated with the destination CI within the VLAN and can encapsulate the frame. In some embodiments, this encapsulation can include GENEVE encapsulation, and specifically L2 GENEVE encapsulation, which includes the L2 (Ethernet) header of the encapsulated frame. The encapsulated frame can identify the destination MAC address, destination interface identifier, source MAC address, and source interface identifier.

[0189] The source VNIC can pass the encapsulated frame to the source switch, which can then direct the frame to the destination VNIC. Upon receiving the frame, the destination VNIC can decapsulate the frame and then provide it to the destination CI.

[0190] Now for reference Figure 8 This illustrates a logical diagram of multiple connected L2 VLAN 800 connections. Figure 8 In the specific embodiment depicted, both VLANs reside within the same VCN. As can be seen, multiple connected L2 VLANs 800 may include a first VLAN (VLAN A 802-A) and a second VLAN (VLAN B 802-B). Each of these VLANs 802-A and 802-B may include one or more CIs, each CI may have an associated L2 VNIC and an associated L2 virtual switch. Additionally, each of these VLANs 802-A and 802-B may include a VSRS.

[0191] Specifically, VLAN A 802-A may include Instance 1 804-A connected to L2 VNIC 1 806-A and Switch 1 808-A, Instance 2 804-B connected to L2 VNIC 2 806-B and Switch 808-B, and Instance 3 804-C connected to L2 VNIC 3806-C and Switch 3 808-C. VLAN B 802-B may include Instance 4 804-D connected to L2 VNIC 4 806-D and Switch 4 808-D, Instance 5 804-E connected to L2 VNIC 5 806-E and Switch 808-E, and Instance 6 804-F connected to L2 VNIC 6 806-F and Switch 3808-F. VLAN A 802-A may also include VSRSA 810-A, and VLAN B 802-B may include VSRS B 810-B. Each of CI 804-A, 804-B, and 804-C of VLAN A 802-A may be communicatively coupled to VSRS A 810-A, and each of CIS 804-D, 804-E, and 804-F of VLAN B 802-B may be communicatively coupled to VSRS B 810-B.

[0192] VLAN A 802-A can be communicatively coupled to VLAN B 802-B via their respective VSRS 810-A, 810-B. Each VSRS can also be coupled to gateway 812, which provides access to other networks outside the VCN where VLANs 802-A and 802-B reside via CI 804-A, 804-B, 804-C, 804-D, 804-E, and 804-F in each VLAN 802-A and 802-B. In some embodiments, these networks may include, for example, one or more on-premises networks, another VCN, a service network, a public network such as the Internet, etc.

[0193] Each of CIs 804-A, 804-B, and 804-C in VLAN A 802-A can communicate with CIs 804-D, 804-E, and 804-F in VLAN B 802-B via VSRS 810A and 810-B of each VLAN 802-A and 802-B. For example, one of CIs 804-A, 804-B, 804-C, 804-D, 804-E, and 804-F in one of VLANs 802-A and 802-B can send a frame to CIs 804-A, 804-B, 804-C, 804-D, 804-E, and 804-F in the other VLAN 802-A and 802-B. This frame can leave the source VLAN via the VSRS of the source VLAN and enter the destination VLAN, and is routed to the destination CI via the destination VSRS.

[0194] In one embodiment, CI 1 804-A can be a source CI, VNIC 806-A can be a source VNIC, and switch 808-A can be a source switch. In this embodiment, CI 5 804-E can be a destination CI, and L2 VNIC 5806-E can be a destination VNIC. VSRS A 810-A can be a source VSRS identified as an SVSRS, and VSRS B 810-B can be a destination VSRS identified as a DVSRS.

[0195] The source CI can send a frame with a MAC address. This frame can be intercepted by the NVD of the instantiated source VNIC and the source switch. The source VNIC encapsulates the frame. In some embodiments, this encapsulation may include GENEVE encapsulation, and specifically L2GENEVE encapsulation. The encapsulated frame can identify the destination address of the destination CI. In some embodiments, this destination address may also include the destination address of the destination VSRS. The destination address of the destination CI may include the destination IP address, the destination MAC address of the destination CI, and / or the destination interface identifier of the destination VNIC associated with the destination CI. The destination address of the destination VSRS may include the IP address of the destination VSRS, the interface identifier of the destination VNIC associated with the destination VSRS, and / or the MAC address of the destination VSRS.

[0196] The source VSRS can receive frames from the source switch, look up the VNIC mapping from the frame's destination address (which can be a destination IP address), and forward the packet to the destination VSRS. The destination VSRS can receive the frame. Based on the destination address contained in the frame, the destination VSRS can forward the frame to the destination VNIC. The destination VNIC can receive and decapsulate the frame, and then provide the frame to the destination CI.

[0197] Now for reference Figure 9 This illustrates a logical diagram of multiple connected L2 VLANs and subnet 900. Figure 9 In the specific embodiment shown, the VLAN and subnet reside within the same VCN. This is indicated by the virtual routers and VSRS of the VLAN and subnet being directly connected, rather than through a gateway.

[0198] As can be seen, this can include a first VLAN (VLAN A 902-A), a second VLAN (VLAN B 902-B), and subnet 930. Each of these VLANs 902-A and 902-B can include one or more CIs, each of which can have an associated L2 VNIC and an associated L2 switch. Additionally, each of these VLANs 902-A and 902-B can include a VSRS. Similarly, subnet 930, which can be an L3 subnet, can include one or more CIs, each of which can have an associated L3 VNIC, and L3 subnet 930 can include a virtual router 916.

[0199] Specifically, VLAN A 902-A may include instance 1 904-A connected to L2 VNIC 1 906-A and switch 1 908-A, instance 2 904-B connected to L2 VNIC 2 906-B and switch 908-B, and instance 3 904-C connected to L2 VNIC 3906-C and switch 3 908-C. VLAN B 902-B may include instance 4 904-D connected to L2 VNIC 4 906-D and switch 4 908-D, instance 5 904-E connected to L2 VNIC 5 906-E and switch 908-E, and instance 6 904-F connected to L2 VNIC 6 906-F and switch 3908-F. VLAN A 902-A may also include VSRSA 910-A, and VLAN B 902-B may include VSRS B 910-B. Each of the CIs 904-A, 904-B, and 904-C of VLAN A 902-A may be communicatively coupled to VSRS A 910-A, and each of the CIs 904-D, 904-E, and 904-F of VLAN B 902-B may be communicatively coupled to VSRS B 910-B. L3 subnet 930 may include one or more CIs, and specifically may include instance 7 904-G, which is communicatively coupled to L3 VNIC 7906-G. L3 subnet 930 may include virtual router 916.

[0200] VLAN A 902-A can be communicatively coupled to VLAN B 902-B via their respective VSRS 910-A, 910-B. L3 subnet 930 can be communicatively coupled to VLAN A 902-A and VLAN B 902-B via virtual router 916. Each virtual router 916 and VSRS instances 910-A, 910-B can also be coupled to gateway 912, which provides access to other networks outside the VCN where VLANs 902-A, 902-B and subnet 930 reside (CI 904-A, 904-B, 904-C, 904-D, 904-E, 904-F, 904-G) reside in each of VLANs 902-A, 902-B and subnet 930. In some embodiments, these networks may include, for example, one or more on-premises networks, another VCN, a service network, a public network such as the Internet.

[0201] Each VSRS instance 910-A, 910-B can provide an exit path for frames leaving the associated VLAN 902-A, 902-B, and an entry path for frames entering the associated VLAN 902-A, 902-B. From VSRS instances 910-A, 910-B in VLAN 902-A, 902-B, frames can be sent to any desired endpoint, including L2 endpoints (such as L2 CIs in another VLAN on the same VCN or on a different VCN or network) and / or L3 endpoints (such as L3 CIs in a subnet on the same VCN or on a different VCN or network).

[0202] In one embodiment, CI 1 904-A can be a source CI, VNIC 906-A can be a source VNIC, and switch 908-A can be a source switch. In this embodiment, CI 7 904-G can be a destination CI, and VNIC 7 906-G can be a destination VNIC. VSRS A 910-A can be a source VSRS identified as an SVSRS, and virtual router (VR) 916 can be a destination VR.

[0203] The source CI can send frames with MAC addresses. These frames can be intercepted by the instantiated source VNIC and the source switch's NVD. The source VNIC encapsulates the frame. In some embodiments, this encapsulation may include GENEVE encapsulation, and specifically L2GENEVE encapsulation. The encapsulated frame can identify the destination address of the destination CI. In some embodiments, this destination address may also include the destination address of the VSRS of the source CI's VLAN. The destination address of the destination CI may include the destination IP address, the destination MAC address of the destination CI, and / or the destination interface identifier of the destination VNIC of the destination CI.

[0204] The source VSRS can receive frames from the source switch, look up the VNIC mapping from the frame's destination address (which can be a destination IP address), and forward the frame to the destination VR. The destination VR can receive the frame. Based on the destination address contained in the frame, the destination VR can forward the frame to the destination VNIC. The destination VNIC can receive and decapsulate the frame, and then provide the frame to the destination CI.

[0205] Learning is performed through L2 VNICs and / or L2 virtual switches within a virtual L2 network.

[0206] Now for reference Figure 10 This diagram illustrates an embodiment of intra-VLAN communication and learning within VLAN 1000. The learning here is specific to how L2 VNICs, the VSRS of the source CI VLAN, and / or L2 virtual switches learn the association between MAC addresses and L2 VNICs / VSRS VNICs (more specifically, the association between MAC addresses associated with L2 compute instances or VSRSs and identifiers associated with the L2 VNICs of these L2 compute instances or with the VSRS VNICs). Generally, learning is based on inbound traffic. For the interface-to-MAC address learning aspect, this learning differs from the learning process (e.g., ARP process) that L2 compute instances can implement to learn destination MAC addresses. Both learning processes (e.g., the learning process of L2 VNICs / L2 virtual switches and the learning process of L2 compute instances) in... Figure 12 The diagram shows a joint implementation.

[0207] As can be seen, VLAN 1000 includes compute instance 1 1000-A, communicatively coupled to NVD 1 1001-A. NVD 1 1001-A instantiates L2 VNIC 1 1002-A and L2 switch 1 1004-A. VLAN 1000 also includes compute instance 2 1000-B, communicatively coupled to NVD 2 1001-B. NVD 2 1001-B instantiates L2 VNIC 2 1002-B and L2 switch 2 1004-A. VLAN 1000 also includes VSRS 1015 running on a server cluster, which includes VSRS VNIC 1002-C and VSRS switch 1004-C. All switches 1004-A, 1004-B, and 1004-C together form L2 distributed switch 1050. VSRS 1015 is communicatively coupled to endpoint 1008, which may include a gateway and, more specifically, may include, for example, another L2 / L3 router in the form of another VSRS, or an L3 router in the form of a virtual router.

[0208] The control plane 1010 of the VCN hosting VLAN 1000 maintains information identifying each L2 VNIC on VLAN 1000 and the network placement of the L2 VNICs. For example, for an L2 VNIC, this information may include the interface identifier associated with the L2 VNIC and / or the physical IP address of the NVD hosting the L2 VNIC. The control plane 1010 uses this information to update (e.g., periodically or on demand) the interfaces in VLAN 1000. Therefore, each L2 VNIC 1002-A, 1002-B, 1002-C in VLAN 1000 receives information identifying the interfaces in the VLAN from the control plane 1010 and populates a table with this information. The table populated by the L2 VNICs can be stored locally in the NVD hosting the L2 VNICs. If VNICs 1002-A, 1002-B, and 1002-C already include a current table, VNICs 1002-A, 1002-B, and 1002-C can determine any differences between their current tables and the information / tables received from control plane 1010. In some embodiments, VNICs 1002-A, 1002-B, and 1002-C can update their tables to match the information received from control plane 1010.

[0209] like Figure 10 As seen in the diagram, frames are transmitted via L2 switches 1004-A, 1004-B, and 1004-C, and received by receiving VNICs 1002-A, 1002-B, and 1002-C. When a frame is received by VNICs 1002-A, 1002-B, and 1002-C, the VNIC learns the mapping between the source interface (source VNIC) and the source MAC address of the frame. Based on a table of information received from control plane 1010, the VNIC can map the source MAC address (from the received frame) to the interface identifier of the source VNIC and the IP address of the VNIC and / or the IP address of the NVD hosting the VNIC (where the interface identifier and (one or more) IP addresses are obtainable from the table). Therefore, L2 VNICs 1002-A, 1002-B, and 1002-C learn the mapping from interface identifier to MAC address based on received communications and / or frames. Each VNIC 1002-A, 1002-B, 1002-C can update its L2 forwarding (FWD) tables 1006-A, 1006-B, 1006-C using this learned mapping information. In some embodiments, the L2 forwarding table includes a MAC address and associates it with at least one of an interface identifier or a physical IP address. In such embodiments, the MAC address refers to the address assigned to the L2 compute instance and may correspond to a port emulated by the L2 VNIC associated with the L2 compute instance. The interface identifier can uniquely identify the L2 VNIC and / or the L2 compute instance. The virtual IP address can be the address of the L2 VNIC. And the physical IP address can be the IP address of the NVD hosting the L2 VNIC. The L2 forwarding updated by the L2 VNIC can be stored locally on the NVD hosting the L2 VNIC and used by the L2 virtual switch associated with the L2 VNIC to guide frames. In some embodiments, VNICs within a shared VLAN can share all or part of their mapping tables with each other.

[0210] Given the network architecture described above, traffic flow will now be described. For clarity, traffic flow will be described in conjunction with compute instance 21000-B, L2 VNIC 2 10002-B, L2 switch 2 1004-B, and NVD 2 1001-B. This description is equivalent to traffic flowing to and / or from other compute instances.

[0211] As explained above, VLANs are implemented within a VCN as an overlay L2 network on top of an L3 physical network. L2 compute instances within a VLAN can send or receive L2 frames that include an overlay MAC address (also known as a virtual MAC address) as the source and destination MAC addresses. L2 frames can also encapsulate packets that include an overlay IP address (also known as a virtual IP address) as the source and destination IP addresses. In some embodiments, the overlay IP address of the compute instance may belong to the CIDR range of the VLAN. Another overlay network IP address may belong to the CIDR range (in which case the L2 frame is within the VLAN) or flow outside the CIDR range (in which case the L2 frame is sent to or received from another network). L2 frames may also include a VLAN tag that uniquely identifies the VLAN and can be used to differentiate multiple L2 VNICs on the same NVD. L2 frames can be received by the NVD via tunnel in encapsulated packets from the host machine of the compute instance, from another NVD, or from a cluster of servers hosting VSRS. In these different cases, the encapsulated packets can be L3 packets sent over the physical network, where the source and destination IP addresses are physical IP addresses. Different types of encapsulation are possible, including GENEVE encapsulation. NVD can decapsulate received packets to extract L2 frames. Similarly, to transmit an L2 frame, NVD can encapsulate it in an L3 packet and transmit it on the physical substrate.

[0212] For intra-VLAN outbound traffic from Instance 2 1000-B, NVD 2 1001-B receives the frame from the host machine of Instance 2 1000-B via the Ethernet link. This frame includes an interface identifier that identifies L2VNIC 2 1000-B. The frame includes the overlay MAC address of Instance 2 1000-B (e.g., M.2) as the source MAC address and the overlay MAC address of Instance 1 1000-A (e.g., M.1) as the destination MAC address. Given the interface identifier, NVD 2 1001-B passes the frame to L2VNIC 2 1002-B for further processing. L2VNIC 2 1002-B forwards the frame to L2 switch 2 1004-B. Based on L2 forwarding table 1006-B, L2 switch 2 1004-B determines whether the destination MAC address is known (e.g., matches an entry in L2 forwarding table 1006-B).

[0213] If known, L2 switch 2 1004-B determines that L2 VNIC 1 1002-A is the relevant tunnel endpoint and forwards the frame to L2 VNIC 1 1002-A. Forwarding may include frame encapsulation in packets and packet decapsulation (e.g., GENEVE encapsulation and decapsulation), where the packet includes the frame, the physical IP address of NVD 1 1001-A (e.g., IP.1) as the destination address, and the physical IP address of NVD 2 1001-B (e.g., IP.2) as the source address.

[0214] If the information is unknown, then L2 switch 2 1004-B broadcasts the frame to the individual VNICs of the VLAN (e.g., including L2VNIC 1 1002-A and any other L2VNICs in the VLAN), where the broadcast frame is processed (e.g., encapsulated, transmitted, decapsulated) among the relevant NVDs. In some embodiments, this broadcast can be performed, or more specifically simulated, at the physical network, thereby encapsulating the frame individually into each L2 VNIC, including the VSRS in the VLAN. Thus, the broadcast is simulated via a series of replicated unicast packets at the physical network. Furthermore, each L2 VNIC receives the frame and learns the association between the interface identifier of L2 VNIC 21002-B and the source MAC address (e.g., M.2) and source physical IP address (e.g., IP.2).

[0215] For intra-VLAN inbound traffic from compute instance 1 1000-A to compute instance 2 1000-B, NVD 2 1001-B receives packets from NVD 1. The packet has IP.1 as the source address and a frame, where the frame includes M.2 as the destination MAC address and M.1 as the source MAC address. The frame also includes the network identifier of L2 VNIC 1 1002-A. After decapsulation, VNIC 2 receives the frame and learns that this interface identifier is associated with M.1 and / or IP.1, and if this was previously unknown, stores the learned information in the L2 forwarding table 1006-B at switch 2 for subsequent outbound traffic. Alternatively, after decapsulation, L2VNIC 2 1002-B receives the frame and learns that this interface identifier is associated with M.1 and / or IP.1, and if this information is known, refreshes the expiration time.

[0216] For egress traffic from instance 2 1000-B in VLAN 1000 to an instance in another VLAN, a similar flow to the egress traffic described above can exist, in addition to using VSRS VNICs and VSRS switches. Specifically, the destination MAC address is not within the L2 broadcast of VLAN 1000 (it is within another L2 VLAN). Therefore, the destination instance's overlay destination IP address (e.g., IP.A) is used for this egress traffic. For example, L2 VNIC 2 1002-B determines that IP.A is outside the CIDR range of VLAN 1000. Therefore, L2 VNIC 2 1002-B sets the destination MAC address to the default gateway MAC address (e.g., M.DG). Based on M.DG, L2 switch 2 1004-B sends the egress traffic to the VSRS VNIC (e.g., via tunnel, with appropriate end-to-end encapsulation). The VSRS VNIC forwards the egress traffic to the VSRS switch. Next, the VSRS switch performs routing functions, where, based on the destination IP address (e.g., IP.A), the VSRS switch in VLAN 1000 sends egress traffic to the VSRS switch in another VLAN (e.g., via a virtual router between the two VLANs, also with appropriate end-to-end encapsulation). The VSRS switch in the other VLAN then performs switching functions by determining that IP.A is within the CIDR range of this VLAN and performs a lookup in its ARP cache based on IP.A to determine the destination MAC address associated with IP.A. If no match is found in the ARP cache, an ARP request is sent to a different L2 VNIC in the other VLAN to determine the destination MAC address. Otherwise, the VSRS switch sends the egress traffic to the relevant VNIC (e.g., via a tunnel, with appropriate encapsulation).

[0217] For inbound traffic from an instance in another VLAN to an instance in VLAN 1000, the traffic flow is similar to the above, only in the reverse direction. For outbound traffic from an instance in VLAN 1000 to the L3 network, the traffic flow is similar to the above, except that the VSRS switch of VLAN 1000 routes packets directly to the destination VNIC in the virtual L3 network via a virtual router (e.g., without routing packets through another VSRS switch). For inbound traffic from the virtual L3 network to an instance in VLAN 1000, the traffic flow is similar to the above, except that packets are received by the VSRS switch of VLAN 1000A, which then sends them as frames within the VLAN. For traffic between VLAN 1000 and other networks (either outbound or inbound), a VSRS switch is used similarly, where its routing function is used for outbound traffic to send packets via appropriate gateways (e.g., IGW, NGW, DRG, SGW, LPG), and its switching function is used for inbound traffic to send frames within VLAN 1000.

[0218] Now for reference Figure 11 The diagram illustrates an embodiment of VLAN 1100 (e.g., a cloud-based virtual L2 network), and more specifically, it shows a view of VLAN implementation methods.

[0219] As described above, a VLAN can include "n" compute instances 1102-A, 1102-B, and 1102-N, each running on a host machine. As discussed earlier, there can be a one-to-one association between compute instances and host machines, or a many-to-one association between multiple compute instances and a single host machine. Each compute instance 1102-A, 1102-B, and 1102-N can be an L2 compute instance, in which case it is associated with at least one virtual interface (e.g., an L2 VNIC) 1104-A, 1104-B, and 1104-N and switches 1106-A, 1106-B, and 1106-N. Switches 1106-A, 1106-B, and 1106-N are L2 virtual switches and together form an L2 distributed switch 1107.

[0220] The pairs of L2 VNICs 1104-A, 1104-B, 1104-N and switches 1106-A, 1106-B, 1106-N associated with compute instances 1102-A, 1102-B, 1102-N on the host machine are a pair of software modules connected to NVDs 1108-A, 1108-B, 1108-N on the host machine. Each L2 VNIC 1104-A, 1104-B, 1104-N represents an L2 port of a single switch (referred to herein as vswitch) perceived by the client. Generally, host machine "i" executes compute instance "i" and connects to NVD "i". NVD "i" then executes L2 VNIC "i" and switch "i". L2 VNIC "i" represents L2 port "i" of vswitch. "i" is a positive integer between 1 and "n". Here, while a one-to-one association is described, other types of associations are also possible. For example, a single NVD can connect to multiple hosts, each executing one or more compute instances belonging to a VLAN. If this is the case, then the NVD hosts multiple pairs of L2 VNICs and switches, each pair corresponding to one of the compute instances.

[0221] VLANs can include instances of VSRS 1110. VSRS 1110 performs switching and routing functions and includes instances of VSRSVNIC 1112 and VSRS switch 1114. VSRS VNIC 1112 represents a port on the vswitch, where this port connects the vswitch to other networks via a virtual router. As shown in the diagram, VSRS 1110 can be instantiated on server cluster 1116.

[0222] Control plane 1118 can track and identify L2 VNICs 1104-A, 1104-B, and 1104-N and their placement within VLANs. Control plane 1110 can also provide this information to L2 interfaces 1104-A, 1104-B, and 1104-N within the VLANs.

[0223] like Figure 11 As shown, the VLAN can be a cloud-based virtual L2 network that can be built on top of the physical network 1120. In some embodiments, this physical network 1120 may include NVD 1108-A, 1108-B, and 1108-N.

[0224] Generally, the first L2 compute instance of a VLAN (e.g., compute instance 1 1102-A) can communicate with the second compute instance of the VLAN (e.g., compute instance 2 1102-B) using L2 protocols. For example, frames can be sent between these two L2 compute instances via VLAN. However, frames can be encapsulated, tunneled, routed, and / or otherwise processed so that frames can be sent through the underlying physical network 1120.

[0225] For example, compute instance 1 1102-A sends a frame destined for compute instance 2 1102-B. Depending on the network connections (e.g., TCP / IP connection, Ethernet connection, tunnel connection, etc.) between host machine 1 and NVD 1, NVD 1 and physical network 1120, physical network 1120 NVD 2, and NVD 2 and host machine 2, different types of processing can be applied to the frame. For example, the frame is received and encapsulated by NVD 1, and so on, until the frame reaches compute instance 2. It is assumed that this processing allows frames to be sent between underlying physical resources, and for the sake of brevity, its description is omitted when describing VLANs and related L2 operations.

[0226] Virtual L2 network communication

[0227] Various forms of communication can occur within or with a virtual L2 network. These can include intra-VLAN communication. In such embodiments, a source compute instance can send communication to a destination compute instance that is in the same VLAN as the source compute instance (CI). Communication can also be sent to an endpoint outside the VLAN of the source CI. This can include, for example, communication between a source CI in a first VLAN and a destination CI in a second VLAN, communication between a source CI in a first VLAN and a destination CI in an L3 subnet, and / or communication from a source CI in a first VLAN to a destination CI outside the VCN of the VLAN containing the source CI. This communication can also include, for example, receiving communication from a source CI outside the VLAN of the destination CI at the destination CI. This source CI can be located in another VLAN, in an L3 subnet, or outside the VCN of the VLAN containing the source CI.

[0228] Each CI within a VLAN can play an active role in traffic flow. This includes learning interface identifiers to MAC addresses (also referred to herein as interface-to-MAC address mapping), mapping instances within the VLAN to maintain the L2 forwarding table within the VLAN, and sending and / or receiving communications (e.g., frames in the case of L2 communication). VSRS can play an active role in communication within the VLAN as well as in communication with source or destination CIs outside the VLAN. VSRS can maintain presence in both L2 and L3 networks to enable egress and ingress communication.

[0229] Now for reference Figure 12 The diagram illustrates a flowchart of one embodiment of a process 1200 for communication within a VLAN. In some embodiments, process 1200 may be executed by a computing instance within a shared VLAN. This process may be specifically executed when a source CI sends communication to a destination CI within the VLAN, but does not know the IP-to-MAC address mapping of that destination CI. For example, this occurs when a source CI sends a packet to a destination CI that has an IP address in the VLAN, but the source CI does not know the MAC address of that IP address. In this case, an ARP procedure may be performed to learn the destination MAC address and the IP-to-MAC address mapping.

[0230] If the source CI knows the IP-to-MAC address mapping, it can send packets directly to the destination CI without performing an ARP procedure. In some embodiments, this packet can be intercepted by the source VNIC, which is an L2 VNIC in intraVLAN communication. If the source VNIC knows the interface-to-MAC address mapping used for the destination MAC address, it can encapsulate the packet, for example, in L2 encapsulation, and forward the corresponding frame to the destination VNIC, which is an L2 VNIC used for the destination MAC address in intraVLAN communication.

[0231] If the source VNIC does not know the interface-to-MAC address mapping used for MAC addresses, it can perform one aspect of the interface-to-MAC address learning process. This can include the source VNIC sending a frame to all interfaces in the VLAN. In some embodiments, this frame can be broadcast to all interfaces within the VLAN. In some embodiments, this broadcast can be implemented at the physical network level as a serial unicast. The frame can include the destination MAC and IP addresses, the interface identifier, and the source VNIC's MAC and IP addresses. Each VNIC in the VLAN can receive this frame and learn the source VNIC's interface-to-MAC address mapping.

[0232] Each receiving VNIC can also decapsulate frames and forward the decapsulated frames (e.g., the corresponding packets) to their associated CI. Each CI may include a network interface that can evaluate the forwarded packets. If the network interface determines that the CI of the received forwarded packet does not match the destination MAC and / or IP address, then the packet is discarded. If the network interface determines that the CI of the received forwarded frame matches the destination MAC and / or IP address, then the packet is received by the CI. In some embodiments, a CI having a MAC and / or IP address that matches the destination MAC and / or IP address of the packet can send a response to the source CI, whereby the source VNIC can learn the interface-to-MAC address mapping of the destination CI, and whereby the source CI can learn the IP-to-MAC address mapping of the destination CI.

[0233] Procedure 1200 can be performed when the source CI is unaware of the IP-to-MAC address mapping, or when the IP-to-MAC address mapping of the destination CI of the source CI is stale. Therefore, when the IP-to-MAC address mapping is known, the source CI can send packets. When the IP-to-MAC address mapping is unknown, procedure 1200 can be performed. When the interface-to-MAC address mapping is unknown, the interface-to-MAC address learning procedure outlined above can be performed. When the interface-to-MAC address mapping is known, the source VNIC can send the corresponding frame to the destination VNIC. Procedure 1200 begins at block 1202, where the source CI determines that the IP-to-MAC address mapping of the destination CI is unknown to the source CI. In some embodiments, this may include the source CI determining the destination IP address for the packet and determining that the destination IP address is not associated with a MAC address stored in the mapping table of the source CI. Alternatively, the source CI may determine that the IP-to-MAC address mapping for the destination CI is stale. In some embodiments, the mapping may be stale if it has not been updated and / or verified within a certain time limit. After determining that the IP-to-MAC address mapping of the destination CI is unknown and / or outdated for the source CI, the source CI initiates an ARP request for the destination IP address and sends the ARP request for Ethernet broadcast.

[0234] At box 1204, the source VNIC (also referred to herein as the source interface) receives ARP requests from the source CI. The source interface identifies all interfaces on the VLAN and sends ARP requests to all interfaces on the VLAN broadcast domain. As mentioned earlier, since the control plane knows all interfaces on the VLAN and provides this information to the interfaces connected to the VLAN, the source interface also knows all interfaces in the VLAN and is able to send ARP requests to each interface in the VLAN. To do this, the source interface copies the ARP requests and encapsulates one of the ARP requests for each interface on the VLAN. Each encapsulated ARP request includes the source CI interface identifier and the source CI MAC and IP address, the destination IP address, and the destination CI interface identifier. The source CI interface replicates Ethernet broadcast by sending the copied and encapsulated ARP requests (e.g., ARP messages) as serial unicast to each interface in the VLAN.

[0235] At box 1206, each interface in the VLAN broadcast domain receives and decapsulates an ARP message. Each interface receiving the ARP message in the VLAN broadcast domain learns the interface-to-MAC address mapping of the source VNIC for the source CI (e.g., the interface identifier to MAC address of the source interface of the source CI), because this message identifies the source CI's MAC and IP addresses as well as the source CI's interface identifier. As part of learning the interface-to-MAC address mapping for the source CI, each interface can update its mapping table (e.g., its L2 forwarding table) and can provide the updated mapping to its associated switches and / or CIs. Except for VSRS, each receiving interface can forward the decapsulated packet to its associated CI. The CI receiving the forwarded decapsulated packet, and specifically the network interface of that CI, can determine whether the destination IP address matches the CI's IP address. If the IP address of the CI associated with that interface does not match the destination CI IP address, then in some embodiments, the packet is dropped by the CI without further action. In the case of VSRS, the VSRS can determine whether the destination IP address matches the VSRS's IP address. If the IP address of the VSRS does not match the destination IP address specified in the received packet, then in some embodiments, the packet is discarded by the VSRS without further action.

[0236] If it is determined that the destination CI IP address specified in the received packet matches the IP address of the CI (destination CI) associated with the receiving interface, then, as indicated in box 1208, the destination CI sends a response, which may be a unicast ARP response to the source interface. This response includes the destination CI MAC address and destination CI IP address, as well as the source CI IP address and MAC address. This response is received by the destination interface encapsulating the unicast ARP response, as indicated in box 1210. In some embodiments, this encapsulation may include a GENEVE encapsulation. The destination interface may forward the encapsulated ARP response to the source interface via a destination switch. The encapsulated ARP response includes the destination CI MAC and IP addresses and the destination CI interface identifier, as well as the source CI MAC and IP addresses and the source CI interface identifier.

[0237] At box 1212, the source interface receives and decapsulates the ARP response. The source interface may further learn the interface-to-MAC address mapping of the destination CI based on the encapsulation and / or information contained in the encapsulated frame. In some embodiments, the source interface may forward the ARP response to the source CI.

[0238] At block 1214, the source CI receives an ARP response. In some embodiments, the source CI may update its mapping table based on the information contained in the ARP response, and specifically, based on the MAC and IP addresses of the destination CI to reflect the IP-to-MAC address mapping. Subsequently, the source CI may send a packet to the destination CI based on this MAC address. This packet may include the MAC address and interface identifier of the source CI as the source MAC address and source interface, and the MAC address and interface identifier of the destination CI as the destination MAC address and destination interface.

[0239] In block 1216, the source interface can receive packets from the source CI. The source interface can encapsulate the packets, and in some embodiments, this encapsulation uses GENEVE encapsulation. The source interface can forward the corresponding frame to the destination CI, specifically to the destination interface. The encapsulated frame can include the MAC address and interface identifier of the source CI as the source MAC address and source interface identifier, and the MAC address and interface identifier of the destination CI as the destination MAC address and destination interface.

[0240] At box 1218, the destination interface receives frames from the source interface. The destination interface can decapsulate the frames and then forward the corresponding packets to the destination CI. At box 1220, the destination CI receives packets from the destination interface.

[0241] Layer 2 Network Information

[0242] An L2 physical network consists of a single switch. Information associated with traffic on the L2 physical network can be determined by querying the switch. For example, a switch may maintain a single L2 forwarding table, which can be queried. In another example, a switch may maintain certain metrics about its ports, which can be queried. In contrast, an L2 virtual network (referred to as a VLAN for brevity) consists of distributed L2 switches (referred to as vswitch to indicate its correspondence to the customer's perception of a single virtual switch), which in turn consists of multiple L2 switches, each associated with a compute instance and hosted by NVD. Each such L2 switch can maintain its own L2 information. This information can be collected from multiple L2 switches to generate collective L2 information about the L2 virtual network, which is then presented to the customer as if it were a single centralized switch with an unlimited or arbitrary number of ports and VLANs (as the customer would expect). For example, each L2 switch is associated with an L2 forwarding table. Different L2 forwarding tables are collected to present a global L2 forwarding table to the customer. Similarly, metrics associated with each L2 switch can be collected to present statistics about the L2 virtual network to the customer. Additionally, L2 virtual networks allow customers flexibility in configuring the ports of their L2 virtual networks. These ports are implemented as L2 virtual network interfaces (e.g., VNICs). The L2 information collected about the L2 virtual switch (e.g., L2 forwarding tables and / or statistics) typically includes snippets of information about the L2 virtual network interfaces. Therefore, before presenting L2 information, these snippets are updated based on the mapping between L2 virtual network interfaces and ports, ensuring that customers receive L2 information formatted according to their port configurations.

[0243] In the example of the L2 forwarding table, in an L2 physical network, a switch maintains an L2 forwarding table that indicates the association between ports and MAC addresses. For example, a user can query the switch to receive the L2 forwarding table for troubleshooting. In contrast, in embodiments of this disclosure, a VLAN's vswitch is a collection of local switches hosted on one or more NVDs. Each local switch is associated with its own L2 forwarding table, which maps MAC addresses to tunnel endpoints (e.g., interface identifiers of VNICs) and / or NVDs (e.g., IP addresses of NVDs). A single view of the vswitch is also presented to the client. Thus, a global L2 forwarding table can be presented to the client, which is synthesized from local L2 forwarding tables and formatted using the client's definition of its VLAN (including the VLAN's port configuration). This global L2 forwarding table can be presented in response to a client query or based on event-based triggering, each of which is described below.

[0244] In the example of L2 statistics, within an L2 physical network, a client can query a switch to receive statistics about frame transmissions. In contrast, in embodiments of this disclosure, a VLAN's vswitch is a collection of local switches hosted on one or more NVDs, where each switch is associated with a VNIC hosted on the same NVD. The VNIC represents an L2 port of the vswitch. In response to a client query, statistics about VLAN frame transmissions can be collected from the various NVDs and presented to the client as vswitch statistics.

[0245] Figure 13 The illustration depicts an example environment suitable for defining an L2 virtual network configuration and providing associated L2 information according to certain embodiments. In an embodiment, the environment includes a computer system 1310 communicating with a client device 1320 via one or more networks (not shown). The computer system 1310 may include a collection of hardware computing resources hosting a VCN 1312. A control plane hosted by one or more of these hardware computing resources can receive and process input from the client device 1320 to deploy an L2 virtual network (shown as...) within the VCN 1312. Figure 13 (L2 VLAN 1314 in the middle).

[0246] In the example, input from client device 1320 can include various types of information. This information can be specified via console or API calls and can include L2 VLAN configuration 1322. Although not explicitly stated in... Figure 13 The diagram illustrates the process, but additional input from client devices can be received via console or API calls, and this additional input can include queries for L2 information (referred to herein as L2 information queries). L2 information queries can include queries for L2 forwarding tables (e.g., referred to herein as L2 forwarding table queries) and / or queries for L2 statistics (referred to herein as L2 statistics queries). L2 forwarding table queries can request entries for a specific set of ports or all ports (e.g., the entire VLAN) within an L2 VLAN. Similarly, L2 statistics queries can request statistics for a specific set of ports or all ports (e.g., the entire VLAN) specific to an L2 VLAN.

[0247] L2 VLAN Configuration 1322 can indicate, for example, the number, type(s), and configuration(s) of L2 compute instances to be included in L2 VLAN 1314. Furthermore, L2 VLAN Configuration 1322 can indicate the customer-specified port names on the customer-aware vswitch, the MAC addresses of the compute instances (which can be L2 compute instances), and the associations between ports and MAC addresses (or more generally, compute instances). For example, a customer can specify that L2 VLAN 1314 needs to include two L2 compute instances, the first with MAC address M.1 and associated with a first port named P1, and the other with MAC address M.2 and associated with a second port named P2. The control plane receives various information and then deploys and manages the different resources of L2 VLAN 1314.

[0248] A system (e.g., a subsystem of computer system 1310, such as a control plane and / or telemetry system) may collect L2 information associated with different L2 virtual switches presented to the client via the vswitch client and translate this collected information into L2 information 1324 associated with the port. L2 information 1324 may be sent to client device 1320 (e.g., in response to an L2 information query, periodically, or based on another triggering event).

[0249] In this embodiment, L2 information 1324 includes an L2 forwarding table with entries corresponding to ports (or subsets of ports) defined by the client. In such embodiments, ports are implemented as L2 VNICs. L2 VNICs are associated with L2 virtual switches. And the pair of L2 VNICs and L2 virtual switches is associated with a compute instance. In this case, the system determines the L2 forwarding table of the L2 virtual switch and associates the entries in the L2 forwarding table with the ports (e.g., although this table indicates a MAC address and an L2 VNIC associated with another compute instance, the information about the L2 VNIC is replaced with information about the port). Similar entries can be collected from different L2 virtual switches and updated to include port information instead of L2 VNIC information. These entries are included in the L2 forwarding table.

[0250] In this embodiment, L2 information 1324 includes L2 statistics with entries corresponding to ports (or subsets of ports) defined by the client. In such embodiments, the ports are also implemented as L2 VNICs. The L2 VNICs are associated with L2 virtual switches. Furthermore, the pair of L2 VNICs and L2 virtual switches is associated with a computation instance. In this case, the system determines the L2 metric associated with that pair (e.g., the frame rate received by the L2 VNIC on the ingress frame stream, such as...). Figure 10 As shown in the figure, and the frame rate transmitted on the egress frame stream by the L2 virtual switch, also as... Figure 10 (As shown in the diagram). The system also updates these metrics to be associated with the ports. Similar metrics can be collected from different L2 VNICs (L2 virtual switches) and updated using their associations with the corresponding ports. These metrics and their associations with the ports can be included in the L2 statistics. Furthermore, the system can perform statistical analysis on the collected metrics (e.g., averaging, percentile calculations, etc.) to indicate L2 statistics for a set of ports or the entire L2 VLAN.

[0251] Figure 14 The illustration shows example L2 information in a Layer 2 virtual network according to certain embodiments. The Layer 2 virtual network is referred to herein as a VLAN. Figure 14 The top of the diagram illustrates a VLAN implementation view 1410. Figure 14 The bottom diagram shows the VLAN client presentation 1420.

[0252] As mentioned above, a VLAN can include "n" compute instances, each executing on the host machine. Although Figure 14 The diagram illustrates a one-to-one association between a compute instance and a host machine, but many-to-one associations are also possible, where one host machine can run multiple compute instances. Each compute instance is associated with at least one virtual interface (e.g., an L2 VNIC) and a switch (e.g., an L2 virtual switch). The pair of VNICs and switches associated with a compute instance on the host machine can be a pair of software modules connected to an NVD on that host machine. Each L2 VNIC represents an L2 port of a client's vswitch. Figure 14 In the diagram, host machine "i" executes compute instance "i" and connects to NVD "i". NVD "i" then executes VNIC "i" and switch "i". VNIC "i" represents the L2 port "i" of the vswitch. "i" is a positive integer between 1 and "n". While a one-to-one association is described here, other types of associations are possible. For example, a single NVD can connect to multiple hosts, each executing one or more compute instances belonging to a VLAN. If this is the case, then the NVD hosts multiple pairs of VNICs and switches, each pair corresponding to one of the compute instances.

[0253] Customer input can be received via a console or API and sent from there to a system that includes the VLAN (e.g., the VCN's control plane and / or telemetry system). Input can be received via API calls and / or a console and can specify VLAN configurations, including at least port configurations (e.g., port naming conventions, MAC addresses associated with the ports, etc., shown as...). Figure 14 (Port configuration 1422 in the file). This configuration information can be used to respond to L2 information queries from clients and / or to provide L2 information associated with the port configuration.

[0254] In the example, and as described above, the NVD's L2 VNIC learns the interface-to-MAC address mapping based on inbound traffic. This mapping can be sent to the system along with the VLAN identifier. The system can receive similar mappings from different NVDs hosting different L2 VNICs and generate mappings between interface identifiers, MAC addresses, (e.g., the NVD's) physical IP addresses, and VLAN identifiers.

[0255] For example, VNIC 1 learns that M.2 (the overlay MAC address of compute instance 2) is associated with ID.2 (the interface identifier of L2 VNIC 2) and IP.2 (the physical address of NVD 2), and Mn (the overlay MAC address of compute instance n) is associated with ID.n (the interface identifier of L2 VNIC n) and IP.n (the physical address of NVD n). Similarly, VNIC 2 learns that M.1 (the overlay MAC address of compute instance 1) is associated with ID.1 (the interface identifier of L2 VNIC 1) and IP.1 (the physical address of NVD 1). These associations are reported to the system as part of a mapping, which can then generate the following mappings: {Client 1; M.1 → ID.1, IP.1; VLAN A}, {Client 1, M.2 → ID.2, IP.2; VLAN A}, ..., {Client 1, Mn → ID.n, IP.n; VLAN A}.

[0256] Additionally, the system can generate and maintain a mapping between port configuration 1422 and the distribution of VLAN resources on the physical network. For example, this mapping indicates that port "i" is configured for compute instance "i" and corresponds to L2 VNIC "i", which in turn is associated with L2 virtual switch "i", and this pair is managed by NVD "i". This mapping can be expressed as {Client 1; CI.1→P.1; P.1→M.1; M.1→ID.1, SW.1, IP.1; VLAN A}, where "CI.1" identifies compute instance 1 (using a customer-defined naming convention), P.1 identifies port 1 (using a customer-defined naming convention), M.1 identifies the MAC address associated with compute instance 1, ID.1 identifies L2 VNIC 1, SW.1 identifies L2 virtual switch 1, IP.1 identifies the IP address of NVD 1, and VLAN A identifies the VLAN.

[0257] The system can receive information associated with the L2 virtual switch hosted by NVD from NVD L2 (in Figure 14 The L2 information 1414 is shown as a separate L2 switch information (1412). This L2 switch is part of a pair associated with a compute instance. The other part of the pair is the L2 VNIC of the emulated port. The separate L2 information 1414 may include the L2 forwarding table of the L2 virtual switch and L2 metrics of traffic flows to the compute instance (e.g., ingress frame flows through the L2 VNIC or egress frame flows through the L2 virtual switch). The system may collect different separate L2 information 1414 from different NVDs, each associated with a different L2 virtual switch.

[0258] Based on the collected individual L2 switch information 1414, the system can generate L2 information associated with the vswitch (in Figure 14 This is illustrated as global L2 information 1414. For example, global L2 information 1414 can be a combination and / or aggregation of individual L2 switch information 1412 collected from different NVDs. For example, global L2 information 1414 is an aggregation of entries from individual L2 forwarding tables, where this aggregation represents the global L2 forwarding table for L2 VLANs. In another example, global L2 information 1414 includes individual L2 metrics and / or L2 statistics derived from individual L2 metrics.

[0259] Based on the mapping between port configuration 1422 and VLAN resources on the physical network, the system can generate L2 information associated with a set of ports or the entire set of ports (e.g., the entire L2 VLAN) of L2 VLANs from global L2 information. This L2 information... Figure 14 This is shown as L2 vswitch information 1424. In this example, L2 vswitch information 1424 includes entries from global L2 information 1414, where these entries are associated with ports (e.g., using a customer-defined naming convention) rather than with L2 virtual switches and / or L2 VNICs. L2 vswitch information 1424 can be sent to customer devices. For example, L2 vswitch information 1424 includes entries from the global L2 forwarding table, where these entries are mapped to ports. In another example, L2 vswitch information 1424 includes separate L2 metrics and / or L2 statistics mapped to ports.

[0260] Figure 15 The illustration shows an example L2 forwarding table in a Layer 2 virtual network according to certain embodiments. The Layer 2 virtual network is referred to herein as a VLAN. Figure 15 The top of the diagram illustrates a VLAN implementation view 1410. Figure 15 The bottom diagram illustrates VLAN client 1420. VLAN configuration and... Figure 14 The similarities or similarities between the two figures will not be repeated in this article.

[0261] Systems such as the control plane receive individual L2 forwarding tables 1512 from the NVD. Each individual L2 forwarding table 1512 is associated with an L2 virtual switch for the VLAN. Based on ingress traffic learning of the L2 VNICs associated with the L2 virtual switches, the individual forwarding tables of the L2 virtual switches associate MAC addresses with the interface identifier of the L2 VNIC and / or the physical IP address of the NVD hosting the L2 VNIC. In such embodiments, the MAC address refers to the address assigned to the L2 compute instance and may correspond to a port emulated by the L2 VNIC associated with the L2 compute instance.

[0262] From the individual L2 forwarding table 1512, the system generates a global L2 forwarding table 1514. For example, the global L2 forwarding table 1514 is a collection of different entries from the individual L2 forwarding table 1512, with redundant entries removed. Therefore, based on ingress traffic learning across different L2VNICs, the global L2 forwarding table 1514 associates MAC addresses with the interface identifier of the L2VNIC and / or the physical IP address of the NVD hosting the L2VNIC.

[0263] The system can generate an L2 vswitch forwarding table 1522 from the global L2 forwarding table 1514 and based on the mapping between port configurations and VLAN resources distributed across the physical network. For example, entries in the global L2 forwarding table 1514 are updated to be associated with ports (e.g., using a customer-defined port naming convention) rather than with L2 VNICs, L2 virtual switches, and / or NVDs. Therefore, the L2 vswitch forwarding table 1522 associates MAC addresses with port identifiers (e.g., M.1 is the MAC address of compute instance 1, and compute instance 1 is after port 1).

[0264] In one embodiment, after a client's L2 forwarding table query, the system can generate a global L2 forwarding table 1514 from the collected L2 forwarding information, and then generate an L2 vswitch forwarding table 1522. Alternatively, the global L2 forwarding table 1514 is generated before the client's L2 forwarding table query, while the L2 vswitch forwarding table 1522 is generated in response to such a query.

[0265] In another example, as an alternative to or supplement to the separate L2 forwarding table 1512 collected by the storage, the system can determine the NVD hosting the L2 virtual switch and request such a table from the NVD in response to a client query.

[0266] Furthermore, partly due to the learning capabilities of L2 VNICs and the flexibility of the network, conflicts can exist between multiple parts of the collected individual L2 forwarding table 1512. For example, NVD 1 might report to the system that compute instance 2 is behind M.2 after L2 VNIC 1 receives and processes a frame from compute instance 2 to compute instance 1. Subsequently, due to network events, M.2 can be reassigned to compute instance 3 associated with L2 VNIC 3. Figure 15 (Computation instance 3 and L2 VNIC 3 are not shown). L2 VNIC 1 may not receive subsequent frames from computation instance 3. Instead, L2 VNIC n can do so by receiving and processing frames from computation instance 3 to computation instance n. Thus, NVD n can report to the system that computation instance 3 is behind M.2. Because NVD 1 previously reported that computation instance 2 is behind M.2, the updated report from NVD n conflicts with it. In this case, the control plane can discard the older conflicting L2 forwarding information (e.g., computation instance 2 is behind M.2) and only use the latest L2 forwarding information (e.g., computation instance 2 is behind M.2) in the global L2 forwarding table.

[0267] The above example of a MAC address reassignment from one compute instance to another is an example of a network event that can trigger the control plane to generate an updated global L2 forwarding table and send the updated global L2 forwarding table to the client. For example, the client may have subscribed to receive such updates. Subscriptions can be received as client input via API calls or the console and can be specific to a particular type of network event or general to any type of network event that causes an update to the global L2 forwarding table. In the example above, the overlay MAC address remains unchanged but is reassigned to another compute instance. Another example of a network event includes a change to the overlay MAC address of the same compute instance. Another example of a network event includes adding an overlay MAC address and / or a compute instance. Yet another example, a network event could include the relocation of a compute instance from one host to another. Other types of events are also possible, such as time-based events. For example, a client could instruct the periodic reporting of L2 vswitch FWD 1522. The system could then periodically collect individual L2 forwarding tables 1512, update the global L2 forwarding table 1514, and then update the vswitch forwarding table 1522 to periodically report to the client.

[0268] Compared to client queries, for event-based triggers, the system can send configuration information to the NVD, requesting the NVD to automatically report its local, individual L2 forwarding tables after detecting an event (e.g., network-based or time-based events). After the NVD detects an event, it sends the local, individual L2 forwarding tables of one or more applicable L2 virtual switches to the system, which then updates the global L2 forwarding table 1514 and the L2 vswitch forwarding table 1522 and notifies the client.

[0269] Figure 16 The illustration shows example L2 metrics and statistics in a layer 2 virtual network according to certain embodiments. Figure 16 The bottom diagram illustrates VLAN client 1420. VLAN configuration and... Figure 14 The similarities or similarities between the two figures will not be repeated in this article.

[0270] Systems such as control plane or telemetry systems receive individual L2 metrics 1612 from the NVD. Each of the individual L2 metrics 1612 is associated with an L2 virtual switch, L2 VNIC, or L2 VNIC-L2 virtual switch pair for the VLAN. Individual L2 metrics include metrics about inbound traffic flows to the L2 VNIC or outbound traffic flows from the L2 virtual switch (e.g., frame rate or any other type of metric indicated by client input). The system associates such metrics with the MAC address, the interface identifier of the L2 VNIC, the identifier of the L2 virtual switch, and / or the physical IP address of the NVD hosting the pair.

[0271] From individual L2 metrics 1612, the system generates global L2 statistics 1614. For example, global L2 statistics 1614 is a statistical combination of individual L2 metrics 1612. Global L2 metrics 1614 are associated with MAC addresses, interface identifiers of L2 VNICs, identifiers of L2 virtual switches, and / or the physical IP addresses of the NVDs hosting the pair.

[0272] From individual L2 statistics (1612) and the mapping between port configuration and VLAN resource distribution on the physical network, the system can generate individual metrics for each port (in... Figure 16 These are shown as individual port metrics 1622. Each of the individual port metrics 1612 corresponds to one of the individual L2 metrics 1616, where the value is mapped to a port rather than a corresponding L2 VNIC, L2 virtual switch, and / or pair. For example, an individual port metric for a port includes a metric about inbound traffic flow to the L2 VNIC or outbound traffic flow from the L2 virtual switch associated with that port (e.g., frame rate or any other type of metric indicated by client input). The system associates such metrics with MAC addresses and user-defined port naming conventions.

[0273] From global L2 statistics 1614 and based on the mapping of port configurations and VLAN resource distribution on the physical network, the system can generate L2 vswitch statistics 1624. For example, entries in global L2 statistics 1614 are updated to be associated with ports (e.g., using a customer-defined port naming convention) rather than with L2 VNICs, L2 virtual switches, and / or NVDs. Therefore, L2 vswitch statistics 1622 associates the values ​​of the statistics with MAC addresses and port identifiers.

[0274] In this embodiment, the client's input includes different dimensions for the statistics. Each dimension indicates the type of metric to be collected and / or the type of statistics to be generated. In the first dimension, the client can specify whether to report the number of frames (e.g., FPS), frame size, and / or the amount of bits (e.g., BPS). In the second dimension, the client can indicate whether to report statistics for the entire VLAN, each port, or a specific set of ports. In the third dimension, the client can request statistics specific to transmitted frames, dropped frames, frame errors, or all frames.

[0275] The NVD of the L2 VNIC hosting the VLAN (e.g., NVD 1 hosting L2 VNIC 1) periodically reports metrics to the system. Metrics can be annotated with metadata identifying the VLAN and L2 VNIC. Thus, the system collects metrics from different NVDs, with these metrics annotated.

[0276] Furthermore, the system generates the statistics requested by the client based on the annotated metrics, port configurations, and mappings. For example, after a client requests the number of frames dropped per hour and per port of the VLAN, given the correspondence between L2 VNIC 1 and port 1, the system does not indicate the number of frames dropped at L2 VNIC 1 in the past hour, but instead presents the number of frames dropped at port 1.

[0277] Figure 17 This is a flowchart illustrating a process 1700 for providing L2 information according to certain embodiments. In some embodiments, process 1700 may be performed by a VCN system, such as a control plane that manages the deployment of L2 virtual networks on the underlying physical network or a telemetry system that monitors data related to the deployment and use of L2 virtual network resources. Process 1700 is described in conjunction with L2 information. This L2 information may include any one or more of L2 forwarding tables, L2 metrics, and L2 statistics, or a combination thereof.

[0278] Process 1700 begins at block 1702, where the system receives client input indicating client configuration for the Layer 2 virtual network. In some embodiments, client input is received from the client device via API calls and / or a console and indicates L2 virtual network configuration (e.g., as combined with...). Figure 13 The L2 VLAN configuration is described. L2 virtual network configuration can include port configurations, such as the number and naming convention of vswitch ports and their association with customer-defined compute instances.

[0279] At box 1704, the system determines a mapping between the client configuration and the distribution of Layer 2 virtual network resources on the physical network. In some embodiments, in addition to compute instances, resources also include L2 VNICs and L2 virtual switches. As explained above, the L2 VNIC-L2 virtual switch pair is hosted by the NVD, while the compute instance is hosted by a host machine communicatively coupled to the NVD. Each L2 VNIC emulates a client-aware vswitch port. The system generates a mapping that indicates a correspondence between each port and an L2 VNIC. This mapping may also associate each port with a MAC address, compute instance, L2 virtual switch, and the NVD hosting the L2 VNIC-L2 virtual switch pair.

[0280] At block 1706, the system identifies an event for collecting L2 information. In some embodiments, this event includes an L2 information query by a client. In other embodiments, the event includes a network event that may lead to changes in previously collected L2 information, such as a change to the network topology or configuration of the L2 virtual network. In still other embodiments, the event includes time-related events, such as predefined periodic time intervals for collecting L2 information.

[0281] At block 1708, the system determines first L2 information associated with a first L2 virtual switch in the client's L2 virtual network. In some embodiments, a pull mechanism is used. For example, the first L2 virtual switch is hosted by a first NVD. After the system detects an event, the system requests L2 information from the NVD. In response, the NVD sends first L2 information with an indication of its association with the first L2 virtual switch, the L2 virtual network interface paired with the first L2 virtual switch, and / or the L2 virtual network interface-L2 virtual switch pair. In other embodiments, a push mechanism is used. For example, the NVD detects an event and sends the first L2 information with the associated indication to the system.

[0282] At block 1710, the system determines Layer 2 information associated with the second L2 virtual switch. In some embodiments, the operation of block 1710 can be similar to that of blocks 1708, thereby receiving Layer 2 information from the same NVD (if such NVD hosts both the first and second L2 virtual switches) or different NVDs (if the first and second L2 virtual switches are hosted by different NVDs). For simplicity, process 1700 describes only two L2 virtual switches. However, if the L2 virtual network includes additional L2 virtual switches, then L2 information associated with each such switch can be determined similarly.

[0283] At block 1712, the system generates third L2 information based on the first and second L2 information. In some embodiments, the third L2 information is a combination or aggregation of the first and second L2 information and any other L2 information determined for any other L2 virtual switches in the L2 virtual network. In some embodiments, the first and / or second L2 information is updated based on a mapping such that the third L2 information indicates port information rather than L2VNIC information. For example, the system determines that a first L2 virtual network interface is associated with a first L2 virtual switch and a second L2 virtual network interface is associated with a second L2 virtual switch. Based on the mapping, the system determines that the first L2 virtual network corresponds to a first port and the second L2 virtual network interface corresponds to a second port. The system then indicates that the third L2 information is associated with the first and second ports, rather than with the first and second L2 virtual network interfaces. This indication can be implemented in different ways. In some embodiments, the entries and / or metadata (e.g., annotations) of the first L2 information identify the first L2 virtual network interface (and / or the first L2 virtual switch). Therefore, the system instead updates the entries and / or metadata to identify the first port. Similar updates can be made to the second L2 information. The updated first and second L2 information (e.g., their entries) can be combined or aggregated to generate the third L2 information. Generally, the update removes information about the distribution of resources specific to the L2 virtual network and replaces it with port-specific information, so that when the third L2 information is presented to the client, the client can understand the information based on how they have configured the L2 virtual network.

[0284] At block 1714, the system sends Layer 3 L2 information to the client's device. In some embodiments, the third L2 information is sent based on an event determined at block 1702. For example, if an L2 information query is received, then the third L2 information is sent as a result of such a query. In this illustration, the L2 information query may indicate a port, a set of ports, or all ports of the L2 virtual network. The result includes a portion or the entire third L2 information corresponding to the indicated port(s). For example, the L2 information query may indicate a first port. In this example, the system determines that the first port corresponds to a first Layer 2 virtual network interface based on a mapping, and determines that the first Layer 2 virtual network interface is associated with a first Layer 2 virtual switch. The system can then generate a query result based on the Layer 2 information and associated with the first port. For example, only the entries in the Layer 2 information are updated and the port is sent as a query result. Alternatively, if the Layer 2 information has been updated and is available, the system can filter it so that only the portion of the query result related to the first port is sent. In another illustration, a network event for collecting L2 information is detected. In this illustration, the entire third L2 information can be sent to the client's device. Alternatively, if a network event only affects a portion of the third L2 information, then only that portion is sent (e.g., the network event corresponds only to a change to a Layer 1 2 virtual switch and / or a Layer 1 2 virtual network interface; in this case, only the updated Layer 1 L2 information is sent). In a further illustration, a time event for collecting L2 information is detected. In this illustration, the entire third L2 information can be sent to the client's device. Alternatively, only changes to the third L2 information can be sent.

[0285] Figure 18 This is a flowchart illustrating process 1800 for generating an L2 forwarding table according to certain embodiments. When, for example, L2 information includes an L2 forwarding table, process 1800 can be implemented as a subprocess of process 1700. For simplicity, process 1800 is described in conjunction with only one L2 virtual switch. However, if the L2 virtual network includes additional L2 virtual switches, then the operations of process 1800 can be similarly repeated to process the L2 forwarding table for each such switch.

[0286] Procedure 1800 begins at block 1802, where the system determines a first L2 forwarding table for a first L2 virtual switch. In some embodiments, this determination may be based on event detection and may use a pull or push mechanism, as described in conjunction with procedure 1700. Here, the NVD hosting the first L2 virtual switch may send the first L2 forwarding table to the system, with indications to associate this table with the first L2 virtual switch, a first L2 virtual network interface paired with the first L2 virtual switch, and / or the pair.

[0287] At box 1804, the system determines that a Layer 2 virtual switch is associated with a Layer 2 virtual network interface. In some embodiments, this determination is based on the distribution of L2 virtual network resources deployed on the physical network.

[0288] At block 1806, the system determines that the first Layer 2 virtual network interface corresponds to the first port. In some embodiments, this determination is based on port mappings (e.g., based on client port configuration) and the distribution of L2 virtual network resources deployed on the physical network. For example, this mapping indicates that the first port corresponds to the first L2 virtual network interface and / or the first L2 virtual network switch.

[0289] At block 1808, the system generates a second L2 forwarding table associated with the first port. In some embodiments, the entries and / or metadata of the first L2 forwarding table are updated to associate the first L2 forwarding table with the first port, rather than with the first L2 virtual switch and / or the first L2 virtual network interface. Additionally, the entries may indicate a mapping of MAC addresses to L2 virtual network interfaces, L2 virtual switches, and / or NVDs (e.g., as learned by the first L2 virtual network interface on inbound traffic). These entries may be updated to remap MAC addresses to the port corresponding to the L2 virtual network interface. The second L2 forwarding table includes the updated entries, with an indication that this table is associated with the first port.

[0290] Although Figure 18 Not shown, but the second L2 forwarding table can be sent to the client's device based on events (e.g., L2 forwarding table queries, network events, or time events). In some embodiments, if an L2 forwarding table query is received and only the first port is indicated, then the second L2 forwarding table is included in the query results. In contrast, if an L2 forwarding table query is received and multiple ports are indicated, then the corresponding L2 forwarding tables (given their association with the ports) are determined and sent in the combined L2 forwarding table in the query results. Alternatively, an L2 vswitch table has already been generated, and entries in this table are filtered according to the indicated ports and sent in the query results.

[0291] At box 1810, the system determines a change to the Layer 2 forwarding table. In some embodiments, an event can cause a change to an entry (e.g., mapping) in the Layer 2 forwarding table. As described above, this event could be the reassociation of a first MAC address from a Layer 2 virtual network interface to a Layer 3 virtual network interface, or the association of a second MAC address with a Layer 2 virtual network interface. Such an event can trigger an update to the Layer 2 forwarding table (based on entry learning of the Layer 2 virtual network interface), whereby the MAC addresses in this table are remapped as appropriate.

[0292] At box 1812, the system updates the Layer 2 forwarding table based on this change. In some embodiments, the previously generated Layer 2 forwarding table is also updated to reflect the MAC address remapping. After detecting a change to the Layer 2 forwarding table, the system can determine that this table is associated with a first port and can associate the change with the port. Therefore, when a client query requesting an L2 forwarding table update is received, this association is used to return the updated Layer 2 forwarding table. Additionally or alternatively, updates to the second L2 forwarding table can be pushed to the client's device.

[0293] Figure 19 This is a flowchart illustrating a process 1900 for generating L2 statistics according to certain embodiments. When, for example, the L2 information includes L2 metrics and / or L2 statistics, process 1900 can be implemented as a subprocess of process 1700. When, for example, the L2 information also includes an L2 forwarding table, process 1900 can be used in conjunction with process 1800. For brevity, L2 metrics and statistics associated with a single L2 virtual switch are described in conjunction with process 1900. However, if the L2 virtual network includes additional L2 virtual switches, then the operation of process 1900 can be similarly repeated to process the L2 metrics and statistics associated with each such switch.

[0294] Procedure 1900 begins at block 1902, where the system determines a first L2 metric associated with a first Layer 2 virtual switch. In some embodiments, this determination may be based on event detection and may use a pull or push mechanism, as described in conjunction with procedure 1700. Here, the NVD hosting the first L2 virtual switch may send the first L2 metric to the system with an indication (e.g., a note in metadata) to associate these metrics with the first L2 virtual switch, a first L2 virtual network interface paired with the first L2 virtual switch, and / or that pair. The type of L2 metric to be collected may be indicated in the client's input. For example, the client may specify to collect metrics about the number of frames, frame size, and / or bits. Thus, the system collects such L2 metrics from different NVDs, each hosting one or more L2 virtual switches.

[0295] At box 1904, the system determines that a Layer 2 virtual switch is associated with a Layer 2 virtual network interface. In some embodiments, this determination is based on the distribution of L2 virtual network resources deployed on the physical network.

[0296] At block 1906, the system determines that a first Layer 2 virtual network interface corresponds to a first port. In some embodiments, this determination is based on a mapping between ports (e.g., based on client port configuration) and the distribution of L2 virtual network resources deployed on the physical network. For example, this mapping indicates that the first port corresponds to a first L2 virtual network interface and / or a first L2 virtual network switch. Furthermore, client input can identify the target for which L2 metrics are to be collected. For example, the client can specify a port, a set of ports, or a Layer 2 virtual network as the target. Therefore, if the input indicates a first port, the system can then collect first L2 metrics and determine that such metrics are associated with the first port.

[0297] At box 1908, the system associates the first L2 metric with the first port. In some embodiments, the indication (e.g., a comment in the metadata) is updated to identify the first port. Furthermore, the identifiers of the first L2 virtual switch and / or the first L2 virtual network interface can be removed from the comment.

[0298] At block 1910, the system generates L2 statistics based on a first L2 metric. In some embodiments, the L2 statistics are specific to a first port and generated from the first L2 metric associated with that port. In other embodiments, the L2 statistics are associated with more than one port, such as with a set of ports or the entire L2 virtual network or vswitch. In this case, the L2 metric can be similarly determined and associated with the applicable port, as described above, and L2 statistics can be generated from such L2 metrics. The type of statistics (e.g., average, sum, or other statistical analysis of frame count, frame size, and / or bit quantity) and / or the target for which statistics are to be generated (e.g., port, set of ports, or layer 2 virtual network) can also be indicated by client input.

[0299] At box 1912, the system associates L2 statistics with a set of ports and / or an L2 virtual network. In some embodiments, this association depends on the target for which the L2 statistics are generated. The association may include one or more annotations (e.g., in metadata) that identify the target (e.g., a port, a set of ports, or a L2 virtual network).

[0300] Although not in Figure 19 As shown, however, L2 metrics and / or L2 statistics, and their association with the target, can be sent to the client's device. In some embodiments, if an L2 statistics query is received and only indicates the first port, then the first L2 metric and L2 statistics determined solely based on such metric can be included in the query results. In other embodiments, once an event occurs (e.g., L2 statistics indicate that the average frame rate exceeds a predefined threshold, or the first L2 metric indicates that the frame rate associated with the first port exceeds a predefined threshold), the relevant L2 statistics and / or metric are sent to the client's device.

[0301] C- Example Infrastructure as a Service architecture

[0302] As noted above, Infrastructure as a Service (IaaS) is a specific type of cloud computing. IaaS can be configured to provide virtualized computing resources over a public network (e.g., the Internet). In the IaaS model, cloud providers can host infrastructure components (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer), etc.). In some cases, IaaS providers can also offer various services accompanying these infrastructure components (e.g., billing, monitoring, logging, security, load balancing, and clustering, etc.). Therefore, because these services may be policy-driven, IaaS users can implement policies to drive load balancing to maintain application availability and performance.

[0303] In some cases, IaaS customers can access resources and services over a wide area network (WAN) such as the Internet and can use the cloud provider's services to install the remaining elements of the application stack. For example, a user can log in to the IaaS platform to create virtual machines (VMs), install an operating system (OS) on each VM, deploy middleware such as databases, create buckets for workloads and backups, and even install enterprise software into that VM. The customer can then use the provider's services to perform various functions, including balancing network traffic, troubleshooting application issues, monitoring performance, and managing disaster recovery.

[0304] In most cases, cloud computing models will require the involvement of cloud providers. Cloud providers can, but are not necessarily, third-party providers specializing in (e.g., provisioning, renting, selling) IaaS services. Entities may also choose to deploy private clouds, thus becoming their own infrastructure service providers.

[0305] In some examples, IaaS deployment is the process of placing a new application or a new version of an application onto a prepared application server, etc. It may also include the processing of server preparation (e.g., installation libraries, daemons, etc.). This is typically managed by the cloud provider, below the hypervisor layer (e.g., servers, storage devices, network hardware, and virtualization). Therefore, the customer can be responsible for processing (OS), middleware, and / or application deployment (e.g., on self-service virtual machines, etc., which can be started on demand).

[0306] In some examples, IaaS provisioning can refer to acquiring computers or virtual hosts for use, or even installing necessary libraries or services on them. In most cases, deployment does not include provisioning, and provisioning may need to be performed first.

[0307] In some cases, IaaS provisioning presents two distinct challenges. First, there's the initial challenge of provisioning the initial infrastructure set before anything is operational. Second, once everything is provisioned, there's the challenge of evolving the existing infrastructure (e.g., adding new services, changing services, removing services, etc.). In some cases, both challenges can be addressed by enabling configuration that declaratively defines the infrastructure. In other words, the infrastructure (e.g., which components are needed and how they interact) can be defined by one or more configuration files. Therefore, the overall topology of the infrastructure (e.g., which resources depend on which resources and how they work together) can be described declaratively. In some cases, once the topology is defined, workflows for creating and / or managing the different components described in the configuration files can be generated.

[0308] In some examples, the infrastructure can have many interconnected components. For example, there may be one or more Virtual Private Clouds (VPCs) (e.g., potential on-demand pools of configurable and / or shared computing resources), also known as the core network. In some examples, one or more security group rules may also be provided to define how the network's security is configured, as well as one or more virtual machines (VMs). Other infrastructure elements, such as load balancers, databases, etc., may also be provided. The infrastructure can evolve incrementally as more and / or more infrastructure elements are expected and added.

[0309] In some cases, continuous deployment techniques can be used to enable the deployment of infrastructure code across various virtual computing environments. Furthermore, the described techniques enable infrastructure management within these environments. In some examples, service teams may write code that they expect to deploy to one or more, but often many, different production environments (e.g., across various geographical locations, sometimes spanning the entire world). However, in some examples, the infrastructure on which the code will be deployed must first be set up. In some cases, provisioning can be done manually, resources can be provisioned using provisioning tools, and / or once the infrastructure is provisioned, the code can be deployed using deployment tools.

[0310] Figure 20 This is a block diagram 2000 illustrating an example pattern of an IaaS architecture according to at least one embodiment. Service provider 2002 may communicatively couple to secure host leasing 2004, which may include a virtual cloud network (VCN) 2006 and a secure host subnet 2008. In some examples, service provider 2002 may use one or more client computing devices, which may be portable handheld devices (e.g., Cellular phone Computing tablets, personal digital assistants (PDAs), or wearable devices (e.g., Google) Head-mounted displays), running software (such as Microsoft Windows) ) and / or various mobile operating systems (such as iOS, Windows Phone, Android, BlackBerry 8, Palm OS, etc.), and support the Internet, email, and short message service (SMS). Or other communication protocols. Alternatively, the client computing device can be a general-purpose personal computer, including, for example, those running various versions of Microsoft... Apple Personal computers and / or laptops running Linux operating systems. Client computing devices can be running various commercially available operating systems. Workstation computers operating systems, including but not limited to any of the various GNU / Linux operating systems (such as, for example, Google Chrome OS), or UNIX-like operating systems. Alternatively or additionally, the client computing device can be any other electronic device, such as a thin client computer, an internet-enabled gaming system (e.g., with or without...). A Microsoft Xbox game console with gesture input devices, and / or a personal messaging device capable of communicating over a network that can access VCN 2006 and / or the Internet.

[0311] VCN 2006 may include a Local Peer Gateway (LPG) 2010, which may be communicatively coupled to a Secure Shell (SSH) VCN 2012 via an LPG 2010 included in SSH VCN 2012. SSH VCN 2012 may include an SSH subnet 2014, and SSH VCN 2012 may be communicatively coupled to a Control Plane VCN 2016 via an LPG 2010 included in a Control Plane VCN 2016. Furthermore, SSH VCN 2012 may be communicatively coupled to a Data Plane VCN 2018 via an LPG 2010. Control Plane VCN 2016 and Data Plane VCN 2018 may be included in a Service Lease 2019 that may be owned and / or operated by an IaaS provider.

[0312] The control plane VCN 2016 may include a control plane demilitarized zone (DMZ) layer 2020 that acts as a peripheral network (e.g., a portion of a corporate network between a corporate intranet and an external network). DMZ-based servers can assume limited liability and help control security vulnerabilities. Furthermore, the DMZ layer 2020 may include one or more load balancer (LB) subnets 2022, a control plane application layer 2024 that may include one or more application subnets 2026, and a control plane data layer 2028 that may include one or more database (DB) subnets 2030 (e.g., one or more front-end database subnets and / or one or more back-end database subnets). One or more LB subnets 2022 included in the control plane DMZ layer 2020 can be communicatively coupled to one or more application subnets 2026 included in the control plane application layer 2024 and an Internet gateway 2034 that can be included in the control plane VCN 2016. The application subnets 2026 can also be communicatively coupled to one or more DB subnets 2030 included in the control plane data layer 2028, as well as a service gateway 2036 and a Network Address Translation (NAT) gateway 2038. The control plane VCN 2016 may include the service gateway 2036 and the NAT gateway 2038.

[0313] The control plane VCN 2016 may include a data plane mirror application layer 2040, which may include one or more application subnets 2026. The one or more application subnets 2026 included in the data plane mirror application layer 2040 may include a virtual network interface controller (VNIC) 2042 capable of executing a compute instance 2044. The compute instance 2044 may communicatively couple the one or more application subnets 2026 of the data plane mirror application layer 2040 to the one or more application subnets 2026 that may be included in the data plane application layer 2046.

[0314] Data plane VCN 2018 may include a data plane application layer 2046, a data plane DMZ layer 2048, and a data plane data layer 2050. Data plane DMZ layer 2048 may include one or more LB subnets 2022 communicatively coupled to one or more application subnets 2026 of data plane application layer 2046 and an Internet gateway 2034 of data plane VCN 2018. One or more application subnets 2026 may be communicatively coupled to a service gateway 2036 and a NAT gateway 2038 of data plane VCN 2018. Data plane data layer 2050 may also include one or more DB subnets 2030 communicatively coupled to one or more application subnets 2026 of data plane application layer 2046.

[0315] The Internet gateway 2034 of the control plane VCN 2016 and data plane VCN 2018 can be communicatively coupled to the metadata management service 2052, which in turn can be communicatively coupled to the public Internet 2054. The public Internet 2054 can be communicatively coupled to the NAT gateway 2038 of the control plane VCN 2016 and data plane VCN 2018. The service gateway 2036 of the control plane VCN 2016 and data plane VCN 2018 can be communicatively coupled to the cloud service 2056.

[0316] In some examples, the service gateway 2036 of the control plane VCN 2016 or data plane VCN 2018 can make application programming interface (API) calls to the cloud service 2056 without traversing the public internet 2054. API calls from the service gateway 2036 to the cloud service 2056 can be unidirectional: the service gateway 2036 can make API calls to the cloud service 2056, and the cloud service 2056 can send requested data to the service gateway 2036. However, the cloud service 2056 cannot initiate API calls to the service gateway 2036.

[0317] In some examples, Secure Host Tenancy 2004 can connect directly to Service Tenancy 2019; otherwise, Service Tenancy 2019 can be isolated. Secure Host Subnet 2008 can communicate with SSH Subnet 2014 via LPG 2010, which can enable bidirectional communication with other isolated systems. Connecting Secure Host Subnet 2008 to SSH Subnet 2014 allows Secure Host Subnet 2008 to access other entities within Service Tenancy 2019.

[0318] The control plane VCN 2016 allows users of Service Tenant 2019 to configure or otherwise provision desired resources. Desired resources provisioned in the control plane VCN 2016 can be deployed in the data plane VCN 2018 or otherwise used. In some examples, the control plane VCN 2016 can be isolated from the data plane VCN 2018, and the data plane mirror application layer 2040 of the control plane VCN 2016 can communicate with the data plane application layer 2046 of the data plane VCN 2018 via VNIC 2042, which can be included in both the data plane mirror application layer 2040 and the data plane application layer 2046.

[0319] In some examples, a user or client of the system may issue a request, such as a Create, Read, Update, or Delete (CRUD) operation, via the public internet 2054. The public internet 2054 can then forward the request to the metadata management service 2052. The metadata management service 2052 can forward the request to the control plane VCN 2016 via the internet gateway 2034. This request can be received by one or more LB subnets 2022 contained in the control plane DMZ layer 2020. The one or more LB subnets 2022 can determine that the request is valid, and in response to this determination, the one or more LB subnets 2022 can forward the request to one or more application subnets 2026 contained in the control plane application layer 2024. If the request is authenticated and requires an invocation of the public internet 2054, then the invocation of the public internet 2054 can be forwarded to the NAT gateway 2038 that can invoke the public internet 2054. The request may expect the storage to be located in one or more DB subnets 2030.

[0320] In some examples, the data plane mirroring application layer 2040 can facilitate direct communication between the control plane VCN 2016 and the data plane VCN 2018. For example, it may be desirable to apply configuration changes, updates, or other verified modifications to resources contained in the data plane VCN 2018. Through VNIC 2042, the control plane VCN 2016 can communicate directly with the resources contained in the data plane VCN 2018, thereby performing configuration changes, updates, or other appropriate modifications.

[0321] In some embodiments, the control plane VCN 2016 and data plane VCN 2018 may be included in a service lease 2019. In this case, the system's users or customers may not own or operate the control plane VCN 2016 or the data plane VCN 2018. Alternatively, the IaaS provider may own or operate both the control plane VCN 2016 and the data plane VCN 2018, both of which may be included in the service lease 2019. This embodiment can enable network isolation, which can prevent users or customers from interacting with the resources of other users or customers. Moreover, this embodiment can allow the system's users or customers to privately store databases without relying on the public Internet 2054, which may not have the desired level of security for storage.

[0322] In other embodiments, one or more LB subnets 2022 included in the control plane VCN 2016 may be configured to receive signals from the service gateway 2036. In this embodiment, the control plane VCN 2016 and the data plane VCN 2018 may be configured to be invoked by the IaaS provider's customers without invoking the public internet 2054. The IaaS provider's customers may expect this embodiment because the database(s) used by the customer can be controlled by the IaaS provider and can be stored on a service lease 2019, which may be isolated from the public internet 2054.

[0323] Figure 21 This is a block diagram 2100 illustrating another example pattern of an IaaS architecture according to at least one embodiment. Service operator 2102 (e.g., Figure 20 Service operators (2002) can communicatively couple to secure host rentals (2104) (e.g., Figure 20 The secure hosting lease 2004), the secure hosting lease 2104 may include a virtual cloud network (VCN) 2106 (e.g., Figure 20 VCN 2006) and Secure Host Subnet 2108 (e.g., Figure 20 The secure host subnet 2008). VCN 2106 may include a local peering gateway (LPG) 2110 (e.g., Figure 20 The LPG 2010, which can be communicatively coupled to the SSH VCN 2112 (e.g., LPG 2010 contained in the Secure Shell (SSH) VCN 2112) Figure 20 SSH VCN 2112 can include SSH subnet 2114 (e.g., Figure 20 The SSH subnet 2014), and the SSH VCN 2112 can be communicatively coupled to the control plane VCN 2116 via the LPG 2110 contained in the control plane VCN 2116 (e.g., Figure 20 Control plane VCN 2116). Control plane VCN 2116 may be included in service lease 2119 (e.g., Figure 20 In the Service Leasing 2019), the data plane VCN 2118 (for example, Figure 20 The data plane (VCN 2018) can be included in a customer lease 2121 that can be owned or operated by the system's users or customers.

[0324] The control plane VCN 2116 may include one or more LB subnets 2122 (e.g., Figure 20 The control plane DMZ layer 2120 of (one or more) LB subnets 2022 (e.g., Figure 20 The control plane DMZ layer 2020), may include (one or more) application subnets 2126 (e.g., Figure 20 The control plane application layer 2124 of (one or more) application subnets 2026 (e.g., Figure 20 The control plane application layer 2024 may include one or more database (DB) subnets 2130 (e.g., similar to...). Figure 20 The control plane data layer 2128 of (one or more) DB subnets 2030 (e.g., Figure 20 The control plane data layer 2028). One or more LB subnets 2122 contained in the control plane DMZ layer 2120 can be communicatively coupled to one or more application subnets 2126 contained in the control plane application layer 2124 and an Internet gateway 2134 that can be contained in the control plane VCN 2116 (e.g., Figure 20 Internet gateway 2034), and application subnet(s) 2126 can communicatively couple to DB subnet(s) 2130 contained in control plane data layer 2128 and service gateway 2136 (e.g., Figure 20 The service gateway) and Network Address Translation (NAT) gateway 2138 (e.g., Figure 20 (NAT gateway 2038). The control plane VCN 2116 may include the service gateway 2136 and the NAT gateway 2138.

[0325] The control plane VCN 2116 may include a data plane mirror of the application layer 2140 (e.g., Figure 20 The data plane mirror application layer 2040 may include one or more application subnets 2126. The application subnets 2126 included in the data plane mirror application layer 2140 may include instances 2144 capable of performing computation (e.g., similar to...). Figure 20 The virtual network interface controller (VNIC) 2142 (e.g., the VNIC of 2042) of the computing instance 2044. The computing instance 2144 may facilitate the mirroring of one or more application subnets 2126 of the application layer 2140 in the data plane with those that may be contained in the application layer 2146 in the data plane (e.g., Figure 20 Communication between one or more application subnets 2126 in the data plane application layer 2046 via VNIC 2142 contained in the data plane mirror application layer 2140 and VNIC 2142 contained in the data plane application layer 2146.

[0326] The Internet gateway 2134 included in the control plane VCN 2116 can be communicatively coupled to the metadata management service 2152 (e.g., Figure 20 Metadata management service 2052), metadata management service 2152 can communicatively couple to public Internet 2154 (e.g., Figure 20 The public internet 2154 can communicatively couple to a NAT gateway 2138 contained in a control plane VCN 2116. The service gateway 2136 contained in the control plane VCN 2116 can communicatively couple to a cloud service 2156 (e.g., ...). Figure 20 Cloud services 2056).

[0327] In some examples, data plane VCN 2118 may be included in customer lease 2121. In this case, the IaaS provider may provide control plane VCN 2116 for each customer, and the IaaS provider may set up a unique compute instance 2144 for each customer, included in service lease 2119. Each compute instance 2144 may allow communication between control plane VCN 2116 included in service lease 2119 and data plane VCN 2118 included in customer lease 2121. Compute instance 2144 may allow resources provisioned in control plane VCN 2116 included in service lease 2119 to be deployed or otherwise used in data plane VCN 2118 included in customer lease 2121.

[0328] In other examples, an IaaS provider's customer may have a database residing in customer tenant 2121. In this example, control plane VCN 2116 may include data plane mirror application layer 2140, which may include one or more application subnets 2126. Data plane mirror application layer 2140 may reside in data plane VCN 2118, but it may not reside in data plane VCN 2118. That is, data plane mirror application layer 2140 may have access to customer tenant 2121, but it may not reside in data plane VCN 2118 or be owned or operated by the IaaS provider's customer. Data plane mirror application layer 2140 may be configured to invoke data plane VCN 2118, but it cannot be configured to invoke any entity contained in control plane VCN 2116. Customers may expect to deploy or otherwise use the resources in the data plane VCN 2118 provided in the control plane VCN 2116, and the data plane mirroring application layer 2140 can facilitate the customer's expected deployment or other use of resources.

[0329] In some embodiments, an IaaS provider's customer may apply filters to data plane VCN 2118. In this embodiment, the customer may determine what data plane VCN 2118 can access, and the customer may restrict access to the public internet 2154 from data plane VCN 2118. The IaaS provider may not be able to apply filters or otherwise control data plane VCN 2118's access to any external networks or databases. Applying filters and controls to data plane VCN 2118 included in customer lease 2121 can help isolate data plane VCN 2118 from other customers and the public internet 2154.

[0330] In some embodiments, cloud service 2156 may be invoked by service gateway 2136 to access services that may not exist on public internet 2154, control plane VCN 2116, or data plane VCN 2118. The connection between cloud service 2156 and control plane VCN 2116 or data plane VCN 2118 may not be real-time or continuous. Cloud service 2156 may reside on different networks owned or operated by an IaaS provider. Cloud service 2156 may be configured to receive calls from service gateway 2136 and may be configured not to receive calls from public internet 2154. Some cloud services 2156 may be isolated from other cloud services 2156, and control plane VCN 2116 may be isolated from cloud services 2156 that may not be in the same region as control plane VCN 2116. For example, control plane VCN 2116 may be located in "Region 1," and cloud service "Deployment 20" may be located in both "Region 1" and "Region 2." If deployment 20 is invoked by service gateway 2136 contained in control plane VCN 2116 located in region 1, then the invocation can be transmitted to deployment 20 in region 1. In this example, control plane VCN 2116 or deployment 20 in region 1 may be coupled to deployment 20 in region 2 without communication or may otherwise communicate.

[0331] Figure 22 This is a block diagram 2200 illustrating another example pattern of an IaaS architecture according to at least one embodiment. Service operator 2202 (e.g., Figure 20 Service operators (2002) can communicatively couple to secure host rentals (2204) (e.g., Figure 20 The secure hosting lease 2204 may include a virtual cloud network (VCN) 2206 (e.g., Figure 20 VCN 2006) and Secure Host Subnet 2208 (e.g., Figure 20 Secure Host Subnet 2008). VCN 2206 can include LPG 2210 (e.g., Figure 20 The LPG 2210 can be used via SSH VCN 2212 (e.g., LPG 2210), which is included in SSH VCN 2212. Figure 20 The LPG 2210 in SSH VCN 2212 is communicatively coupled to SSH VCN 2212. SSH VCN 2212 may include SSH subnet 2214 (e.g., Figure 20 The SSH subnet 2014), and the SSH VCN 2212 can be communicatively coupled to the control plane VCN 2216 via the LPG 2210 contained in the control plane VCN 2216 (e.g., Figure 20 The control plane VCN 2216) and coupled to the data plane VCN 2218 via the LPG 2210 contained in the data plane VCN 2218 (e.g., Figure 20 Data plane 2218). Control plane VCN 2216 and data plane VCN 2218 may be included in service lease 2219 (e.g., Figure 20 In the service rental (2019).

[0332] The control plane VCN 2216 may include a load balancer (LB) subnet 2222 (e.g., Figure 20 The control plane DMZ layer 2220 of (one or more) LB subnets 2022 (e.g., Figure 20 The control plane DMZ layer 2020 may include one or more application subnets 2226 (e.g., similar to...). Figure 20 The control plane application layer 2224 of (one or more) application subnets 2026 (e.g., Figure 20 The control plane application layer 2024), and may include (one or more) DB subnets 2230, and the control plane data layer 2228 (e.g., Figure 20 The control plane data layer 2228). One or more LB subnets 2222 contained in the control plane DMZ layer 2220 can be communicatively coupled to one or more application subnets 2226 contained in the control plane application layer 2224 and an Internet gateway 2234 that can be contained in the control plane VCN 2216 (e.g., Figure 20 Internet gateway 2034), and application subnet(s) 2226 can communicatively couple to DB subnet(s) 2230 contained in control plane data layer 2228 and service gateway 2236 (e.g., Figure 20 The service gateway) and Network Address Translation (NAT) gateway 2238 (e.g., Figure 20 (NAT gateway 2038). The control plane VCN 2216 may include the service gateway 2236 and the NAT gateway 2238.

[0333] The data plane VCN 2218 may include the data plane application layer 2246 (e.g., Figure 20 Data plane application layer 2046), data plane DMZ layer 2248 (e.g., Figure 20 Data plane DMZ layer 2048), and data plane data layer 2250 (e.g., Figure 20 The data plane data layer 2050). The data plane DMZ layer 2248 may include one or more trusted application subnets 2260 and one or more untrusted application subnets 2262 that can be communicatively coupled to the data plane application layer 2246, and one or more LB subnets 2222 of the Internet gateway 2234 contained in the data plane VCN 2218.

[0334] One or more trusted application subnets 2260 may be communicatively coupled to a service gateway 2236 contained in data plane VCN 2218, a NAT gateway 2238 contained in data plane VCN 2218, and one or more DB subnets 2230 contained in data plane data layer 2250. One or more untrusted application subnets 2262 may be communicatively coupled to a service gateway 2236 contained in data plane VCN 2218 and one or more DB subnets 2230 contained in data plane data layer 2250. Data plane data layer 2250 may include one or more DB subnets 2230 that may be communicatively coupled to a service gateway 2236 contained in data plane VCN 2218.

[0335] One or more untrusted application subnets 2262 may include one or more primary VNICs 2264(1)-(N) that are communicatively coupled to tenant virtual machines (VMs) 2266(1)-(N). Each tenant VM 2266(1)-(N) may be communicatively coupled to a corresponding application subnet 2267(1)-(N) that may be contained in a corresponding container egress VCN 2268(1)-(N), which may be contained in a corresponding customer lease 2270(1)-(N). A corresponding secondary VNIC 2272(1)-(N) may facilitate communication between the one or more untrusted application subnets 2262 contained in the data plane VCN 2218 and the application subnets contained in the container egress VCN 2268(1)-(N). Each container egress VCN 2268(1)-(N) may include a NAT gateway 2238 that can communicatively couple to the public internet 2254 (e.g., Figure 20 The public internet in 2054).

[0336] Internet gateway 2234, contained in control plane VCN 2216 and data plane VCN 2218, can be communicatively coupled to metadata management service 2252 (e.g., Figure 20 A metadata management system 2252 is communicatively coupled to a public internet 2254. The public internet 2254 is communicatively coupled to a NAT gateway 2238 contained in a control plane VCN 2216 and a data plane VCN 2218. A service gateway 2236 contained in the control plane VCN 2216 and the data plane VCN 2218 is communicatively coupled to a cloud service 2256.

[0337] In some embodiments, the data plane VCN 2218 may be integrated with the customer lease 2270. Such integration may be useful or desired by the IaaS provider's customers in certain situations, such as when support may be expected during code execution. Customers may provide code that could be destructive, might communicate with other customer resources, or might otherwise cause undesirable effects. In response, the IaaS provider may determine whether to run the code provided by the customer.

[0338] In some examples, an IaaS provider's customer can grant temporary network access to the IaaS provider and request functionality to be attached to data plane layer application 2246. The code running this functionality can execute in VM 2266(1)-(N) and the code does not need to be configured to run anywhere else on data plane VCN 2218. Each VM 2266(1)-(N) can be connected to a customer lease 2270. The corresponding container 2271(1)-(N) contained in VM 2266(1)-(N) can be configured to run the code. In this scenario, dual isolation can exist (e.g., container 2271(1)-(N) runs code, where container 2271(1)-(N) may be contained at least within VM 2266(1)-(N), which is contained within one or more untrusted application subnets 2262), which can help prevent incorrect or otherwise undesirable code from damaging the IaaS provider's network or the networks of different customers. Container 2271(1)-(N) may be communicatively coupled to customer lease 2270 and may be configured to transfer or receive data from customer lease 2270. Container 2271(1)-(N) may not be configured to transfer or receive data from any other entity in the data plane VCN 2218. After the code runs, the IaaS provider may kill or otherwise dispose of container 2271(1)-(N).

[0339] In some embodiments, one or more trusted application subnets 2260 may run code that can be owned or operated by an IaaS provider. In this embodiment, one or more trusted application subnets 2260 may be communicatively coupled to one or more database subnets 2230 and configured to perform CRUD operations in one or more database subnets 2230. One or more untrusted application subnets 2262 may be communicatively coupled to one or more database subnets 2230, but in this embodiment, one or more untrusted application subnets may be configured to perform read operations in one or more database subnets 2230. Containers 2271(1)-(N) that may be contained in each customer's VM 2266(1)-(N) and may run code from the customer may not be communicatively coupled to one or more database subnets 2230.

[0340] In other embodiments, the control plane VCN 2216 and the data plane VCN 2218 may be coupled without direct communication. In this embodiment, there may be no direct communication between the control plane VCN 2216 and the data plane VCN 2218. However, communication may occur indirectly through at least one method. The LPG 2210 may be established by an IaaS provider, which can facilitate communication between the control plane VCN 2216 and the data plane VCN 2218. In another example, either the control plane VCN 2216 or the data plane VCN 2218 may invoke the cloud service 2256 via the service gateway 2236. For example, an invocation from the control plane VCN 2216 to the cloud service 2256 may include a request for a service that can communicate with the data plane VCN 2218.

[0341] Figure 23 This is a block diagram 2300 illustrating another example pattern of an IaaS architecture according to at least one embodiment. Service operator 2302 (e.g., Figure 20 Service operators (2002) can communicatively couple to secure host rental (2304) (e.g., Figure 20 Secure hosting rental 2004), Secure hosting rental 2304 may include Virtual Cloud Network (VCN) 2306 (e.g., Figure 20 VCN 2006) and Secure Host Subnet 2308 (e.g., Figure 20 Secure Host Subnet 2008). VCN 2306 can include LPG 2310 (e.g., Figure 20 The LPG 2310 can be communicatively coupled to the SSH VCN 2312 via the LPG 2310 included in the SSH VCN 2312 (e.g., LPG 2310). Figure 20 SSH VCN 2012). SSH VCN 2312 can include SSH subnet 2314 (e.g., Figure 20 The SSH subnet 2014), and the SSH VCN 2312 can be communicatively coupled to the control plane VCN 2316 via the LPG 2310 contained in the control plane VCN 2316 (e.g., Figure 20 The control plane VCN 2316) and coupled to the data plane VCN 2318 via the LPG 2310 contained in the data plane VCN 2318 (e.g., Figure 20 Data plane 2018). Control plane VCN 2316 and data plane VCN 2318 may be included in service lease 2319 (e.g., Figure 20 In the service rental (2019).

[0342] The control plane VCN 2316 may include: one or more LB subnets 2322 (e.g., Figure 20 The control plane DMZ layer 2320 of (one or more) LB subnets 2022 (e.g., Figure 20 The control plane DMZ layer 2020), may include application subnet 2326 (e.g., Figure 20 Application plane application layer 2324 of the application subnet 2026 (e.g., Figure 20 The control plane application layer 2024) may include (one or more) DB subnets 2330 (e.g., Figure 22 The control plane data layer 2328 of (one or more) DB subnets 2230 (e.g., Figure 20 The control plane data layer 2328). One or more LB subnets 2322 contained in the control plane DMZ layer 2320 can be communicatively coupled to one or more application subnets 2326 contained in the control plane application layer 2324 and to an Internet gateway 2334 that can be contained in the control plane VCN 2316 (e.g., Figure 20 Internet gateway 2034), and application subnet(s) 2326 can communicatively couple to DB subnet(s) 2330 contained in control plane data layer 2328 and to service gateway 2336 (e.g., Figure 20 The service gateway) and Network Address Translation (NAT) gateway 2338 (e.g., Figure 20 (NAT gateway 2038). The control plane VCN 2316 may include the service gateway 2336 and the NAT gateway 2338.

[0343] The data plane VCN 2318 may include the data plane application layer 2346 (e.g., Figure 20 Data plane application layer 2046), data plane DMZ layer 2348 (e.g., Figure 20 Data plane DMZ layer 2048), and data plane data layer 2350 (e.g., Figure 20 The data plane data layer 2050). The data plane DMZ layer 2348 may include one or more LB subnets 2322, which may be communicatively coupled to one or more trusted application subnets 2360 of the data plane application layer 2346 (e.g., Figure 22 (one or more) trusted application subnets 2260) and (one or more) untrusted application subnets 2362 (e.g., Figure 22 The data plane VCN 2318 may include one or more untrusted application subnets 2262 and an Internet gateway 2334. One or more trusted application subnets 2360 may communicatively couple to a service gateway 2336, a NAT gateway 2338, and a DB subnet 2330 contained in the data plane VCN 2318. One or more untrusted application subnets 2362 may communicatively couple to a service gateway 2336 and a DB subnet 2330 contained in the data plane VCN 2318 and the data plane data layer 2350. The data plane data layer 2350 may include one or more DB subnets 2330 that may communicatively couple to a service gateway 2336 contained in the data plane VCN 2318.

[0344] One or more untrusted application subnets 2362 may include a primary VNIC 2364(1)-(N) communicatively coupled to tenant virtual machines (VMs) 2366(1)-(N) residing within one or more untrusted application subnets 2362. Each tenant VM 2366(1)-(N) may run code in a corresponding container 2367(1)-(N) and communicatively coupled to an application subnet 2326 that may be included in a data plane application layer 2346, which may be included in a container egress VCN 2368. A corresponding secondary VNIC 2372(1)-(N) may facilitate communication between one or more untrusted application subnets 2362 included in the data plane VCN 2318 and the application subnets included in the container egress VCN 2368. The container egress VCN may include a primary VNIC 2364(1)-(N) communicatively coupled to the public Internet 2354 (e.g., Figure 20 NAT gateway 2338 for the public Internet (2054).

[0345] Internet gateway 2334, contained in control plane VCN 2316 and data plane VCN 2318, can be communicatively coupled to metadata management service 2352 (e.g., Figure 20 A metadata management system 2052, which is communicatively coupled to the public internet 2354, is also communicatively coupled to a NAT gateway 2338 contained in a control plane VCN 2316 and a data plane VCN 2318. A service gateway 2336 contained in a control plane VCN 2316 and a data plane VCN 2318 is communicatively coupled to a cloud service 2356.

[0346] In some examples, Figure 23 The architecture shown in the block diagram 2300 can be considered as... Figure 22 This is an exception to the pattern shown in the architecture diagram 2200, and this pattern may be what the IaaS provider's customers would expect if the IaaS provider cannot communicate directly with the customer (e.g., in a disconnected region). The customer can access in real time the corresponding container 2367(1)-(N) contained in each customer's VM 2366(1)-(N). Container 2367(1)-(N) can be configured to invoke a corresponding auxiliary VNIC 2372(1)-(N) contained in one or more application subnets 2326 of the data plane application layer 2346, which may be contained in the container egress VCN 2368. The auxiliary VNIC 2372(1)-(N) can transmit the call to a NAT gateway 2338, which can then transmit the call to the public internet 2354. In this example, containers 2367(1)-(N), which can be accessed by clients in real time, can be isolated from the control plane VCN 2316 and from other entities contained in the data plane VCN 2318. Containers 2367(1)-(N) can also be isolated from resources from other clients.

[0347] In other examples, a client can use container 2367(1)-(N) to invoke cloud service 2356. In this example, the client can run code requesting services from cloud service 2356 within container 2367(1)-(N). Container 2367(1)-(N) can then transmit the request to auxiliary VNIC 2372(1)-(N), which can then transmit the request to a NAT gateway, which can then transmit the request to the public internet 2354. The public internet 2354 can then transmit the request via internet gateway 2334 to one or more LB subnets 2322 contained in control plane VCN 2316. In response to determining that the request is valid, the one or more LB subnets can then transmit the request to one or more application subnets 2326, which can then transmit the request to cloud service 2356 via service gateway 2336.

[0348] It should be recognized that the IaaS architectures 2000, 2100, 2200, and 2300 depicted in the figures may have other components besides those depicted. Furthermore, the embodiments shown in the figures are merely some examples of cloud infrastructure systems that can be incorporated into embodiments of this disclosure. In some other embodiments, the IaaS system may have more or fewer components than shown in the figures, may combine two or more components, or may have different configurations or component arrangements.

[0349] In some embodiments, the IaaS system described herein may include application suites, middleware, and database service offerings delivered to customers in a self-service, subscription-based, elastically scalable, reliable, highly available, and secure manner. An example of such an IaaS system is the Oracle Cloud Infrastructure (OCI) provided by this assignee.

[0350] Figure 24 An example computer system 2400 in which various embodiments can be implemented is illustrated. System 2400 can be used to implement any of the computer systems described above. As shown, computer system 2400 includes a processing unit 2404 that communicates with a plurality of peripheral subsystems via a bus subsystem 2402. These peripheral subsystems may include a processing acceleration unit 2406, an I / O subsystem 2408, a storage subsystem 2418, and a communication subsystem 2424. Storage subsystem 2418 includes a tangible computer-readable storage medium 2422 and system memory 2410.

[0351] Bus subsystem 2402 provides a mechanism for allowing various components and subsystems of computer system 2400 to communicate with each other as intended. While bus subsystem 2402 is schematically shown as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses. Bus subsystem 2402 can be any of several types of bus architectures, including memory buses or memory controllers, peripheral buses, and local buses using any of the various bus architectures. For example, such architectures may include Industry Standard Architecture (ISA) buses, Microchannel Architecture (MCA) buses, Enhanced ISA (EISA) buses, Video Electronics Standards Association (VESA) local buses, and Peripheral Component Interconnect (PCI) buses, which may be implemented as Mezzanine buses manufactured according to the IEEE P1386.1 standard.

[0352] A processing unit 2404, which may be implemented as one or more integrated circuits (e.g., a conventional microprocessor or microcontroller), controls the operation of the computer system 2400. One or more processors may be included in the processing unit 2404. These processors may include single-core or multi-core processors. In some embodiments, the processing unit 2404 may be implemented as one or more independent processing units 2432 and / or 2434, wherein each processing unit includes a single-core or multi-core processor. In other embodiments, the processing unit 2404 may also be implemented as a quad-core processing unit formed by integrating two dual-core processors into a single chip.

[0353] In various embodiments, processing unit 2404 can execute various programs in response to program code and can maintain multiple concurrently executing programs or processes. At any given time, some or all of the program code to be executed can reside in processor(s) 2404 and / or storage subsystem 2418. With appropriate programming, processor(s) 2404 can provide the various functions described above. Computer system 2400 may additionally include processing acceleration unit 2406, which may include digital signal processor (DSP), dedicated processor, etc.

[0354] The I / O subsystem 2408 may include user interface input devices and user interface output devices. User interface input devices may include keyboards, pointing devices such as mice or trackballs, touchpads or touchscreens integrated into a display, scroll wheels, click wheels, dials, buttons, switches, audio input devices with voice command recognition systems, microphones, and other types of input devices. User interface input devices may include, for example, motion sensing and / or gesture recognition devices, such as those from Microsoft… Motion sensors enable users to control devices such as Microsoft products via a natural user interface using gestures and voice commands. The 360 ​​game controller's input device interacts with it. The user interface input device may also include eye gesture recognition devices, such as detecting eye movements from the user (e.g., "blinking" when taking a photo and / or making a menu selection) and translating the eye gestures into the input device (e.g., Google). Google input in ) Blink detector. Additionally, the user interface input device may include enabling the user to interact with a voice recognition system (e.g., ...) via voice commands. Voice recognition sensing devices for interaction with navigators.

[0355] User interface input devices may also include, but are not limited to, 3D mice, joysticks or pointing sticks, game panels and drawing tablets, as well as audio / video devices such as speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, barcode readers, 3D scanners, 3D printers, laser rangefinders, and eye-tracking devices. Furthermore, user interface input devices may include, for example, medical imaging input devices such as computed tomography, magnetic resonance imaging, positron emission tomography, or medical ultrasound equipment. User interface input devices may also include, for example, audio input devices such as MIDI keyboards, digital musical instruments, etc.

[0356] User interface output devices may include display subsystems, indicator lights, or non-visual displays such as audio output devices, etc. Display subsystems may be cathode ray tubes (CRTs), flat panel devices such as those using liquid crystal displays (LCDs) or plasma displays, projection devices, touchscreens, etc. Generally, the term "output device" is intended to include all possible types of devices and mechanisms for outputting information from computer system 2400 to the user or other computers. For example, user interface output devices may include, but are not limited to, various display devices that visually convey text, graphics, and audio / video information, such as monitors, printers, speakers, headphones, car navigation systems, plotters, voice output devices, and modems.

[0357] Computer system 2400 may include a storage subsystem 2418 containing software elements, shown as currently located in system memory 2410. System memory 2410 may store program instructions that can be loaded and executed on processing unit 2404, as well as data generated during the execution of these programs.

[0358] Depending on the configuration and type of the computer system 2400, the system memory 2410 may be volatile (such as random access memory (RAM)) and / or non-volatile (such as read-only memory (ROM), flash memory, etc.). RAM typically contains data and / or program modules that can be immediately accessed by the processing unit 2404 and / or are currently being operated and executed by the processing unit 2404. In some implementations, the system memory 2410 may include various different types of memory, such as static random access memory (SRAM) or dynamic random access memory (DRAM). In some implementations, a basic input / output system (BIOS), which contains basic routines that facilitate the transfer of information between the elements of the computer system 2400 during startup, may typically be stored in ROM. As an example, but not a limitation, the system memory 2410 also includes application programs 2412, program data 2414, and an operating system 2416, which may include client applications, web browsers, middleware applications, relational database management systems (RDBMS), etc. As an example, the operating system 2416 may include various versions of Microsoft... Apple and / or Linux operating system, and various commercially available... Or a UNIX-like operating system (including but not limited to various GNU / Linux operating systems, Google...) OS, etc.) and / or such as iOS, Phone OS 24OS and A mobile operating system based on the OS operating system.

[0359] Storage subsystem 2418 may also provide a tangible computer-readable storage medium for storing basic programming and data structures that provide the functionality of some embodiments. Software (programs, code modules, instructions) that provides the above-described functionality when executed by a processor may be stored in storage subsystem 2418. These software modules or instructions may be executed by processing unit 2404. Storage subsystem 2418 may also provide a repository for storing data used according to this disclosure.

[0360] Storage subsystem 2400 may also include a computer-readable storage medium reader 2420 that can be further connected to computer-readable storage medium 2422. Together with and optionally in conjunction with system memory 2410, computer-readable storage medium 2422 can comprehensively represent a remote, local, fixed, and / or removable storage device plus storage medium for temporarily and / or more persistently containing, storing, transmitting, and retrieving computer-readable information.

[0361] The computer-readable storage medium 2422 containing code or portions thereof may also include any suitable medium known or used in the art, including storage and communication media, such as, but not limited to, volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing and / or transmitting information. This may include tangible computer-readable storage media such as RAM, ROM, electrically erasable programmable ROM (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical storage, magnetic tape cassette, magnetic tape, disk storage or other magnetic storage devices, or other tangible computer-readable media. This may also include non-tangible computer-readable media such as data signals, data transmissions, or any other medium that can be used to transmit desired information and can be accessed by the computing system 2400.

[0362] For example, computer-readable storage medium 2422 may include a hard disk drive that reads or writes to a non-removable non-volatile magnetic medium, a disk drive that reads or writes to a removable non-volatile magnetic disk, and a removable non-volatile optical disc (such as a CD-ROM, DVD, etc.). An optical disc drive that reads from or writes to a disk or other optical medium. Computer-readable storage medium 2422 may include, but is not limited to, Disk drives, flash memory cards, Universal Serial Bus (USB) flash drives, Secure Digital (SD) cards, DVDs, digital audio tapes, and so on. Computer-readable storage media 2422 may also include solid-state drives (SSDs) based on non-volatile memory (such as flash memory-based SSDs, enterprise flash drives, solid-state ROMs, etc.), volatile memory-based SSDs (such as solid-state RAM, dynamic RAM, static RAM), DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs using a combination of DRAM-based and flash memory-based SSDs. Disk drives and their associated computer-readable media can provide non-volatile storage of computer-readable instructions, data structures, program modules, and other data for computer system 2400.

[0363] The communication subsystem 2424 provides an interface to other computer systems and networks. The communication subsystem 2424 serves as an interface for receiving data from other systems and sending data from computer system 2400 to other systems. For example, the communication subsystem 2424 enables computer system 2400 to connect to one or more devices via the Internet. In some embodiments, the communication subsystem 2424 may include radio frequency (RF) transceiver components (e.g., advanced data network technologies using cellular telephone technologies, such as 3G, 4G, or EDGE (Enhanced Data Rates for Global Evolution), WiFi (IEEE 802.11 series standards), or other mobile communication technologies, or any combination thereof), GPS receiver components, and / or other components for accessing wireless voice and / or data networks. In some embodiments, as an addition to or alternative to the wireless interface, the communication subsystem 2424 may provide a wired network connection (e.g., Ethernet).

[0364] In some embodiments, the communication subsystem 2424 may also represent one or more users who can use the computer system 2400 to receive input communications in the form of structured and / or unstructured data feeds 2426, event streams 2428, event updates 2430, etc.

[0365] For example, the communication subsystem 2424 can be configured to receive data feeds 2426 in real time from users of social networks and / or other communication services, such as... feed, Updates, web feeds such as Rich Site Summary (RSS) feeds, and / or real-time updates from one or more third-party information sources.

[0366] Furthermore, the communication subsystem 2424 can also be configured to receive data in the form of a continuous data stream, which may include event streams 2428 and / or event updates 2430 that are essentially continuous or unbounded real-time events without a clearly defined termination. Examples of applications that generate continuous data may include, for example, sensor data applications, financial quote machines, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, vehicle traffic monitoring, and so on.

[0367] The communication subsystem 2424 can also be configured to output structured and / or unstructured data feeds 2426, event streams 2428, event updates 2430, etc. to one or more databases, which can communicate with one or more streaming data source computers coupled to the computer system 2400.

[0368] Computer system 2400 can be one of various types, including handheld portable devices (e.g., Cellular phone Computing tablets, PDAs), and wearable devices (e.g., Glass head-mounted displays, PCs, workstations, mainframes, information stations, server racks, or any other data processing systems.

[0369] In the foregoing description, specific details have been set forth for purposes of explanation to provide a thorough understanding of the examples of this disclosure. However, it will be apparent that various examples can be practiced without these specific details. The following description is illustrative only and is not intended to limit the scope, applicability, or configuration of this disclosure. Rather, the subsequent description of the examples will provide an enabling description for those skilled in the art to implement the examples. It should be understood that various changes can be made to the function and arrangement of elements without departing from the spirit and scope of this disclosure as set forth in the appended claims. The drawings and descriptions are not intended to be restrictive. Circuits, systems, networks, processes, and other components may be shown as components in block diagram form to avoid obscuring the examples with unnecessary details. In other cases, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary details to avoid obscuring the examples. The teachings disclosed herein can also be applied to various types of applications, such as mobile applications, non-mobile applications, desktop applications, web applications, enterprise applications, etc. Furthermore, the teachings of this disclosure are not limited to a particular operating environment (e.g., operating system, device, platform, etc.), but can instead be applied to multiple different operating environments.

[0370] Furthermore, it should be noted that individual examples can be described as processes, which may be represented as flowcharts, flow diagrams, data flow diagrams, structure diagrams, or block diagrams. While flowcharts can describe operations as sequential processes, many operations can be executed in parallel or concurrently. Additionally, the order of operations can be rearranged. A process terminates when its operations are completed, but a process can have additional steps not included in the diagram. Processes can correspond to methods, functions, procedures, subroutines, subroutines, etc. When a process corresponds to a function, its termination can correspond to the function returning to the calling function or the main function.

[0371] The terms "example" and "exemplary" are used herein to mean "serving as an example, instance, or illustration." Any embodiment or design described herein as "exemplary" or "example" is not necessarily to be construed as preferred or advantageous over other embodiments or designs.

[0372] The terms "machine-readable storage medium" or "computer-readable storage medium" include, but are not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying one or more instructions and / or data. Machine-readable or computer-readable storage media may include non-transitory media in which data can be stored but do not include carrier waves and / or transient electronic signals propagated wirelessly or via a wired connection. Examples of non-transitory media may include, but are not limited to, disks or tapes, optical storage media such as CDs or DVDs, flash memory, or memory or storage devices thereof. Computer program products may include code and / or machine-executable instructions that may represent procedures, functions, subroutines, programs, routines, subroutines, modules, software packages, classes, or any combination of instructions, data structures, or program statements. Code segments may be coupled to another code segment or hardware circuitry by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc., may be passed, forwarded, or transmitted via any suitable means, including memory sharing, messaging, token passing, network transmission, etc.

[0373] Furthermore, the examples can be implemented using hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented as software, firmware, middleware, or microcode, program code or code segments (e.g., a computer program product) that perform the necessary tasks can be stored on a machine-readable medium. One or more processors can perform the necessary tasks. Some of the systems depicted in these figures can be provided in various configurations. In some examples, the system can be configured as a distributed system, where one or more components of the system are distributed across one or more networks in a cloud computing system. Where a component is described as being "configured" to perform certain operations, this configuration can be accomplished, for example, by designing electronic circuits or other hardware to perform the operations, by programming or controlling electronic circuits (e.g., a microprocessor or other suitable electronic circuitry) to perform the operations, or any combination thereof.

[0374] While specific embodiments of this disclosure have been described, various modifications, alterations, alternative constructions, and equivalents are also included within the scope of this disclosure. The embodiments of this disclosure are not limited to operation within certain specific data processing environments, but can be freely operated within multiple data processing environments. Furthermore, although embodiments of this disclosure have been described using a specific series of transactions and steps, those skilled in the art will understand that the scope of this disclosure is not limited to the described series of transactions and steps. Various features and aspects of the above embodiments can be used individually or in combination.

[0375] Furthermore, while embodiments of this disclosure have been described using specific combinations of hardware and software, it should be recognized that other combinations of hardware and software are also within the scope of this disclosure. Embodiments of this disclosure may be implemented using only hardware, or only software, or a combination thereof. The various processes described herein may be implemented in any combination on the same processor or on different processors. Thus, where a component or module is described as being configured to perform certain operations, such configuration may be accomplished, for example, by designing electronic circuitry to perform the operations, by programming programmable electronic circuitry (such as a microprocessor), or any combination thereof. Processes may communicate using a variety of technologies, including but not limited to conventional technologies for inter-process communication, and different pairs of processes may use different technologies, or the same pair of processes may use different technologies at different times.

[0376] Therefore, the specification and drawings are to be considered illustrative rather than restrictive. However, it will be apparent that additions, omissions, deletions, and other modifications and changes can be made thereto without departing from the broader spirit and scope set forth in the claims. Thus, while specific disclosed embodiments have been described, they are not intended to be limiting. Various modifications and equivalents are within the scope of the following claims.

[0377] Examples of embodiments of this disclosure may be described by considering the following terms:

[0378] Clause 1. A method comprising: determining first Layer 2 information associated with a first Layer 2 virtual switch of a client's Layer 2 virtual network, wherein: the Layer 2 virtual network is hosted by a physical network and includes a first compute instance, a second compute instance, a first Layer 2 virtual network interface, a second Layer 2 virtual network interface, a first Layer 2 virtual switch, and a second Layer 2 virtual switch; the first compute instance and the second compute instance are hosted by the same host machine or different host machines of the physical network; the first Layer 2 virtual network interface and the first Layer 2 virtual switch are hosted by a network virtualization device (NVD) of the physical network and associated with the first compute instance; and the second Layer 2 virtual network interface and the second Layer 2 virtual switch are hosted by the NVD of the physical network or different NVDs and associated with the second compute instance; determining second Layer 2 information associated with the second Layer 2 virtual switch; generating third Layer 2 information based on the first Layer 2 information and the second Layer 2 information; and sending the third Layer 2 information to a client's device.

[0379] Clause 2. The method of Clause 1 further includes: receiving input from a client indicating a client configuration for the Layer 2 virtual network; and determining a mapping between the client configuration and the distribution of resources of the Layer 2 virtual network on the physical network, wherein Layer 2 information is further generated based on the mapping.

[0380] Clause 3. The method of Clause 2, wherein the customer configuration indicates a first port and a second port of a Layer 2 virtual network, wherein the mapping indicates that the first port corresponds to a first Layer 2 virtual network interface and the second port corresponds to a second Layer 2 virtual network interface, and wherein the method further includes: determining that the first Layer 2 virtual network interface is associated with a first Layer 2 virtual switch; determining that the second Layer 2 virtual network interface is associated with a second Layer 2 virtual switch; and indicating that third Layer 2 information is associated with the first port and the second port.

[0381] Clause 4. The method of Clause 2, wherein the customer configuration indicates a first port, and wherein the method further comprises: receiving a customer information query indicating a first port; determining, based on the mapping, that the first port corresponds to a Layer 2 virtual network interface; determining that the Layer 2 virtual network interface is associated with a Layer 2 virtual switch; and generating a query result based on Layer 2 information and associating the result with the first port, wherein the Layer 2 information includes the query result.

[0382] Clause 5.2 The method of which the customer configuration indicates the first port, wherein the method further comprises: receiving from the NVD first layer 2 information associated with a first layer 2 virtual switch; determining that the first layer 2 virtual switch is associated with a first layer 2 virtual network interface; determining that the first layer 2 virtual network interface corresponds to the first port based on the mapping; and including at least a portion of the first layer 2 information associated with the first port in third layer 2 information.

[0383] Clause 6. The method of Clause 5, wherein first Layer 2 information is received from the NVD based at least in part on a client’s Layer 2 information query or a change in the distribution of Layer 2 virtual network resources on the physical network.

[0384] Clause 7. The method of any one of Clauses 1-6, wherein the Layer 2 information includes at least one of the following: a Layer 2 forwarding table of the Layer 2 virtual network or statistical information about the Layer 2 virtual network.

[0385] Clause 8. The method of any one of Clauses 1-7, wherein the first layer 2 information includes a first layer 2 forwarding table of a first layer 2 virtual switch, wherein the second layer 2 information includes a second layer 2 forwarding table of a second layer 2 virtual switch, and wherein the third layer 2 information includes a third layer 2 forwarding table of a layer 2 virtual network generated based on the first and second layer 2 forwarding tables.

[0386] Clause 9. The method of Clause 8 further includes: receiving input from a client indicating a first port of a Layer 2 virtual network; determining that the first port corresponds to a Layer 2 virtual network interface; determining that the Layer 2 virtual network interface is associated with a Layer 2 virtual switch; determining, based on a Layer 2 forwarding table, that the Media Access Control (MAC) address of the Layer 2 virtual network interface is associated with a first computing instance; and including in a Layer 2 forwarding table an indication that the MAC address corresponds to the first port and is associated with the first computing instance.

[0387] The method of Clause 10. Clause 8 further includes: receiving input from a client indicating a first port of a Layer 2 virtual network; receiving a Layer 2 forwarding table query from a client indicating the first port; determining that the first port corresponds to a Layer 2 virtual network interface; determining that the Layer 2 virtual network interface is associated with a Layer 2 virtual switch; and generating a query result based on Layer 2 information and associating the result with the first port, wherein the Layer 2 information includes the query result.

[0388] The method of Clause 11.8 further includes: receiving input from a client indicating a first port of a Layer 2 virtual network; determining a change to a Layer 2 forwarding table associated with a Layer 2 virtual network interface; determining that the Layer 2 virtual network interface corresponds to the first port; associating the change with the first port; and including the change associated with the first port in a Layer 2 forwarding table.

[0389] Clause 12. The method of Clause 11, wherein the change includes at least one of the following: reassociating a first media access control (MAC) address from a Layer 2 virtual network interface to another Layer 2 virtual network interface, or associating a second MAC address with a Layer 2 virtual network interface.

[0390] Clause 13. The method of any one of Clauses 1-12, wherein the first Layer 2 information includes a first metric about a first frame stream of a first Layer 2 virtual switch, wherein the second Layer 2 information includes a second metric about a second frame stream of a second Layer 2 virtual switch, and wherein the third Layer 2 information includes statistical information about a third frame stream of a Layer 2 virtual network generated based on the first and second metrics.

[0391] The methods of Clause 14. Clause 13 also include:

[0392] Receive client input indicating the type and target of statistical information, wherein the type includes at least one of the following: number of frames, frame size, or amount of bits, and wherein the target includes at least one of the following: port, set of ports, or layer 2 virtual network.

[0393] The methods of Clause 15. Clause 13 also include:

[0394] Receive a first metric and metadata associated with the first metric from NVD, the metadata identifying at least one of a Layer 2 virtual network or a Layer 2 virtual network interface, wherein the statistics are further generated at least in part based on the metadata.

[0395] The method of Clause 16.15 further includes: receiving input from a client indicating a first port of a Layer 2 virtual network; determining, based on metadata, that a first metric is associated with a Layer 2 virtual network interface; determining that the Layer 2 virtual network interface corresponds to the first port; and associating the first metric with the first port.

[0396] Clause 17. A system comprising: one or more processors; and one or more computer-readable storage media storing instructions, which, when executed by the one or more processors, configure the system to: determine first Layer 2 information associated with a first Layer 2 virtual switch of a client's Layer 2 virtual network, wherein: the Layer 2 virtual network is hosted by a physical network and includes a first computing instance, a second computing instance, a first Layer 2 virtual network interface, a second Layer 2 virtual network interface, a first Layer 2 virtual switch, and a second Layer 2 virtual switch; the first computing instance and the second computing instance are hosted by the same host machine or different host machines of the physical network; the first Layer 2 virtual network interface and the first Layer 2 virtual switch are hosted by a network virtualization device (NVD) of the physical network and associated with the first computing instance; and the second Layer 2 virtual network interface and the second Layer 2 virtual switch are hosted by the NVD of the physical network or different NVDs and associated with the second computing instance; determine second Layer 2 information associated with the second Layer 2 virtual switch; generate third Layer 2 information based on the first Layer 2 information and the second Layer 2 information; and send the third Layer 2 information to a client's device.

[0397] Clause 18. The system of Clause 17, wherein the first layer 2 information includes the first layer 2 forwarding table of the first layer 2 virtual switch, wherein the second layer 2 information includes the second layer 2 forwarding table of the second layer 2 virtual switch, and wherein the third layer 2 information includes the third layer 2 forwarding table of the layer 2 virtual network generated based on the first layer 2 forwarding table and the second layer 2 forwarding table.

[0398] Clause 19. A non-transitory computer-readable storage medium containing one or more storage instructions that, when executed on a system, cause the system to perform operations, including: determining first Layer 2 information associated with a first Layer 2 virtual switch of a client's Layer 2 virtual network, wherein: the Layer 2 virtual network is hosted by a physical network and includes a first computing instance, a second computing instance, a first Layer 2 virtual network interface, a second Layer 2 virtual network interface, a first Layer 2 virtual switch, and a second Layer 2 virtual switch; the first computing instance and the second computing instance are hosted by the same host machine or different host machines of the physical network; the first Layer 2 virtual network interface and the first Layer 2 virtual switch are hosted by a network virtualization device (NVD) of the physical network and associated with the first computing instance; and the second Layer 2 virtual network interface and the second Layer 2 virtual switch are hosted by the NVD of the physical network or different NVDs and associated with the second computing instance; determining second Layer 2 information associated with the second Layer 2 virtual switch; generating third Layer 2 information based on the first Layer 2 information and the second Layer 2 information; and sending the third Layer 2 information to the client's device.

[0399] Clause 20. One or more non-transitory computer-readable storage media of Clause 19, wherein the first layer 2 information includes a first metric about a first frame stream of a first layer 2 virtual switch, wherein the second layer 2 information includes a second metric about a second frame stream of a second layer 2 virtual switch, and wherein the third layer 2 information includes statistical information about a third frame stream of a layer 2 virtual network generated based on the first and second metrics.

[0400] In the context of describing the disclosed embodiments (particularly in the context of the following claims), the terms "a," "an," and "the," and similar designations, are to be interpreted as covering both singular and plural, unless otherwise indicated herein or obviously contradicted by the context. Unless otherwise stated, the terms "comprising," "having," "including," and "containing" are to be interpreted as open-ended terms (i.e., meaning "including but not limited to"). The term "connected" should be interpreted as partially or wholly contained in, attached to, or joined together, even if something exists in between. Unless otherwise indicated herein, the enumeration of value ranges herein is intended only as a shorthand method for individually referencing each individual value falling within that range, and each individual value is incorporated into the specification as if it were individually enumerated herein. Unless otherwise indicated herein or obviously contradicted by the context, all methods described herein can be performed in any suitable order. The use of any and all examples or exemplary language (e.g., "such as") provided herein is intended only to better illustrate embodiments of this disclosure and does not constitute a limitation on the scope of this disclosure, unless otherwise stated. Nothing in the specification should be construed as indicating that any unclaimed element is essential to the practice of this disclosure.

[0401] Exclusion language, such as the phrase "at least one of X, Y, or Z", is intended to be understood in the context generally used to represent items, terms, etc., unless otherwise explicitly stated, and may be X, Y, or Z, or any combination thereof (e.g., X, Y, and / or Z). Therefore, such exclusion language is generally not intended to, and should not, imply that some embodiments require the presence of at least one of X, at least one of Y, or at least one of Z.

[0402] This document describes preferred embodiments of the present disclosure, including the best modes known for carrying out the present disclosure. Variations of those preferred embodiments will become apparent to those skilled in the art upon reading the foregoing description. Those skilled in the art should be able to suitably employ such variations and may practice the present disclosure in ways other than those specifically described herein. Thus, the present disclosure includes all modifications and equivalents to the subject matter recited in the appended claims, where permitted by applicable law. Furthermore, unless otherwise indicated herein, the present disclosure includes any combination of the foregoing elements in all its possible variations.

[0403] All references cited in this article, including publications, patent applications and patents, are incorporated into this article by reference to the same extent as if each reference individually and specifically indicated to be incorporated by reference and elaborated in full in this article.

[0404] In the foregoing specification, various aspects of this disclosure have been described with reference to specific embodiments thereof; however, those skilled in the art will recognize that this disclosure is not limited thereto. The various features and aspects of the foregoing disclosure may be used individually or in combination. Furthermore, embodiments may be used in any number of settings and applications other than those described herein without departing from the broader spirit and scope of this specification. Therefore, this specification and the accompanying drawings should be considered illustrative rather than restrictive.< / realm>

Claims

1. A method for providing layer 2 information, comprising: Determine first layer 2 information associated with a first layer 2 virtual switch in the client's layer 2 virtual network, wherein the first layer 2 information includes at least one of a first layer 2 forwarding table of the first layer 2 virtual switch or a first metric about a first frame stream of the first layer 2 virtual switch, wherein: The Layer 2 virtual network is hosted by the physical network and includes a first compute instance, a second compute instance, a first Layer 2 virtual network interface, a second Layer 2 virtual network interface, a first Layer 2 virtual switch, and a second Layer 2 virtual switch. The first computing instance and the second computing instance are hosted by the same host machine or different host machines in the physical network. The Layer 1 2 virtual network interface and the Layer 1 2 virtual switch are hosted by the physical network virtualization device (NVD) and associated with the first compute instance. The Layer 2 virtual network interface and the Layer 2 virtual switch are hosted by the NVD or different NVDs of the physical network and associated with the second computing instance; Determine the Layer 2 information associated with the Layer 2 virtual switch, wherein the Layer 2 information includes at least one of the Layer 2 forwarding table of the Layer 2 virtual switch or a second metric about the second frame stream of the Layer 2 virtual switch; Third Layer 2 information is generated based on the first and second Layer 2 information, wherein the third Layer 2 information includes at least one of the following: a third Layer 2 forwarding table of the Layer 2 virtual network generated based on the first and second Layer 2 forwarding tables; or statistical information about the third frame stream of the Layer 2 virtual network generated based on the first and second metrics; and Send the Layer 3 2 information to the customer's device.

2. The method of claim 1, further comprising: Receive input from the client, which indicates the client configuration for the Layer 2 virtual network; as well as Determine the mapping between the client configuration and the distribution of resources in the Layer 2 virtual network on the physical network, wherein Layer 2 information is further generated based on this mapping.

3. The method of claim 2, wherein the client configuration indicates a first port and a second port of the Layer 2 virtual network, wherein the mapping indicates that the first port corresponds to a first Layer 2 virtual network interface and the second port corresponds to a second Layer 2 virtual network interface, and wherein the method further comprises: Associat the Layer 12 virtual network interface with the Layer 12 virtual switch; Determine the association between the Layer 2 virtual network interface and the Layer 2 virtual switch; as well as This indicates that the information in the third layer 2 is associated with the first and second ports.

4. The method of claim 2, wherein the client configuration indicates the first port, and wherein the method further comprises: Receive customer information queries, which are directed to the first port; Based on the mapping, the first port is determined to correspond to the first layer 2 virtual network interface; Associat the Layer 12 virtual network interface with the Layer 12 virtual switch; as well as The query result is generated based on the first layer 2 information and is associated with the first port, wherein the third layer 2 information includes the query result.

5. The method of claim 2, wherein the client configuration indicates the first port, and the method further comprises: Receive Layer 2 information associated with the Layer 2 Virtual Switch from NVD; Determine the association between the Layer 12 virtual switch and the Layer 12 virtual network interface; Based on the mapping, the first layer 2 virtual network interface is determined to correspond to the first port; and The third layer 2 information includes at least a portion of the first layer 2 information associated with the first port.

6. The method of claim 5, wherein the first Layer 2 information is received from the NVD at least in part based on a client’s Layer 2 information query or a change in the distribution of Layer 2 virtual network resources on the physical network.

7. The method of claim 1, further comprising: Receive a second input from the client, which indicates the first port of the Layer 2 virtual network; Determine the correspondence between the first port and the Layer 1 2 virtual network interface; Associat the Layer 12 virtual network interface with the Layer 12 virtual switch; Based on the Layer 2 forwarding table, the Media Access Control (MAC) address of the Layer 2 virtual network interface is determined to be associated with the first computing instance; and The Layer 3 2 forwarding table includes an indication that the MAC address corresponds to the first port and is associated with the first computing instance.

8. The method of claim 1, further comprising: Receive a second input from the client, which indicates the first port of the Layer 2 virtual network; Receives a Layer 2 forwarding table query from a client, which indicates the first port; Determine the correspondence between the first port and the Layer 1 2 virtual network interface; Associat the Layer 12 virtual network interface with the Layer 12 virtual switch; as well as The query result is generated based on the first layer 2 information and is associated with the first port, wherein the third layer 2 information includes the query result.

9. The method of claim 1, further comprising: Receive a second input from the client, which indicates the first port of the Layer 2 virtual network; Identify the changes to the Layer 1 2 forwarding table, which are associated with the Layer 1 2 virtual network interface; Determine the correspondence between the Layer 1 2 virtual network interface and the first port; The change will be associated with the first port; and The changes associated with the first port are included in the Layer 3 2 forwarding table.

10. The method of claim 9, wherein the change includes at least one of: reassociating a first media access control (MAC) address from a second layer 2 virtual network interface to another layer 2 virtual network interface, or associating a second MAC address with a second layer 2 virtual network interface.

11. The method of claim 1, further comprising: Receive a second input from the client, the second input indicating the type and target of the statistics, wherein the type includes at least one of the following: number of frames, frame size, or amount of bits, and wherein the target includes at least one of the following: port, set of ports, or layer 2 virtual network.

12. The method of claim 1, further comprising: Receive a first metric and metadata associated with the first metric from NVD, the metadata identifying at least one of a Layer 2 virtual network or a Layer 2 virtual network interface, wherein the statistics are further generated at least in part based on the metadata.

13. The method of claim 12, further comprising: Receive a second input from the client, which indicates the first port of the Layer 2 virtual network; Based on metadata, the first metric is associated with the Layer 1 2 virtual network interface; Determine the correspondence between the Layer 1 2 virtual network interface and the first port; and Associate the first metric with the first port.

14. A system for providing Layer 2 information, comprising: One or more processors; as well as One or more computer-readable storage media storing instructions that, when executed by the one or more processors, configure the system to: Determine first layer 2 information associated with a first layer 2 virtual switch in the client's layer 2 virtual network, wherein the first layer 2 information includes at least one of a first layer 2 forwarding table of the first layer 2 virtual switch or a first metric about a first frame stream of the first layer 2 virtual switch, wherein: The Layer 2 virtual network is hosted by the physical network and includes a first compute instance, a second compute instance, a first Layer 2 virtual network interface, a second Layer 2 virtual network interface, a first Layer 2 virtual switch, and a second Layer 2 virtual switch. The first computing instance and the second computing instance are hosted by the same host machine or different host machines in the physical network. The Layer 1 2 virtual network interface and the Layer 1 2 virtual switch are hosted by the physical network virtualization device (NVD) and associated with the first compute instance. The Layer 2 virtual network interface and the Layer 2 virtual switch are hosted by the NVD or different NVDs of the physical network and associated with the second computing instance; Determine the Layer 2 information associated with the Layer 2 virtual switch, wherein the Layer 2 information includes at least one of the Layer 2 forwarding table of the Layer 2 virtual switch or a second metric about the second frame stream of the Layer 2 virtual switch; The third layer 2 information is generated based on the first layer 2 information and the second layer 2 information, wherein the third layer 2 information includes at least one of the following: a third layer 2 forwarding table of the layer 2 virtual network generated based on the first layer 2 forwarding table and the second layer 2 forwarding table, or statistical information about the third frame stream of the layer 2 virtual network generated based on the first metric and the second metric. as well as Send the Layer 3 2 information to the customer's device.

15. The system of claim 14, wherein the one or more computer-readable storage media further stores instructions that, when executed by the one or more processors, configure the system to perform the following operations: Receive client input indicating the client configuration for the Layer 2 virtual network; and Determine the mapping between the client configuration and the distribution of resources in the Layer 2 virtual network on the physical network, wherein Layer 2 information is further generated based on this mapping.

16. The system of claim 15, wherein the client configuration indicates a first port and a second port of the Layer 2 virtual network, wherein the mapping indicates that the first port corresponds to a first Layer 2 virtual network interface and the second port corresponds to a second Layer 2 virtual network interface, and wherein the one or more computer-readable storage media further stores instructions that, when executed by the one or more processors, configure the system to perform the following operations: Associat the Layer 12 virtual network interface with the Layer 12 virtual switch; Associating the Layer 2 2 virtual network interface with the Layer 2 2 virtual switch; and This indicates that the information in the third layer 2 is associated with the first and second ports.

17. One or more non-transitory computer-readable storage media storing instructions, which, when executed on a system, cause the system to perform operations including: Determine first layer 2 information associated with a first layer 2 virtual switch in the client's layer 2 virtual network, wherein the first layer 2 information includes at least one of a first layer 2 forwarding table of the first layer 2 virtual switch or a first metric about a first frame stream of the first layer 2 virtual switch, wherein: The Layer 2 virtual network is hosted by the physical network and includes a first compute instance, a second compute instance, a first Layer 2 virtual network interface, a second Layer 2 virtual network interface, a first Layer 2 virtual switch, and a second Layer 2 virtual switch. The first computing instance and the second computing instance are hosted by the same host machine or different host machines in the physical network. The Layer 1 2 virtual network interface and the Layer 1 2 virtual switch are hosted by the physical network virtualization device (NVD) and associated with the first compute instance. The Layer 2 virtual network interface and the Layer 2 virtual switch are hosted by the NVD or different NVDs of the physical network and associated with the second computing instance; Determine the Layer 2 information associated with the Layer 2 virtual switch, wherein the Layer 2 information includes at least one of the Layer 2 forwarding table of the Layer 2 virtual switch or a second metric about the second frame stream of the Layer 2 virtual switch; The third layer 2 information is generated based on the first layer 2 information and the second layer 2 information, wherein the third layer 2 information includes at least one of the following: a third layer 2 forwarding table of the layer 2 virtual network generated based on the first layer 2 forwarding table and the second layer 2 forwarding table, or statistical information about the third frame stream of the layer 2 virtual network generated based on the first metric and the second metric. as well as Send the Layer 3 2 information to the customer's device.

18. The one or more non-transitory computer-readable storage media of claim 17, wherein the one or more non-transitory computer-readable storage media further stores instructions that, when the system is executed, configure the system to perform the following operations: Receive client input indicating the client configuration for the Layer 2 virtual network; and Determine the mapping between the client configuration and the distribution of resources in the Layer 2 virtual network on the physical network, wherein Layer 2 information is further generated based on this mapping.

19. The one or more non-transitory computer-readable storage media of claim 18, wherein the client configuration indicates a first port and a second port of a Layer 2 virtual network, wherein the mapping indicates that the first port corresponds to a first Layer 2 virtual network interface and the second port corresponds to a second Layer 2 virtual network interface, and wherein the one or more non-transitory computer-readable storage media also stores instructions that, when the system is executed, configure the system as follows: Associat the Layer 12 virtual network interface with the Layer 12 virtual switch; Associating the Layer 2 2 virtual network interface with the Layer 2 2 virtual switch; and This indicates that the information in the third layer 2 is associated with the first and second ports.

Images