System and method for vlan switching and routing services

Abstract

Systems and methods for VLAN switching and routing services (VSRS) are disclosed herein. A method can include generating a table for instances of a VSRS that couples a first virtual layer 2 network (VLAN) with a second network. The table can contain information identifying IP addresses, MAC addresses, and virtual interface identifiers of the instances in the virtual layer 2 network. The method can include receiving a packet from a first instance with the VSRS, the packet designated for delivery to a second instance within the virtual layer 2 network, identifying the second instance within the virtual layer 2 network with the VSRS to deliver the packet based on information received with the packet and information contained within the table, and delivering the packet to the identified second instance.

Description

[0001] Cross-reference to related applications

[0002] This application claims the rights of the following applications:

[0003] (1) U.S. Provisional Application No. 63 / 051,728, filed on July 14, 2020, entitled "VLAN Switching And Routing Service And Layer-2 Networking In A Virtualized Cloud Environment", and

[0004] (2) U.S. Provisional Application No. 63 / 132,377, filed on December 30, 2020, entitled “Layer-2 Networking In A Virtualized Cloud Environment”. The entire contents of the above provisional application are incorporated herein by reference for all purposes.

[0005] This application is also related to U.S. Application No. __________________ entitled "VIRTUAL LAYER-2 NETWORK" filed on July 14, 2021 (Attorney Docket No. 088325-1203134-276500US), and U.S. Application No. __________________ entitled "INTERFACE-BASED ACLS IN A LAYER-2 NETWORK" filed on July 14, 2021 (Attorney Docket No. 088325-1256549-276520US), the entire contents of each of which are incorporated herein by reference for all purposes. Background Technology

[0006] Cloud computing provides on-demand availability of computing resources. It can be based on data centers accessible to users via the internet. Cloud computing can provide Infrastructure as a Service (IaaS). Virtual networks can be created for user use. However, these virtual networks have limitations that restrict their functionality and value. Therefore, further improvements are expected. Summary of the Invention

[0007] One aspect of this disclosure relates to a computer-implemented method. The method includes providing a virtual layer 3 network and a virtual layer 2 network in a virtualized cloud environment, the virtual layer 3 network being hosted by an underlying physical network, and the virtual layer 2 network being hosted by the underlying physical network.

[0008] In some embodiments, a virtual layer 2 network may be a virtual local area network (VLAN). In some embodiments, a VLAN includes multiple endpoints. In some embodiments, multiple endpoints may be multiple computing instances. In some embodiments, a VLAN includes multiple L2 virtual network interface cards (L2 VNICs) and multiple switches.

[0009] In some embodiments, each of the plurality of computing instances is communicatively coupled to a single L2 Virtual Network Interface Card (L2VNIC) and a single switch. In some embodiments, the plurality of switches may together form a distributed switch. In some embodiments, each of the plurality of switches routes outbound traffic according to a mapping table received from the L2 VNIC paired with the switch. In some embodiments, the mapping table is an endpoint identification interface-to-MAC address mapping within a VLAN.

[0010] In some embodiments, the method further includes instantiating a pair comprising a unique L2VNIC and a unique switch on a network virtualization device (NVD). In some embodiments, the method includes receiving packets addressed to one of the plurality of compute instances from another endpoint within a VLAN at the unique L2VNIC of one of the plurality of compute instances, and learning a mapping of the other endpoint using the unique L2VNIC of one of the plurality of compute instances. In some embodiments, the mapping of the other endpoint includes an interface-to-MAC address mapping of the other endpoint.

[0011] In some embodiments, the method includes decapsulating a received packet with a unique L2VNIC of one of a plurality of computing instances and forwarding the decapsulated packet to one of the plurality of computing instances. In some embodiments, the method includes learning an IP address-to-MAC address mapping for another endpoint with one of the plurality of computing instances.

[0012] In some embodiments, the method includes sending an IP packet from a first computing instance in a VLAN, the IP packet including a destination IP address of a second computing instance in the VLAN; receiving the IP packet at a first L2 VNIC associated with the first computing instance; encapsulating the IP packet at the first L2 VNIC; and forwarding the IP packet to the second computing instance via a first switch. In some embodiments, the first switch and the first L2 VNIC together serve as a pair communicatively coupled to the first computing instance. In some embodiments, the method further includes receiving the IP packet at a second VNIC associated with the second computing instance; decapsulating the IP packet at the second VNIC; and forwarding the IP packet from the second VNIC to the second computing instance.

[0013] In some embodiments, the Virtual Layer 2 network includes multiple Virtual Local Area Networks (VLANs). In some embodiments, each of the multiple VLANs includes multiple endpoints. In some embodiments, the multiple VLANs include a first VLAN and a second VLAN. In some embodiments, the first VLAN includes multiple first endpoints, and the second VLAN includes multiple second endpoints. In some embodiments, each of the multiple VLANs has a unique identifier. In some embodiments, one of the multiple first endpoints in the first VLAN communicates with one of the multiple second endpoints in the second VLAN.

[0014] One aspect of this disclosure relates to a system comprising a physical network. The physical network includes at least one host machine and at least one network virtualization device. The physical network can provide a virtual layer 3 network and a virtual layer 2 network in a virtualized cloud environment, the virtual layer 3 network being hosted by the underlying physical network, the virtual layer 2 network being hosted by the underlying physical network.

[0015] One aspect of this disclosure relates to a non-transitory computer-readable storage medium storing a plurality of instructions executable by one or more processors. When executed by one or more processors, the plurality of instructions cause the one or more processors to provide a virtual layer 3 network and a virtual layer 2 network in a virtualized cloud environment, the virtual layer 3 network being hosted by an underlying physical network, the virtual layer 2 network being hosted by the underlying physical network.

[0016] One aspect of this disclosure relates to a method including generating a table for instances of a VLAN Switching and Routing Service (VSRS), which couples a first Virtual Layer 2 network to a second network. In some embodiments, the table contains information identifying IP addresses, MAC addresses, and virtual interface identifiers of instances within the first Virtual Layer 2 network. The method includes receiving packets from a first instance using the VSRS, specifying packets for delivery to a second instance in the first Virtual Layer 2 network; identifying the second instance in the first Virtual Layer 2 network using the VSRS to deliver the packets based on information received with the packets and information contained in the table; and delivering the packets to the identified second instance.

[0017] In some embodiments, the first virtual Layer 2 network includes multiple instances. In some embodiments, the first virtual Layer 2 network includes multiple L2 virtual network interface cards (L2 VNICs) and multiple switches. In some embodiments, each of the multiple instances is communicatively coupled to a single L2 virtual network interface card (L2 VNIC) and a single switch.

[0018] In some embodiments, identifying a second instance for packet delivery within a first virtual layer 2 network using VSRS based on information received with the packet and information contained in a table includes determining with VSRS that the table does not include mapping information for the second instance, suspending packet delivery with VSRS, broadcasting an ARP request containing the IP address of the second instance to an L2 VNIC in the first virtual layer 2 network with VSRS, and receiving an ARP response from the L2 VNIC of the second instance with VSRS.

[0019] In some embodiments, the method further includes updating the table based on received ARP responses. In some embodiments, the first instance is outside a first Virtual Layer 2 network and in a second network. In some embodiments, the second network may be an L3 network. In some embodiments, the second network may be a second Virtual Layer 2 network. In some embodiments, the table is generated based on communications received by VSRS.

[0020] In some embodiments, the method includes instantiating VSRS as a service on multiple hardware nodes. In some embodiments, the method includes distributing tables across hardware nodes. In some embodiments, the tables distributed across hardware nodes can be accessed by another VSRS instantiation. In some embodiments, the first instance is within a first Virtual Layer 2 network.

[0021] In some embodiments, the method includes receiving packets from a third instance within a first Virtual Layer 2 network using VSRS. In some embodiments, the packets are designated for delivery to a fourth instance outside the first Virtual Layer 2 network and for forwarding the packets to the fourth instance. In some embodiments, the method includes receiving packets from a third instance within a first Virtual Layer 2 network using VSRS. In some embodiments, the packets are designated for delivery to a service used by the third instance within the first Virtual Layer 2 network. In some embodiments, the service may be at least one of the following: DHCP; NTP; and DNS.

[0022] In some embodiments, the method includes receiving packets from a third instance within a first virtual layer 2 network using VSRS. In some embodiments, the packets are designated for delivery to a fourth instance in a second virtual layer 2 network. In some embodiments, the method includes distributing a table of VSRS instances with layer 2 and layer 3 network information across a set of service nodes to provide highly reliable and highly scalable instantiation of VSRS. In some embodiments, the method includes receiving packets from a third instance within a first virtual layer 2 network using VSRS and learning a mapping of the third instance using VSRS.

[0023] One aspect of this disclosure relates to a system. The system includes a physical network. The physical network includes at least one processor and a network virtualization device. The at least one processor can instantiate an instance of a VLAN Switching and Routing Service (VSRS), which couples a first Virtual Layer 2 network to a second network, and generate a table for the VSRS instances. In some embodiments, the table contains information identifying an instance's IP address, MAC address, and virtual interface identifier within the first Virtual Layer 2 network. The at least one processor can use the VSRS to receive packets from a first instance designated for delivery to a second instance within the first Virtual Layer 2 network, identify the second instance within the first Virtual Layer 2 network using the VSRS, deliver the packets based on information received with the packets and information contained in the table, and deliver the packets to the identified second instance.

[0024] One aspect of this disclosure relates to a non-transitory computer-readable storage medium storing a plurality of instructions executable by one or more processors. When executed by the one or more processors, the plurality of instructions cause the one or more processors to: instantiate an instance of a VLAN Switching and Routing Service (VSRS) that couples a first Virtual Layer 2 network to a second network, and generate a table for the instance of the VSRS. In some embodiments, the table contains information identifying IP addresses, MAC addresses, and virtual interface identifiers for instances within the first Virtual Layer 2 network. When executed by the one or more processors, the plurality of instructions cause the one or more processors to: receive packets from a first instance using the VSRS that are designated for delivery to a second instance within the first Virtual Layer 2 network; identify the second instance within the first Virtual Layer 2 network using the VSRS to deliver the packets based on information received with the packets and information contained in the table; and deliver the packets to the identified second instance.

[0025] One aspect of this disclosure relates to a method. The method includes: sending packets from a source computing instance in a virtual network to a destination computing instance via a destination L2 virtual network interface card (destination L2 VNIC) in a first virtual layer 2 network; evaluating access control lists (ACLs) for packets having a source virtual network interface card (source VNIC); embedding packet-related ACL information in the packet; forwarding the encapsulated packet to a Virtual Switching and Routing Service (VSRS) that couples the first virtual layer 2 network (VLAN) to a second network; using the VSRS to identify the destination L2 VNIC within the first virtual layer 2 network to deliver the packet based on information received with the packet and mapping information contained in a mapping table; accessing ACL information from the packet using the VSRS; and applying the accessed ACL information to the packet.

[0026] In some embodiments, the packets include IP packets. In some embodiments, the source compute instance resides in a virtual L3 network. In some embodiments, the source compute instance resides in a second virtual Layer 2 network.

[0027] In some embodiments, the method includes encapsulating packets with a source VNIC. In some embodiments, the method includes receiving and decapsulating packets using VSRS. In some embodiments, identifying a destination L2 VNIC within a first Virtual Layer 2 network to deliver packets using VSRS based on information received with the packets and mapping information contained in a mapping table includes determining with VSRS that the mapping table does not include mapping information for the destination compute instance, suspending VSRS forwarding of the packets, broadcasting an ARP request containing the IP address of the destination compute instance to an L2 VNIC in the first Virtual Layer 2 network using VSRS, and receiving an ARP response from the L2 VNIC of the destination compute instance using VSRS. In some embodiments, one of the L2 VNICs is the L2 VNIC of the destination compute instance.

[0028] In some embodiments, the method includes updating a table based on a received ARP response. In some embodiments, identifying a destination L2 VNIC within a first Virtual Layer 2 network for packet delivery using VSRS based on information received with the packet and mapping information contained in the mapping table includes determining that the mapping table includes mapping information for a destination computation instance and identifying the destination L2 VNIC based on the mapping information contained in the mapping table. In some embodiments, embedding packet-related ACL information in the packet includes storing the ACL information as metadata in the packet. In some embodiments, accessing ACL information from the packet using VSRS includes extracting the metadata containing the ACL information in the packet.

[0029] In some embodiments, applying accessed ACL information to a packet includes determining that the ACL information is not related to the destination L2 VNIC. In some embodiments, applying accessed ACL information to a packet also includes forwarding the packet to the destination compute instance via the destination L2 VNIC. In some embodiments, applying accessed ACL information to a packet includes determining, using VSRS, that the ACL information is related to the destination L2 VNIC. In some embodiments, applying accessed ACL information to a packet also includes: determining, using VSRS, that the destination L2 VNIC conforms to the ACL information; and forwarding the packet to the destination compute instance via the destination L2 VNIC using VSRS.

[0030] In some embodiments, applying the accessed ACL information to the packet further includes: using VSRS to determine that the destination L2VNIC does not conform to the ACL information; and VSRS discarding the packet. In some embodiments, applying the accessed ACL information to the packet further includes using VSRS to send a response instructing the source compute instance to discard the packet.

[0031] One aspect of this disclosure relates to a system including a physical network. The physical network includes at least one first processor, a network virtualization device, and at least one second processor. The at least one processor can send packets from a source computing instance in a virtual network instantiated on the physical network to a destination L2 virtual network interface card (destination L2 VNIC) within a first virtual layer 2 network instantiated on the physical network. The network virtualization device can instantiate the source VNIC. The source VNIC can evaluate an access control list (ACL) for the packet, embed packet-related ACL information in the packet, and forward the packet to a Virtual Switching and Routing Service (VSRS) that couples the first virtual layer 2 network (VLAN) to a second network. The at least one second processor can instantiate the VSRS. The VSRS can identify the destination L2 VNIC for packet delivery based on information received with the packet and mapping information contained in a mapping table, access ACL information from the packet, and apply the accessed ACL information to the packet.

[0032] In some embodiments, applying the accessed ACL information to a packet includes determining that the ACL information is associated with the destination L2VNIC, determining that the destination L2VNIC conforms to the ACL information, and forwarding the packet to the destination compute instance via the destination L2VNIC using VSRS.

[0033] One aspect of this disclosure relates to a non-transitory computer-readable storage medium storing a plurality of instructions executable by one or more processors. When executed by one or more processors, the plurality of instructions cause one or more processors to send packets from a source computing instance in a virtual network to a destination computing instance via a destination L2 virtual network interface card (destination L2 VNIC) in a first virtual layer 2 network, evaluate an access control list (ACL) for the packet using the source virtual network interface card (source VNIC), embed packet-related ACL information in the packet, forward the packet to a Virtual Switching and Routing Service (VSRS) that couples the first virtual layer 2 network (VLAN) to a second network, identify the destination L2 VNIC in the first virtual layer 2 network for packet delivery using the VSRS based on information received with the packet and mapping information contained in a mapping table, access ACL information from the packet using the VSRS, and apply the accessed ACL information to the packet. Attached Figure Description

[0034] Figure 1 This is a high-level diagram of a distributed environment, illustrating a virtual or overlay cloud network hosted by a cloud service provider infrastructure according to certain embodiments.

[0035] Figure 2 A simplified architecture diagram of the physical components in the physical network within the CSPI according to certain embodiments is depicted.

[0036] Figure 3 An example arrangement within CSPI according to certain embodiments is shown, in which a host machine is connected to multiple network virtualization devices (NVDs).

[0037] Figure 4 The connectivity between the host machine and the NVD, according to certain embodiments, is described for providing I / O virtualization to support multiple tenancy.

[0038] Figure 5 A simplified block diagram of a physical network provided by CSPI according to certain embodiments is depicted.

[0039] Figure 6 This is a schematic diagram of one embodiment of a computing network.

[0040] Figure 7 This is a logical and hardware diagram of a Virtual Local Area Network (VLAN).

[0041] Figure 8 This is a logical diagram of multiple connected L2 VLANs.

[0042] Figure 9 This is a logical diagram of multiple connected L2 VLANs and subnets.

[0043] Figure 10 This is a schematic diagram of an embodiment of intra-VLAN communication and learning within a VLAN.

[0044] Figure 11 This is a schematic diagram of an embodiment of a VLAN implementation view.

[0045] Figure 12 This is a flowchart illustrating one embodiment of a process used for communication within a VLAN.

[0046] Figure 13 This is a schematic diagram of the process used for communication within a VLAN.

[0047] Figure 14 This is a flowchart illustrating one embodiment of the process for inter-VLAN communication in a virtual L2 network.

[0048] Figure 15This is a schematic diagram of the process used for inter-VLAN communication.

[0049] Figure 16 This is a flowchart illustrating one embodiment of the process used for ingress grouping flow.

[0050] Figure 17 This is a schematic diagram of the process used for ingress communication.

[0051] Figure 18 This is a flowchart illustrating one embodiment of the process for an outgoing packet flow from a VLAN.

[0052] Figure 19 This is a schematic diagram of the process used for outbound grouping flow.

[0053] Figure 20 This is a flowchart illustrating one embodiment of the process for classifying Access Control Lists (ACLs) for delays.

[0054] Figure 21 This is a flowchart illustrating one embodiment of the process used for early classification of ACLs.

[0055] Figure 22 This is a flowchart illustrating one embodiment of a process for next-hop routing based on the sender.

[0056] Figure 23 This is a flowchart illustrating one embodiment of the process for delaying next-hop routing.

[0057] Figure 24 This is a block diagram illustrating a pattern for implementing cloud infrastructure as a service system according to at least one embodiment.

[0058] Figure 25 This is a block diagram illustrating another pattern for implementing cloud infrastructure as a service system according to at least one embodiment.

[0059] Figure 26 This is a block diagram illustrating another pattern for implementing cloud infrastructure as a service system according to at least one embodiment.

[0060] Figure 27 This is a block diagram illustrating another pattern for implementing cloud infrastructure as a service system according to at least one embodiment.

[0061] Figure 28 This is a block diagram illustrating an example computer system according to at least one embodiment. Detailed Implementation

[0062] In the following description, specific details are set forth for purposes of explanation in order to provide a thorough understanding of certain embodiments. However, it will be apparent that various embodiments may be practiced without these specific details. The accompanying drawings and description are not intended to be limiting. The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment or design described herein as "exemplary" is not necessarily to be construed as preferred or superior to other embodiments or designs.

[0063] Example Virtual Networking Architecture

[0064] The term cloud service generally refers to services provided by a cloud service provider (CSP) to users or customers on demand (e.g., via a subscription model) using systems and infrastructure (cloud infrastructure) provided by the CSP. Typically, the servers and systems that make up the CSP's infrastructure are separate from the customer's own on-premises servers and systems. Therefore, customers can utilize cloud services provided by the CSP without having to purchase separate hardware and software resources for the service. Cloud services are designed to provide subscribers with simple, scalable access to applications and computing resources without requiring customers to invest in the infrastructure used to provide the service.

[0065] Several cloud service providers offer various types of cloud services. There are various types or models of cloud services, including Software as a Service (SaaS), Platform as a Service (PaaS), Infrastructure as a Service (IaaS), etc.

[0066] Customers can subscribe to one or more cloud services provided by a CSP. Customers can be any entity, such as individuals, organizations, or businesses. When a customer subscribes to or registers for a service provided by a CSP, a lease or account is created for that customer. The customer can then access one or more cloud resources associated with that account through the subscription.

[0067] As mentioned above, Infrastructure as a Service (IaaS) is a specific type of cloud computing service. In the IaaS model, a CSP provides infrastructure (called Cloud Service Provider Infrastructure or CSPI) that customers can use to build their own customizable networks and deploy customer resources. Therefore, the customer's resources and network are hosted in a distributed environment by the infrastructure provided by the CSP. This differs from traditional computing, where the customer's resources and network are hosted by the infrastructure provided by the customer.

[0068] CSPI can include interconnected high-performance computing resources forming a physical network, including various host machines, memory resources, and network resources. This physical network is also known as the substrate network or underlying network. Resources in the CSPI can be distributed across one or more data centers, which may be geographically dispersed across one or more geographic regions. Virtualization software can be executed by these physical resources to provide a virtualized distributed environment. Virtualization creates overlay networks (also known as software-based networks, software-defined networks, or virtual networks) on the physical network. The CSPI physical network provides the underlying foundation for creating one or more overlay or virtual networks on top of the physical network. The physical network (or substrate network or underlying network) includes physical network devices such as physical switches, routers, computers, and host machines. An overlay network is a logical (or virtual) network that runs on top of the physical substrate network. A given physical network can support one or more overlay networks. Overlay networks typically use encapsulation techniques to distinguish traffic belonging to different overlay networks. Virtual or overlay networks are also known as Virtual Cloud Networks (VCNs). A virtual network is a layer of network abstraction that can run on top of a physical network, implemented using software virtualization technologies (e.g., hypervisors, virtualization functions implemented by network virtualization devices (NVDs) (e.g., smart NICs), top-of-rack (TOR) switches, smart TORs that implement one or more functions performed by NVDs, and other mechanisms). Virtual networks can take many forms, including peer-to-peer networks, IP networks, etc. Virtual networks are typically either Layer 3 IP networks or Layer 2 VLANs. This method of virtual or overlay networking is often referred to as virtual or overlay Layer 3 networking. Examples of protocols developed for virtual networks include IP-in-IP (or Generic Routing Encapsulation (GRE)), Virtual Extensible LAN (VXLAN—IETF RFC 7348), Virtual Private Network (VPN) (e.g., MPLS Layer 3 Virtual Private Network (RFC 4364)), VMware's NSX, GENEVE (Generic Network Virtualization Encapsulation), etc.

[0069] For IaaS, the infrastructure provided by a CSP (Center for Service Providers) can be configured to offer virtualized computing resources over a public network (e.g., the Internet). In the IaaS model, cloud service providers can host infrastructure components (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer), etc.). In some cases, IaaS providers can also provision various services to accompany those infrastructure components (e.g., billing, monitoring, logging, security, load balancing, and clustering, etc.). Therefore, since these services can be policy-driven, IaaS users can implement policies to drive load balancing to maintain application availability and performance. CSPI provides a collection of infrastructure and complementary cloud services that enable customers to build and run a wide range of applications and services in a highly available, hosted, distributed environment. CSPI provides high-performance computing resources and capabilities, as well as storage capacity, in a flexible virtual network securely accessible from various networked locations, such as the customer's on-premises network. When a customer subscribes to or enrolls in an IaaS service provided by a CSP, the lease created for that customer is a secure and isolated partition within the CSP, where the customer can create, organize, and manage their cloud resources.

[0070] Customers can build their own virtual networks using the compute, storage, and networking resources provided by CSPI. One or more customer resources or workloads, such as compute instances, can be deployed on these virtual networks. For example, a customer can use resources provided by CSPI to build one or more customizable and private virtual networks, called Virtual Cloud Networks (VCNs). A customer can deploy one or more customer resources, such as compute instances, on a customer VCN. Compute instances can take the form of virtual machines, bare metal instances, etc. Therefore, CSPI provides a collection of infrastructure and complementary cloud services that enable customers to build and run a wide range of applications and services in a highly available, virtually managed environment. Customers do not manage or control the underlying physical resources provided by CSPI, but they can control the operating system, storage devices, and deployed applications; and may have limited control over selected networking components (e.g., firewalls).

[0071] The CSP can provide a console that enables customers and network administrators to configure, access, and manage resources deployed in the cloud using CSPI resources. In some embodiments, the console provides a web-based user interface that can be used to access and manage the CSPI. In other embodiments, the console is a web-based application provided by the CSP.

[0072] CSPI can support both single-tenant and multi-tenant architectures. In a single-tenant architecture, software (e.g., applications, databases) or hardware components (e.g., host machines or servers) serve a single customer or tenant. In a multi-tenant architecture, the software or hardware components serve multiple customers or tenants. Therefore, in a multi-tenant architecture, CSPI resources are shared among multiple customers or tenants. In a multi-tenant scenario, precautions and safeguards are implemented in CSPI to ensure that each tenant's data is isolated and invisible to other tenants.

[0073] In a physical network, a network endpoint (“endpoint”) is a computing device or system that connects to and communicates with the physical network. Network endpoints in a physical network can connect to a Local Area Network (LAN), a Wide Area Network (WAN), or other types of physical networks. Examples of traditional endpoints in a physical network include modems, hubs, bridges, switches, routers and other network devices, physical computers (or host machines), etc. Each physical device in a physical network has a fixed network address that can be used to communicate with the device. This fixed network address can be a Layer 2 address (e.g., a MAC address), a fixed Layer 3 address (e.g., an IP address), etc. In a virtualized environment or virtual network, endpoints can include various virtual endpoints, such as virtual machines hosted by components of the physical network (e.g., hosted by physical host machines). These endpoints in a virtual network are addressed by overlay addresses, such as overlay Layer 2 addresses (e.g., overlay MAC addresses) and overlay Layer 3 addresses (e.g., overlay IP addresses). Network overlays provide flexibility by allowing network administrators to move around the overlay addresses associated with network endpoints using software management (e.g., via software implementing a control plane for virtual networks). Therefore, unlike physical networks, in virtual networks, overlay addresses (e.g., overlay IP addresses) can be moved from one endpoint to another using network management software. Since virtual networks are built on top of physical networks, communication between components within a virtual network involves both the virtual network and the underlying physical network. To facilitate this communication, CSPI components are configured to learn and store mappings that map overlay addresses in the virtual network to actual physical addresses in the baseboard network, and vice versa. These mappings are then used to facilitate communication. Client traffic is encapsulated to facilitate routing within the virtual network.

[0074] Therefore, physical addresses (e.g., physical IP addresses) are associated with components in a physical network, and overlay addresses (e.g., overlay IP addresses) are associated with entities in a virtual or overlay network. A physical IP address is an IP address associated with a physical device (e.g., a network device) in a substrate or physical network. For example, each NVD has an associated physical IP address. An overlay IP address is an overlay address associated with an entity in an overlay network (such as a compute instance in a customer's Virtual Cloud Network (VCN)). Two different customers or tenants, each with their own private VCN, could potentially use the same overlay IP address in their VCNs without each other's knowledge. Both physical IP addresses and overlay IP addresses are types of real IP addresses. These are separate from virtual IP addresses. A virtual IP address is typically a single IP address that represents or maps to multiple real IP addresses. A virtual IP address provides a one-to-many mapping between virtual IP addresses and multiple real IP addresses. For example, a load balancer can use a VIP to map or represent multiple servers, each with its own real IP address.

[0075] Cloud infrastructure, or CSPI, is physically hosted in one or more data centers in one or more regions of the world. CSPI can include components in a physical or substrate network and virtualized components (e.g., virtual networks, compute instances, virtual machines, etc.) in a virtual network built on top of the physical network components. In some embodiments, CSPI is organized and hosted in domains, regions, and availability domains. A region is typically a localized geographical area containing one or more data centers. Regions are generally independent of each other and can be geographically distant, for example, spanning countries or even continents. For example, one region might be in Australia, another in Japan, another in India, and so on. CSPI resources are partitioned between regions so that each region has its own independent subset of CSPI resources. Each region can provide a set of core infrastructure services and resources, such as compute resources (e.g., bare metal servers, virtual machines, containers, and related infrastructure); storage resources (e.g., block volume storage devices, file storage devices, object storage devices, archive storage devices); network resources (e.g., virtual cloud networks (VCNs), load balancing resources, connectivity to on-premises networks), database resources; edge networking resources (e.g., DNS); and access management and monitoring resources, etc. Each region typically has multiple paths connecting it to other regions within the domain.

[0076] Generally, applications are deployed in the areas where they are used most frequently (i.e., on infrastructure associated with that area) because using nearby resources is faster than using distant ones. Applications may also be deployed in different areas for various reasons, such as redundancy to mitigate the risk of events within a region (such as large weather systems or earthquakes), or to meet different requirements of legal jurisdictions, tax zones, and other business or social standards.

[0077] Data centers within a region can be further organized and subdivided into Availability Domains (ADs). An Availability Domain can correspond to one or more data centers located within a region. A region can consist of one or more Availability Domains. In this distributed environment, CSPI resources are either region-specific, such as Virtual Cloud Networks (VCNs), or Availability Domain-specific, such as compute instances.

[0078] Availability Zones (ADs) within a region are isolated from each other, fault-tolerant, and configured to make simultaneous failures highly unlikely. This is achieved by ensuring that ADs do not share critical infrastructure resources (such as networking, physical cabling, cable paths, cable entry points, etc.), making a failure at one AD within a region unlikely to affect the availability of other ADs in the same region. ADs within the same region can be interconnected via low-latency, high-bandwidth networks, enabling high-availability connectivity to other networks (e.g., the internet, customer on-premises networks, etc.) and allowing for replication systems across multiple ADs to achieve high availability and disaster recovery. Cloud services use multiple ADs to ensure high availability and prevent resource failures. As the infrastructure provided by an IaaS provider grows, more regions and ADs, along with additional capacity, can be added. Traffic between availability domains is typically encrypted.

[0079] In some embodiments, regions are grouped into domains. A domain is a logical collection of regions. Domains are isolated from each other and do not share any data. Regions within the same domain can communicate with each other, but regions in different domains cannot. A customer's lease or account with the CSP resides in a single domain and can be distributed across one or more regions belonging to that domain. Typically, when a customer subscribes to an IaaS service, a lease or account is created for that customer in a region within the domain that the customer designates (called the "primary" region). A customer can extend their lease to one or more other regions within the domain. A customer cannot access regions that are not within the domain where their lease is located.

[0080] IaaS providers can offer multiple domains, each catering to a specific set of customers or users. For example, a business domain can be offered for business customers. As another example, a domain can be offered for customers within a specific country. As yet another example, a government domain can be offered for governments, etc. For instance, a government domain can cater to a specific government and may have a higher level of security than the business domain. For example, Oracle Cloud Infrastructure (OCI) currently offers domains for the business region and two domains for the government cloud region (e.g., FedRAMP licensed and IL5 licensed).

[0081] In some embodiments, an Active Directory (AD) can be subdivided into one or more fault domains. A fault domain is a grouping of infrastructure resources within an AD to provide anti-affinity. Fault domains allow for the distribution of compute instances such that these instances do not reside on the same physical hardware within a single AD. This is known as anti-affinity. A fault domain refers to a collection of hardware components (computers, switches, etc.) that share a single point of failure. Compute pools are logically divided into fault domains. Therefore, a hardware failure or compute hardware maintenance event affecting one fault domain does not affect instances in other fault domains. Depending on the embodiment, the number of fault domains for each AD can vary. For example, in some embodiments, each AD contains three fault domains. Fault domains act as logical data centers within the AD.

[0082] When a customer subscribes to IaaS services, resources from CSPI are provisioned to the customer and associated with the customer's lease. Customers can use these provisioned resources to build private networks and deploy resources on those networks. Customer networks hosted in the cloud by CSPI are called Virtual Cloud Networks (VCNs). Customers can use the CSPI resources allocated to them to set up one or more VCNs. A VCN is a virtual or software-defined private network. Customer resources deployed in a customer's VCN can include compute instances (e.g., virtual machines, bare metal instances) and other resources. These compute instances can represent various customer workloads, such as applications, load balancers, databases, etc. Compute instances deployed on a VCN can communicate with publicly accessible endpoints (“public endpoints”) via public networks (such as the Internet), with other instances in the same VCN or other VCNs (e.g., other VCNs belonging to the customer or VCNs not belonging to the customer), with the customer's on-premises data center or network, and with service endpoints and other types of endpoints.

[0083] CSPs can use CSPIs to provide various services. In some cases, CSPI clients can function as service providers themselves and use CSPI resources to provide services. Service providers can expose service endpoints, which are characterized by identifying information such as IP addresses, DNS names, and ports. Client resources (e.g., compute instances) can access a specific service by accessing the service endpoints exposed by the service for that specific service. These service endpoints are generally publicly accessible to users via public communication networks (such as the Internet) using the public IP addresses associated with the endpoints. Publicly accessible network endpoints are sometimes also called public endpoints.

[0084] In some embodiments, a service provider may expose the service via an endpoint (sometimes referred to as a service endpoint) for the service. Clients of the service can then use this service endpoint to access the service. In some implementations, the service endpoint provided for the service can be accessed by multiple clients intending to consume the service. In other implementations, a dedicated service endpoint can be provided to a client, such that only that client can use that dedicated service endpoint to access the service.

[0085] In some embodiments, when a VCN is created, it is associated with a Private Overlay Classless Inter-Domain Routing (CIDR) address space, which refers to a set of private overlay IP addresses (e.g., 10.0 / 16) assigned to the VCN. A VCN includes associated subnets, routing tables, and gateways. A VCN resides within a single region but can span one or more of the availability domains within that region. A gateway is a virtual interface configured for the VCN and enables communication between the VCN and one or more endpoints outside the VCN. One or more different types of gateways can be configured for a VCN to enable communication to and from different types of endpoints.

[0086] A VCN can be subdivided into one or more subnets, such as one or more subnets. Therefore, a subnet is a configurable unit or subdivision that can be created within a VCN. A VCN can have one or more subnets. Each subnet within a VCN is associated with a contiguous range of overlay IP addresses (e.g., 10.0.0.0 / 24 and 10.0.1.0 / 24), which do not overlap with other subnets within that VCN and represent a subset of the VCN's address space.

[0087] Each compute instance is associated with a Virtual Network Interface Card (VNIC), which enables the compute instance to participate in subnets within a VCN. A VNIC is the logical representation of a physical network interface card (NIC). Generally, a VNIC is the interface between an entity (e.g., a compute instance, a service) and the virtual network. A VNIC exists within a subnet and has one or more associated IP addresses, along with associated security rules or policies. A VNIC is equivalent to a Layer 2 port on a switch. A VNIC is attached to both the compute instance and the subnet within the VCN. The VNIC associated with a compute instance makes the compute instance part of the VCN's subnet and enables the compute instance to communicate (e.g., send and receive packets) with endpoints on the same subnet as the compute instance, with endpoints in different subnets within the VCN, or with endpoints outside the VCN. Therefore, the VNIC associated with a compute instance determines how the compute instance connects to endpoints inside and outside the VCN. When a compute instance is created and added to a subnet within a VCN, a VNIC for the compute instance is created and associated with that compute instance. For a subnet that includes a set of compute instances, the subnet contains a VNIC corresponding to that set of compute instances, and each VNIC is attached to a compute instance within that set of compute instances.

[0088] A private overlay IP address is assigned to each compute instance via the VNIC associated with it. This private overlay network IP address is assigned to the VNIC associated with the compute instance when the compute instance is created and is used to route traffic to and from the compute instance. All VNICs within a given subnet use the same routing table, security list, and DHCP options. As described above, each subnet within a VCN is associated with a contiguous range of overlay IP addresses (e.g., 10.0.0.0 / 24 and 10.0.1.0 / 24) that do not overlap with other subnets within that VCN and represent a subset of the address space within the VCN's address space. For a VNIC on a specific subnet of a VCN, the private overlay IP address assigned to that VNIC is an address from the contiguous range of overlay IP addresses allocated to the subnet.

[0089] In some embodiments, in addition to a private overlay IP address, a compute instance may optionally be assigned additional overlay IP addresses, such as one or more public IP addresses, for example, if in a public subnet. These multiple addresses may be assigned either on the same VNIC or on multiple VNICs associated with the compute instance. However, each instance has a primary VNIC, which is created during instance startup and associated with the overlay private IP address assigned to that instance—this primary VNIC cannot be deleted. Additional VNICs, called secondary VNICs, can be added to existing instances in the same availability domain as the primary VNIC. All VNICs are in the same availability domain as the instance. Secondary VNICs may reside in a subnet within the same VCN as the primary VNIC, or in a different subnet within the same VCN or different VCNs.

[0090] If a compute instance is in a public subnet, it can optionally be assigned a public IP address. When creating a subnet, it can be specified as either a public or private subnet. A private subnet means that resources within the subnet (such as compute instances) and associated VNICs cannot have public overriding IP addresses. A public subnet means that resources within the subnet and associated VNICs can have public IP addresses. Customers can specify that a subnet exists within a single availability domain or across multiple availability domains in regions or domains.

[0091] As described above, a VCN can be subdivided into one or more subnets. In some embodiments, a virtual router (VR) configured for the VCN (referred to as a VCN VR or simply VR) enables communication between the subnets of the VCN. For a subnet within the VCN, the VR represents a logical gateway for that subnet, enabling that subnet (i.e., the compute instances on that subnet) to communicate with endpoints on other subnets within the VCN and other endpoints outside the VCN. The VCN VR is a logical entity configured to route traffic between the VNIC within the VCN and the virtual gateway (“gateway”) associated with the VCN. The following section discusses… Figure 1Further description of the gateway. A VCN VR is a Layer 3 / IP layer concept. In one embodiment, there exists a VCN VR for a VCN, where the VCN VR has a potentially unlimited number of IP-addressable ports, one port for each subnet of the VCN. In this way, the VCN VR has a different IP address for each subnet within the VCN to which the VCN VR is attached. The VR also connects to various gateways configured for the VCN. In some embodiments, a specific overlay IP address within the overlay IP address range for a subnet is reserved for the port of the VCN VR for that subnet. For example, consider a VCN with two subnets, associated with address ranges 10.0 / 16 and 10.1 / 16. For the first subnet in the VCN with an address range of 10.0 / 16, addresses within this range are reserved for the port of the VCN VR for that subnet. In some cases, the first IP address within the range can be reserved for the VCN VR. For example, for a subnet covering the IP address range of 10.0 / 16, the IP address 10.0.0.1 could be reserved for the port of the VCN VR for that subnet. For a second subnet within the same VCN with an address range of 10.1 / 16, the VCN VR can have a port for the second subnet with the IP address 10.1.0.1. The VCN VR has a different IP address for each subnet within the VCN.

[0092] In some other embodiments, each subnet within a VCN may have its own associated VR, which can be addressed by the subnet using a reserved or default IP address associated with the VR. For example, the reserved or default IP address may be the first IP address in a range of IP addresses associated with the subnet. The VNICs within the subnet can use this default or reserved IP address to communicate with the VR associated with the subnet (e.g., send and receive packets). In this embodiment, the VR is the ingress / egress point of the subnet. The VR associated with a subnet within the VCN can communicate with other VRs associated with other subnets within the VCN. The VR can also communicate with the gateway associated with the VCN. The VR functionality of a subnet runs on or is performed by one or more NVDs that perform VNIC functionality for the VNICs within the subnet.

[0093] You can configure routing tables, security rules, and DHCP options for a VCN. A routing table is a virtual routing table used by the VCN and contains rules that route traffic from subnets within the VCN to destinations outside the VCN via gateways or specially configured instances. You can customize the VCN's routing table to control how packets are forwarded / routed to and from the VCN. DHCP options refer to configuration information automatically provided to the instance when it starts up.

[0094] Security rules configured for a VCN represent overlay firewall rules used by the VCN. Security rules can include ingress and egress rules, specifying the types of traffic allowed to enter and exit the VCN instance (e.g., based on protocol and port). Clients can choose whether a given rule is stateful or stateless. For example, a client can allow a set of incoming SSH traffic from anywhere to the instance by setting a stateful ingress rule with source CIDR 0.0.0.0 / 0 and destination TCP port 22. Security rules can be implemented using network security groups or security lists. A network security group consists of a set of security rules that apply only to resources within that group. A security list, on the other hand, includes rules applicable to all resources in any subnet using that security list. A default security list with default security rules can be provided to the VCN. DHCP options configured for the VCN provide configuration information that is automatically provided to instances within the VCN at instance startup.

[0095] In some embodiments, configuration information for a VCN is determined and stored by the VCN control plane. For example, configuration information for a VCN may include information about: the address range associated with the VCN, subnets and associated information within the VCN, one or more VRs associated with the VCN, compute instances and associated VNICs within the VCN, NVDs (e.g., VNICs, VRs, gateways) performing various virtualization network functions associated with the VCN, status information for the VCN, and other VCN-related information. In some embodiments, a VCN distribution service publishes configuration information stored by the VCN control plane or a portion thereof to the NVD. The distributed information can be used to update information stored and used by the NVD (e.g., forwarding tables, routing tables, etc.) to forward packets to or from compute instances within the VCN.

[0096] In some embodiments, the creation of VCNs and subnets is handled by the VCN control plane (CP), and the initiation of compute instances is handled by the compute control plane. The compute control plane is responsible for allocating physical resources to the compute instances and then invoking the VCN control plane to create VNICs and attach them to the compute instances. The VCN CP also maps VCN data to the VCN data plane, which is configured to perform packet forwarding and routing functions. In some embodiments, the VCN CP provides a distribution service responsible for providing updates to the VCN data plane. Examples of VCN control planes are also available in... Figure 24 , Figure 25 , Figure 26 and Figure 27 It is depicted in (see reference numerals 24116, 2516, 2616 and 2716) and described below.

[0097] Customers can create one or more VCNs using resources hosted by CSPI. Compute instances deployed on a customer's VCN can communicate with different endpoints. These endpoints can include endpoints hosted by CSPI and endpoints external to CSPI.

[0098] Figure 1 , Figure 2 , Figure 3 , Figure 4 , Figure 5 , Figure 24 , Figure 25 , Figure 26 and Figure 28 The document describes various architectures for implementing cloud-based services using CSPI, and these are described below. Figure 1 This is a high-level diagram of a distributed environment 100, illustrating an overlay or client VCN hosted by CSPI according to certain embodiments. Figure 1 The distributed environment described includes multiple components in the overlay network. Figure 1 The distributed environment 100 depicted herein is merely an example and is not intended to unduly limit the scope of the claimed embodiments. Many variations, substitutions, and modifications are possible. For example, in some implementations, Figure 1 The distributed environment described in the text can have more than Figure 1 The more or fewer systems or components shown can be combined into two or more systems, or can have different system configurations or arrangements.

[0099] like Figure 1 As illustrated in the example, distributed environment 100 includes a CSPI 101 that provides services and resources that customers can subscribe to and use to build their Virtual Cloud Network (VCN). In some embodiments, CSPI 101 provides IaaS services to subscribing customers. Data centers within CSPI 101 can be organized into one or more regions. Figure 1 The example shown is a region "Region US" 102. The customer has already configured customer VCN 104 for Region 102. The customer can deploy various compute instances on VCN 104, which can include virtual machines or bare metal instances. Examples of instances include applications, databases, load balancers, etc.

[0100] exist Figure 1 In the embodiment depicted, customer VCN 104 includes two subnets, namely "Subnet-1" and "Subnet-2", each with its own CIDR IP address range. Figure 1In this configuration, subnet-1 covers the IP address range of 10.0 / 16, and subnet-2 covers the address range of 10.1 / 16. VCN Virtual Router 105 represents the logical gateway for the VCN, enabling communication between subnets of VCN 104 and with other endpoints outside the VCN. VCN VR 105 is configured to route traffic between the VNICs within VCN 104 and the gateway associated with VCN 104. VCN VR 105 provides a port for each subnet of VCN 104. For example, VR 105 could provide a port with IP address 10.0.0.1 for subnet-1 and a port with IP address 10.1.0.1 for subnet-2.

[0101] Multiple compute instances can be deployed on each subnet, where compute instances can be virtual machine instances and / or bare metal instances. Compute instances within a subnet can be hosted by one or more host machines within CSPI 101. Compute instances participate in the subnet via a VNIC associated with the compute instance. For example, as... Figure 1 As shown, compute instance C1 becomes part of subnet-1 via the VNIC associated with it. Similarly, compute instance C2 becomes part of subnet-1 via the VNIC associated with it. In a similar manner, multiple compute instances (which can be virtual machine instances or bare metal instances) can be part of subnet-1. Each compute instance is assigned a private overlay IP address and a Media Access Control (MAC) address via its associated VNIC. For example, in... Figure 1 In this subnet, compute instance C1 has an overlay IP address of 10.0.0.2 and a MAC address of M1, while compute instance C2 has a private overlay IP address of 10.0.0.3 and a MAC address of M2. Each compute instance in subnet-1 (including compute instances C1 and C2) has a default route to VCN VR 105 using IP address 10.0.0.1, which is the IP address of the port of VCN VR 105 used in subnet-1.

[0102] Multiple compute instances, including virtual machine instances and / or bare metal instances, can be deployed on subnet-2. For example, such as Figure 1 As shown, compute instances D1 and D2 become part of subnet-2 via the VNIC associated with the respective compute instance. Figure 1In the illustrated embodiment, compute instance D1 has an overlay IP address of 10.1.0.2 and a MAC address of MM1, while compute instance D2 has a private overlay IP address of 10.1.0.3 and a MAC address of MM2. Each compute instance in subnet-2 (including compute instances D1 and D2) has a default route to VCN VR 105 using IP address 10.1.0.1, which is the IP address of the port of VCN VR 105 in subnet-2.

[0103] VCN A 104 may also include one or more load balancers. For example, a load balancer can be provided for a subnet, and the load balancer can be configured to load balance traffic across multiple compute instances on the subnet. Load balancers can also be provided to load balance traffic across subnets within a VCN.

[0104] A specific compute instance deployed on VCN 104 can communicate with a variety of different endpoints. These endpoints can include endpoints hosted by CSPI 200 and endpoints outside of CSPI 200. Endpoints hosted by CSPI 101 can include: endpoints located on the same subnet as the specific compute instance (e.g., communication between two compute instances in subnet-1); endpoints located on different subnets but within the same VCN (e.g., communication between a compute instance in subnet-1 and a compute instance in subnet-2); endpoints in different VCNs within the same region (e.g., communication between a compute instance in subnet-1 and an endpoint in a VCN in the same region 106 or 110, or communication between a compute instance in subnet-1 and an endpoint in service point 110 within the same region); or endpoints in VCNs in different regions (e.g., communication between a compute instance in subnet-1 and an endpoint in a VCN in a different region 108). Compute instances in a subnet hosted by CSPI 101 can also communicate with endpoints not hosted by CSPI 101 (i.e., outside of CSPI 101). These external endpoints include endpoints in the customer’s on-premises network 116, endpoints in other remote cloud-hosted networks 118, public endpoints 114 accessible via public networks such as the Internet, and other endpoints.

[0105] VNICs associated with source and destination compute instances facilitate communication between compute instances on the same subnet. For example, compute instance C1 in subnet-1 might want to send a packet to compute instance C2 in subnet-1. For a packet originating from the source compute instance and destined for another compute instance in the same subnet, the packet is first processed by the VNIC associated with the source compute instance. The processing performed by the VNIC associated with the source compute instance may include determining the packet's destination information from the packet header, identifying any policies (e.g., security lists) configured for the VNIC associated with the source compute instance, determining the next hop for the packet, performing any packet encapsulation / decapsulation functions as needed, and then forwarding / routing the packet to the next hop to facilitate communication to its intended destination. When the destination compute instance is in the same subnet as the source compute instance, the VNIC associated with the source compute instance is configured to identify the VNIC associated with the destination compute instance and forward the packet to that VNIC for processing. The VNIC associated with the destination compute instance then performs the processing and forwards the packet to the destination compute instance.

[0106] For packets to be transferred from compute instances in a subnet to endpoints in different subnets within the same VCN, communication is facilitated by the VNIC associated with the source and destination compute instances, as well as the VCN VR. For example, if Figure 1 If compute instance C1 in subnet-1 wants to send a packet to compute instance D1 in subnet-2, the packet is first processed by the VNIC associated with compute instance C1. The VNIC associated with compute instance C1 is configured to route the packet to VCN VR 105 using the default route or port 10.0.0.1 of the VCN VR. VCN VR 105 is configured to route the packet to subnet-2 using port 10.1.0.1. Then, the VNIC associated with D1 receives and processes the packet and forwards it to compute instance D1.

[0107] For packets to be transferred from a compute instance in VCN 104 to an endpoint outside VCN 104, communication is facilitated by the VNIC associated with the source compute instance, VCN VR 105, and the gateway associated with VCN 104. One or more types of gateways can be associated with VCN 104. A gateway is an interface between a VCN and another endpoint located outside the VCN. A gateway is a Layer 3 / IP concept and enables a VCN to communicate with endpoints outside the VCN. Therefore, a gateway facilitates traffic flow between a VCN and other VCNs or networks. Various different types of gateways can be configured for a VCN to facilitate different types of communication with different types of endpoints. Depending on the gateway, communication can occur over a public network (e.g., the Internet) or over a private network. Various communication protocols can be used for these communications.

[0108] For example, compute instance C1 might want to communicate with an endpoint outside of VCN 104. The packet can first be processed by the VNIC associated with the source compute instance C1. The VNIC processing determines that the packet's destination is outside C1's subnet-1. The VNIC associated with C1 can then forward the packet to VCN VR 105 for VCN 104. VCN VR 105 then processes the packet and, as part of the processing, determines a specific gateway associated with VCN 104 as the next hop for the packet based on its destination. VCN VR 105 can then forward the packet to the specific identified gateway. For example, if the destination is an endpoint within a customer's on-premises network, the packet can be forwarded by VCN VR 105 to the Dynamic Routing Gateway (DRG) gateway 122 configured for VCN 104. The packet can then be forwarded from the gateway to the next hop to facilitate delivery to its final intended destination.

[0109] Various types of gateways can be configured for a VCN. Examples of gateways that can be configured for a VCN are available in [link to example]. Figure 1 It is depicted in [the text] and described below. Examples of gateways associated with VCNs are also [described in the text]. Figure 24 , Figure 25 , Figure 26 and Figure 27 The gateways are depicted (e.g., referenced by reference numerals 2434, 2436, 2438, 2534, 2536, 2538, 2634, 2636, 2638, 2734, 2736, and 2738) and are described below. Figure 1As depicted in the embodiments, a Dynamic Routing Gateway (DRG) 122 can be added to or associated with a customer VCN 104 and provides a path for private network traffic communication between the customer VCN 104 and another endpoint, which can be the customer's on-premises network 116, a VCN 108 in a different region of CSPI 101, or another remote cloud network 118 not hosted by CSPI 101. The customer's on-premises network 116 can be a customer network or customer data center built using the customer's resources. Access to the customer's on-premises network 116 is generally very restricted. For customers who have both a customer's on-premises network 116 and one or more VCNs 104 deployed or hosted in the cloud by CSPI 101, the customer may want their on-premises network 116 and their cloud-based VCN 104 to be able to communicate with each other. This allows customers to build extended hybrid environments that include the customer's VCN 104 hosted by CSPI 101 and their on-premises network 116. The DRG 122 enables this communication. To enable this type of communication, a communication channel 124 is set up, with one endpoint located in the customer's on-premises network 116 and the other endpoint located in CSPI 101 and connected to the customer's VCN 104. Communication channel 124 can be over a public communication network (such as the Internet) or a private communication network. Various communication protocols can be used, such as IPsec VPN technology over a public communication network (such as the Internet), Oracle's FastConnect technology using a private network instead of a public network, etc. The device or equipment forming one endpoint of communication channel 124 in the customer's on-premises network 116 is referred to as a customer field equipment (CPE), such as... Figure 1 The CPE126 is depicted in the diagram. On the CSPI 101 side, the endpoint can be a host machine executing DRG 122.

[0110] In some embodiments, a remote peering connection (RPC) can be added to the DRG, which allows a customer to peer one VCN to another VCN in a different region. Using such an RPC, customer VCN 104 can use DRG 122 to connect to VCN 108 in another region. DRG 122 can also be used to communicate with other remote cloud networks 118 (such as Microsoft Azure cloud, Amazon AWS cloud, etc.) not hosted by CSPI 101.

[0111] like Figure 1As shown, an Internet Gateway (IGW) 120 can be configured for customer VCN 104, enabling compute instances on VCN 104 to communicate with a public endpoint 114 accessible via a public network, such as the Internet. IGW 120 is a gateway connecting the VCN to a public network such as the Internet. IGW 120 enables public subnets within the VCN (such as VCN 104), where resources within the public subnet have publicly overriding IP addresses, to directly access a public endpoint 112 on the public network 114 (such as the Internet). Using IGW 120, connections can be initiated from subnets within VCN 104 or from the Internet.

[0112] A Network Address Translation (NAT) gateway 128 can be configured for a customer's VCN 104, enabling cloud resources within the customer's VCN that do not have dedicated public overlay IP addresses to access the Internet, and doing so to direct incoming Internet connections (e.g., L4-L7 connections) without exposing those resources. This allows private subnets within the VCN (such as private subnet-1 in VCN 104) to privately access public endpoints on the Internet. In a NAT gateway, connections can only originate from private subnets to the public Internet, and not from the Internet to private subnets.

[0113] In some embodiments, a Service Gateway (SGW) 126 can be configured for customer VCN 104 to provide a path for private network traffic between VCN 104 and service endpoints supported in service network 110. In some embodiments, service network 110 may be provided by a CSP and may offer a variety of services. An example of such a service network is Oracle's service network, which provides a variety of services available to customers. For example, compute instances (e.g., database systems) in a private subnet of customer VCN 104 can back up data to service endpoints (e.g., object storage devices) without requiring a public IP address or access to the Internet. In some embodiments, a VCN may have only one SGW, and connections can only originate from subnets within the VCN, not from service network 110. If a VCN is peered to another, resources in the other VCN typically cannot access the SGW. Resources in the on-premises network of a VCN connected using FastConnect or VPNConnect can also use the service gateway configured for that VCN.

[0114] In some implementations, the SGW 126 uses the concept of a Service Classless Inter-Domain Routing (CIDR) label, which is a string representing all regional public IP address ranges for the service or group of services of interest. Customers use the Service CIDR label when configuring the SGW and associated routing rules to control traffic to the service. If the public IP address of the service changes in the future, customers can optionally use it when configuring security rules without having to adjust them.

[0115] Local peering gateway (LPG) 132 is a gateway that can be added to customer VCN 104 and enable VCN 104 to peer with another VCN in the same region. Peering means that VCNs communicate using private IP addresses, and traffic does not need to traverse public networks (such as the Internet) or be routed through the customer's on-premises network 116. In a preferred embodiment, a VCN has a separate LPG for each peer it establishes. Local peering or VCN peering is a common practice for establishing network connectivity between different applications or infrastructure management functions.

[0116] Service providers (such as those providing services in service network 110) can offer access to services using different access models. Under the public access model, a service can be exposed as a public endpoint that is publicly accessible by compute instances within a customer's VCN via a public network (such as the Internet), and / or can be privately accessed via SGW 126. Under a specific private access model, a service can be accessed as a private IP endpoint within a private subnet of the customer's VCN. This is called Private Endpoint (PE) access and enables service providers to expose their services as instances within the customer's private network. A private endpoint resource represents a service within the customer's VCN. Each PE is represented as a VNIC (called a PE-VNIC, with one or more private IPs) within the customer's VCN in a subnet selected by the customer. Thus, the PE provides a way to present services within a private customer VCN subnet using a VNIC. Since the endpoint is exposed as a VNIC, all characteristics associated with the VNIC (such as routing rules, security lists, etc.) are now available for the PE VNIC.

[0117] Service providers can register their services to enable access via PE. Providers can associate policies with services, which restricts the visibility of the service to customer leases. Providers can register multiple services under a single Virtual IP address (VIP), especially for multi-tenant services. Multiple such private endpoints (across multiple VCNs) can represent the same service.

[0118] Compute instances in a private subnet can then access the service using the private IP address or service DNS name of the PE VNIC. Compute instances in a customer VCN can access the service by sending traffic to the private IP address of the PE in the customer VCN. A Private Access Gateway (PAGW) 130 is a gateway resource that can be attached to a service provider VCN (e.g., a VCN in service network 110), which acts as the ingress / egress point for all traffic from / to the private endpoints of the customer subnet. PAGW 130 enables providers to scale the number of PE connections without utilizing their internal IP address resources. A provider only needs to configure one PAGW for any number of services registered in a single VCN. A provider can represent a service as a private endpoint in multiple VCNs of one or more customers. From the customer's perspective, the PE VNIC is not an instance attached to the customer, but rather appears to be attached to the service that the customer wants to interact with. Traffic to the private endpoint is routed to the service via PAGW 130. These are referred to as customer-to-service private connections (C2S connections).

[0119] The PE concept can also be used to extend private access to services to the customer's on-premises network and data center by allowing traffic to flow through FastConnect / IPsec links and private endpoints in the customer's VCN. Private access to services can also be extended to the customer's peering VCN by allowing traffic to flow between the LPG 132 and the PE in the customer's VCN.

[0120] Customers can control routing within a VCN at the subnet level, allowing them to specify which subnets within a customer's VCN (such as VCN104) use each gateway. The VCN's routing table is used to determine whether traffic is allowed to leave the VCN via a specific gateway. For example, in a given instance, a routing table for a public subnet within customer VCN 104 might send non-local traffic via IGW 120. A routing table for a private subnet within the same customer VCN 104 might send traffic destined for a CSP service via SGW 126. All remaining traffic can be sent via NAT gateway 128. The routing table only controls traffic flowing out of the VCN.

[0121] Security lists associated with a VCN are used to control traffic entering the VCN via a gateway through inbound connections. All resources within a subnet use the same routing tables and security lists. Security lists can be used to control specific types of traffic allowed to enter or leave instances within a subnet of the VCN. Security list rules can include inbound and outbound rules. For example, inbound rules can specify allowed source address ranges, while outbound rules can specify allowed destination address ranges. Security rules can specify specific protocols (e.g., TCP, ICMP), specific ports (e.g., port 22 for SSH, port 3389 for Windows RDP), etc. In some implementations, the instance's operating system can enforce its own firewall rules that conform to the security list rules. Rules can be stateful (e.g., tracking connections and automatically allowing responses without explicit security list rules for response traffic) or stateless.

[0122] Access from a customer's VCN (i.e., through resources or compute instances deployed on VCN 104) can be categorized as public access, private access, or dedicated access. Public access refers to an access model that uses a public IP address or NAT to access a public endpoint. Private access enables customer workloads with private IP addresses within VCN 104 (e.g., resources in a private subnet) to access services without traversing a public network such as the Internet. In some embodiments, CSPI 101 enables customer VCN workloads with private IP addresses to access services (public service endpoints) using a service gateway. Thus, the service gateway provides a private access model by establishing a virtual link between the customer's VCN and the public endpoint of the service residing outside the customer's private network.

[0123] Furthermore, CSPI can provide dedicated public access using technologies such as FastConnect public peering, where on-premises instances can access one or more services within a customer's VCN using FastConnect connections without traversing public networks such as the internet. CSPI can also provide dedicated private access using FastConnect private peering, where on-premises instances with private IP addresses can access a customer's VCN workloads using FastConnect connections. FastConnect is a network connectivity alternative to using the public internet to connect a customer's on-premises network to CSPI and its services. Compared to internet-based connections, FastConnect offers a simple, flexible, and cost-effective way to create dedicated and private connections with higher bandwidth options and a more reliable and consistent network experience.

[0124] Figure 1The accompanying description above describes the various virtualized components in the example virtual network. As mentioned above, the virtual network is built on the underlying physical or substrate network. Figure 2 A simplified architecture diagram of the physical components within the physical network of the CSPI 200, which provides the underlying layer for virtual networks according to certain embodiments, is depicted. As shown, the CSPI 200 provides a distributed environment including components and resources (e.g., compute, memory, and network resources) provided by a cloud service provider (CSP). These components and resources are used to provide cloud services (e.g., IaaS services) to subscribed customers (i.e., customers who have subscribed to one or more services provided by the CSP). Based on the services subscribed to by the customer, a subset of the resources of the CSPI 200 (e.g., compute, memory, and network resources) is provisioned to the customer. The customer can then use the physical compute, memory, and networking resources provided by the CSPI 200 to build their own cloud-based (i.e., CSPI-hosted) customizable and private virtual networks. As indicated above, these customer networks are referred to as Virtual Cloud Networks (VCNs). Customers can deploy one or more customer resources, such as compute instances, on these customer VCNs. Compute instances can take the form of virtual machines, bare metal instances, etc. The CSPI 200 provides a collection of infrastructure and complementary cloud services, enabling customers to build and run a wide range of applications and services in a highly available managed environment.

[0125] exist Figure 2 In the example embodiment depicted, the physical components of CSPI 200 include one or more physical host machines or physical servers (e.g., 202, 206, 208), network virtualization devices (NVDs) (e.g., 210, 212), top-of-rack (TOR) switches (e.g., 214, 216), and a physical network (e.g., 218), as well as switches within physical network 218. The physical host machines or servers can host and execute various compute instances participating in one or more subnets of the VCN. Compute instances can include virtual machine instances and bare metal instances. For example, Figure 1 The various computational examples described in the text can be derived from... Figure 2 The physical host machine described in the diagram is used for hosting virtual machine compute instances in a VCN. Virtual machine compute instances in a VCN can be executed by one host machine or multiple different host machines. A physical host machine can also host virtual host machines, container-based hosts, or functions, etc. Figure 1 The VNIC and VCN VR described in the text can be generated by Figure 2 The NVD execution described in the text. Figure 1 The gateway described in the text can be... Figure 2 The host machine and / or NVD execution described in the document.

[0126] A host machine or server can run a hypervisor (also known as a virtual machine monitor or VMM) that creates and enables virtualized environments on the host machine. Virtualization or virtualized environments facilitate cloud-based computing. One or more compute instances can be created, executed, and managed on the host machine by a hypervisor on that host machine. The hypervisor on the host machine enables the host machine's physical computing resources (e.g., compute, memory, and network resources) to be shared among various compute instances executed by the host machine.

[0127] For example, such as Figure 2 As depicted, host machines 202 and 208 execute hypervisors 260 and 266, respectively. These hypervisors can be implemented using software, firmware, or hardware, or a combination thereof. Typically, a hypervisor is a process or software layer residing above the host machine's operating system (OS), which in turn executes on the host machine's hardware processor. Hypervisors provide a virtualized environment by enabling the host machine's physical computing resources (e.g., processing resources such as processors / cores, memory resources, network resources) to be shared among various virtual machine computing instances executed by the host machine. For example, in... Figure 2 In this configuration, the hypervisor 260 can reside on top of the operating system of the host machine 202 and enable the computing resources (e.g., processing, memory, and network resources) of the host machine 202 to be shared among computing instances (e.g., virtual machines) executed by the host machine 202. A virtual machine can have its own operating system (called a guest operating system), which may be the same as or different from the host machine's operating system. The operating system of a virtual machine executed by the host machine can be the same as or different from the operating system of another virtual machine executed by the same host machine. Therefore, the hypervisor enables multiple operating systems to be executed simultaneously, while sharing the same computing resources of the host machine. Figure 2 The host machines described in the text may have the same or different types of management programs.

[0128] A compute instance can be a virtual machine instance or a bare metal instance. Figure 2 In the diagram, compute instance 268 on host machine 202 and compute instance 274 on host machine 208 are examples of virtual machine instances. Host machine 206 is an example of a bare metal instance provided to a customer.

[0129] In some cases, an entire host machine can be provisioned to a single customer, and one or more compute instances (or virtual machines or bare metal instances) hosted by that host machine all belong to the same customer. In other cases, the host machine can be shared among multiple customers (i.e., multiple tenants). In this multi-tenancy scenario, the host machine can host virtual machine compute instances belonging to different customers. These compute instances can be members of different VCNs for different customers. In some embodiments, bare metal compute instances are hosted by bare metal servers without a hypervisor. When provisioning bare metal compute instances, a single customer or tenant maintains control over the physical CPU, memory, and network interfaces of the host machine hosting the bare metal instance, and the host machine is not shared with other customers or tenants.

[0130] As previously mentioned, each compute instance as part of a VCN is associated with a VNIC that enables that compute instance to become a member of a subnet of the VCN. The VNIC associated with a compute instance facilitates communication to and from the compute instance. The VNIC is associated with the compute instance when the compute instance is created. In some embodiments, for compute instances executed by a host machine, the VNIC associated with that compute instance is executed by an NVD connected to the host machine. For example, in Figure 2 In this example, host machine 202 executes a virtual machine compute instance 268 associated with VNIC 276, and VNIC 276 is executed by NVD 210 connected to host machine 202. As another example, a bare metal instance 272 hosted by host machine 206 is associated with VNIC 280 executed by NVD 212 connected to host machine 206. As yet another example, VNIC 284 is associated with compute instance 274 executed by host machine 208, and VNIC 284 is executed by NVD 212 connected to host machine 208.

[0131] For compute instances hosted by a host machine, NVDs connected to that host machine also execute VCN VR corresponding to the VCN where the compute instance is a member. For example, in Figure 2 In the embodiment depicted, NVD 210 executes VCN VR 277 corresponding to a VCN that is a member of compute instance 268. NVD 212 may also execute one or more VCN VR 283 corresponding to a VCN that corresponds to a compute instance hosted by host machines 206 and 208.

[0132] The host machine may include one or more network interface cards (NICs) that enable the host machine to connect to other devices. The NIC on the host machine can provide one or more ports (or interfaces) that allow the host machine to communicate with another device. For example, the host machine can use one or more ports (or interfaces) provided on the host machine and the NVD to connect to the NVD. The host machine can also connect to other devices (such as another host machine).

[0133] For example, in Figure 2 In this configuration, host machine 202 is connected to NVD 210 via link 220, which extends between port 234 provided by NIC 232 of host machine 202 and port 236 of NVD 210. Host machine 206 is connected to NVD 212 via link 224, which extends between port 246 provided by NIC 244 of host machine 206 and port 248 of NVD 212. Host machine 208 is connected to NVD 212 via link 226, which extends between port 252 provided by NIC 250 of host machine 208 and port 254 of NVD 212.

[0134] The NVD is then connected to top-of-rack (TOR) switches via communication links, which are connected to physical network 218 (also known as a switch architecture). In some embodiments, the links between the host machine and the NVD, and between the NVD and the TOR switches, are Ethernet links. For example, in Figure 2 In this configuration, NVDs 210 and 212 are connected to TOR switches 214 and 216 via links 228 and 230, respectively. In some embodiments, links 220, 224, 226, 228, and 230 are Ethernet links. The collection of host machines and NVDs connected to the TOR is sometimes referred to as a rack.

[0135] Physical network 218 provides a communication architecture that enables TOR switches to communicate with each other. Physical network 218 can be a multi-layer network. In some implementations, physical network 218 is a multi-layer Clos network of switches, where TOR switches 214 and 216 represent leaf-level nodes of the multi-layer and multi-node physical switching network 218. Different Clos network configurations are possible, including but not limited to Layer 2 networks, Layer 3 networks, Layer 4 networks, Layer 5 networks, and general "n"-layer networks. Examples of Clos networks are provided in... Figure 5 It is depicted in the middle and described below.

[0136] Various connection configurations can exist between the host machine and the NVD, such as one-to-one, many-to-one, and one-to-many configurations. In a one-to-one configuration, each host machine connects to its own individual NVD. For example, in... Figure 2 In this configuration, host machine 202 connects to NVD 210 via its NIC 232. In a many-to-one configuration, multiple host machines connect to a single NVD. For example, in... Figure 2 In this configuration, host machines 206 and 208 are connected to the same NVD 212 via NICs 244 and 250, respectively.

[0137] In a one-to-many configuration, a host machine connects to multiple NVDs. Figure 3 An example within the CSPI 300 is shown, where a host machine is connected to multiple NVDs. (Example follows) Figure 3 As shown, host machine 302 includes a network interface card (NIC) 304, which includes multiple ports 306 and 308. Host machine 300 is connected to a first NVD 310 via port 306 and link 320, and to a second NVD 312 via port 308 and link 322. Ports 306 and 308 may be Ethernet ports, and links 320 and 322 between host machine 302 and NVDs 310 and 312 may be Ethernet links. NVD 310 is further connected to a first TOR switch 314, and NVD 312 is connected to a second TOR switch 316. Links between NVDs 310 and 312 and TOR switches 314 and 316 may be Ethernet links. TOR switches 314 and 316 represent Layer 0 switching devices in a multi-layer physical network 318.

[0138] Figure 3 The arrangement depicted provides two separate physical network paths from physical switch network 318 to host machine 302: the first path passes through TOR switch 314 to NVD 310 and then to host machine 302, and the second path passes through TOR switch 316 to NVD 312 and then to host machine 302. These separate paths provide enhanced availability (referred to as high availability) for host machine 302. If one of the paths (e.g., a link in one of the paths breaks) or a device (e.g., a particular NVD is not running) experiences a problem, the other path can be used for communication with host machine 302.

[0139] exist Figure 3 In the configuration depicted, the host machine connects to two different NVDs using two different ports provided by the host machine's NIC. In other embodiments, the host machine may include multiple NICs that enable the host machine to connect to multiple NVDs.

[0140] Go back to reference Figure 2An NVD is a physical device or component that performs one or more network and / or storage virtualization functions. An NVD can be any device with one or more processing units (e.g., CPU, Network Processing Unit (NPU), FPGA, packet processing pipeline, etc.), memory (including cache), and ports. Various virtualization functions can be executed by software / firmware performed by one or more processing units of the NVD.

[0141] NVDs can be implemented in various different forms. For example, in some embodiments, an NVD is implemented as an interface card called a smartNIC or a smart NIC with an onboard embedded processor. A smartNIC is a device that is independent of the NIC on the host machine. Figure 2 In this context, NVD 210 and 212 can be implemented as smartNICs connected to host machine 202 and host machines 206 and 208, respectively.

[0142] However, smartNIC is just one example of an NVD implementation. Various other implementations are possible. For example, in some other implementations, the NVD, or one or more functions performed by the NVD, may be integrated into or performed by one or more host machines, one or more TOR switches, and other components of the CSPI 200. For instance, the NVD may be implemented within a host machine, where the functions performed by the NVD are performed by the host machine. As another example, the NVD may be part of a TOR switch, or the TOR switch may be configured to perform functions performed by the NVD, enabling the TOR switch to perform various complex packet transformations for public clouds. A TOR performing the functions of the NVD is sometimes referred to as a smart TOR. In other implementations that serve virtual machine (VM) instances rather than bare metal (BM) instances to customers, the functions performed by the NVD may be implemented within the hypervisor of the host machine. In some other implementations, some of the functions of the NVD may be offloaded to a centralized service running on a set of host machines.

[0143] In some embodiments, such as when implemented as Figure 2 As shown in the smartNIC diagram, the NVD can include multiple physical ports that enable it to connect to one or more host machines and one or more TOR switches. Ports on the NVD can be categorized as host-facing ports (also known as "south ports") or network-facing or TOR-facing ports (also known as "north ports"). The host-facing ports of the NVD are those used to connect the NVD to the host machine. Figure 2 Examples of host-facing ports include port 236 on the NVD 210 and ports 248 and 254 on the NVD 212. Network-facing ports on the NVD are used to connect the NVD to a TOR switch. Figure 2 Examples of network-facing ports include port 256 on the NVD 210 and port 258 on the NVD 212. Figure 2 As shown, NVD 210 is connected to TOR switch 214 via link 228, which extends from port 256 of NVD 210 to TOR switch 214. Similarly, NVD 212 is connected to TOR switch 216 via link 230, which extends from port 258 of NVD 212 to TOR switch 216.

[0144] The NVD receives packets and frames from the host machine (e.g., packets and frames generated by compute instances hosted by the host machine) via its host-facing port, and after performing the necessary packet processing, can forward the packets and frames to the TOR switch via its network-facing port. The NVD can also receive packets and frames from the TOR switch via its network-facing port, and after performing the necessary packet processing, can forward the packets and frames to the host machine via its host-facing port.

[0145] In some embodiments, there can be multiple ports and associated links between the NVD and TOR switches. These ports and links can be aggregated to form a link aggregation group (called a LAG) of multiple ports or links. Link aggregation allows multiple physical links between two endpoints (e.g., between the NVD and TOR switches) to be treated as a single logical link. All physical links in a given LAG can operate at the same speed in full-duplex mode. LAGs help increase the bandwidth and reliability of the connection between two endpoints. If one of the physical links in the LAG fails, traffic is dynamically and transparently reassigned to one of the other physical links in the LAG. Aggregated physical links deliver higher bandwidth than each individual link. Multiple ports associated with an LAG are treated as a single logical port. Traffic can be load-balanced across the multiple physical links of the LAG. One or more LAGs can be configured between two endpoints. These endpoints can be located between the NVD and TOR switches, between a host machine and the NVD, etc.

[0146] The NVD implements or performs network virtualization functions. These functions are performed by software / firmware executed by the NVD. Examples of network virtualization functions include, but are not limited to: packet encapsulation and decapsulation functions; functions for creating VCN networks; functions for implementing network policies, such as VCN security list (firewall) functionality; functions for facilitating packet routing and forwarding to and from compute instances within the VCN; and so on. In some embodiments, upon receiving a packet, the NVD is configured to perform a packet processing pipeline to process the packet and determine how to forward or route it. As part of this packet processing pipeline, the NVD may perform one or more virtual functions associated with the overlay network, such as performing VNICs associated with compute instances within the VCN, performing virtual routers (VRs) associated with the VCN, packet encapsulation and decapsulation to facilitate forwarding or routing within the virtual network, execution of certain gateways (e.g., local peer gateways), implementation of security lists, network security groups, Network Address Translation (NAT) functionality (e.g., host-by-host translation of public IPs to private IPs), throttling functions, and other functions.

[0147] In some embodiments, the packet processing data path in the NVD may include multiple packet pipelines, each consisting of a series of packet transformation stages. In some implementations, upon receiving a packet, the packet is parsed and classified into a single pipeline. The packet is then processed linearly, stage by stage, until the packet is either dropped or sent out through the NVD's interface. These stages provide basic functional packet processing building blocks (e.g., header verification, throttling, insertion of new Layer 2 headers, L4 firewall enforcement, VCN encapsulation / decapsulation, etc.) so that new pipelines can be built by combining existing stages, and new functionality can be added by creating new stages and inserting them into existing pipelines.

[0148] NVD can perform control plane and data plane functions corresponding to those of VCN's control plane and data plane. An example of the VCN control plane is also available in... Figure 24 , Figure 25 , Figure 26 and Figure 27 The VCN data plane is depicted (see reference numerals 2416, 2516, 2616, and 2716) and described below. An example of the VCN data plane is... Figure 24 , Figure 25 , Figure 26 and Figure 27The following describes (see reference numerals 2418, 2518, 2618, and 2718) and is illustrated in the text. Control plane functions include those for configuring how control data is forwarded on the network (e.g., setting routes and routing tables, configuring VNICs, etc.). In some embodiments, a VCN control plane is provided that centrally computes all overlay mappings to the baseboard and publishes them to the NVD and virtual network edge devices (such as various gateways, such as DRGs, SGWs, IGWs, etc.). Firewall rules can also be published using the same mechanism. In some embodiments, the NVD only receives mappings associated with that NVD. Data plane functions include those for actually routing / forwarding packets based on the configuration set using the control plane. The VCN data plane is implemented by encapsulating client network packets before they traverse the baseboard network. Encapsulation / decapsulation functionality is implemented on the NVD. In some embodiments, the NVD is configured to intercept all network packets entering and leaving the host machine and perform network virtualization functions.

[0149] As indicated above, NVD performs various virtualization functions, including VNIC and VCN VR. NVD can execute VNICs associated with compute instances hosted on one or more host machines connected to the VNIC. For example, as... Figure 2 As depicted, NVD 210 performs the functionality of VNIC 276 associated with compute instance 268 hosted by host machine 202 connected to NVD 210. As another example, NVD 212 performs VNIC 280 associated with bare-metal compute instance 272 hosted by host machine 206, and VNIC 284 associated with compute instance 274 hosted by host machine 208. Host machines can host compute instances belonging to different VCNs (belonging to different customers), and NVDs connected to host machines can perform VNICs corresponding to compute instances (i.e., perform VNIC-related functionality).

[0150] NVD also executes a VCN virtual router corresponding to the VCN of the compute instance. For example, in Figure 2 In the embodiments depicted, NVD 210 executes VCN VR 277 corresponding to the VCN to which compute instance 268 belongs. NVD 212 executes one or more VCN VR 283 corresponding to one or more VCNs to which compute instances hosted by host machines 206 and 208 belong. In some embodiments, the VCN VR corresponding to a VCN is executed by all NVDs connected to host machines hosting at least one compute instance belonging to that VCN. If a host machine hosts compute instances belonging to different VCNs, then NVDs connected to that host machine can execute VCN VR corresponding to those different VCNs.

[0151] In addition to VNIC and VCN VR, NVD can execute various software (e.g., daemons) and include one or more hardware components that facilitate various network virtualization functions performed by NVD. For simplicity, these various components are grouped together as... Figure 2 The term "packet processing component" is shown in the diagram. For example, NVD 210 includes packet processing component 286 and NVD 212 includes packet processing component 288. For instance, a packet processing component for an NVD may include a packet processor configured to interact with the NVD's ports and hardware interfaces to monitor all packets received by and transmitted using the NVD and to store network information. This network information may include, for example, network flow information identifying different network flows handled by the NVD and per-flow information (e.g., per-flow statistics). In some embodiments, network flow information may be stored on a per-VNIC basis. The packet processor may perform per-packet manipulation and implement stateful NAT and L4 firewall (FW). As another example, a packet processing component may include a replication agent configured to replicate information stored by the NVD to one or more different replication target repositories. As yet another example, a packet processing component may include a logging agent configured to perform logging functions of the NVD. The packet processing component may also include software for monitoring the performance and health of the NVD and may also monitor the status and health of other components connected to the NVD.

[0152] Figure 1 The components of an example virtual or overlay network are shown, including a VCN, subnets within the VCN, compute instances deployed on the subnets, VNICs associated with the compute instances, VRs for the VCN, and a collection of gateways configured for the VCN. Figure 1 The overlay component described in the text can be made by Figure 2 One or more executions or hosts are described in the physical components. For example, a compute instance in a VCN can be executed or managed by... Figure 2 The VNIC described herein is executed or hosted by one or more host machines. For a compute instance hosted by a host machine, the VNIC associated with that compute instance is typically executed by an NVD connected to that host machine (i.e., VNIC functionality is provided by an NVD connected to that host machine). The VCN VR functionality for a VCN is executed by all NVDs connected to the host machine hosting or executing a compute instance as part of that VCN. The gateway associated with a VCN can be executed by one or more different types of NVDs. For example, some gateways can be executed by smartNICs, while others can be executed by one or more host machines or other implementations of NVDs.

[0153] As described above, compute instances in a client VCN can communicate with various endpoints, which can be within the same subnet as the source compute instance, in a different subnet but within the same VCN as the source compute instance, or communicate with endpoints located outside the VCN of the source compute instance. These communications are facilitated using VNICs, VCN VRs, and gateways associated with the VCNs.

[0154] For communication between two compute instances on the same subnet within a VCN, a VNIC associated with both the source and destination compute instances facilitates the communication. The source and destination compute instances can be hosted by the same host machine or different host machines. Packets originating from the source compute instance can be forwarded from the host machine hosting the source compute instance to an NVD connected to that host machine. On the NVD, packets are processed using a packet processing pipeline, which may include the execution of the VNIC associated with the source compute instance. Because the destination endpoint for the packet is within the same subnet, the execution of the VNIC associated with the source compute instance results in the packet being forwarded to the NVD executing the VNIC associated with the destination compute instance, whereupon the NVD processes the packet and forwards it to the destination compute instance. The VNIC associated with the source and destination compute instances can execute on the same NVD (e.g., when the source and destination compute instances are hosted by the same host machine) or on different NVDs (e.g., when the source and destination compute instances are hosted by different host machines connected to different NVDs). The VNIC can use a routing / forwarding table stored by the NVD to determine the next hop for the packet.

[0155] For packets destined for endpoints in different subnets within the same VCN, the packet originating from the source compute instance is forwarded from the host machine hosting the source compute instance to the NVD connected to that host machine. On the NVD, the packet is processed using a packet processing pipeline, which may include the execution of one or more VNICs and the VR associated with the VCN. For example, as part of the packet processing pipeline, the NVD executes or invokes functionality associated with the VNIC associated with the source compute instance (also known as executing the VNIC). Functionality executed by the VNIC may include viewing the VLAN identifier on the packet. Since the packet's destination is outside the subnet, the VCN VR functionality is then invoked and executed by the NVD. The VCN VR then routes the packet to the NVD executing the VNIC associated with the destination compute instance. The VNIC associated with the destination compute instance then processes the packet and forwards it to the destination compute instance. The VNICs associated with the source and destination compute instances may execute on the same NVD (e.g., when the source and destination compute instances are hosted by the same host machine) or on different NVDs (e.g., when the source and destination compute instances are hosted by different host machines connected to different NVDs).

[0156] If the destination of the packet is outside the VCN of the source compute instance, the packet originating from the source compute instance is forwarded from the host machine hosting the source compute instance to the NVD connected to that host machine. The NVD executes the VNIC associated with the source compute instance. Since the destination endpoint of the packet is outside the VCN, the packet is subsequently processed by the VCN VR used by that VCN. The NVD invokes the VCN VR functionality, which results in the packet being forwarded to the NVD executing the appropriate gateway associated with the VCN. For example, if the destination is an endpoint within a customer's on-premises network, the packet may be forwarded by the VCN VR to the NVD executing the DRG gateway configured for the VCN. The VCN VR may execute on the same NVD as the NVD executing the VNIC associated with the source compute instance, or it may be executed by a different NVD. The gateway may be executed by the NVD, which may be a smartNIC, a host machine, or another NVD implementation. The packet is then processed by the gateway and forwarded to the next hop, which facilitates the forwarding of the packet to its intended destination endpoint. For example, in Figure 2 In the embodiment depicted, packets originating from compute instance 268 can be transmitted from host machine 202 to NVD 210 via link 220 (using NIC 232). On NVD 210, VNIC 276 is invoked because it is the VNIC associated with the source compute instance 268. VNIC 276 is configured to examine the information encapsulated in the packet and determine the next hop for forwarding the packet, with the aim of facilitating the transmission of the packet to its intended destination endpoint, and then forwarding the packet to the determined next hop.

[0157] Compute instances deployed on a VCN can communicate with a variety of endpoints. These endpoints can include endpoints hosted by CSPI 200 and endpoints outside of CSPI 200. Endpoints hosted by CSPI 200 can include instances within the same VCN or other VCNs, which can be the customer's VCN or a VCN not belonging to the customer. Communication between endpoints hosted by CSPI 200 can be performed over physical network 218. Compute instances can also communicate with endpoints not hosted by CSPI 200 or outside of CSPI 200. Examples of these endpoints include endpoints within the customer's on-premises network or data center, or public endpoints accessible via public networks such as the Internet. Communication with endpoints outside of CSPI 200 can use various communication protocols over public networks (e.g., the Internet). Figure 2 (not shown in the image) or a dedicated network ( Figure 2 (Not shown in the image) to execute.

[0158] Figure 2The architecture of the CSPI 200 depicted herein is merely an example and is not intended to be limiting. Variations, alternatives, and modifications are possible in alternative embodiments. For example, in some implementations, the CSPI 200 may have a more advanced architecture than... Figure 2 The systems or components shown may include more or fewer systems or components, and may combine two or more systems, or may have different system configurations or arrangements. Figure 2 The systems, subsystems, and other components described herein may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective system, using hardware, or a combination thereof. The software may be stored on a non-transitory storage medium (e.g., a memory device).

[0159] Figure 4 The diagram depicts a connection between a host machine and an NVD, according to certain embodiments, for providing I / O virtualization to support multi-tenancy. For example... Figure 4 As depicted, host machine 402 executes a hypervisor 404 that provides a virtualized environment. Host machine 402 executes two virtual machine instances, VM1 406 belonging to customer / tenant #1 and VM2 408 belonging to customer / tenant #2. Host machine 402 includes a physical NIC 410 connected to NVD 412 via link 414. Each compute instance is attached to a VNIC executed by NVD 412. Figure 4 In the embodiment, VM1 406 is attached to VNIC-VM1 420 and VM2 408 is attached to VNIC-VM2 422.

[0160] like Figure 4 As shown, NIC 410 includes two logical NICs, logical NIC A 416 and logical NIC B 418. Each virtual machine is attached to its own logical NIC and configured to work with its own logical NIC. For example, VM1 406 is attached to logical NIC A 416 and VM2 408 is attached to logical NIC B 418. Although host machine 402 includes only one physical NIC 410 shared by multiple tenants, due to the logical NICs, each tenant's virtual machines believe they have their own host machine and network interface card.

[0161] In some embodiments, each logical NIC is assigned its own VLAN ID. Therefore, a specific VLAN ID is assigned to logical NIC A 416 for tenant #1, and a separate VLAN ID is assigned to logical NIC B 418 for tenant #2. When a packet is transmitted from VM1 406, the hypervisor appends a tag assigned to tenant #1 to the packet, and the packet is then transmitted from host machine 402 to NVD 412 via link 414. Similarly, when a packet is transmitted from VM2 408, the hypervisor appends a tag assigned to tenant #2 to the packet, and the packet is then transmitted from host machine 402 to NVD 412 via link 414. Thus, a packet 424 transmitted from host machine 402 to NVD 412 has an associated tag 426 identifying the specific tenant and the associated VM. On the NVD, for a packet 424 received from the host machine 402, the tag 426 associated with the packet is used to determine whether the packet is processed by VNIC-VM1 420 or VNIC-VM2 422. The packet is then processed by the corresponding VNIC. Figure 4 The configuration described in [the document] enables each tenant's compute instance to believe that they own their own host machine and NIC. Figure 4 The setup described in [the document] provides I / O virtualization to support multi-tenancy.

[0162] Figure 5 A simplified block diagram of a physical network 500 according to certain embodiments is depicted. Figure 5 The embodiments depicted are structured as Clos networks. Clos networks are a specific type of network topology designed to provide connectivity redundancy while maintaining high bandwidth and maximum resource utilization. Clos networks are non-blocking, multi-stage or multi-layer switching networks, where the number of stages or layers can be two, three, four, five, etc. Figure 5 The embodiment depicted is a Layer 3 network, including Layer 1, Layer 2, and Layer 3. TOR switch 504 represents a Layer 0 switch in a Clos network. One or more NVDs are connected to the TOR switch. Layer 0 switches are also referred to as edge devices of the physical network. Layer 0 switches are connected to Layer 1 switches, also known as leaf switches. Figure 5In the embodiments depicted, a set of "n" Layer 0 TOR switches is connected to a set of "n" Layer 1 switches, forming a pod. Each Layer 0 switch in the pod is interconnected to all Layer 1 switches in that pod, but there is no switch connectivity between pods. In some implementations, two pods are referred to as blocks. Each block is served by or connected to a set of "n" Layer 2 switches (sometimes called backbone switches). There can be several blocks in the physical network topology. The Layer 2 switches are then connected to "n" Layer 3 switches (sometimes called super backbone switches). Packet communication over the physical network 500 is typically performed using one or more Layer 3 communication protocols. Typically, all layers of the physical network (except the TOR layer) are n-way redundant, thus allowing high availability. Policies can be specified for pods and blocks to control the visibility of switches to each other in the physical network, thereby enabling the scaling of the physical network.

[0163] A key characteristic of Clos networks is that the maximum number of hops from one Layer 0 switch to another (or from an NVD connected to a Layer 0 switch to another NVD connected to a Layer 0 switch) is fixed. For example, in a Layer 3 Clos network, a packet takes a maximum of seven hops to reach another NVD, where the source and destination NVDs are connected to the leaf layers of the Clos network. Similarly, in a Layer 4 Clos network, a packet takes a maximum of nine hops to reach another NVD, where the source and destination NVDs are connected to the leaf layers of the Clos network. Therefore, the Clos network architecture maintains consistent latency throughout the network, which is important for communication within and between data centers. Clos topologies are horizontally scalable and cost-effective. Network bandwidth / throughput capacity can be easily increased by adding more switches at each layer (e.g., more leaf switches and backbone switches) and by increasing the number of links between switches in adjacent layers.

[0164] In some embodiments, each resource within the CSPI is assigned a unique identifier called a Cloud Identifier (CID). This identifier is included as part of the resource's information and can be used to manage the resource, for example, via a console or API. An example syntax for a CID is:

[0165]

[0166] in,

[0167] ocid1: A text string indicating the version of the CID;

[0168] resource type: The type of resource (e.g., instance, volume, VCN, subnet, user, group, etc.);

[0169] realm: The realm where the resource resides. Example values ​​are "c1" for the commercial realm, "c2" for the government cloud realm, or "c3" for the federal government cloud realm, etc. Each realm can have its own domain name;

[0170] region: The region where the resource is located. This section may be empty if the region is not applicable to the resource.

[0171] future use: to be reserved for future use.

[0172] Unique ID: The unique part of the ID. The format may vary depending on the type of resource or service.

[0173] L2 Virtual Network

[0174] The number of enterprise customers transitioning their on-premises applications to cloud environments provided by cloud service providers (CSPs) continues to grow rapidly. However, many of these customers quickly realize that the transition to a cloud environment can be very bumpy, requiring them to rebuild and redesign their existing applications to make them work in the cloud environment. This is because applications written for on-premises environments often depend on the physical network's monitoring, availability, and scalability features. Therefore, these on-premises applications need to be rebuilt and redesigned before they can work in a cloud environment.

[0175] There are several reasons why on-premises applications cannot easily transition to cloud environments. One major reason is that current cloud virtual networks operate at Layer 3 of the OSI model, such as at the IP layer, and do not provide the Layer 2 capabilities required by applications. Layer 3-based routing or forwarding involves determining where a packet should be sent (e.g., to which customer instance) based on information contained in the Layer 3 header of the packet (e.g., the destination IP address contained in the Layer 3 header of the packet). To facilitate this, the location of IP addresses in the virtualized cloud network is determined through a centralized control and orchestration system or controller. These can include, for example, IP addresses associated with customer entities or resources within the virtualized cloud environment.

[0176] Many customers run applications in their on-premises environments that have stringent requirements for Layer 2 networking capabilities, issues that current cloud offerings and IaaS providers do not currently address. For example, traffic in current cloud offerings is routed using Layer 3 protocols with Layer 3 headers and does not support the Layer 2 features required by applications. These Layer 2 features may include Address Resolution Protocol (ARP) processing, Media Access Control (MAC) address learning and Layer 2 broadcast capabilities, Layer 2 (MAC-based) forwarding, Layer 2 networking constructs, and others. By providing virtualized Layer 2 networking functionality within a virtualized cloud network, as described in this disclosure, customers can now seamlessly migrate their legacy applications to cloud environments without any substantial rebuilding or redesign. For example, the virtualized Layer 2 networking capabilities described herein enable such applications (e.g., VMware vSphere, vCenter, vSAN, and NSX-T components) to communicate at Layer 2 as they would in an on-premises environment. These applications can run the same versions and configurations in the public cloud, allowing customers to utilize their legacy on-premises applications, including existing knowledge, tools, and processes associated with those applications. Customers are also able to access native cloud services from their applications (e.g., using VMware software-defined data centers (SDDC)).

[0177] As another example, several legacy on-premises applications (e.g., enterprise clustering software applications, network virtual appliances) require Layer 2 broadcast support for failover. Example applications include Fortinet FortiGate, IBM QRadar, Palo Alto firewall, Cisco ASA, Juniper SRX, and Oracle RAC (True Application Clustering). By providing virtualized Layer 2 networking in a virtualized public cloud as described in this disclosure, these applications can now run unchanged in a virtualized public cloud environment. Virtualized Layer 2 networking functionality equivalent to on-premises deployment is provided as described herein. The virtualized Layer 2 networking functionality described in this disclosure supports traditional Layer 2 networking. This includes support for customer-defined VLANs and unicast, broadcast, and multicast Layer 2 traffic functionality. Layer 2-based packet routing and forwarding involves routing or forwarding packets using Layer 2 protocols and information contained in the Layer 2 header of the packet, such as the destination MAC address contained in the Layer 2 header. Protocols used by enterprise applications (such as clustering software applications), such as ARP, gratuitous address resolution protocol (GARP), and reverse address resolution protocol (RARP), can now also work in cloud environments.

[0178] Traditional virtualized cloud infrastructures support Layer 3 networking but not Layer 2 networking for several reasons. Layer 2 networks typically cannot scale as quickly as Layer 3 networks. Layer 2 network control protocols lack the complexity expected of scaling. For example, Layer 3 networks do not have to worry about packet looping, a problem that Layer 2 networks must solve. IP packets (i.e., Layer 3 packets) have a Time-to-Live (TTL) concept, while Layer 2 packets do not. IP addresses contained in Layer 3 packets have topological concepts such as subnets and CIDR ranges, while Layer 2 addresses (e.g., MAC addresses) do not. Layer 3 IP networks have built-in tools that facilitate troubleshooting, such as ping and traceroute for finding path information. Such tools are unavailable for Layer 2. Layer 3 networks support multipathing, which is unavailable in Layer 2 networks. Due to the lack of sophisticated control protocols specifically designed for exchanging information between entities in the network (e.g., Border Gateway Protocol (BGP) and Open Shortest Path First (OSPF)), Layer 2 networks must rely on broadcast and multicast to learn the network sequentially, which negatively impacts network performance. As networks change, the learning process used for Layer 2 must be repeated, while that is not necessary for Layer 3. For these and other reasons, cloud IaaS service providers prefer to provide infrastructure that operates at Layer 3 rather than Layer 2.

[0179] However, despite several drawbacks, many on-premises applications still require Layer 2 functionality. For example, consider a virtualized cloud configuration where a customer (customer 1) has two instances in a virtual network "V," instance A with IP1 and instance B with IP2. These instances can be compute instances (e.g., bare metal, virtual machines, or containers) or service instances (such as load balancers, NFS mount points, or other service instances). Virtual network V is a unique address space isolated from other virtual networks and the underlying physical network. This isolation can be achieved using various techniques, including packet encapsulation or NAT. For this reason, the IP address of the instance in the virtual network used by the customer is different from the address in the physical network hosting it. A centralized SDN (Software-Defined Networking) control plane is provided, which knows the physical IP and virtual interface of all virtual IP addresses. When a packet is sent from instance A to its destination IP2 in virtual network V, the virtual network SDN stack needs to know the location of IP2. It must know this in advance so that it can send the packet to the IP in the physical network hosting the virtual IP address IP2 used for V. The location of virtual IP addresses can be modified in the cloud, thereby altering the relationship between physical IP addresses and virtual IP addresses. Whenever a virtual IP address needs to be moved (e.g., moving an IP address associated with a virtual machine to another virtual machine or migrating a virtual machine to a new physical host), an API call must be made to the SDN control plane to inform the controller that the IP is moving so it can update all participants in the SDN stack, including the packet processor (data plane). However, some application classes do not make such API calls. Examples include various on-premises applications and applications provided by various virtualization software vendors (such as VMware). The value of facilitating virtual Layer 2 networking in a virtualized cloud environment lies in enabling applications not programmed to make such API calls or applications that rely on other Layer 2 networking features, such as supporting non-IP Layer 3 and MAC learning.

[0180] Virtual Layer 2 networks create broadcast domains, where learning is performed by members of the broadcast domain. Within a virtual Layer 2 domain, any IP address can be on any MAC address of any host within that domain, and the system will learn to use standard Layer 2 networking protocols. These networking primitives are virtualized without the central controller explicitly informing the MAC and IP addresses of their location within the virtual Layer 2 network. This enables applications requiring low-latency failover, applications supporting broadcast or multicast protocols to multiple nodes, and legacy applications that don't know how to make API calls to the SDN control plane or API endpoints to determine IP and MAC address locations. Therefore, Layer 2 networking capabilities are needed in virtualized cloud environments to support functionality not available at the IP Layer 3 level.

[0181] Another technical advantage of providing virtual Layer 2 in a virtualized cloud environment is that it enables support for a variety of different Layer 3 protocols (such as IPv4 and IPv6), including non-IP protocols. For example, various non-IP protocols, such as IPX and AppleTalk, can be supported. Because existing cloud IaaS providers do not provide Layer 2 functionality in their virtualized cloud networks, they cannot support these non-IP protocols. By providing the Layer 2 networking functionality described in this disclosure, support can be provided for protocols at Layer 3, as well as for applications that require and rely on the availability of Layer 2 level functionality.

[0182] Using the techniques described in this disclosure, both Layer 3 and Layer 2 functionality are provided in a virtualized cloud infrastructure. As previously mentioned, Layer 3-based networking offers certain efficiencies, particularly suited for scalability, which Layer 2 networking does not provide. Providing Layer 2 functionality in addition to Layer 3 functionality allows full utilization of such efficiencies offered by Layer 3 (e.g., providing a more scalable solution) while providing Layer 2 functionality in a more scalable manner. For example, virtualized Layer 3 avoids the use of broadcasting for learning purposes. By providing Layer 3 for increased efficiency, while providing virtualized Layer 2 to enable applications that require it and those that cannot function without Layer 2 functionality, and supporting non-IP protocols, etc., customers are provided with complete flexibility in a virtualized cloud environment.

[0183] Customers typically have hybrid environments where Layer 2 and Layer 3 environments coexist, and the virtualized cloud environment can now support both. Customers can have Layer 3 networks such as subnets and / or Layer 2 networks such as VLANs, and these two environments can communicate with each other within the virtualized cloud environment.

[0184] Virtualized cloud environments also need to support multi-tenancy. Multi-tenancy makes it technically difficult and complex to provision both Layer 3 and Layer 2 functionality within the same virtualized cloud environment. For example, a Layer 2 broadcast domain must be managed across many different customers in the cloud provider's infrastructure. The embodiments described in this disclosure overcome these technical problems.

[0185] For virtualization providers (e.g., VMware), virtualized Layer 2 networks that emulate physical Layer 2 networks allow workloads to run without modification. Applications provided by such providers can then run on virtualized Layer 2 networks provided by cloud infrastructure. For example, such applications may include a collection of instances that require a Layer 2 network to run. When customers want to elevate such applications from their on-premises environments to virtualized cloud environments, they cannot simply acquire the application and run it in the cloud because those applications rely on the underlying Layer 2 network (e.g., Layer 2 network features are used to perform virtual machine migrations or move the location of MAC and IP addresses), which is not currently provided by virtualized cloud providers. For these reasons, such applications cannot run natively in virtualized cloud environments. Using the techniques described herein, cloud providers provide virtualized Layer 2 networks in addition to providing virtualized Layer 3 networks. Now, such application stacks can run in cloud environments without modification, and nested virtualizations can run in cloud environments. Customers can now run and manage their own Layer 2 applications in the cloud. Application providers do not need to make any changes to their software to facilitate this. Such legacy applications or workloads (e.g., legacy load balancers, legacy applications, KVM, OpenStack, clustering software) can now run unchanged in a virtualized cloud environment.

[0186] By providing the Layer 2 functionality of virtualization as described in this article, virtualized cloud environments can now support a wide range of Layer 3 protocols, including non-IP protocols. Taking Ethernet as an example, various different EtherTypes (a field in the Layer 2 header that tells the type of Layer 3 packet being sent; it tells what protocol is expected at Layer 3) can be supported, including a variety of non-IP protocols. EtherType is a two-octet field in an Ethernet frame. It is used to indicate which protocol is encapsulated in the frame's payload and is used by the data link layer at the receiving end to determine how to process the payload. EtherType also serves as the basis for 802.1Q VLAN tagging and encapsulating packets from VLANs for multiplexing with other VLAN traffic over Ethernet relay. Examples of EtherTypes include IPv4, IPv6, Address Resolution Protocol (ARP), AppleTalk, IPX, etc. Cloud networks that support Layer 2 protocols can support any protocol at Layer 3. Similarly, when cloud infrastructure provides support for Layer 3 protocols, it can support a variety of protocols at Layer 4, such as TCP, UDP, ICMP, etc. When virtualization is provided at Layer 3, the network can be unaffected by Layer 4 protocols. Similarly, when virtualization is provided at Layer 2, the network can be unaffected by Layer 3 protocols. This technology can be extended to support any Layer 2 network type, including FDDI, Infiniband, etc.

[0187] Therefore, many applications written for physical networks (especially those working with clusters of computer nodes sharing a broadcast domain) utilize Layer 2 features not supported by L3 virtual networks. The following six examples highlight the complexities that can result from not providing Layer 2 networking capabilities:

[0188] (1) Assigning MAC and IP addresses without prior API calls. Network appliances and management programs (such as VMware) are not built for cloud virtual networks. They assume they can use MAC addresses as long as they are unique, and either obtain dynamic addresses from a DHCP server or use any IP address assigned to the cluster. Often there is no mechanism to configure them to inform the control plane about the assignment of these Layer 2 and Layer 3 addresses. If the location of the MAC and IP addresses is unknown, then the Layer 3 virtual network does not know where to send traffic.

[0189] (2) Low-latency reallocation of MAC and IP addresses for high availability and live migration. Many on-premises applications use ARP to reassign IP and MAC addresses for high availability—when an instance in a cluster or HA pair stops responding, the new active instance sends a gratuitous ARP (GARP) to reassign the service IP to its MAC address or a reverse ARP (RARP) to reassign the service MAC address to its interface. This is also important when live migrating instances on a hypervisor: the new host must send RARP after a guest migration to redirect guest traffic to the new host. The assignment not only requires no API calls but also extremely low latency (sub-milliseconds). This cannot be accomplished via HTTPS calls to REST endpoints.

[0190] (3) Interface multiplexing via MAC addresses. When a hypervisor hosts multiple virtual machines on a single host, all of these virtual machines are on the same network, and guest interfaces are distinguished by their MAC addresses. This requires support for multiple MAC addresses on the same virtual interface.

[0191] (4) VLAN support. A single physical virtual machine host will need to be located on multiple broadcast domains, as indicated by the VLAN tag. For example, VMware ESX uses VLANs for traffic separation (e.g., a guest virtual machine can communicate on one VLAN, store on another VLAN, and host the virtual machine on yet another VLAN).

[0192] (5) Use of broadcast and multicast traffic. ARP requires L2 broadcast, and there are examples of on-premises applications using broadcast and multicast traffic for cluster and HA applications.

[0193] (6) Support for non-IP traffic. Since L3 networks require IPv4 or IPv6 headers for communication, using any L3 protocol other than IP will not work. L2 virtualization means that the network within a VLAN can be independent of L3 protocols—the L3 header can be IPv4, IPv6, IPX or anything else—or even not exist at all.

[0194] As disclosed herein, a Layer 2 (L2) network can be created within a cloud network. This virtual L2 network comprises one or more virtualized L2 VLANs (hereinafter referred to as VLANs). Each VLAN may include multiple compute instances, each of which may be associated with at least one L2 virtual interface (e.g., an L2 VNIC) and a local switch. In some embodiments, each pair of L2 VNICs and switches is hosted on an NVD. An NVD may host multiple such pairs, each of which is associated with a different compute instance. The set of local switches represents a simulated single switch for the VLAN. An L2 VNIC represents a set of ports on a simulated single switch. VLANs may be connected to other VLANs, Layer 3 (L3) networks, on-premises networks, and / or other networks via VLAN Switching and Routing Service (VSRS) (also referred to herein as a Real Virtual Router (RVR) or L2VSRS).

[0195] Now for reference Figure 6The diagram illustrates one embodiment of a computing network. VCN 602 resides within CSPI 601. VCN 602 includes multiple gateways connecting VCN 602 to other networks. These gateways include DRG 604, which can connect VCN 602 to, for example, an on-premises network (such as on-premises data center 606). Gateways may also include gateway 600, which may include, for example, an LPG for connecting VCN 602 to another VCN, and / or an IGW and / or NAT gateway for connecting VCN 602 to the Internet. Gateways of VCN 602 may also include service gateway 610, which can connect VCN 602 to service network 612. Service network 612 may include one or more databases and / or repositories, including, for example, a self-contained database 614 and / or an object repository 616. Service network may include a conceptual network comprising an aggregation of IP ranges, such as public IP ranges. In some embodiments, these IP ranges may cover some or all of the public services provided by the CSPI 601 provider. For example, these services can be accessed through an Internet gateway or a NAT gateway. In some embodiments, the service network provides a way for services within the service network to be accessed from a local area via a dedicated gateway (service gateway) for that purpose. In some embodiments, the backends of these services may be implemented, for example, in their own private network. In some embodiments, service network 612 may include additional databases.

[0196] VCN 602 can include multiple virtual networks. Each of these networks can include one or more compute instances that can communicate within their respective networks, between networks, or outside of VCN 602. One of the virtual networks of VCN 602 is L3 subnet 620. L3 subnet 620 is a unit subdivision of a configuration created within VCN 602. Subnet 620 can include a virtual Layer 3 network within the virtualized cloud environment of VCN 602, which is hosted on the underlying physical network of CPSI 601. Although Figure 6 A single subnet 620 is depicted, but VCN 602 may have one or more subnets. Each subnet within VCN 602 may be associated with a contiguous range of IP addresses (e.g., 10.0.0.0 / 24 and 10.0.1.0 / 24) that does not overlap with other subnets within that VCN and represents a subset of the address space within that VCN's address space. In some embodiments, this IP address space may be isolated from the address space associated with CPSI 601.

[0197] Subnet 620 includes one or more computing instances, and specifically includes a first computing instance 622-A and a second computing instance 622-B. Computing instances 622-A and 622-B can communicate with each other within subnet 620, or they can communicate with other instances, devices, and / or networks outside subnet 620. Communication outside subnet 620 is enabled by a virtual router (VR) 624. VR 624 enables communication between subnet 620 and other networks of VCN 602. For subnet 620, VR 624 represents a logical gateway that enables subnet 620 (i.e., computing instances 622-A and 622-B) to communicate with endpoints on other networks within VCN 602 and with other endpoints outside VCN 602.

[0198] The VCN 602 may also include additional networks, and specifically may include one or more L2 VLANs (referred to herein as VLANs), which are examples of virtual L2 networks. These one or more VLANs may each comprise a virtual Layer 2 network located in the cloud environment of the VCN 602 and / or hosted by the underlying physical network of the CPSI 601. Figure 6 In this embodiment, VCN 602 includes VLAN A 630 and VLAN B 640. Each VLAN 630, 640 within VCN 602 may be associated with a contiguous range of IP addresses (e.g., 10.0.0.0 / 24 and 10.0.1.0 / 24) that do not overlap with other networks within the VCN (such as other subnets or VLANs within the VCN) and represent a subset of the address space within the VCN's address space. In some embodiments, this IP address space of the VLAN may be isolated from the address space associated with CPSI 601.

[0199] Each of VLANs 630 and 640 may include one or more compute instances, and specifically, VLAN A 630 may include, for example, a first compute instance 632-A and a second compute instance 632-B. In some embodiments, VLAN A 630 may include additional compute instances. VLAN B 640 may include, for example, a first compute instance 642-A and a second compute instance 642-B. Each of compute instances 632-A, 632-B, 642-A, and 642-B may have an IP address and a MAC address. These addresses may be assigned or generated in any desired manner. In some embodiments, these addresses may be within the CIDR of the VLAN of the compute instance, and in some embodiments, these addresses may be any addresses. In embodiments where the compute instance of the VLAN communicates with an endpoint outside the VLAN, one or both of these addresses may come from the VLAN CIDR, while when all communication is within the VLAN, these addresses are not limited to addresses within the VLAN CIDR. Unlike networks where addresses are assigned by the control plane, the IP and / or MAC addresses of compute instances in a VLAN can be assigned by the users / clients of that VLAN, and these IP and / or MAC addresses can then be discovered and / or learned by the compute instances in the VLAN according to the learning process discussed below.

[0200] Each VLAN may include a VLAN Switching and Routing Service (VSRS), and specifically, VLAN A 630 includes VSRS A 634 and VLAN B 640 includes VSRS B 644. Each VSRS 634, 644 participates in Layer 2 switching and local learning within the VLAN and also performs all necessary Layer 3 network functions, including ARP, NDP, and routing. VSRS performs ARP (a Layer 2 protocol) because VSRS must map IPs to MAC addresses.

[0201] Within these cloud-based VLANs, each virtual interface or virtual gateway can be associated with one or more Media Access Control (MAC) addresses, which can be virtual MAC addresses. Within a VLAN, one or more compute instances 632-A, 632-B, 642-A, 642-B (e.g., bare metal, VMs, or containers, and / or one or more service instances) can communicate directly with each other via virtual switches. Communication outside the VLAN, such as with other VLANs or with L3 networks, is enabled via VSRS 634, 644. VSRS 634, 644 are distributed services that provide Layer 3 functionality, such as IP routing, to the VLAN network. In some embodiments, VSRS 634, 644 are horizontally scalable, highly available routing services that can reside at the intersection of IP and L2 networks and participate in IP routing and L2 learning within the cloud-based L2 domain.

[0202] VSRS 634 and 644 can be distributed across multiple nodes within an infrastructure, and their functionality can be scalable, particularly horizontally scalable. In some embodiments, each node implementing the functionality of VSRS 634 and 644 shares and replicates the functionality of the router and / or switch with each other. Furthermore, these nodes can present themselves as a single VSRS 634 and 644 to all instances in VLANs 630 and 640. VSRS 634 and 644 can be implemented on any virtualization device within the CSPI 601, specifically within a virtual network. Therefore, in some embodiments, VSRS 634 and 644 can be implemented on any virtual network virtualization device, including NICs, SmartNICs, switches, intelligent switches, or general-purpose computing hosts.

[0203] VSRS 634 and 644 can be services residing on one or more hardware nodes supporting a cloud network. These hardware nodes may be, for example, one or more servers, such as one or more x86 servers, or one or more networking devices supporting the cloud network, such as one or more NICs, and specifically one or more SmartNICs. In some embodiments, VSRS 634 and 644 can be implemented on a server cluster. Therefore, VSRS 634 and 644 can be services distributed across a cluster of nodes, which can be a centrally managed cluster or distributed at the edge of virtual networking executors that participate in and share L2 and L3 learning and evaluate routing and security policies. In some embodiments, each VSRS instance can update other VSRS instances with new mapping information, as this new mapping information is learned by the VSRS instance. For example, when a VSRS instance learns the IP, interface, and / or MAC mappings of one or more CIs in its VLAN, the VSRS instance can provide this updated information to other VSRS instances within the VCN. Through this cross-update, the VSRS instance associated with the first VLAN can know the mappings for CIs in other VLANs (in some embodiments, for CIs in other VLANs within VCN 602), including IP, interface, and / or MAC mappings. These updates can be significantly accelerated when the VSRS resides on a server cluster and / or is distributed across nodes.

[0204] In some embodiments, VSRS 634, 644 may also host one or more higher-level services necessary for networking, including but not limited to: DHCP relay; DHCP (managed); DHCPv6; neighbor discovery protocols (such as IPv6 neighbor discovery protocol); DNS; managed DNSv6; SLAAC for IPv6; NTP; metadata services; and blockstore mount points. In some embodiments, VSRS may support one or more Network Address Translation (NAT) functions for translation between network address spaces. In some embodiments, VSRS may incorporate anti-spoofing, anti-MAC spoofing, ARP cache poisoning protection for IPv4, IPv6 Route Advertisement (RA) protection, DHCP protection, packet filtering using Access Control Lists (ACLs); and / or reverse path forwarding checks. Functions that VSRS can implement include, for example, ARP, GARP, packet filtering (ACLs), DHCP relay, and / or IP routing protocols. For example, VSRS 634 and 644 can learn MAC addresses, invalidate expired MAC addresses, handle MAC address migration, vet MAC address information, handle MAC information flooding, handle storm control, prevent loops, multicast via Layer 2 protocols such as IGMP in the cloud, collect statistics including logs, collect statistics using SNMP, monitor, and / or collect and use statistics for broadcast, total traffic, bits, cross-tree groups, etc.

[0205] Within a virtual network, VSRS 634 and 644 can manifest as distinct instantiations. In some embodiments, each of these VSRS instantiations can be associated with VLAN 630 and 640, and in some embodiments, each VLAN 630 and 640 can have an instantiation of VSRS 634 and 644. In some embodiments, each instantiation of VSRS 634 and 644 can have one or more unique tables corresponding to the VLAN 630 and 640 associated with each instantiation of VSRS 634 and 644. Each instantiation of VSRS 634 and 644 can generate and / or orchestrate a unique table associated with that instantiation of VSRS 634 and 644. Therefore, while a single service can provide VSRS 634 and 644 functionality for one or more cloud networks, individual instantiations of VSRS 634 and 644 within a cloud network can have unique Layer 2 and Layer 3 forwarding tables, while multiple such customer networks can have overlapping Layer 2 and Layer 3 forwarding tables.

[0206] In some embodiments, VSRS 634, 644 can support conflicting VLANs and IP spaces across multiple tenants. This can include multiple tenants on the same VSRS 634, 644. In some embodiments, some or all of these tenants may choose to use some or all of the following: the same IP address space, the same MAC space, and the same VLAN space. This provides users with great flexibility in address selection. In some embodiments, this multi-tenancy is supported by providing each tenant with a different virtual network, which is a private network within a cloud network. Each virtual network is assigned a unique identifier. Similarly, in some embodiments, each host may have a unique identifier, and / or each virtual interface or virtual gateway may have a unique identifier. In some embodiments, these unique identifiers, and specifically the unique identifiers of the tenant's virtual network, may be encoded in each communication. By providing a unique identifier for each virtual network and including it in communications, a single instantiation of VSRS 634, 644 can serve multiple tenants with overlapping addresses and / or namespaces.

[0207] VSRS 634 and 644 can perform these switching and / or routing functions to facilitate and / or enable the creation and / or communication with L2 networks within VLANs 630 and 640. These VLANs 630 and 640 can be found within a cloud computing environment, and more specifically, within a virtual network within that cloud computing environment.

[0208] For example, each of VLANs 630 and 640 includes multiple compute instances 632-A, 632-B, 642-A, and 642-B. VSRS 634 and 644 enable communication between compute instances in one VLAN 630 or 640 and compute instances in another VLAN 630 or 640 or in subnet 620. In some embodiments, VSRS 634 and 644 enable communication between compute instances in one VLAN 630 or 640 and another VCN, or another network outside that VCN (including the Internet, on-premises data centers, etc.). In such embodiments, for example, a compute instance (such as compute instance 632-A) may send communication to an endpoint outside the VLAN (in this example, an endpoint outside L2 VLAN A 630). Compute instance (632-A) may send communication to VSRS A 634, which may direct the communication to routers 624 and 644 or gateways 604, 608, and 610 coupled to the desired endpoint. Routers 624, 644 or gateways 604, 608, 610 coupled to the desired endpoint can receive communication from the computing instance (632-A) and can direct the communication to the desired endpoint.

[0209] Now for reference Figure 7 The diagram illustrates the logical and hardware schematics of VLAN 700. As can be seen, VLAN 700 includes multiple endpoints, specifically multiple compute instances and VSRS. Multiple compute instances (CIs) are instantiated on one or more host machines. In some embodiments, this can be a one-to-one relationship, such that each CI is instantiated on a unique host machine, and / or in some embodiments, this can be a many-to-one relationship, such that multiple CIs are instantiated on a single shared host machine. In various embodiments, CIs can be Layer 2 CIs by being configured to communicate with each other using the L2 protocol. Figure 7 This describes a scenario where some CIs are instantiated on a single host machine, and some CIs share a common host machine. For example... Figure 7 As seen in the diagram, instance 1 (CI1) 704-A is instantiated on host machine 1 702-A, instance 2 (CI2) 704-B is instantiated on host machine 2 702-B, and instance 3 (CI3) 704-AC and instance 4 (CI4) 704-D are instantiated on the shared host machine 702-C.

[0210] Each of CI 704-A, 704-B, 704-C, and 704-D is communicatively coupled to the other CI 704-A, 704-B, 704-C, and 704-D in VLAN 700 and to VSRS 714. Specifically, each of CI 704-A, 704-B, 704-C, and 704-D is connected to the other CI 704-A, 704-B, 704-C, and 704-D in VLAN 700 via an L2 VNIC and a switch, and is connected to VSRS 714. Each CI 704-A, 704-B, 704-C, and 704-D is associated with a unique L2 VNIC and switch. The switch can be a local L2 virtual switch and is uniquely associated with and deployed for the L2 VNIC. Specifically, CI1 704-A is associated with L2 VNIC 1 708-A and switch 1 710-A, CI2 704-B is associated with L2 VNIC 2708-B and switch 710-B, CI3 704-C is associated with L2 VNIC 3 708-C and switch 3 710-C, and CI4 704-D is associated with L2 VNIC 4 708-D and switch 4 710-D.

[0211] In some embodiments, each L2 VNIC 708 and its associated switch 710 can be instantiated on NVD 706. This instantiation can be a one-to-one relationship, whereby a single L2 VNIC 708 and its associated switch 710 are instantiated on a unique NVD 706, or it can be a many-to-one relationship, where multiple L2 VNIC 708s and their associated switches 710s are instantiated on a single shared NVD 706. Specifically, L2 VNIC 1 708-A and switch 1 710-A are instantiated on NVD 1 706-A, L2 VNIC 2 708-B and switch 2 710-B are instantiated on NVD 2, and L2 VNIC 3 708-C and switch 3 710-C, as well as L2 VNIC 4 708-D and switch 710-D, are all instantiated on a shared NVD (i.e., NVD 706-C).

[0212] In some embodiments, the VSRS 714 can support conflicting VLANs and IP spaces across multiple tenants. This can include multiple tenants on the same VSRS 714. In some embodiments, some or all of these tenants may choose to use some or all of the following: the same IP address space, the same MAC space, and the same VLAN space. This provides users with great flexibility in address selection. In some embodiments, this multi-tenancy is supported by providing each tenant with a different virtual network, which is a private network within the cloud network. Each virtual network (e.g., each VLAN or VCN) is assigned a unique identifier, such as a VCN identifier that can be a VLAN identifier. This unique identifier may be selected by, for example, the control plane, and specifically by the CSPI control plane. In some embodiments, this unique identifier may include one or more bits, which may be included and / or used in packet encapsulation.

[0213] Similarly, in some embodiments, each host may have a unique identifier, and / or each virtual interface or virtual gateway may have a unique identifier. In some embodiments, these unique identifiers, and specifically unique identifiers of a tenant's virtual network, may be encoded in each communication. By providing a unique identifier for each virtual network and including it in communications, a single instantiation of VSRS can serve multiple tenants with overlapping addresses and / or namespaces.

[0214] In some embodiments, the VSRS 714 can determine which tenant a packet belongs to based on the VCN identifier and / or VLAN identifier associated with the communication, specifically within the VCN header of the communication. In the embodiments disclosed herein, communication leaving or entering a VLAN may have a VCN header that may include a VLAN identifier. Based on the VCN header containing the VLAN identifier, the VSRS can determine the lease, or in other words, the receiving VSRS can determine which VLAN and / or which tenant the communication is sent to.

[0215] Furthermore, each compute instance belonging to a VLAN (e.g., an L2 compute instance) is assigned a unique interface identifier that identifies the L2 VNIC associated with the compute instance. The interface identifier can be included in traffic originating from and / or destined for the compute instance (e.g., by including it in the frame header) and can be used by the NVD to identify the L2 VNIC associated with the compute instance. In other words, the interface identifier can uniquely indicate a compute instance and / or its associated L2 VNIC. Figure 7 As indicated in the document, switches 710-A, 710-B, 710-C, and 710-D can be combined to form an L2 distributed switch 712, also referred to herein as a distributed switch 712. From the customer's perspective, each switch 710-A, 710-B, 710-C, and 710-D in the L2 distributed switch 712 is a single switch connected to all CIs in the VLAN. However, the distributed switches, simulating the user experience of a single switch, are infinitely scalable and include a collection of local switches (e.g., in...). Figure 7 In the illustrative examples, switches 710-A, 710-B, 710-C, and 710-D are shown. Figure 7 As shown, each CI executes on a host machine connected to the NVD. For each CI on a host connected to the NVD, the NVD hosts a Layer 2 VNIC and a local switch associated with the compute instance (e.g., an L2 virtual switch, local to the NVD, associated with a Layer 2 VNIC, and a member or component of an L2 distributed switch 712). A Layer 2 VNIC represents a port of the compute instance on a Layer 2 VLAN. The local switch connects L2 VNICs to other L2 VNICs (e.g., other ports) associated with other compute instances on other Layer 2 VLANs.

[0216] Each of CI 704-A, 704-B, 704-C, and 704-D can communicate with the other CI 704-A, 704-B, 704-C, and 704-D in VLAN 700, or with VSRS 714. One of CI 704-A, 704-B, 704-C, and 704-D sends a packet to another of CI 704-A, 704-B, 704-C, and 704-D or to VSRS 714 by sending the MAC address and interface identifier of the recipient in CI 704-A, 704-B, 704-C, and 704-D, or to VSRS 714. The MAC address and interface identifier can be included in the packet header. As explained above, the interface identifier can indicate the receiver of CI 704-A, 704-B, 704-C, 704-D or the L2 VNIC of VSRS 714.

[0217] In one embodiment, CI1 704-A can be the source CI, L2 VNIC 708-A can be the source VNIC, and switch 710-A can be the source switch. In this embodiment, CI3 704-C can be the destination CI, and L2 VNIC 3708-C can be the destination VNIC. The source CI can send packets with source and destination MAC addresses. This packet can be intercepted by NVD 706-A, thereby instantiating the source VNIC and the source switch.

[0218] For VLAN 700, L2 VNICs 708-A, 708-B, 708-C, and 708-D can each learn a mapping from MAC addresses to L2 VNIC interface identifiers. This mapping can be learned based on packets and / or communications received within VLAN 700. Based on this pre-determined mapping, the source VNIC can determine the interface identifier of the destination interface associated with the destination CI within the VLAN and can encapsulate the packet. In some embodiments, this encapsulation can include GENEVE encapsulation, and specifically L2 GENEVE encapsulation, which includes the L2 (Ethernet) header of the encapsulated packet. The encapsulated packet can identify the destination MAC, destination interface identifier, source MAC, and source interface identifier.

[0219] The source VNIC can forward the encapsulated packet to the source switch, which can then direct the packet to the destination VNIC. Upon receiving the packet, the destination VNIC can decapsulate it and then provide it to the destination CI.

[0220] Now for reference Figure 8 This illustrates a logical diagram of multiple connected L2 VLAN 800 connections. Figure 8In the specific embodiment depicted, both VLANs reside within the same VCN. As can be seen, multiple connected L2 VLANs 800 may include a first VLAN (VLAN A 802-A) and a second VLAN (VLAN B 802-B). Each of these VLANs 802-A and 802-B may include one or more CIs, each CI may have an associated L2 VNIC and an associated L2 virtual switch. Additionally, each of these VLANs 802-A and 802-B may include a VSRS.

[0221] Specifically, VLAN A 802-A may include Instance 1 804-A connected to L2 VNIC 1 806-A and Switch 1 808-A, Instance 2 804-B connected to L2 VNIC 2 806-B and Switch 808-B, and Instance 3 804-C connected to L2 VNIC 3806-C and Switch 3 808-C. VLAN B 802-B may include Instance 4 804-D connected to L2 VNIC 4 806-D and Switch 4 808-D, Instance 5 804-E connected to L2 VNIC 5 806-E and Switch 808-E, and Instance 6 804-F connected to L2 VNIC 6 806-F and Switch 3 808-F. VLAN A 802-A may also include VSRS A 810-A, and VLAN B 802-B may include VSRS B 810-B. Each of CI 804-A, 804-B, and 804-C of VLAN A 802-A may be communicatively coupled to VSRS A 810-A, and each of CIS 804-D, 804-E, and 804-F of VLAN B 802-B may be communicatively coupled to VSRS B 810-B.

[0222] VLAN A 802-A can be communicatively coupled to VLAN B 802-B via their respective VSRS 810-A, 810-B. Each VSRS can also be coupled to gateway 812, which provides access to other networks outside the VCN where VLANs 802-A and 802-B reside via CI 804-A, 804-B, 804-C, 804-D, 804-E, and 804-F in each VLAN 802-A and 802-B. In some embodiments, these networks may include, for example, one or more on-premises networks, another VCN, a service network, a public network such as the Internet, etc.

[0223] Each of the CIs 804-A, 804-B, and 804-C in VLAN A 802-A can communicate with CIs 804-D, 804-E, and 804-F in VLAN B 802-B via the VSRSs 810A and 810-B of each VLAN 802-A and 802-B. For example, one of the CIs 804-A, 804-B, 804-C, 804-D, 804-E, and 804-F in one of VLANs 802-A and 802-B can send packets to CIs 804-A, 804-B, 804-C, 804-D, 804-E, and 804-F in the other VLAN 802-A and 802-B. This packet can leave the source VLAN via the VSRS of the source VLAN and enter the destination VLAN, and is routed to the destination CI via the destination VSRS.

[0224] In one embodiment, CI 1 804-A can be a source CI, L2 VNIC 806-A can be a source VNIC, and switch 808-A can be a source switch. In this embodiment, CI 5 804-E can be a destination CI, and L2 VNIC 5806-E can be a destination VNIC. VSRS A 810-A can be a source VSRS identified as an SVSRS, and VSRS B 810-B can be a destination VSRS identified as a DVSRS.

[0225] The source CI can send packets with MAC addresses. These packets can be intercepted by the NVD of the instantiated source VNIC and the source switch. The source VNIC encapsulates the packet. In some embodiments, this encapsulation may include Geneve encapsulation, and specifically L2 Geneve encapsulation. The encapsulated packet can identify the destination address of the destination CI. In some embodiments, this destination address may also include the destination address of the destination VSRS. The destination address of the destination CI may include the destination IP address, the destination MAC address of the destination CI, and / or the destination interface identifier of the destination VNIC of the destination CI. The destination address of the destination VSRS may include the IP address of the destination VSRS, the interface identifier of the destination VNIC associated with the destination VSRS, and / or the MAC address of the destination VSRS.

[0226] The source VSRS can receive packets from the source switch, look up the VNIC mapping from the packet's destination address (which can be a destination IP address), and forward the packet to the destination VSRS. The destination VSRS can receive the packet. Based on the destination address contained in the packet, the destination VSRS can forward the packet to the destination VNIC. The destination VNIC can receive and decapsulate the packet, and then provide the packet to the destination CI.

[0227] Now for reference Figure 9 This illustrates a logical diagram of multiple connected L2 VLANs and subnet 900. Figure 9 In the specific embodiment shown, the VLAN and subnet reside within the same VCN. This is indicated by the virtual routers and VSRS of the VLAN and subnet being directly connected, rather than through a gateway.

[0228] As can be seen, this can include a first VLAN (VLAN A 902-A), a second VLAN (VLAN B 902-B), and subnet 930. Each of these VLANs 902-A and 902-B can include one or more CIs, each of which can have an associated L2 VNIC and an associated L2 switch. Additionally, each of these VLANs 902-A and 902-B can include a VSRS. Similarly, subnet 930, which can be an L3 subnet, can include one or more CIs, each of which can have an associated L3 VNIC, and L3 subnet 930 can include a virtual router 916.

[0229] Specifically, VLAN A 902-A may include Instance 1 904-A connected to L2 VNIC 1 906-A and Switch 1 908-A, Instance 2 904-B connected to L2 VNIC 2 906-B and Switch 908-B, and Instance 3 904-C connected to L2 VNIC 3 906-C and Switch 3 908-C. VLAN B 902-B may include Instance 4 904-D connected to L2 VNIC 4 906-D and Switch 4 908-D, Instance 5 904-E connected to L2 VNIC 5 906-E and Switch 908-E, and Instance 6 904-F connected to L2 VNIC 6 906-F and Switch 3 908-F. VLAN A 902-A may also include VSRS A 910-A, and VLAN B 902-B may include VSRS B 910-B. Each of the CIs 904-A, 904-B, and 904-C of VLAN A 902-A may be communicatively coupled to VSRS A 910-A, and each of the CIs 904-D, 904-E, and 904-F of VLAN B 902-B may be communicatively coupled to VSRS B 910-B. L3 subnet 930 may include one or more CIs, and specifically may include instance 7 904-G, which is communicatively coupled to L3 VNIC 7 906-G. L3 subnet 930 may include virtual router 916.

[0230] VLAN A 902-A can be communicatively coupled to VLAN B 902-B via their respective VSRS 910-A, 910-B. L3 subnet 930 can be communicatively coupled to VLAN A 902-A and VLAN B 902-B via virtual router 916. Each virtual router 916 and VSRS instances 910-A, 910-B can also be coupled to gateway 912, which provides access to other networks outside the VCN where VLANs 902-A, 902-B and subnet 930 reside (CI 904-A, 904-B, 904-C, 904-D, 904-E, 904-F, 904-G) reside in each of VLANs 902-A, 902-B and subnet 930. In some embodiments, these networks may include, for example, one or more on-premises networks, another VCN, a service network, a public network such as the Internet.

[0231] Each VSRS instance 910-A, 910-B can provide an exit path for packets leaving the associated VLAN 902-A, 902-B, and an entry path for packets entering the associated VLAN 902-A, 902-B. From VSRS instances 910-A, 910-B of VLAN 902-A, 902-B, packets can be sent to any desired endpoint, including L2 endpoints (such as L2CIs in another VLAN on the same VCN or on a different VCN or network) and / or L3 endpoints (such as L3 CIs in a subnet on the same VCN or on a different VCN or network).

[0232] In one embodiment, CI 1 904-A can be a source CI, L2 VNIC 906-A can be a source VNIC, and switch 908-A can be a source switch. In this embodiment, CI 7 904-G can be a destination CI, and VNIC 7906-G can be a destination VNIC. VSRS A 910-A can be a source VSRS identified as an SVSRS, and virtual router (VR) 916 can be a destination VR.

[0233] The source CI can send packets with MAC addresses. These packets can be intercepted by the instantiated source VNIC and the source switch's NVD. The source VNIC encapsulates the packet. In some embodiments, this encapsulation may include Geneve encapsulation, and specifically L2 Geneve encapsulation. The encapsulated packet can identify the destination address of the destination CI. In some embodiments, this destination address may also include the destination address of the VSRS of the source CI's VLAN. The destination address of the destination CI may include the destination IP address, the destination MAC address of the destination CI, and / or the destination interface identifier of the destination VNIC of the destination CI.

[0234] The source VSRS can receive packets from the source switch, look up the VNIC mapping from the packet's destination address (which can be a destination IP address), and forward the packet to the destination VR. The destination VR can receive the packet. Based on the destination address contained in the packet, the destination VR can forward the packet to the destination VNIC. The destination VNIC can receive and decapsulate the packet, and then provide the packet to the destination CI.

[0235] Learning within a virtual L2 network

[0236] Now for reference Figure 10This diagram illustrates an embodiment of intra-VLAN communication and learning within VLAN 1000. The learning here specifically refers to how L2 VNICs, VSRS VNICs, and / or L2 virtual switches learn the association between MAC addresses and L2 VNICs / VSRS VNICs (more specifically, the association between MAC addresses associated with L2 compute instances or VSRS and identifiers associated with the L2 VNICs of these L2 compute instances or with the VSRS VNICs). Generally, learning is based on inbound traffic. For the interface-to-MAC address learning aspect, this learning differs from the learning process (e.g., ARP process) that L2 compute instances can implement to learn destination MAC addresses. Both learning processes (e.g., the learning process of L2 VNICs / L2 virtual switches and the learning process of L2 compute instances) in... Figure 12 The diagram shows a joint implementation.

[0237] As can be seen, VLAN 1000 includes compute instance 1 1000-A, which is communicatively coupled to NVD 1 1001-A. NVD1 1001-A instantiates L2 VNIC 1 1002-A and L2 switch 1 1004-A. VLAN 1000 also includes compute instance 2 1000-B, which is communicatively coupled to NVD 2 1001-B. NVD 2 1001-B instantiates L2 VNIC 2 1002-B and L2 switch 2 1004-A. VLAN 1000 also includes VSRS 1010 running on a server cluster, which includes VSRS VNIC 1002-C and VSRS switch 1004-C. All switches 1004-A, 1004-B, and 1004-C together form a distributed switch. VSRS 1010 is communicatively coupled to endpoint 1008, which may include a gateway and, more specifically, may include, for example, another L2 / L3 router in the form of a VSRS, or, for example, an L3 router in the form of a virtual router.

[0238] The control plane 1001 of the VCN hosting VLAN 1000 maintains information identifying each L2 VNIC on VLAN 1000 and the network placement of the L2 VNICs. For example, for an L2 VNIC, this information may include the interface identifier associated with the L2 VNIC and / or the physical IP address of the NVD hosting the L2 VNIC. The control plane 1001 uses this information to update the interfaces in VLAN 1000, such as periodically or on demand. Therefore, each L2 VNIC 1002-A, 1002-B, and 1002-C in the VLAN receives information from the control plane 1001 identifying the interfaces in the VLAN and populates a table with this information. The table populated by the L2 VNICs can be stored locally in the NVD hosting the L2 VNICs. If L2 VNICs 1002-A, 1002-B, and 1002-C already include a current table, L2 VNICs 1002-A, 1002-B, and 1002-C can determine any differences between their current tables and the information / tables received from control plane 1001. In some embodiments, L2 VNICs 1002-A, 1002-B, and 1002-C can update their tables to match the information received from control plane 1001.

[0239] like Figure 10As seen in the diagram, packets are transmitted via L2 switches 1004-A, 1004-B, and 1004-C, and received by receiving L2 VNICs 1002-A, 1002-B, and 1002-C. When a packet is received by an L2 VNIC 1002-A, 1002-B, or 1002-C, the VNIC learns the mapping between the source interface (source VNIC) and the source MAC address of the packet. Based on a table of information received from control plane 1010, the VNIC can map the source MAC address (from the received packet, also referred to herein as a frame) to the interface identifier of the source VNIC and the IP address of the VNIC and / or the IP address of the NVD hosting the VNIC (where the interface identifier and (one or more) IP addresses are obtained from the table). Therefore, L2 VNICs 1002-A, 1002-B, and 1002-C learn the mapping from received communication and / or packet learning to MAC addresses. L2 VNICs 1002-A, 1002-B, and 1002-C can update their tables, L2 FWD tables 1006-A, 1006-B, and 1006-C, with this learned mapping information. In some embodiments, the L2 forwarding table includes MAC addresses and associates them with at least one of an interface identifier or a physical IP address. In such embodiments, a MAC address refers to the address assigned to an L2 compute instance and may correspond to a port emulated by the L2 VNIC associated with the L2 compute instance. The interface identifier can uniquely identify the L2 VNIC and / or the L2 compute instance. The virtual IP address can be the address of the L2 VNIC. And the physical IP address can be the IP address of the NVD hosting the L2 VNIC. L2 forwarding updates from an L2 VNIC can be stored locally on the NVD hosting the L2 VNIC and used by the L2 virtual switch associated with the L2 VNIC for guide frames. In some embodiments, L2 VNICs within a shared VLAN can share all or part of their mapping tables with each other.

[0240] Given the network architecture described above, traffic flow will now be described. For clarity, traffic flow will be described in conjunction with compute instance 21000-B, L2 VNIC 2 10002-B, L2 switch 2 1004-B, and NVD 2 1001-B. This description is equivalent to traffic flowing to and / or from other compute instances.

[0241] As explained above, a VLAN is implemented within a VCN as an overlay L2 network on top of an L3 physical network. An L2 compute instance of a VLAN can send or receive L2 frames that include an overlay MAC address (also known as a virtual MAC address) as the source and destination MAC addresses. L2 frames can also encapsulate packets that include an overlay IP address (also known as a virtual IP address) as the source and destination IP addresses. In some embodiments, the overlay IP address of the compute instance may belong to the CIDR range of the VLAN. Another overlay network IP address may belong to the CIDR range (in which case the L2 frame is within the VLAN) or flow outside the CIDR range (in which case the L2 frame is sent to or received from another network). L2 frames may also include a VLAN tag that uniquely identifies the VLAN and a VLAN tag that can be used to distinguish multiple L2 VNICs on the same NVD. L2 frames can be received by the NVD via a tunnel in encapsulated packets from the host machine of the compute instance, from another NVD, or from a cluster of servers hosting VSRS. In these different cases, the encapsulated packets can be L3 packets sent over the physical network, where the source and destination IP addresses are physical IP addresses. Different types of encapsulation are possible, including Geneve encapsulation. NVD can decapsulate received packets to extract L2 frames. Similarly, to transmit an L2 frame, NVD can encapsulate it in an L3 packet and transmit it on the physical substrate.

[0242] For intra-VLAN outbound traffic from Instance 2 1000-B, NVD 2 1001-B receives the frame from the host machine of Instance 2 1000-B via the Ethernet link. This frame includes an interface identifier that identifies L2VNIC 2 1000-B. The frame includes the overlay MAC address of Instance 2 1000-B (e.g., M.2) as the source MAC address and the overlay MAC address of Instance 1 1000-A (e.g., M.1) as the destination MAC address. Given the interface identifier, NVD 2 1001-B passes the frame to L2VNIC 2 1002-B for further processing. L2VNIC 2 1002-B forwards the frame to L2 switch 2 1004-B. Based on L2 forwarding table 1006-B, L2 switch 2 1004-B determines whether the destination MAC address is known (e.g., matches an entry in L2 forwarding table 1006-B).

[0243] If known, L2 switch 2 1004-B determines that L2 VNIC 1 1002-A is the relevant tunnel endpoint and forwards the frame to L2 VNIC 1 1002-A. Forwarding may include frame encapsulation in packets and packet decapsulation (e.g., Geneve encapsulation and decapsulation), wherein the packet includes the frame, the physical IP address of NVD 1 1001-A (e.g., IP.1) as the destination address, and the physical IP address of NVD 2 1001-B (e.g., IP.2) as the source address.

[0244] If the information is unknown, then L2 switch 2 1004-B broadcasts the frame to each L2 VNIC in the VLAN (e.g., including L2VNIC 1 1002-A and any other L2VNICs in the VLAN), where the broadcast frame is processed (e.g., encapsulated, transmitted, decapsulated) among the relevant NVDs. In some embodiments, this broadcast can be performed, or more specifically simulated, at the physical network, thereby encapsulating the frame individually into each L2 VNIC, including the VSRS in the VLAN. Thus, the broadcast is simulated via a series of replicated unicast packets at the physical network. Furthermore, each L2 VNIC receives the frame and learns the association between the interface identifier of L2 VNIC 21002-B and the source MAC address (e.g., M.2) and source physical IP address (e.g., IP.2).

[0245] For intra-VLAN inbound traffic from Compute Instance 1 1000-A to Compute Instance 2 1000-B, NVD 2 1001-B receives packets from NVD 1. The packet has IP.1 as the source address and a frame, where the frame includes M.2 as the destination MAC address and M.1 as the source MAC address. The frame also includes the network identifier of L2 VNIC 1 1002-A. After decapsulation, L2VNIC 2 receives the frame and learns that this interface identifier is associated with M.1 and / or IP.1, and if this was previously unknown, stores the learned information in the L2 forwarding table 1006-B at switch 2 for subsequent outbound traffic. Alternatively, after decapsulation, L2VNIC 2 receives the frame and learns that this interface identifier is associated with M.1 and / or IP.1, and if this information is known, refreshes the expiration time.

[0246] For egress traffic from instance 2 1000-B in VLAN 1000 to an instance in another VLAN, a similar flow to the egress traffic described above can exist, in addition to using VSRS VNICs and VSRS switches. Specifically, the destination MAC address is not within the L2 broadcast of VLAN 1000 (it is within another L2 VLAN). Therefore, the destination instance's overlay destination IP address (e.g., IP.A) is used for this egress traffic. For example, L2 VNIC 2 1002-B determines that IP.A is outside the CIDR range of VLAN 1000. Therefore, L2 VNIC 2 1002-B sets the destination MAC address to the default gateway MAC address (e.g., M.DG). Based on M.DG, L2 switch 2 1004-B sends the egress traffic to the VSRS VNIC (e.g., via tunnel, with appropriate end-to-end encapsulation). The VSRS VNIC forwards the egress traffic to the VSRS switch. Next, the VSRS switch performs routing functions, where, based on the destination IP address (e.g., IP.A), the VSRS switch in VLAN 1000 sends egress traffic to the VSRS switch in another VLAN (e.g., via a virtual router between the two VLANs, also with appropriate end-to-end encapsulation). The VSRS switch in the other VLAN then performs switching functions by determining that IP.A is within the CIDR range of this VLAN and performs a lookup in its ARP cache based on IP.A to determine the destination MAC address associated with IP.A. If no match is found in the ARP cache, an ARP request is sent to a different L2 VNIC in the other VLAN to determine the destination MAC address. Otherwise, the VSRS switch sends the egress traffic to the relevant VNIC (e.g., via a tunnel, with appropriate encapsulation).

[0247] For inbound traffic from an instance in another VLAN to an instance in VLAN 1000, the traffic flow is similar to the above, only in the reverse direction. For outbound traffic from an instance in VLAN 1000 to the L3 network, the traffic flow is similar to the above, except that the VSRS switch of VLAN 1000 routes packets directly to the destination VNIC in the virtual L3 network via a virtual router (e.g., without routing packets through another VSRS switch). For inbound traffic from the virtual L3 network to an instance in VLAN 1000, the traffic flow is similar to the above, except that packets are received by the VSRS switch of VLAN 1000A, which then sends them as frames within the VLAN. For traffic between VLAN 1000 and other networks (either outbound or inbound), a VSRS switch is used similarly, where its routing function is used for outbound traffic to send packets via appropriate gateways (e.g., IGW, NGW, DRG, SGW, LPG), and its switching function is used for inbound traffic to send frames within VLAN 1000.

[0248] Now for reference Figure 11 The diagram illustrates an embodiment of VLAN 1100 (e.g., a cloud-based virtual L2 network), and more specifically, it shows a view of VLAN implementation methods.

[0249] As described above, a VLAN can include "n" compute instances 1102-A, 1102-B, and 1102-N, each running on a host machine. As discussed earlier, there can be a one-to-one association between compute instances and host machines, or a many-to-one association between multiple compute instances and a single host machine. Each compute instance 1102-A, 1102-B, and 1102-N can be an L2 compute instance, in which case it is associated with at least one virtual interface (e.g., an L2 VNIC) 1104-A, 1104-B, and 1104-N and switches 1106-A, 1106-B, and 1106-N. Switches 1106-A, 1106-B, and 1106-N are L2 virtual switches and together form an L2 distributed switch 1107.

[0250] The pairs of L2 VNICs 1104-A, 1104-B, 1104-N and switches 1106-A, 1106-B, 1106-N associated with compute instances 1102-A, 1102-B, 1102-N on the host machine are a pair of software modules connected to NVDs 1108-A, 1108-B, 1108-N on the host machine. Each L2 VNIC 1104-A, 1104-B, 1104-N represents an L2 port of a single switch (referred to herein as vswitch) perceived by the client. Generally, host machine “i” executes compute instance “i” and connects to NVD “i”. NVD “i” then executes L2 VNIC “i” and switch “i”. L2 VNIC “i” represents L2 port “i” of vswitch. “i” is a positive integer between 1 and “n”. Here, while a one-to-one association is described, other types of associations are also possible. For example, a single NVD can connect to multiple hosts, each executing one or more compute instances belonging to a VLAN. If this is the case, then the NVD hosts multiple pairs of L2 VNICs and switches, each pair corresponding to one of the compute instances.

[0251] VLANs can include instances of VSRS 1110. VSRS 1110 performs switching and routing functions and includes instances of VSRSVNIC 1112 and VSRS switch 1114. VSRS VNIC 1112 represents a port on the vswitch, where this port connects the vswitch to other networks via a virtual router. As shown in the diagram, VSRS 1110 can be instantiated on server cluster 1116.

[0252] Control plane 1118 can track and identify L2 VNICs 1104-A, 1104-B, and 1104-N and their placement within VLANs. Control plane 1110 can also provide this information to interfaces 1104-A, 1104-B, and 1104-N within the VLANs.

[0253] like Figure 11 As shown, the VLAN can be a cloud-based virtual L2 network that can be built on top of the physical network 1120. In some embodiments, this physical network 1120 may include NVD 1108-A, 1108-B, and 1108-N.

[0254] Generally, the first L2 compute instance of a VLAN (e.g., compute instance 1 1102-A) can communicate with the second compute instance of the VLAN (e.g., compute instance 2 1102-B) using L2 protocols. For example, frames can be sent between these two L2 compute instances via VLAN. However, frames can be encapsulated, tunneled, routed, and / or otherwise processed so that frames can be sent through the underlying physical network 1120.

[0255] For example, compute instance 1 1102-A sends a frame destined for compute instance 2 1102-B. Depending on the network connections (e.g., TCP / IP connection, Ethernet connection, tunnel connection, etc.) between host machine 1 and NVD 1, NVD 1 and physical network 1120, physical network 1120 NVD 2, and NVD 2 and host machine 2, different types of processing can be applied to the frame. For example, the frame is received and encapsulated by NVD 1, and so on, until the frame reaches compute instance 2. It is assumed that this processing allows frames to be sent between underlying physical resources, and for the sake of brevity, its description is omitted when describing VLANs and related L2 operations.

[0256] Virtual L2 network communication

[0257] Various forms of communication can occur within or with a virtual L2 network. These can include intra-VLAN communication. In such embodiments, a source compute instance can send packets to a destination compute instance that is in the same VLAN as the source compute instance (CI). Communication can also include sending packets to an endpoint outside the VLAN of the source CI. This can include, for example, communication between a source CI in a first VLAN and a destination CI in a second VLAN, communication between a source CI in a first VLAN and a destination CI in an L3 subnet, and / or communication from a source CI in a first VLAN to a destination CI outside the VCN of the VLAN containing the source CI. This communication can also include, for example, receiving communication from a source CI outside the VLAN of the destination CI at the destination CI. This source CI can be located in another VLAN, in an L3 subnet, or outside the VCN of the VLAN containing the source CI.

[0258] Each CI within a VLAN can play an active role in traffic flow. This includes learning interface identifiers to MAC addresses (also referred to as interface-to-MAC addresses in this document), mapping instances within the VLAN to maintain the L2 forwarding table within the VLAN, and sending and / or receiving communication packets. VSRS can play an active role in communication within the VLAN as well as in communication with source or destination CIs outside the VLAN. VSRS can maintain presence in both L2 and L3 networks to enable egress and ingress communication.

[0259] Communication within VLAN

[0260] Now for reference Figure 12 The diagram illustrates a flowchart of one embodiment of a process 1200 for intra-VLAN communication. In some embodiments, process 1200 may be executed by a computing instance within a shared VLAN. This process may be specifically executed when a source CI sends a packet to a destination CI within the VLAN but does not know the IP-to-MAC address mapping of that destination CI. For example, this occurs when a source CI sends a packet to a destination CI that has an IP address in the VLAN, but the source CI does not know the MAC address of that IP address. In this case, an ARP procedure may be performed to learn the destination MAC address and the IP-to-MAC address mapping.

[0261] If the source CI knows the IP-to-MAC address mapping, it can send packets directly to the destination CI without performing an ARP procedure. In some embodiments, this packet can be intercepted by the source VNIC, which is an L2 VNIC in intraVLAN communication. If the source VNIC knows the interface-to-MAC address mapping used for the destination MAC address, it can encapsulate the packet, for example, in L2 encapsulation, and forward the encapsulated packet to the destination VNIC, which is an L2 VNIC used for the destination MAC address in intraVLAN communication.

[0262] If the source VNIC does not know the interface-to-MAC address mapping used for MAC addresses, it can perform one aspect of the interface-to-MAC address learning process. This can include the source VNIC sending packets to all interfaces in the VLAN. In some embodiments, this packet can be broadcast to all interfaces within the VLAN. In some embodiments, this broadcast can be implemented at the physical network level as a serial unicast. This packet can include the destination MAC and IP addresses, as well as the source VNIC's interface, MAC address, and IP address. Each VNIC in the VLAN can receive this packet and learn the source VNIC's interface-to-MAC address mapping.

[0263] Each receiving VNIC can also decapsulate packets and forward the decapsulated packets to their associated CI. Each CI may include a network interface that can evaluate forwarded packets. If the network interface determines that the CI of the received forwarded packet does not match the destination MAC and / or IP address, then the packet is discarded. If the network interface determines that the CI of the received forwarded packet matches the destination MAC and / or IP address, then the packet is received by the CI. In some embodiments, a CI having a MAC and / or IP address that matches the destination MAC and / or IP address of the packet can send a response to the source CI, whereby the source VNIC can learn the interface-to-MAC address mapping of the destination CI, and whereby the source CI can learn the IP-to-MAC address mapping of the destination CI.

[0264] Procedure 1200 can be executed when the source CI does not know the IP to MAC address mapping, or when the IP to MAC address mapping of the destination CI of the source CI is outdated.

[0265] Therefore, when the IP-to-MAC address mapping is known, the source CI can send packets. When the IP-to-MAC address mapping is unknown, procedure 1200 can be executed. When the interface-to-MAC address mapping is unknown, the interface-to-MAC address learning procedure outlined above can be executed. When the interface-to-MAC address mapping is known, the VNIC can send packets to the destination CI.

[0266] Process 1200 begins at block 1202, where the source CI determines that the IP-to-MAC address mapping for the destination CI is unknown to the source CI. In some embodiments, this may include the source CI determining the destination IP address for the packet and determining that the destination IP address is not associated with a MAC address stored in the source CI's mapping table. Alternatively, the source CI may determine that the IP-to-MAC address mapping for the destination CI is stale. In some embodiments, the mapping may be stale if it has not been updated and / or validated within a certain time limit. After determining that the IP-to-MAC address mapping for the destination CI is unknown and / or stale to the source CI, the source CI initiates an ARP request for the destination IP and sends the ARP request for Ethernet broadcast.

[0267] At box 1204, the source VNIC (also referred to herein as the source interface) receives ARP requests from the source CI. The source interface identifies all interfaces on the VLAN and sends ARP requests to all interfaces on the VLAN broadcast domain. As mentioned earlier, since the control plane knows all interfaces on the VLAN and provides this information to the interfaces connected to the VLAN, the source interface also knows all interfaces in the VLAN and is able to send ARP requests to each interface in the VLAN. To do this, the source interface copies the ARP requests and encapsulates one of the ARP requests for each interface on the VLAN. Each encapsulated ARP request includes the source CI interface identifier and the source CI MAC and IP address, the destination IP address, and the destination CI interface identifier. The source CI interface replicates Ethernet broadcast by sending the copied and encapsulated ARP requests as serial unicast to each interface in the VLAN.

[0268] At box 1206, all interfaces in the VLAN broadcast domain receive and decapsulate packets. Each interface receiving packets in the VLAN broadcast domain learns the interface-to-MAC address mapping of the source VNIC of the source CI (e.g., the interface identifier to MAC address of the source interface of the source CI) as a packet to identify the source CI MAC and IP addresses and the source CI interface identifier. As part of learning the interface-to-MAC address mapping for the source CI, each interface may update its mapping table (e.g., its L2 forwarding table) and may provide the updated mapping to its associated switch and / or CI. Except for VSRS, each receiving interface may forward the decapsulated packets to its associated CI. The CI receiving the forwarded decapsulated packets, and specifically the network interface of that CI, may determine whether the destination IP address of the packet matches the IP address of the CI. If the IP address of the CI associated with that interface does not match the destination CI IP address specified in the received packet, then in some embodiments, the packet is dropped by the CI without further action. In the case of VSRS, the VSRS may determine whether the destination IP address of the packet matches the IP address of the VSRS. If the IP address of the VSRS does not match the destination IP address specified in the received packet, in some embodiments the packet is discarded by the VSRS without further action.

[0269] If the destination CI IP address specified in the received packet is determined to match the IP address of the CI (destination CI) associated with the receiving interface, then, and as indicated in box 1208, the destination CI sends a response, which may be a unicast ARP response to the source interface. This response includes the destination CI MAC address and destination CI IP address, as well as the source CI IP and MAC address. As will be discussed below, if the VSRS determines that the destination IP address matches the VSRS IP address, then the VSRS may send an ARP response.

[0270] This response is received by the destination interface encapsulating the unicast ARP response, as indicated in box 1210. In some embodiments, this encapsulation may include a Geneve encapsulation. The destination interface may forward the encapsulated packet to the source interface via the destination switch. The encapsulated packet includes the destination CI MAC and IP address and the destination CI interface identifier, as well as the source CI MAC and IP address and the source CI interface identifier.

[0271] At box 1212, the source interface receives and decapsulates the ARP response. The source interface may further learn the interface-to-MAC address mapping of the destination CI based on the encapsulation and / or information contained in the encapsulated packet. In some embodiments, the source interface may forward the ARP response to the source CI.

[0272] At box 1214, the source CI receives an ARP response. In some embodiments, the source CI may update its mapping table based on information contained in the ARP response, and specifically based on the MAC and IP addresses of the destination CI to reflect the IP-to-MAC address mapping. The source CI may then send a packet, which can be any packet including IP packets, and specifically an IPv4 or IPv6 packet destined for the destination CI. This packet may include the MAC and IP addresses of the source CI as the source MAC and source IP addresses of the packet, and the MAC and IP addresses of the destination CI as the destination MAC and destination IP addresses.

[0273] At box 1216, the source interface can receive packets from the source CI. The source interface can encapsulate the packets, and in some embodiments, Geneve encapsulation can be used to encapsulate the packets. The source interface can forward the encapsulated packets to the destination CI, and more specifically, to the destination interface. The encapsulated packets may include the IP and MAC addresses and interface identifiers of the source CI as the source MAC address, source IP address, and source interface identifier, and the MAC address, IP address, and interface identifier of the destination CI as the destination MAC address, IP address, and destination interface identifier.

[0274] At box 1218, the destination interface receives packets from the source interface. The destination interface can decapsulate the packets and then forward them to the destination CI. At box 1220, the destination CI receives packets from the destination interface.

[0275] Now for reference Figure 13 A schematic diagram 1300 illustrates a process 1300 for communication within a VLAN. As can be seen, the VLAN CIDR for VLAN A 1302 is 10.0.3.0 / 24. VLAN A 1302 includes a VSRS VNIC (VRVI) 1304, which can be instantiated on one or more hardware devices, and specifically on a server cluster 1306. VRVI 1304 can have an IP address of 10.0.3.1. A VLAN can include a compute instance 1 (CI1) 1310 with an IP address of 10.0.3.2 and communicatively coupled to an NVD 1 (SN1) 1312 that can instantiate an L2 VNIC 1 (VI1) 1314 and an L2 switch 1. VLANs can include compute instance 2 (CI2) 1320 with IP address 10.0.3.3 and communicatively coupled to NVD 2 (SN2) 1322, which can instantiate L2 VNIC 2 (VI2) 1324 and L2 switch 2. VLANs can also include compute instance 3 (CI3) 1330 with IP address 10.0.3.4 and communicatively coupled to NVD 3 (SN3) 1332, which can instantiate L2 VNIC 3 (VI3) 1334 and L2 switch 3.

[0276] exist Figure 13 In the example, and applied Figure 12 In method 1200, CI3 1330 is the source CI and VI3 1334 is the source interface. Additionally, CI2 1320 is the destination CI and VI2 1324 is the destination interface. CI3 determines that it does not have an IP-to-MAC mapping for the destination IP address (10.0.3.3), causing CI3 1330 to send an ARP request. The ARP request can be directed to a known address, and specifically to the known IP address of CI2 1320. Therefore, in some embodiments, the ARP request can be directed to 10.0.3.3.

[0277] This ARP request is received by SN3 1332 and VI3 1334. VI3 1334 copies the ARP request to create an ARP request for each CI in VLAN 1302. VI3 1334 encapsulates each ARP request and sends it to each interface in the VLAN. These encapsulated ARP requests may include information identifying the MAC address of the source CI (CI3), the interface identifier of the source interface (VI3), and the destination MAC address. These requests can be sent to each interface in the VLAN, as indicated by arrow 1350. Within the VLAN, these requests can be broadcast ARP requests.

[0278] Interfaces within a VLAN receive encapsulated ARP requests and decapsulate them. Based on the information contained in and / or associated with the ARP request, the interfaces in the VLAN update their mappings. Specifically, for example, each of VI1 1314, VI2 1324, and VRVI 1304 receives an ARP request from CI 3 1330, decapsulates the ARP request, and learns the interface-to-MAC address mapping of the source CI, with both the interface identifier and MAC address included in the encapsulated packet. VRVI 1304 can further update the IP-to-MAC address mapping used for the source CI based on the information contained in the encapsulated packet.

[0279] An interface in a VLAN with a CI containing the requested IP address can send an ARP response to the CI 31330 as indicated by arrow 1352. Specifically, as... Figure 13 As shown, VI2 1324 is the interface of CI2 1320 and therefore can send ARP responses. An ARP response from CI2 can be received by VI2, encapsulated, and sent as an ARP unicast to the requesting interface, specifically VI3. As previously indicated in this application, sending this ARP response may include providing the encapsulated ARP response to the associated switch, which can then send the encapsulated ARP response to VI3.

[0280] ARP responses can be received by VI3 and can be decapsulated. VI3 can learn CI2's interface-to-MAC address mapping based on the received ARP responses and can provide the updated learned mapping to VI3's associated switches. Decapsulated packets can be provided to CI3, which can learn CI2 1320's IP-to-MAC address mapping based on the decapsulated packets. CI3 can send packets to CI2, which can be IP packets, such as IPv4 or IPv6 packets. This packet can use CI3's MAC and IP addresses as the source address and CI2's IP and MAC addresses as the destination address.

[0281] Packets sent by CI3 can be received by VI3. VI3 can encapsulate packets and forward them to interface VI2. VI2 can receive packets, decapsulate packets, and forward packets to CI2.

[0282] Inter-VLAN communication

[0283] Now for reference Figure 14 This diagram illustrates a flowchart of one embodiment of a process 1400 for inter-VLAN communication in a virtual L2 network. Process 1400 can be performed by all or part of two connected VLANs (such as...). Figure 8 The process 1400 is executed in the context of multiple connected L2 VLANs (800). In some embodiments, process 1400 can be executed when a compute instance (source CI) in the first VLAN (source VLAN) sends a packet to a destination compute instance (destination CI) in the second VLAN (destination VLAN). In some embodiments, the source CI can determine that the destination CI is outside the source VLAN based on the IP address of the destination CI. For example, the source CI can determine that the destination IP is outside the source VLAN CIDR. In this case, the source CI can determine to send the IP packet to the destination CI via the VSRS of the source VLAN. If the source CI already knows the mapping of the VSRS in the first VLAN (source VSRS), then the source CI can directly point the packet to the source VSRS. If the source CI does not know the mapping of the source VSRS, then the source CI and its associated VNIC (source interface or source VNIC) first learn the mapping of the source VSRS. In embodiments of inter-VLAN communication, both the source and destination VNICs are L2 VNICs. The first step of process 1400 (steps 1402 to 1410) involves learning the source VSRS mapping by the source VNIC and the source CI.

[0284] Procedure 1400 begins at box 1402, where the source CI with the destination IP address initiates an ARP request. The ARP request is used to determine the IP->MAC address mapping of the source VSRS. The ARP request is sent by the source CI via Ethernet broadcast. The ARP request includes the source CI's IP address as the source IP and the MAC address.

[0285] At box 1404, the source VNIC receives and copies an ARP request. Specifically, the source VNIC receives an ARP request from the source CI, identifies all interfaces on the VLAN, and sends the ARP request to all interfaces on the VLAN broadcast domain. As mentioned earlier, since the control plane knows all interfaces on the VLAN and provides this information to the interfaces connected to the VLAN, the source interface also knows all interfaces in the VLAN and can send ARP requests to each interface in the VLAN. To do this, the source interface copies the ARP request and encapsulates one of the ARP requests for each interface on the VLAN. Each encapsulated ARP request includes the source CI interface identifier and the source CI IP and MAC address as the source address, and the destination CI interface identifier and IP address as the destination address. The source CI interface performs Ethernet broadcast by sending the copied and encapsulated ARP request to each interface in the VLAN via serial unicast. In some embodiments, the source VNIC may use Geneve encapsulation to encapsulate the ARP request.

[0286] At box 1406, all interfaces in the VLAN broadcast domain receive and decapsulate packets. Each interface receiving packets in the VLAN broadcast domain learns the interface-to-MAC address mapping of the source VNIC for the source CI, as the packet identifies the source CI MAC and IP addresses as well as the source interface identifier. As part of learning the interface-to-MAC address mapping for the source CI, each interface can update its mapping table and provide the updated mapping table to its associated switches and / or CIs. Except for VSRS, each receiving interface forwards the decapsulated packets to its associated CI. The CI receiver (specifically, the network interface of that CI) that forwards the decapsulated packets can determine whether the destination IP address of the packet matches the IP address of the CI. If the IP address of the CI associated with that interface does not match the destination CI IP address specified in the received packet, no further action is taken.

[0287] The source VSRS determines that the destination IP address matches the source VSRS's IP address, and as indicated in step 1408, the source VSRS encapsulates and sends a response, which may be a unicast ARP response to the source interface. This response includes the source CI MAC address, IP address, and source CI interface identifier as the destination address. The response also includes the source VSRS MAC address, IP address, and VSRS interface identifier as the source address. In some embodiments, the encapsulation of the ARP response may include Geneve encapsulation.

[0288] At box 1410, the source interface receives and decapsulates the ARP response. The source interface can also learn the interface-to-MAC address mapping for the source VSRS based on information contained in the encapsulation and / or the encapsulated packet. In some embodiments, the source interface may forward the ARP response to the source CI.

[0289] At block 1412, the source CI receives an ARP response. In some embodiments, the source CI may update its mapping table based on information contained in the ARP response, and specifically based on the MAC address and IP address of the source VSRS. In some embodiments, for example, the source CI may update its mapping table to reflect the IP-to-MAC address mapping of the source VSRS. The source CI may then send a packet to the source VSRS, which can be any packet, including IP packets, specifically IPv4 or IPv6 packets. In some embodiments, this may include sending an IP packet with the IP address of the destination CI as the destination address. In some embodiments, the IP address of the destination CI may be included in the packet header, such as, for example, in the packet's L3 header. The header may also include the MAC address of the source VSRS in another header of the packet, such as, for example, in the packet's L2 header. The packet may also include the MAC address and IP address of the source CI as the source MAC address and source IP address.

[0290] At box 1414, the source interface can receive packets from the source CI. The source interface can encapsulate packets. The source interface can forward the encapsulated packets to the source VSRS, and more specifically, to the source VSRS VNIC. In addition to the packet address, the encapsulated packet may include the MAC address and interface identifier of the source CI as the source MAC address and source interface identifier, and the MAC address and interface identifier of the source VSRS as the destination MAC address and destination interface identifier.

[0291] At box 1416, the source VSRS receives the encapsulated packet. The source VSRS decapsulates the packet and strips away any address information associated with the source VSRS, including, for example, the source VSRS IP address, MAC address, and / or source VSRS interface identifier. The source VSRS identifies the destination CI of the packet. In some embodiments, the source VSRS identifies the destination CI based on the IP address of the destination CI included in the packet. The source VSRS looks up a mapping for the destination IP address of the packet. If the IP address is within the IP address space of the VCN, then the source VSRS looks up a mapping for the destination IP address of the packet in the IP address space used by the VCN. The source VSRS then recapsulates the packet using L3 encapsulation. In some embodiments, this L3 encapsulation may include, for example, MPLSoUDPL3 encapsulation. The source VSRS then forwards the packet to the VSRS of the destination VLAN (the destination VSRS). In some embodiments, the source VSRS may forward the packet to the destination VSRS, such that the destination VSRS is the tunnel endpoint (TEP) of the packet. The L3-encapsulated packet includes the source CI IP address and MAC address, as well as the IP address of the destination CI.

[0292] At block 1418, the destination VSRS receives the packet and decapsulates it. In some embodiments, this decapsulation may include removing L3 encapsulation. If the destination VSRS knows the IP-to-MAC and MAC-to-interface mappings used for the destination CI, then the destination VSRS identifies the interface and MAC address of the destination CI within the destination VLAN. Alternatively, if the destination VSRS does not know the destination CI mappings, then steps 1612 to 1622 of procedure 1600 may be performed.

[0293] At block 1420, the destination VSRS forwards the packet to the destination interface of the destination CI. In some embodiments, this may include encapsulating the packet with L2 encapsulation. In some embodiments, the packet may include the IP address and MAC address of the source CI and / or the MAC address and interface identifier of the destination VSRS. In some embodiments, the packet may also include the MAC address and interface identifier of the destination CI.

[0294] At block 1422, the destination interface receives and decapsulates the packet. Specifically, the destination VNIC removes the L2 encapsulation. In some embodiments, the destination VNIC forwards the packet to the destination CI. At block 1424, the destination CI receives the packet.

[0295] Now for reference Figure 15A schematic diagram 1500 illustrates the process 1400 for inter-VLAN communication. As can be seen, the VLAN CIDR for VLAN A 1502-A is 10.0.3.0 / 24. VLAN A 1502-A includes VSRS VNIC A (VRVI A) 1504-A, which can be instantiated on one or more hardware devices, and specifically on server cluster 1506. VRVI A 1504-A can have an IP address of 10.0.3.1. A VLAN can include a compute instance 1 (CI1) 1510 with an IP address of 10.0.3.2 and communicatively coupled to NVD 1 (SN1) 1512, which can instantiate L2 VNIC 1 (VI1) 1514 and L2 switch 1. VLANs can include compute instance 2 (CI2) 1520 with IP address 10.0.3.3 and communicatively coupled to NVD 2 (SN2) 1522, which can instantiate L2VNIC 2 (VI2) 1524 and L2 switch 2. VLANs can also include compute instance 3 (CI3) 1530 with IP address 10.0.3.4 and communicatively coupled to NVD 3 (SN3) 1532, which can instantiate L2VNIC 3 (VI3) 1514 and L2 switch 3.

[0296] VLAN B 1502-B has a VLAN CIDR of 10.0.34.0 / 24. VLAN B 1502-B includes a VSRS VNIC B (VRVI B) 1504-B, which can be instantiated on one or more hardware devices, and specifically on server cluster 1506. The IP address of VRVI B 1504-B can be 10.0.4.1. VLANs can include compute instance 4 (CI4) 1540 with IP address 10.0.4.2 and communicatively coupled to NVD 4 (SN4) 1542, which can instantiate L2 VNIC 4 (VI3) 1544 and L2 switch 4.

[0297] exist Figure 15 In the example, and applied Figure 14 In method 1400, CI3 1530 is the source CI and VI3 1534 is the source interface. Additionally, CI4 1540 is the destination CI and VI4 1544 is the destination interface. CI3 determines that it does not have an IP-to-MAC mapping for VRVI A1504-A, causing CI3 1530 to send an ARP request. The ARP request can be targeted at an address, and specifically, at a known IP address of VRVI A 1504-A. Therefore, in some embodiments, the ARP request can be targeted at 10.0.3.1.

[0298] This ARP request is received by SN3 1532 and VI3 1534. VI3 1534 copies the ARP request to create an ARP request for each CI in VLAN 1502-A. VI3 1534 encapsulates each ARP request with L2 encapsulation and sends the ARP request to each interface in the VLAN. These encapsulated ARP requests may include information identifying the MAC and IP addresses of the source CI (CI3), the interface identifier of the source interface VI3, and the destination IP address. These requests can be sent to each interface in the VLAN as shown by arrow 1550, and specifically, they can be sent as a serial unicast, causing each interface in the VLAN to receive the ARP request.

[0299] Interfaces within a VLAN receive encapsulated ARP requests and decapsulate them. Based on the information contained in and / or associated with the ARP request, the interfaces in the VLAN update their mappings. Specifically, for example, each of VI1 1514, VI2 1524, and VRVI A 1504-A receives a unicast ARP request from CI 3 1530, decapsulates the unicast ARP request, and learns the interface-to-MAC address mapping of the source CI, with both the interface identifier and MAC address included in the encapsulated packet.

[0300] The VRVI A1504-A determines that its IP address matches the requested IP address and sends an ARP response to CI 3 1530 as indicated by arrow 1552. The ARP response from the VRVI A1504-A can be encapsulated by the VRVI A1504-A using, for example, L2 encapsulation, and can be sent as an ARP unicast to the interface that issued the request, specifically VI3. As previously indicated in this application, the sending of this ARP response may include providing the encapsulated ARP response to the associated switch of the VRVI A1504-A, which can then send the encapsulated ARP response to VI3.

[0301] ARP responses can be received by VI3 and can be decapsulated. VI3 can learn the interface-to-MAC address mapping of VRVI A 1504-A based on the received ARP responses and can provide the updated learned mapping to the associated switches of VI3. The decapsulated packets can be provided to CI3. CI3 can learn the IP address-to-MAC address mapping of VRVI A 1504-A and can send packets, which can be IP packets, such as IPv4 or IPv6 packets. This packet can have CI3's MAC address and IP address as the source address. This packet can also include the IP address of the destination CI (CI4 1540) as the destination address and can have VRVI A 1504-A MAC address and IP address.

[0302] Packets sent by CI3 can be received by VI3, which can encapsulate the packet and forward it to VRVI A1504-A. VRVI A1504-A can receive the packet, decapsulate the packet, and look up the mapping of the packet's destination IP address (CI4's IP address). In some embodiments, packet decapsulation may include stripping the L2 header from the packet, or in other words, stripping information related to VLAN A1502-A from the header. This stripped information may include, for example, the MAC address of CI3, the interface identifier of the interface of VI3, and / or the MAC address and interface identifier of VRVI A1504-A. In some embodiments, VSRS, and therefore VRVI A1504-A, may reside in both L2 and L3 networks. In the L2 network, and therefore within the VLAN, VSRS can use L2 communication protocols, while VSRS can use L3 communication protocols when communicating with the L3 network. Unlike learning performed in a VLAN, VSRS can learn the mapping of endpoints in the L3 network from the control plane. In some embodiments, for example, the control plane can provide information that maps the IP addresses and / or MAC addresses of instances in the L3 network.

[0303] VRVI A 1504-A can look up the mapping of IP destination addresses contained in packets, or in other words, it can look up the mapping of IP addresses for destination CIs. In some embodiments, looking up this mapping may include a VSRS that identifies VRVI B1504-B as VLAN B1502-B. VRVI A 1504-A can encapsulate packets and forward the encapsulated packets to VRVI B 1504-B, which may be a Tunneling Endpoint (TEP) for the destination VLAN and / or the destination interface. This forwarding of the encapsulated packets is indicated by box 1556. The forwarded packets may include the IP address CI3 of the source CI as the source address and the IP address of the destination CI (CI4) as the destination address.

[0304] VRVI B 1504-B can receive and decapsulate packets. VRVI B 1504-B can also identify interfaces within VLAN B 1502-B corresponding to destination IP addresses. VRVI B 1504-B can encapsulate packets and add L2 headers to establish tunnels within VLAN B1504-B. VRVI B 1504-B can then forward packets to the destination CI. Destination interface VI4 1544 can receive and decapsulate packets, and then forward Ethernet frames or packets to destination CI 1540.

[0305] Ingress Packet Flow

[0306] Now for reference Figure 16 The diagram illustrates a flowchart of one embodiment of the process 1600 for ingress grouping flow. Specifically, Figure 16 An embodiment of a process 1600 for ingress packets from a subnet is shown. This process can be performed by all or part of the system 600, and specifically by the VLAN entity and some external source CI (for the VLAN), which may reside on the L3 subnet.

[0307] Process 1600 begins at block 1602, where the source L3 CI determines to send a packet to the destination CI, and more specifically, to the destination IP address within the VLAN. The source L3 CI is unaware of this mapping and therefore sends an ARP request for the MAC address of the virtual router in the subnet containing the source L3 CI. In some embodiments, sending this ARP request may include sending the ARP request via Ethernet broadcast. At step 1604, the source interface of the source L3 CI replies to the ARP request with the MAC address of the VR. In some embodiments, the source interface may determine the MAC address of the VR based on mapping information accessed by the source interface.

[0308] After receiving an ARP reply from the source interface, the source L3 CI sends an IP packet to the VR, as indicated in box 1606. In some embodiments, this may include the source L3 CI sending the packet, and the source L3 interface receiving, encapsulating, and forwarding the packet. In some embodiments, the source L3 interface may encapsulate the packet using L3 encapsulation, which includes the original packet starting with an L3 header. The source L3 interface may forward the packet to the VR. This IP packet may be sent to the VR MAC address and the VR interface, and may include the source L3 CI MAC address as the MAC address and the source L3 CI interface identifier as the source interface identifier.

[0309] The VR receives and decapsulates packets. The VR can then look up the VNIC mapping for the packet's destination IP address. The VR can determine if the packet is in a VLAN CIDR, and then encapsulate the packet and forward the encapsulated packet to the VSRS of the VLAN containing the destination CI. In some embodiments, the VR can use L3 encapsulation to encapsulate the packet. The encapsulated packet may include the destination IP address, the MAC address and interface identifier of the VSRS, and the source IP address.

[0310] The VSRS receives and decapsulates packets, as indicated in box 1610. When the VSRS knows the mapping of the destination CI, and specifically the mapping of the destination IP address in the received packet, the VSRS forwards the IP packet to the TEP of the CI corresponding to the destination IP address. This may include generating and encapsulating L2 packets with a destination MAC address corresponding to the MAC address of the destination CI and a destination interface identifier corresponding to the destination interface. In an embodiment of the ingress packet flow, the destination interface is an L2 VNIC.

[0311] If the VSRS does not know the destination CI mapping, then process 1600 continues at box 1612, where the VSRS receives and decapsulates the IP packets.

[0312] When the VSRS does not know the destination MAC address to VNIC mapping, it can perform an interface-to-MAC address learning process. This can include the VSRS sending packets to all interfaces within the VLAN. In some embodiments, this packet can be broadcast to all interfaces within the VLAN. This packet can include the destination MAC and IP addresses, the VSRS's interface identifier and MAC address, and the source CI's IP address. Each VNIC in the VLAN can receive this packet and learn the VSRS's interface-to-MAC address mapping.

[0313] Each receiving VNIC can also decapsulate packets and forward the decapsulated packets to their associated CI. Each CI may include a network interface that can evaluate forwarded packets. If the network interface determines that the CI of the received forwarded packet does not match the destination MAC and / or IP address, then the packet is discarded. If the network interface determines that the CI of the received forwarded packet matches the destination MAC and / or IP address, then the packet is received by the CI. In some embodiments, a CI with a MAC and / or IP address that matches the destination MAC and / or IP address of the packet can send a response to the VSRS, whereby the VSRS can learn the interface-to-MAC address mapping and IP-to-MAC address mapping of the destination CI.

[0314] Alternatively, when the VSRS is unaware of the destination CI mapping, and specifically, the mapping from the destination IP address to the MAC address, the VSRS suspends the IP packet. The VSRS then generates an ARP request for the destination IP address on all interfaces within its VLAN broadcast domain. This ARP request includes the VSRS MAC address as the source MAC address, the VSRS interface's interface identifier as the source interface, and the destination IP address as the destination IP address. In some embodiments, this ARP request may be broadcast to all interfaces in the VLAN.

[0315] At box 1614, all interfaces in the VLAN broadcast domain receive and decapsulate packets. Each interface in the VLAN broadcast domain that receives the packets learns the VSRS interface-to-MAC address mapping, as the packets identify the VSRS MAC address and interface identifier. As part of learning the interface-to-MAC address mapping for VSRS, each interface can update its mapping table and provide the updated mapping table to its associated switch and / or CI.

[0316] Each receiving interface can forward the decapsulated packets to its associated CI. The CI receiving the forwarded decapsulated packets, and specifically the network interface of that CI, can determine whether the destination IP address of the packet matches the IP address of the CI. If the IP address of the CI associated with that interface does not match the destination CI IP address specified in the received packet, then no further action is taken.

[0317] If it is determined that the destination CI IP address specified in the received packet matches the IP address of the CI (destination CI) associated with the receiving interface, then, as indicated in box 1616, the destination CI sends a response, which may be a unicast ARP response to the source interface. This response includes the destination CI MAC and IP address, and the VSRS MAC and IP address. This response is received by the destination interface that encapsulated the unicast ARP response, as indicated in box 1618. The destination interface may forward the encapsulated packet to the VSRS via the destination switch. The encapsulated packet includes the destination CI MAC and IP address and the destination CI interface identifier, and the VSRS MAC and IP address and the VSRS interface identifier.

[0318] At box 1620, VSRS, and specifically the VSRS interface, receives and decapsulates ARP responses. VSRS, and specifically the VSRS interface, can further learn the interface-to-MAC address mapping for destination CI based on information contained in the encapsulation and / or the encapsulated packets.

[0319] At box 1622, the VSRS can then encapsulate an L2 header and add it to the previously suspended IP packet, which can then be forwarded to the destination CI, specifically to the destination interface. The destination interface can decapsulate the packet and provide the decapsulated packet to the destination CI. This packet forwarded by the VSRS can use the VSRS's MAC address and interface identifier as the source MAC address and source interface identifier, and the destination CI's MAC address and interface identifier as the destination MAC address and destination interface identifier. This packet may also include the IP address of the destination CI and the IP address of the source CI.

[0320] The destination interface receives packets from the VSRS and then decapsulates them. This decapsulation may include removing headers added by the VSRS, which may be VCN headers. The destination interface can then forward the packets to the destination CI, and the destination CI can receive packets from the destination interface.

[0321] Now for reference Figure 17A schematic diagram 1700 illustrates the process 1600 for ingress communication. As can be seen, the VLAN CIDR for VLAN A 1502-A is 10.0.3.0 / 24. VLAN A 1502-A includes VSRS VNIC A (VRVI A) 1504-A, which can be instantiated on one or more hardware devices, and specifically on server cluster 1506. VRVI A 1504-A can have an IP address of 10.0.3.1. The VLAN can include a compute instance 1 (CI1) 1510 with an IP address of 10.0.3.2 and communicatively coupled to NVD 1 (SN1) 1512, which can instantiate L2 VNIC 1 (VI1) 1514 and L2 switch 1. VLANs can include compute instance 2 (CI2) 1520 with IP address 10.0.3.3 and communicatively coupled to NVD 2 (SN2) 1522, which can instantiate L2 VNIC 2 (VI2) 1524 and L2 switch 2. VLANs can also include compute instance 3 (CI3) 1530 with IP address 10.0.3.4 and communicatively coupled to NVD 3 (SN3) 1532, which can instantiate L2 VNIC 3 (VI3) 1514 and L2 switch 3.

[0322] The compute instance of L3 compute instance 4 (CI4) 1744 can reside on subnet 1739 outside VLAN A 1702-A. CI4 can have IP address 10.0.4.4 and can be communicatively coupled to NVD4 (SN4) 1742, which can instantiate L3 VNIC 4 (VI4) 1744.

[0323] exist Figure 17 In the example, and applied Figure 16 In method 1600, CI4 1740 is the source CI and VI4 1744 is the source interface. Additionally, CI3 1730 is the destination CI and VI3 1734 is the destination interface. CI4 determines that it does not know the mapping for sending packets to CI3. Therefore, CI4 sends an ARP request for the IP address of the subnet virtual router. In response to this request, in some embodiments, VI4 residing on the NVD containing the subnet virtual router instance may directly reply to CI4 with the VR IP address. CI4 learns the VR IP address and sends the packet to the VR. The VR receives the packet, then encapsulates the packet based on the mapping information and forwards the packet to the VSRS, as indicated by arrow 1748.

[0324] VSRS receives packets, and if VSRS does not know the mapping, specifically the IP-to-MAC address mapping to the destination CI, then VSRS suspends the packets and sends an ARP request to all interfaces in VLAN A 1704-A. This ARP request includes the VSRS MAC address and interface identifier, and each receiving interface in VLAN A 1704-A learns the VSRS mapping based on the ARP request. Each interface also decapsulates the packets and sends them to its CI. Upon receiving the decapsulated packets, CI3 determines that it is the CI identified in the packets, and CI3 generates an ARP reply with its MAC address. CI3 receives and encapsulates the ARP reply and forwards the encapsulated ARP reply to VSRS. The encapsulated ARP reply includes the VSRS MAC address and interface identifier, as well as the MAC address and interface identifier of CI3's interface.

[0325] VSRS receives ARP replies and learns the IP address-to-MAC address and MAC address-to-interface mappings for CI3. VSRS then forwards previously pending packets to CI3, which are received, decapsulated, and forwarded to CI3 by VSRS.

[0326] Export group flow

[0327] Now for reference Figure 18 The diagram illustrates a flowchart of one embodiment of a process 1800 for an outgoing packet flow from a VLAN. In some embodiments, packets may flow out of a VLAN to another VLAN, subnet, or network. Process 1800 may be performed in whole or in part by system 600, and specifically by an entity of the VLAN. In some embodiments, part of the process may be performed by some external destination CI (for the source VLAN), which may reside on an L3 subnet.

[0328] In some embodiments, procedure 1800 may be performed when a compute instance (source CI) in a VLAN (source VLAN) sends a packet to a destination compute instance (destination CI) outside the VLAN. If the source CI already knows the mapping of the VSRS (source VSRS) in the first VLAN, then the source CI can directly direct the packet to the source VSRS. If the source CI does not know the mapping of the source VSRS, then the source CI and its associated VNIC (source interface or source VNIC) first learn the mapping of the source VSRS, specifically the IP-to-MAC address mapping. In embodiments of outgoing packet flows from L2 VLANs, the source VNIC is an L2 VNIC. The first step of procedure 1800 (steps 1802 to 1810) involves the learning of the source VSRS mapping by the source VNIC and the source CI.

[0329] Procedure 1800 begins at box 1802, where the source CI initiates an ARP request. The ARP request is used to determine the IP-to-MAC address mapping of the source VSRS. The ARP request is sent by the source CI via Ethernet broadcast. The ARP request includes the source CI's MAC and IP addresses, as well as the source CI's interface identifier as the source address and source interface identifier. The ARP request also includes the source VSRS's IP address.

[0330] At box 1804, the source VNIC receives and replicates the ARP request. Specifically, the source VNIC receives the ARP request from the source CI, identifies all interfaces on the VLAN, and sends the ARP request to all interfaces on the VLAN broadcast domain. As mentioned earlier, since the control plane knows all interfaces on the VLAN and provides this information to the interfaces connected to the VLAN, the source interface also knows all interfaces in the VLAN and can send the ARP request to each interface in the VLAN. To do this, the source interface replicates the ARP request and encapsulates one of the ARP requests for each interface on the VLAN. Each encapsulated ARP request includes the source CI interface identifier and source CI MAC and / or IP address as the source address, and the destination CI interface identifier as the destination address. The source CI interface replicates the Ethernet broadcast by sending the replicated and encapsulated ARP request via serial unicast. In some embodiments, the source VNIC may use Geneve encapsulation to encapsulate the ARP request.

[0331] At box 1806, all interfaces in the VLAN broadcast domain receive and decapsulate packets. Each interface in the VLAN broadcast domain that receives packets learns the interface-to-MAC address mapping of the source CI, as the packet identifies the source CI MAC address and interface identifier. In addition, the VSRS learns the IP-to-MAC address mapping of the source CI. As part of learning the interface-to-MAC address mapping for the source CI, each interface can update its mapping table and provide the updated mapping table to its associated switches and / or CIs. Except for the VSRS, each receiving interface can forward the decapsulated packets to its associated CI. The CI receiving the forwarded decapsulated packets, and specifically the network interface of that CI, can determine whether the destination IP address of the packet matches the IP address of the CI. If the IP address of the CI associated with that interface does not match the destination CI IP address specified in the received packet, no further action is taken.

[0332] The source VSRS determines that the destination IP address matches the source VSRS IP address, and as indicated in step 1808, the source VSRS encapsulates and sends a response, which may be a unicast ARP response to the source interface. This response includes the source CI MAC address and IP address as the destination address, and the source CI interface identifier as the destination interface identifier. The response also includes the source VSRS MAC address and IP address as the source address, and the source VSRS interface identifier as the source interface identifier.

[0333] At box 1810, the source interface receives and decapsulates the ARP response. The source interface can also learn the source VSRS interface-to-MAC address mapping based on information contained in the encapsulation and / or the encapsulated packet. In some embodiments, the source interface may forward the ARP response to the source CI.

[0334] At block 1812, the source CI receives an ARP response. In some embodiments, the source CI may update its mapping table based on information contained in the ARP response, specifically based on the MAC address and IP address of the source VSRS. The source CI may then send a packet to the source VSRS, which can be any packet, including IP packets, and specifically IPv4 or IPv6 packets. In some embodiments, this may include sending an IP packet with a destination address having the source VSRS MAC address and the source VSRS IP address. The packet may also include the source CI's MAC address and IP address as the source address.

[0335] At box 1814, the source interface receives packets from the source CI. The source interface encapsulates the packets. The source interface can forward the encapsulated packets to the source VSRS, and more specifically, to the source VSRS VNIC. The encapsulated packets may include the MAC address and interface identifier of the source CI as the source MAC address and source interface identifier, and the MAC address and interface identifier of the source VSRS as the destination MAC address and destination interface identifier. The encapsulated packets may also include the IP address of the source CI and the IP address of the source VSRS.

[0336] At box 1816, the source VSRS receives the encapsulated packet. The source VSRS identifies the destination CI of the packet. In some embodiments, the source VSRS identifies the destination CI of the packet based on the IP address of the destination CI included in the packet. The source VSRS looks up a mapping for the destination IP address of the packet. If the IP address is within the IP address space of the VCN, then the source VSRS looks up a mapping for the destination IP address of the packet in the IP address space of the VCN. The source VSRS then re-encapsulates the packet using L3 encapsulation.

[0337] The source VSRS then forwards the packet to the destination CI. In some embodiments, this may include forwarding the encapsulated packet to a VR associated with the subnet containing the destination CI, and / or forwarding the packet to a gateway to allow the packet to exit the VCN. In some embodiments, forwarding the packet to the destination CI may include forwarding the packet to a TEP associated with the destination CI. The L3-encapsulated packet includes the IP address of the source CI as the source address and the IP address of the destination CI as the destination address.

[0338] At block 1820, the destination interface receives and decapsulates the packet. In some embodiments, the destination VNIC forwards the packet to the destination CI. At block 1822, the destination CI receives the packet.

[0339] Now for reference Figure 19 A schematic diagram 1900 illustrates the process 1800 for outgoing packet flows. As shown, the VLAN CIDR for VLAN A 1502-A is 10.0.3.0 / 24. VLAN A 1502-A includes VSRS VNIC A (VRVI A) 1504-A, which can be instantiated on one or more hardware devices, specifically on server cluster 1506. VRVI A 1504-A can have an IP address of 10.0.3.1. A VLAN can include a compute instance 1 (CI1) 1510 with an IP address of 10.0.3.2 and communicatively coupled to NVD 1 (SN1) 1512, which can instantiate L2 VNIC 1 (VI1) 1514 and L2 switch 1. VLANs can include compute instance 2 (CI2) 1520 with IP address 10.0.3.3 and communicatively coupled to NVD 2 (SN2) 1522, which can instantiate L2 VNIC 2 (VI2) 1524 and L2 switch 2. VLANs can also include compute instance 3 (CI3) 1530 with IP address 10.0.3.4 and communicatively coupled to NVD 3 (SN3) 1532, which can instantiate L2 VNIC 3 (VI3) 1514 and L2 switch 3.

[0340] A compute instance of L3 compute instance 4 (CI4) 1944 can reside on subnet 1939 outside VLAN A 1902-A. CI4 can have an IP address of 10.0.4.4 and can be communicatively coupled to NVD4 (SN4) 1942, which can instantiate L3 VNIC 4 (VI4) 1944. Subnet 1939 can include virtual router (VR) 1948. VR 1948 can have an IP address of 10.0.4.1. VR 1948 can be instantiated on, for example, a SmartNIC, a server, a server cluster, etc.

[0341] exist Figure 19 In the example, and applied Figure 18 In method 1800, CI3 1930 is the source CI and VI3 1934 is the source interface. Additionally, CI4 1940 is the destination CI and VI4 1944 is the destination interface. CI3 determines that it does not have an IP-to-MAC mapping for VRVI A1904-A, causing CI3 1930 to send an ARP request. The ARP request can be directed to a known address, and specifically, to a known IP address of VRVI A 1904-A. Therefore, in some embodiments, the ARP request can be directed to 10.0.3.1.

[0342] This ARP request is received by SN3 1932 and VI3 1934. VI3 1934 copies the ARP request to create an ARP request for each CI in VLAN 1902-A. VI3 1934 encapsulates each ARP request with L2 encapsulation and sends the ARP request to each interface in the VLAN. These encapsulated ARP requests may include information identifying the MAC address of the source CI (CI3), the source interface identifier VI3, and the destination IP address. These requests can be sent to each interface in the VLAN as shown by arrow 1950, and specifically, they can be broadcast so that each interface in the VLAN receives the ARP request.

[0343] Interfaces within a VLAN receive encapsulated ARP requests and decapsulate them. Based on the information contained in and / or associated with the ARP request, the interfaces in the VLAN update their mappings. Specifically, for example, each of VI1 1914, VI2 1924, and VRVI A 1904-A receives a unicast ARP request from CI 3 1930, decapsulates the unicast ARP request, and learns the interface-to-MAC address mapping of the source CI, with both the interface identifier and MAC address included in the encapsulated packet.

[0344] VRVI A 1904-A determines that its IP address matches the requested IP address and sends an ARP response to CI 3 1930 as indicated by arrow 1952. The ARP response from VRVI A 1904-A can be encapsulated by VRVI A 1904-A and can be sent as an ARP unicast to the interface that issued the request, specifically VI3. As previously indicated in this application, the sending of this ARP response may include providing the encapsulated ARP response to the associated switch of VRVI A 1904-A, which can then send the encapsulated ARP response to VI3.

[0345] ARP responses can be received by VI3 and can be decapsulated. VI3 can learn the interface-to-MAC mapping of VRVI A 1904-A based on the received ARP responses and can provide the updated learned mapping to the associated switch of VI3. Decapsulated packets can be provided to CI3. CI3 can learn the IP-MAC address mapping based on the received packets. CI3 can send packets, which can be IP packets, such as IPv4 or IPv6 packets. This packet can have CI3's MAC and IP addresses as the source address and the destination CI (CI4 1940)'s IP address as the destination address. In some embodiments, the destination address may also include VRVI A's 1904-A MAC address.

[0346] A pause packet sent by CI3 can be received by VI3, which can encapsulate the packet and forward it to VRVI A 1904-A. VRVI A 1904-A can receive the packet, decapsulate it, and look up the mapping for the destination IP address (the IP address of CI4) of the packet. In some embodiments, VSRS, and therefore VRVI A 1904-A, can reside in both L2 and L3 networks. In the L2 network, and therefore within a VLAN, VSRS can use L2 communication protocols, while VSRS can use L3 communication protocols when communicating with the L3 network. In contrast to the learning performed in a VLAN, VSRS can learn the mapping of endpoints in the L3 network from the L3 control plane. In some embodiments, for example, the L3 control plane can provide information mapping the IP addresses, MAC addresses, and / or interface identifiers of instances in the L3 network.

[0347] VRVI A 1904-A can look up a mapping for the IP destination address contained in a packet, or in other words, a mapping for the IP address of the destination CI. In some embodiments, looking up this mapping may include identifying subnet 1939 where CI41940 resides and / or identifying the VR associated with subnet 1939 where CI4 resides. VRVI A 1904-A can encapsulate the packet and forward the encapsulated packet to VR 1948, which may be a tunnel endpoint (TEP) for subnet 1939 and / or destination CI 1940. This forwarding of the encapsulated packet is indicated by box 1956.

[0348] VR 1948 can receive and decapsulate packets. VR 1948 can also identify the interface within subnet 1939 corresponding to the destination IP address. In some embodiments, VR 1948 may reside on the same NVD as the destination interface VI4 1944. Therefore, VR 1948 can forward packets directly to the destination CI, and specifically to CI4 1940, as indicated in box 1958.

[0349] Interface-based access control list filtering

[0350] VSRS can provide interface-based access control list (ACL) filtering. This can include evaluating ingress security policies for VLANs. It can also include evaluating egress security policies based on the learned interface-to-VLAN MAC and IP address mapping at the VSRS when determining where to send received packets. This results in delayed ACL classification.

[0351] In some embodiments, an ACL may include a list of permissions associated with objects within the system. These objects may include hardware within a physical network, such as one or more servers, smart NICs, host machines, etc. These objects may also include one or more virtual objects within a virtual network. These virtual objects may include, for example, one or more interfaces, compute instances, addresses (such as IP addresses and / or MAC addresses), etc. In some embodiments, an ACL may specify which users, objects, and / or system processes are granted access to objects and / or are allowed to perform which operations on a given object.

[0352] An ACL can be specific to one or more CIs. Therefore, in some embodiments, some or all CIs may have a unique ACL and / or maintain a unique ACL. In some embodiments, a CI ACL may specify which interfaces and / or addresses (or MAC or IP addresses) a CI can send packets to, which interfaces and / or addresses (or MAC or IP addresses) a CI cannot send packets to, allow a CI to send one or more types of packets to one or more interfaces and / or addresses, and / or prohibit a CI from sending one or more types of packets to one or more interfaces. In some embodiments, the CI's ACL may be stored in a location accessible to other entities within the network, including other entities such as one or more VRs or VSRSs that can enforce the ACL.

[0353] For example, when receiving communication from one or more intended recipients within a VLAN on an IP network, the VSRS can determine and apply filtering and / or restrictions on the delivery of the communication based on the sender's ACL. In some embodiments, this can be achieved, for example, by: (1) the VSRS making a communication decision based on accessing a copy of the sender's ACL; or (2) the VSRS making a communication decision on the communication received by the VSRS based on information encoded in the packet metadata of the communication.

[0354] Now for reference Figure 20 The diagram illustrates a flowchart of one embodiment of a process 2000 for classifying delayed access control lists (ACLs). Process 2000 can be executed in whole or in part by system 600, and specifically by VSRS 624, 634.

[0355] Process 2000 begins at block 2002, where the source CI sends packets to the destination MAC or IP address. In some embodiments, the source CI may be outside the VLAN containing the destination MAC or IP address to which the packets are sent.

[0356] At block 2006, the VSRS of the VLAN containing the destination MAC or IP address receives packets. In some embodiments, the packets may be encapsulated using L2 encapsulation, and in some embodiments, the packets may be encapsulated using L3 encapsulation. The VSRS may decapsulate the packets and identify the source CI, as indicated in block 2008. After identifying the source CI, the VSRS may access the ACL used for the source CI. In some embodiments, for example, the ACL used for the source CI may be stored in a location accessible to the VSRS. In some embodiments, accessing the ACL used for the source CI may include retrieving information from the ACL of the source CI, such as information that identifies one or more restrictions on packet delivery. In some embodiments, this information may include one or more rules based on one or more IP addresses, MAC addresses, TCP and / or UDP source and destination ports, EtherType, etc.

[0357] At box 2010, the VSRS identifies the destination interface used for the packet. In embodiments where the VSRS does not have a mapping for the destination address, the VSRS can determine the mapping information, as described above regarding... Figure 16As discussed in steps 1612 to 1620 of process 1600. In embodiments where the mapping has been previously learned, or learned via some or all of steps 1612 to 1620 of process 1600, the VSRS can identify the destination interface based on the mapping learned by the VSRS through communication with the interface within the VSRS's VLAN. In some embodiments, identifying the destination interface may include looking up the destination interface based on the destination address, and more specifically, the destination IP address and / or MAC address of the packet.

[0358] At block 2012, VSRS applies the source CI ACL to the destination interface. This may include determining whether any portion of the source CI ACL is relevant to the destination interface, and if so, applying that portion of the source CI ACL. At block 2014, if the destination interface conforms to the source CI ACL, or in other words, if the source CI ACL allows packets to be sent from the source CI to the destination interface, then VSRS forwards the packet to the destination interface. In some embodiments, this forwarding of the packet may include encapsulating the packet. In some embodiments, the packet may be encapsulated according to L2 encapsulation (such as, for example, L2 Geneve encapsulation). VSRS may forward the packet to the destination interface, and more specifically, to the destination CI that uses the destination interface as its TEP. The destination interface may receive the packet, decapsulate the packet, and forward the packet to the destination CI.

[0359] Alternatively, at block 2016, if the destination interface does not conform to the source CI ACL, or in other words, if the source CI ACL does not allow sending packets from the source CI to the destination interface, then the VSRS discards the packet. In some embodiments, the VSRS may respond to the source CI indicating to discard the packet and / or indicating the reason for discarding the packet. In some embodiments, the VSRS may update the operational metrics / statistics associated with the sending interface to reflect the ACL decision.

[0360] Now for reference Figure 21 The diagram illustrates a flowchart of one embodiment of a process 2100 for early classification of ACLs and incorporating that classification into metadata. Process 2100 can be performed by all or part of system 600, and specifically by source CI and VSRS 624, 634.

[0361] At block 2102, the source CI determines which packet to send to the destination CI. In some embodiments, this may include the source CI determining the MAC address and / or IP address to which the packet is sent to the destination CI. As indicated in block 2104, the source VNIC may send the packet. The source VNIC may receive the packet, evaluate the ACL used for the packet, and embed ACL information associated with the packet in a portion of the packet. In some embodiments, this information may include one or more rules based on one or more IP addresses, MAC addresses, TCP and / or UDP source and destination ports, EtherType, etc. The source VNIC may also encapsulate the packet. In some embodiments, the ACL may be stored as metadata in the encapsulated packet, and specifically in the packet header. In some embodiments, the source CI may be outside the VLAN containing the destination CI to which the packet is sent.

[0362] At box 2106, the pause indicates a VSRS receiving packet containing the destination MAC or IP address of the VLAN. In some embodiments, the packet may be encapsulated using L2 encapsulation, and in some embodiments, the packet may be encapsulated using L3 encapsulation. The VSRS can decapsulate the packet. In some embodiments, the VSRS can extract information from the packet that identifies the packet's destination, and specifically, the destination IP address.

[0363] At box 2108, the VSRS identifies the destination interface used for the packet. In embodiments where the VSRS does not have a mapping for the destination address, the VSRS can determine the mapping information, as described above regarding... Figure 16 As discussed in steps 1612 to 1620 of process 1600. In embodiments where the mapping has been previously learned, or learned by performing some or all of steps 1612 to 1620 of process 1600, the VSRS can identify the destination interface based on the mapping learned by the VSRS through communication with the interface within the VSRS's VLAN. Therefore, in some embodiments, the VSRS can determine that it has mapping information for the destination address and can identify the destination interface based on this mapping information. In some embodiments, identifying the destination interface may include looking up the destination interface based on the destination address, and more specifically, the destination IP address and / or MAC address of the packet.

[0364] At block 2110, VSRS accesses ACL information contained in a portion of the packet. In some embodiments, this may include extracting metadata from the packet, and more specifically, extracting metadata from the packet header. In some embodiments, this may include decoding information encoded in the packet metadata within the packet header.

[0365] At block 2112, VSRS applies the ACL information retrieved from this portion of the packet. Specifically, this may include applying security information encoded in the ACL to the destination interface. This may include determining whether any portion of the ACL information is relevant to the destination interface, and if so, applying that portion of the ACL information. At block 2114, if the destination interface conforms to the ACL information and / or conforms to one or more rules of the ACL information, or in other words, if the ACL information allows packets to be sent from the source CI to the destination interface, then VSRS forwards the packet to the destination interface. In some embodiments, this forwarding of the packet may include encapsulating the packet. In some embodiments, the packet may be encapsulated according to L2 encapsulation (such as, for example, L2Geneve encapsulation). VSRS may forward the packet to the destination interface, and more specifically to the destination CI that uses the destination interface as its TEP. The destination interface may receive the packet, decapsulate the packet, and forward the packet to the destination CI.

[0366] Alternatively, at block 2116, if the destination interface does not conform to the ACL information, or in other words, if the ACL information does not allow packets to be sent from the source CI to the destination interface, then the VSRS discards the packet. In some embodiments, the VSRS may respond to the source CI by instructing the packet to be discarded and / or indicating the reason for discarding the packet.

[0367] Next-hop routing

[0368] Some implementations of VSRS can facilitate next-hop routing and, specifically, can delay next-hop evaluation until communication is received at the VSRS. Because the sender of communication outside the VLAN may not know the updated virtual IP address used for the instance within the VLAN, the sender outside the VLAN cannot accurately specify the next-hop route.

[0369] In some embodiments, the VSRS can facilitate next-hop route specification. In some embodiments, this can be achieved, for example, by: (1) the sender making an initial specification of the next-hop route for the communication, and the VSRS re-evaluating the next-hop specification after receiving the communication; or (2) delaying the next-hop specification until the VSRS receives the communication.

[0370] Now for reference Figure 22The diagram illustrates a flowchart of one embodiment of a process 2200 for sender-based next-hop routing. In some embodiments, this may include the separation of next-hop route evaluation from next-hop route specification. This results in the source CI evaluating the routing policy and encoding the next-hop route within the packet metadata of the virtual packet header of the communication. In such embodiments, the packet may include the expected virtual IP address of the next-hop destination, and the next-hop route may be encoded in the packet metadata.

[0371] This communication is received by the VSRS, which then uses the encoded next-hop route from the packet metadata to determine the instance receiving the communication within the VLAN, and the destination virtual IP address of that instance. This determination of the instance can use tables generated and / or curated by the VSRS, and specifically, tables linking virtual IP addresses, MAC addresses, and / or virtual interface IDs. Using these tables, the VSRS can identify the virtual IP address corresponding to the MAC address and / or virtual interface ID of the expected next-hop destination. In some embodiments, this identification of the receiving instance by the VSRS based on the encoded next-hop route contained in the packet metadata can cause the VSRS to send communication to a different destination virtual IP address as the next-hop destination indicated by the virtual IP address in the packet and specified by the sender of the packet.

[0372] Process 2200 can be executed by all or part of system 600, and specifically by source CI and VSRS624, 634.

[0373] The process begins at block 2202, where the source CI can send packets. In some embodiments, the source CI can be outside the VLAN containing the destination MAC address or IP address to which the packets are sent.

[0374] At block 2204, the source VNIC may make a routing decision based on one or more routing rules. In some embodiments, this may include the source VNIC retrieving and / or accessing one or more routing rules and then making a routing decision based on those rules. In some embodiments, this routing decision may be a next-hop routing decision for packets. In some embodiments, these routing rules may be stored in a routing table in, for example, the network of the source CI. In some embodiments, this may include, for example, a subnet routing table.

[0375] At block 2206, the source VNIC may embed routing decisions within a portion of a packet. In some embodiments, this may include encapsulating the packet and embedding the routing decisions within the packet's metadata, which may be encoded, for example, in the packet's header. After encapsulating the packet and embedding the routing decisions within the encapsulated packet, the source VNIC may forward the encapsulated packet to the VSRS.

[0376] At block 2208, a VSRS receive packet containing a VLAN with a destination MAC or IP address. In some embodiments, the packet may be encapsulated using L2 encapsulation, and in some embodiments, the packet may be encapsulated using L3 encapsulation. In some embodiments, receiving a packet by VSRS may include decapsulating the packet. In some embodiments, receiving a packet by VSRS may include extracting routing decisions embedded in a portion of the packet. In some embodiments, this may include decoding encoded metadata containing the routing decisions.

[0377] At block 2210, the VSRS applies a routing decision to the VSRS routing information to determine the destination interface of the packet. In some embodiments, this may include applying a decoded routing decision to the VSRS routing information to determine the destination CI in the VLAN corresponding to the routing information. At block 2212, the VSRS sends the packet to the CI determined in the VLAN. In some embodiments, this may include the VSRS forwarding the packet to the destination interface. In some embodiments, such forwarding of the packet may include encapsulating the packet, which may be based on L2 encapsulation (such as, for example, L2 Geneve encapsulation). The VSRS may forward the packet to the destination interface, more specifically to the destination CI that uses the destination interface as its TEP. The destination interface may receive the packet, decapsulate the packet, and forward the packet to the destination CI.

[0378] Now for reference Figure 23 The diagram illustrates a flowchart of one embodiment of a process 2300 for delayed next-hop routing. Process 2300 may be performed by all or part of system 600, and specifically by source VNIC and VSRS 624, 634.

[0379] In some embodiments, for example, the source VNIC can make routing decisions for communication over a VLAN based on routing rules that may be included in a routing table. This communication can be received by a VSRS, which can re-evaluate the next-hop designation and query a copy of the same routing table to identify the routing rule. Based on this routing rule, the VSRS determines the instance within the VLAN corresponding to the routing rule, the virtual IP address of that instance, and sends the communication to that instance. Determining the instance corresponding to the routing rule may include the VSRS retrieving information from a table linking virtual IP addresses, MAC addresses, and / or virtual interface IDs, and determining the virtual IP address associated with the virtual interface and / or instance as the intended next-hop destination.

[0380] Process 2300 begins at block 2302, where the source CI may send packets. In some embodiments, this may include the source CI sending packets to the MAC address and / or IP address of the destination CI. In some embodiments, the source CI may be outside the VLAN containing the destination MAC address or IP address to which the packets are sent.

[0381] At box 2204, the source VNIC can receive packets. The source VNIC can then make a routing decision based on one or more routing rules. In some embodiments, this may include the source VNIC retrieving and / or accessing one or more routing rules and then making a routing decision based on those rules. In some embodiments, this routing decision may be a next-hop routing decision for the packet. In some embodiments, these routing rules may be stored in a routing table in, for example, the source CI's network. In some embodiments, this may include, for example, a subnet routing table. The source VNIC can then encapsulate the packet and send it to the VSRS.

[0382] At box 2206, a VSRS receive packet containing a VLAN with a destination MAC or IP address. In some embodiments, the packet may be encapsulated using L2 encapsulation, and in some embodiments, the packet may be encapsulated using L3 encapsulation. In some embodiments, receiving a packet by VSRS may include decapsulating the packet.

[0383] At block 2308, the VSRS retrieves routing information associated with the packet. In some embodiments, this may include retrieving a routing table and / or a portion of a routing table associated with the received packet. In some embodiments, this routing table may be received from, for example, the control plane. After receiving the routing information, the VSRS identifies routing rules associated with the received packet.

[0384] At block 2310, the VSRS applies routing rules to the VSRS routing information to determine the destination interface for packets within the VLAN. In some embodiments, this may include applying routing rules to the VSRS routing information to determine the destination CI within the VLAN corresponding to the routing information. At block 2312, the VSRS sends packets to the CI determined within the VLAN. In some embodiments, this may include the VSRS forwarding packets to the destination interface. In some embodiments, such forwarding of packets may include encapsulating the packets, which may be based on L2 encapsulation (such as, for example, L2Geneve encapsulation). The VSRS may forward packets to the destination interface, and more specifically, to the destination CI that uses the destination interface as its TEP. The destination interface may receive packets, decapsulate packets, and forward packets to the destination CI.

[0385] Example Implementation

[0386] As noted above, Infrastructure as a Service (IaaS) is a specific type of cloud computing. IaaS can be configured to provide virtualized computing resources over a public network (e.g., the Internet). In the IaaS model, cloud providers can host infrastructure components (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer), etc.). In some cases, IaaS providers can also offer various services accompanying these infrastructure components (e.g., billing, monitoring, logging, load balancing, and clustering, etc.). Therefore, because these services may be policy-driven, IaaS users can implement policies to drive load balancing to maintain application availability and performance.

[0387] In some cases, IaaS customers can access resources and services over a wide area network (WAN) such as the Internet and can use the cloud provider's services to install the remaining elements of the application stack. For example, a user can log in to the IaaS platform to create virtual machines (VMs), install an operating system (OS) on each VM, deploy middleware such as databases, create buckets for workloads and backups, and even install enterprise software into that VM. The customer can then use the provider's services to perform various functions, including balancing network traffic, troubleshooting application issues, monitoring performance, and managing disaster recovery.

[0388] In most cases, cloud computing models will require the involvement of cloud providers. Cloud providers can, but are not necessarily, third-party providers specializing in (e.g., provisioning, renting, selling) IaaS services. Entities may also choose to deploy private clouds, thus becoming their own infrastructure service providers.

[0389] In some examples, IaaS deployment is the process of placing a new application or a new version of an application onto a prepared application server, etc. It may also include the processing of server preparation (e.g., installation libraries, daemons, etc.). This is typically managed by the cloud provider, below the hypervisor layer (e.g., servers, storage devices, network hardware, and virtualization). Therefore, the customer can be responsible for processing (OS), middleware, and / or application deployment (e.g., on self-service virtual machines, etc., which can be started on demand).

[0390] In some examples, IaaS provisioning can refer to acquiring computers or virtual hosts for use, or even installing necessary libraries or services on them. In most cases, deployment does not include provisioning, and provisioning may need to be performed first.

[0391] In some cases, IaaS provisioning presents two distinct challenges. First, there's the initial challenge of provisioning the initial infrastructure set before anything is operational. Second, once everything is provisioned, there's the challenge of evolving the existing infrastructure (e.g., adding new services, changing services, removing services, etc.). In some cases, both challenges can be addressed by enabling configuration that declaratively defines the infrastructure. In other words, the infrastructure (e.g., which components are needed and how they interact) can be defined by one or more configuration files. Therefore, the overall topology of the infrastructure (e.g., which resources depend on which resources and how they work together) can be described declaratively. In some cases, once the topology is defined, workflows for creating and / or managing the different components described in the configuration files can be generated.

[0392] In some examples, the infrastructure can have many interconnected components. For example, there may be one or more Virtual Private Clouds (VPCs) (e.g., potential on-demand pools of configurable and / or shared computing resources), also known as the core network. In some examples, one or more inbound / outbound traffic group rules may also be provided to define how inbound / outbound traffic to the network and one or more virtual machines (VMs). Other infrastructure elements, such as load balancers, databases, etc., may also be provided. The infrastructure can evolve incrementally as more and / or more infrastructure elements are expected and added.

[0393] In some cases, continuous deployment techniques can be used to enable the deployment of infrastructure code across various virtual computing environments. Furthermore, the described techniques enable infrastructure management within these environments. In some examples, service teams may write code that they expect to deploy to one or more, but often many, different production environments (e.g., across various geographical locations, sometimes spanning the entire world). However, in some examples, the infrastructure on which the code will be deployed must first be set up. In some cases, provisioning can be done manually, resources can be provisioned using provisioning tools, and / or once the infrastructure is provisioned, the code can be deployed using deployment tools.

[0394] Figure 24 This is a block diagram 2400 illustrating an example pattern of an IaaS architecture according to at least one embodiment. Service operator 2402 may be communicatively coupled to a secure host lease 2404, which may include a virtual cloud network (VCN) 2406 and a secure host subnet 2408. In some examples, service operator 2402 may use one or more client computing devices, which may be portable handheld devices (e.g., Cellular phone Computing tablets, personal digital assistants (PDAs), or wearable devices (e.g., Google) Head-mounted displays), running software (such as Microsoft Windows) ) and / or various mobile operating systems (such as iOS, Windows Phone, Android, BlackBerry 8, Palm OS, etc.), and support the Internet, email, and short message service (SMS). Or other communication protocols. Alternatively, the client computing device can be a general-purpose personal computer, including, for example, those running various versions of Microsoft... Apple Personal computers and / or laptops running Linux operating systems. Client computing devices can be running various commercially available operating systems. Or a workstation computer operating a UNIX-like operating system, including but not limited to any of the various GNU / Linux operating systems (such as, for example, Google Chrome OS). Alternatively or additionally, the client computing device can be any other electronic device, such as a thin client computer, an internet-enabled gaming system (e.g., with or without...). A Microsoft Xbox game console with gesture input devices, and / or a personal messaging device capable of communicating via a network that can access the VCN2406 and / or the Internet.

[0395] VCN 2406 may include a local peering gateway (LPG) 2410, which may be communicatively coupled to a secure shell (SSH) VCN 2412 via the LPG 2410 included in SSH VCN 2412. SSH VCN 2412 may include an SSH subnet 2414, and SSH VCN 2412 may be communicatively coupled to a control plane VCN 2416 via the LPG 2410 included in control plane VCN 2416. Furthermore, SSH VCN 2412 may be communicatively coupled to a data plane VCN 2418 via the LPG 2410. Control plane VCN 2416 and data plane VCN 2418 may be contained within a service lease 2419 that may be owned and / or operated by an IaaS provider.

[0396] The control plane VCN 2416 may include a control plane demilitarized zone (DMZ) layer 2420 that acts as a peripheral network (e.g., a portion of a corporate network between a corporate intranet and an external network). DMZ-based servers can assume limited liability and help control vulnerabilities. Furthermore, the DMZ layer 2420 may include one or more load balancer (LB) subnets 2422, a control plane application layer 2424 that may include one or more application subnets 2426, and a control plane data layer 2428 that may include one or more database (DB) subnets 2430 (e.g., one or more front-end DB subnets and / or one or more back-end DB subnets). One or more LB subnets 2422 contained in the control plane DMZ layer 2420 can be communicatively coupled to one or more application subnets 2426 contained in the control plane application layer 2424 and an Internet gateway 2434 that can be contained in the control plane VCN 2416. The application subnets 2426 can be communicatively coupled to one or more DB subnets 2430 contained in the control plane data layer 2428, as well as a service gateway 2436 and a Network Address Translation (NAT) gateway 2438. The control plane VCN 2416 may include the service gateway 2436 and the NAT gateway 2438.

[0397] The control plane VCN 2416 may include a data plane mirror application layer 2440, which may include one or more application subnets 2426. The one or more application subnets 2426 included in the data plane mirror application layer 2440 may include a virtual network interface controller (VNIC) 2442 capable of executing a compute instance 2444. The compute instance 2444 may communicatively couple the one or more application subnets 2426 of the data plane mirror application layer 2440 to the one or more application subnets 2426 that may be included in the data plane application layer 2446.

[0398] Data plane VCN 2418 may include data plane application layer 2446, data plane DMZ layer 2448, and data plane data layer 2450. Data plane DMZ layer 2448 may include one or more LB subnets 2422 communicatively coupled to one or more application subnets 2426 of data plane application layer 2446 and Internet gateway 2434 of data plane VCN 2418. One or more application subnets 2426 may be communicatively coupled to service gateway 2436 and NAT gateway 2438 of data plane VCN 2418. Data plane data layer 2450 may also include one or more DB subnets 2430 communicatively coupled to one or more application subnets 2426 of data plane application layer 2446.

[0399] The Internet gateway 2434 of the control plane VCN 2416 and data plane VCN 2418 can be communicatively coupled to the metadata management service 2452, which in turn can be communicatively coupled to the public Internet 2454. The public Internet 2454 can be communicatively coupled to the NAT gateway 2438 of the control plane VCN 2416 and data plane VCN 2418. The service gateway 2436 of the control plane VCN 2416 and data plane VCN 2418 can be communicatively coupled to the cloud service 2456.

[0400] In some examples, the service gateway 2436 of the control plane VCN 2416 or data plane VCN 2418 can make application programming interface (API) calls to the cloud service 2456 without traversing the public internet 2454. API calls from the service gateway 2436 to the cloud service 2456 can be unidirectional: the service gateway 2436 can make API calls to the cloud service 2456, and the cloud service 2456 can send requested data to the service gateway 2436. However, the cloud service 2456 may not initiate API calls to the service gateway 2436.

[0401] In some examples, secure host lease 2404 can be directly connected to service lease 2419, which would otherwise be isolated. Secure host subnet 2408 can communicate with SSH subnet 2414 via LPG 2410, which enables bidirectional communication between otherwise isolated systems. Connecting secure host subnet 2408 to SSH subnet 2414 allows secure host subnet 2408 to access other entities within service lease 2419.

[0402] Control plane VCN 2416 may allow users of service lease 2419 to configure or otherwise provision desired resources. Desired resources provisioned in control plane VCN 2416 may be deployed or otherwise used in data plane VCN 2418. In some examples, control plane VCN 2416 may be isolated from data plane VCN 2418, and the data plane mirror application layer 2440 of control plane VCN 2416 may communicate with the data plane application layer 2446 of data plane VCN 2418 via VNIC 2442, which may be included in both the data plane mirror application layer 2440 and the data plane application layer 2446.

[0403] In some examples, users or clients of the system can make requests, such as create, read, update, or delete (CRUD) operations, via the public internet 2454, which can transmit requests to the metadata management service 2452. The metadata management service 2452 can transmit requests to the control plane VCN 2416 via internet gateway 2434. Requests can be received by one or more LB subnets 2422 contained in the control plane DMZ layer 2420. The LB subnets 2422 can determine that the request is valid, and in response to this determination, they can transmit the request to one or more application subnets 2426 contained in the control plane application layer 2424. If the request is validated and requires a call to the public internet 2454, the call to the public internet 2454 can be transmitted to a NAT gateway 2438 that can make calls to the public internet 2454. The request may expect the storage to be located in one or more DB subnets 2430.

[0404] In some examples, the data plane mirroring application layer 2440 can facilitate direct communication between the control plane VCN 2416 and the data plane VCN 2418. For example, it may be desirable to apply configuration changes, updates, or other appropriate modifications to resources contained in the data plane VCN 2418. Through VNIC 2442, the control plane VCN 2416 can communicate directly with the resources contained in the data plane VCN 2418, and thus can perform configuration changes, updates, or other appropriate modifications.

[0405] In some embodiments, the control plane VCN 2416 and data plane VCN 2418 may be contained within a service lease 2419. In this case, the system's users or customers may not own or operate the control plane VCN 2416 or the data plane VCN 2418. Alternatively, the IaaS provider may own or operate both the control plane VCN 2416 and the data plane VCN 2418, both of which may be contained within the service lease 2419. This embodiment enables the isolation of networks that might prevent users or customers from interacting with resources of other users or customers. Furthermore, this embodiment allows users or customers of the system to privately store databases without relying on the public internet 2454, which may not have the desired level of threat prevention for storage.

[0406] In other embodiments, one or more LB subnets 2422 included in the control plane VCN 2416 may be configured to receive signals from the service gateway 2436. In this embodiment, the control plane VCN 2416 and the data plane VCN 2418 may be configured to be invoked by the IaaS provider's customers without invoking the public internet 2454. The IaaS provider's customers may expect this embodiment because the database(s) used by the customer can be controlled by the IaaS provider and can be stored on service lease 2419, which may be isolated from the public internet 2454.

[0407] Figure 25 This is a block diagram 2500 illustrating another example pattern of an IaaS architecture according to at least one embodiment. Service operator 2502 (e.g., Figure 24 The service provider 2402) can communicatively couple to the secure host lease 2504 (e.g., Figure 24 The secure hosting lease 2404 may include a virtual cloud network (VCN) 2506 (e.g., Figure 24 VCN 2406) and Secure Host Subnet 2508 (e.g., Figure 24 The secure host subnet 2408). VCN 2506 may include a local peering gateway (LPG) 2510 (e.g., Figure 24 The LPG 2410, which can be communicatively coupled to the Secure Shell (SSH) VCN 2512 (e.g., LPG 2410 contained in the SSH VCN 2512) via the LPG 2410 included in the SSH VCN 2512. Figure 24 SSH VCN 2412). SSH VCN 2512 can include SSH subnet 2514 (e.g., Figure 24 The SSH subnet 2414), and the SSH VCN 2512 can be communicatively coupled to the control plane VCN 2516 via the LPG 2510 included in the control plane VCN 2516 (e.g., Figure 24 Control plane VCN 2416). Control plane VCN 2516 may be included in service lease 2519 (e.g., Figure 24 In the service lease 2419), and the data plane VCN 2518 (e.g., Figure 24 The data plane VCN 2418 may be included in a customer lease 2521 that may be owned or operated by the system’s users or customers.

[0408] Control plane VCN 2516 may include control plane DMZ layer 2520 (e.g., Figure 24 The control plane DMZ layer 2420 may include one or more LB subnets 2522 (e.g., Figure 24 (one or more) LB subnets 2422), may include (one or more) application subnets 2526 (e.g., Figure 24 The control plane application layer 2524 of (one or more) application subnets 2426 (e.g., Figure 24 The control plane application layer 2424 may include one or more database (DB) subnets 2530 (e.g., similar to...). Figure 24 The control plane data layer 2528 of (one or more) DB subnets 2430 (e.g., Figure 24 The control plane data layer 2428). One or more LB subnets 2522 contained in the control plane DMZ layer 2520 can be communicatively coupled to one or more application subnets 2526 contained in the control plane application layer 2524 and an Internet gateway 2534 that can be contained in the control plane VCN 2516 (e.g., Figure 24 Internet gateway 2434), and application subnet(s) 2526 can communicatively couple to DB subnet(s) 2530 contained in control plane data layer 2528 and service gateway 2536 (e.g., Figure 24 The service gateway) and Network Address Translation (NAT) gateway 2538 (e.g., Figure 24 (NAT gateway 2438). The control plane VCN 2516 may include the service gateway 2536 and the NAT gateway 2538.

[0409] The control plane VCN 2516 may include a data plane mirror of the application layer 2540, which may include one or more application subnets 2526 (e.g., Figure 24 The data plane mirror application layer 2440). One or more application subnets 2526 contained in the data plane mirror application layer 2540 may include computational instances 2544 (e.g., similar to...). Figure 24 The virtual network interface controller (VNIC) 2542 (e.g., the VNIC of 2442) of the computing instance 2444. The computing instance 2544 can facilitate the mirroring of the data plane application layer 2540 with one or more application subnets 2526 and can be included in the data plane application layer 2546 (e.g., Figure 24 Communication between one or more application subnets 2526 in the data plane application layer 2446 via VNIC 2542 contained in the data plane mirror application layer 2540 and VNIC 2542 contained in the data plane application layer 2546.

[0410] The Internet gateway 2534 included in the control plane VCN 2516 can be communicatively coupled to the metadata management service 2552 (e.g., Figure 24 Metadata management service 2452), which can communicatively couple to the public Internet 2554 (e.g., Figure 24 The public internet 2554 can communicatively couple to a NAT gateway 2538 contained in a control plane VCN 2516. The service gateway 2536 contained in the control plane VCN 2516 can communicatively couple to a cloud service 2556 (e.g., ...). Figure 24 Cloud services (2456).

[0411] In some examples, data plane VCN 2518 may be included in customer lease 2521. In this case, the IaaS provider may provide control plane VCN 2516 for each customer, and the IaaS provider may set up a unique compute instance 2544 for each customer, included in service lease 2519. Each compute instance 2544 may allow communication between control plane VCN 2516 included in service lease 2519 and data plane VCN 2518 included in customer lease 2521. Compute instance 2544 may allow resources provisioned in control plane VCN 2516 included in service lease 2519 to be deployed or otherwise used in data plane VCN 2518 included in customer lease 2521.

[0412] In other examples, an IaaS provider's customer may have a database residing in customer lease 2521. In this example, control plane VCN 2516 may include data plane mirror application layer 2540, which may include one or more application subnets 2526. Data plane mirror application layer 2540 may reside in data plane VCN 2518, but may not reside in data plane VCN 2518. That is, data plane mirror application layer 2540 may have access to customer lease 2521, but may not reside in data plane VCN 2518 or be owned or operated by an IaaS provider's customer. Data plane mirror application layer 2540 may be configured to invoke data plane VCN 2518, but may not be configured to invoke any entity contained in control plane VCN 2516. Customers may expect to deploy or otherwise use resources provisioned in the control plane VCN 2516 in the data plane VCN 2518, and the data plane mirroring application layer 2540 can facilitate the customer's desired deployment or other use of resources.

[0413] In some embodiments, an IaaS provider's customer can apply filters to data plane VCN 2518. In this embodiment, the customer can determine what data plane VCN 2518 can access, and the customer can restrict access from data plane VCN 2518 to the public Internet 2554. The IaaS provider may not be able to apply filters or otherwise control data plane VCN 2518's access to any external networks or databases. Applying filters and controls to data plane VCN 2518 contained in customer lease 2521 can help isolate data plane VCN 2518 from other customers and the public Internet 2554.

[0414] In some embodiments, cloud service 2556 may be invoked by service gateway 2536 to access services that may not exist on public internet 2554, control plane VCN 2516, or data plane VCN 2518. The connection between cloud service 2556 and control plane VCN 2516 or data plane VCN 2518 may not be real-time or continuous. Cloud service 2556 may reside on different networks owned or operated by an IaaS provider. Cloud service 2556 may be configured to receive calls from service gateway 2536 and may be configured not to receive calls from public internet 2554. Some cloud services 2556 may be isolated from other cloud services 2556, and control plane VCN 2516 may be isolated from cloud services 2556 that may not be in the same region as control plane VCN 2516. For example, control plane VCN 2516 may be located in "Region 1," and cloud service "Deployment 24" may be located in both "Region 1" and "Region 2." If the service gateway 2536, contained in the control plane VCN 2516 located in region 1, makes a call to deployment 24, then that call can be transmitted to deployment 24 in region 1. In this example, the control plane VCN 2516 or deployment 24 in region 1 may not be communicatively coupled to or otherwise communicate with deployment 24 in region 2.

[0415] Figure 26 This is a block diagram 2600 illustrating another example pattern of an IaaS architecture according to at least one embodiment. Service operator 2602 (e.g., Figure 24 The service provider 2402) can communicatively couple to the secure host lease 2604 (e.g., Figure 24 Secure hosting lease 2404), the secure hosting lease 2604 may include a virtual cloud network (VCN) 2606 (e.g., Figure 24 VCN 2406) and Secure Host Subnet 2608 (e.g., Figure 24 The secure host subnet 2408). VCN 2606 may include LPG 2610 (e.g., Figure 24 The LPG 2410), which can be communicatively coupled to the SSH VCN 2612 via the LPG 2610 included in the SSH VCN 2612 (e.g., Figure 24 SSH VCN 2412). SSH VCN 2612 can include SSH subnet 2614 (e.g., Figure 24 The SSH subnet 2414), and the SSH VCN 2612 can be communicatively coupled to the control plane VCN 2616 via the LPG 2610 included in the control plane VCN 2616 (e.g., Figure 24 The control plane VCN 2416) and coupled to the data plane VCN 2618 via the LPG 2610 contained in the data plane VCN 2618 (e.g., Figure 24 Data plane 2418). Control plane VCN 2616 and data plane VCN 2618 may be included in service lease 2619 (e.g., Figure 24 (Service rental 2419).

[0416] The control plane VCN 2616 may include a load balancer (LB) subnet 2622 (e.g., Figure 24 The control plane DMZ layer 2620 of (one or more) LB subnets 2422) (e.g., Figure 24 The control plane DMZ layer 2420 may include one or more application subnets 2626 (e.g., similar to...). Figure 24 The control plane application layer 2624 of (one or more) application subnets 2426 (e.g., Figure 24 The control plane application layer 2424), may include a control plane data layer 2628 (e.g., DB subnet 2630) that may include one or more DB subnets 2630. Figure 24 The control plane data layer 2428). One or more LB subnets 2622 contained in the control plane DMZ layer 2620 can be communicatively coupled to one or more application subnets 2626 contained in the control plane application layer 2624 and an Internet gateway 2634 that can be contained in the control plane VCN 2616 (e.g., Figure 24 Internet gateway 2434), and application subnet(s) 2626 can communicatively couple to DB subnet(s) 2630 contained in control plane data layer 2628 and service gateway 2636 (e.g., Figure 24 The service gateway) and Network Address Translation (NAT) gateway 2638 (e.g., Figure 24 (NAT gateway 2438). The control plane VCN 2616 may include the service gateway 2636 and the NAT gateway 2638.

[0417] The data plane VCN 2618 may include the data plane application layer 2646 (e.g., Figure 24 Data plane application layer 2446), data plane DMZ layer 2648 (e.g., Figure 24 Data plane DMZ layer 2448), and data plane data layer 2650 (e.g., Figure 24 The data plane data layer 2450). The data plane DMZ layer 2648 may include one or more trusted application subnets 2660 and one or more untrusted application subnets 2662 that can be communicatively coupled to the data plane application layer 2646, and one or more LB subnets 2622 of the Internet gateway 2634 contained in the data plane VCN 2618. The one or more trusted application subnets 2660 may be communicatively coupled to the service gateway 2636 contained in the data plane VCN 2618, the NAT gateway 2638 contained in the data plane VCN 2618, and one or more DB subnets 2630 contained in the data plane data layer 2650. The one or more untrusted application subnets 2662 may be communicatively coupled to the service gateway 2636 contained in the data plane VCN 2618 and the one or more DB subnets 2630 contained in the data plane data layer 2650. The data plane data layer 2650 may include one or more DB subnets 2630 that can be communicatively coupled to the service gateway 2636 contained in the data plane VCN 2618.

[0418] One or more untrusted application subnets 2662 may include one or more primary VNICs 2664(1)-(N) that can be communicatively coupled to tenant virtual machines (VMs) 2666(1)-(N). Each tenant VM 2666(1)-(N) may be communicatively coupled to a corresponding application subnet 2667(1)-(N) that may be contained in a corresponding container egress VCN 2668(1)-(N), which may be contained in a corresponding customer lease 2670(1)-(N). A corresponding secondary VNIC 2672(1)-(N) may facilitate communication between one or more untrusted application subnets 2662 contained in the data plane VCN 2618 and the application subnets contained in the container egress VCN 2668(1)-(N). Each container exit VCN 2668(1)-(N) may include a NAT gateway 2638, which can communicatively couple to the public Internet 2654 (e.g., Figure 24 (Public Internet 2454).

[0419] The Internet gateway 2634, contained in the control plane VCN 2616 and the data plane VCN 2618, can be communicatively coupled to the metadata management service 2652 (e.g., Figure 24 A metadata management system 2452, which is communicatively coupled to the public internet 2654, is also communicatively coupled to a NAT gateway 2638 contained in a control plane VCN 2616 and a data plane VCN 2618. A service gateway 2636 contained in both the control plane VCN 2616 and the data plane VCN 2618 is communicatively coupled to a cloud service 2656.

[0420] In some embodiments, the data plane VCN 2618 may be integrated with the customer lease 2670. Such integration may be useful or desired by the IaaS provider's customers in certain situations, such as when support may be expected during code execution. Customers may provide code that could be destructive, might communicate with other customer resources, or might otherwise cause undesirable effects. In response, the IaaS provider may determine whether to run the code provided by the customer to the IaaS provider.

[0421] In some examples, an IaaS provider's customer can grant temporary network access to the IaaS provider and request functionality attached to data plane layer application 2646. The code running this functionality can execute within VMs 2666(1)-(N), and this code may not be configured to run anywhere else on data plane VCN 2618. Each VM 2666(1)-(N) can be connected to a customer lease 2670. A corresponding container 2671(1)-(N) contained within VMs 2666(1)-(N) can be configured to run the code. In this scenario, dual isolation can exist (e.g., container 2671(1)-(N) runs the code, where container 2671(1)-(N) may be contained within at least one VM 2666(1)-(N) contained in untrusted application subnet 2662), which can help prevent incorrect or otherwise unintended code from corrupting the IaaS provider's network or the networks of different customers. Container 2671(1)-(N) may be communicatively coupled to customer lease 2670 and may be configured to transmit or receive data from customer lease 2670. Container 2671(1)-(N) may not be configured to transmit or receive data from any other entity in the data plane VCN 2618. After the code execution is complete, the IaaS provider may terminate or otherwise dispose of container 2671(1)-(N).

[0422] In some embodiments, one or more trusted application subnets 2660 may run code that can be owned or operated by an IaaS provider. In this embodiment, one or more trusted application subnets 2660 may be communicatively coupled to one or more database subnets 2630 and configured to perform CRUD operations in one or more database subnets 2630. One or more untrusted application subnets 2662 may be communicatively coupled to one or more database subnets 2630, but in this embodiment, one or more untrusted application subnets may be configured to perform read operations in one or more database subnets 2630. Containers 2671(1)-(N) that may be contained in each customer's VM 2666(1)-(N) and may run code from the customer may not be communicatively coupled to one or more database subnets 2630.

[0423] In other embodiments, the control plane VCN 2616 and the data plane VCN 2618 may be coupled without direct communication. In this embodiment, there may be no direct communication between the control plane VCN 2616 and the data plane VCN 2618. However, communication can occur indirectly through at least one method. The LPG 2610 may be established by an IaaS provider, which can facilitate communication between the control plane VCN 2616 and the data plane VCN 2618. In another example, the control plane VCN 2616 or the data plane VCN 2618 may invoke the cloud service 2656 via the service gateway 2636. For example, an invocation of the cloud service 2656 from the control plane VCN 2616 may include a request for a service that can communicate with the data plane VCN 2618.

[0424] Figure 27 This is a block diagram 2700 illustrating another example pattern of an IaaS architecture according to at least one embodiment. Service operator 2702 (e.g., Figure 24 The service provider 2402) can communicatively couple to the secure host lease 2704 (e.g., Figure 24 Secure hosting lease 2404), the secure hosting lease 2704 may include a virtual cloud network (VCN) 2706 (e.g., Figure 24 VCN 2406) and Secure Host Subnet 2708 (e.g., Figure 24 The secure host subnet 2408). VCN 2706 can include LPG 2710 (e.g., Figure 24 The LPG 2410), the LPG 2710 can be accessed via SSH VCN 2712 (e.g., Figure 24The LPG 2710 in the SSH VCN 2412 is communicatively coupled to the SSH VCN 2712. The SSH VCN 2712 may include the SSH subnet 2714 (e.g., Figure 24 The SSH subnet 2414), and the SSH VCN 2712 can be communicatively coupled to the control plane VCN 2716 via the LPG 2710 included in the control plane VCN 2716 (e.g., Figure 24 The control plane VCN 2416) and coupled to the data plane VCN 2718 via the LPG 2710 contained in the data plane VCN 2718 (e.g., Figure 24 Data plane 2418). Control plane VCN 2716 and data plane VCN 2718 may be included in service lease 2719 (e.g., Figure 24 (Service rental 2419).

[0425] The control plane VCN 2716 may include one or more LB subnets 2722 (e.g., Figure 24 The control plane DMZ layer 2720 of (one or more) LB subnets 2422) (e.g., Figure 24 The control plane DMZ layer 2420 may include (one or more) application subnets 2726 (e.g., Figure 24 The control plane application layer 2724 of (one or more) application subnets 2426 (e.g., Figure 24 The control plane application layer 2424) may include (one or more) DB subnets 2730 (e.g., Figure 26 The control plane data layer 2728 of (one or more) DB subnets 2630 (e.g., Figure 24 The control plane data layer 2428). One or more LB subnets 2722 contained in the control plane DMZ layer 2720 can be communicatively coupled to one or more application subnets 2726 contained in the control plane application layer 2724 and an Internet gateway 2734 that can be contained in the control plane VCN 2716 (e.g., Figure 24 Internet gateway 2434), and application subnet(s) 2726 can communicatively couple to DB subnet(s) 2730 contained in control plane data layer 2728 and service gateway 2736 (e.g., Figure 24 The service gateway) and Network Address Translation (NAT) gateway 2738 (e.g., Figure 24 (NAT gateway 2438). The control plane VCN 2716 may include the service gateway 2736 and the NAT gateway 2738.

[0426] Data plane VCN 2718 may include data plane application layer 2746 (e.g., Figure 24 Data plane application layer 2446), data plane DMZ layer 2748 (e.g., Figure 24 Data plane DMZ layer 2448), and data plane data layer 2750 (e.g., Figure 24 The data plane data layer 2450). The data plane DMZ layer 2748 may include one or more trusted application subnets 2760 that can be communicatively coupled to the data plane application layer 2746 (e.g., Figure 26 (one or more) trusted application subnets 2660) and (one or more) untrusted application subnets 2762 (e.g., Figure 26 The data plane VCN 2718 may include one or more untrusted application subnets 2762 and one or more LB subnets 2722. One or more trusted application subnets 2760 may communicatively couple to a service gateway 2736, a NAT gateway 2738, and a DB subnet 2730 in the data plane VCN 2750. One or more untrusted application subnets 2762 may communicatively couple to a service gateway 2736 and a DB subnet 2730 in the data plane VCN 2718. The data plane VCN 2750 may include one or more DB subnets 2730 that may communicatively couple to a service gateway 2736 in the data plane VCN 2718.

[0427] One or more untrusted application subnets 2762 may include a primary VNIC 2764(1)-(N) communicatively coupled to tenant virtual machines (VMs) 2766(1)-(N) residing within one or more untrusted application subnets 2762. Each tenant VM 2766(1)-(N) may run code in a corresponding container 2767(1)-(N) and is communicatively coupled to an application subnet 2726 that may be contained in a data plane application layer 2746 contained in a container egress VCN 2768. A corresponding secondary VNIC 2772(1)-(N) may facilitate communication between one or more untrusted application subnets 2762 contained in a data plane VCN 2718 and the application subnets contained in a container egress VCN 2768. The container egress VCN may include a public internet 2754 (e.g., Figure 24 NAT gateway 2738 for the public Internet 2454.

[0428] Internet gateway 2734, contained in control plane VCN 2716 and data plane VCN 2718, can be communicatively coupled to metadata management service 2752 (e.g., Figure 24 A metadata management system 2452, which is communicatively coupled to the public internet 2754, is also communicatively coupled to a NAT gateway 2738 contained in a control plane VCN 2716 and a data plane VCN 2718. A service gateway 2736 contained in both the control plane VCN 2716 and the data plane VCN 2718 is communicatively coupled to a cloud service 2756.

[0429] In some examples, Figure 27 The architecture shown in the block diagram 2700 can be considered as... Figure 10 This is an exception to the pattern shown in the architecture diagram 1000, and this pattern may be what the IaaS provider's customers would expect if the IaaS provider cannot communicate directly with the customer (e.g., in a disconnected region). The customer can access in real time the corresponding container 2767(1)-(N) contained in each customer's VM 2766(1)-(N). Container 2767(1)-(N) can be configured to invoke a corresponding auxiliary VNIC 2772(1)-(N) contained in one or more application subnets 2726 of the data plane application layer 2746, which may be contained in the container egress VCN 2768. The auxiliary VNIC 2772(1)-(N) can transmit the call to a NAT gateway 2738, which can then transmit the call to the public internet 2754. In this example, containers 2767(1)-(N), which can be accessed by clients in real time, can be isolated from the control plane VCN 2716 and from other entities contained in the data plane VCN 2718. Containers 2767(1)-(N) can also be isolated from resources from other clients.

[0430] In other examples, a client can use container 2767(1)-(N) to invoke cloud service 2756. In this example, the client can run code within container 2767(1)-(N) requesting a service from cloud service 2756. Container 2767(1)-(N) can then transmit the request to auxiliary VNIC 2772(1)-(N), which can then transmit the request to a NAT gateway, which can then transmit the request to the public internet 2754. The public internet 2754 can then transmit the request via internet gateway 2734 to one or more LB subnets 2722 contained in control plane VCN 2716. In response to determining that the request is valid, the one or more LB subnets can then transmit the request to one or more application subnets 2726, which can then transmit the request to cloud service 2756 via service gateway 2736.

[0431] It should be recognized that the IaaS architectures 2400, 2500, 2600, and 2700 depicted in the figures may have other components besides those depicted. Furthermore, the embodiments shown in the figures are merely some examples of cloud infrastructure systems that can be incorporated into embodiments of this disclosure. In some other embodiments, the IaaS system may have more or fewer components than shown in the figures, may combine two or more components, or may have different configurations or component arrangements.

[0432] In some embodiments, the IaaS system described herein may include application suites, middleware, and database service offerings delivered to customers in a self-service, subscription-based, elastically scalable, reliable, highly available, and secure manner. An example of such an IaaS system is the Oracle Cloud Infrastructure (OCI) provided by this assignee.

[0433] Figure 28 An example computer system 2800, in which various embodiments can be implemented, is illustrated. System 2800 can be used to implement any of the computer systems described above. As shown, computer system 2800 includes a processing unit 2804 that communicates with a plurality of peripheral subsystems via a bus subsystem 2802. These peripheral subsystems may include a processing acceleration unit 2806, an I / O subsystem 2808, a storage subsystem 2818, and a communication subsystem 2824. Storage subsystem 2818 includes a tangible computer-readable storage medium 2822 and system memory 2810.

[0434] Bus subsystem 2802 provides a mechanism for allowing various components and subsystems of computer system 2800 to communicate with each other as intended. While bus subsystem 2802 is schematically shown as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses. Bus subsystem 2802 can be any of several types of bus architectures, including memory buses or memory controllers, peripheral buses, and local buses using any of the various bus architectures. For example, such architectures may include Industry Standard Architecture (ISA) buses, Microchannel Architecture (MCA) buses, Enhanced ISA (EISA) buses, Video Electronics Standards Association (VESA) local buses, and Peripheral Component Interconnect (PCI) buses, which may be implemented as Mezzanine buses manufactured according to the IEEE P1386.1 standard.

[0435] A processing unit 2804, which may be implemented as one or more integrated circuits (e.g., a conventional microprocessor or microcontroller), controls the operation of the computer system 2800. One or more processors may be included in the processing unit 2804. These processors may include single-core or multi-core processors. In some embodiments, the processing unit 2804 may be implemented as one or more independent processing units 2832 and / or 2834, each including a single-core or multi-core processor. In other embodiments, ...

Claims

1. A method comprising: A table is generated for instances of VLAN Switching and Routing Service (VSRS), which couples a first Virtual Layer 2 network (VLAN) to a second network. This table contains information that identifies instances within the Virtual Layer 2 network, including their IP addresses, MAC addresses, and virtual interface identifiers. Receive packets from the first instance using VSRS, packets designated for delivery to the second instance within the Virtual Layer 2 network; Based on the information received along with the packets and the information contained in the table, a second instance within the Virtual Layer 2 network is identified using VSRS to deliver packets; and The group will be delivered to the identified second instance.

2. The method of claim 1, wherein the first virtual layer 2 network comprises multiple instances.

3. The method of claim 2, wherein the VLAN includes a plurality of Layer 2 Virtual Network Interface Cards (VNICs) and a plurality of switches, wherein each of the plurality of instances is communicatively coupled to a pair including a unique Layer 2 Virtual Network Interface Card (VNIC) and a unique switch.

4. The method of any one of claims 1 to 3, wherein identifying a second instance within the Virtual Layer 2 network for packet delivery using VSRS based on information received along with the packet and information contained in a table comprises: VSRS was used to determine that the table does not include mapping information for the second instance; Suspend packet delivery using VSRS; Use VSRS to broadcast an ARP request to the VNIC in the VLAN, which includes the IP address of the second instance; and Use VSRS to receive ARP responses from the VNIC of the second instance.

5. The method of claim 4, further comprising updating the table based on the received ARP response.

6. The method of claim 2 or 3, wherein the first instance is outside the virtual layer 2 network and in the second network.

7. The method of claim 6, wherein the second network comprises a layer 3 network.

8. The method of claim 6, wherein the second network includes a second virtual layer 2 network.

9. The method of claim 2 or 3, wherein the table is generated based on communications received by VSRS.

10. The method of claim 9, further comprising instantiating VSRS as a service on multiple hardware nodes.

11. The method of claim 10, further comprising a cross-hardware node distribution table.

12. The method of claim 11, wherein the table distributed across hardware nodes can be accessed by another VSRS instantiation.

13. The method of any one of claims 1 to 3, wherein the first instance is within the first virtual layer 2 network.

14. The method of any one of claims 1 to 13, further comprising: VSRS is used to receive packets from a third instance inside the Virtual Layer 2 network, where the packets are designated for delivery to a fourth instance outside the Virtual Layer 2 network and forwarded to the fourth instance.

15. The method of any one of claims 1 to 13, further comprising: VSRS is used to receive packets from a third instance within the Virtual Layer 2 network, where the packets are designated for delivery to a service used by the third instance within the Virtual Layer 2 network.

16. The method of claim 15, wherein the service includes at least one of: DHCP; NTP; and DNS.

17. The method of any one of claims 1 to 13, further comprising: VSRS receives packets from a third instance within a virtual layer 2 network, where the packets are designated for delivery to a fourth instance in a second virtual layer 2 network.

18. The method of any one of claims 1 to 9, further comprising: A table for VSRS instances with Layer 2 and Layer 3 network information is distributed across service nodes to provide highly reliable and scalable instantiation of VSRS.

19. The method of claim 1, further comprising: Receive packets from a third instance within the first virtual layer 2 network using VSRS; and Learn the mapping of the third instance using VSRS.

20. A system comprising: Physical networks include: At least one processor; and Network virtualization devices, The at least one processor is configured to: Instantiate an instance of VLAN Switching and Routing Service (VSRS), which couples the first Virtual Layer 2 network with the second network; Generate a table for VSRS instances, which contains information that identifies the instances within the Virtual Layer 2 network, including their IP addresses, MAC addresses, and virtual interface identifiers. Receive packets from the first instance using VSRS, packets designated for delivery to the second instance within the Virtual Layer 2 network; Based on the information received along with the packets and the information contained in the table, a second instance within the Virtual Layer 2 network is identified using VSRS to deliver packets; and The group will be delivered to the identified second instance.

21. A non-transitory computer-readable storage medium storing a plurality of instructions executable by one or more processors, the plurality of instructions causing the one or more processors, when executed by the one or more processors, to: Instantiate an instance of VLAN Switching and Routing Service (VSRS), which couples the first Virtual Layer 2 network with the second network; Generate a table for VSRS instances, which contains information that identifies the instances within the Virtual Layer 2 network, including their IP addresses, MAC addresses, and virtual interface identifiers. Receive packets from the first instance using VSRS, packets designated for delivery to the second instance within the Virtual Layer 2 network; Based on the information received along with the packets and the information contained in the table, a second instance within the Virtual Layer 2 network is identified using VSRS to deliver packets; and The group will be delivered to the identified second instance.

Images