System and method for VLAN switching and routing services

Abstract

This invention provides methods, systems, and storage media to improve limitations that restrict the functionality and value of virtual networks. [Solution] The method generates a table for instances of a VLAN Switching and Routing Service (VSRS). The VSRS connects a first virtual Layer 2 network (VLAN) to a second network. The table contains information for identifying the IP addresses, MAC addresses, and virtual interface identifiers of instances within the virtual Layer 2 network. The method also uses the VSRS to receive packets being delivered from the first instance to the second instance within the virtual Layer 2 network, uses the VSRS to identify the second instance within the virtual Layer 2 network to which the packets should be delivered based on the information received with the packets and the information contained in the table, and delivers the packets to the identified second instance.

Description

Technical Field

[0001] Cross - Reference to Related Applications This application claims priority to the following applications: (1) U.S. Provisional Application No. 63 / 051728, filed on July 14, 2020, entitled "VLAN Switching and Routing Services and Layer 2 Virtual Networking in a Virtual Cloud Environment", and (2) U.S. Provisional Application No. 63 / 132377, filed on December 30, 2020, entitled "Layer 2 Virtual Networking in a Virtual Cloud Environment". All the content of the provisional applications mentioned above is incorporated herein by reference for all purposes.

[0002] This application is also related to U.S. Application No. 17 / 375999, filed on July 14, 2021, entitled "Virtual Layer 2 Network" (Attorney Docket No. 088325 - 1203134 - 276500US), and U.S. Application No. 17 / 376004, filed on July 14, 2021, entitled "Interface - Based ACL in a Layer 2 Network" (Attorney Docket No. 088325 - 1256549 - 276520US). All the content of each related application is incorporated herein by reference for all purposes.

Background Art

[0003] Background Cloud computing provides on - demand availability of computing resources. Cloud computing can be based on data centers that are accessible to users via the Internet. Cloud computing can provide IaaS (Infrastructure as a Service). Virtual networks may be created for user use. However, these virtual networks have limitations that restrict their functionality and value. Therefore, further improvements are desired. However, these virtual networks have limitations that restrict their functionality and value. Therefore, further improvements are desired.

Summary of the Invention

[0004] overview One aspect of this disclosure relates to a computer implementation method. This method includes providing a virtual Layer 3 network hosted by an underlying physical network in a virtualized cloud environment, and providing a virtual Layer 2 network hosted by an underlying physical network in a virtualized cloud environment.

[0005] In some embodiments, the virtual Layer 2 network may be a virtual local area network (VLAN). In some embodiments, the VLAN includes multiple endpoints. In some embodiments, the multiple endpoints may be multiple compute instances. In some embodiments, the VLAN includes multiple L2 virtual network interface cards (L2VNICs) and multiple switches.

[0006] In some embodiments, each of the multiple computing instances is commutably connected to a pair comprising a unique L2 virtual network interface card (L2VNIC) and a unique switch. In some embodiments, the multiple switches together can form a distributed switch. In some embodiments, each of the multiple switches routes outbound traffic according to a mapping table received from the L2VNIC paired with the switch. In some embodiments, mapping The table identifies the interface-to-MAC address mapping of endpoints within a VLAN.

[0007] In some embodiments, the method further includes instantiating a pair, which includes a unique L2VNIC and a unique switch, on a network virtualization device (NVD). In some embodiments, the method includes receiving a packet addressed to one of the multiple compute instances from another endpoint in a VLAN on the unique L2VNIC of one of the multiple compute instances, and learning the mapping of the other endpoint using the unique L2VNIC of one of the multiple compute instances. In some embodiments, the mapping of the other endpoint includes an interface-to-MAC address mapping of the other endpoint.

[0008] In some embodiments, the method includes decapsulating an incoming packet using a unique L2VNIC of one of a plurality of computing instances, and forwarding the decapsulated packet to one of the plurality of computing instances. In some embodiments, the method includes learning an IP address-to-MAC address mapping for another endpoint using one of the plurality of computing instances.

[0009] In some embodiments, the method includes sending an IP packet from a first compute instance in a VLAN to a second compute instance in a VLAN, including the destination IP address of a second compute instance in a VLAN; receiving the IP packet on a first L2VNIC associated with the first compute instance; encapsulating the IP packet on the first L2VNIC; and forwarding the IP packet to the second compute instance via a first switch. In some embodiments, the first switch and the first L2VNIC are communicateable to the first compute instance as a pair. In some embodiments, the method further includes receiving the IP packet on a second VNIC associated with the second compute instance; decapsulating the IP packet on the second VNIC; and forwarding the IP packet from the second VNIC to the second compute instance.

[0010] In some embodiments, the virtual Layer 2 network includes multiple virtual local area networks (VLANs). In some embodiments, each of the multiple VLANs includes multiple endpoints. In some embodiments, the multiple VLANs include a first VLAN and a second VLAN. In some embodiments, the first VLAN includes multiple first endpoints, and the second VLAN includes multiple second endpoints. In some embodiments, each of the multiple VLANs has a unique identifier. In some embodiments, one of the multiple first endpoints in the first VLAN communicates with one of the multiple second endpoints in the second VLAN.

[0011] One aspect of this disclosure relates to a system including a physical network. The physical network includes at least one host machine and at least one network virtualization device. The physical network can provide a virtual Layer 3 network hosted by the underlying physical network in a virtualized cloud environment, and can provide a virtual Layer 2 network hosted by the underlying physical network in a virtualized cloud environment.

[0012] One aspect of this disclosure relates to a non-temporary computer-readable storage medium that stores a plurality of instructions executable by one or more processors. When the plurality of instructions are executed by one or more processors, they cause one or more processors to provide a virtual Layer 3 network hosted by an underlying physical network in a virtualized cloud environment. In a UDO environment, a virtual Layer 2 network hosted by the underlying physical network is provided.

[0013] One aspect of the present disclosure relates to a method comprising generating a table for instances of a VLAN Switching and Routing Service (VSRS), the VSRS connecting a first virtual Layer 2 network to a second network. In some embodiments, the table includes information for identifying the IP addresses, MAC addresses, and virtual interface identifiers of instances within the first virtual Layer 2 network. The method includes using the VSRS to receive packets being delivered from a first instance to a second instance within the virtual Layer 2 network, using the VSRS to identify a second instance within the virtual Layer 2 network to which the packets should be delivered based on information received with the packets and information contained in the table, and delivering the packets to the identified second instance.

[0014] In some embodiments, the first virtual Layer 2 network includes multiple instances. In some embodiments, the first virtual Layer 2 network includes multiple L2 virtual network interface cards (L2VNICs) and multiple switches. In some embodiments, each of the multiple instances is communicably connected to a pair including its own L2 virtual network interface card (L2VNIC) and its own switch.

[0015] In some embodiments, using VSRS to identify a second instance in a first virtual Layer 2 network for packet delivery based on information received with the packet and information contained in a table includes using VSRS to determine that the table does not contain mapping information for the second instance, withholding packet delivery by VSRS, and using VSRS to broadcast an ARP request to a VNIC in the first virtual Layer 2 network, the ARP request including the IP address of the second instance, and using VSRS to receive an ARP response from the L2VNIC of the second instance.

[0016] In some embodiments, the method further includes updating a table based on received ARP responses. In some embodiments, the first instance is located outside a first virtual Layer 2 network and inside a second network. In some embodiments, the second network may be an L3 network. In some embodiments, the second network may be a second virtual Layer 2 network. In some embodiments, the table is generated based on information received by the VSRS.

[0017] In some embodiments, the method includes instantiating VSRS as a service on multiple hardware nodes. In some embodiments, the method includes distributing tables among the hardware nodes. In some embodiments, the tables distributed among the hardware nodes are available through another VSRS instantiation. In some embodiments, the first instance resides within a first virtual Layer 2 network.

[0018] In some embodiments, the method includes receiving packets from a third instance within a first virtual Layer 2 network using VSRS. In some embodiments, the packets are delivered to a fourth instance outside the first virtual Layer 2 network. In some embodiments, the method includes receiving packets from a third instance within a first virtual Layer 2 network using VSRS. In one embodiment, the packet is delivered to a service used by a third instance within a first virtual Layer 2 network. In some embodiments, the service may be at least one of DHCP, NTP, and DNS.

[0019] In some embodiments, the method includes using VSRS to receive packets from a third instance in a first virtual Layer 2 network. In some embodiments, the packets are delivered to a fourth instance in a second virtual Layer 2 network. In some embodiments, the method further includes distributing tables for instances of VSRS having Layer 2 and Layer 3 network information across multiple service nodes to provide highly reliable and scalable VSRS instantiation. In some embodiments, the method includes using VSRS to receive packets from a third instance in a first virtual Layer 2 network and using VSRS to learn the mapping of the third instance.

[0020] One aspect of the present disclosure relates to a system. The system includes a physical network. The physical network includes at least one processor and a network virtualization device. At least one processor can instantiate an instance of a VLAN Switching and Routing Service (VSRS), which connects a first virtual Layer 2 network to a second network, and at least one processor can generate a table for the VSRS instance. In some embodiments, the table includes information for identifying the IP addresses, MAC addresses, and virtual interface identifiers of the instance in the first virtual Layer 2 network. At least one processor can use the VSRS to receive packets being delivered from a first instance to a second instance in the virtual Layer 2 network, and can use the VSRS to identify a second instance in the first virtual Layer 2 network for packet delivery based on the information received with the packet and the information contained in the table, and deliver the packet to the identified second instance.

[0021] One aspect of the present disclosure relates to a computer-readable storage medium for storing a plurality of instructions that can be executed by one or more processors. When the plurality of instructions are executed by one or more processors, they cause one or more processors to instantiate an instance of a VLAN Switching and Routing Service (VSRS), the VSRS to connect a first virtual Layer 2 network to a second network, and to generate a table for the VSRS instance. In some embodiments, the table includes information for identifying the IP addresses, MAC addresses, and virtual interface identifiers of the instances in the first virtual Layer 2 network. When the plurality of instructions are executed by one or more processors, they cause one or more processors to use the VSRS to receive packets to be delivered from a first instance to a second instance in the virtual Layer 2 network, to identify a second instance in the virtual Layer 2 network to deliver the packets to based on the information received with the packets and the information contained in the table, and to deliver the packets to the identified second instance.

[0022] One aspect of this disclosure relates to a method. This method involves sending a packet from a source compute instance in a virtual network to a destination compute instance via a destination L2 virtual network interface card (destination L2VNIC) in a first virtual Layer 2 network; evaluating the access control list (ACL) of the packet using the source virtual network interface card (source VNIC); embedding the ACL information associated with the packet into the packet; forwarding the encapsulated packet into a virtual switching and routing service (VSRS) for connecting the first virtual Layer 2 network (VLAN) to a second network; and using the VSRS, together with the packet Identifying a destination L2VNIC in a first virtual layer 2 network for delivering a packet based on the received information and the mapping information included in the mapping table, obtaining ACL information from the packet using VSRS, and applying the obtained ACL information to the packet.

[0023] In some embodiments, the packet includes an IP packet. In some embodiments, the source computing instance is located in a virtual L3 network. In some embodiments, the source computing instance is located in a second virtual layer 2 network.

[0024] In some embodiments, the method includes encapsulating the packet using a source VNIC. In some embodiments, the method includes receiving and decapsulating the packet using VSRS. In some embodiments, identifying a destination L2VNIC in a first virtual layer 2 network for delivering a packet based on the information received with the packet and the mapping information included in the mapping table using VSRS includes determining using VSRS that the mapping table does not include mapping information of the destination computing instance, holding the transfer of the packet using VSRS, broadcasting an ARP request including the IP address of the destination computing instance to an L2VNIC in the first virtual layer 2 network using VSRS, and receiving an ARP response from the L2VNIC of the destination computing instance using VSRS. In some embodiments, one of the L2VNICs is the L2VNIC of the destination computing instance.

[0025] In some embodiments, the method includes updating a table based on an received ARP response. In some embodiments, using VSRS to identify a destination L2VNIC in a first virtual Layer 2 network for delivering a packet, based on information received with the packet and mapping information contained in a mapping table, includes determining that the mapping table contains mapping information for a destination compute instance and identifying the destination L2VNIC based on the mapping information contained in the mapping table. In some embodiments, embedding ACL information related to a packet into the packet includes storing the ACL information in the packet as metadata. In some embodiments, obtaining ACL information from a packet using VSRS includes extracting metadata containing the ACL information from the packet.

[0026] In some embodiments, applying the acquired ACL information to a packet includes determining that the ACL information is not associated with the destination L2VNIC. In some embodiments, applying the acquired ACL information to a packet further includes forwarding the packet to the destination compute instance via the destination L2VNIC. In some embodiments, applying the acquired ACL information to a packet includes determining, using VSRS, that the ACL information is associated with the destination L2VNIC. In some embodiments, applying the acquired ACL information to a packet further includes determining, using VSRS, that the destination L2VNIC conforms to the ACL information, and forwarding the packet to the destination compute instance via the destination L2VNIC using VSRS.

[0027] In some embodiments, applying the acquired ACL information to a packet includes using VSRS to determine that the destination L2VNIC does not comply with the ACL information, and the VSRS discards the packet. In some embodiments, applying the acquired ACL information to a packet further includes using VSRS to send a response indicating packet discard to the source compute instance.

[0028] One aspect of the present disclosure relates to a system including a physical network. The physical network includes at least one first processor, a network virtualization device, and at least one second processor. At least one processor can send a packet from a source computing instance in a virtual network instantiated on the physical network to a destination computing instance via a destination Layer 2 virtual network interface card (destination L2VNIC) in a first virtual Layer 2 network instantiated on the physical network. The network virtualization device can instantiate a source VNIC. The source VNIC can evaluate an access control list (ACL) of the packet, embed ACL information related to the packet in the packet, and transfer the packet to virtual switching and routing services (VSRS), and the VSRS connects a first virtual Layer 2 network (VLAN) to a second network. At least one second processor can instantiate VSRS. VSRS can identify a destination L2VNIC for delivering the packet based on the information received with the packet and the mapping information included in the mapping table, obtain ACL information from the packet, and apply the obtained ACL information to the packet.

[0029] In some embodiments, applying the obtained ACL information to the packet includes determining that the ACL information is related to the destination L2VNIC, determining that the destination L2VNIC complies with the ACL information, and using VSRS to transfer the packet to the destination computing instance via the destination L2VNIC.

[0030] One aspect of the present disclosure relates to a computer-readable storage medium for storing a plurality of instructions that can be executed by one or more processors. When the plurality of instructions are executed by one or more processors, they cause one or more processors to send a packet from a source compute instance in a virtual network to a destination compute instance in a virtual network via a destination L2 virtual network interface card (destination L2VNIC) in a first virtual Layer 2 network; to use the source virtual network interface card (source VNIC) to evaluate the access control list (ACL) of the packet; to embed the ACL information associated with the packet into the packet; to forward the packet to a virtual switching and routing service (VSRS) for connecting the first virtual Layer 2 network (VLAN) to a second network; to use the VSRS to identify a destination L2VNIC in the first virtual Layer 2 network for delivering the packet based on information received with the packet and mapping information contained in a mapping table; to use the VSRS to obtain the ACL information from the packet; and to apply the obtained ACL information to the packet. [Brief explanation of the drawing]

[0031] [Figure 1] This is a high-level diagram of a distributed environment showing a virtual or overlay cloud network hosted by a cloud service provider infrastructure, according to a specific embodiment. [Figure 2] This is an architectural schematic diagram showing the physical elements of the physical network within the CSPI according to a specific embodiment. [Figure 3] This figure shows an exemplary configuration of a CSPI in which a host machine is connected to multiple network virtualization devices (NVDs) according to a particular embodiment. [Figure 4] This diagram shows the connection between a host machine and an NVD that provides I / O virtualization to support multi-tenancy functionality, according to a specific embodiment. [Figure 5]This is a schematic block diagram showing the physical network provided by CSPI according to a specific embodiment. [Figure 6] This is a schematic diagram illustrating one embodiment of a computing network. [Figure 7] This is a schematic diagram of the logic and hardware of a virtual local area network (VLAN). [Figure 8] This is a logical schematic diagram showing multiple connected L2VLANs. [Figure 9] This is a logical schematic diagram showing multiple connected L2VLANs and subnets. [Figure 10] This is a schematic diagram illustrating one embodiment of intra-VLAN communication and learning within a VLAN. [Figure 11] This is a schematic diagram showing one embodiment of VLAN implementation. [Figure 12] This is a flowchart showing one embodiment of the process for performing communication within a VLAN. [Figure 13] This is a schematic diagram illustrating the process for performing communication within a VLAN. [Figure 14] This is a flowchart illustrating one embodiment of the process for performing inter-VLAN communication within a virtual L2 network. [Figure 15] This is a schematic diagram illustrating the process for performing inter-VLAN communication. [Figure 16] This is a flowchart illustrating one embodiment of the process for performing a received packet flow. [Figure 17] This is a schematic diagram illustrating the process for receiving communications. [Figure 18] This is a flowchart showing one embodiment of a process for performing outgoing packet flow from a VLAN. [Figure 19] This is a schematic diagram illustrating the process for performing a packet transmission flow. [Figure 20]This is a flowchart illustrating one embodiment of the process for performing delayed access control list (ACL) classification. [Figure 21] This is a flowchart illustrating one embodiment of the process for performing early classification of ACLs. [Figure 22] This is a flowchart illustrating one embodiment of a process for performing sender-based next-hop routing. [Figure 23] This is a flowchart illustrating one embodiment of the process for performing delayed next-hop routing. [Figure 24] This block diagram shows one pattern for implementing a cloud infrastructure as a service system, according to at least one embodiment. [Figure 25] This block diagram shows another pattern for implementing cloud infrastructure as a service system, according to at least one embodiment. [Figure 26] This block diagram shows another pattern for implementing cloud infrastructure as a service system, according to at least one embodiment. [Figure 27] This block diagram shows another pattern for implementing cloud infrastructure as a service system, according to at least one embodiment. [Figure 28] A block diagram showing an exemplary computer system according to at least one embodiment. [Modes for carrying out the invention]

[0032] Detailed explanation In the following description, certain details are included for illustrative purposes to provide a complete understanding of a particular embodiment. However, it will be apparent that various embodiments may be carried out without these specific details. The drawings and description are not intended to be limiting. The term “exemplary” is used in this disclosure to mean “serving as an example, illustration, or depiction.” Any embodiment or design described as “exemplary” in this disclosure should not necessarily be construed as being preferable or advantageous over other embodiments or designs.

[0033] Examples of virtual network architectures The term "cloud service" generally refers to a service provided by a cloud service provider (CSP) using its systems and infrastructure (cloud infrastructure) to be available to users or customers on demand (e.g., via a subscription model). This refers to a service that enables customers to access and utilize cloud services. Typically, the servers and systems that make up the CSP's infrastructure are separate from the customer's own on-premises servers and systems. Therefore, customers can use cloud services provided by the CSP without having to purchase hardware and software resources separately for the service. Cloud services are designed to provide subscribers with easy and scalable access to applications and computing resources without requiring them to invest in procuring the infrastructure used to provide the service.

[0034] Several cloud service providers (CSPs) offer various types of cloud services. Cloud services include various different types or models such as SaaS (Software-as-a-Service), PaaS (Platform-as-a-Service), and IaaS (Infrastructure-as-a-Service).

[0035] A customer can subscribe to one or more cloud services provided by a CSP. A customer may be any entity, such as an individual, organization, or company. When a customer subscribes to or registers for a service provided by a CSP, a tenant or account is created for that customer. The customer can then access one or more subscribed cloud resources associated with this account.

[0036] As mentioned above, IaaS (Infrastructure as a Service) is a single specific type of This is a cloud computing service. In the IaaS model, the CSP provides infrastructure (called Cloud Service Provider Infrastructure or CSPI) that customers can use to build their own customizable networks and deploy their customer resources. Therefore, the customer's resources and network are hosted in a distributed environment by the infrastructure provided by the CSP. This differs from traditional computing where the customer's infrastructure hosts the customer's resources and network.

[0037] A CSPI may include interconnected high-performance computing resources, including various host machines, memory resources, and network resources, forming a physical network also known as an underlay network. The resources of a CSPI may be distributed across one or more data centers geographically distributed across one or more geographical regions. Virtualization software can run on these physical resources to provide a virtualized distributed environment. Virtualization creates an overlay network (also known as a software-based network, software-defined network, or virtual network) on top of a physical network. The CSPI physical network provides the foundation for creating one or more overlay or virtual networks on top of the physical network. The physical network (or underlay network) includes physical network devices such as physical switches, routers, computers, and host machines. An overlay network is a logical (or virtual) network operating on top of the physical underlay network. A given physical network can support one or more overlay networks. Overlay networks typically use encapsulation techniques to distinguish traffic belonging to different overlay networks. A virtual network or overlay network is also known as a virtual cloud network (VCN). Virtual networks are implemented using software virtualization technologies (e.g., hypervisors, virtualization functions implemented by network virtualization devices (NVDs) (e.g., smart NICs), top-of-rack (TOR) switches, smart TORs that implement one or more functions performed by NVDs, and other mechanisms) to create a network abstraction layer that can run on top of a physical network. Virtual networks can take various forms, such as peer-to-peer networks and IP networks. A virtual network is typically either a Layer 3 IP network or a Layer 2 VLAN. Such virtual or overlay networks are often called virtual Layer 3 networks or overlay Layer 3 networks. Examples of protocols developed for virtual networks include IP-in-IP (or GRE (Generic Routing Encapsulation)) and Virtual Extensible LAN (VXLAN-IETF). This includes RFC7348), virtual private networks (VPNs) (e.g., MPLS Layer 3 virtual private network (RFC4364)), VMware NSX, GENEVE (Generic Network Virtualization Encapsulation), etc.

[0038] In the case of IaaS, the infrastructure provided by the CSP (CSPI) may be configured to deliver virtualized computing resources over a public network (e.g., the internet). In the IaaS model, the cloud computing service provider can host infrastructure elements (e.g., servers, storage, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer)). In some cases, the IaaS provider can provide various services associated with these infrastructure elements (e.g., billing, monitoring, logging, security, load balancing, and clustering). Because these services are policy-driven, IaaS users can maintain application availability and performance by implementing policies to drive load balancing. The CSPI provides infrastructure and a set of complementary cloud services. This allows customers to build and run a wide range of applications and services in a highly available hosted distributed environment. The CSPI provides high-performance computing resources and capabilities, as well as storage capacity, on a flexible virtual network that can be securely accessed from various network locations, such as the customer's on-premises network. When a customer subscribes to or registers for an IaaS service provided by a CSP, the tenancy created for that customer becomes a securely isolated partition from the CSP, allowing the customer to create, organize, and manage cloud resources.

[0039] Customers can build their own virtual networks using the computing, memory, and networking resources provided by CSPI. On these virtual networks, one or more customer resources or workspaces, such as compute instances, can be placed. Loads can be deployed. For example, a customer can use resources provided by CSPI to build one or more customizable private virtual networks called Virtual Cloud Networks (VCNs). On a customer VCN, a customer can deploy one or more customer resources, such as compute instances. Compute instances may be virtual machines, bare metal instances, etc. Thus, CSPI provides the infrastructure and a set of complementary cloud services that enable customers to build and run various applications and services in a highly available virtual host environment. Customers do not manage or control the underlying physical resources provided by CSPI, but they do control the operating system, storage, and deployed applications, and in some cases have limited control over certain networking components (e.g., firewalls).

[0040] A CSP can provide a console that enables customers and network administrators to configure, access, and manage resources deployed to the cloud using CSPI resources. In certain embodiments, the console provides a web-based user interface that can be used to utilize and manage CSPI. In some embodiments, the console is a web-based application provided by the CSP.

[0041] CSPI can support single-tenancy or multi-tenancy architectures. In a single-tenancy architecture, software (e.g., applications, databases) or hardware elements (e.g., host machines or servers) serve a single customer or tenant. In a multi-tenancy architecture, software or hardware elements serve multiple customers or tenants. Therefore, in a multi-tenancy architecture, CSPI resources are shared among multiple customers or tenants. In a multi-tenancy environment, CSPI implements precautions and safeguards to isolate each tenant's data and prevent it from being visible to other tenants.

[0042] In a physical network, a network endpoint (endpoint) refers to a computing device or system that is connected to the physical network and communicates bidirectionally with the connected network. A network endpoint in a physical network may be connected to a local area network (LAN), a wide area network (WAN), or other types of physical networks. Examples of traditional endpoints in a physical network include modems, hubs, bridges, switches, routers, and other networking devices, as well as physical computers (or host machines). Each physical device in a physical network has a fixed network address that can be used to communicate with that device. This fixed network address may be a Layer 2 address (e.g., a MAC address), a fixed Layer 3 address (e.g., an IP address), etc. In a virtualized environment or virtual network, endpoints can include various virtual endpoints, such as virtual machines hosted by elements of the physical network (e.g., hosted by a physical host machine). These endpoints in a virtual network are addressed by overlay addresses, such as overlay Layer 2 addresses (e.g., overlay MAC addresses) and overlay Layer 3 addresses (e.g., overlay IP addresses). Network overlays provide flexibility by allowing network administrators to move overlay addresses associated with network endpoints using software management (e.g., through software implementing the control plane of the virtual network). Therefore, unlike physical networks, in virtual networks, network management software can be used to move overlay addresses (e.g., overlay IP addresses) from one endpoint to another. Because virtual networks are built on top of physical networks, both the virtual network and the underlying physical network are involved in communication between elements of the virtual network.To facilitate such communication, each element of the CSPI is configured to learn and store mappings that map the overlay address of the virtual network to the actual physical address of the underlying network, or vice versa. These mappings are used to facilitate communication. To facilitate routing within the virtual network, customer traffic is encapsulated.

[0043] Therefore, physical addresses (e.g., physical IP addresses) are associated with elements of a physical network, while overlay addresses (e.g., overlay IP addresses) are associated with entities in a virtual network. A physical IP address is an IP address associated with a physical device (e.g., a network device) within the underlying network or physical network. For example, each NVD has an associated physical IP address. An overlay IP address is an overlay address associated with an entity within the overlay network, such as a compute instance within a customer's virtual cloud network (VCN). Two different customers or tenants, each with their own private VCN, can potentially use the same overlay IP address in their VCNs without knowing each other. Both physical IP addresses and overlay IP addresses are real IP addresses. These are different from virtual IP addresses. A virtual IP address is typically a single IP address that represents or maps to multiple real IP addresses. A virtual IP address provides a one-to-many mapping between a virtual IP address and multiple real IP addresses. For example, a load balancer can use a VIP to map to or represent multiple servers, each of which has its own real IP address.

[0044] The cloud infrastructure or CSPI is physically hosted in one or more data centers in one or more regions worldwide. The CSPI may include elements of a physical network or underlayment network and virtualization elements of a virtual network built on the physical network elements (e.g., virtual networks, compute instances, virtual machines). In certain embodiments, the CSPI is realm, region, and CSPI resources are organized and hosted within available domains. A region is typically a local geographic area containing one or more data centers. Regions are generally independent of each other and may be separated by vast distances, for example, across countries or continents. For example, one region may be in Australia, another in Japan, and yet another in India. CSPI resources are divided among these regions such that each region has an independent subset of CSPI resources. Each region may provide a set of core infrastructure services and resources, such as computing resources (e.g., bare metal servers, virtual machines, containers and related infrastructure), storage resources (e.g., block volume storage, file storage, object storage, archive storage), networking resources (e.g., virtual cloud networks (VCNs), load balancing resources, connectivity to on-premises networks), database resources, edge networking resources (e.g., DNS), access management, and monitoring resources. Each region generally has multiple paths to connect it to other regions within the realm.

[0045] Generally, applications are deployed in the region where they are most frequently used (i.e., on infrastructure relevant to that region) because using nearby resources is faster than using distant resources. Applications may also be deployed in different regions for various reasons, such as redundancy to mitigate the risks of large-scale weather systems or region-wide events like earthquakes, or redundancy to meet various requirements for legal jurisdictions, tax domains, and other business or social standards.

[0046] Data centers within a region may be further organized and subdivided into availability domains (ADs). An availability domain is one or more domains located within a given region. The above data centers may also be supported. A region may consist of one or more available domains. In such a distributed environment, CSPI resources may be region-specific, such as virtual cloud networks (VCNs), or domain-specific, such as compute instances.

[0047] ADs within a single region are configured to be fault-tolerant, isolated from one another, and configured to be extremely unlikely to fail simultaneously. This is achieved by configuring ADs so that a failure in one AD within a region has little impact on the availability of other ADs within the same region, by not sharing critical infrastructure resources such as networking, physical cabling, cabling routes, and cabling entry points. By connecting ADs within the same region with a low-latency, high-bandwidth network, highly available connectivity to other networks (e.g., the internet, customer on-premises networks) can be provided, and a replication system for both high availability and disaster recovery can be built across multiple ADs. Cloud services utilize multiple ADs to achieve high availability. This ensures security and protects against resource failures. As the infrastructure provided by the IaaS provider grows, more regions and ADs may be added along with additional capacity. Traffic between available domains is typically encrypted.

[0048] In certain embodiments, regions are grouped into realms. A realm is a logical set of regions. Realms are isolated from each other and do not share any data. Regions within the same realm can communicate with each other, but regions in different realms cannot. A CSP customer's tenancy or account may reside in a single realm and span one or more regions belonging to that single realm. Typically, when a customer subscribes to an IaaS service, their tenancy or account is created in a customer-designated region within a realm (referred to as the "home" region). The customer can extend their tenancy to one or more other regions within a realm. A customer cannot access regions that do not reside in the realm in which their tenancy resides.

[0049] An IaaS provider can offer multiple realms, each corresponding to a specific set of customers or users. For example, a commercial realm may be offered for commercial customers. Another example is that a realm may be offered for a specific country or for customers in that country. Yet another example is that a government realm may be offered for a government, for example. For example, a government realm may be created for a specific government and may have a higher security level than a commercial realm. For example, Oracle Cloud Infrastructure (OCI) currently offers a realm for the commercial domain and two realms for the government cloud domain (e.g., FedRAMP accreditation and IL5 accreditation).

[0050] In certain embodiments, an Active Directory (AD) can be subdivided into one or more fault domains. A fault domain is a grouping of infrastructure resources within the AD to provide anti-affinity. Fault domains can distribute compute instances so that they are not located on the same physical hardware within a single AD. This is known as anti-affinity. A fault domain refers to a collection of hardware elements (computers, switches, etc.) that share a single point of failure. The compute pool is logically divided into fault domains. Therefore, a hardware failure or compute hardware maintenance event affecting one fault domain does not affect instances in other fault domains. Depending on the embodiment, the number of fault domains in each AD may vary. For example, in certain embodiments, each AD may contain three fault domains. Fault domains function as logical data centers within the AD.

[0051] When a customer subscribes to an IaaS service, resources from CSPI are provisioned to the customer and associated with the customer's tenancy. The customer can use these provisioned resources to build private networks and deploy resources on these networks. Customer networks hosted on the cloud by CSPI are called Virtual Cloud Networks (VCNs). A customer can configure one or more Virtual Cloud Networks (VCNs) using CSPI resources allocated to them. A VCN is a virtual or software-defined private network. Customer resources deployed in a customer's VCN can include compute instances (e.g., virtual machines, bare metal instances) and other resources. These compute instances may represent various customer workloads such as applications, load balancers, and databases. Compute instances deployed on a VCN can communicate with publicly accessible endpoints (public endpoints) over public networks such as the internet, communicate with other instances within the same VCN or other VCNs (e.g., other VCNs of the customer, or VCNs not belonging to the customer), and communicate with the customer's on-premises data center or network. It can communicate with service endpoints and other types of endpoints.

[0052] A CSP can provide a variety of services using a CSPI. In some cases, a CSPI customer can act like a service provider themselves and provide services using CSPI resources. A service provider can expose service endpoints characterized by identifying information (e.g., IP address, DNS name, and port). A customer's resources (e.g., compute instances) can consume a specific service by accessing the service endpoint of that particular service exposed by the service. These service endpoints are generally publicly accessible endpoints that users can access via public communication networks such as the internet using the public IP address associated with the endpoint. Publicly accessible network endpoints are sometimes called public endpoints.

[0053] In certain embodiments, a service provider may expose a service through an endpoint of the service (sometimes called a service endpoint). Customers of the service can access the service using this service endpoint. In certain embodiments, the service endpoint provided for a service may be accessible to multiple customers who wish to consume that service. In other implementations, a dedicated service endpoint may be provided to a customer. Thus, only that customer can access the service using that dedicated service endpoint.

[0054] In certain embodiments, the VCN, once created, is assigned a private overlay IP address range (e.g., 10.0 / 16) to that VCN for private overlay classless inter-domain routing. A VCN is associated with a CIDR address space. A VCN includes associated subnets, route tables, and gateways. While a VCN resides within a single region, it can extend to one or more or all available domains within a region. A gateway is a virtual interface configured for a VCN, enabling traffic communication between the VCN and one or more endpoints outside the VCN. By configuring one or more different types of gateways for a VCN, communication between different types of endpoints can be enabled.

[0055] A VCN may be subdivided into one or more subnets, such as one or more subnets. Thus, a subnet is a constituent unit or partition that can be created within a VCN. A VCN can have one or more subnets. Each subnet within a VCN does not overlap with other subnets within that VCN and is associated with a contiguous range of overlay IP addresses (e.g., 10.0.0.0 / 24 and 10.0.1.0 / 24) that represent an address space subset of the VCN's address space.

[0056] Each compute instance is associated with a virtual network interface card (VNIC). This allows each compute instance to participate in a subnet of the VCN. A VNIC is a logical representation of a physical network interface card (NIC). Generally, a VNIC is the interface between an entity (e.g., compute instance, service) and a virtual network. A VNIC resides in a subnet and has one or more associated IP addresses and associated security rules or policies. A VNIC is equivalent to a Layer 2 port on a switch. A VNIC connects a compute instance to a subnet within the VCN. It is associated with a compute instance. A VNIC enables a compute instance to be part of a VCN subnet and allows the compute instance to communicate (e.g., send and receive packets) with endpoints on the same subnet as the compute instance, endpoints in different subnets within the VCN, or endpoints outside the VCN. Therefore, the VNIC associated with a compute instance determines how the compute instance connects to internal and external endpoints within the VCN. A compute instance's VNIC is created and associated with that compute instance when the compute instance is created and added to a subnet within the VCN. If a subnet consists of a set of compute instances, it contains a VNIC corresponding to that set of compute instances, and each VNIC connects to a compute instance within a set of computer instances.

[0057] Each compute instance is assigned a private overlay IP address via the VNIC associated with it. This private overlay IP address is assigned to the VNIC associated with the compute instance when the compute instance is created and is used to route the compute instance's traffic. All VNICs within a given subnet use the same route table, security lists, and DHCP options. As mentioned above, each subnet within a VCN does not overlap with other subnets within that VCN and is associated with a contiguous range of overlay IP addresses (e.g., 10.0.0.0 / 24 and 10.0.1.0 / 24) that represents a subset of the address space of that VCN. For a VNIC on a particular subnet of a VCN, the overlay IP address assigned to the VNIC is an address from the contiguous range of overlay IP addresses assigned to the subnet.

[0058] In certain embodiments, a compute instance may be assigned additional overlay IP addresses, such as one or more public IP addresses in the case of a public subnet, in addition to its private overlay IP address, as needed. These multiple addresses may be assigned to the same VNIC or to multiple VNICs associated with the compute instance. However, each instance has a primary VNIC that is created at instance launch and associated with the overlay private IP address assigned to the instance. This primary VNIC cannot be deleted. Additional VNICs, called secondary VNICs, can be added to an existing instance in the same available domain as the primary VNIC. All VNICs are in the same available domain as the instance. Secondary VNICs may be in the same VCN subnet as the primary VNIC, or they may be in the same VCN or different VCN subnets.

[0059] Compute instances can optionally be assigned a public IP address if they are located in a public subnet. When creating a subnet, you can specify that subnet as either a public or private subnet. A private subnet means that resources within that subnet (e.g., compute instances) and associated VNICs cannot have public overlay IP addresses. A public subnet means that resources within that subnet and associated VNICs can have public IP addresses. Customers can specify subnets that exist across a single available domain or multiple available domains within a region or realm.

[0060] As described above, a VCN may be subdivided into one or more subnets. In certain embodiments, a virtual router configured for the VCN (referred to as a VCN VR or simply a VR) enables communication between subnets of the VCN. For subnets within a VCN, the VR communicates between the subnet (i.e., compute instances on that subnet) and within the VCN. This represents the logical gateway of a subnet that enables communication with endpoints on other subnets and other endpoints outside the VCN. A VCN VR is a logical entity configured to route traffic between VNICs within a VCN and virtual gateways (gateways) associated with the VCN. Gateways are described further below with respect to Figure 1. A VCN VR is a Layer 3 / IP layer concept. In one embodiment, there is one VCN VR for one VCN. This VCN VR has a potentially unlimited number of ports addressed by IP addresses, with one port for each subnet of the VCN. In this way, the VCN VR has a different IP address for each subnet of the VCN to which the VCN VR is connected. The VR is also connected to various gateways configured for the VCN. In a particular embodiment, a specific overlay IP address from the overlay IP address range of a subnet is reserved for a port on the VCN VR of that subnet. For example, consider a VCN having two subnets, each having the associated address ranges 10.0 / 16 and 10.1 / 16, respectively. For a first subnet of a VCN with the address range 10.0 / 16, addresses from this range are held on the VCN VR ports for that subnet. In some cases, the first IP address from this range may also be held on the VCN VR. For example, for a subnet with the overlay IP address range 10.0 / 16, the IP address 10.0.0.1 may be held on the VCN VR ports for that subnet. For a second subnet within the same VCN with the address range 10.1 / 16, the VCN VR may have a port for the second subnet with the IP address 10.1.0.1. The VCN VR has a different IP address for each subnet within the VCN.

[0061] In some other embodiments, each subnet within a VCN may have its own associated VR, which is addressable by the subnet using a reserved or default IP address associated with the VR. The reserved or default IP address may be, for example, a first IP address from a range of IP addresses associated with that subnet. A VNIC within a subnet can use this default or reserved IP address to communicate with the VR associated with the subnet (e.g., send and receive packets). In such embodiments, the VR is the incoming / outgoing point for that subnet. A VR associated with a subnet within a VCN can communicate with other VRs associated with other subnets within the VCN. A VR can also communicate with gateways associated with the VCN. The VR functionality of a subnet is performed on, or by, one or more NVDs that perform the VNIC functionality of the VNICs within the subnet.

[0062] Route tables, security rules, and DHCP options may be configured for the VCN. The route table is the VCN's virtual route table and contains rules for routing traffic from subnets within the VCN to destinations outside the VCN, via a gateway or specially configured instance. The VCN's route table can be customized to control packet forwarding / routing to and from the VCN. This is possible. DHCP options refer to configuration information that is automatically provided to an instance when the instance starts up.

[0063] Security rules configured for a VCN represent the VCN's overlay firewall rules. Security rules can include inbound and outbound rules and can specify the types of traffic allowed to enter and leave instances within the VCN (e.g., based on protocol and port). Customers can choose whether certain rules are stateful or stateless. For example, a customer can define a stateful inbound rule with source CIDR 0.0.0.0 / 0 and destination TCP port 22. By configuring rules, you can allow incoming SSH traffic to a pair of instances from any location. Security rules may be implemented using network security groups or security lists. A network security group consists of a set of security rules that apply only to resources within that group. A security list, on the other hand, contains rules that apply to all resources in a subnet that uses that security list. A VCN may include default security rules and a default security list. DHCP options configured for a VCN provide configuration information that is automatically provided when instances in the VCN start up.

[0064] In certain embodiments, VCN configuration information is determined and stored by the VCN control plane. VCN configuration information may include, for example, address ranges associated with the VCN, subnets and associated information within the VCN, one or more VRs associated with the VCN, compute instances and associated VNICs within the VCN, NVDs that perform various virtualized network functions associated with the VCN (e.g., VNICs, VRs, gateways), VCN status information, and other VCN-related information. In certain embodiments, the VCN distribution service exposes the configuration information or a portion thereof stored by the VCN control plane to the NVD. Using the distributed information, packets can be forwarded to compute instances within the VCN by updating information stored and used by the NVD (e.g., forwarding tables, routing tables, etc.).

[0065] In certain embodiments, the creation of VCNs and subnets is handled by the VCN control plane (CP), and the startup of compute instances is handled by the compute control plane. The compute control plane is configured to allocate physical resources for compute instances and then call the VCN control plane to create VNICs and connect to the compute instances. The VCN CP also sends VCN data mappings to the VCN data plane, which is configured to perform packet forwarding and routing functions. In certain embodiments, the VCN CP provides a distribution service configured to provide updates to the VCN data plane. Examples of VCN control planes are shown in Figures 24, 25, 26, and 27 (see reference numbers 2416, 2516, 2616, and 2716) and are described below.

[0066] Customers can create one or more VCNs using resources hosted by CSPI. Compute instances deployed on a customer VCN can communicate with different endpoints. These endpoints can include endpoints hosted by CSPI and endpoints outside of CSPL.

[0067] Various different architectures for implementing cloud-based services using CSPI are shown in Figures 1, 2, 3, 4, 5, 24, 25, 26, and 28, and are described below. Figure 1 is a high-level diagram of a distributed environment 100 showing an overlay VCN or customer VCN hosted by CSPI according to a particular embodiment. The distributed environment shown in Figure 1 includes multiple elements within the overlay network. The distributed environment 100 shown in Figure 1 is merely an example and is not intended to unduly limit the scope of the claimed embodiments. Many variations, alternatives, and modifications are possible. For example, in some implementations, the distributed environment shown in Figure 1 may have more or fewer systems or elements than those shown in Figure 1, may combine two or more systems, or may have different system configurations or arrangements.

[0068] As shown in the example in Figure 1, the distributed environment 100 provides services and resources that customers can subscribe to and use to build a virtual cloud network (VCN). This includes CSPI101. In a particular embodiment, CSPI101 provides IaaS services to subscriber customers. The data centers within CSPI101 may be organized into one or more regions. Figure 1 shows an example of a region, “US Region” 102. A customer has configured a customer VCN 104 for region 102. The customer can deploy various compute instances on VCN 104, which may include virtual machines or bare metal instances. Examples of instances include applications, databases, load balancers, etc.

[0069] In the embodiment shown in Figure 1, customer VCN 104 includes two subnets, namely "Subnet-1" and "Subnet-2," each subnet having its own CIDR IP address range. In Figure 1, the overlay IP address range for Subnet-1 is 10.0 / 16, and the address range for Subnet-2 is 10.1 / 16. The VCN virtual router 105 represents the logical gateway of the VCN, enabling communication between subnets of VCN 104 and communication with other endpoints outside the VCN. VR105 is configured to route traffic between VNICs within VCN104 and gateways associated with VCN104. VCN VR105 provides ports to each subnet of VCN104. For example, VR105 can provide a port with IP address 10.0.0.1 to subnet-1 and a port with IP address 10.1.0.1 to subnet-2.

[0070] Multiple compute instances can be deployed on each subnet. In this case, compute instances may be virtual machine instances and / or bare metal instances. Compute instances within a subnet may be hosted by one or more host machines within CSPI101. Compute instances join the subnet via the VNIC associated with them. For example, as shown in Figure 1, compute instance C1 is part of subnet-1 via the VNIC associated with it. Similarly, compute instance C2 is part of subnet-1 via the VNIC associated with C2. Similarly, multiple compute instances, which may be virtual machine instances or bare metal instances, may be part of subnet-1. Each compute instance is assigned a private overlay IP address and a media access control address (MAC address) via the associated VNIC. For example, in Figure 1, compute instance C1 has the overlay IP address 10.0.0.2 and MAC address M1, and compute instance C2 has the private overlay IP address 10.0.0.3 and MAC address M2. Each compute instance in subnet-1, including compute instances C1 and C2, has a default route to VCN VR105 using IP address 10.0.0.1, which is the IP address of the port of VCN VR105 in subnet-1.

[0071] Multiple compute instances, including virtual machine instances and / or bare metal instances, can be deployed in subnet-2. For example, as shown in Figure 1, compute instances Dl and D2 are part of subnet-2 via the VNIC associated with each compute instance. In the embodiment shown in Figure 1, compute instance D1 has the overlay IP address 10.1.0.2 and MAC address MM1, and compute instance D2 has the private overlay IP address 10.1.0.3 and MAC address MM2. Each compute instance in subnet-2, including compute instances D1 and D2, has a default route to VCN VR105 using IP address 10.1.0.1, which is the IP address of the port of VCN VR105 in subnet-2.

[0072] Furthermore, VCN A104 may include one or more load balancers. For example, A load balancer may be provided for a subnet and configured to load balance traffic among multiple compute instances on that subnet. Alternatively, a load balancer may be provided to load balance traffic among subnets within a VCN.

[0073] A specific compute instance deployed on VCN104 can communicate with various different endpoints. These endpoints may include endpoints hosted by CSPI200 and endpoints outside of CSPI200. Endpoints hosted by CSPI101 may include endpoints on the same subnet as a particular compute instance (e.g., communication between two compute instances in subnet-1), endpoints on different subnets but within the same VCN (e.g., communication between a compute instance in subnet-1 and a compute instance in subnet-2), endpoints in different VCNs within the same region (e.g., communication between a compute instance in subnet-1 and an endpoint in a VCN in the same region 106 or 110, or between a compute instance in subnet-1 and an endpoint in service network 110 in the same region), or endpoints in VCNs in different regions (e.g., communication between a compute instance in subnet-1 and an endpoint in a VCN in a different region 108). In addition, compute instances in subnets hosted by CSPI101 can communicate with endpoints not hosted by CSPI101 (i.e., outside of CSPI101). These external endpoints include endpoints within the customer's on-premises network 116, endpoints within other remote cloud host networks 118, public endpoints 114 accessible via public networks such as the internet, and other endpoints.

[0074] Communication between compute instances on the same subnet is facilitated using VNICs associated with the source and destination compute instances. For example, compute instance C1 in subnet-1 may want to send a packet to compute instance C2, also in subnet-1. For a packet sent from the source compute instance, whose destination is another compute instance on the same subnet, this packet is first processed by the VNIC associated with the source compute instance. The processing performed by the VNIC associated with the source compute instance may include determining the packet's destination information from the packet header, identifying any policies (e.g., security lists) configured for the VNIC associated with the source compute instance, determining the packet's next hop, performing any packet encapsulation / decapsulation functions as needed, and forwarding / routing the packet to the next hop to facilitate communication to its intended destination. If the destination compute instance is on the same subnet as the source compute instance, the VNIC associated with the source compute instance is configured to identify the VNIC associated with the destination compute instance and forward packets to that VNIC for processing. The VNIC associated with the destination compute instance then performs the processing and forwards the packet to the destination compute instance.

[0075] When packets are communicated from a compute instance within a subnet to an endpoint in a different subnet of the same VCN, communication is facilitated by the VNICs associated with the source and destination compute instances, and the VCN VR. For example, if compute instance C1 in subnet-1 in Figure 1 wants to send a packet to compute instance D1 in subnet-2, the packet is first processed by the VNIC associated with compute instance C1. The VNIC associated with compute instance C1 is configured to route the packet to VCN VR105 using the VCN VR's default route or port 10.0.0.1. VCN VR105 uses port 10.1.0.1. It is configured to route packets to subnet-2. The packets are then received and processed by the VNIC associated with D1, and the VNIC forwards the packets to compute instance D1.

[0076] To transmit packets from compute instances within VCN104 to endpoints outside VCN104, communication is facilitated by a VNIC associated with the source compute instance, VCN VR105, and a gateway associated with VCN104. One or more types of gateways can be associated with VCN104. A gateway is an interface between the VCN and another endpoint, which is outside the VCN. A gateway is a Layer 3 / IP layer concept that enables communication between the VCN and endpoints outside the VCN. Therefore, gateways facilitate traffic flow between the VCN and other VCNs or networks. Different types of gateways can be configured in the VCN to facilitate different types of communication with different types of endpoints. Through gateways, communication may take place over a public network (e.g., the internet) or a private network. Various communication protocols may be used for these communications.

[0077] For example, compute instance C1 may want to communicate with an endpoint outside of VCN104. The packet may first be processed by the VNIC associated with source compute instance C1. The VNIC processing determines that the packet's destination is outside subnet-1 of C1. The VNIC associated with C1 can forward the packet to VCN VR105 of VCN104. VCN VR105 then processes the packet and, as part of the processing, determines a specific gateway associated with VCN104 as the packet's next hop based on the packet's destination. VCN VR105 can then forward the packet to the specific gateway. For example, if the destination is an endpoint within the customer's operation-premise network, the packet may be forwarded by VCN VR105 to a dynamic routing gateway (DRG) 122 configured for VCN104. The packet is then forwarded from the gateway to the next hop, facilitating communication of the packet to its intended final destination.

[0078] Various different types of gateways may be configured for the VCN. Examples of gateways that may be configured for a VCN are shown in Figure 1 and described below. Examples of gateways associated with a VCN are also shown in Figures 24, 25, 26, and 27 (for example, gateways shown by reference numbers 2434, 2436, 2438, 2534, 2536, 2538, 2634, 2636, 2638, 2734, 2736, and 2738) and described below. As shown in the embodiment shown in Figure 1, a dynamic routing gateway (DRG) 122 may be added to or associated with the customer VCN 104. The DRG 122 provides a path for private network traffic communication between the customer VCN 104 and another endpoint. The other endpoint may be the customer on-premises network 116, VCN 108 in a different region of CSPI 101, or another remote cloud network 118 not hosted by CSPI 101. The customer on-premises network 116 may be a customer network or customer data center built using customer resources. Access to the customer on-premises network 116 is generally strictly restricted. For customers who have both the customer on-premises network 116 and one or more VCNs 104 deployed or hosted in the cloud by CSPI 101, the customer may want the on-premises network 116 and the cloud-based VCNs 104 to be able to communicate with each other. This would allow the customer to build an enhanced hybrid environment that includes the customer's VCNs 104 hosted by CSPI 101 and the on-premises network 116. DRG 122 enables such communication. To enable such communication, communication channels Channel 124 is configured. In this case, one endpoint of the communication channel is located on the customer's on-premises network 116, and the other endpoint is located on CSPI 101 and connected to the customer's VCN 104. The communication channel 124 can travel over a public communication network such as the internet, or a private communication network. Various different communication protocols can be used, such as IPsec VPN technology on a public communication network such as the internet, or Oracle® FastConnect technology which uses a private network instead of a public network. A device or equipment within the customer on-premises network 116 that forms one endpoint of communication channel 124 is called customer premises equipment (CPE), such as CPE126 shown in Figure 1. The endpoint on the CSPI101 side may be a host machine running DRG122.

[0079] In certain embodiments, a Remote Peering Connection (RPC) can be added to the DRG. This allows a customer to peer one VCN with another VCN in a different region. Using such an RPC, customer VCN 104 can connect to VCN 108 in a different region using DRG 122. DRG 122 can also connect to other remote cloud networks 118 not hosted by CSPI 101, such as Microsoft® Azure Cloud, Amazon® AWS Cloud, etc. It may be used to communicate with Udo.

[0080] As shown in Figure 1, an Internet Gateway (IGW) 120 can be configured on the customer VCN 104 to enable compute instances on the customer VCN 104 to communicate with a public endpoint 114 accessible via a public network such as the Internet. The IGW 120 is a gateway for connecting the VCN to a public network such as the Internet. The IGW 120 enables public subnets within a VCN, such as VCN 104 (resources within public subnets have public overlay IP addresses), to directly access a public endpoint 112 on a public network such as the Internet 114. Connections can be initiated from subnets within VCN 104 or from the Internet using the IGW 120.

[0081] A Network Address Translation (NAT) gateway 128 can be configured in customer VCN 104. The NAT gateway 128 enables cloud resources within the customer VCN that do not have dedicated public overlay IP addresses to access the internet without directly exposing them to incoming internet connectivity (e.g., L4-L7 connectivity). This allows private subnets within the VCN, such as private subnet-1 of VCN 104, to have private access to public endpoints on the internet. With the NAT gateway, private subnets can initiate connections to the public internet, but connections cannot be initiated from the internet to the private subnets.

[0082] In certain embodiments, a service gateway (SGW) 126 can be configured in the customer VCN 104. The SGW 126 provides a route for private network traffic between the VCN 104 and service endpoints supported by the service network 110. In certain embodiments, the service network 110 may be provided by a CSP and can provide a variety of services. An example of such a service network is the Oracle® service network, which provides a variety of services that can be used by the customer. For example, compute instances (e.g., database systems) within the private subnet of the customer VCN 104 can access service endpoints (e.g., without requiring a public IP address or access to the internet). For example, data can be backed up to object storage. In some embodiments, a VCN may have only one SGW and connections can only be initiated from subnets within the VCN, and connections cannot be initiated from the service network 110. When peering a VCN with another VCN, resources in the other VCN typically cannot access the SGW. Resources in an on-premises network connected to a VCN via FastConnect or VPN Connect can also use the service gateway configured for that VCN.

[0083] In some implementations, SGW126 uses service-classless inter-domain routing (CIDR) labels. A CIDR label is a string representing all regionally exposed IP address ranges for a service or group of services of interest. Customers use service CIDR labels to control traffic to services when configuring SGW and associated routing rules. Customers can optionally use service CIDR labels when configuring security rules without having to adjust security rules if the public IP addresses of services change in the future.

[0084] The Local Peering Gateway (LPG) 132 is an addable gateway to the customer VCN 104 that enables the VCN 104 to peer with other VCNs within the same region. Peering means that VCNs communicate using private IP addresses without traffic traversing a public network such as the internet or routing traffic through the customer's on-premises network 116. In a preferred embodiment, the VCN has a separate LPG for each established peering. Local peering, or VCN peering, is a common practice used to establish network connectivity between different applications or infrastructure management functions.

[0085] Service providers, such as service providers on service network 110, can provide access to their services using different access models. According to the public access model, a service may be exposed as a public endpoint accessible publicly by compute instances within the customer VCN via a public network such as the internet, or it may be accessed privately via SGW126. According to a specific private access model, a service may be accessed as a private IP endpoint within a private subnet within the customer VCN. This is called private endpoint (PE) access and allows service providers to expose their services as instances within the customer's private network. A private endpoint resource represents a service within the customer VCN. Each PE appears as a VNIC (referred to as a PE-VNIC, having one or more private IPs) selected by the customer from a subnet within the customer VCN. Thus, a PE provides a way to provide services within the customer's private VCN subnet using a VNIC. Because the endpoint is exposed as a VNIC, the PE VNIC can utilize all the features associated with a VNIC, such as routing rules and security lists.

[0086] Service providers enable access via PE by registering their services. Providers can associate policies with services that restrict their visibility to customer tenants. Providers can register multiple services under a single virtual IP address (VIP), especially in the case of multi-tenant services. Multiple private endpoints may exist representing the same service (across multiple VCNs).

[0087] Subsequently, compute instances within the private subnet can access the service using the PE VNIC's private IP address or service DNS name. Compute instances within the customer VCN can access the service by sending traffic to the PE's private IP address within the customer VCN. The Private Access Gateway (PAGW) 130 is a gateway resource that can connect to a service provider VCN (e.g., a VCN within service network 110) and act as the receiving / transmitting point for all traffic to and from the customer subnet private endpoint. The PAGW 130 allows the provider to scale the number of PE connections without utilizing internal IP address resources. The provider only needs to configure one PAGW for any number of services registered in a single VCN. The provider can present a service as a private endpoint in multiple VCNs for one or more customers. From the customer's perspective, the PE VNIC appears to be connected to the service the customer wants to interact with, rather than to the customer's instances. Traffic directed to the private endpoint is routed to the service via the PAGW 130. These are called customer-to-service private connections (C2S connections).

[0088] Furthermore, using the PE concept, traffic is transmitted over FastConnect / IPsec links and customer VCNs. By allowing traffic to flow through private endpoints within the network, private access to the service can also be extended to the customer's on-premises network and data center. Furthermore, by allowing traffic to flow between LPG132 and PEs within the customer's VCN, private access to the service can also be extended to the customer's peering VCN.

[0089] Customers can control VCN routing at the subnet level, allowing them to specify which subnets use which gateways within their VCN, such as VCN104. The VCN's route table can be used to determine whether traffic can be routed outside the VCN through a particular gateway. For example, in a specific case, the route table for a public subnet within customer VCN104 might allow non-local traffic to be sent via IGW120. The route table for a private subnet within the same customer VCN104 might allow traffic to CSP services via SGW126. All remaining traffic could be sent via NAT gateway 128. The route table only controls traffic leaving the VCN.

[0090] Security lists associated with a VCN are used to control inbound connections and traffic entering the VCN via gateways. All resources within a subnet use the same mute table and security lists. Security lists may be used to control specific types of traffic entering and leaving instances within a VCN subnet. Security list rules may include inbound and outbound rules. For example, inbound rules may specify allowed source address ranges, and outbound rules may specify allowed destination address ranges. Security rules may specify specific protocols (e.g., TCP, ICMP), specific ports (e.g., port 22 for SSH, port 3389 for Windows® RDP), etc. In certain implementations, the instance's operating system may enforce its own firewall rules that match the security list rules. Rules may be stateful (e.g., connections are tracked and responses are automatically allowed without explicit security list rules for response traffic) or stateless.

[0091] Access from a customer VCN (i.e., resources or compute instances deployed on VCN104) may be classified as public access, private access, or dedicated access. Public access refers to an access model for accessing public endpoints using public IP addresses or NAT. Private access allows customer workloads within VCN104 with private IP addresses (e.g., resources in a private subnet) to access services without traversing a public network such as the internet. In certain embodiments, CSPI101 allows customer VCN workloads with private IP addresses to access the public service endpoint of a service using a service gateway. Thus, the service gateway provides a private access model by establishing a virtual link between the customer VCN and the public endpoint of a service that resides outside the customer's private network.

[0092] Furthermore, CSPI uses dedicated platforms that utilize technologies such as FastConnect public peering. Brick access can be provided. In this case, the customer's on-premises instance can connect via FastConnect without going through a public network such as the internet. It can be used to access one or more services within the customer VCN. Additionally, CSPI provides dedicated private access using FastConnect private peering. It is also possible to do this. In this case, the customer's on-premises instance with a private IP address can access the customer's VCN workload using a FastConnect connection. FastConnect uses the public internet to connect customers' on-premises networks. FastConnect is a network connection used instead of connecting to CSPI and its services. Compared to internet-based connections, FastConnect offers higher bandwidth options and It provides an easy, flexible, and economical way to create dedicated private connections with a reliable and consistent networking experience.

[0093] Figure 1 and the accompanying description above illustrate various virtualization elements in an exemplary virtual network. As mentioned above, the virtual network is built on an underlying physical network or infrastructure network. Figure 2 is a simplified architectural diagram showing the physical elements within the physical network in the CSPI200 that provide the foundation for the virtual network, according to a particular embodiment. As shown, the CSPI200 provides a distributed environment including elements and resources (e.g., compute resources, memory resources, and networking resources) provided by a Cloud Service Provider (CSP). These elements and resources are used to provide cloud services (e.g., IaaS services) to subscribers, i.e., customers who subscribe to one or more services provided by the CSP. Based on the services a customer subscribes to, the CSPI200 provides some resources (e.g., compute resources, memory resources, and networking resources) to the customer. The customer can then use the physical compute resources, memory resources, and networking resources provided by the CSPI200 to build their own cloud-based (i.e., CSPI-hosted) customizable private virtual network. As mentioned above, these customer networks are called virtual cloud networks (VCNs). Customers can deploy one or more customer resources, such as compute instances, to these customer VCNs. Compute instances may be virtual machines, bare metal instances, etc. CSPI200 provides infrastructure and a suite of complementary cloud services that enable customers to build and run a wide range of applications and services in a highly available host environment.

[0094] In the exemplary embodiment shown in Figure 2, the physical elements of the CSPI200 include one or more physical host machines or physical servers (e.g., 202, 206, 208), and network virtual machines. This includes a NVD (Non-Virtual Device) (e.g., 210, 212), a top-of-rack (TOR) switch (e.g., 214, 216), a physical network (e.g., 218), and switches within physical network 218. A physical host machine or server can host and run various compute instances participating in one or more subnets of the VCN. Compute instances may include virtual machine instances and bare metal instances. For example, the various compute instances shown in Figure 1 may be hosted by the physical host machine shown in Figure 2. Virtual machine compute instances in the VCN may run on one host machine or on several different host machines. A physical host machine can also host virtual host machines, container-based hosts or functions, etc. The VIC and VCN VR shown in Figure 1 may run on the FTVD shown in Figure 2. The gateway shown in Figure 1 may run on the host machine and / or NVD shown in Figure 2.

[0095] A host machine or server can run a hypervisor (also known as a virtual machine monitor or VMM) that creates and enables a virtualized environment on the host machine. Virtualization or a virtualized environment facilitates cloud-based computing. One or more compute instances may be created, run, and managed on the host machine by a hypervisor on the host machine. The hypervisor on the host machine can share the host machine's physical compute resources (e.g., compute resources, memory resources, and networking resources) among various compute instances running on the host machine.

[0096] For example, as shown in Figure 2, host machines 202 and 208 run hypervisors 260 and 266, respectively. These hypervisors may be implemented using software, firmware, hardware, or a combination thereof. Typically, a hypervisor is a process or software layer residing in the host machine's operating system (OS), which runs on the host machine's hardware processors. A hypervisor provides a virtualization environment that allows the host machine's physical computing resources (e.g., processing resources such as processors / cores, memory resources, and networking resources) to be shared among various virtual machine computing instances running on the host machine. For example, in Figure 2, hypervisor 260 resides in the OS of host machine 202 and allows the host machine 202's computing resources (e.g., processing resources, memory resources, and networking resources) to be shared among computing instances (e.g., virtual machines) running on host machine 202. A virtual machine can have its own OS (called a guest OS). This guest OS may be the same as or different from the host machine's OS. The operating system (OS) of a virtual machine running on a host machine may be the same as, or different from, the operating systems of other virtual machines running on the same host machine. Therefore, a hypervisor can run multiple OSs in parallel while sharing the same computing resources of the host machine. The host machines shown in Figure 2 may have the same type of hypervisor or different types of hypervisors.

[0097] Compute instances may be virtual machine instances or bare metal instances. In Figure 2, compute instance 268 on host machine 202 and compute instance 274 on host machine 208 are examples of virtual machine instances. Host machine 206 is an example of a bare metal instance provided to a customer.

[0098] In certain examples, the entire host machine may be provided to a single customer, and one or more compute instances (either virtual machines or bare metal instances) hosted on that host machine may all belong to the same customer. In other examples, A host machine may be shared among multiple customers (i.e., multiple tenants). In such a multi-tenant scenario, the host machine can host virtual machine compute instances belonging to different customers. These compute instances may be members of different VCNs belonging to different customers. In certain embodiments, bare metal compute instances are hosted by bare metal servers that do not have a hypervisor. When bare metal compute instances are provided, a single customer or tenant maintains control of the physical CPU, memory, and network interfaces of the host machine hosting the bare metal instance, and the host machine is not shared with other customers or tenants.

[0099] As mentioned above, each compute instance that is part of a VCN is associated with a VNIC that enables the compute instance to be a member of the VCN's subnet. The VNIC associated with a compute instance facilitates the communication of packets or frames to and from the compute instance. The VNIC is associated with the compute instance when it is created. In certain embodiments, for a compute instance run by a host machine, the VNIC associated with that compute instance is run by an NVD connected to the host machine. For example, in Figure 2, host machine 202 runs virtual machine compute instance 268 associated with VNIC 276, and VNIC 276 is run by an NVD 210 connected to host machine 202. In another example, bare metal instance 272 hosted by host machine 206 is associated with VNIC 280, which is run by an NVD 212 connected to host machine 206. In yet another example, VNIC 284 is associated with compute instance 274 run by host machine 208, and VNIC 284 is run by an NVD 212 connected to host machine 208.

[0100] For compute instances hosted by a host machine, an NVD connected to that host machine executes a VCN VR corresponding to the VCN of which the compute instance is a member. For example, in the embodiment shown in Figure 2, NVD210 executes VCN VR277 corresponding to the VCN of which compute instance 268 is a member. Additionally, NVD212 can execute one or more VCN VR283 corresponding to the VCNs of compute instances hosted by host machines 206 and 208.

[0101] A host machine may include one or more network interface cards (NICs) for connecting it to other devices. The NICs on the host machine may provide one or more ports (or interfaces) for communicating with another device. For example, a host machine can be connected to an NVD using one or more ports (or interfaces) provided on the host machine and the NVD. Alternatively, a host machine can be connected to other devices, such as another host machine.

[0102] For example, in Figure 2, host machine 202 is connected to NVD210 using a link 220 that extends between port 234 provided by NIC 232 of host machine 202 and port 236 of NVD210. Host machine 206 is connected to NVD212 using a link 224 that extends between port 246 provided by NIC 244 of host machine 206 and port 248 of NVD212. Host machine 208 is connected to NVD212 using a link 226 that extends between port 252 provided by NIC 250 of host machine 208 and port 254 of NVD212.

[0103] Similarly, NVDs communicate via the physical (also known as switch fabric) over the communication link. They are connected to a top-of-rack (TOR) switch connected to network 218. In certain embodiments, the links between the host machine and the NVD and between the NVD and the TOR switch are Ethernet® links. For example, in Figure 2, NVDs 210 and 212 are connected to TOR switches 214 and 216, respectively, via links 228 and 230. In certain embodiments, links 220, 224, 226, 228, and 230 are Ethernet links. The collection of host machines and NVDs connected to the TOR is sometimes referred to as a rack.

[0104] The physical network 218 provides a communication fabric that enables interoperability between TOR switches. The physical network 218 may be a multi-layer network. In a particular implementation, the physical network 218 is a multi-layer Clos network of switches, and TOR switches 214 and 216 represent leaf-level nodes of the multi-layer and multi-node physical switching network 218. Different Clos network configurations are possible, including but not limited to 2-layer, 3-layer, 4-layer, 5-layer networks, and generally "n"-layer networks. An example of a Clos network is shown in Figure 5 and described below.

[0105] Various connection configurations are possible between the host machine and N VDs, including one-to-one, many-to-one, and one-to-many configurations. In an example of a one-to-one configuration, each host machine is connected to its own separate NVD. For example, in Figure 2, host machine 202 is connected to NVD210 via host machine 202's NIC232. In a many-to-one configuration, multiple host machines are connected to a single NVD. For example, in Figure 2, host machines 206 and 208 are connected to the same NVD212 via NIC244 and 250, respectively.

[0106] In a one-to-many configuration, one host machine is connected to multiple NVDs. Figure 3 shows an example within CSPI300 where a host machine is connected to multiple NVDs. As shown in Figure 3, the host machine 302 has a network interface card (NIC) 304 which includes multiple ports 306 and 30S. The host machine 300 is connected to the first NVD 310 via port 306 and link 320, and to the second NVD 312 via port 308 and link 322. Ports 306 and 308 may be Ethernet® ports, and links 320 and 322 between the host machine 302 and the NVDs 310 and 312 may be Ethernet® links. The NVD 310 is connected to the first TOR switch 314, and the NVD 312 is connected to the second TOR switch 316. The links between the NVDs 310 and 312 and the TOR switches 314 and 316 may be Ethernet® links. TOR switches 314 and 316 represent Tier-0 switching devices within a multilayer physical network 318.

[0107] The configuration shown in Figure 3 provides two separate physical network paths from the physical switch network 318 to the host machine 302: a first path from TOR switch 314 through NVD 310 to the host machine 302, and a second path from TOR switch 316 through NVD 312 to the host machine 302. These separate paths provide enhanced availability (referred to as high availability) for the host machine 302. If one path experiences a problem (e.g., one link in the path fails) or if there is a problem with a device (e.g., a particular NVD is not functioning), the other path can be used for communication with the host machine 302.

[0108] In the configuration shown in Figure 3, the host machine is connected to two different NVDs using two different ports provided by the host machine's NIC. In other embodiments... In this configuration, the host machine may include multiple NICs that enable connections between the host machine and multiple NVDs.

[0109] Referring again to Figure 2, the NVD is a physical device or element that performs one or more network virtualization functions and / or memory virtualization functions. The NVD may be any device having one or more processing units (e.g., a CPU, a network processing unit (NPU), an FPGA, a packet processing pipeline), memory including a cache, and ports. Various virtualization functions may be performed by software / firmware executed by one or more processing units of the NVD.

[0110] NVDs may be implemented in various different forms. For example, in certain embodiments, an NVD may be implemented as an interface card called a smart NIC or intelligent NIC with an integrated processor. A smart NIC is a separate device from the NIC on the host machine. In Figure 2, NVD210 may be implemented as a smart NIC connected to host machine 202, and NVD212 may be implemented as smart NICs connected to host machines 206 and 208.

[0111] However, the smart NIC is just one example of an NVD implementation. Various other implementations are possible. For example, in some other implementations, the NVD or one or more functions performed by the NVD may be incorporated into or performed by one or more host machines, one or more TOR switches, and other elements of the CSPI200. For example, the NVD may be integrated into the host machine. In this case, the functions performed by the NVD are performed by the host machine. As another example, the NVD may be part of a TOR switch, or the TOR switch may be configured to perform functions performed by the NVD, enabling the TOR switch to perform various complex packet translations used in public clouds. A TOR that performs the functions of the NVD is sometimes called a smart TOR. In yet another implementation that provides customers with virtual machine (VM) instances rather than bare metal (BM) instances, the functions provided by the NVD may be implemented within the hypervisor of the host machine. In some other implementations, some of the functions of the NVD may be offloaded to a centralized service running on a set of host machines.

[0112] As shown in Figure 2, in certain embodiments, such as when implemented as a smart NIC, the NVD may have multiple physical ports that enable it to connect to one or more host machines and one or more TOR switches. The ports on the NVD can be classified as host-facing ports (also called "south ports") or network-facing or TOR-facing ports (also called "north ports"). Host-facing ports on the NVD are the ports used to connect the NVD to a host machine. Examples of host-facing ports in Figure 2 include port 236 on the NVD210 and ports 248 and 254 on the NVD212. Network-facing ports on the NVD are the ports used to connect the NVD to a TOR switch. Examples of network-facing ports in Figure 2 include port 256 on the NVD210 and port 258 on the NVD212. As shown in Figure 2, the NVD210 is connected to the TOR switch 214 via a link 228 extending from port 256 on the NVD210 to the TOR switch 214. Similarly, the NVD212 is connected to the TOR switch 216 via a link 230 that extends from port 258 of the NVD212 to the TOR switch 216.

[0113] NVD receives packets and frames from the host machine (e.g., packets and frames generated by compute instances hosted by the host machine) via the host-facing port, performs the necessary packet processing, and then the NVD's network... Packets and frames can be forwarded to the TOR switch via the network-facing port. The NVD can receive packets and frames from the TOR switch via its network-facing port, perform the necessary packet processing, and then forward the packets and frames to the host machine via its host-facing port.

[0114] In certain embodiments, multiple ports and associated links may be provided between the NVD and the TOR switch. By aggregating these ports and links, a link aggregator group (LAG) of multiple ports or links can be formed. Link aggregation allows multiple physical links between two endpoints (e.g., between the NVD and the TOR switch) to be treated as a single logical link. All physical links within a given LAG can operate in full-duplex mode at the same speed. LAGs help to increase the bandwidth and reliability of the connection between the two endpoints. If one of the physical links in the LAG fails, traffic is dynamically and transparently reassigned to another physical link within the LAG. Aggregated physical links provide higher bandwidth than individual links. Multiple ports associated with a LAG are treated as a single logical port. Traffic can be load-balanced across the multiple physical links of the LAG. One or more LAGs can be configured between two endpoints. The two endpoints may be, for example, between the NVD and the TOR switch, or between a host machine and the NVD.

[0115] NVD implements or performs network virtualization functions. These functions are performed by software / firmware run by NVD. Examples of network virtualization functions include, but are not limited to, packet encapsulation and decapsulation functions, functions for creating VCN networks, functions for implementing network policies such as VCN security list (firewall) functions, and functions for facilitating the routing and forwarding of packets to and from compute instances within the VCN. In certain embodiments, upon receiving a packet, NVD is configured to run a packet processing pipeline that processes the packet and determines how to forward or route it. As part of this packet processing pipeline, NVD provides the execution of one or more virtual functions related to the overlay network, e.g., running a VNIC related to compute instances within the VCN, running a virtual router (VR) related to the VCN, packet encapsulation and decapsulation to facilitate forwarding or routing within the virtual network, running a specific gateway (e.g., a local peering gateway), implementing security lists, network security groups, network address translation (NAT) functions (e.g., translation from public IP to private IP on a per-host basis), throttling functions, and other functions.

[0116] In some embodiments, the packet processing data path within the NVD may include multiple packet pipelines. Each packet pipeline consists of a set of packet translation stages. In some implementations, upon receiving a packet, it is parsed and classified into a single pipeline. The packet is then processed linearly, stage by stage, until it is discarded or sent out through the NVD's interface. These stages provide the basic functional packet processing building blocks (e.g., header validation, throttling, insertion of new Layer 2 headers, L4 firewall execution, VCN encapsulation / decapsulation), and as a result, new pipelines can be constructed by assembling existing stages, and new functionality can be added by creating new stages and inserting them into existing pipelines.

[0117] NVD can perform both control plane and data plane functions corresponding to the VCN's control plane and data plane. An example of the VCN control plane is shown in Figure The VCN data plane is shown in Figures 24, 25, 26, and 27 (see reference numbers 2416, 2516, 2616, and 2716) and is described below. An example of the VCN data plane is shown in Figures 24, 25, 26, and 27 (see reference numbers 2418, 2518, 2618, and 2718) and is described below. Control plane functions include functions used to configure the network to control how data is forwarded (e.g., setting routes and route tables, configuring VNICs). In certain embodiments, a VCN control plane is provided that centrally calculates the mapping of all overlays to the substrate and exposes it to the NVD and virtual network edge devices (e.g., various gateways such as DRG, SGW, IGW). Firewall rules can also be exposed using the same mechanism. In certain embodiments, the NVD retrieves only the mappings relevant to that NVD. Data plane functions include functions that perform the actual routing / forwarding of packets based on the configuration set using the control plane. The VCN data plane is implemented by encapsulating customer network packets before they pass through the backbone network. The encapsulation / decapsulation function is implemented in the NVD. In certain embodiments, the NVD is configured to intercept all network packets entering and leaving the host machine and to perform network virtualization functions.

[0118] As described above, NVD performs various virtualization functions, including VNICs and VCN VRs. An NVD can run VNICs associated with compute instances hosted by one or more host machines connected to a VNIC. For example, as shown in Figure 2, NVD210 runs the functions of VNIC276 associated with compute instance 268 hosted by host machine 202 connected to NVD210. As another example, NVD212 runs VNIC280 associated with bare-metal compute instance 272 hosted by host machine 206 and VNIC284 associated with compute instance 274 hosted by host machine 208. A host machine can host compute instances belonging to different VCNs belonging to different customers. An NVD connected to a host machine can run VNICs corresponding to compute instances (i.e., perform functions associated with VNICs).

[0119] Furthermore, the NVD runs a VCN virtual router corresponding to the VCN of the compute instance. For example, in the embodiment shown in Figure 2, NVD210 runs VCN VR277 corresponding to the VCN to which compute instance 268 belongs. NVD212 runs one or more VCN VR283 corresponding to one or more VCNs to which compute instances hosted on host machines 206 and 208 belong. In a particular embodiment, a VCN VR corresponding to a VCN is run by all NVDs connected to a host machine that hosts at least one compute instance belonging to that VCN. If a host machine hosts compute instances belonging to a different VCN, the NVDs connected to that host machine can run VCN VRs corresponding to different VCNs.

[0120] In addition to VNICs and VCN VRs, an NVD may include one or more hardware elements that run various software (e.g., daemons) and facilitate various network virtualization functions performed by the NVD. For simplicity, these various elements are grouped as “packet processing elements” as shown in Figure 2. For example, NVD210 includes packet processing element 286, and NVD212 includes packet processing element 288. For example, a packet processing element of an NVD may include a packet processor configured to monitor all packets received and communicated using the NVD and to store network information by interacting with the NVD’s ports and hardware interfaces. Network information may include, for example, network flow information to identify different network flows processed by the NVD and information about each flow (e.g., statistics for each flow). In a particular embodiment, network flow information is: It may be stored on a per-VNIC basis. As another example, the packet processing element may include a replication agent configured to replicate the information stored by the NVD to one or more different replication target stores. As yet another example, The packet processing element may include a logging agent configured to perform the NVD's logging function. The packet processing element also considers the NVD's performance and It may also include software to monitor health and, in some cases, the status and health of other elements connected to the NVD.

[0121] Figure 1 shows the elements of an exemplary virtual or overlay network, including a VCN, subnets within the VCN, compute instances deployed on the subnets, VNICs associated with the compute instances, a VR for the VCN, and a set of gateways configured for the VCN. The overlay elements shown in Figure 1 may be run or hosted by one or more of the physical elements shown in Figure 2. For example, compute instances within a VCN may be run or hosted by one or more host machines shown in Figure 2. In the case of compute instances hosted by host machines, the VNICs associated with those compute instances are typically run by NVDs connected to that host machine (i.e., VNIC functionality is provided by NVDs connected to that host machine). The VCN VR functionality of the VCN is run by all NVDs connected to the host machines that host or run the compute instances that are part of that VCN. Gateways associated with the VCN may be run by one or more different types of NVDs. For example, some gateways may be run by smart NICs, and others may be run by one or more host machines or other implementations of NVDs.

[0122] As described above, compute instances within a customer VCN can communicate with a variety of different endpoints. These endpoints may be on the same subnet as the source compute instance, on a different subnet but still within the same VCN, or may include endpoints outside the source compute instance's VCN. This communication is facilitated using the VNIC associated with the compute instance, the VCN VR, and the gateway associated with the VCN.

[0123] Communication between two compute instances on the same subnet within a VCN is facilitated using VNICs associated with the source and destination compute instances. The source and destination compute instances may be hosted on the same host machine or on different host machines. Packets originating from the source compute instance may be forwarded from the host machine hosting the source compute instance to an NVD connected to that host machine. In the NVD, packets are processed using a packet processing pipeline, which may include the execution of the VNIC associated with the source compute instance. Because the destination endpoint of the packets is on the same subnet, the execution of the VNIC associated with the source compute instance forwards the packets to the NVD running the VNIC associated with the destination compute instance, where the NVD processes the packets and forwards them to the destination compute instance. The VNICs associated with the source and destination compute instances may run on the same NVD (for example, if both the source and destination compute instances are hosted on the same host machine) or on different NVDs (for example, if the source and destination compute instances are hosted on different host machines connected to different NVDs). The VNIC can use the routing / forwarding table stored by the NVD to determine the next hop of a packet.

[0124] When a packet is communicated from a compute instance within a subnet to an endpoint in a different subnet within the same VCN, the packet originating from the source compute instance is communicated from the host machine hosting the source compute instance to the NVD connected to that host machine. In the NVD, the packet is processed using a packet processing pipeline and a VR associated with the VCN, which may include the execution of one or more VNICs. For example, as part of the packet processing pipeline, the NVD executes or invokes a function corresponding to the VNIC associated with the source compute instance (also called executing the VNIC). The function executed by the VNIC may include looking up the VLAN identifier on the packet. Because the packet's destination is outside the subnet, the VCN VR function is invoked and executed by the NVD. The VCN VR then routes the packet to the NVD executing the VNIC associated with the destination compute instance. The VNIC associated with the destination compute instance then processes the packet and forwards it to the destination compute instance. The VNICs associated with the source compute instance and the destination compute instance may run on the same NVD (for example, if both the source compute instance and the destination compute instance are hosted by the same host machine), or they may run on different NVDs (for example, if the source compute instance and the destination compute instance are hosted by different host machines connected to different NVDs).

[0125] If the packet's destination is outside the source compute instance's VCN, the packet originating from the source compute instance is communicated from the host machine hosting the source compute instance to the NVD connected to that host machine. The NVD runs the VNIC associated with the source compute instance. Because the packet's destination endpoint is outside the VCN, the packet is processed by the VCN VR of that VCN. The NVD invokes VCN VR functionality, which may result in the packet being forwarded to an NVD running the appropriate gateway associated with the VCN. For example, if the destination is an endpoint within the customer's on-premises network, the packet may be forwarded by the VCN VR to an NVD running the DRG gateway configured for the VCN. The VCN VR may run on the same NVD running the VNIC associated with the source compute instance, or it may run on a different NVD. The gateway may run on an NVD that is a smart NIC, a host machine, or another NVD implementation. The packet is then processed by the gateway and forwarded to the next hop to facilitate communication of the packet to its intended destination endpoint. For example, in the embodiment shown in Figure 2, a packet originating from compute instance 268 may be communicated from host machine 202 to NVD210 via link 220 (using NIC 232). VNIC 276 on NVD210 is invoked because it is the VNIC associated with source compute instance 268. VNIC 276 is configured to examine the encapsulation information in the packet, determine the next hop for forwarding the packet to facilitate communication of the packet to its intended destination endpoint, and forward the packet to the determined next hop.

[0126] Compute instances deployed on a VCN can communicate with various different endpoints. These endpoints may include endpoints hosted by CSPI200 and endpoints outside of CSPI200. Endpoints hosted by CSPI200 may include instances within the same VCN or other VCNs (which may be customer VCNs or VCNs not belonging to a customer). Communication between endpoints hosted by CSPI200 may be performed over the physical network 218. Compute instances can also communicate with endpoints not hosted by CSPI200 or located outside of CSPI200. Examples of these endpoints are located within the customer's on-premises network or data center. This includes endpoints, or public endpoints accessible via a public network such as the Internet. Communication with endpoints outside of the CSPI200 may be performed over a public network (e.g., the Internet) (not shown in Figure 2) or a private network (not shown in Figure 2) using various communication protocols.

[0127] The architecture of the CSPI200 shown in Figure 2 is merely an example and is not intended to be limiting. Alternative embodiments are possible, and variations, substitutions, and modifications are possible. For example, in some implementations, the CSPI200 may have more or fewer systems or elements than those shown in Figure 2, may combine two or more systems, or may have different system configurations or arrangements. The systems, subsystems, and other elements shown in Figure 2 may be implemented as software (e.g., code, instructions, programs), hardware, or a combination thereof, executed by one or more processing units (e.g., processors, cores) of each system. The software may be stored in a non-temporary storage medium (e.g., a memory device).

[0128] Figure 4 shows a connection between a host machine and an NVD to provide I / O virtualization to support multi-tenancy functionality, according to a particular embodiment. As shown in Figure 4, the host machine 402 runs a hypervisor 404 that provides the virtualization environment. The host machine 402 runs two virtual machine instances, namely VM1 406 belonging to customer / tenant #1 and VM2 408 belonging to customer / tenant #2. The host machine 402 includes a physical NIC 410 connected to the NVD 412 via link 414. Each compute instance is connected to a VNIC run by the NVD 412. In the embodiment of Figure 4, VM1 406 is connected to VNIC-VM1 420 and VM2 408 is connected to VNIC-VM2 422.

[0129] As shown in Figure 4, NIC410 includes two logical NICs, namely logical NIC A 416 and logical NIC B 418. Each virtual machine is connected to its own logical NIC and configured to operate with its own logical NIC. For example, VM1 406 is connected to logical NIC A 416, and VM2 408 is connected to logical NIC B 418. Although the host machine 402 consists of only one physical NIC 410 shared by multiple tenants, the logical NICs allow each tenant's virtual machine to believe that it owns its own host machine and NIC.

[0130] In a particular embodiment, each logical NIC is assigned its own VLAN ID. Thus, logical NIC A 416 for tenant #1 is assigned a specific VLAN ID, and logical NIC B 418 for tenant #2 is assigned a different VLAN ID. When a packet is communicated from VM1 406, the hypervisor attaches the tag assigned to tenant #1 to the packet and then communicates the packet from host machine 402 to NVD412 via link 414. Similarly, when a packet is communicated from VM2 408, the hypervisor attaches the tag assigned to tenant #2 to the packet and then communicates the packet from host machine 402 to NVD412 via link 414. Thus, the packet 424 communicated from host machine 402 to NVD412 has an associated tag 426 that identifies a specific tenant and associated VM. When packet 424 is received from host machine 402 on NVD, the tag 426 associated with the packet is used to determine whether the packet should be processed by VNIC-VM1 420 or VNIC-VM2 422. The packet is then processed by the corresponding VNIC. The configuration shown in Figure 4 allows each tenant's compute instance to believe it owns its own host machine and NIC. The configuration shown in Figure 4 provides I / O virtualization to support multi-tenancy functionality. ru.

[0131] Figure 5 is a schematic block diagram showing a physical network 500 according to a particular embodiment. The embodiment shown in Figure 5 is constructed as a Clos network. A Clos network is a specific type of network topology designed to provide connectivity redundancy while maintaining high bimodal bandwidth and maximum resource utilization. A Clos network is a type of non-blocking, multi-stage or multi-layer switching network, where the number of stages or layers may be 2, 3, 4, 5, etc. The embodiment shown in Figure 5 is a 3-layer network including layers 1, 2, and 3. A TOR switch 504 represents a layer-0 switch in the Clos network. One or more NVDs are connected to the TOR switch. Layer-0 switches are also called edge devices in the physical network. Layer-0 switches are connected to layer-1 switches, also called leaf switches. In the embodiment shown in Figure 5, "n" layer-0 TOR switches are connected to "n" layer-1 switches to form pods. Each layer-0 switch in a pod is interconnected to all layer-1 switches in the pod, but switches between pods are not connected. In a particular implementation, two pods are referred to as a block. Each block is serviced by or connected to n Layer-2 switches (also called spine switches). The physical network topology may contain multiple blocks. Similarly, the Layer-2 switches are connected to n Layer-3 switches (also called superspine switches). Packet communication over the physical network 500 is typically performed using one or more Layer 3 communication protocols. Typically, all layers of the physical network except the TOR layer are n-way redundant, thus achieving high availability. The physical network can be extended by specifying policies for pods and blocks to control the mutual visibility of switches in the physical network.

[0132] A key feature of Clos networks is that the maximum hop count required to reach one Layer-0 switch from another Layer-0 switch (or from an NVD connected to a Layer-0 switch to another NVD connected to a Layer-0 switch) remains constant. For example, in a Layer 3 Clos network, a packet requires a maximum of 7 hops to reach one NVD from another. In this case, the source and target NVDs are connected to the leaf layer of the Clos network. Similarly, in a Layer 4 Clos network, a packet requires a maximum of 9 hops to reach one NVD from another. In this case, the source and target NVDs are connected to the leaf layer of the Clos network. Therefore, the Clos network architecture maintains a constant overall network latency, which is crucial for communication within and between data centers. Clos topologies are horizontally scalable and cost-effective. Network bandwidth / throughput capacity can be easily increased by adding more switches to each layer (e.g., more leaf and spine switches) and increasing the number of links between switches in adjacent layers.

[0133] In certain embodiments, each resource within the CSPI is assigned a unique identifier called a Cloud Identifier (CID). This identifier is included as part of the resource's information. This identifier can be used to manage the resource, for example, via a console or API. An example syntax for a CID is as follows:

[0134] ocid1.<RESOURCE TYPE> . <realm>[REGION] [FUTURE USE]<UNIQUE ID> That is the case. During the ceremony, "ocid1" is a string that indicates the CID version.

[0135] "RESOURCE TYPE" refers to the type of resource (e.g., instance, volume, VCN). This represents subnets, users, and groups.

[0136] "REALM" represents the area where the resource resides. As an example value, "c1" is commercial. The domain names represent different domains; "c2" represents the government cloud domain, and "c3" represents the federal government cloud domain. Each domain can have its own domain name.

[0137] "REGION" represents the region to which the resource belongs. If no region applies to the resource, this section may be left blank.

[0138] "FUTURE USE" indicates that it is reserved for future use. "UNIQUE ID" is the unique ID part. This format is used for resources or services. This may vary depending on the type of screw.

[0139] L2 Virtual Network The number of enterprise customers migrating on-premises applications to cloud environments provided by cloud service providers (CSPs) continues to increase rapidly. However, many of these customers quickly realize that migrating to the cloud environment is quite difficult and requires redesigning and rebuilding their existing applications to run in the cloud. This is because applications developed in on-premises environments often rely on the capabilities of the physical network in terms of monitoring, availability, and scalability. Such on-premises applications need to be redesigned and rebuilt before they can run in the cloud environment.

[0140] There are several reasons why migrating on-premises applications to a cloud environment is not easy. One of the main reasons is that current cloud virtual networks operate at Layer 3 of the OSI model, for example, the IP layer, and therefore cannot provide the Layer 2 functionality required by applications. Layer 3-based routing or forwarding involves determining where a packet should be sent (e.g., to which customer instance) based on information contained in the Layer 3 header of the packet, for example, based on the destination IP address contained in the Layer 3 header of the packet. To facilitate this, the location of IP addresses within the virtualized cloud network is determined via a centralized control and orchestration system or controller. This location may include, for example, IP addresses associated with customer entities or resources within the virtualized cloud environment.

[0141] Many customers are running applications in on-premises environments with stringent requirements for Layer 2 networking capabilities not addressed by current cloud and IaaS service providers. For example, traffic to current cloud services is routed using Layer 3 protocols with Layer 3 headers, and the Layer 2 capabilities required by the application are not supported. These Layer 2 capabilities include Address Resolution Protocol (ARP) processing, media access The features may include MAC address learning and Layer 2 broadcast functionality, Layer 2 (MAC-based) forwarding, and Layer 2 networking configuration. As described in this disclosure, providing virtualized Layer 2 networking capabilities in a virtualized cloud network enables customers to smoothly migrate legacy applications to a cloud environment without requiring substantial redesign or reconstruction. For example, the virtualized Layer 2 networking capabilities described in this disclosure enable such applications (e.g., VMware vSphere, vCenter, vSAN, NSX-T components) to operate on-premises. This enables communication at Layer 2, similar to the mismatched environment. Because these applications can run on the public cloud with the same version and configuration, customers can utilize their existing knowledge, tools, and processes associated with legacy on-premises applications. Misapplications can be used. Additionally, customers can access native cloud services from their applications (for example, using VMware Software-Defined Data Centers (SDDCs)).

[0142] As another example, some legacy on-premises applications (e.g., enterprise clustering software applications, network virtual appliances) require Layer 2 broadcast support to handle failover. Illustrative applications include Fortinet FortiGate, IBM® QRadar, Palo Alto Firewalls, Cisco ASA, Juniper SRX, and Oracle® RAC (Real Application Clustering). By providing virtualized Layer 2 networking in a virtualized public cloud as described in this disclosure, these applications can operate in a virtualized public cloud environment without modification. Virtualized Layer 2 networking capabilities comparable to on-premises are provided as described in this disclosure. The virtualized Layer 2 networking capabilities described in this disclosure support traditional Layer 2 networking. This includes support for unicast Layer 2 traffic, broadcast Layer 2 traffic, and multicast Layer 2 traffic, along with support for customer-defined VLANs. Layer 2-based packet routing and forwarding includes, for example, routing or forwarding packets based on the destination MAC address contained in the Layer 2 header, using the Layer 2 protocol and information contained in the Layer 2 header of the packet. Protocols used by enterprise applications (e.g., clustering software applications), such as ARP (Address Resolution Protocol), GARP (Gratuitous ARP), and RARP (Reverse Addressing Protocol). The Reverse ARP (Reverse ARP) resolution protocol can now also operate in cloud environments.

[0143] There are several reasons why traditional virtualized cloud infrastructure supports virtualized Layer 3 networks and not Layer 2 networks. Layer 2 networks are typically not as scalable as Layer 3 networks. Layer 2 network control protocols do not have the advanced level required for scalability. For example, Layer 3 networks do not need to worry about packet looping, which Layer 2 networks must address. IP packets (i.e., Layer 3 packets) have a Time To Live (TTL), while Layer 2 packets do not. IP addresses contained within Layer 3 packets have topology, such as subnets and CIDR ranges, while Layer 2 addresses (e.g., MAC addresses) do not. Layer 3 IP networks have built-in tools to facilitate troubleshooting, such as ping and traceroute for finding routing information. Such tools are not available to Layer 2. Layer 3 networks support multipathing, which is not available to Layer 2. Layer 2 networks have advanced control protocols, particularly for exchanging information between entities within the network (e.g., BGP (Border Gateway Protocol)). Furthermore, because it lacks OSPF (Open Shortest Path First), it must rely on broadcast and multicast to learn the network, which can negatively impact network performance. Also, as the network changes, Layer 2 requires repeating the learning process, whereas Layer 3 does not. For these and other reasons, it is more desirable for cloud IaaS service providers to provide infrastructure that operates at Layer 3 rather than Layer 2.

[0144] However, despite its numerous drawbacks, many on-premises applications require Layer 2 functionality. For example, instances can be compute instances (e.g., bare metal, virtual machines, or containers) or service instances (e.g., load bars). Assume a virtualized cloud configuration in a virtual network "V" (which could be a Lancer, NFS mount point, or other service instance) where a customer (Customer 1) has two instances: instance A with IP1 and instance B with IP2. Virtual network V is a separate address space isolated from other virtual networks and the underlying physical network. This isolation can be achieved using various techniques, including packet encapsulation or NAT. Therefore, the IP addresses of instances within the customer's virtual network are different from the addresses in the physical network hosting those instances. A centralized SDN (Software Defined Networking) control plane is provided that knows the physical IP and virtual interface of all virtual IP addresses. When sending a packet from instance A to destination IP2 in virtual network V, the virtual network SDN stack needs to know the location of IP2. The virtual network SDN stack needs to know the location of IP2 in advance so that it can send the packet to the IP of the physical network hosting the virtual IP address IP2 in virtual network V. Because the location of virtual IP addresses can be changed on the cloud, the relationship between physical IPs and virtual IP addresses can also be changed. Each time a virtual IP address is moved (for example, moving an IP address associated with a virtual machine to another virtual machine, or migrating a virtual machine to a new physical host), an API call must be made to the SDN control plane to inform the controller that the IP has moved, so that all participants in the SDN stack, including the packet processor (data plane), can be updated. However, there are types of applications that do not make such API calls. Examples include various on-premises applications and applications provided by various virtualization software vendors, such as VMware.The usefulness of facilitating virtual Layer 2 networking in a virtualized cloud environment lies in enabling support for applications that are not programmed to make API calls, or applications that rely on other Layer 2 networking features, such as non-IP Layer 3 and MAC learning support.

[0145] A virtual Layer 2 network creates a broadcast domain, and members of the broadcast domain learn. Within a virtual Layer 2 domain, any IP can reside on any MAC on any host within that Layer 2 domain, and the system learns using standard Layer 2 networking protocols. The system virtualizes these networking primitives without the need for a central controller to explicitly inform it of the location of MACs and IPs present in that virtual Layer 2 network. This enables the execution of applications requiring low-latency failover, applications that need to support broadcast or multicast protocols to multiple nodes, and legacy applications that do not know how to make API calls to the SDN control plane or API endpoints to determine the location of IP addresses and MAC addresses. Therefore, providing Layer 2 networking capabilities in a virtualized cloud environment is necessary to support features that are not available at the IP Layer 3 level.

[0146] Another technical advantage of providing virtual Layer 2 in a virtualized cloud environment is the ability to support various different Layer 3 protocols (such as IPv4 and IPv6), including non-IP protocols. Tokol can be supported. Existing cloud IaaS providers cannot support these non-IP protocols because they do not provide Layer 2 functionality in their virtualized cloud networks. By providing Layer 2 networking functionality as described in this disclosure, support can be provided to applications that require and rely on the availability of Layer 3 protocols and Layer 2 level functionality.

[0147] The technologies described in this disclosure provide both Layer 3 and Layer 2 functionality in a virtualized cloud infrastructure. As previously stated, Layer 3-based networking provides certain capabilities not provided by Layer 2 networking, particularly scalability. By providing Layer 2 functionality in addition to Layer 3 functionality, it is possible to provide Layer 2 functionality in a more scalable way while leveraging the capabilities provided by Layer 3 (for example, to provide a more scalable solution). For example, virtualizing Layer 3 avoids the need to use broadcast for learning. By providing Layer 3 capabilities while enabling applications that require Layer 2 functionality and applications that cannot function without Layer 2 functionality, and by providing virtualized Layer 2 to support non-IP protocols, etc., it is possible to provide customers with the full flexibility of a virtualized cloud environment.

[0148] The customer has a hybrid environment where both Layer 3 and Layer 2 environments exist, and the virtualized cloud environment can support both of these environments. The customer may have Layer 3 networks such as subnets and / or Layer 2 networks such as VLANs, and these two environments can communicate with each other within the virtualized cloud environment.

[0149] Furthermore, virtualized cloud environments need to support multi-tenancy. Multi-tenancy makes provisioning both Layer 3 and Layer 2 functionalities in the same virtualized cloud environment technically difficult and complex. For example, a Layer 2 broadcast domain must be managed across many different customers within the cloud provider's infrastructure. The embodiments described in this disclosure overcome these technical challenges.

[0150] For virtualization providers (e.g., VMware), a virtualized Layer 2 network that emulates a physical Layer 2 network can run without modifying the workload. Applications provided by such virtualization providers can run on a virtualized Layer 2 network provided by the cloud infrastructure. For example, such an application may include a set of instances that need to run on a Layer 2 network. If a customer wants to lift and shift such applications from their on-premises environment to a virtualized cloud environment, they cannot run these applications in the cloud environment because these applications rely on an underlying Layer 2 network (e.g., Layer 2 networking capabilities used to migrate or move virtual machines to locations where MAC and IP addresses exist) that are not provided by the current virtualized cloud provider. For these reasons, such applications cannot run natively in a virtualized cloud environment. Cloud providers can provide not only virtualized Layer 3 networks but also virtualized Layer 2 networks using the technologies described in this disclosure. This allows such application stacks to run in the cloud environment without modification and to perform nested virtualization in the cloud environment. Customers can run and manage their own Layer 2 applications in the cloud. Application providers do not need to make any software changes to facilitate this. This allows such legacy applications or workloads (e.g., legacy load balancers, legacy applications, KVM, OpenStack, clustering software) to be virtualized without modification. It may be run in a loud environment.

[0151] By providing virtualization Layer 2 functionality as described in this disclosure, the virtualized cloud environment can support a variety of Layer 3 protocols, including non-IP protocols. Yes, it is possible. Taking Ethernet (registered trademark) as an example, it can support various different EtherTypes (the types of Layer 3 packets being transmitted or the Layer 2 header fields containing the expected Layer 3 protocol), including various non-IP protocols. The EtherType is a two-octet field within the Ethernet frame. The Ethernet type is used to indicate which protocol is encapsulated in the frame's payload and is used by the data link layer at the receiving end to determine how to process the payload. The Ethernet type is used as the basis for 802.1Q VLANs, which tag and encapsulate packets from VLANs for multiplexed transmission with other VLAN traffic over Ethernet trunks. Examples of Ethernet types include IPv4, IPv6, Address Resolution Protocol (ARP), AppleTalk®, and IP This includes X, etc. A cloud network that supports Layer 2 protocols can support any Layer 3 protocol. Similarly, if the cloud infrastructure supports Layer 3 protocols, it can support various Layer 4 protocols such as TCP, UDP, and ICMP. If the network is virtualized at Layer 3, it does not depend on Layer 4 protocols. Similarly, if the network is virtualized at Layer 2, it does not depend on Layer 3 protocols. This technology can be extended to support any Layer 2 network, including FDDI, InfiniBand®, etc.

[0152] Therefore, many applications written for physical networks, particularly those running on clusters of computer nodes sharing a broadcast domain, use Layer 2 features that are not supported by L3 virtual networks. The following six examples highlight the complexities that can arise from the lack of Layer 2 networking capabilities.

[0153] (1) MAC and IP assignment without preceding API calls. Network devices and hypervisors (e.g., VMware) were not built for virtual cloud networks. They assume that they can use MACs as long as the MAC is unique, and can either obtain a dynamic address from a DHCP server or use any IP assigned to the cluster. There is often no mechanism that can be configured to inform the control plane of these Layer 2 and Layer 3 address assignments. If the MAC and IP are unknown, the Layer 3 virtual network does not know where to send the traffic.

[0154] (2) Low-latency reassignment of MAC and IP for high availability and live migration. Many on-premises applications reassign IP and MAC using ARP for high availability. When an instance in a cluster or HA pair becomes unresponsive, a newly activated instance reassigns the service IP to its MAC by sending a Gratuitous ARP (GARP) or the service MAC to its interface by sending a Reverse ARP (RARP). Reallocate the host. This is also important when live migrating instances on the hypervisor. When a guest is migrated, the new host must send a RARP so that guest traffic is sent to the new host. This reallocation must not only be done without API calls, but also with very low latency (sub-milliseconds). This cannot be achieved with HTTPS calls to a REST endpoint.

[0155] (3) Interface multiplexing by MAC address. When a hypervisor hosts multiple virtual machines on a single host and all virtual machines are on the same network, guest interfaces are distinguished by MAC address. Therefore, the same virtual interface It is necessary to support multiple MAC addresses on the same interface.

[0156] (4) VLAN support. A single physical virtual machine host must be located on multiple broadcast domains, as indicated by VLAN tags. For example, VMware ESX uses VLANs to separate traffic (for example, a guest virtual machine can communicate on one VLAN, remember on another VLAN, and host virtual machines on yet another VLAN).

[0157] (5) Use of broadcast and multicast traffic. ARP requires L2 broadcasting, and exemplary on-premises applications use broadcast and multicast traffic for cluster and HA purposes.

[0158] (6) Support for non-IP traffic. Because L3 networks require IPv4 or IPv6 headers for communication, they will not work with L3 protocols other than IP. With L2 virtualization, networks within a VLAN become independent of L3 protocols. In this case, the L3 header may be IPv4, IPv6, IPX, or something else, or it may not be present at all.

[0159] As described in this disclosure, a Layer 2 (L2) network can be created within a cloud network. This virtual L2 network includes one or more virtual L2 VLANs (referred to as VLANs in this disclosure). Each VLAN may include multiple compute instances, each compute instance may be associated with at least one L2 virtual interface (e.g., an L2VNIC) and a local switch. In some embodiments, each pair of L2VNICs and switches is hosted on an NVD. The NVD may host multiple such pairs, each pair being associated with a different compute instance. A set of local switches represents a single switch for an emulated VLAN. An L2VNIC represents a set of ports on an emulated single switch. VLANs can connect to other VLANs, Layer 3 (L3) networks, on-premises networks, and / or other networks via a VLAN Switching and Routing Service (VSRS), also referred to in this disclosure as a Real Virtual Router (RVR) or L2VSRS.

[0160] Referring to Figure 6, which shows a schematic diagram of a computing network in one embodiment, VCN602 resides in CSPI601. VCN602 includes multiple gateways for connecting VCN602 to other networks. These gateways include, for example, DRG604 which can connect VCN602 to an on-premises network such as an on-premises data center 606. The gateways may further include gateway 600 which may include, for example, an LPG for connecting VCN602 to another VCN, and / or an IGW and / or NAT gateway for connecting VCN602 to the Internet. The gateways of VCN602 may further include a service gateway 610 for connecting VCN602 to a service network 612. The service network 612 may include one or more databases and / or stores, for example, an autonomous database 614 and / or an object store 616. The service network may include a conceptual network which includes a set of IP ranges, which may be, for example, a public IP range. In some embodiments, these IP ranges may cover some or all of the public services provided by the CSPI601 provider. These services can be accessed, for example, via an internet gateway or NAT gateway. In some embodiments, the service network can be accessed from the local area through a dedicated gateway (service gateway) for accessing the services within the service network. It provides a method for accessing the service. In some embodiments, the backend of these services is, for example, its own private network. This can be implemented. In some embodiments, the service network 612 may include additional databases.

[0161] VCN602 can contain multiple virtual networks. Each of these networks can contain one or more compute instances that can communicate within each network, between networks, or outside of VCN602. One of the virtual networks in VCN602 is L3 subnet 620. L3 subnet 620 is a configuration unit or partition created within VCN602. Subnet 620 can contain a virtual Layer 3 network within the virtualized cloud environment of VCN602, which is hosted on the underlying physical network of CPSI601. Figure 6 shows only one subnet 620, but VCN602 can contain one or more subnets. Each subnet within VCN602 can be associated with a contiguous range of overlay IP addresses (e.g., 10.0.0.0 / 24 and 10.0.1.0 / 24) that do not overlap with other subnets within that VCN and represent an address space subset of the VCN's address space. In some embodiments, this IP address space can be isolated from the address space associated with CPSI601.

[0162] Subnet 620 contains one or more compute instances, specifically a first compute instance 622-A and a second compute instance 622-B. Compute instances 622-A and 622-B can communicate with each other within subnet 620, or with other instances, devices, and / or networks outside subnet 620. Virtual router (VR) 624 enables communication outside subnet 620. VR624 enables communication between subnet 620 and other networks in VCN602. For subnet 620, VR624 represents a logical gateway that enables subnet 620 (i.e., compute instances 622-A and 622-B) to communicate with endpoints on other networks within VCN602 and with other endpoints outside VCN602.

[0163] VCN602 may further include additional networks, specifically one or more L2 VLANs (referred to as VLANs in this disclosure) which are examples of virtual L2 networks. Each of these one or more VLANs may include a virtual Layer 2 network located in the cloud environment of VCN602 and / or hosted by the underlying physical network of CPSI601. In the embodiment of Figure 6, VCN602 includes VLAN A630 and VLAN B640. Each VLAN 630, 640 within VCN602 may not overlap with other networks, such as other subnets or VLANs within the VCN, and may be associated with a contiguous range of overlay IP addresses (e.g., 10.0.0.0 / 24 and 10.0.1.0 / 24) representing an address space subset of the VCN's address space. In some embodiments, the IP address space of this VLAN may be isolated from the address space associated with CPSI601.

[0164] VLANs 630 and 640 can each contain one or more compute instances. Specifically, VLAN A630 can contain, for example, a first compute instance 632-A and a second compute instance 632-B. In some embodiments, VLAN A630 can contain additional compute instances. VLAN B640 can contain, for example, a first compute instance 642-A and a second compute instance 642-B. Each of compute instances 632-A, 632-B, 642-A, and 642-B can have an IP address and a MAC address. Addresses may be assigned or generated in any desired manner. In some embodiments, these addresses may be within the CIDR of the compute instance's VLAN. In some embodiments, these addresses may be arbitrary addresses. In embodiments where a compute instance of a VLAN communicates with an endpoint outside the VLAN, one or both of these addresses are from the VLAN CIDR, but if all communication is intra-VLAN communication, these addresses are not limited to addresses within the VLAN CIDR. In contrast to networks where addresses are assigned by the control plane, the IP addresses and / or MAC addresses of compute instances within a VLAN may be assigned by the users / customers of that VLAN, and these IP addresses and / or MAC addresses may be discovered and / or learned by compute instances within the VLAN according to the learning process described below.

[0165] Each VLAN can include a VLAN Switching and Routing Service (VSRS). Specifically, VLAN A630 includes VSRS A634, and VLAN B640 includes VSRS B644. Each VSRS634, 644 participates in Layer 2 switching and local learning within the VLAN and performs all necessary Layer 3 networking functions, including ARP, NDP, and routing. VSRS performs ARP (a Layer 2 protocol) because it needs to map IP to MAC addresses.

[0166] In these cloud-based VLANs, each virtual interface or virtual gateway may be associated with one or more media access control (MAC) addresses, and one or more MAC addresses may be virtual MAC addresses. For example, one or more compute instances 632-A, 632-B, 642-A, 642-B, and / or one or more service instances within a VLAN, which may be bare metal, VMs, or containers, can communicate directly with each other via a virtual switch. External communication with other VLANs or L3 networks is enabled via VSRS634,644. VSRS634,644 is a distributed service that provides Layer 3 functionality, such as IP routing, to the VLAN network. In some embodiments, VSRS634,644 is a horizontally scalable, highly available routing service that is located at the intersection of IP and L2 networks and can participate in IP routing and L2 learning within a cloud-based L2 domain.

[0167] VSRS634,644 may be distributed across multiple nodes in the infrastructure. The VSRS634,644 functionality is extensible, specifically horizontally. In some embodiments, each node implementing the VSRS634,644 functionality shares and replicates router and / or switch functionality with one another. These nodes can also present themselves as a single VSRS634,644 to all instances within VLAN630,640. VSRS634,644 may be implemented on any virtualization device within CSPI601, specifically on a virtual network. Therefore, in some embodiments, VSRS634,644 may be implemented on any virtual network virtualization device, including NICs, smart NICs, switches, smart switches, or general-purpose compute hosts.

[0168] VSRS634,644 may reside on one or more hardware nodes, such as one or more x86 servers, or on one or more networking devices, such as one or more NICs, particularly one or more smart NICs, and may be a service that supports a cloud network. In some embodiments, VSRS634,644 may be implemented on a group of servers. Therefore, VSRS634,644 may be a service distributed across a group of nodes. This group of nodes may handle routing and other networking. The security policies are evaluated and participate in and share L2 and L3 learning, and may be a centrally managed group of virtual networking enforcers, or distributed at the edges of a group of virtual networking enforcers. In some embodiments, each VSRS instance can update other VSRS instances with new mapping information learned by one VSRS instance. For example, if one VSRS instance learns the IP, interface, and / or MAC mappings of one or more CIs in a VLAN, that VSRS instance can provide its updated information to other VSRS instances in the VCN. Through this mutual updating, a VSRS instance associated with a first VLAN can know the mappings, including the IP, interface, and / or MAC mappings of CIs in other VLANs, and in some embodiments, the CIs of other VLANs in VCN602. These updates can be significantly accelerated if the VSRS resides on a group of servers and / or is distributed across a group of nodes.

[0169] In some embodiments, VSRS634, 644 can host one or more higher-level networking services, including but not limited to DHCP relay, DHCP (hosting), DHCPv6, IPv6 neighbor discovery protocols, DNS, hosting DNSv6, SLAAC for IPv6, NTP, metadata services, and blockstore mount points. In some embodiments, VSRS can support one or more Network Address Translation (NAT) functions for translating the network address space. In some embodiments, VSRS can incorporate anti-spoofing, anti-MAC spoofing, IPv4 ARP cache poisoning protection, IPv6 route advertisement (RA) protection, DHCP protection, packet filtering with access control lists (ACLs), and / or reverse path forwarding checks. VSRS can implement functions including, for example, ARP, GARP, packet filtering (ACLs), DHCP relay, and / or IP routing protocols. VSRS634 and 644 can, for example, learn MAC addresses, invalidate expired MAC addresses, process MAC address migrations, retrieve MAC address information, process MAC information flooding, process storms, prevent loops, perform Layer 2 multicast via protocols such as IGMP in the cloud, collect statistics including logs and statistics using SNMP, and / or monitor, collect, and use statistics such as broadcast, total traffic, bits, spanning tree packets, etc.

[0170] In a virtual network, VSRS634,644 can appear as different instantiations. In some embodiments, each of these VSRS instantiations can be associated with VLAN630,640, and in some embodiments, each VLAN630,640 can have an instantiation of VSRS634,644. In some embodiments, each instantiation of VSRS634,644 can have one or more unique tables corresponding to the VLAN630,640 to which the VSRS634,644 instantiation is associated. Each instantiation of VSRS634,644 can generate and / or curate unique tables related to the VSRS634,644 instantiation. Thus, a single service can provide VSRS634,644 functionality to one or more cloud networks, but individual instantiations of VSRS634,644 within a cloud network can have their own Layer 2 and Layer 3 forwarding tables, and multiple such customer networks can have overlapping Layer 2 and Layer 3 forwarding tables.

[0171] In some embodiments, VSRS634,644 spans multiple tenants It can support competing VLANs and IP spaces. This includes having multiple tenants on the same VSRS634,644. In some embodiments, some or all of these tenants can selectively use some or all of the same IP address space, the same MAC space, and the same VLAN space. This provides users with a very high degree of flexibility in choosing addresses. In some embodiments, this multi-tenancy is supported by providing each tenant with a separate virtual network, which is a private network within the cloud network. Each virtual network is given a unique identifier. Similarly, in some embodiments, each host may have a unique identifier, and / or each virtual interface or virtual gateway may have a unique identifier. In some embodiments, these unique identifiers, specifically the unique identifiers of the tenant's virtual network, can be encoded in each communication. By assigning a unique identifier to each virtual network and including it in the communication, a single instance of VSRS634,644 can serve multiple tenants with overlapping addresses and / or namespaces.

[0172] VSRS634, 644 can facilitate and / or enable the creation of L2 networks within VLANs 630, 640 and / or communication with such L2 networks by performing these switching and / or routing functions. VLANs 630, 640 may be located in a cloud computing environment, more specifically, in a virtual network within that cloud computing environment.

[0173] For example, each of VLANs 630 and 640 includes multiple compute instances 632-A, 632-B, 642-A, and 642-B. VSRS634, 644 enables communication between compute instances in one VLAN 630 or VLAN 640 and compute instances in another VLAN 630, VLAN 640, or subnet 620. In some embodiments, VSRS634, 644 enables communication between compute instances in one VLAN 630 or VLAN 640 and another network outside the VCN, including another VCN, the Internet, or an on-premises data center. In such embodiments, compute instances, such as compute instance 632-A, can send information to an endpoint outside the VLAN, in this example, an endpoint outside L2VLAN A630. The compute instance (632-A) can send information to VSRS A634, which can then send the information to routers 624, 644 or gateways 604, 608, 610 that are communicatively connected to the desired endpoint. Routers 624, 644 or gateways 604, 608, 610 that are communicatively connected to the desired endpoint can receive information from the compute instance (632-A) and send it to the desired endpoint.

[0174] Referring to Figure 7, which shows a schematic diagram of the logic and hardware of VLAN 700. As shown, VLAN 700 includes multiple endpoints, specifically multiple compute instances and VSRS. Multiple compute instances (CIs) are instantiated on one or more host machines. In some embodiments, this may be a one-to-one relationship where each CI is instantiated on its own host machine, and / or in some embodiments, this may be a many-to-one relationship where multiple CIs are instantiated on a single common host machine. In various embodiments, CIs may be Layer 2 CIs configured to communicate with each other using an L2 protocol. Figure 7 shows a scenario where several CIs are instantiated on their own host machines and several CIs share a common host machine. As shown in Figure 7, instance 1 (CI1) 704-A is instantiated on host machine 1 702-A, instance 2 (CI2) 704-B is instantiated on host machine 2 702- Instances 3 (CI3) 704-C and 4 (CI4) 704-D are instantiated on B, while instances 3 (CI3) 704-C and 4 (CI4) 704-D are instantiated on the common host machine 702-C.

[0175] Each of the CI704-A, 704-B, 704-C, and 704-D is communicatively connected to other CI704-A, 704-B, 704-C, and 704-D within VLAN 700 and communicatively connected to VSRS714. Specifically, each of the CI704-A, 704-B, 704-C, and 704-D is connected to other CI704-A, 704-B, 704-C, and 704-D within VLAN 700 and connected to VSRS714 via L2VNICs and switches. Each CI704-A, 704-B, 704-C, and 704-D is associated with its own L2VNIC and switch. The switch may be local, uniquely associated with the L2VNIC, and may be an L2 virtual switch deployed for the L2VNIC. Specifically, CI1 704-A is associated with L2VNIC 1 708-A and switch 1 710-A; CI2 704-B is associated with L2VNIC 2 708-B and switch 710-B; CI3 704-C is associated with L2VNIC 3 708-C and switch 3 710-C; and CI4 704-D is associated with L2VNIC 4 708-D and switch 4 710-D.

[0176] In some embodiments, each L2VNIC708 and associated switch 710 may be instantiated on an NVD706. This instantiation may be a one-to-one relationship, where a single L2VNIC708 and associated switch 710 are instantiated on a specific NVD706, or it may be a many-to-one relationship, where multiple L2VNIC708s and associated switches 710 are instantiated on a single common NVD706. Specifically, L2VNIC1 708-A and switch 1 710-A are instantiated on NVD1 706-A, L2VNIC2 708-B and switch 2 710-B are instantiated on NVD2, and both L2VNIC3 708-C and switch 3 710-C, and L2VNIC4 708-D and switch 710-D are instantiated on a common NVD, i.e., NVD 706-C.

[0177] In some embodiments, the VSRS714 can support competing VLANs and IP spaces across multiple tenants. This may include having multiple tenants on the same VSRS714. In some embodiments, some or all of these tenants may selectively use some or all of the same IP address space, the same MAC space, and the same VLAN space. This can provide users with a very high degree of flexibility when selecting addresses. In some embodiments, this multi-tenancy is supported by providing each tenant with a separate virtual network, which is a private network within the cloud network. Each virtual network (e.g., each VLAN or VCN) is given a unique identifier, such as a VCN identifier which may be a VLAN identifier. This unique identifier may be selected, for example, by the control plane, specifically by the CSPI control plane. In some embodiments, this unique identifier may include one or more bits which may be included in and / or used in packet encapsulation.

[0178] Similarly, in some embodiments, each host may have a unique identifier, and / or each virtual interface or virtual gateway may have a unique identifier. In some embodiments, these unique identifiers, specifically the unique identifiers of a tenant's virtual network, can be encoded in each communication. By assigning a unique identifier to each virtual network and including it in the communication, a single instantiation of VSRS can have multiple tenants with overlapping addresses and / or namespaces. We can provide services to [the target audience].

[0179] In some embodiments, the VSRS714 can determine which tenant a packet belongs to based on the VCN identifier and / or VLAN identifier associated with the communication, specifically based on the VCN identifier and / or VLAN identifier in the VCN header of the communication. In embodiments disclosed herein, communication entering or leaving a VLAN may have a VCN header which may include a VLAN identifier. Based on the VCN header which includes the VLAN identifier, the VSRS can determine the tenant. In other words, the receiving VSRS can determine which VLAN and / or tenant to send the communication to.

[0180] Furthermore, each compute instance belonging to a VLAN (e.g., an L2 compute instance) is given a unique interface identifier that identifies the L2VNIC associated with that compute instance. The interface identifier may be included in the traffic from and / or to the computer instance (e.g., the frame header) and may be used by the NVD to identify the L2VNIC associated with the compute instance. In other words, the interface identifier can uniquely identify the compute instance and / or the L2VNIC associated with it. As shown in Figure 7, switches 710-A, 710-B, 710-C, and 710-D together can form an L2 distributed switch 712, also referred to in this disclosure as a distributed switch 712. From the customer's perspective, each switch 710-A, 710-B, 710-C, and 710-D within the L2 distributed switch 712 is a single switch connected to all CIs in the VLAN. However, a distributed switch that emulates the user experience of a single switch is infinitely scalable and includes a collection of local switches (e.g., switches 710-A, 710-B, 710-C, and 710-D in the example in Figure 7). As shown in Figure 7, each CI operates on a host machine connected to the NVD. For each CI on a host connected to the NVD, the NVD hosts a Layer 2 VNIC and a local switch associated with the compute instance (e.g., an L2 virtual switch located locally in the NVD, associated with the Layer 2 VNIC, and a member or component of the L2 distributed switch 712). The Layer 2 VNIC represents a port for a compute instance within a Layer 2 VLAN. The local switch connects the L2 VNIC to other L2 VNICs (e.g., other ports) associated with other compute instances within the Layer 2 VLAN.

[0181] Each of CI704-A, 704-B, 704-C, and 704-D can communicate with other CI704-A, 704-B, 704-C, and 704-D within VLAN 700, or with VSRS714. One of CI704-A, 704-B, 704-C, and 704-D sends a packet to another of CI704-A, 704-B, 704-C, and 704-D or to VSRS714 by sending the packet to the MAC address and interface of the receiving CI of one of CI704-A, 704-B, 704-C, and 704-D or to VSRS714. The MAC address and interface identifier may be included in the packet header. As explained above, the interface identifier can indicate the L2VNIC of one of the receiving CIs, CI704-A, 704-B, 704-C, and 704-D, or the L2VNIC of the VSRS714.

[0182] In one embodiment, CI1 704-A may be source C1, L2VNIC708-A may be source VNIC, and switch 710-A may be source switch. In this embodiment, CI3 704-C may be destination CI, and L2VNIC3 708-C may be destination VNIC. The source CI can transmit packets including the source MAC address and destination MAC address. This packet may be intercepted by the NVD706-A that instantiates the source VNIC and source switch.

[0183] Each of the L2VNICs 708-A, 708-B, 708-C, and 708-D of VLAN 700 can learn the MAC address-to-interface identifier mapping of the L2VNIC. This mapping can be learned based on packets and / or information received from VLAN 700. Based on this predetermined mapping, the source VNIC can determine the interface identifier of the destination interface associated with the destination CI within the VLAN and encapsulate the packet. In some embodiments, this encapsulation can include GENEVE encapsulation, specifically L2GENEVE encapsulation, which includes encapsulation of the packet's L2(Ethernet®) header. The encapsulated packet can identify the destination MAC, destination interface identifier, source MAC, and source interface identifier.

[0184] The source VNIC can send the encapsulated packet to the source switch, which can then send the packet to the destination VNIC. Upon receiving the packet, the destination VNIC can decapsulate it and then provide the packet to the destination CI.

[0185] Referring to Figure 8, a logical schematic diagram of multiple connected L2VLANs 800 is shown. In the particular embodiment shown in Figure 8, both VLANs are located in the same VCN. As shown in the figure, the multiple connected L2VLANs 800 are VLANs that are the first VLAN. It can include VLAN A802-A and a second VLAN, VLAN B802-B. Each of these VLANs 802-A and 802-B can contain one or more CIs, each of which can have associated L2VNICs and associated L2 virtual switches. Additionally, each of these VLANs 802-A and 802-B can contain a VSRS.

[0186] Specifically, VLAN A 802-A can include instance 1 804-A connected to L2VNIC1 806-A and switch 1 808-A, instance 2 804-B connected to L2VNIC2 806-B and switch 808-B, and instance 3 804-C connected to L2VNIC3 806-C and switch 3 808-C. VLAN B 802-B can include instance 4 804-D connected to L2VNIC4 806-D and switch 4 808-D, instance 5 804-E connected to L2VNIC5 806-E and switch 808-E, and instance 6 connected to L2VNIC6 806-F and switch 3 808-F. It can include 804-F. Also, VLAN A 802-A is VSRS A VLAN B 802-B may include VSRS B 810-B, and VLAN B 802-B may include VSRS B 810-B. Each of the CI804-A, 804-B, and 804-C of VLAN A 802-A may be communicatively connected to VSRS A 810-A, and each of the CIS804-D, 804-E, and 804-F of VLAN B 802-B may be communicatively connected to VSRS B 810-B.

[0187] VLAN A 802-A may be communicably connected to VLAN B 802-B via the corresponding VSRS 810-A, 810-B. Similarly, each VSRS may be connected to a gateway 812, which can provide access from CI 804-A, 804-B, 804-C, 804-D, 804-E, and 804-F within each VLAN 802-A, 802-B to other networks outside the VCN where VLANs 802-A and 802-B are located. In some embodiments, these The network can include, for example, one or more on-premises networks, another VCN, a service network, and a public network such as the internet.

[0188] Each of the CI804-A, 804-B, and 804-C within VLAN A 802-A connects to the VLAN via the VSRS810-A and 810-B of each VLAN 802-A and 802-B. CI804-D, 804-E, and 804-F within VLAN 802-B can communicate with each other. For example, one of CI804-A, 804-B, 804-C, 804-D, 804-E, or 804-F located in one of VLANs 802-A or 802-B can send a packet to CI804-A, 804-B, 804-C, 804-D, 804-E, or 804-F located in the other VLAN 802-A or 802-B. This packet may be sent from the source VLAN via the source VLAN's VSRS, received by the destination VLAN, and forwarded to the destination CI via the destination VSRS.

[0189] In one embodiment, C1 1 804-A may be a source C1, L2VNIC 806-A may be a source VNIC, and switch 808-A may be a source switch. In this embodiment, C1 5 804-E may be a destination CI, and L2VNIC5 806-E may be a destination VNIC. VSRS A 810-A may be a source VSRS identified as an SVSRS, and VSRS B 810-B may be a destination VSRS identified as a DVSRS.

[0190] The source CI can send a packet containing a MAC address. This packet may be intercepted by the NVD and source switch that instantiate the source VNIC. The source VNIC encapsulates the packet. In some embodiments, this encapsulation may include Geneve encapsulation, specifically L2Geneve encapsulation. The encapsulated packet can identify the destination address of the destination CI. In some embodiments, this destination address may also include the destination address of the destination VSRS. The destination address of the destination CI may include the destination IP address, the destination MAC address of the destination CI, and / or the destination interface identifier of the destination VNIC of the destination CI. The destination address of the destination VSRS may include the IP address of the destination VSRS, the interface identifier of the destination VNIC associated with the destination VSRS, and / or the MAC address of the destination VSRS.

[0191] The source VSRS can receive packets from the source switch, look up the VNIC mapping from the packet's destination address (which may be the destination IP address), and forward the packets to the destination VSRS. The destination VSRS can receive the packets. Based on the destination address contained in the packets, the destination VSRS can forward the packets to the destination VNIC. The destination VNIC can receive and decapsulate the packets and then provide them to the destination CI.

[0192] Referring to Figure 9, which shows a logical schematic diagram of multiple connected L2 VLANs and subnets 900. In the particular embodiment shown in Figure 9, both the VLANs and subnets are located in the same VCN. This indicates that the virtual routers and VSRSs for both the VLANs and subnets are connected directly, rather than through a gateway.

[0193] As shown in the diagram, this can include a first VLAN, VLAN A 902-A, a second VLAN, VLAN B 902-B, and subnet 930. Each of these VLANs 902-A and 902-B can contain one or more CIs, each of which can contain associated L2VNICs and associated L2 switches. Also, each of these VLANs 902-A and 902-B can contain a VSRS. Subnet 930, which may be an L3 subnet, can contain one or more CIs, each CI can contain an associated L3 VNIC, and L3 subnet 930 can contain a virtual router 916.

[0194] Specifically, VLAN A 902-A can include instance 1 904-A connected to L2VNIC1 906-A and switch 1 908-A, instance 2 904-B connected to L2VNIC2 906-B and switch 908-B, and instance 3 904-C connected to L2VNIC3 906-C and switch 3 908-C. VLAN B 902-B can include instance 4 904-D connected to L2VNIC4 906-D and switch 4 908-D, instance 5 904-E connected to L2VNIC5 906-E and switch 908-E, and instance 6 connected to L2VNIC6 906-F and switch 3 908-F. It can include 904-F. VLAN A 902-A can further include VSRS A 910-A, and VLAN B 902-B can include VSRS B 910-B. Each of the CIs 904-A, 904-B, and 904-C of VLAN A 902-A may be communicatively connected to VSRS A 910-A, and each of the CIS 904-D, 904-E, and 904-F of VLAN B 902-B may be communicatively connected to VSRS B 910-B. L3 subnet 930 can include one or more CIs, specifically instance 7 904-G which is communicatively connected to L3 VNIC 7 906-G. L3 subnet 930 can include virtual router 916.

[0195] VLAN A 902-A may be communicably connected to VLAN B 902-B via the corresponding VSRS 910-A, 910-B. L3 subnet 930 may be communicably connected to VLAN A 902-A and VLAN B 902-B via virtual router 916. Similarly, virtual router 916 and each of VSRS instances 910-A, 910-B may be connected to gateway 912, which can provide access from CI 904-A, 904-B, 904-C, 904-D, 904-E, 904-F, 904-G within each VLAN 902-A, 902-B and subnet 930 to other networks outside the VCN where VLAN 902-A, 902-B and subnet 930 are located. In some embodiments, these networks may include, for example, one or more on-premises networks, another VCN, a service network, or a public network such as the Internet.

[0196] Each VSRS instance 910-A, 910-B can provide a transmission path for packets leaving the associated VLANs 902-A, 902-B and a reception path for packets entering the associated VLANs 902-A, 902-B. Packets may be sent from VSRS instances 910-A, 910-B in VLANs 902-A, 902-B to any desired endpoint, including L2 endpoints in the same VCN, a different VCN, or another VLAN on the network, such as L2 CIs, and / or L3 endpoints in the same VCN, a different VCN, or a subnet on the network, such as L3 CIs.

[0197] In one embodiment, CI1 904-A may be a source C1, L2VNIC 906-A may be a source VNIC, and switch 908-A may be a source switch. In this embodiment, CI7 904-G may be a destination CI, and VNIC7 906-G may be a destination VNIC. VSRS A 910-A may be a source VSRS identified as an SVSRS, and virtual router (VR) 916 may be a destination VR.

[0198] The source CI can send packets containing the MAC address. This may be intercepted by the NVD and source switch that instantiate the source VNIC. The source VNIC encapsulates the packet. In some embodiments, this encapsulation may include Geneve encapsulation, specifically L2Geneve encapsulation. The encapsulated packet can identify the destination address of the destination CI. In some embodiments, this destination address may also include the destination address of the VSRS of the VLAN of the source CI. The destination address of the destination CI may include the destination IP address, the destination MAC address of the destination CI, and / or the destination interface identifier of the destination VNIC of the destination CI.

[0199] The source VSRS can receive packets from the source switch, look up the VNIC mapping from the packet's destination address (which may be the destination IP address), and forward the packet to the destination VR. The destination VR can receive the packets. Based on the destination address contained in the packets, the destination VR can forward the packets to the destination VNIC. The destination VNIC can receive and decapsulate the packets and then provide them to the destination CI.

[0200] Learning within a virtual L2 network Referring to Figure 10, a schematic diagram of one embodiment of intra-VLAN communication and learning within VLAN 1000 is shown. This learning involves L2VNIC and VSRS. This is specific to how VNICs and / or L2 virtual switches learn associations between MAC addresses and L2VNICs / VSRS VNICs (more specifically, between MAC addresses associated with L2 compute instances or VSRSs and interface identifiers associated with the L2VNICs or VSRS VNICs of those L2 compute instances). Generally, learning is based on ingress traffic. In the case of interface-versus-MAC address learning, This learning process is different from the learning process (e.g., the ARP process) that an L2 compute instance performs to learn the destination MAC address. The two learning processes (e.g., L2VNIC / L2 virtual switch and L2 compute instance) are shown to be performed jointly in Figure 12.

[0201] As shown in the diagram, VLAN 1000 includes compute instance 1 1000-A, which is communicatively connected to NVD1 1001-A, which instantiates L2VNIC1 1002-A and L2 switch 1 1004-A. VLAN 1000 also includes compute instance 2 1000-B, which is communicatively connected to NVD2 1001-B, which instantiates L2VNIC2 1002-B and L2 switch 2 1004-A. Furthermore, VLAN 1000 operates on a server cluster and includes VSRS1010, which contains VSRS VNIC1002-C and VSRS switch 1004-C. Switches 1004-A, 1004-B, and 1004-C all together form a distributed switch. VSRS1010 is communicatively connected to endpoint 1008. Endpoint 1008 may include a gateway, specifically an L2 / L3 router in the form of another VSRS, or an L3 router in the form of a virtual router.

[0202] The control plane 1001 of the VCN hosting VLAN 1000 holds information to identify each L2VNIC on VLAN 1000 and the network configuration of the L2VNICs. For example, this information may include the interface identifier associated with the L2VNIC and / or the physical IP address of the NVD hosting the L2VNIC. The control plane 1001 uses this information to update the interfaces within VLAN 1000 (e.g., periodically or on demand). Thus, each L2VNIC 1002-A, 1002-B, 1002-C in the VLAN receives information from the control plane 1001 to identify the interface within the VLAN, and this information The tables are populated. The tables populated by the L2VNIC may be stored locally on the NVD hosting the L2VNIC. If L2VNICs 1002-A, 1002-B, and 1002-C already contain the current tables, they can determine any discrepancies between their current tables and the information / table received from the control plane 1001. In some embodiments, L2VNICs 1002-A, 1002-B, and 1002-C can update their tables to match the information received from the control plane 1001.

[0203] As shown in Figure 10, packets are transmitted via L2 switches 1004-A, 1004-B, and 1004-C and received by receiving L2 VNICs 1002-A, 1002-B, and 1002-C. When a packet is received by L2 VNICs 1002-A, 1002-B, and 1002-C, the VNIC learns the mapping of the packet's source interface (source VNIC) and source MAC address. Based on a table of information received from the control plane 1010, the VNIC can map the source MAC address (from the received packet, also referred to in this disclosure as a frame) to the interface identifier of the source VNIC, the IP address of the VNIC, and / or the IP address of the NVD hosting the VNIC (the interface identifier and IP address are available from the table). Therefore, L2VNICs 1002-A, 1002-B, and 1002-C learn interface identifier-to-MAC address mappings based on received information and / or packets, and L2VNICs 1002-A, 1002-B, and 1002-C can use this learned mapping information to update tables, i.e., L2 forwarding tables 1006-A, 1006-B, and 1006-C. In some embodiments, the L2 forwarding table includes MAC addresses, associating each MAC address with at least one of either an interface identifier or a physical IP address. In such embodiments, the MAC address is an address assigned to an L2 compute instance, corresponding to a port emulated by the L2VNIC associated with the L2 compute instance. The interface identifier can uniquely identify the L2VNIC and / or L2 compute instance. The virtual IP address may be the virtual IP address of the L2VNIC. The physical IP address may be the IP address of the NVD hosting the L2VNIC. The L2 forwarding table updated by the L2VNIC is stored locally on the NVD hosting the L2VNIC and can be used by the L2 virtual switch associated with the L2VNIC to send frames.In some embodiments, L2VNICs within a common VLAN can share all or part of the mapping table.

[0204] The following describes the traffic flow, referring to the network architecture described above. To clarify the explanation, the traffic flow will be described in relation to compute instance 2 1000-B, L2VNIC2 10002-B, L2 switch 2 1004-B, and NVD2 1001-B. This explanation applies similarly to traffic flows to and / or from other compute instances.

[0205] As described above, VLANs are implemented in a VCN as overlay L2 networks on the L3 physical network. An L2 compute instance of a VLAN can send or receive L2 frames that include an overlay MAC address (also called a virtual MAC address) as the source MAC address and destination MAC address. The L2 frame can also encapsulate packets that include an overlay IP address (also called a virtual IP address) as the source IP address and destination IP address. In some embodiments, the compute instance's overlay IP address is the VLAN It can belong to the CIDR range. Other overlay IP addresses can be located within the CIDR range (in which case L2 frames flow within the VLAN) or outside the CIDR range (in which case L2 frames are sent to or received from other networks). L2 frames can include a VLAN tag to uniquely identify the VLAN. This VLAN tag can be used to distinguish multiple L2VNICs on the same NVD. L2 frames may be received via a tunnel as packets encapsulated by the NVD from the host machine of a compute instance, from another NVD, or from a group of servers hosting a VSRS. In these different cases, the encapsulated packet may also be an L3 packet transmitted over the physical network, and the source and destination IP addresses are physical IP addresses. Different types of encapsulation are possible, including Geneve encapsulation. The NVD can decapsulate a received packet to extract an L2 frame. Similarly, the NVD can encapsulate an L2 frame into an L3 packet and transmit it over the physical board in order to transmit the L2 frame.

[0206] For intra-VLAN outbound traffic from compute instance 2 1000-B, NVD2 1001-B receives frames from the host machine of instance 2 1000-B via the Ethernet® link. The frames contain an interface identifier to identify L2VNIC2 1000-B. This frame contains the overlay MAC address of compute instance 2 1000-B (e.g., M.2) as the source MAC address and the overlay MAC address of compute instance 1 1000-A (e.g., M.1) as the destination MAC address. Given the interface identifier, NVD2 1001-B sends the frame to L2VNIC2 for further processing. The frame is passed to 1002-B. L2VNIC2 1002-B forwards the frame to L2 switch 2 1004-B. L2 switch 2 1004-B determines, based on the L2 forwarding table 1006-B, whether the destination MAC address is known (for example, whether it matches an entry in the L2 forwarding table 1006-B).

[0207] If known, L2 switch 2 1004-B determines that L2VNIC1 1002-A is the relevant tunnel endpoint and forwards the frame to L2VNIC1 1002-A. This forwarding may involve encapsulating the frame into a packet and decapsulating the packet (e.g., Geneve encapsulation and decapsulation), where the packet contains the frame, the physical IP address of NVD1 1001-A as the destination address (e.g., IP.1), and the physical IP address of NVD2 1001-B as the source address (e.g., IP.2).

[0208] If unknown, L2 switch 2 1004-B broadcasts the frame to various L2VNICs in the VLAN (including, for example, L2VNIC1 1002-A and any other L2VNICs in the VLAN). The broadcasted frame is processed (e.g., encapsulated, transmitted, decapsulated) among the relevant NVDs. In some embodiments, this broadcast is performed in a physical network, and more specifically, emulated. This physical network can encapsulate the frame to each L2VNIC, including the VSRS of the VLAN. Thus, the broadcast is emulated in the physical network via a series of replicated unicast packets. Each L2VNIC then receives the frame and learns the association between the interface identifier of L2VNIC2 1002-B and the source MAC address (e.g., M.2) and source physical IP address (e.g., IP.2).

[0209] For inbound traffic within a VLAN from compute instance 1 1000-A to compute instance 2 1000-B, NVD2 1001-B receives packets from NVD1. The packet contains IP.1 as the source address and a frame, and the frame contains M.2 as the destination MAC address and M.1 as the source MAC address. The frame also contains the network identifier of L2VNIC1 1002-A. During decapsulation, L2VNIC2 receives the frame, learns that this interface identifier is associated with M.1 and / or IP.1, and if previously unknown, stores this learned information in switch 2's L2 forwarding table 1006-B for subsequent outgoing traffic. Alternatively, during decapsulation, L2VNIC2 receives the frame, learns that this interface identifier is associated with M.1 and / or IP.1, and if this information is known, refreshes the expiration time.

[0210] For outbound traffic sent from instance 2 1000-B in VLAN 1000 to an instance in another VLAN, the flow is similar to the outbound traffic flow described above, except that a VSRS VNIC and VSRS switch are used. Specifically, the destination MAC address is not located within the L2 broadcast of VLAN 1000 (it is located within another L2 VLAN). Therefore, this outbound traffic is sent using the destination instance's overlay destination IP address (e.g., IP.A). For example, L2VNIC2 1002-B determines that IP.A is outside the CIDR range of VLAN 1000. Therefore, L2VNIC2 1002-B sets the destination MAC address to the default gateway MAC address (e.g., M.DG). L2 switch 2 1004-B, based on M.DG, sends the outbound traffic to the VSRS VNIC (e.g., via a tunnel with appropriate end-to-end encapsulation). The VSRS VNIC forwards the outbound traffic to the VSRS switch. The VSRS switch performs routing functions. The VSRS switch on VLAN 1000 sends the outgoing traffic (e.g., through a virtual router between these two VLANs, with appropriate end-to-end encapsulation) to the VSRS switch on the other VLAN based on the overlay destination IP address (e.g., IP.A). The VSRS switch on the other VLAN then determines that IP.A is within the CIDR range of this VLAN and performs its switching function, determining the destination MAC address associated with IP.A by searching its ARP cache based on IP.A. If no match is found in the ARP cache, it determines the destination MAC address by sending an ARP request to a different L2VNIC on the other VLAN. Otherwise, the VSRS switch sends the outgoing traffic (e.g., through a tunnel, with appropriate encapsulation) to the relevant VNIC.

[0211] For incoming traffic from an instance in another VLAN to an instance in VLAN 1000, the traffic flow is the same as above, except that it is in the reverse direction. For outgoing traffic from an instance in VLAN 1000 to the L3 network, the traffic flow is the same as above, except that the VSRS switch in VLAN 1000 routes the packet directly to the destination VNIC in the virtual L3 network via the virtual router (without routing the packet through another VSRS switch, for example). For incoming traffic from the virtual L3 network to an instance in VLAN 1000, the traffic flow is the same as above, except that the VSRS switch in VLAN 1000A that sent the packet as a frame within the VLAN receives the packet. For (outgoing or incoming) traffic between VLAN 1000 and other networks, the VSRS switch similarly uses its routing function to send packets through the appropriate gateway (e.g., IGW, NGW, DRG, SGW, LPG) in the case of outgoing traffic, and uses its switching function to send frames within VLAN 1000 in the case of incoming traffic.

[0212] Referring to Figure 11, a schematic diagram of one embodiment of VLAN 1100 (for example, a cloud-based virtual L2 network) is shown, specifically an implementation diagram of the VLAN. .

[0213] As described above in this disclosure, a VLAN may include n compute instances 1102-A, 1102-B, 1102-N, each operating on a host machine. As previously stated, there may be a one-to-one association between a compute instance and a host machine, or there may be a many-to-one association between multiple compute instances and a single host machine. Each compute instance 1102-A, 1102-B, 1102-N may be an L2 compute instance and is associated with at least one virtual interface (e.g., L2VNIC) 1104-A, 1104-B, 1104-N and switches 1106-A, 1106-B, 1106-N. The switches 1106-A, 1106-B, 1106-N are L2 virtual switches and together form an L2 distributed switch 1107.

[0214] Pairs of L2VNICs 1104-A, 1104-B, 1104-N and switches 1106-A, 1106-B, 1106-N associated with compute instances 1102-A, 1102-B, 1102-N on the host machine are pairs of software modules on NVDs 1108-A, 1108-B, 1108-N connected to the host machine. Each L2VNIC 1104-A, 1104-B, 1104-N represents an L2 port of a single customer-recognized switch (referred to as a v-switch in this disclosure). Generally, host machine "i" runs compute instance "i" and is connected to NVD "i". Similarly, NVD "i" runs L2VNIC "i" and switch "i". L2VNIC "i" represents L2 port "i" of the v-switch, where i is a positive integer between 1 and n. Here, we've described a one-to-one association, but other types of associations are also possible. For example, a single NVD can be connected to multiple hosts, each host running one or more compute instances belonging to a VLAN. In this case, the NVD hosts multiple pairs of L2VNICs and switches, each corresponding to one of the compute instances.

[0215] A VLAN may contain an instance of VSRS1110. VSRS1110 performs switching and routing functions and includes instances of VSRS VNIC1112 and VSRS switch 1114. VSRS VNIC1112 represents a port on the v-switch, which connects the v-switch to other networks via a virtual router. As shown in the diagram, VSRS1110 may be instantiated on server cluster 1116.

[0216] The control plane 1118 can track information to identify L2VNICs 1104-A, 1104-B, and 1104-N within the VLAN and their placement. The control plane 1110 can also provide this information to interfaces 1104-A, 1104-B, and 1104-N within the VLAN.

[0217] As shown in Figure 11, the VLAN may be a cloud-based virtual L2 network that can be built on the physical network 1120. In some embodiments, this physical network 1120 may include NVD1108-A, 1108-B, and 1108-N.

[0218] Generally, a first L2 compute instance in a VLAN (e.g., compute instance 1 1102-A) can communicate with a second compute instance in the VLAN (e.g., compute instance 2 1102-B) using an L2 protocol. For example, frames may be sent between the two L2 compute instances over the VLAN. Nevertheless, frames may be encapsulated, tunneled, routed, and / or processed by other means so as to be sent over the underlying physical network 1120.

[0219] For example, compute instance 1 1102-A sends a frame to compute instance 2 1102-B. Depending on the network connections between host machine 1 and NVD1, between NVD1 and physical network 1120, between physical network 1120 and NVD2, and between NVD2 and host machine 2 (e.g., TCP / IP connection, Ethernet® connection, tunneling connection), different types of processing may be applied to the frame. For example, this processing is repeated until the frame is received by NVD1, encapsulated, and reaches compute instance 2. This processing assumes that the frame can be transmitted between underlying physical resources, and for brevity and clarity, its explanation is omitted from the explanation of VLANs and related L2 operations.

[0220] Communication on a virtual L2 network Multiple types of communication can take place within or between virtual L2 networks. These communications can include intra-VLAN communications. In such embodiments, a source compute instance can send packets to a destination compute instance located in the same VLAN as the source compute instance (CI). The communication may further include sending packets to an endpoint outside the VLAN of the source CI. The communication may include, for example, communication from a source CI in a first VLAN to a destination CI in a second VLAN, communication from a source CI in a first VLAN to a destination CI in an L3 subnet, and / or communication from a source CI in a first VLAN to a destination CI outside the VCN containing the VLAN of the source CI. The communication may further include, for example, the destination CI receiving information from a source CI outside the VLAN of the destination CI. This source CI may be located in another VLAN, in an L3 subnet, or outside the VCN containing the VLAN of the source CI.

[0221] Each CI within a VLAN can play an active role in the traffic flow. This includes learning interface identifiers versus MAC addresses (also referred to in this disclosure as interface versus MAC address), mapping instances within the VLAN to maintain the L2 forwarding table within the VLAN, and sending and / or receiving communication packets. VSRS can play an active role in communication within the VLAN and with source or destination CIs outside the VLAN. VSRS can reside within and enable transmission and reception in both L2 and L3 networks.

[0222] Intra-VLAN Communication Referring to Figure 12, which is a flowchart illustrating one embodiment of process 1200 for performing intra-VLAN communication. In some embodiments, process 1200 may be performed by a compute instance in a common VLAN. This process is specifically performed when a source CI sends a packet to a destination CI in a VLAN, but does not know the IP-to-MAC address mapping of that destination CI. This can occur, for example, when a source CI sends a packet to a destination CI that has an IP address in a VLAN, but the source CI does not know the MAC address of that IP address. In this case, the destination MAC address and IP-to-MAC address mapping can be learned by performing an ARP process.

[0223] If the source CI knows the IP-to-MAC address mapping, the source CI can send the packet directly to the destination CI without having to perform the ARP process. In some embodiments, this packet may be intercepted by the source VNIC, which is the L2VNIC during intra-VLAN communication. The source VNIC knows the interface-to-MAC address mapping of the destination MAC address. For example, packets can be encapsulated using L2 encapsulation, and the encapsulated packets can be forwarded to the destination VNIC. This destination VNIC is the L2 VNIC for the destination MAC address during intra-VLAN communication.

[0224] If the source VNIC does not know the interface-to-MAC address mapping of the MAC address, it can perform one interface-to-MAC address learning process. This learning process may include the source VNIC sending a packet to all interfaces in the VLAN. In some embodiments, this packet may be sent to all interfaces in the VLAN via broadcast. In some embodiments, this broadcast may be implemented on the physical network in the form of serial unicast. This packet may include the destination MAC address, destination IP address, the source VNIC's interface, MAC address, and IP address. Each VNIC in the VLAN can receive this packet and learn the source VNIC's interface-to-MAC address mapping.

[0225] Furthermore, each receiving VNIC can decapsulate a packet and forward the decapsulated packet to the associated CI. Each CI may include a network interface that can evaluate the forwarded packet. If the network interface determines that the CI receiving the forwarded packet does not match the destination MAC and / or IP address, the packet is discarded. If the network interface determines that the CI receiving the forwarded packet does match the destination MAC and / or IP address, the packet is received by the CI. In some embodiments, a CI having a MAC and / or IP address that matches the destination MAC and / or IP address of a packet can send a response to the source CI, thereby allowing the source VNIC to learn the interface-to-MAC address mapping of the destination CI, and thus allowing the source CI to learn the IP-to-MAC address mapping of the destination CI.

[0226] If the source CI does not know the IP-to-MAC address mapping, or if the source CI's IP-to-MAC address mapping to the destination CI is outdated, process 1200 can be executed.

[0227] Therefore, if the IP-to-MAC address mapping is known, the source CI can send the packet. If the IP-to-MAC address mapping is unknown, process 1200 can be executed. If the interface-to-MAC address mapping is unknown, the interface-to-MAC address learning process outlined above can be executed. If the interface-to-MAC address mapping is known, the VNIC can send the packet to the destination CI.

[0228] Process 1200 begins in block 1202. In block 1202, the source CI determines that the IP-to-MAC address mapping of the destination CI is unknown to the source CI. In some embodiments, this may include the source CI determining the destination IP address of the packet and determining that the destination IP address is not associated with any MAC address stored in the source CI's mapping table. Alternatively, the source CI may determine that the IP-to-MAC address mapping of the destination CI is old. In some embodiments, the mapping may be determined to be old if it has not been updated and / or validated within a certain time limit. Once the source CI determines that the IP-to-MAC address mapping of the destination CI is unknown and / or old, the source CI initiates an ARP request for the destination IP and sends the ARP request to Send to Sanet Broadcast.

[0229] In block 1204, the source VNIC, also referred to in this disclosure as the source interface, receives an ARP request from the source CI. The source interface identifies all interfaces on the VLAN and sends the ARP request to all interfaces in the VLAN broadcast domain. As previously stated, the control plane knows all interfaces on the VLAN and provides this information to the interfaces on the VLAN; therefore, the source interface also knows all interfaces in the VLAN and can send the ARP request to each interface in the VLAN. For this purpose, the source interface duplicates the ARP request and encapsulates it for each interface on the VLAN. Each encapsulated ARP request includes the interface identifier of the source CI, the MAC address of the source CI, the IP address of the source CI, the target IP address, and the interface identifier of the destination CI. The source CI interface duplicates the Ethernet broadcast by sending the duplicated and encapsulated ARP requests one by one as serial unicast to each interface in the VLAN.

[0230] In block 1206, all interfaces within the VLAN broadcast domain receive and decapsulate the packet. Since the packet identifies the MAC address, IP address, and interface identifier of the source CI, each interface within the VLAN broadcast domain that receives the packet learns the source VNIC interface-to-MAC address mapping of the source CI (e.g., the mapping of the source interface's interface identifier to the source CI's MAC address). As part of learning the source CI's interface-to-MAC address mapping, each interface can update its mapping table (e.g., the L2 forwarding table) and provide the updated mapping to the relevant switch and / or CI. Each receiving interface, except for the VSRS, can forward the decapsulated packet to the relevant CI. The CI that receives the forwarded and decapsulated packet, specifically its network interface, can determine whether the packet's target IP address matches the CI's IP address. In some embodiments, if the IP address of the CI associated with the interface does not match the IP address of the destination CI specified in the received packet, the CI discards the packet and takes no further action. In the case of VSRS, VSRS can determine whether the target IP address of a packet matches the IP address of VSRS. In some embodiments, if the IP address of VSRS does not match the target IP address specified in the received packet, VSRS discards the packet and takes no further action.

[0231] If the IP address of the destination CI specified in the received packet is determined to match the IP address of the CI (destination CI) associated with the receiving interface, the destination CI sends a response, which may be a unicast ARP response, to the source interface, as shown in block 1208. This response includes the MAC address and IP address of the destination CI, and the IP address and MAC address of the source CI. As described later, the VSRS may send an ARP response if it determines that the target IP address matches the IP address of the VSRS.

[0232] As shown in block 1210, this response is received by the destination interface for encapsulating the unicast ARP response. In some embodiments, this encapsulation may include Geneve encapsulation. The destination interface forwards the encapsulated packet to the source interface via the destination switch. This is possible. The encapsulated packet includes the MAC address, IP address, and interface identifier of the destination CI, and the MAC address, IP address, and interface identifier of the source CI.

[0233] In block 1212, the source interface receives and decapsulates the ARP response. The source interface can also learn the interface-to-MAC address mapping of the destination CI based on the encapsulation and / or the information contained in the encapsulated packet. In some embodiments, the source interface can forward the ARP response to the source CI.

[0234] In block 1214, the source CI receives an ARP response. In some embodiments, the source CI can update its mapping table based on the information contained in the ARP response, specifically, it can update the mapping table to reflect the IP-to-MAC address mapping based on the destination CI's MAC and IP addresses. The source CI can then send a packet to the destination CI that may be any packet, specifically an IP packet, including an IPv4 or IPv6 packet. This packet may include the source CI's MAC and IP addresses as the packet's source MAC and source IP addresses, and the destination CI's MAC and IP addresses as the packet's destination MAC and destination IP addresses.

[0235] In block 1216, the source interface can receive packets from the source CI. The source interface can encapsulate packets, and in some embodiments, it can encapsulate packets using Geneve encapsulation. The source interface can forward the encapsulated packets to the destination CI, specifically to the destination interface. The encapsulated packets may include the IP address, MAC address, and interface identifier of the source CI as the source MAC address, source IP address, and source interface identifier, and may include the MAC address, IP address, and interface identifier of the destination CI as the destination MAC address, destination IP address, and destination interface identifier.

[0236] In block 1218, the destination interface receives the packet from the source interface. The destination interface can decapsulate the packet and then forward it to the destination CI. In block 1220, the destination CI receives the packet from the destination interface.

[0237] Referring to Figure 13, Figure 13 is a schematic diagram 1300 showing process 1200 for performing VLAN intra-region communication. As shown in the diagram, VLAN A 1302 has a VLAN CIDR with 10.0.3.0 / 24. VLAN A 1302 can be instantiated on one or more hardware, specifically including a VSRS VNIC (VRVI) 1304 which can be instantiated on server group 1306. VRVI 1304 may have an IP address of 10.0.3.1. The VLAN may include compute instance 1 (CI1) 1310 which has an IP address of 10.0.3.2 and is communicably connected to L2VNIC 1 (VI1) 1314 and L2 switch 1 (NVD 1 (SN1) 1312 which can be instantiated). The VLAN may include compute instance 2 (CI2) 1320, which has IP address 10.0.3.3 and is communicably connected to L2VNIC2 (VI2) 1324 and L2 switch 2 and NVD2 (SN2) 1322. The VLAN may also include compute instance 3 (SN3) 1332, which has IP address 10.0.3.4 and is communicably connected to L2VNIC3 (VI3) 1334 and L2 switch 3 and NVD3 (SN3) 1332. It can include NC3 (CI3)1330.

[0238] Applying Method 1200 of Figure 12 to the example in Figure 13, CI3 1330 is the source CI, and VI3 1334 is the source interface. Also, CI2 1320 is the destination CI, and VI2 1324 is the destination interface. If CI3 determines that it does not have an IP-to-MAC mapping for the destination IP address (10.0.3.3), CI3 1330 sends an ARP request. The ARP request may be for a known address, specifically for a known IP address of CI2 1320. Therefore, in some embodiments, the ARP request may be for 10.0.3.3.

[0239] This ARP request is received by SN3 1332 and VI3 1334. VI3 1334 creates an ARP request for each CI in VLAN 1302 by duplicating the ARP request. VI3 1334 encapsulates each ARP request and sends it to each interface in the VLAN. These encapsulated ARP requests may include information to identify the MAC address of the source CI (CI3), the interface identifier of the source interface (VI3), and the destination MAC address. These requests may also be sent to each interface in the VLAN, as indicated by arrow 1350. In a VLAN, these requests may also be broadcast ARP requests.

[0240] Interfaces within a VLAN receive encapsulated ARP requests and decapsulate them. Based on the information contained in the ARP request and / or related information, interfaces within the VLAN update their mappings. Specifically, for example, VI1 1314, VI2 1324, and VRVI1304 each receive an ARP request from CI3 1330, decapsulate it, and learn the interface-to-MAC address mapping of the source CI. Both the interface identifier and MAC address of the source CI are included in the encapsulated packet. Furthermore, VRVI1304 can update the IP-to-MAC address mapping of the source CI based on the information contained in the encapsulated packet.

[0241] As indicated by arrow 1352, the interface within the VLAN having a CI with the requested IP address can send an ARP response to CI3 1330. Specifically, as shown in FIG. 13, VI2 1324 is the interface of CI2 1320 and can send an ARP response. The ARP response from CI2 is received by VI2, encapsulated, and sent as an ARP unicast to the requested interface, specifically VI3. As described above in the present application, the sending of this ARP response includes providing the encapsulated ARP response to the relevant switch, and the relevant switch can send the encapsulated ARP response to VI3.

[0242] The ARP response may be received and decapsulated by VI3. VI3 can learn the interface-to-MAC mapping of CI2 based on the received ARP response and provide the learned mapping to the switch associated with VI3. The decapsulated packet may be provided to CI3. CI3 can learn the IP-to-MAC address mapping of CI2 1320 based on the decapsulated packet. CI3 can send the packet to CI2. This packet may be an IP packet such as an IPv4 or IPv6 packet. This packet can include the MAC and IP addresses of CI3 as the source address and the IP and MAC addresses of CI2 as the destination address.

[0243] The packet sent by CI3 may be received by VI3. VI3 can encapsulate the packet and transfer the packet to interface VI2. VI2 can receive the packet, decapsulate the packet, and transfer the packet to CI2.

[0244] Inter-VLAN communication Next, referring to FIG. 14, FIG. 14 is a flowchart showing an embodiment of a process 1400 for performing inter-VLAN communication within a virtual L2 network. Process 1400 may be performed by two connected VLANs, for example, all or part of a plurality of connected L2 VLANs 800 as shown in FIG. 8. In some embodiments, process 1400 may be performed when a computing instance (source CI) within a first VLAN (source VLAN) transmits a packet to a destination computing instance (destination CI) within a second VLAN (destination VLAN). In some embodiments, the source CI may determine that the destination CI is outside the source VLAN based on the IP address of the destination CI. For example, the source CI may determine that the destination IP is outside the source VLAN CIDR. In such a case, the source CI may decide to send the IP packet to the destination CI via the VSRS of the source VLAN. If the source CI already knows the mapping of the VSRS of the first VLAN (source VSRS), the source CI may send the packet directly to the source VSRS. If the source CI does not know the mapping of the source VSRS, the source CI and the associated VNIC (source interface or source VNIC) first learn the mapping of the source VSRS. In an embodiment of inter-VLAN communication, both the source VNIC and the destination VNIC are L2 VNICs. The first steps 1402-1410 of process 1400 relate to the learning of the source VSRS mapping by the source VNIC and the source CI.

[0245] Process 1400 starts at block 1402. At block 1402, the source CI having the destination IP address initiates an ARP request. Using the ARP request, the IP-to-MAC address mapping of the source VSRS is determined. The source CI sends the ARP request as an Ethernet broadcast. The ARP request includes the IP address of the source CI as the source IP address and MAC address.

[0246] In block 1404, the source VNIC receives and replicates the ARP request. Specifically, the source VNIC receives the ARP request from the source CI, identifies all interfaces on the VLAN, and sends the ARP request to all interfaces on the VLAN broadcast domain. As previously mentioned, the control plane knows all interfaces on the VLAN and provides this information to the interfaces within the VLAN; therefore, the source interface also knows all interfaces within the VLAN and can send the ARP request to each interface within the VLAN. For this reason, the source interface replicates the ARP request and encapsulates the ARP request for each interface on the VLAN. Each encapsulated ARP request includes the source CI's interface identifier and the source CI's IP address and MAC address as the source address, and the destination CI's interface identifier and IP address as the destination address. The source CI interface achieves Ethernet broadcasting by sending the replicated and encapsulated ARP requests to each interface within the VLAN via serial unicast. In some embodiments, the source VNIC can encapsulate the ARP requests using Geneve encapsulation.

[0247] In block 1406, all interfaces within the VLAN broadcast domain receive and decapsulate packets. The MAC address of the source CI To identify the source interface identifier, IP address, and source interface, each interface in the VLAN broadcast domain that receives the packet learns the interface-to-MAC address mapping of the source VNIC of the source CI. As part of learning the source CI's interface-to-MAC address mapping, each interface can update its corresponding mapping table and provide the updated mapping table to the relevant switch and / or CI. Each receiving interface, except VSRS, forwards the decapsulated packet to the relevant CI. The CI that receives the forwarded decapsulated packet, specifically its network interface, can determine whether the packet's target IP address matches its own IP address. If the IP address of the CI associated with that interface does not match the IP address of the destination CI specified in the received packet, no further action is taken.

[0248] If the source VSRS determines that the target IP address matches the source VSRS's IP address, it encapsulates a response, which may be a unicast ARP response, as shown in block 1408, and sends it to the source interface. This response includes the source CI's MAC address, IP address, and interface identifier as the destination address. Furthermore, this response includes the source VSRS's MAC address, IP address, and interface identifier as the source address. In some embodiments, the encapsulation of the ARP response may include Geneve encapsulation.

[0249] In block 1410, the source interface receives and decapsulates the ARP response. The source interface can also learn the interface-to-MAC address mapping of the source VSRS based on the encapsulation and / or the information contained in the encapsulated packet. In some embodiments, the source interface can forward the ARP response to the source CI.

[0250] In block 1412, the source CI receives an ARP response. In some embodiments, the source CI can update its mapping table based on the information contained in the ARP response, specifically based on the MAC address and IP address of the source VSRS. In some embodiments, for example, the source CI can update its mapping table to reflect the IP-to-MAC address mapping of the source VSRS. The source CI can then send a packet to the source VSRS, which may be any packet, specifically an IP packet, specifically an IPv4 or IPv6 packet. In some embodiments, this may include sending an IP packet that includes the IP address of the destination CI as the destination address. In some embodiments, the IP address of the destination CI may be included in the packet header, for example, the L3 header of the packet. This header may further include the MAC address of the source VSRS in another header of the packet, for example, the L2 header of the packet. The packet may further include the MAC address of the source CI and the IP address of the source CI as the source MAC address and source IP address.

[0251] In block 1414, the source interface can receive packets from the source CI. The source interface can encapsulate the packets. The source interface can forward the encapsulated packets to the source VSRS, specifically to the VNIC of the source VSRS. In addition to the packet's address, the encapsulated packet may include the MAC address and interface identifier of the source CI as the source MAC address and source interface identifier, and the MAC address and interface identifier of the source VSRS as the destination MAC address and destination interface identifier.

[0252] In block 1416, the source VSRS receives the encapsulated packet. The source VSRS decapsulates the packet, removing any address information related to the source VSRS (e.g., including the source VSRS's IP address, MAC address, and / or interface identifier). The source VSRS identifies the packet's destination CI. In some embodiments, the source VSRS identifies the packet's destination CI based on the IP address of the destination CI contained in the packet. The source VSRS searches for a mapping of the destination IP address in the packet. If the IP address is within the VCN's IP address space, the source VSRS searches for a mapping of the packet's destination IP address in the VCN's IP address space. The source VSRS then recaptures the packet using L3 encapsulation. In some embodiments, this L3 encapsulation may include, for example, MPLSoUDP L3 encapsulation. The source VSRS then forwards the packet to the VSRS of the destination VLAN (destination VSRS). In some embodiments, the source VSRS may forward the packet to the destination VSRS such that the destination VSRS becomes the packet's tunnel endpoint (TEP). L3-encapsulated packets include the IP address and MAC address of the source CI, and the IP address of the destination CI.

[0253] In block 1418, the destination VSRS receives the packet and decapsulates it. In some embodiments, this decapsulation may include removing the L3 encapsulation. If the destination VSRS knows the IP-to-MAC and MAC-to-interface mappings of the destination CI, it identifies the interface and MAC address of the destination CI within the destination VLAN. Alternatively, if the destination VSRS does not know the mappings of the destination CI, it may perform steps 1612 through 1622 of process 1600.

[0254] In block 1420, the destination VSRS forwards the packet to the destination interface of the destination CI. In some embodiments, this may include encapsulating the packet using L2 encapsulation. In some embodiments, the packet may include the IP address and MAC address of the source CI and / or the MAC address and interface identifier of the destination VSRS. In some embodiments, the packet may further include the MAC address and interface identifier of the destination CI.

[0255] In block 1422, the destination interface receives and decapsulates the packet. Specifically, the destination VNIC removes the L2 encapsulation. In some embodiments, the destination VNIC forwards the packet to the destination CI. In block 1424, the destination CI receives the packet.

[0256] Referring to Figure 15, which is a schematic diagram 1500 showing process 1400 for inter-VLAN communication. As shown in the diagram, VLAN A 1502-A has a VLAN CIDR of 10.0.3.0 / 24. VLAN A 1502-A can be instantiated on one or more hardware, specifically including VSRS VNIC A (VRVI A) 1504-A which can be instantiated on server group 1506. VRVI A 1504-A may have the IP address 10.0.3.1. The VLAN may include compute instance 1 (CI1) 1510 which has the IP address 10.0.3.2 and is communicably connected to NVD1 (SN1) 1512 which can instantiate L2VNIC1 (VI1) 1514 and L2 switch 1. The VLAN has the IP address 10.0.3.3 and can instantiate L2VNIC2(VI2)1524 and L2 switch 2 on NVD2(SN2)1522. It can include a compute instance 2 (CI2) 1520 that is connected via a communication interface. The VLAN can include a compute instance 3 (CI3) 1530 that has the IP address 10.0.3.4 and is connected via a communication interface to an NVD3 (SN3) 1532 that can instantiate an L2 VNIC 3 (VI3) 1534 and an L2 switch 3.

[0257] VLAN B 1502-B has a VLAN CIDR of 10.0.34.0 / 24. VLAN B 1502-B can be instantiated on one or more hardware, specifically including VSRS VNIC B (VRVI B) 1504-B which can be instantiated on server group 1506. VRVI B 1504-B may have the IP address 10.0.4.1. The VLAN may include compute instance 4 (CI4) 1540 which has the IP address 10.0.4.2 and is communicably connected to NVD4 (SN4) 1542 which can instantiate L2 VNIC 4 (VI3) 1544 and L2 switch 4.

[0258] Applying method 1400 of Figure 14 to the example in Figure 15, CI3 1530 is the source CI and VI3 1534 is the source interface. Also, CI4 1540 is the destination CI and VI4 1544 is the destination interface. If CI3 determines that it does not have an IP-MAC mapping for VRVI A 1504-A, CI3 1530 sends an ARP request. The ARP request may be for an address, specifically for a known IP address of VRVI A 1504-A. Thus, in some embodiments, the ARP request may be for 10.0.3.1.

[0259] This ARP request is received by SN3 1532 and VI3 1534. VI3 1534 creates an ARP request for each CI in VLAN 1502-A by duplicating the ARP request. VI3 1534 encapsulates each ARP request using L2 encapsulation and sends the ARP request to each interface in the VLAN. These encapsulated ARP requests may include the MAC address and IP address of the source CI (CI3), the interface identifier of the source interface (VI3), and information to identify the target IP address. These requests may be sent to each interface in the VLAN, as indicated by arrow 1550, specifically, as serial unicast so that each interface in the VLAN receives the ARP request.

[0260] Interfaces within a VLAN receive encapsulated ARP requests and decapsulate them. Based on the information contained in the ARP request and / or related information, the interfaces within the VLAN update their mappings. Specifically, for example, VI1 1514, VI2 1524, and VRVI A 1504-A each receive a unicast ARP request from CI3 1530, decapsulate the unicast ARP request, and learn the interface-to-MAC address mapping of the source CI. Both the interface identifier and MAC address are included in the encapsulated packet.

[0261] As indicated by arrow 1552, VRVI A 1504-A sends an ARP response to CI3 1530 when it determines that its IP address matches the requested IP address. For example, VRVI A 1504-A can encapsulate the ARP response from VRVI A 1504-A using L2 encapsulation and send it as an ARP unicast to the requesting interface, specifically to VI3. As previously stated in this application, this transmission of the ARP response may include providing the encapsulated ARP response to a switch associated with VRVI A 1504-A, which then encapsulates the ARP response. The received ARP response can be sent to VI3.

[0262] The ARP response may be received by VI3 and may be decapsulated. Based on the received ARP response, VI3 can learn the interface-to-MAC mapping of VRVI A 1504-A and provide the learned mapping to the switch associated with VI3. The decapsulated packet may be provided to CI3. CI3 can learn the IP-to-MAC address mapping of VRVI A 1504-A and send a packet. This packet may be an IP packet, such as an IPv4 or IPv6 packet. This packet may contain CI3's MAC address and IP address as source addresses. This packet may further contain the destination IP address of the destination CI (CI4 1540) as the destination address and may contain VRVI A 1504-A's MAC address and IP address.

[0263] VI3 can receive packets transmitted by CI3, encapsulate the packets, and forward them to VRVI A 1504-A. VRVI A 1504-A can receive the packets, decapsulate them, and look up the mapping of the packet's destination IP address (the IP address of CI4). In some embodiments, packet decapsulation may include removing the L2 header from the packet, in other words, removing information related to VLAN A 1502-A from the header. The removed information may include, for example, the MAC address of CI3, the interface identifier of interface VI3, and / or the MAC address and interface identifier of VRVI A 1504-A. In some embodiments, VSRS, and therefore VRVI A 1504-A, can reside in both L2 and L3 networks. In the L2 network, and therefore the VLAN, VSRS can utilize L2 communication protocols, but when communicating with the L3 network, it can utilize L3 communication protocols. In contrast to learning performed in VLANs, VSRS can learn the mapping of endpoints in the L3 network from the control plane. In some embodiments, for example, the control plane can provide mapping information for the IP addresses and / or MAC addresses of instances in the L3 network.

[0264] VRVI A 1504-A can retrieve the mapping of IP destination addresses contained in a packet, or in other words, it can retrieve the mapping of IP addresses of destination CIs. In some embodiments, retrieving this mapping is done by VRVI It can include identifying B 1504-B as the VSRS of VLAN B 1502-B. VRVI A 1504-A can encapsulate packets and transfer the encapsulated packets to VRVI B 1504-B. VRVI B 1504-B may be a tunnel endpoint (TEP) of the destination VLAN and / or the destination interface. The transfer of this encapsulated packet is indicated by block 1556. This transferred packet can include the IP address of the source CI (CI3) as the source address and the IP address of the destination CI (CI4) as the destination address.

[0265] VRVI B 1504-B can receive and decapsulate packets. VRVI B 1504-B can further identify an interface within VLAN B 1502-B corresponding to the destination IP address. VRVI B 1504-B can encapsulate the packet and add an L2 header to tunnel it within VLAN B 1504-B. Then, VRVI B 1504-B can transfer the packet to the destination CI. After receiving and decapsulating the packet, the destination interface VI4 1544 can transfer the Ethernet (registered trademark) frame or packet to the destination CI1540.

[0266] Received Packet Flow Referring to FIG. 16, FIG. 16 is a flowchart showing an embodiment of a process 1600 for performing a received packet flow. Specifically, FIG. 16 shows an embodiment of a process 1600 for receiving packets from a subnet. This process may be executed by all or part of the system 600, specifically, by an entity of the VLAN and an external source CI (with respect to the VLAN). The external source CI can reside in the L3 subnet.

[0267] ​Process 1600 begins in block 1602. In block 1602, the source L3 CI decides to send the packet to the destination CI, specifically to the destination IP address within the VLAN. Since the source L3 CI does not know the mapping, it sends an ARP request for the MAC address of the virtual router in the subnet containing the source L3 CI. In some embodiments, sending this ARP request may include sending the ARP request to an Ethernet broadcast. In block 1604, the source L3 The CI's source interface responds to ARP requests with the VR's MAC address. In some embodiments, the source interface can determine the VR's MAC address based on mapping information accessed by the source interface.

[0268] Upon receiving an ARP response from the source interface, the source L3 C1 sends an IP packet to the VR, as shown in block 1606. In some embodiments, this may include the source L3 CI sending the packet and the source L3 interface receiving, encapsulating, and forwarding this packet. In some embodiments, the source L3 interface may encapsulate the packet using L3 encapsulation. The L3 encapsulation includes the original packet, beginning with an L3 header. The source L3 interface may forward the packet to the VR. This IP packet may be sent to the VR MAC address and VR interface, and may include the MAC address of the source L3 CI as the MAC address and the interface identifier of the source L3 CI as the source interface identifier.

[0269] The VR receives the packet and decapsulates it. The VR can then look up the VNIC mapping for the destination IP address within the packet. The VR then determines if the packet is VLAN It can be determined that the packet is within a CIDR, and then the packet can be encapsulated and forwarded to the VSRS of the VLAN containing the destination CI. In some embodiments, the VR can encapsulate the packet using L3 encapsulation. The encapsulated packet may include the destination IP address, MAC address, and interface identifier of the VSRS, as well as the source IP address.

[0270] As shown in block 1610, the VSRS receives and decapsulates the packet. If the VSRS knows the mapping of the destination CI, specifically the mapping of the destination IP address in the received packet, it forwards the IP packet to the TEP of the CI corresponding to the destination IP address. This may involve generating and encapsulating an L2 packet having a destination MAC corresponding to the MAC address of the destination CI and a destination interface identifier corresponding to the destination interface. In an embodiment of the received packet flow, the destination interface is an L2VNIC.

[0271] If VSRS does not know the destination CI mapping, process 1600 proceeds to block 1612, where VSRS receives the IP packet and decapsulates it.

[0272] If the VSRS does not know the destination MAC address to VNIC mapping, it can perform an interface to MAC address learning process. This may involve the VSRS sending a packet to all interfaces in the VLAN. In some embodiments, this packet may be sent to all interfaces in the VLAN via broadcast. This packet may include the destination MAC and IP address, the VSRS interface identifier and MAC address, and the source CI's IP address. Each VNIC in the VLAN can receive this packet and learn the VSRS interface to MAC address mapping.

[0273] Each receiving VNIC can further decapsulate the packet and forward the decapsulated packet to the associated CI. Each CI may include a network interface that can evaluate the forwarded packet. If the network interface determines that the CI receiving the forwarded packet does not match the destination MAC and / or IP address, the packet is discarded. If the network interface determines that the CI receiving the forwarded packet matches the destination MAC and / or IP address, the packet is received by the CI. In some embodiments, a CI having a MAC and / or IP address that matches the packet's destination MAC and / or IP address can send a response to the VSRS. This allows the VSRS to learn the interface-to-MAC address mapping and IP-to-MAC address mapping of the destination CI.

[0274] Alternatively, if the VSRS does not know the destination CI mapping, specifically the destination IP address to MAC address mapping, it holds the IP packet. The VSRS then generates an ARP request for the destination IP address on all interfaces of the VSRS within the VLAN broadcast domain. This ARP request includes the VSRS MAC as the source MAC, the interface identifier of the VSRS interface as the source interface, and the destination IP address as the target IP address. In some embodiments, this ARP request can be broadcast to all interfaces within the VLAN.

[0275] In block 1614, all interfaces within the VLAN broadcast domain receive and decapsulate packets. Because the packets identify the MAC address and interface identifier of the VSRS, each interface within the VLAN broadcast domain that receives the packet learns the interface-to-MAC address mapping of the VSRS. As part of learning the interface-to-MAC address mapping of the VSRS, each interface can update its mapping table and provide the updated mapping table to the relevant switch and / or CI.

[0276] Each receiving interface can forward the decapsulated packet to the associated CI. The CI that receives the forwarded decapsulated packet, specifically its network interface, can determine whether the packet's destination IP address matches the IP address of that CI. If the IP address of the CI associated with the interface does not match the IP address of the destination CI specified in the received packet, no further action is taken.

[0277] If the IP address of the destination CI specified in the received packet is determined to match the IP address of the CI associated with the receiving interface (destination CI), block 1616 will be executed. As shown, the destination CI sends a response to the source interface that may be a unicast ARP response. This response includes the MAC address and IP address of the destination CI and the MAC address and IP address of the VSRS. As shown in block 1618, this response is received by the destination interface encapsulating the unicast ARP response. The destination interface can then forward the encapsulated packet to the VSRS via the destination switch. The encapsulated packet includes the MAC address, IP address and interface identifier of the destination CI and the MAC address, IP address and interface identifier of the VSRS.

[0278] In block 1620, VSRS, specifically the VSRS interface, receives and decapsulates the ARP response. VSRS, specifically the VSRS interface, can further learn the interface-to-MAC address mapping of the destination CI based on the encapsulation and / or the information contained in the encapsulated packet.

[0279] In block 1622, the VSRS can encapsulate the L2 header and append it to the previously held IP packet, and then forward the previously held IP packet to the destination CI, specifically to the destination interface. The destination interface can decapsulate the packet and provide the decapsulated packet to the destination CI. This packet forwarded by the VSRS may include the VSRS's MAC address and interface identifier as the source MAC address and source interface identifier, and the destination CI's MAC address and interface identifier as the destination MAC address and destination interface identifier. This packet may further include the destination CI's IP address and the source CI's IP address.

[0280] The destination interface receives the packet from the VSRS and decapsulates it. This decapsulation may include removing headers added by the VSRS, which may be VCN headers. The destination interface can then forward the packet to the destination CI, which can receive the packet from the destination interface.

[0281] Referring to Figure 17, which is a schematic diagram 1700 showing process 1600 for receiving communications. As shown, VLAN A 1502-A has a VLAN CIDR with 10.0.3.0 / 24. VLAN A 1502-A can be instantiated on one or more hardware, specifically including VSRS VNIC A (VRVI A) 1504-A which can be instantiated on server group 1506. VRVI A 1504-A may have an IP address of 10.0.3.1. The VLAN may include compute instance 1 (CI1) 1510 which has an IP address of 10.0.3.2 and is communicably connected to NVD1 (SN1) 1512 which can instantiate L2VNIC1 (VI1) 1514 and L2 switch 1. The VLAN may include compute instance 2 (CI2) 1520, which has the IP address 10.0.3.3 and is communicatively connected to L2VNIC2 (VI2) 1524 and L2 switch 2 and NVD2 (SN2) 1522. The VLAN may also include compute instance 3 (CI3) 1530, which has the IP address 10.0.3.4 and is communicatively connected to L2VNIC3 (VI3) 1534 and L2 switch 3 and NVD3 (SN3) 1532.

[0282] A possible L3 compute instance 4 (CI4) 1744 is a VLAN. A 1702-A can reside on subnet 1739 outside of A. CI4 can have the IP address 10.0.4.4 and can communicate with NVD4(SN4)1742, which can instantiate L3 VNIC4(VI4)1744. It is possible.

[0283] Applying Method 1600 of Figure 16 to the example in Figure 17, CI4 1740 is the source CI and VI4 1744 is the source interface. Also, CI3 1730 is the destination CI and VI3 1734 is the destination interface. CI4 determines that it does not know the mapping to send the packet to CI3. Therefore, CI4 sends an ARP request to the IP address of the subnet virtual router. In response to this request, in some embodiments, VI4, which may reside on the NVD containing an instance of the subnet virtual router, responds directly to CI4 with the IP address of the VR. CI4 learns the IP address of the VR and sends the packet to the VR. The VR receives the packet, encapsulates the packet based on the mapping information, and forwards the packet to the VSRS as indicated by arrow 1748.

[0284] When VSRS receives a packet and does not know the mapping to the destination CI, specifically the IP-to-MAC address mapping, VSRS holds the packet and sends an ARP request to all interfaces in VLAN A 1704-A. This ARP request includes the MAC address and interface identifier of VSRS. Each receiving interface in VLAN A 1704-A learns the VSRS mapping based on the ARP request. Similarly, each interface decapsulates the packet and sends it to its CI. Upon receiving the decapsulated packet, CI3 determines that it is the CI specified in the packet and generates an ARP response with its MAC address. VI3 receives and encapsulates the ARP response and forwards the encapsulated ARP response to VSRS. The encapsulated ARP response includes the MAC address and interface identifier of VSRS and the MAC address and interface identifier of CI3's interface.

[0285] VSRS receives the ARP response and learns the IP address-to-MAC address mapping and MAC address-to-interface mapping for CI3. VSRS then forwards the previously held packets to CI3. These packets are received by VI3, decapsulated, and forwarded to CI3.

[0286] Outgoing packet flow Next, referring to Figure 18, which is a flowchart of one embodiment of process 1800 for performing outgoing packet flow from a VLAN. In some embodiments, packets can be sent from one VLAN to another VLAN, to a subnet, or to another network. Process 1800 may be performed by all or part of system 600, specifically by the entities of the VLAN. In some embodiments, part of the process may be performed by an external source CI (for the VLAN). The external source CI may reside in an L3 subnet.

[0287] In some embodiments, process 1800 may be performed when a compute instance (source CI) within a VLAN (source VLAN) sends a packet to a destination compute instance (destination CI) outside the VLAN. If the source CI already knows the mapping of the VSRS (source VSRS) of the first VLAN, it can send the packet directly to the source VSRS. If the source CI does not know the mapping of the source VSRS, the source CI and the associated VNIC (source interface or source VNIC) first learn the mapping of the source VSRS, specifically the IP-to-MAC address mapping of the source VSRS. In embodiments of the outgoing packet flow from an L2 VLAN, the source VNIC is an L2 VNIC. The first step 1802 of process 1800... 1810 concerns learning source VSRS mapping using source VNIC and source CI.

[0288] Process 1800 begins at block 1802. In block 1802, the source CI initiates an ARP request. The ARP request is used to determine the IP-to-MAC address mapping of the source VSRS. The ARP request is sent by the source CI to the Ethernet broadcast. The ARP request includes the source CI's MAC address, IP address, and interface identifier as the source address and source interface identifier. The ARP request further includes the source VSRS's IP address.

[0289] In block 1804, the source VNIC receives and replicates the ARP request. Specifically, the source VNIC receives the ARP request from the source CI, identifies all interfaces on the VLAN, and sends the ARP request to all interfaces on the VLAN broadcast domain. As previously mentioned, the control plane knows all interfaces on the VLAN and provides this information to the interfaces within the VLAN; therefore, the source interface can similarly know all interfaces within the VLAN and send an ARP request to each interface within the VLAN. To do this, the source interface replicates the ARP request and encapsulates one of the ARP requests for each interface on the VLAN. Each encapsulated ARP request includes the source CI's interface identifier and the source CI's MAC and / or IP address as the source address, and the destination CI's interface identifier as the destination address. The source CI's interfaces replicate the Ethernet broadcast by sending the replicated and encapsulated ARP requests via serial unicast. In some embodiments, the source VNIC can encapsulate the ARP request using Geneve encapsulation.

[0290] In block 1806, all interfaces within the VLAN broadcast domain receive and decapsulate the packet. Since the packet identifies the source CI's MAC address and interface identifier, each interface within the VLAN broadcast domain that receives the packet learns the source CI's interface-to-MAC address mapping. In addition, the VSRS learns the source CI's IP-to-MAC address mapping. As part of learning the source CI's interface-to-MAC address mapping, each interface can update its mapping table and provide the updated mapping table to the relevant switch and / or CI. Each receiving interface, except the VSRS, can forward the decapsulated packet to the relevant CI. The CI that receives the forwarded decapsulated packet, specifically its network interface, can determine whether the packet's destination IP address matches the CI's IP address. If the IP address of the CI associated with the interface does not match the destination CI's IP address specified in the received packet, no further action is taken.

[0291] If the source VSRS determines that the destination IP address matches the source VSRS's IP address, it encapsulates a response that may be a unicast ARP response, as shown in block 1808, and sends it to the source interface. This response includes the source CI's MAC address and IP address as the destination address and the source CI's interface identifier as the destination interface identifier. This response further includes the source VSRS's MAC address and IP address as the source address and the source VSRS's interface identifier as the source interface identifier.

[0292] In block 1810, the source interface receives an ARP response and decouples. Cellularization is performed. The source interface can further learn the interface-to-MAC address mapping of the source VSRS based on the encapsulation and / or the information contained in the encapsulated packets. In some embodiments, the source interface can forward ARP responses to the source CI.

[0293] In block 1812, the source CI receives an ARP response. In some embodiments, the source CI can update its mapping table based on the information contained in the ARP response, specifically based on the MAC address and IP address of the source VSRS. The source CI can then send a packet to the source VSRS, which may be any packet, specifically an IP packet, specifically an IPv4 or IPv6 packet. In some embodiments, this may include sending an IP packet with the MAC address and IP address of the source VSRS as the destination address. The packet may further include the MAC address and IP address of the source CI as the source address.

[0294] In block 1814, the source interface receives a packet from the source CI. The source interface encapsulates the packet. The source interface can then forward the encapsulated packet to the source VSRS, specifically to the VNIC of the source VSRS. The encapsulated packet may include the MAC address and interface identifier of the source CI as the source MAC address and source interface identifier, and the MAC address and interface identifier of the source VSRS as the destination MAC address and destination interface identifier. The encapsulated packet may further include the IP address of the source CI and the IP address of the source VSRS.

[0295] In block 1816, the source VSRS receives the encapsulated packet. The source VSRS identifies the packet's destination CI. In some embodiments, the source VSRS identifies the packet's destination CI based on the IP address of the destination CI contained in the packet. The source VSRS searches for a mapping of the packet's destination IP address. If the IP address is within the VCN's IP address space, the source VSRS searches for a mapping of the packet's destination IP address within the VCN's IP address space. The source VSRS then re-encapsulates the packet using L3 encapsulation.

[0296] The source VSRS then forwards the packet to the destination CI. In some embodiments, this may include forwarding the encapsulated packet to a VR associated with the subnet containing the destination CI, and / or forwarding the packet to a gateway to allow the packet to leave the VCN. In some embodiments, forwarding the packet to the destination CI may include forwarding the packet to a TEP associated with the destination CI. The L3 encapsulated packet includes the IP address of the source CI as the source address and the IP address of the destination CI as the destination address.

[0297] In block 1820, the destination interface receives and decapsulates the packet. In some embodiments, the destination VNIC forwards the packet to the destination CI. In block 1822, the destination CI receives the packet.

[0298] Referring to Figure 19, which is a schematic diagram 1900 showing process 1800 for performing the transmit packet flow. As shown in the diagram, VLAN A 1502-A has a VLAN CIDR with 10.0.3.0 / 24. VLAN A 1502-A can be instantiated on one or more hardware, specifically on server group 1506, VSRS VNIC A (VRVI A) 1504-A This includes: VRVI A 1504-A may have the IP address 10.0.3.1. The VLAN may include compute instance 1 (CI1) 1510, which has the IP address 10.0.3.2 and is communicatively connected to NVD1 (SN1) 1512, which can instantiate L2VNIC1 (VI1) 1514 and L2 switch 1. The VLAN may include compute instance 2 (CI2) 1520, which has the IP address 10.0.3.3 and is communicatively connected to NVD2 (SN2) 1522, which can instantiate L2VNIC2 (VI2) 1524 and L2 switch 2. The VLAN may include compute instance 3 (CI3) 1530, which has the IP address 10.0.3.4 and is communicatively connected to NVD3 (SN3) 1532, which can instantiate L2VNIC3 (VI3) 1534 and L2 switch 3.

[0299] A possible compute instance is L3 compute instance 4 (CI4) 1944, which is VLAN It can reside on subnet 1939 outside of A1902-A. CI4 can have the IP address 10.0.4.4 and can communicate with NVD4(SN4)1942, which can instantiate L3 VNIC4(VI4)1944. Subnet 1939 can contain a virtual router (VR)1948. VR1948 can have the IP address 10.0.4.1. VR1948 may be instantiated on, for example, a smart NIC, a server, or a group of servers.

[0300] Applying Method 1800 of Figure 18 to the example in Figure 19, CI3 1930 is the source CI and VI3 1934 is the source interface. Also, CI4 1940 is the destination CI and VI4 1944 is the destination interface. If CI3 determines that it does not have an IP-to-MAC mapping for VRVI A 1904-A, CI3 1930 sends an ARP request. The ARP request may be for a known address, specifically for a known IP address of VRVI A 1904-A. Thus, in some embodiments, the ARP request may be for 10.0.3.1.

[0301] This ARP request is received by SN3 1932 and VI3 1934. VI3 1934 creates an ARP request for each CI in VLAN 1902-A by duplicating the ARP request. VI3 1934 encapsulates each ARP request using L2 encapsulation and sends the ARP request to each interface in the VLAN. These encapsulated ARP requests may include the MAC address of the source CI (CI3), the source interface identifier VI3, and information to identify the target IP address. These requests may also be sent to each interface in the VLAN, as indicated by arrow 1950, or more specifically, they may be broadcast so that each interface in the VLAN receives the ARP request.

[0302] Interfaces within a VLAN receive encapsulated ARP requests and decapsulate them. Based on the information contained in the ARP request and / or related information, the interfaces within the VLAN update their mappings. Specifically, for example, VI1 1914, VI2 1924, and VRVI A 1904-A each receive a unicast ARP request from CI3 1930, decapsulate the unicast ARP request, and learn the interface-to-MAC address mapping of the source CI. Both the interface identifier and MAC address are included in the encapsulated packet.

[0303] As indicated by arrow 1952, VRVI A 1904-A determines that its IP address matches the requested IP address and sends an ARP response to CI3 1930. The ARP response from VRVI A 1904-A is, for example, sent to VRVI A 1904-A. Therefore, it may be encapsulated and sent as an ARP unicast to the requesting interface, specifically to VI3. As described above, the transmission of this ARP response may include providing the encapsulated ARP response to a switch associated with VRVI A 1904-A, which can then transmit the encapsulated ARP response to VI3.

[0304] The ARP response may be received by VI3 and may be decapsulated. Based on the received ARP response, VI3 can learn the interface-to-MAC mapping of VRVI A 1904-A and provide the learned mapping to the switch associated with VI3. The decapsulated packet may be provided to CI3. Based on the received packet, CI3 can learn the IP-to-MAC address mapping. CI3 can send a packet, which may be an IP packet such as an IPv4 or IPv6 packet. This packet may contain the MAC address and IP address of CI3 as the source address and the IP address of the destination CI (CI4 1940) as the destination address. In some embodiments, the destination address may further contain the MAC address of VRVI A 1904-A.

[0305] Packets transmitted by CI3 can be received by VI3, which can encapsulate the packets and forward them to VRVI A 1904-A. VRVI A 1904-A can receive the packets, decapsulate them, and retrieve the mapping of the packet's destination IP address (the IP address of CI4). In some embodiments, VSRS, and therefore VRVI A 1904-A, can reside in both L2 and L3 networks. In an L2 network, and therefore VLAN, VSRS can utilize L2 communication protocols, but when communicating with an L3 network, it can utilize L3 communication protocols. In contrast to learning performed in a VLAN, VSRS can learn the mapping of endpoints in the L3 network from the L3 control plane. In some embodiments, for example, the L3 control plane can provide information mapping the IP addresses, MAC addresses, and / or interface identifiers of instances in the L3 network.

[0306] VRVI A 1904-A can look up the mapping of IP destination addresses contained in a packet, or in other words, it can look up the mapping of IP addresses of destination CIs. In some embodiments, looking up this mapping may include identifying subnet 1939 where CI4 1940 resides, and / or identifying the VR associated with subnet 1939 where CI4 resides. VRVI A 1904-A can encapsulate a packet and forward the encapsulated packet to VR1948. VR1948 may be a tunnel endpoint (TEP) for subnet 1939 and / or destination CI1940. This forwarding of the encapsulated packet is shown by block 1956.

[0307] VR1948 can receive and decapsulate packets. VR1948 can further identify the interface within subnet 1939 corresponding to the destination IP address. In some embodiments, VR1948 may be located on the same NVD as the destination interface VI4 1944. Thus, VR1948 can forward packets directly to the destination CI, specifically to CI4 1940 as indicated by block 1958.

[0308] Interface-based access control list filtering VSRS can provide interface-based access control list (ACL) filtering. This may include evaluating the incoming security policy of a VLAN. It may also include evaluating the outgoing security policy of the VSRS sender based on learned VLAN interface-to-MAC address and IP address mappings when VSRS determines where to send incoming packets. This may result in a delay in ACL classification.

[0309] In some embodiments, an ACL may include an allow list associated with objects in the system. These objects may include hardware in a physical network, which may include, for example, one or more servers, smart NICs, host machines, etc. These objects may include one or more virtual objects in a virtual network, which may include, for example, one or more interfaces, compute instances, addresses such as IP addresses and / or MAC addresses. In some embodiments, an ACL may specify users who are allowed access to an object, the object and / or system processes, and / or actions permitted for a given object.

[0310] An ACL may be unique to one or more CIs. Thus, in some embodiments, some or all CIs may have and / or retain unique ACLs. In some embodiments, an ACL for a CI may specify interfaces and / or addresses (either MAC addresses or IP addresses) on which the CI can send packets, interfaces and / or addresses (either MAC addresses or IP addresses) on which the CI cannot send packets, one or more interfaces and / or addresses on which the CI can send one or more types of packets, and / or one or more interfaces on which the CI cannot send one or more types of packets. In some embodiments, an ACL for a CI may be stored in a location accessible by other entities in the network, including one or more other entities such as VRs or VSRSs that can enforce the ACL.

[0311] For example, when receiving communications from an IP network for one or more intended recipients within a VLAN, the VSRS can determine and apply filtering and / or delivery restrictions to the communications based on the ACL of the sender of those communications. In some embodiments, this can be achieved, for example, by (1) the VSRS making communications decisions based on access to a copy of the sender's ACL, or by (2) the VSRS making communications decisions based on information encoded in the packet metadata of the received communications.

[0312] Referring to Figure 20, which is a flowchart of one embodiment of process 2000 for performing delayed access control list (ACL) classification. Process 2000 may be performed by all or part of system 600, specifically by VSRS624,634.

[0313] Process 2000 begins in block 2002. In block 2002, the source CI sends the packet to the destination MAC or IP address. In some embodiments, the source CI may be outside the VLAN containing the destination MAC or IP address to which the packet is sent.

[0314] In block 2006, the VSRS of the VLAN, which includes the destination MAC or IP address, receives the packet. In some embodiments, the packet undergoes L2 encapsulation. Packets may be encapsulated using L3 encapsulation, and in some embodiments, packets may be encapsulated using L3 encapsulation. As shown in block 2008, the VSRS can decapsulate the packet and identify the source CI. Once the source CI is identified, the VSRS can obtain the source CI's ACL. In some embodiments, for example, the source CI's ACL may be stored in a location accessible by the VSRS. In some embodiments, accessing the source CI's ACL may include retrieving information within the source CI's ACL. This information may, for example, identify one or more restrictions on the delivery of the packet. In some embodiments, this information may include one or more rules based on one or more IP addresses, MAC addresses, source and destination TCP and / or UDP ports, EtherType, etc.

[0315] In block 2010, the VSRS identifies the destination interface of the packet. In embodiments where the VSRS does not have a destination address mapping, the VSRS can determine the mapping information as described above with reference to steps 1612 to 1620 of process 1600 in Figure 16. In embodiments where the mapping has been previously learned, or where the mapping has been learned by performing some or all of steps 1612 to 1620 of process 1600, the VSRS can identify the destination interface based on the mapping learned by the VSRS through communication with the VSRS interface in the VLAN. In some embodiments, identifying the destination interface may include looking up the destination interface based on the destination address, specifically the destination IP address and / or MAC address of the packet.

[0316] In block 2012, the VSRS applies the ACL of the source CI to the destination interface. This may include determining whether any portion of the source CI's ACL is relevant to the destination interface, and if so, applying that portion of the source CI's ACL. In block 2014, if the destination interface conforms to the source CI's ACL, in other words, if the source CI's ACL permits the transmission of packets from the source CI to the destination interface, the VSRS forwards the packets to the destination interface. In some embodiments, this packet forwarding may include encapsulating the packets. In some embodiments, the packets may be encapsulated according to L2 encapsulation, such as L2Geneve encapsulation. The VSRS can forward the packets to the destination interface, more specifically to the destination CI having the destination interface as the TEP. The destination interface can receive the packets, decapsulate them, and forward them to the destination CI.

[0317] Alternatively, in block 2016, if the destination interface does not conform to the source CI's ACL, in other words, if the source CI's ACL does not permit the transmission of the packet from the source CI to the destination interface, the VSRS will drop the packet. In some embodiments, the VSRS may respond to the source CI indicating the packet drop and / or the reason for the packet drop. In some embodiments, the VSRS may update the operational metrics / statistics related to the transmitting interface to reflect the ACL decision.

[0318] Referring to Figure 21, Figure 21 is a flowchart of one embodiment of process 2100 for early classification of ACLs and incorporating the classification into metadata. Process 2100 may be performed by all or part of system 600, specifically by source C1 and VSRS624,634.

[0319] In block 2102, the source CI decides to send the packet to the destination CI. In some embodiments, this may include the source CI deciding to send the packet to the destination CI's MAC address and / or IP address. The source VNIC may send the packet as shown in block 2104. The source VNIC may receive the packet, evaluate the packet's ACL, and embed ACL information relevant to the packet into a portion of the packet. In some embodiments, this information may include one or more rules based on one or more IP addresses, MAC addresses, source and destination TCP and / or UDP ports, Ethernet type, etc. The source VNIC may further encapsulate the packet. In some embodiments, the ACL may be stored as metadata in the encapsulated packet, specifically in the packet header. In some embodiments, the source CI may be outside the VLAN containing the destination CI to which the packet is sent.

[0320] In block 2106, the VSRS of the VLAN containing the destination MAC or IP address receives the packet. In some embodiments, the packet may be encapsulated using L2 encapsulation, and in some embodiments, the packet may be encapsulated using L3 encapsulation. The VSRS can decapsulate the packet. In some embodiments, the VSRS can extract from the packet information that identifies the packet's destination, specifically information that identifies the destination IP address.

[0321] In block 2108, the VSRS identifies the destination interface of the packet. In embodiments where the VSRS does not have destination address mappings, the VSRS can determine mapping information as described above with reference to steps 1612 to 1620 of process 1600 in Figure 16. In embodiments where mappings have been previously learned, or where mappings have been learned by performing some or all of steps 1612 to 1620 of process 1600, the VSRS can identify the destination interface based on the mappings learned by the VSRS via communication with the VSRS interface in the VLAN. Thus, in some embodiments, the VSRS can determine that it has destination address mapping information and identify the destination interface based on this mapping information. In some embodiments, identifying the destination interface may include looking up the destination interface based on the destination address, specifically the destination IP address and / or MAC address of the packet.

[0322] In block 2110, the VSRS obtains ACL information contained in a portion of the packet. In some embodiments, this may include extracting metadata from the packet, specifically from the packet header. In some embodiments, this may include decoding the information encoded in the packet metadata of the packet header.

[0323] In block 2112, the VSRS applies ACL information extracted from a portion of the packet. Specifically, this may include applying encoded security information to the destination interface. This may include determining any portion of the ACL information relevant to the destination interface and, if relevant, applying that portion of the ACL information. In block 2114, if the destination interface conforms to the ACL information and / or one or more rules of the ACL information, in other words, if the ACL information permits the packet to be sent from the source CI to the destination interface, the VSRS forwards the packet to the destination interface. In some embodiments... The forwarding of this packet may include encapsulating the packet. In some embodiments, the packet may be encapsulated according to L2 encapsulation, such as L2Geneve encapsulation. The VSRS can forward the packet to the destination interface, more specifically to the destination CI which has the destination interface as the TEP. The destination interface can receive the packet, decapsulate the packet, and forward the packet to the destination CI.

[0324] Alternatively, in block 2116, if the destination interface does not conform to the ACL information, in other words, if the ACL information does not permit the packet to be sent from the source CI to the destination interface, the VSRS discards the packet. In some embodiments, the VSRS may respond to the source CI indicating the packet discard and / or the reason for the packet discard.

[0325] Next-hop routing Some embodiments of VSRS can facilitate next-hop routing, specifically by delaying the evaluation of the next hop until VSRS receives the communication. Since the sender of communication outside the VLAN may not know the updated virtual IP address of the instance within the VLAN, the sender outside the VLAN cannot accurately specify the next-hop routing.

[0326] In some embodiments, VSRS can facilitate next-hop routing specifications. In some embodiments, this can be achieved, for example, by (1) the sender making an initial specification for next-hop routing for communication, and VSRS re-evaluating the next-hop specification when it receives the communication, or (2) delaying the next-hop specification until VSRS receives the communication.

[0327] Referring to Figure 22, Figure 22 is a flowchart illustrating one embodiment of process 2200 for performing sender-based next-hop routing. In some embodiments, this may include separating the evaluation of the next-hop route from the specification of the next-hop route. This may result in the source CI evaluating the route policy and encoding the next-hop route in the packet metadata of the virtual packet header of the communication. In such embodiments, the packet may contain the intended virtual IP address of the next-hop destination, and the next-hop route may be encoded in the packet metadata.

[0328] This communication is received by VSRS, which uses the next-hop route encoded in the packet metadata to determine the instance in the VLAN receiving the communication and the destination virtual IP address of that instance. This instance can be determined using tables generated and / or curated by VSRS, specifically tables linking virtual IP addresses, MAC addresses, and / or virtual interface IDs. Using these tables, VSRS can identify the virtual IP address corresponding to the MAC address and / or virtual interface ID of the intended next-hop destination. In some embodiments, VSRS identifying the recipient instance based on the encoded next-hop route contained in the packet metadata can result in VSRS sending the communication to a destination virtual IP address different from the next-hop destination indicated by the virtual IP address in the packet and specified by the packet sender.

[0329] Process 2200 may be executed by all or part of system 600, specifically by source CI and VSRS624,634.

[0330] The process begins at block 2202. In block 2202, the source CI can send a packet. In some embodiments, the source CI may be outside the VLAN containing the destination MAC address or IP address to which the packet is sent.

[0331] In block 2204, the source VNIC can make a routing decision based on one or more routing rules. In some embodiments, this may include the source VNIC searching for and / or obtaining one or more routing rules and making a routing decision based on these routing rules. In some embodiments, this routing decision may be a next-hop routing decision for a packet. In some embodiments, these routing rules may be stored, for example, in a route table in the source CI's network. In some embodiments, this may include, for example, a subnet route table.

[0332] In block 2206, the source VNIC can embed the routing decision within a portion of the packet. In some embodiments, this may include encapsulating the packet and embedding the routing decision in the packet's metadata. The metadata may, for example, be encoded in the packet header. After encapsulating the packet and embedding the routing decision within the encapsulated packet, the source VNIC can forward the encapsulated packet to the VSRS.

[0333] In block 2208, a VSRS within a VLAN containing a destination MAC address or IP address receives a packet. In some embodiments, the packet may be encapsulated using L2 encapsulation, and in some embodiments, the packet may be encapsulated using L3 encapsulation. In some embodiments, the VSRS receiving a packet may include decapsulating the packet. In some embodiments, the VSRS receiving a packet may include extracting routing decisions embedded in a portion of the packet. In some embodiments, this may include decoding encoded metadata containing routing decisions.

[0334] In block 2210, the VSRS determines the destination interface of the packet by applying the routing decision to the VSRS routing information. In some embodiments, this may include determining the destination CI in the VLAN corresponding to the routing information by applying the decoded routing decision to the VSRS routing information. In block 2212, the VSRS sends the packet to the determined CI in the VLAN. In some embodiments, this may include the VSRS forwarding the packet to the destination interface. In some embodiments, this packet forwarding may include encapsulating the packet, which may be done according to L2 encapsulation, such as L2Geneve encapsulation. The VSRS can forward the packet to the destination interface, more specifically to the destination CI having the destination interface as the TEP. The destination interface can receive the packet, decapsulate the packet, and forward the packet to the destination CI.

[0335] Referring to Figure 23, a flowchart is shown illustrating one embodiment of process 2300 for performing delayed next-hop routing. Process 2300 may be performed by all or part of system 600, specifically by source VNIC and VSRS624,634.

[0336] In some embodiments, for example, a source VNIC can make routing decisions for communications traversing a VLAN based on routing rules that may be included in its routing table. This communications are received by a VSRS, which can re-evaluate the next-hop specification and identify routing rules by querying a copy of the same routing table. Based on these routing rules, the VSRS determines the instance in the VLAN corresponding to the routing rule and the virtual IP address of that instance, and sends the communications to that instance. Determining the instance corresponding to the routing rule may include the VSRS retrieving information from a table linking virtual IP addresses, MAC addresses, and / or virtual interface IDs, and determining the virtual IP address associated with the virtual interface and / or instance that is the intended next-hop destination.

[0337] Process 2300 begins at block 2302. In block 2302, the source CI can send a packet. In some embodiments, this may include the source CI sending the packet to the MAC address and / or IP address of the destination CI. In some embodiments, the source CI may be outside the VLAN containing the destination MAC address or IP address to which the packet is sent.

[0338] In block 2204, the source VNIC can receive the packet. The source VNIC can then make a routing decision based on one or more routing rules. In some embodiments, this may include the source VNIC searching for and / or obtaining one or more routing rules and making a routing decision based on those rules. In some embodiments, this routing decision may be a next-hop routing decision for the packet. In some embodiments, these routing rules may be stored, for example, in a route table in the source CI's network. In some embodiments, this may include, for example, a subnet route table. The source VNIC can encapsulate the packet and send the packet to the VSRS.

[0339] In block 2206, a VSRS within a VLAN containing the destination MAC or IP address receives a packet. In some embodiments, the packet may be encapsulated using L2 encapsulation, and in some embodiments, the packet may be encapsulated using L3 encapsulation. In some embodiments, the VSRS receiving a packet may include decapsulating the packet.

[0340] In block 2308, the VSRS obtains routing information related to the packet. In some embodiments, this may include obtaining a routing table and / or a portion of the routing table related to the received packet. In some embodiments, this routing table may be received, for example, from the control plane. Upon receiving the routing information, the VSRS identifies the routing rules related to the received packet.

[0341] In block 2310, the VSRS determines the destination interface of a packet for a VLAN by applying routing rules to the VSRS routing information. In some embodiments, this may include determining the destination CI for the VLAN corresponding to the routing information by applying routing rules to the VSRS routing information. In block 2312, the VSRS sends the packet to the determined VLAN CI. In some embodiments, this may include the VSRS forwarding the packet to the destination interface. The forwarding of this packet may include encapsulating the packet, and the encapsulation may be performed according to L2 encapsulation, such as L2Geneve encapsulation. The VSRS can forward the packet to the destination interface, more specifically to the destination CI which has the destination interface as the TEP. The destination interface can receive the packet, decapsulate it, and forward it to the destination CI.

[0342] Example Implementation As mentioned above, IaaS (Infrastructure as a Service) is one specific type This is cloud computing. IaaS may be configured to provide virtualized computing resources over a public network (e.g., the internet). In the IaaS model, a cloud computing provider can host infrastructure elements (e.g., servers, storage, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer)). In some cases, the IaaS provider can provide various services associated with the infrastructure elements (e.g., billing, monitoring, logging, load balancing, and clustering). Therefore, since these services can be policy-driven, IaaS users can implement policies to drive load balancing in order to maintain application availability and performance.

[0343] In some cases, IaaS customers can access resources and services over a wide area network (WAN), such as the internet, and install the rest of their application stack using the cloud provider's services. For example, a user can log into an IaaS platform, create virtual machines (VMs), install an operating system (OS) on each VM, deploy middleware such as databases, create storage buckets for workloads and backups, and install enterprise software on the VMs. Customers can use the provider's services to perform a variety of functions, including balancing network traffic, troubleshooting applications, monitoring performance, and managing disaster recovery.

[0344] In most cases, the cloud computing model requires the participation of a cloud provider. This cloud provider may or may not be a third-party service specializing in IaaS provision (e.g., offering, renting, or selling). Alternatively, a company can become a provider of private clouds and infrastructure services.

[0345] In some cases, IaaS deployment is the process of deploying a new application or a new version of an application to a pre-configured application server. IaaS deployment may also include the process of preparing the server (e.g., installing libraries, daemons, etc.). IaaS deployment is often managed by the cloud provider under the hypervisor layer (e.g., servers, storage, network hardware, and virtualization). Therefore, customers can deploy the OS, middleware, and / or applications (e.g., self-service virtual machines, which can be spun up on demand).

[0346] In some cases, IaaS provisioning may include acquiring the computers or virtual hosts to be used and installing the necessary libraries or services on those computers or virtual hosts. In most cases, deployment does not include provisioning, and provisioning must be performed first.

[0347] In some cases, IaaS provisioning presents two distinct challenges. First, there's the challenge of provisioning an initial set of infrastructure before doing anything. Second, there's the challenge of evolving existing infrastructure after everything has been provisioned (e.g., adding new services, modifying services, removing services). In some cases, these two challenges can be addressed by enabling the declarative definition of infrastructure configuration. In other words, the infrastructure (e.g., what elements are needed and how these elements interact) may be defined by one or more configuration files. Thus, the overall topology of the infrastructure (e.g., which resources depend on which and how they work together) can be described declaratively. In some examples, once the topology is defined, workflows can be generated to create and / or manage the different elements described in the configuration files.

[0348] In some examples, infrastructure can include many interconnected elements. For example, there may be one or more virtual private clouds (VPCs), also known as core networks (e.g., configurable compute resources and / or potential on-demand pools of shared compute resources). In some examples, there may be one or more inbound and / or outbound traffic group rules and one or more virtual machines (VMs) provisioned to define how inbound and / or outbound traffic is configured for the network. Other infrastructure elements such as load balancers and databases may also be provisioned. As more and more infrastructure elements are desired and / or added, the infrastructure can evolve incrementally.

[0349] In some examples, sequential deployment techniques may be employed to enable the deployment of infrastructure code across various virtual computing environments. Furthermore, the techniques described can enable infrastructure management within these environments. In some examples, a service team may write code that is intended to be deployed to one or more different production environments, typically many different geographical locations, sometimes across the globe. However, in some examples, the infrastructure for deploying the code must first be configured. In some examples, provisioning can be done manually, resources can be provisioned using provisioning tools, and / or the code can be deployed using deployment tools after the infrastructure has been provisioned.

[0350] Figure 24 is a block diagram 2400 showing an exemplary pattern of an IaaS architecture according to at least one embodiment. The service operator 2402 may be communicably connected to a secure host tenancy 2404 which may include a virtual cloud network (VCN) 2406 and a secure host subnet 2408. In some examples, the service operator 2402 may use one or more client computing devices. One or more client computing devices may be handheld mobile devices (e.g., iPhone®, mobile phone, iPad®, tablet, personal digital assistant (PDA) or wearable device (Google® Glass® head-mounted display)) with Internet, email, short message service (SMS), BlackBerry® or other communication protocols enabled, and which may run software such as Microsoft Windows Mobile® and / or various mobile operating systems such as iOS®, Windows Phone, Android®, BlackBerry 8 and Palm OS. The client computing device is, as an example, Microsoft Windows (registered trademark). Operating System, Apple Macintosh® Operating System This refers to various versions of the Linux® operating system. The client computing device may be a general-purpose personal computer, including personal computers and / or laptop computers. Alternatively, the client computing device may be a workstation computer running various commercially available UNIX® or UNIX-like operating systems, including but not limited to various GNU / Linux operating systems, such as Google Chrome® OS. Alternatively or additionally, the client computing device may be other electronic devices that can communicate via the VCN2406 and / or a network with internet access, such as thin client computers, internet-enabled game systems (e.g., Microsoft Xbox® game consoles with or without Kinect® gesture input devices), and / or personal messaging devices.

[0351] VCN2406 may include a local peering gateway (LPG) 2410 that can communicate with Secure Shell (SSH) VCN2412 via LPG2410 included in SSH VCN2412. SSH VCN2412 may include an SSH subnet 2414, and SSH VCN2412 may communicate with control plane VCN2416 via LPG2410 included in control plane VCN2416. SSH VCN2412 may also communicate with data plane VCN2418 via LPG2410. Control plane VCN2416 and data plane VCN2418 may be included in a service tenancy 2419, which may be owned and / or operated by an IaaS provider.

[0352] The control plane VCN2416 may include a control plane DMZ (demilitarized zone) layer 2420 that functions as a perimeter network (e.g., the portion of the corporate network between the corporate intranet and the external network). DMZ-based servers have a certain level of reliability and can contain security breaches. Furthermore, the DMZ layer 2420 may include a control plane application layer 2424 that may include one or more load balancer (LB) subnets 2422 and application subnets 2426, and a control plane data layer 2428 that may include database (DB) subnets 2430 (e.g., a front-end DB subnet and / or a back-end DB subnet). The LB subnet 2422 included in the control plane DMZ layer 2420 may be communicatively connected to the application subnet 2426 included in the control plane application layer 2424 and the internet gateway 2434 which may be included in the control plane VCN 2416. The application subnet 2426 may be communicatively connected to the DB subnet 2430 included in the control plane data layer 2428, the service gateway 2436, and the network address translation (NAT) gateway 2438. The control plane VCN 2416 may include the service gateway 2436 and the NAT gateway 2438.

[0353] The control plane VCN2416 may include a data plane mirror application layer 2440, which may include an application subnet 2426. The application subnet 2426 included in the data plane mirror application layer 2440 may include a virtual network interface controller (VNIC) 2442 on which compute instance 2444 can run. Compute instance 2444 can communicate with the application subnet 2426 of the data plane mirror application layer 2440 to any application subnet 2426 that may be included in the data plane application layer 2446.

[0354] The data plane VCN2418 may include a data plane application layer 2446, a data plane DMZ layer 2448, and a data plane data layer 2450. The data plane DMZ layer 2448 may include an LB subnet 2422 that can be communicated to the application subnet 2426 of the data plane application layer 2446 and the internet gateway 2434 of the data plane VCN2418. The application subnet 2426 may be communicated to the service gateway 2436 of the data plane VCN2418 and the NAT gateway 2438 of the data plane VCN2418. Additionally, the data plane data layer 2450 may include a DB subnet 2430 that can be communicated to the application subnet 2426 of the data plane application layer 2446.

[0355] The Internet gateway 2434 of the control plane VCN2416 and the Internet gateway 2434 of the data plane VCN2418 may be communicatively connected to a metadata management service 2452, which may be communicatively connected to the public internet 2454. The public internet 2454 may be communicatively connected to the NAT gateway 2438 of the control plane VCN2416 and the NAT gateway 2438 of the data plane VCN2418. The service gateway 2436 of the control plane VCN2416 and the service gateway 2436 of the data plane VCN2418 may be communicatively connected to a cloud service 2456.

[0356] In some cases, a service gateway 2436 of the control plane VCN2416 or data plane VCN2418 can make application programming interface (API) calls to a cloud service 2456 without going through the public internet 2454. API calls from the service gateway 2436 to the cloud service 2456 can be one-way. The service gateway 2436 can make API calls to the cloud service 2456, and the cloud service 2456 can send request data to the service gateway 2436. However, the cloud service 2456 may not initiate an API call to the service gateway 2436.

[0357] In some examples, secure host tenancy 2404 may be directly connected to a potentially orphaned service tenancy 2419. Secure host subnet 2408 can communicate with SSH subnet 2414 via LPG 2410, which enables bidirectional communication with the orphaned system. By connecting secure host subnet 2408 to SSH subnet 2414, secure host subnet 2408 can access other entities within service tenancy 2419.

[0358] The control plane VCN2416 allows users of service tenancy 2419 to configure or provision desired resources. Desired resources provisioned in the control plane VCN2416 may be deployed or used in the data plane VCN2418. In some examples, the control plane VCN2416 may be isolated from the data plane VCN2418, and the data plane mirror application layer 2440 of the control plane VCN2416 can communicate with the data plane application layer 2446 of the data plane VCN2418 via a VNIC 2442 which may be included in the data plane mirror application layer 2440 and the data plane application layer 2446.

[0359] In some examples, a system user or customer may make requests, such as create, read, update, or delete (CRUD) operations, via the public internet 2454, which can communicate the requests to the metadata management service 2452. The metadata management service 2452 can communicate the requests to the control plane VCN 2416 via the internet gateway 2434. The requests are then sent to the control plane DMZ. The request may be received by the LB subnet 2422, which is included in layer 2420. The LB subnet 2422 may determine that the request is valid, and in response to this determination, the LB subnet 2422 may send the request to the application subnet 2426, which is included in the control plane application layer 2424. If the request is validated and requires a call to the public internet 2454, the call to the public internet 2454 may be sent to the NAT gateway 2438, which is capable of making calls to the public internet 2454. Memory for storing the request may be stored in the DB subnet 2430.

[0360] In some cases, the data plane mirror application layer 2440 can facilitate direct communication between the control plane VCN2416 and the data plane VCN2418. For example, it may be desirable that changes, updates, or other appropriate modifications to the configuration be applied to the resources contained in the data plane VCN2418. Since the control plane VCN2416 can communicate directly with the resources contained in the data plane VCN2418 via VNIC2442, it can perform changes, updates, or other appropriate modifications to the configuration.

[0361] In some embodiments, the control plane VCN2416 and the data plane VCN2418 may be included in the service tenancy 2419. In this case, the system user or customer does not have to own or operate either the control plane VCN2416 or the data plane VCN2418. Instead, the IaaS provider may own or operate the control plane VCN2416 and the data plane VCN2418, and both of these may be included in the service tenancy 2419. This embodiment can prevent a user or customer from interacting with other users' resources or other customers' resources by enabling network isolation. This embodiment can also enable a system user or customer to store databases privately without having to rely on the public internet 2454, which may not have the desired level of threat protection for storage.

[0362] In another embodiment, the LB subnet 2422 included in the control plane VCN2416 may be configured to receive signals from the service gateway 2436. In this embodiment, the control plane VCN2416 and the data plane VCN2418 may be configured to be invoked by the IaaS provider's customers without calling the public internet 2454. The IaaS provider's customers may prefer this embodiment because the database used by the customer may be stored in a service tenancy 2419 that is controlled by the IaaS provider and can be isolated from the public internet 2454.

[0363] Figure 25 is a block diagram 2500 showing another exemplary parameter of an IaaS architecture according to at least one embodiment. A service operator 2502 (e.g., service operator 2402 in Figure 24) may be communicably connected to a secure host tenancy 2504 (e.g., secure host tenancy 2404 in Figure 24), which may include a virtual cloud network (VCN) 2506 (e.g., VCN2406 in Figure 24) and a secure host subnet 2508 (e.g., secure host subnet 2408 in Figure 24). VCN2506 may include a local peering gateway (LPG) 2510 (e.g., LPG2410 in Figure 24), which may be communicably connected to a secure shell (SSH) VCN2512 (e.g., SSH VCN2412 in Figure 24) via an LPG2410 included in an SSH VCN2512. SSH VCN2512 may include SSH subnet 2514 (for example, SSH subnet 2414 in Figure 24), and SSH VCN2512 is accessed via LPG2510, which is included in the control plane VCN2516. It can be communicatively connected to the control plane VCN2524 (for example, the control plane VCN2416 in Figure 24). The control plane VCN2524 may be included in the service tenancy 2519 (for example, the service tenancy 2419 in Figure 24), and the data plane VCN2518 (for example, the data plane VCN2418 in Figure 24) may be included in the customer tenancy 2521, which may be owned or operated by a user or customer of the system.

[0364] The control plane VCN2516 may include a control plane DMZ layer 2520 (e.g., control plane DMZ layer 2420 in Figure 24) which may include an LB subnet 2522 (e.g., LB subnet 2422 in Figure 24), a control plane application layer 2516 (e.g., control plane application layer 2424 in Figure 24) which may include an application subnet 2526 (e.g., application subnet 2426 in Figure 24), and a control plane data layer 2528 (e.g., control plane data layer 2428 in Figure 24) which may include a database (DB) subnet 2530 (e.g., similar to DB subnet 2430 in Figure 24). The LB subnet 2522 included in the control plane DMZ layer 2520 may be communicated to the application subnet 2526 included in the control plane application layer 2516 and to an internet gateway 2534 (e.g., internet gateway 2434 in Figure 24) which may be included in the control plane VCN2516. The application subnet 2526 may be communicably connected to the DB subnet 2530, the service gateway 2536 (e.g., the service gateway in Figure 24), and the network address translation (NAT) gateway 2538 (e.g., the NAT gateway 2438 in Figure 24), which are included in the control plane data layer 2528. The control plane VCN 2516 may include the service gateway 2536 and the NAT gateway 2538.

[0365] The control plane VCN2516 may include a data plane mirror application layer 2540 (e.g., data plane mirror application layer 2440 in Figure 24) which may include an application subnet 2526. The application subnet 2526 included in the data plane mirror application layer 2540 may include a virtual network interface controller (VNIC) 2542 (e.g., VNIC2442) which can run a compute instance 2544 (e.g., similar to compute instance 2444 in Figure 24). The compute instance 2544 can facilitate communication between the application subnet 2526 of the data plane mirror application layer 2540 and the application subnet 2526 that may be included in the data plane application layer 2546 (e.g., data plane application layer 2446 in Figure 24) via the VNIC2542 included in the data plane mirror application layer 2540 and the VNIC2542 included in the data plane application layer 2546.

[0366] The Internet gateway 2534 included in the control plane VCN2516 may be communicably connected to a metadata management service 2552 (e.g., metadata management service 2452 in Figure 24), which may be communicably connected to the public internet 2554 (e.g., public internet 2454 in Figure 24). The public internet 2554 may be communicably connected to a NAT gateway 2538 included in the control plane VCN2516. The service gateway 2536 included in the control plane VCN2516 may be communicably connected to a cloud service 2556 (e.g., cloud service 2456 in Figure 24).

[0367] In some examples, the data plane VCN2518 may be included in the customer tenancy 2521. In this case, the IaaS provider can provide a control plane VCN2516 for each customer, and the IaaS provider can configure a unique compute instance 2544 included in the service tenancy 2519 for each customer. Each compute instance 2544 allows communication between the control plane VCN2516 included in the service tenancy 2519 and the data plane VCN2518 included in the customer tenancy 2521. Compute instance 2544 can allow resources provisioned in the control plane VCN2516, which is included in service tenancy 2519, to be deployed to or used in the data plane VCN2518, which is included in customer tenancy 2521.

[0368] In another example, an IaaS provider's customer may have a database residing in customer tenancy 2521. In this example, control plane VCN 2516 may include a data plane miner application tier 2540 that can include application subnet 2526. A data plane mirror application tier 2540 may reside in data plane VCN 2518, but does not have to reside in data plane VCN 2518. That is, a data plane mirror application tier 2540 can access customer tenancy 2521, but does not have to reside in data plane VCN 2518 and does not have to be owned or operated by an IaaS provider's customer. A data plane mirror application tier 2540 may be configured to make calls to data plane VCN 2518, but does not have to be configured to make calls to any entity contained in control plane VCN 2516. The customer may wish to deploy or use resources within the data plane VCN2518 provisioned to the control plane VCN2516, and the data plane mirror application tier 2540 can facilitate the customer's desired deployment or other use of resources.

[0369] In some embodiments, a customer of the IaaS provider can apply filters to the data plane VCN2518. In this embodiment, the customer can determine what the data plane VCN2518 can access and can restrict access from the data plane VCN2518 to the public internet 2554. The IaaS provider may not be able to apply filters or control access from the data plane VCN2518 to any external network or database. Applying filters and controls to the data plane VCN2518 included in the customer tenancy 2521 can help isolate the data plane VCN2518 from other customers and the public internet 2554.

[0370] In some embodiments, the cloud service 2556 can be invoked by the service gateway 2536 to access services that may not reside on the public internet 2554, on the control plane VCN 2516, or on the data plane VCN 2518. The connection between the cloud service 2556 and the control plane VCN 2516 or data plane VCN 2518 does not have to be live or continuous. The cloud service 2556 may reside on another network owned or operated by the IaaS provider. The cloud service 2556 may be configured to receive calls from the service gateway 2536 and not to receive calls from the public internet 2554. Some cloud services 2556 may be isolated from other cloud services 2556, and the control plane VCN 2516 may be isolated from cloud services 2556 that may not be located in the same region as the control plane VCN 2516. For example, the control plane VCN 2516 may be located in "Region 1", and the cloud service "Deployment 24" may be located in "Region 1" and "Region 2". If a call to deployment 24 is made by a service gateway 2536 included in the control plane VCN2516 located in region 1, this call may be sent to deployment 24 in region 1. In this example, the control plane VCN2516 or deployment 24 in region 1 does not need to be communicatively connected to deployment 24 in region 2.

[0371] Figure 26 is a block diagram 2600 showing another exemplary pattern of an IaaS architecture according to at least one embodiment. A service operator 2602 (e.g., service operator 2402 in Figure 24) may be communicably connected to a secure host tenancy 2604 (e.g., secure host tenancy 2404 in Figure 24), which may include a virtual cloud network (VCN) 2606 (e.g., VCN2406 in Figure 24) and a secure host subnet 2608 (e.g., secure host subnet 2408 in Figure 24). VCN2606 may include an LPG2610 (e.g., LPG2410 in Figure 24), which may be communicably connected to SSH VCN2612 (e.g., SSH VCN2412 in Figure 24) via an LPG2610 included in SSH VCN2612. SSH VCN2612 may include SSH subnet 2614 (e.g., SSH subnet 2414 in Figure 24), and SSH VCN2612 may be communicatively connected to control plane VCN2616 (e.g., control plane VCN2416 in Figure 24) via LPG2610 included in control plane VCN2616, and may be communicatively connected to data plane VCN2618 (e.g., data plane 2418 in Figure 24) via LPG2610 included in data plane VCN2618. Control plane VCN2616 and data plane VCN2618 may be included in service tenancy 2619 (e.g., service tenant 2419 in Figure 24).

[0372] The control plane VCN2616 may include a control plane DMZ layer 2620 (e.g., control plane DMZ layer 2420 in Figure 24) which may include a load balancer (LB) subnet 2622 (e.g., LB subnet 2422 in Figure 24), a control plane application layer 2624 (e.g., control plane application layer 2424 in Figure 24) which may include an application subnet 2626 (e.g., similar to application subnet 2426 in Figure 24), and a control plane data layer 2628 (e.g., control plane data layer 2428 in Figure 24) which may include a DB subnet 2630. The LB subnet 2622 included in the control plane DMZ layer 2620 may be communicably connected to the application subnet 2626 included in the control plane application layer 2624 and to an internet gateway 2634 (e.g., internet gateway 2434 in Figure 24) which may be included in the control plane VCN2616. The application subnet 2626 may be communicatively connected to the DB subnet 2630 included in the control plane data layer 2628, and to the service gateway 2636 (e.g., the service gateway in Figure 24) and the network address translation (NAT) gateway 2638 (e.g., the NAT gateway 2438 in Figure 24). The control plane VCN 2616 may include the service gateway 2636 and the NAT gateway 2638.

[0373] The data plane VCN2618 may include a data plane application layer 2646 (e.g., data plane application layer 2446 in Figure 24), a data plane DMZ layer 2648 (e.g., data plane DMZ layer 2448 in Figure 24), and a data plane data layer 2650 (e.g., data plane data layer 2450 in Figure 24). The data plane DMZ layer 2648 may include an LB subnet 2622 that can be communicatively connected to the trusted application subnet 2660 and untrusted application subnet 2662 of the data plane application layer 2646 and Internet gateway 2634 included in the data plane VCN2618. The trusted application subnet 2660 may be communicatively connected to the service gateway 2636 included in the data plane VCN2618, the NAT gateway 2638 included in the data plane VCN2618, and the DB subnet 2630 included in the data plane data layer 2650. The untrusted application subnet 2662 may be communicatively connected to the service gateway 2636 included in the data plane VCN 2618, and to the DB subnet 2630 included in the data plane data layer 2650. The data plane data layer 2650 may be communicatively connected to the DB subnet 2630 included in the data plane VCN 2618, which is the service gateway 2636 included in the data plane VCN 2618. It can include Net 2630.

[0374] An untrusted application subnet 2662 may include one or more primary VNICs 2664(1)-(N) that can be communicated to tenant virtual machines (VMs) 2666(1)-(N). Each tenant VM 2666(1)-(N) may be communicated to each application subnet 2667(1)-(N) that may be included in each container transmission VCN 2668(1)-(N) that may be included in each customer tenancy 2670(1)-(N). Each secondary VNIC 2672(1)-(N) can facilitate communication between the untrusted application subnet 2662 included in data plane VCN 2618 and the application subnets included in container transmission VCN 2668(1)-(N). Each container transmission VCN 2668(1)-(N) may include a NAT gateway 2638 that can be communicated to the public internet 2654 (e.g., public internet 2454 in Figure 24).

[0375] The Internet gateway 2634 included in the control plane VCN2616 and the Internet gateway 2634 included in the data plane VCN2618 may be communicatively connected to a metadata management service 2652 (e.g., the metadata management system 2452 in Figure 24), which can be communicatively connected to the public internet 2654. The public internet 2654 may be communicatively connected to the NAT gateway 2638 included in the control plane VCN2616 and the NAT gateway 2638 included in the data plane VCN2618. The service gateway 2636 included in the control plane VCN2616 and the service gateway 2636 included in the data plane VCN2618 may be communicatively connected to a cloud service 2656.

[0376] In some embodiments, the data plane VCN2618 may be integrated with the customer tenancy 2670. This integration may be useful or desirable for the IaaS provider's customer in some cases, such as when they may want support when executing code. The customer may provide code that, when executed, could be destructive, could communicate with other customer resources, or could cause undesirable effects. Thus, the IaaS provider can determine whether or not to execute the code that the customer has provided to the IaaS provider.

[0377] In some examples, an IaaS provider's customer may grant the IaaS provider temporary network access and request functionality to be added to the dataplane application layer 2646. The code to perform the functionality may run in VM2666(1)~(N), but cannot be configured to run elsewhere on the dataplane VCN2618. Each VM2666(1)~(N) may be connected to one customer tenancy 2670. Each container 2671(1)~(N) contained within VM2666(1)~(N) may be configured to run the code. In this case, a double isolation may exist (for example, container 2671(1)~(N) runs the code, and container 2671(1)~(N) may be contained in at least VM2666(1)~(N) that are in an untrusted application subnet 2662), which can help prevent erroneous or undesirable code from damaging the IaaS provider's network or the networks of different customers. Containers 2671(1)-(N) may be communicatively connected to customer tenancy 2670 and may be configured to send or receive data from customer tenancy 2670. Containers 2671(1)-(N) do not have to be configured to send or receive data from any other entities in the data plane VCN2618. Once code execution is complete, the IaaS provider may kill or discard containers 2671(I)-(N).

[0378] In some embodiments, a trusted application subnet 2660 may execute code that may be owned or operated by the IaaS provider. In this embodiment, the trusted application subnet 2660 may be communicatively connected to the DB subnet 2630 and configured to perform CRUD operations in the DB subnet 2630. An untrusted application subnet 2662 may be communicatively connected to the DB subnet 2630, but in this embodiment, the untrusted application subnet may be configured to perform read operations within the DB subnet 2630. Containers 2671(1)~(N) contained in each customer's VM2666(1)~(N) and capable of executing code from the customer do not need to be communicatively connected to the DB subnet 2630.

[0379] In other embodiments, the control plane VCN2616 and the data plane VCN2618 do not have to be directly coupled in a communicative manner. In this embodiment, direct communication between the control plane VCN2616 and the data plane VCN2618 may not exist. However, indirect communication by at least one method may exist. An LPG2610 that facilitates communication between the control plane VCN2616 and the data plane VCN2618 may be established by the IaaS provider. In another example, the control plane VCN2616 or the data plane VCN2618 can make a call to the cloud service 2656 via the service gateway 2636. For example, a call from the control plane VCN2616 to the cloud service 2656 may include a request for a service that can communicate with the data plane VCN2618.

[0380] Figure 27 is a block diagram 2700 showing another exemplary parameter of an IaaS architecture according to at least one embodiment. A service operator 2702 (e.g., service operator 2402 in Figure 24) may be communicably connected to a secure host tenancy 2704 (e.g., secure host tenancy 2404 in Figure 24), which may include a virtual cloud network (VCN) 2706 (e.g., VCN2406 in Figure 24) and a secure host subnet 2708 (e.g., secure host subnet 2408 in Figure 24). VCN2706 may include an LPG2710 (e.g., LPG2410 in Figure 24), which may be communicably connected to an SSH VCN2712 (e.g., SSH VCN2412 in Figure 24) via an LPG2710 included in SSH VCN2712. VCN2712 may include SSH subnet 2714 (e.g., SSH subnet 2414 in Figure 24), and SSH VCN2712 may be communicatively connected to control plane VCN2716 (e.g., control plane VCN2416 in Figure 24) via LPG2710 included in control plane VCN2716, and may be communicatively connected to data plane VCN2718 (e.g., data plane 2418 in Figure 24) via LPG2710 included in data plane VCN2718. Control plane VCN2716 and data plane VCN2718 may be included in service tenancy 2719 (e.g., service tenancy 2419 in Figure 24).

[0381] The control plane VCN2716 may include a control plane DMZ layer 2720 (e.g., control plane DMZ layer 2420 in Figure 24) which may include an LB subnet 2722 (e.g., LB subnet 2422 in Figure 24), a control plane application layer 2724 (e.g., control plane application layer 2424 in Figure 24) which may include an application subnet 2726 (e.g., application subnet 2426 in Figure 24), and a control plane data layer 2728 (e.g., control plane data layer 2428 in Figure 24) which may include a DB subnet 2730 (e.g., DB subnet 2630 in Figure 26). The LB subnet 2722 included in the control plane DMZ layer 2720 may include an application subnet 2726 included in the control plane application layer 2724 and an internet gateway 2734 (e.g., The application subnet 2726 may be communicatively connected to the DB subnet 2730 included in the control plane data layer 2728, and to the service gateway 2736 (e.g., the service gateway in Figure 24) and the network address translation (NAT) gateway 2738 (e.g., the NAT gateway 2438 in Figure 24). The control plane VCN 2716 may include the service gateway 2736 and the NAT gateway 2738.

[0382] The data plane VCN2718 may include a data plane application layer 2746 (e.g., data plane application layer 2446 in Figure 24), a data plane DMZ layer 2748 (e.g., data plane DMZ layer 2448 in Figure 24), and a data plane data layer 2750 (e.g., data plane data layer 2450 in Figure 24). The data plane DMZ layer 2748 may include a trusted application subnet 2760 (e.g., trusted application subnet 2660 in Figure 26) and an untrusted application subnet 2762 (e.g., untrusted application subnet 2662 in Figure 26) of the data plane application layer 2746, and an LB subnet 2722 that can be communicably connected to the internet gateway 2734 included in the data plane VCN2718. A trusted application subnet 2760 may be communicatively connected to a service gateway 2736 included in data plane VCN 2718, a NAT gateway 2738 included in data plane VCN 2718, and a DB subnet 2730 included in data plane data layer 2750. An untrusted application subnet 2762 may be communicatively connected to a service gateway 2736 included in data plane VCN 2718, and a DB subnet 2730 included in data plane data layer 2750. The data plane data layer 2750 may include a DB subnet 2730 that can be communicatively connected to a service gateway 2736 included in data plane VCN 2718.

[0383] An untrusted application subnet 2762 may include primary YNICs 2764(1)-(N) that can communicate with tenant virtual machines (VMs) 2766(1)-(N) residing in the untrusted application subnet 2762. Each tenant VM 2766(1)-(N) may execute code in its respective container 2767(1)-(N) and may be communicated with an application subnet 2726 that may be included in a data plane application layer 2746 that may be included in a container-transmitting VCN 2768. Each secondary VNIC 2772(1)-(N) can facilitate communication between the untrusted application subnet 2762 included in a data plane VCN 2718 and the application subnet included in a container-transmitting VCN 2768. The container-transmitting VCN may include a NAT gateway 2738 that can communicate with the public internet 2754 (e.g., public internet 2454 in Figure 24).

[0384] The Internet gateway 2734 included in the control plane VCN2716 and the Internet gateway 2734 included in the data plane VCN2718 may be communicatively connected to a metadata management service 2752 (e.g., the metadata management system 2452 in Figure 24), which can be communicatively connected to the public internet 2754. The public internet 2754 may be communicatively connected to the Internet gateway 2734 included in the control plane VCN2716 and the NAT gateway 2738 included in the data plane VCN2718. The Internet gateway 2734 included in the control plane VCN2716 and the service gateway 2736 included in the data plane VCN2718 may be communicatively connected to a cloud service 2756.

[0385] In some examples, the architecture is shown in block diagram 2700 of Figure 27. This pattern can be considered an exception to the pattern shown by the architecture in block diagram 2600 of Figure 26, and may be desirable for the IaaS provider's customers when the IaaS provider cannot communicate directly with the customers (e.g., in a disconnected area). Customers can access in real time each container 2767(1)~(N) contained within each customer's VM2766(1)~(N). Containers 2767(1)~(N) may be configured to call each secondary VNIC 2772(1)~(N) contained within the application subnet 2726 of the data plane application layer 2746, which may be contained within the container sending VCN 2768. Secondary VNICs 2772(1)~(N) can send calls to a NAT gateway 2738 which can send calls to the public internet 2754. In this example, the containers 2767(1) to (N) that customers can access in real time may be isolated from the control plane VCN2716 and from other entities included in the data plane VCN2718. Furthermore, the containers 2767(1) to (N) may be isolated from other customers' resources.

[0386] In another example, a customer can use containers 2767(1)-(N) to invoke cloud service 2756. In this example, the customer can execute code in containers 2767(1)-(N) to request a service from cloud service 2756. Containers 2767(1)-(N) can then send this request to secondary VNICs 2772(1)-(N), which can send the request to a NAT gateway that can send the request to the public internet 2754. The public internet 2754 can then send this request to LB subnet 2722, which is included in control plane VCN 2716, via internet gateway 2734. In response to determining that the request is valid, the LB subnet can send this request to application subnet 2726, which can then send this request to cloud service 2756 via service gateway 2736.

[0387] Note that the illustrated IaaS architectures 2400, 2500, 2600, and 2700 may include elements other than those shown. Furthermore, the illustrated embodiments are only examples of some cloud infrastructure systems that may incorporate embodiments of this disclosure. In some other embodiments, the IaaS system may have more or fewer elements than those shown, may combine two or more elements, or may have different configurations or arrangements of elements.

[0388] In certain embodiments, the IaaS systems described herein may include a suite of applications, middleware, and database services delivered to customers in a self-service, subscription-based, flexibly scalable, reliable, highly available, and secure manner. An example of such an IaaS system is Oracle® Cloud Infrastructure (OCI), provided by the applicant.

[0389] Figure 28 shows an exemplary computer system 2800 in which various embodiments may be implemented. System 2800 may be used to implement any of the computer systems described above. As shown, computer system 2800 includes a processing unit 2804 that communicates with a number of peripheral subsystems via a bus subsystem 2802. These peripheral subsystems may include a processing acceleration unit 2806, an I / O subsystem 2808, a storage subsystem 2818, and a communication subsystem 2824. The storage subsystem 2818 includes a tangible computer-readable storage medium 2822 and system memory 2810.

[0390] The bus subsystem 2802 provides a mechanism for various components and subsystems of the computer system 2800 to communicate with each other as intended. While the M2802 is schematically shown as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses. The bus subsystem 2802 may be any of several types of bus st...

Claims

1. It is a method, This includes generating a table for instances of a VLAN Switching and Routing Service (VSRS), wherein the VSRS connects a first virtual Layer 2 network (VLAN) to a second network, and the table includes information for identifying the IP addresses, MAC addresses, and virtual interface identifiers of instances within the virtual Layer 2 network. Using the aforementioned VSRS, packets are received that are delivered from the first instance to the second instance within the virtual Layer 2 network. Using the VSRS, the second instance in the virtual Layer 2 network for distributing the packet is identified based on the information received with the packet and the information contained in the table. A method comprising delivering the packet to the identified second instance.

2. The method according to claim 1, wherein the first virtual Layer 2 network includes a plurality of instances.

3. The VLAN comprises multiple L2 virtual network interface cards (VNICs) and multiple switches. The method according to claim 2, wherein each of the plurality of instances is commutably connected to a pair comprising a unique L2 virtual network interface card (VNIC) and a unique switch.

4. Using the VSRS, to identify the second instance in the virtual Layer 2 network for distributing the packet based on the information received with the packet and the information contained in the table, Using the VSRS, it is determined that the table does not contain mapping information for the second instance, Using the aforementioned VSRS, the distribution of the aforementioned packets is suspended, The process includes broadcasting an ARP request to a VNIC within the VLAN using the VSRS, wherein the ARP request includes the IP address of the second instance. The method according to any one of claims 1 to 3, comprising receiving an ARP response from the VNIC of the second instance using the VSRS.

5. The method according to claim 4, further comprising updating the table based on the received ARP response.

6. The method according to claim 2 or 3, wherein the first instance is located outside the virtual layer 2 network and inside the second network.

7. The method according to claim 6, wherein the second network includes an L3 network.

8. The method according to claim 6, wherein the second network includes a second virtual Layer 2 network.

9. The method according to claim 2 or 3, wherein the table is generated based on the information received by the VSRS.

10. The aforementioned VSRS is instantiated as a service on multiple hardware nodes. The method according to claim 9, further comprising:

11. The method according to claim 10, further comprising distributing the table among the hardware nodes.

12. The method according to claim 11, wherein the table distributed among the hardware nodes is available through another VSRS instantiation.

13. The method according to any one of claims 1 to 3, wherein the first instance is located inside the first virtual layer 2 network.

14. The VSRS further includes receiving packets from a third instance within the virtual Layer 2 network, and the packets being delivered to a fourth instance outside the virtual Layer 2 network. The method according to any one of claims 1 to 13, further comprising forwarding the packet to the fourth instance.

15. The VSRS further includes receiving packets from a third instance within the virtual Layer 2 network, The method according to any one of claims 1 to 13, wherein the packet is delivered to a service used by the third instance in the virtual Layer 2 network.

16. The method according to claim 15, wherein the service includes at least one of DHCP, NTP, and DNS.

17. The VSRS further includes receiving packets from a third instance within a virtual Layer 2 network, The method according to any one of claims 1 to 13, wherein the packet is delivered to a fourth instance in a second virtual Layer 2 network.

18. The method according to any one of claims 1 to 9, further comprising distributing the table for the instance of VSRS having Layer 2 and Layer 3 network information across a plurality of service nodes in order to provide highly reliable and highly scalable VSRS instantiation.

19. Using the aforementioned VSRS, packets are received from a third instance within the first virtual Layer 2 network, The method according to claim 1, further comprising learning the mapping of the third instance using the VSRS.

20. It is a system, It comprises at least one processor and a physical network including a network virtualization device, The aforementioned at least one processor is configured to perform the following operations: The aforementioned operation is, This includes instantiating an instance of a VLAN Switching and Routing Service (VSRS), the VSRS connecting a first virtual Layer 2 network to a second network, This includes generating a table for the instance of the VSRS, the table containing the IP address, MAC address, and other information of the instance in the virtual Layer 2 network. This includes information for identifying the virtual interface identifier, Using the aforementioned VSRS, packets are received that are delivered from the first instance to the second instance within the virtual Layer 2 network. Using the VSRS, the second instance in the virtual Layer 2 network for distributing the packet is identified based on the information received with the packet and the information contained in the table. A system including delivering the packets to the identified second instance.

21. A non-temporary computer-readable storage medium for storing multiple instructions that can be executed by one or more processors, wherein, when the multiple instructions are executed by the one or more processors, the one or more processors cause the following operations to be performed: The aforementioned operation is, This includes instantiating an instance of a VLAN Switching and Routing Service (VSRS), the VSRS connecting a first virtual Layer 2 network to a second network, This includes generating a table for the instance of the VSRS, the table including information for identifying the IP address, MAC address, and virtual interface identifier of the instance in the virtual Layer 2 network, Using the aforementioned VSRS, packets are received that are delivered from the first instance to the second instance within the virtual Layer 2 network. Using the VSRS, the second instance in the virtual Layer 2 network for distributing the packet is identified based on the information received with the packet and the information contained in the table. A non-temporary computer-readable storage medium, which includes delivering the packets to the identified second instance.

Images