Geometry-based Flow Programming

Abstract

This specification discloses a system and method for geometric-based flow programming. The method includes receiving, at a first network virtualization device ("NVD"), at least one compiled rule, each of the at least one compiled rule being applicable to a class of packets received by the first NVD for delivery to a virtual network interface card ("VNIC"). The method can include receiving, at the first NVD, a first packet for delivery to a first VNIC, using the first NVD to determine that a first rule of the at least one compiled rule is applicable to the first packet, and using the first NVD to process the first packet according to the first rule.

Description

Technical Field

[0001] Cross - reference to Related Applications This application claims priority to U.S. Patent Application No. 17 / 845,779, filed on June 21, 2022, entitled "GEOMETRIC BASED FLOW PROGRAMMING", which is hereby incorporated by reference in its entirety.

Background Art

[0002] Background This application relates to packet processing within a network. Specifically, communication within a Virtual Cloud Network (VCN) passes between Virtual Network Interface Cards (VNICs) through one or more Network Virtualization Devices (NVDs). Rules can be pre - loaded into the NVDs, and one of those rules can be used to process packets. These rules provided to the NVDs can grow cumulatively large and become difficult to handle. Therefore, improvement is desired.

Summary of the Invention

[0003] Brief Summary One aspect of the present disclosure relates to a method that includes receiving, at a first network virtualization device (“NVD”), at least one compiled rule. In some embodiments, each of the at least one compiled rules is applicable to a class of packets received by the first NVD for delivery to a virtual network interface card (“VNIC”). The method can include receiving, at the first NVD, a first packet for delivery to a first VNIC; determining, using the first NVD, that a first rule of the at least one compiled rules is applicable to the first packet; and processing, using the first NVD, the first packet according to the first rule.

[0004] In some embodiments, the first packet is processed according to the first rule and according to at least one attribute of the first packet. In some embodiments, the at least one attribute of the packet includes at least one of a destination address and a source address. In some embodiments, the source address includes at least one of a source IP address and a source port identifier. In some embodiments, the destination address includes at least one of a destination IP address and a destination port identifier.

[0005] In some embodiments, each of the at least one compiled rules includes a flow key, a security policy, and mapping information. In some embodiments, the security policy can include a local security policy and / or a remote security policy. In some embodiments, each of the at least one compiled rules further includes packet transformation information and throttle information.

[0006] In some embodiments, the method includes receiving, at a first NVD, a second packet for delivery to a second VNIC, determining, using the first NVD, that the first NVD does not have a rule for processing the second packet, and forwarding the second packet to at least one server. In some embodiments, the at least one server can process the second packet. In some embodiments, the at least one server can include configuration information for a set of VNICs within at least one virtual network.

[0007] In some embodiments, the method includes determining, using at least one server, a rule for processing the second packet, and processing, using the at least one server, the second packet according to the determined rule. In some embodiments, the rule is determined based on configuration information and at least one attribute of the second packet. In some embodiments, the at least one attribute of the packet includes at least one of a destination address and a source address. In some embodiments, the source address includes at least one of a source IP address and a source port identifier. In some embodiments, the destination address includes at least one of a destination IP address and a destination port identifier.

[0008] In some embodiments, the method includes generating a new compiled rule and providing the new compiled rule to a first NVD. In some embodiments, the new compiled rule includes a second rule. In some embodiments, generating the new compiled rule comprises identifying possible rules for each of a plurality of stages, the possible rules being associated with a plurality of packet flows, and forming the new compiled rule from common portions between the possible rules for each of the plurality of stages of the flow. In some embodiments, the method includes identifying a plurality of NVDs as being similar to the first NVD and providing the new compiled rule to the plurality of NVDs that are similar to the first NVD. In some embodiments, the plurality of stages includes two or more of a routing stage, a security stage, a throttle stage, and a translation stage.

[0009] One aspect of the present disclosure relates to a system. The system includes at least one server that includes a plurality of compiled rules, and a first network virtualization device (“NVD”). The first NVD can receive at least one compiled rule. In some embodiments, each of the at least one compiled rules is applicable to a class of packets received by the first NVD for delivery to a virtual network interface card (“VNIC”). The first NVD receives a first packet for delivery to a first VNIC, determines that a first rule of the at least one compiled rules is applicable to the first packet, and can be further configured to process the first packet according to the first rule. In some embodiments, the first NVD further receives a second packet for delivery to a second VNIC, determines that the first NVD does not have a rule for processing the second packet, and can transfer the second packet to at least one server. In some embodiments, the at least one server can include configuration information of a set of VNICs within at least one virtual network, and the at least one server can determine a rule for processing the second packet, where the rule is determined based on the configuration information and at least one attribute of the second packet, and process the second packet according to the determined rule.

[0010] One aspect of the present disclosure relates to a non-transitory computer-readable storage medium storing a plurality of instructions executable by one or more processors of a first network virtualization device ("NVD"). In some embodiments, when the plurality of instructions are executed by one or more processors of the first NVD, the one or more processors of the first NVD are caused to receive at least one compiled rule, receive a first packet for delivery to a first VNIC, determine that a first rule of the at least one compiled rule is applicable to the first packet, and process the first packet according to the first rule. In some embodiments, each of the at least one compiled rules is applicable to a class of packets received by the first NVD for delivery to a virtual network interface card ("VNIC"). BRIEF DESCRIPTION OF THE DRAWINGS

[0011]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

DETAILED DESCRIPTION OF THE INVENTION

[0012] Detailed Description Exemplary Virtual Network Architecture The term cloud service is usually used to refer to services that are made available on demand (e.g., by a subscription model) by a cloud service provider (CSP) for use by a user or customer who uses the systems and infrastructure (cloud infrastructure) provided by the CSP. Typically, the servers and systems that make up the CSP's infrastructure are separate from the customer's own on-premises servers and systems. Thus, the customer can utilize the cloud services provided by the CSP without having to purchase separate hardware resources and software resources for the services. Cloud services are designed to provide subscribing customers with easy and scalable access to applications and computing resources without the customer having to invest in procuring the infrastructure used to provide the services.

[0013] There are multiple cloud service providers that offer various types of cloud services. There are various types or models of cloud services, including SaaS (Software-as-a-Service), PaaS (Platform-as-a-Service), IaaS (Infrastructure-as-a-Service), and the like.

[0014] A customer can subscribe to one or more cloud services provided by a CSP. The customer can be any entity such as an individual, an organization, a business, etc. When a customer subscribes or registers for the services provided by a CSP, a tenant or account for that customer is created. The customer can then access, via that account, one or more subscribed cloud resources associated with the account.

[0015] As described above, IaaS (infrastructure as a service) is a specific type of cloud computing service. In the IaaS model, the CSP constructs a customizable network for the customer and provides the infrastructure (referred to as cloud service provider infrastructure or CSPI) that can be used by the customer to deploy the customer's resources. Thus, the customer's resources and network are hosted in a distributed environment by the infrastructure provided by the CSP. This is different from conventional computing where the customer's resources and network are hosted by the infrastructure provided by the customer.

[0016] CSPI can comprise interconnected high-performance computing resources, including various host machines, memory resources, and network resources that form a physical network also known as the underlying network or substrate network. The resources in CSPI can be geographically distributed across one or more data centers that can be geographically dispersed across one or more geographical regions. To provide a virtualized distributed environment, virtualization software may be executed by these physical resources. This virtualization creates an overlay network (also known as a software-based network, software-defined network, or virtual network) on top of the physical network. The physical network of CSPI provides the foundation for creating one or more overlay networks or virtual networks on top of the physical network. The physical network (or underlying network or substrate network) includes physical network devices such as physical switches, routers, computers, and host machines. An overlay network is a logical (or virtual) network that operates on top of the physical underlying network. A particular physical network can support one or more overlay networks. Overlay networks typically use encapsulation techniques to distinguish traffic belonging to different overlay networks. The virtual network or overlay network is also referred to as a virtual cloud network (VCN). The virtual network is implemented using software virtualization techniques (e.g., hypervisors, network virtualization devices (NVDs) (e.g., smart NICs), top-of-rack (TOR) switches, virtualization functions implemented by smart TORs that implement one or more functions executed by NVDs, and other mechanisms) to create a layer of network abstraction that can be executed on top of the physical network. The virtual network can take many forms, including peer-to-peer networks, IP networks, etc.A virtual network is typically either a Layer 3 IP network or a Layer 2 VLAN. This approach to virtual networks or overlay networks is often referred to as a virtual Layer 3 network or an overlay Layer 3 network. Examples of protocols developed for virtual networks include IP-in-IP (or Generic Routing Encapsulation (GRE)), Virtual Extensible LAN (VXLAN - IETF RFC 7348), Virtual Private Networks (VPNs) (e.g., MPLS Layer 3 Virtual Private Networks (RFC 4364)), VMware's NSX, GENEVE (Generic Network Virtualization Encapsulation), and others.

[0017] In the case of IaaS, the infrastructure provided by the CSP (CSPI) can be configured to provide virtualized computing resources via a public network (e.g., the Internet). In the IaaS model, a cloud computing service provider can host infrastructure components (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer), etc.). In some cases, the IaaS provider may provide various services (e.g., billing, monitoring, logging, security, load balancing, and clustering, etc.) as they arise in connection with those infrastructure components. Thus, since these services can be policy-driven, an IaaS user may be able to implement policies to drive load balancing to maintain the availability and performance of an application. CSPI provides infrastructure and a set of complementary cloud services that enable customers to build and run a wide range of applications and services within a highly available hosted distributed environment. CSPI provides high-performance computing resources, computing power, and storage capacity within a flexible virtual network that can be securely accessed from various networked locations, such as from the customer's on-premises network. When a customer subscribes to or registers for the IaaS services provided by the CSP, the tenancy created for that customer is a secure and isolated partition within the CSPI where the customer can create, orchestrate, and manage their cloud resources.

[0018] Customers can build their own virtual networks using the computing resources, memory resources, and network resources provided by the CSPI. One or more customer resources or workloads, such as computing instances, can be deployed to these virtual networks. For example, customers can use the resources provided by the CSPI to build one or more customizable private virtual networks called virtual cloud networks (VCNs). Customers can deploy one or more customer resources, such as computing instances, to their VCNs. Computing instances can take forms such as virtual machines, bare metal instances, etc. Thus, the CSPI provides infrastructure and a set of complementary cloud services that enable customers to build and run a wide range of applications and services within a highly available hosted virtual environment. Customers do not manage or control the underlying physical resources provided by the CSPI, but can control the operating system, storage, and deployed applications, and in some cases, can limitedly control selected network components (e.g., firewalls).

[0019] The CSP may provide a console that enables customers and network administrators to configure, access, and manage resources deployed in the cloud using CSPI resources. In one embodiment, this console provides a web-based user interface that can be used to access and manage the CSPI. In some implementations, the console is a web-based application provided by the CSP.

[0020] CSPI may support a single-tenancy architecture or a multi-tenancy architecture. In a single-tenancy architecture, a software component (e.g., an application, a database) or a hardware component (e.g., a host machine or a server) works for a single customer or tenant. In a multi-tenancy architecture, a software component or a hardware component works for multiple customers or tenants. Therefore, in a multi-tenancy architecture, CSPI resources are shared among multiple customers or tenants. In a multi-tenancy situation, preventive measures are taken and preventive means are introduced into CSPI to ensure that the data of each tenant is separated and remains invisible to other tenants.

[0021] In a physical network, a network endpoint (the "endpoint") refers to a computing device or system that is connected to the physical network and communicates with the destination network. Network endpoints within a physical network may be connected to a local area network (LAN), a wide area network (WAN), or other types of physical networks. Examples of conventional endpoints within a physical network include modems, hubs, bridges, switches, routers, and other network devices, physical computers (or host machines), etc. Each physical device within a physical network has a fixed network address that can be used to communicate with the device. This fixed network address can be a layer 2 address (e.g., a MAC address), a fixed layer 3 address (e.g., an IP address), etc. In a virtual environment or virtual network, endpoints can include various virtual endpoints such as virtual machines hosted by components of the physical network (e.g., hosted by a physical host machine). These endpoints within a virtual network are addressed by overlay addresses such as an overlay layer 2 address (e.g., an overlay MAC address) and an overlay layer 3 address (e.g., an overlay IP address). The network overlay enables flexibility by allowing network administrators to move between overlay addresses associated with network endpoints using software management (e.g., by software implementing the control plane of the virtual network). Thus, unlike a physical network, in a virtual network, an overlay address (e.g., an overlay IP address) can be moved from one endpoint to another using network management software. Since a virtual network is built on top of a physical network, communication between components within the virtual network involves both the virtual network and the underlying physical network.To facilitate such communication, the components of the CSPI are configured to learn and store mappings that map overlay addresses within the virtual network to actual physical addresses within the underlying network and vice versa. These mappings are then used to facilitate communication. Customer traffic is encapsulated to facilitate routing within the virtual network.

[0022] Thus, a physical address (e.g., a physical IP address) is associated with a component within the physical network, and an overlay address (e.g., an overlay IP address) is associated with an entity within the virtual network or overlay network. A physical IP address is an IP address associated with a physical device (e.g., a network device) within the underlying network or physical network. For example, each NVD has an associated physical IP address. An overlay IP address is an overlay address associated with an entity within the overlay network, such as being associated with a compute instance within a customer's virtual cloud network (VCN). Two different customers or tenants, each having their own private VCN, may be able to use the same overlay IP address within their VCNs without having any knowledge of each other. Both physical IP addresses and overlay IP addresses are types of real IP addresses. These IP addresses exist separately from virtual IP addresses. A virtual IP address is typically a single IP address that represents or maps to multiple real IP addresses. A virtual IP address provides a one-to-many mapping between the virtual IP address and multiple real IP addresses. For example, a load balancer may use a VIP to map to or represent multiple servers, and each server has its own real IP address.

[0023] Cloud infrastructure or CSPI is physically hosted within one or more data centers in one or more regions around the world. CSPI may include components within a physical network or underlying network, and virtual components (e.g., virtual networks, compute instances, virtual machines, etc.) within a virtual network built on top of the components of the physical network. In certain embodiments, CSPI is organized and hosted within realms, regions, and availability domains. A region is typically a local geographic area that includes one or more data centers. Regions are generally independent of each other and can be separated, for example, by vast distances spanning multiple countries or continents. For example, a first region may be in Australia, another region may be in Japan, and yet another region may be in India, etc. CSPI resources are divided among regions such that each region contains its own independent subset of CSPI resources. Each region may provide a set of core infrastructure services and resources such as compute resources (e.g., bare metal servers, virtual machines, containers, and related infrastructure), storage resources (e.g., block volume storage, file storage, object storage, archive storage), network resources (e.g., virtual cloud network (VCN), load balancing resources, connection to on-premises networks), database resources, edge network resources (e.g., DNS), as well as access management and monitoring resources. Each region generally has multiple paths connecting it to other regions within the realm.

[0024] Since using nearby resources is faster than using distant resources, generally an application is deployed within the region where the application is most frequently used (i.e., deployed to the infrastructure associated with that region). An application can also be deployed in different regions for various reasons, such as reducing the risk of region-wide events like large weather systems or earthquakes, meeting changing requirements regarding legal jurisdictions, tax regions, and having redundancy to meet other business or social criteria.

[0025] Data centers within a region can be further organized and subdivided into availability domains (ADs). An availability domain can correspond to one or more data centers located within a region. A region can be composed of one or more availability domains. In such a distributed environment, CSPI resources can be region-specific, such as a virtual cloud network (VCN), or availability domain-specific, such as a compute instance.

[0026] The ADs within a region are configured to be separated from each other, fault-tolerant, and have an extremely low probability of failing simultaneously. This is achieved by ADs that do not share important infrastructure resources such as networks, physical cables, cable routes, cable entry points, etc., so that a failure in one AD within a region is less likely to affect the availability of other ADs in the same region. ADs within the same region may be connected to each other by a low-latency high-bandwidth network that enables them to provide high-availability connections to other networks (such as the Internet, the customer's on-premises network, etc.), as well as to build systems replicated across multiple ADs for both high availability and disaster recovery. Cloud services use multiple ADs to ensure high availability and protect against resource failures. As the infrastructure provided by the IaaS provider increases, more regions and ADs with additional capabilities may be added. Traffic between availability domains is typically encrypted.

[0027] In certain embodiments, regions are grouped into realms. A realm is a logical collection of regions. Realms are separated from each other and do not share any data. Regions within the same realm may communicate with each other, but regions in different realms cannot communicate. A customer's tenancy or account exists within a single realm with the CSP and may be distributed across one or more regions belonging to that realm. Typically, when a customer subscribes to an IaaS service, a tenancy or account for that customer is created within the region (referred to as the "home" region) specified by the customer within the realm. The customer may extend the customer's tenancy across one or more other regions within the realm. A customer cannot access regions that are not within the realm in which the customer's tenancy exists.

[0028] An IaaS provider can offer multiple realms, and each realm caters to the requirements of a specific set of customers or users. For example, a commercial realm may be provided to commercial customers. As another example, a realm may be provided for customers within a certain country, specific to that country. As yet another example, a government realm may be provided to a government, etc. For example, a government realm may well be in response to the requirements of a specific government and may have a higher level of security than a commercial realm. For example, Oracle Cloud Infrastructure (OCI) currently offers a realm for commercial regions and two realms (e.g., FedRAMP certified and IL5 certified) for government cloud regions.

[0029] In some embodiments, an AD can be subdivided into one or more failure domains. A failure domain is a group of infrastructure resources within an AD that provides anti-affinity. A failure domain enables the distribution of compute instances such that multiple compute instances do not exist on the same physical hardware within a single AD. This distribution is known as anti-affinity. A failure domain refers to a set of hardware components (computers, switches, etc.) that share a single point of failure. The compute pool is logically divided into failure domains. Thus, a hardware failure or a compute hardware maintenance event that affects one failure domain does not affect instances within other failure domains. Depending on the embodiment, the number of failure domains per AD may vary. For example, in some embodiments, each AD includes three failure domains. A failure domain functions as a logical data center within an AD.

[0030] When a customer subscribes to an IaaS service, resources from the CSPI are provisioned for the customer and associated with the customer's tenancy. The customer can use these provisioned resources to build a private network and deploy resources to these networks. The customer's network hosted within the cloud by the CSPI is called a Virtual Cloud Network (VCN). The customer can use the CSPI resources allocated to the customer to set up one or more Virtual Cloud Networks (VCNs). A VCN is a virtual private network or software-defined private network. The customer's resources deployed within the customer's VCN can include compute instances (e.g., virtual machines, bare metal instances) and other resources. These compute instances can represent various customer workloads such as applications, load balancers, databases, etc. Compute instances deployed to a VCN can communicate with public endpoints that are publicly accessible via a public network such as the Internet ("public endpoints"), other instances within the same VCN or other VCNs (e.g., other VCNs of the customer, or VCNs not belonging to the customer), the customer's on-premises data center or network, as well as service endpoints and other types of endpoints.

[0031] CSP may provide various services using CSPI. In some cases, CSPI customers may act like service providers themselves and provide services using CSPI resources. A service provider may expose service endpoints characterized by identification information (e.g., IP address, DNS name, and DNS port). Customer resources (e.g., compute instances) can consume a particular service by accessing the service endpoints exposed by the service for that particular service. These service endpoints are generally endpoints that can be publicly accessed by users using the public IP address associated with the endpoint via a public communication network such as the Internet. Network endpoints that can be publicly accessed are sometimes called public endpoints.

[0032] In one embodiment, a service provider may expose a service via an endpoint for the service (which may be referred to as a service endpoint). A customer of the service can then use this service endpoint to access the service. In one implementation, the service endpoints provided for a service can be accessed by multiple customers attempting to consume that service. In other implementations, dedicated service endpoints may be provided to a customer such that only that customer can access the service using that dedicated service endpoint.

[0033] In one embodiment, when a VCN is created, the VCN is associated with a private overlay classless inter-domain routing (CIDR) address space that is the range of private overlay IP addresses assigned to the VCN (e.g., 10.0 / 16). The VCN includes associated subnets, route tables, and gateways. The VCN exists within a single region, but can span one or more or all of the region's availability domains. A gateway is a virtual interface configured for the VCN that enables communication of traffic between the VCN and one or more endpoints outside the VCN. One or more different types of gateways for the VCN may be configured to enable communication between different types of endpoints.

[0034] The VCN can be subdivided into one or more subnetwork such as one or more subnets. Thus, a subnet is a unit of configuration or subdivision that can be created within the VCN. The VCN can include one or more subnets. Each subnet within the VCN is associated with a contiguous range of overlay IP addresses (e.g., 10.0.0.0 / 24 and 10.0.1.0 / 24) that represents a subset of the address space within the VCN's address space and does not overlap with other subnets within that VCN.

[0035] Each compute instance is associated with a virtual network interface card (VNIC) that enables the compute instance to participate in a subnet of a VCN. A VNIC is a logical representation of a physical network interface card (NIC). Generally, a VNIC is an interface between an entity (e.g., a compute instance, a service) and a virtual network. A VNIC exists within a subnet and has one or more associated IP addresses, and associated security rules or security policies. A VNIC is equivalent to a layer 2 port on a switch. A VNIC is connected to a compute instance and to a subnet within a VCN. The VNIC associated with a compute instance enables the compute instance to become part of a subnet of the VCN and enables the compute instance to communicate (e.g., send and receive packets) with endpoints on the same subnet as the compute instance, endpoints in different subnets within the VCN, or endpoints outside the VCN. Thus, the VNIC associated with a compute instance determines how the compute instance connects to endpoints inside and outside the VCN. A VNIC for a compute instance is created and associated with that compute instance when the compute instance is created and added to a subnet within a VCN. In the case of a subnet that contains a set of compute instances, the subnet contains VNICs corresponding to the set of compute instances, and each VNIC is connected to one compute instance within the set of compute instances.

[0036] Each compute instance is assigned a private overlay IP address via the VNIC associated with the compute instance. This private overlay IP address is assigned to the VNIC associated with the compute instance when the compute instance is created and is used to route traffic to and from the compute instance. All VNICs within a particular subnet use the same route table, security list, and DHCP options. As described above, each subnet within a VCN represents a contiguous range of overlay IP addresses (e.g., 10.0.0.0 / 24 and 10.0.1.0 / 24) within the VCN's address space that do not overlap with other subnets within that VCN. For a VNIC on a particular subnet of a VCN, the private overlay IP address assigned to the VNIC is an address from the contiguous range of overlay IP addresses assigned to this subnet.

[0037] In some embodiments, a compute instance may optionally be assigned additional overlay IP addresses, such as one or more public IP addresses if, for example, it is within a public subnet, in addition to the private overlay IP address. These multiple addresses may be assigned to the same VNIC or across multiple VNICs associated with the compute instance. However, each instance has a primary VNIC that is created during instance startup and is associated with the private overlay IP address assigned to the instance, and this primary VNIC cannot be removed. Additional VNICs, called secondary VNICs, can be added to an existing instance within the same availability domain as the primary VNIC. All VNICs are within the same availability domain as the instance. A secondary VNIC can be within the same subnet within the same VCN as the primary VNIC, or within a different subnet within the same VCN or a different VCN.

[0038] If a compute instance is in a public subnet, the compute instance may optionally be assigned a public IP address. When a subnet is created, the subnet can be specified as either a public subnet or a private subnet. A private subnet means that resources within the subnet (e.g., compute instances) and associated VNICs cannot have public overlay IP addresses. A public subnet means that resources within the subnet and associated VNICs can have public IP addresses. A customer can specify subnets to exist within a single availability domain or across multiple availability domains within a region or realm.

[0039] As described above, a VCN may be subdivided into one or more subnets. In certain embodiments, a virtual router (VR) configured for a VCN (referred to as a VCN VR or simply a VR) enables communication between subnets of the VCN. For subnets within a VCN, the VR represents the logical gateway for that subnet, enabling the subnet (i.e., compute instances on that subnet) to communicate with endpoints on other subnets within the VCN and with other endpoints outside the VCN. The VCN VR is a logical entity configured to route traffic between VNICs within the VCN and a virtual gateway (the “gateway”) associated with the VCN. The gateway is further described below with respect to FIG. 1. The VCN VR is a Layer 3 / IP layer concept. In one embodiment, there is one VCN VR for one VCN, and the VCN VR has an unlimited number of ports, which are sometimes addressed by IP addresses, with one port for each subnet of the VCN. In this way, the VCN VR has a different IP address for each subnet within the VCN to which it is connected. The VR is also connected to various gateways configured for the VCN. In certain embodiments, a particular overlay IP address from the overlay IP address range of a subnet is reserved for the port of the VCN VR for that subnet. For example, consider a VCN that includes two subnets having associated address ranges 10.0 / 16 and 10.1 / 16, respectively. For the first subnet within the VCN having the address range 10.0 / 16, an address from this range is reserved for the port of the VCN VR for that subnet. In some cases, the first IP address from this range may be reserved for the VCN VR. For example, for a subnet having an overlay IP address range of 10.0 / 16, the IP address 10.0.0.1 may be reserved for the port of the VCN VR for that subnet.In the case of a second subnet within the same VCN having an address range of 10.1 / 16, the VCN VR may have a port for that second subnet having an IP address of 10.1.0.1. The VCN VR has a different IP address for each of the subnets within the VCN.

[0040] In some other embodiments, each subnet within the VCN may include a VR associated with itself that can be addressed by the subnet using a reserved IP address or a default IP address associated with the VR. The reserved IP address or default IP address may be, for example, the first IP address from the range of IP addresses associated with that subnet. VNICs within the subnet can use this default IP address or reserved IP address to communicate (e.g., send and receive packets) with the VR associated with the subnet. In such embodiments, the VR is the ingress / egress point of that subnet. The VRs associated with subnets within the VCN can communicate with other VRs associated with other subnets within the VCN. The VR can also communicate with the gateway associated with the VCN. The VR functionality of the subnet operates on or is executed by one or more NVDs that are executing the VNIC functionality of the VNICs within the subnet.

[0041] A route table, security rules, and DHCP options may be configured for the VCN. The route table is the virtual route table of the VCN and includes rules for routing traffic from subnets within the VCN to destinations outside the VCN via a gateway or a specially configured instance. The route table of the VCN can be customized to control how packets are transferred / routed between the VCN. DHCP options refer to the configuration information that is automatically provided to an instance when the instance is launched.

[0042] Security rules configured for a VCN represent the overlay firewall rules for the VCN. Security rules include ingress and egress rules and can specify the types of traffic (e.g., based on protocol and port) permitted into and out of instances within the VCN. A customer can choose whether a particular rule is stateful or stateless. For example, a customer can permit incoming SSH traffic to a set of instances from any location by setting a stateful ingress rule that includes source CIDR 0.0.0.0 / 0 and destination TCP port 22. Network security groups or security lists can be used to implement security rules. A network security group is composed of a set of security rules that apply only to resources within that group. A security list, on the other hand, includes rules that apply to all resources within any subnet that uses the security list. A VCN may have a default security list that includes default security rules. DHCP options configured for a VCN provide configuration information that is automatically provided to instances within the VCN when the instances are launched.

[0043] In one embodiment, the configuration information of the VCN is determined and stored by the VCN control plane. The configuration information of the VCN may include, for example, information regarding the address range associated with the VCN, subnets within the VCN and associated information, one or more VRs associated with the VCN, computing instances within the VCN and associated VNICs, NVDs (such as VNICs, VRs, gateways) that execute various virtualized network functions associated with the VCN, the status information of the VCN, as well as information related to other VCNs. In one embodiment, a VCN distribution service publishes the configuration information stored by the VCN control plane or a part thereof to the NVD. The distributed information may be used to update the information (such as forwarding tables, routing tables, etc.) stored and used by the NVD for transferring packets between computing instances within the VCN.

[0044] In one embodiment, the creation of the VCN and subnets is processed by the VCN control plane (CP: Control Plane), and the startup of computing instances is processed by the computing control plane. The computing control plane is responsible for allocating physical resources to the computing instances, and then calls the VCN control plane to create VNICs and connect them to the computing instances. The VCN CP also sends the VCN data mapping to the VCN data plane configured to perform packet transfer and routing functions. In one embodiment, the VCN CP provides a distribution service that is responsible for providing updates to the VCN data plane. Examples of the VCN control plane are also shown in FIGS. 12, 13, 14, and 15 (see reference numerals 1216, 1316, 1416, and 1516), and will be described below.

[0045] Customers may create one or more VCNs using resources hosted by CSPI. Compute instances deployed to a customer's VCN may communicate with various endpoints. These endpoints may include endpoints hosted by CSPI and endpoints outside of CSPI.

[0046] Various different architectures for implementing cloud-based services using CSPI are shown in FIGS. 1, 2, 3, 4, 5, 12, 13, 14, and 16 and are described below. FIG. 1 is a high-level diagram of a distributed environment 100 showing an overlay VCN or a customer's VCN hosted by CSPI according to one embodiment. The distributed environment shown in FIG. 1 includes multiple components within an overlay network. The distributed environment 100 shown in FIG. 1 is merely an example and is not intended to overly limit the scope of the claimed embodiments. Many variations, alternatives, and modifications are possible. For example, in some implementations, the distributed environment shown in FIG. 1 may include more or fewer systems or components than those shown in FIG. 1, may combine two or more subsystems, or may include different configurations or arrangements of the systems.

[0047] As shown in the example illustrated in FIG. 1, the distributed environment 100 includes CSPI 101 that provides services and resources that a customer may subscribe to and use to construct their virtual cloud network (VCN). In one embodiment, CSPI 101 provides to customers subscribed to IaaS services. The data centers within CSPI 101 may be organized into one or more regions. One exemplary region, "Region US" 102, is shown in FIG. 1. The customer configures their VCN 104 with respect to region 102. The customer may deploy various compute instances to VCN 104, and the compute instances may include virtual machines or bare metal instances. Examples of instances include applications, databases, load balancers, and the like.

[0048] In the embodiment shown in FIG. 1, the customer's VCN 104 includes two subnets, namely, "Subnet 1" and "Subnet 2", and each subnet has its own CIDR IP address range. In FIG. 1, the overlay IP address range of Subnet 1 is 10.0 / 16, and the address range of Subnet 2 is 10.1 / 16. The VCN virtual router 105 represents the logical gateway of the VCN that enables communication between the subnets of the VCN 104 and other endpoints outside the VCN. The VCN VR 105 is configured to route traffic between the VNICs within the VCN 104 and the gateways associated with the VCN 104. The VCN VR 105 provides ports for each subnet of the VCN 104. For example, the VR 105 may provide a port with the IP address 10.0.0.1 of Subnet 1 and a port with the IP address 10.1.0.1 of Subnet 2.

[0049] Multiple computing instances may be deployed to each subnet, and the computing instances can be virtual machine instances and / or bare metal instances. The computing instances within a subnet may be hosted by one or more host machines within CSPI 101. The computing instances participate in the subnet via VNICs associated with the computing instances. For example, as shown in FIG. 1, computing instance C1 becomes part of subnet 1 via the VNIC associated with this computing instance. Similarly, computing instance C2 becomes part of subnet 1 via the VNIC associated with C2. In a similar manner, multiple computing instances, which may be virtual machine instances or bare metal instances, may become part of subnet 1. Each computing instance is assigned a private overlay IP address and a MAC address via the VNIC associated with it. For example, in FIG. 1, computing instance C1 has an overlay IP address of 10.0.0.2 and a MAC address of M1, while computing instance C2 has a private overlay IP address of 10.0.0.3 and a MAC address of M2. Each computing instance within subnet 1, including computing instances C1 and C2, has a default route to VCN VR105 using the IP address 10.0.0.1, which is the IP address of the port of VCN VR105 of subnet 1.

[0050] Multiple compute instances, including virtual machine instances and / or bare metal instances, can be deployed in subnet 2. For example, as shown in FIG. 1, compute instances D1 and D2 become part of subnet 2 via their respective VNICs associated with each compute instance. In the embodiment shown in FIG. 1, compute instance D1 has an overlay IP address of 10.1.0.2 and a MAC address of MM1, while compute instance D2 has a private overlay IP address of 10.1.0.3 and a MAC address of MM2. Each compute instance within subnet 2, including compute instances D1 and D2, has a default route to VCN VR105 that uses IP address 10.1.0.1, which is the IP address of the port of VCN VR105 of subnet 2.

[0051] VCN A104 may include one or more load balancers. For example, a load balancer may be provided to a subnet and configured to load balance traffic across multiple compute instances on the subnet. A load balancer may be provided to load balance traffic across multiple subnets within a VCN.

[0052] Specific compute instances deployed on VCN104 can communicate with various endpoints. These endpoints may include endpoints hosted by CSPI200 and endpoints outside CSPI200. Endpoints hosted by CSPI101 can be endpoints on the same subnet as the specific compute instance (e.g., communication between two compute instances within subnet 1), endpoints on different subnets but within the same VCN (e.g., communication between a compute instance in subnet 1 and a compute instance in subnet 2), endpoints in different VCNs within the same region (e.g., communication between a compute instance in subnet 1 and an endpoint in a VCN within the same region 106 or 110, communication between a compute instance in subnet 1 and an endpoint in the service network 110 within the same region), or endpoints in VCNs in different regions (e.g., communication between a compute instance in subnet 1 and an endpoint in a VCN in a different region 108). Compute instances within the subnet hosted by CSPI101 may communicate with endpoints not hosted by CSPI101 (i.e., outside CSPI101). These outside endpoints include endpoints within the customer's on-premises network 116, endpoints within a network 118 hosted by another remote cloud, public endpoints 114 accessible via a public network such as the Internet, and other endpoints.

[0053] Communication between compute instances on the same subnet is facilitated using the VNICs associated with the source compute instance and the destination compute instance. For example, a compute instance C1 within subnet 1 may want to send a packet to a compute instance C2 within subnet 1. In the case of a packet originating from a source compute instance and having a destination that is another compute instance within the same subnet, the packet is first processed by the VNIC associated with the source compute instance. The processing performed by the VNIC associated with the source compute instance can include determining the destination information of the packet from the packet header, identifying any policies (e.g., security lists) configured for the VNIC associated with the source compute instance, determining the next hop of the packet, performing any packet encapsulation / decapsulation functions as necessary, and then forwarding / routing the packet to the next hop for the purpose of facilitating communication of the packet with the intended destination. If the destination compute instance is within the same subnet as the source compute instance, the VNIC associated with the source compute instance is configured to identify the VNIC associated with the destination compute instance and forward the packet to that VNIC for processing. Next, the VNIC associated with the destination compute instance is executed and the packet is forwarded to the destination compute instance.

[0054] In the case of packets transmitted from a compute instance within a subnet to an endpoint within a different subnet within the same VCN, this communication is facilitated by the VNICs associated with the source and destination compute instances as well as the VCN VR. For example, if compute instance C1 within subnet 1 of FIG. 1 wants to send a packet to compute instance D1 within subnet 2, this packet is first processed by the VNIC associated with compute instance C1. The VNIC associated with compute instance C1 is configured to route the packet to VCN VR105 using the default route of the VCN VR or port 10.0.0.1. VCN VR105 is configured to route the packet to subnet 2 using port 10.1.0.1. The packet is then received and processed by the VNIC associated with D1, and the VNIC transfers the packet to compute instance D1.

[0055] For packets transmitted from a compute instance within VCN104 to an endpoint outside of VCN104, communication is facilitated by the VNIC associated with the source compute instance, VCN VR105, and the gateway associated with VCN104. One or more types of gateways may be associated with VCN104. A gateway is an interface between a VCN and another endpoint, where the other endpoint is outside of the VCN. A gateway is a layer 3 / IP layer concept that enables a VCN to communicate with endpoints outside of the VCN. Thus, a gateway facilitates the traffic flow between a VCN and other VCNs or networks. Different types of gateways may be configured for a VCN to facilitate different types of communication with different types of endpoints. Depending on the gateway, communication may be via a public network (e.g., the Internet) or via a private network. Different communication protocols may be used for these communications.

[0056] For example, compute instance C1 may want to communicate with an endpoint outside of VCN104. The packet may first be processed by the VNIC associated with the source compute instance C1. This VNIC processing determines that the destination of the packet is outside of C1's subnet 1. The VNIC associated with C1 may forward the packet to VCN VR105 of VCN104. Next, VCN VR105 processes the packet and, as part of this processing, determines a specific gateway associated with VCN104 as the next hop of the packet based on the destination of the packet. Thereafter, VCN VR105 may forward the packet to the identified specific gateway. For example, if the destination is an endpoint within the customer's on-premises network, the packet may be forwarded by VCN VR105 to the Dynamic Routing Gateway (DRG) gateway 122 configured for VCN104. Next, the packet may be forwarded from the gateway to the next hop to facilitate delivery of the packet to the final intended destination.

[0057] Various types of gateways may be configured for a VCN. Examples of gateways that may be configured for a VCN are shown in FIG. 1 and described below. Examples of gateways associated with a VCN are also shown in FIGS. 12, 13, 14, and 15 (e.g., the gateways referenced by reference numerals 1234, 1236, 1238, 1334, 1336, 1338, 1434, 1436, 1438, 1534, 1536, and 1538) and described below. As shown in the embodiment illustrated in FIG. 1, a Dynamic Routing Gateway (DRG) 122 may be added to or associated with a customer's VCN 104 and provides a path for private network traffic communication between the customer's VCN 104 and another endpoint, which can be the customer's on-premises network 116, a VCN 108 in a different region of the CSPI 101, or another remote cloud network 118 not hosted by the CSPI 101. The customer's on-premises network 116 may be a customer's network or customer's data center built using the customer's resources. Access to the customer's on-premises network 116 is typically highly restricted. In the case of a customer having both the customer's on-premises network 116 and one or more VCNs 104 deployed or hosted within the cloud by the CSPI 101, the customer may desire that the customer's on-premises network 116 and the customer's cloud-based VCN 104 be able to communicate with each other. This enables the customer to build an extended hybrid environment that includes the customer's VCN 104 hosted by the CSPI 101 and the customer's on-premises network 116. The DRG 122 enables such communication. To enable such communication, a communication channel 124 is established, with one endpoint of this channel being within the customer's on-premises network 116 and the other endpoint being within the CSPI 101 and connected to the customer's VCN 104. The communication channel 124 can pass through a public communication network such as the Internet or a private communication network.Various communication protocols may be used, such as IPsec VPN technology via a public communication network such as the Internet, and Oracle's FastConnect technology that uses a private network instead of a public network. A device or equipment within the customer's on-premises network 116 that forms one endpoint of the communication channel 124 is called customer premise equipment (CPE), such as the CPE 126 shown in FIG. 1. On the CSPI 101 side, the endpoint may be a host machine that executes the DRG 122.

[0058] In one embodiment, a Remote Peering Connection (RPC) that enables a customer to peer one VCN with another VCN in a different region may be added to the DRG. Using such an RPC, the customer's VCN 104 can be connected to a VCN 108 in a different region using the DRG 122. The DRG 122 may also be used to communicate with other remote cloud networks 118 that are not hosted by the CSPI 101, such as the Microsoft Azure cloud, the Amazon AWS cloud, etc.

[0059] As shown in FIG. 1, an Internet Gateway (IGW) 120 may be configured for a customer's VCN 104 to enable a computing instance on the VCN 104 to communicate with a public endpoint 114 accessible via a public network such as the Internet. The IGW 120 is a gateway that connects the VCN to a public network such as the Internet. The IGW 120 enables direct access from a public subnet within the VCN, such as the VCN 104 (resources within the public subnet have public overlay IP addresses), to a public endpoint 112 on the public network 114 such as the Internet. Using the IGW 120, connections can be initiated from within the subnets of the VCN 104 or from the Internet.

[0060] A Network Address Translation (NAT) gateway 128 is configured for a customer's VCN 104 to enable access to the Internet for cloud resources within the customer's VCN that do not have dedicated public overlay IP addresses, and the NAT gateway 128 enables such access without exposing those resources to direct incoming Internet connections (e.g., L4-L7 connections). This enables private access from a private subnet within the VCN, such as the private subnet 1 within the VCN 104, to a public endpoint on the Internet. With the NAT gateway, only connections from the private subnet to the public Internet can be initiated, and connections from the Internet to the private subnet cannot be initiated.

[0061] In one embodiment, a Service Gateway (SGW) 126 can be configured for a customer's VCN 104 and provide a route for private network traffic between the VCN 104 and supported service endpoints within the service network 110. In one embodiment, the service network 110 may be provided by a CSP and may provide various services. An example of such a service network is Oracle's Services Network that provides various services that can be used by a customer. For example, a compute instance (e.g., a database system) within a private subnet of a customer's VCN 104 can back up data to a service endpoint (e.g., object storage) without the need for a public IP address or access to the Internet. In one embodiment, a VCN can include only one SGW and connections can be initiated only from subnets within the VCN and not from the service network 110. When a VCN is peered with another VCN, typically, resources within the other VCN do not have access to the SGW. Resources within an on-premises network connected to a VCN using FastConnect or a VPN connection can also use the service gateway configured for that VCN.

[0062] In one implementation, the SGW 126 uses the concept of a service classless inter-domain routing (CIDR) label, which is a string representing the public IP address range of all regions for a target service or group of services. A customer uses the service CIDR label when configuring the SGW and related route rules to control traffic to the service. Optionally, a customer can utilize the service CIDR label when configuring security rules so that if the public IP address of the service changes in the future, they do not need to adjust those security rules.

[0063] A Local Peering Gateway (LPG) 132 is a gateway that can be added to a customer's VCN 104 and enables the VCN 104 to peer with another VCN within the same region. Peering means that traffic communicates using private IP addresses without crossing a public network such as the Internet or routing traffic through the customer's on-premises network 116. In a preferred embodiment, a VCN includes a separate LPG for each peering established. Local peering or VCN peering is a common way used to establish network connections between different applications or infrastructure management functions.

[0064] Service providers, such as providers of services within the service network 110, may provide access to services using various access models. According to the public access model, the service may be publicly accessible by a compute instance within the customer's VCN via a public network such as the Internet and / or privately accessible via the SGW126 and may be published as a public endpoint. According to a particular private access model, the service is made accessible as a private IP endpoint within a private subnet within the customer's VCN. This access is referred to as private endpoint (PE:Private Endpoint) access and enables the service provider to publish the service as an instance within the customer's private network. The private endpoint resource represents the service within the customer's VCN. Each PE appears as a VNIC (referred to as a PE-VNIC having one or more private IPs) within a subnet selected by the customer within the customer's VCN. Thus, the PE provides a way to present the service within the private subnet of the customer's VCN using the VNIC. Since the endpoint is published as a VNIC, all features associated with the VNIC, such as routing rules, security lists, etc., are available for use by the PE VNIC.

[0065] The service provider can register the service to enable access via the PE. The provider can associate a policy that restricts the visibility of the service to the customer's tenancy with the service. The provider can register multiple services under a single virtual IP address (VIP:virtual IP address), especially in the case of multi-tenant services. There may be multiple such private endpoints (within multiple VCNs) representing the same service.

[0066] Next, the compute instances within the private subnet can access the service using the private IP address of the PE VNIC or the DNS name of the service. Compute instances within the customer's VCN can access the service by sending traffic to the private IP address of the PE within the customer's VCN. The Private Access Gateway (PAGW) 130 is a gateway resource that can be connected to the service provider's VCN (e.g., a VCN within the service network 110) and functions as the ingress / egress point for all traffic between the private endpoints of the customer's subnets. The PAGW 130 enables the provider to scale the number of connections to the PE without using internal IP address resources. The provider only needs to configure one PAGW for any number of services registered within a single VCN. The provider can represent the service as a private endpoint within the private endpoints of multiple VCNs of one or more customers. From the customer's perspective, the PE VNIC appears to be connected to the service the customer wishes to interact with instead of being connected to the customer's instance. Traffic going to the private endpoint is routed to the service via the PAGW 130. These are called customer-to-service (C2S) private connections.

[0067] The concept of the PE can also be used to extend private access to the service to the customer's on-premises network and data center by enabling traffic to flow through the FastConnect / IPsec link and the private endpoints within the customer's VCN. Private access to the service can also be extended to the customer's peered VCNs by enabling traffic to flow between the LPG 132 and the PE within the customer's VCN.

[0068] Customers can control routing within a VCN at the subnet level, so they can specify which subnets within their VCN, such as VCN 104, use each gateway. The VCN's route table is used to determine whether traffic is allowed to go outside the VCN through a particular gateway. For example, in a specific case, the route table for the public subnet within the customer's VCN 104 may send non-local traffic via IGW 120. The route table for the private subnet within the same customer's VCN 104 may send traffic going to the CSP service via SGW 126. All remaining traffic may be sent via NAT gateway 128. The route table controls only traffic going out of the VCN.

[0069] The security lists associated with a VCN are used to control traffic entering the VCN via an inbound connection through a gateway. All resources within a subnet use the same route table and security list. Security lists may be used to control specific types of traffic permitted to enter and leave instances within a VCN's subnet. Security list rules may include ingress (inbound) rules and egress (outbound) rules. For example, an ingress rule may specify an allowed source address range, while an egress rule may specify an allowed destination address range. Security rules may specify a particular protocol (e.g., TCP, ICMP), a particular port (e.g., 22 for SSH, 3389 for Windows RDP), etc. In one implementation, an instance's operating system may enforce its own firewall rules that match the security list rules. The rules may be stateful (e.g., the connection is tracked and responses are automatically allowed without using explicit security list rules for response traffic) or stateless.

[0070] Access from a customer's VCN (i.e., by resources or compute instances deployed to VCN 104) can be classified as public access, private access, or dedicated access. Public access refers to an access model in which a public IP address or NAT is used to access a public endpoint. Private access enables a customer's workload within VCN 104 having a private IP address (e.g., a resource within a private subnet) to access a service without traversing a public network such as the Internet. In some embodiments, CSPI 101 enables a customer's VCN workload having a private IP address to access a service (at its public service endpoint) using a service gateway. Thus, the service gateway provides a private access model by establishing a virtual link between the customer's VCN and the public endpoint of a service existing outside the customer's private network.

[0071] Furthermore, CSPI may provide dedicated public access using technologies such as FastConnect public peering, where a customer's on-premises instance can access one or more services within the customer's VCN using a FastConnect connection without traversing a public network such as the Internet. CSPI may also provide dedicated private access using FastConnect private peering, where a customer's on-premises instance with a private IP address can access the customer's VCN workloads using a FastConnect connection. FastConnect is a network connection that provides an alternative to using the public Internet to connect a customer's on-premises network to CSPI and its services. FastConnect provides an easy, flexible, and cost-effective way to create a dedicated private connection with higher bandwidth options and a more reliable and consistent network experience compared to Internet-based connections.

[0072] FIG. 1 and the accompanying description above describe various virtual components within an exemplary virtual network. As previously described, a virtual network is built on top of an underlying physical network or substrate network. FIG. 2 shows a simplified architecture diagram of the physical components in the physical network within CSPI200 that serves as the substrate for a virtual network, according to one embodiment. As shown in the figure, CSPI200 provides a distributed environment that includes components and resources (e.g., computing resources, memory resources, and network resources) provided by a cloud service provider (CSP). These components and resources are used to provide cloud services (e.g., IaaS services) to subscribing customers, i.e., customers who subscribe to one or more services provided by the CSP. Based on the services subscribed to by the customer, a subset of the resources of CSPI200 (e.g., computing resources, memory resources, and network resources) are provisioned for the customer. The customer can then use the physical computing resources, memory resources, and network resources provided by CSPI200 to build their own cloud-based (i.e., hosted by CSPI) customizable private virtual network. As already shown, these customer networks are referred to as virtual cloud networks (VCNs). The customer can deploy one or more customer resources, such as computing instances, to these customer VCNs. The computing instances can be in the form of virtual machines, bare metal instances, etc. CSPI200 provides the infrastructure and a set of complementary cloud services that enable customers to build and run a wide range of applications and services in a highly available hosted environment.

[0073] In the embodiment example shown in FIG. 2, the physical components of CSPI200 include one or more physical host machines or physical servers (e.g., 202, 206, 208), network virtualization devices (NVDs) (e.g., 210, 212), top-of-rack (TOR) switches (e.g., 214, 216), as well as a physical network (e.g., 218), and switches within the physical network 218. The physical host machine or physical server may host and execute various computing instances participating in one or more subnets of the VCN. The computing instances may include virtual machine instances and bare metal instances. For example, the various computing instances shown in FIG. 1 may be hosted by the physical host machines shown in FIG. 2. The virtual machine computing instances within the VCN may be executed by one host machine or by multiple different host machines. The physical host machine may host a virtual host machine, a container-based host, or functions, etc. The VNIC and VCN VR shown in FIG. 1 may be executed by the NVD shown in FIG. 2. The gateway shown in FIG. 1 may be executed by a host machine and / or by the NVD shown in FIG. 2.

[0074] The host machine or server may execute a hypervisor (also called a virtual machine monitor or VMM) that creates and enables a virtual environment on the host machine. The virtualized environment or virtual environment facilitates cloud-based computing. On the host machine, one or more computing instances may be created, executed, and managed by the hypervisor on that host machine. The hypervisor on the host machine enables the physical computing resources of the host machine (e.g., computing resources, memory resources, and network resources) to be shared among the various computing instances executed by the host machine.

[0075] For example, as shown in FIG. 2, host machines 202 and 208 respectively execute hypervisors 260 and 266. These hypervisors may be implemented using software, firmware, hardware, or a combination thereof. Typically, a hypervisor is a process or software layer that exists on top of the host machine's operating system (OS), and the operating system is executed on the host machine's hardware processor. The hypervisor provides a virtual environment by enabling the host machine's physical computing resources (e.g., processing resources such as processors / cores, memory resources, network resources) to be shared among the various virtual machine computing instances executed by the host machine. For example, in FIG. 2, hypervisor 260 may exist on top of the OS of host machine 202, enabling the computing resources (e.g., processing resources, memory resources, and network resources) of host machine 202 to be shared among the computing instances (e.g., virtual machines) executed by host machine 202. The virtual machines can have their own operating system (referred to as a guest operating system), which may be the same as or different from the host machine's OS. The operating systems of the virtual machines executed by a host machine may be the same as or different from the operating systems of other virtual machines executed by the same host machine. Thus, the hypervisor enables multiple operating systems to execute in parallel while sharing the same computing resources of the host machine. The host machines shown in FIG. 2 may have the same type or different types of hypervisors.

[0076] A compute instance can be a virtual machine instance or a bare metal instance. In FIG. 2, compute instance 268 on host machine 202 and compute instance 274 on host machine 208 are examples of virtual machine instances. Host machine 206 is an example of a bare metal instance provided to a customer.

[0077] In one example, an entire host machine may be provisioned to a single customer, and all of the one or more compute instances (either virtual machine or bare metal instances) hosted by that host machine belong to that same customer. In other examples, a host machine may be shared among multiple customers (i.e., multiple tenants). In such a multi-tenant situation, the host machine may host virtual machine compute instances belonging to different customers. These compute instances may be members of different VCNs of different customers. In some embodiments, a bare metal compute instance is hosted by a bare metal server without a hypervisor. When a bare metal compute instance is provisioned, a single customer or tenant maintains control of the physical CPUs, memory, and network interfaces of the host machine hosting the bare metal instance, and the host machine is not shared with other customers or tenants.

[0078] As described above, each computing instance that is part of a VCN is associated with a VNIC that enables the computing instance to be a member of a subnet of the VCN. The VNIC associated with the computing instance facilitates packet or frame communication between the computing instance and the VNIC. The VNIC is associated with the computing instance when the computing instance is created. In one embodiment, in the case of a computing instance executed by a host machine, the VNIC associated with the computing instance is executed by an NVD connected to the host machine. For example, in FIG. 2, host machine 202 executes virtual machine computing instance 268 associated with VNIC 276, and VNIC 276 is executed by NVD 210 connected to host machine 202. As another example, bare metal instance 272 hosted by host machine 206 is associated with VNIC 280 executed by NVD 212 connected to host machine 206. As yet another example, VNIC 284 is associated with computing instance 274 executed by host machine 208, and VNIC 284 is executed by NVD 212 connected to host machine 208.

[0079] In the case of a computing instance hosted by a host machine, the NVD connected to the host machine also executes the VCN VR corresponding to the VCN of which those computing instances are members. For example, in the embodiment shown in FIG. 2, NVD 210 executes VCN VR 277 corresponding to the VCN of which computing instance 268 is a member. NVDs 212 may execute one or more VCN VRs 283 corresponding to the VCNs corresponding to the computing instances hosted by host machines 206 and 208.

[0080] The host machine may include one or more network interface cards (NICs) that enable the host machine to be connected to other devices. The NIC on the host machine may provide one or more ports (or interfaces) that enable the host machine to be communicatively connected to another device. For example, the host machine may be connected to the NVD using one or more ports (or interfaces) provided on the host machine and the NVD. The host machine may also be connected to other devices such as another host machine.

[0081] For example, in FIG. 2, host machine 202 is connected to NVD 210 using link 220 that extends between port 234 provided by NIC 232 of host machine 202 and port 236 of NVD 210. Host machine 206 is connected to NVD 212 using link 224 that extends between port 246 provided by NIC 244 of host machine 206 and port 248 of NVD 212. Host machine 208 is connected to NVD 212 using link 226 that extends between port 252 provided by NIC 250 of host machine 208 and port 254 of NVD 212.

[0082] Next, the NVD is connected via a communication link to a top-of-rack (TOR) switch that is connected to the physical network 218 (also referred to as a switch fabric). In some embodiments, the links between the host machine and the NVD and between the NVD and the TOR switch are Ethernet® links. For example, in FIG. 2, links 228 and 230 are used to connect NVDs 210 and 212 to TOR switches 214 and 216, respectively. In some embodiments, links 220, 224, 226, 228, and 230 are Ethernet links. The collection of host machines and NVDs connected to the TOR is sometimes referred to as a rack.

[0083] The physical network 218 provides a communication fabric that enables the TOR switches to communicate with each other. The physical network 218 can be a multi-layer network. In one implementation, the physical network 218 is a multi-layer Clos network, and the TOR switches 214 and 216 represent the leaf-level nodes of the multi-layer multi-node physical switching network 218. Various Clos network configurations are possible, including, but not limited to, 2-layer networks, 3-layer networks, 4-layer networks, 5-layer networks, and generally, "n" layer networks. An example of a Clos network is shown in FIG. 5 and is described below.

[0084] Various connection configurations are possible between the host machine and the NVD, such as a one-to-one configuration, a many-to-one configuration, and a one-to-many configuration. In a one-to-one configuration implementation, each host machine is connected to its own separate NVD. For example, in FIG. 2, the host machine 202 is connected to the NVD 210 via the NIC 232 of the host machine 202. In a many-to-one configuration, multiple host machines are connected to one NVD. For example, in FIG. 2, the host machines 206 and 208 are each connected to the same NVD 212 via the NICs 244 and 250.

[0085] In a one-to-many configuration, one host machine is connected to multiple NVDs. FIG. 3 shows an example within CSPI 300 where a host machine is connected to multiple NVDs. As shown in FIG. 3, host machine 302 includes a network interface card (NIC) 304 that includes multiple ports 306 and 308. Host machine 300 is connected to a first NVD 310 via port 306 and link 320, and is connected to a second NVD 312 via port 308 and link 322. Ports 306 and 308 may be Ethernet ports, and links 320 and 322 between host machine 302 and NVDs 310 and 312 may be Ethernet links. Next, NVD 310 is connected to a first TOR switch 314, and NVD 312 is connected to a second TOR switch 316. The links between NVDs 310 and 312 and TOR switches 314 and 316 may be Ethernet links. TOR switches 314 and 316 represent layer 0 switching devices within a multi-layer physical network 318.

[0086] The arrangement shown in FIG. 3 provides two separate physical network paths between physical switch network 318 and host machine 302: a first path that crosses TOR switch 314 and passes through NVD 310 towards host machine 302, and a second path that crosses TOR switch 316 and passes through NVD 312 towards host machine 302. The separate paths result in improved availability (referred to as high availability) of host machine 302. If there is a problem with one of the paths (e.g., one of the links in a path fails) or a device (e.g., a particular NVD is not functioning), the other path may be used for communication with host machine 302.

[0087] In the configuration shown in FIG. 3, two different ports provided by the host machine's NIC are used for the host machine to be connected to two different NVDs. In other embodiments, the host machine may include multiple NICs that enable connection of the host machine to multiple NVDs.

[0088] Referring back to FIG. 2, the NVD is a physical device or component that executes one or more network and / or storage virtualization functions. The NVD can be any device having one or more processing units (e.g., CPU, network processing units (NPUs), FPGAs, packet processing pipelines, etc.), memory including caches, and ports. Various virtualization functions may be executed by software / firmware executed by one or more processing units of the NVD.

[0089] The NVD may be implemented in various forms. For example, in one embodiment, the NVD is implemented as an interface card called a smart NIC or intelligent NIC that incorporates an embedded processor. The smart NIC is a device separate from the NIC on the host machine. In FIG. 2, NVDs 210 and 212 may be implemented as smart NICs connected to host machines 202, and host machines 206 and 208, respectively.

[0090] However, the Smart NIC is but one example of an NVD implementation. Various other implementations are possible. For example, in some other implementations, the NVD or one or more functions performed by the NVD may be incorporated into or performed by one or more host machines, one or more TOR switches, and other components of the CSPI200. For example, the NVD may be embodied in a host machine, and the functions performed by the NVD are performed by the host machine. As another example, the NVD may be part of a TOR switch, or the TOR switch may be configured to perform the functions performed by the NVD, whereby the TOR switch can perform various complex packet conversions used in public clouds. A TOR that performs the functions of the NVD may sometimes be referred to as a Smart TOR. In yet other implementations where virtual machines (VM) instances rather than bare metal (BM) instances are provided to customers, the functions performed by the NVD may be implemented inside the hypervisor of the host machine. In some other implementations, some of the functions of the NVD may be offloaded to a centralized service operating on a series of host machines.

[0091] In one embodiment, such as when implemented as a Smart NIC as shown in FIG. 2, the NVD may include a plurality of physical ports that enable the NVD to be connected to one or more host machines and one or more TOR switches. The ports on the NVD can be classified as host-facing ports (also referred to as "south ports") or network-facing or TOR-facing ports (also referred to as "north ports"). The host-facing ports of the NVD are the ports used to connect the NVD to a host machine. Examples of host-facing ports in FIG. 2 include port 236 on NVD210, and ports 248 and 254 on NVD212. The network-facing ports of the NVD are the ports used to connect the NVD to a TOR switch. Examples of network-facing ports in FIG. 2 include port 256 on NVD210 and port 258 on NVD212. As shown in FIG. 2, NVD210 is connected to TOR switch 214 using link 228 that extends from port 256 of NVD210 to TOR switch 214. Similarly, NVD212 is connected to TOR switch 216 using link 230 that extends from port 258 of NVD212 to TOR switch 216.

[0092] The NVD may receive packets and frames (e.g., packets and frames generated by a compute instance hosted by the host machine) from the host machine via the host-facing ports, perform any necessary packet processing, and then transfer those packets and frames to the TOR switch via the network-facing ports of the NVD. The NVD may receive packets and frames from the TOR switch via the network-facing ports of the NVD, perform any necessary packet processing, and then transfer those packets and frames to the host machine via the host-facing ports of the NVD.

[0093] In some embodiments, there may be a plurality of ports and associated links between the NVD and the TOR switch. These ports and links may be aggregated to form a link aggregator group (referred to as a LAG (link aggregator group)) of the plurality of ports or links. Aggregation of the links enables multiple physical links between two endpoints (e.g., between the NVD and the TOR switch) to be treated as a single logical link. All physical links within a particular LAG may operate in full-duplex mode at the same speed. A LAG helps increase bandwidth and improve the reliability of the connection between two endpoints. If one of the physical links within the LAG fails, traffic is dynamically and transparently reallocated to one of the other physical links within the LAG. The aggregated physical links provide a higher bandwidth than individual links. The plurality of ports associated with the LAG are treated as a single logical port. Traffic may be load balanced across the plurality of physical links of the LAG. One or more LAGs may be configured between two endpoints. The two endpoints may exist, for example, between the NVD and the TOR switch, between a host machine and the NVD, etc.

[0094] The NVD implements or executes network virtualization functions. These functions are executed by the software / firmware executed by the NVD. Examples of network virtualization functions include, but are not limited to, packet encapsulation and decapsulation functions, functions for creating a VCN network, functions for implementing network policies such as VCN security list (firewall) functions, functions for facilitating packet routing and forwarding between compute instances within a VCN, etc. In one embodiment, upon receiving a packet, the NVD is configured to execute a packet processing pipeline to process the packet and determine how the packet is to be forwarded or routed. As part of this packet processing pipeline, the NVD may execute a VNIC associated with a compute instance within a VCN, execute a virtual router (VR) associated with the VCN, perform packet encapsulation and decapsulation to facilitate transfer or routing within a virtual network, execute a certain gateway (e.g., a local peering gateway), enforce security lists, network security groups, perform a network address translation (NAT) function (e.g., conversion from a public IP per host to a private IP), perform a bandwidth throttling function, and execute one or more virtual functions associated with an overlay network such as other functions.

[0095] In one embodiment, the packet processing data path within the NVD may comprise a plurality of packet pipelines, each configured with a series of packet transformation stages. In one implementation, upon receipt of a packet, the packet is parsed and classified into a single pipeline. Next, the packet is processed in a linear fashion, one stage at a time, until the packet is removed or transmitted via an NVD interface. These stages provide basic functional packet processing components (e.g., header authentication, bandwidth throttling, new layer 2 header insertion, L4 firewall implementation, VCN encapsulation / decapsulation, etc.) such that new pipelines can be constructed by assembling existing stages and new functionality can be added by creating new stages and inserting them into existing pipelines.

[0096] The NVD may perform both control plane functions and data plane functions corresponding to the control plane and data plane of the VCN. Examples of the VCN control plane are also shown in FIGS. 12, 13, 14, and 15 (see reference numerals 1216, 1316, 1416, and 1516), and are described below. Examples of the VCN data plane are shown in FIGS. 12, 13, 14, and 15 (see reference numerals 1218, 1318, 1418, and 1518), and are described below. The control plane functions include functions used to configure a network that controls how data is transferred (e.g., setting routes and route tables, configuring VNICs, etc.). In some embodiments, a VCN control plane is provided that centrally computes the mappings between all overlays and the substrate and publishes those mappings to the NVD and to virtual network edge devices such as various gateways like DRG, SGW, IGW. Firewall rules may be published using the same mechanism. In some embodiments, the NVD retrieves only the mappings relevant to that NVD. The data plane functions include functions for the actual routing / transfer of packets based on the configuration settings that use the control plane. The VCN data plane is implemented by encapsulating customer network packets before they cross the substrate network. The encapsulation / decapsulation function is implemented in the NVD. In some embodiments, the NVD is configured to intercept all network packets entering and leaving the host machine and perform network virtualization functions.

[0097] As shown above, the NVD executes various virtualization functions including VNIC and VCN VR. The NVD may execute a VNIC associated with a compute instance hosted by one or more host machines connected to the VNIC. For example, as shown in FIG. 2, the NVD 210 executes the functions of the VNIC 276 associated with the compute instance 268 hosted by the host machine 202 connected to the NVD 210. As another example, the NVD 212 executes the VNIC 280 associated with the bare metal compute instance 272 hosted by the host machine 206 and executes the VNIC 284 associated with the compute instance 274 hosted by the host machine 208. A host machine may host compute instances belonging to different VCNs belonging to different customers, and the NVD connected to the host machine may execute the VNIC corresponding to the compute instance (i.e., may execute the functions associated with the VNIC).

[0098] The NVD also executes a VCN virtual router corresponding to the VCN of the compute instance. For example, in the embodiment shown in FIG. 2, the NVD 210 executes the VCN VR 277 corresponding to the VCN to which the compute instance 268 belongs. The NVD 212 executes one or more VCN VRs 283 corresponding to one or more VCNs to which the compute instances hosted by the host machines 206 and 208 belong. In one embodiment, the VCN VR corresponding to the VCN is executed by all NVDs connected to the host machine hosting at least one compute instance belonging to that VCN. When a host machine hosts compute instances belonging to different VCNs, the NVD connected to that host machine may execute the VCN VRs corresponding to those different VCNs.

[0099] In addition to the VNIC and VCN VR, the NVD may run various software (e.g., daemons) and may include one or more hardware components that facilitate the various network virtualization functions performed by the NVD. For simplicity purposes, these various components are grouped together as the "packet processing components" shown in FIG. 2. For example, NVD210 includes packet processing component 286, and NVD212 includes packet processing component 288. For example, the packet processing components of the NVD may communicate with the ports and hardware interfaces of the NVD and may include a packet processor configured to monitor all packets received by the NVD and packets transmitted using the NVD and store network information. The network information may include, for example, network flow information identifying the various network flows processed by the NVD and flow-by-flow information (e.g., flow-by-flow statistics). In one embodiment, the network flow information may be stored per VNIC. In addition to performing per-packet operations, the packet processor may implement stateful NAT and L4 firewall (FW: firewall). As another example, the packet processing component may include a replication agent configured to replicate information stored by the NVD to one or more different replicated target stores. As yet another example, the packet processing component may include a logging agent configured to perform the logging function of the NVD. The packet processing component may also include software for monitoring the performance and health of the NVD and, optionally, the state and health of other components connected to the NVD.

[0100] FIG. 1 shows components of an exemplary virtual network or overlay network that includes a VCN, subnets within the VCN, compute instances deployed in the subnets, VNICs associated with the compute instances, a VR of the VCN, and a set of gateways configured for the VCN. The overlay components shown in FIG. 1 may be executed or hosted by one or more of the physical components shown in FIG. 2. For example, a compute instance within a VCN may be executed or hosted by one or more of the host machines shown in FIG. 2. In the case of a compute instance hosted by a host machine, the VNIC associated with that compute instance is typically executed by an NVD connected to that host machine (i.e., the VNIC functionality is provided by an NVD connected to that host machine). The VCN VR functionality of the VCN is executed by all NVDs connected to the host machine that hosts or executes a compute instance that is part of that VCN. The gateway associated with the VCN may be executed by one or more different types of NVDs. For example, one gateway may be executed by a smart NIC, while another gateway may be executed by one or more host machines or other implementations of NVDs.

[0101] As described above, compute instances within a customer's VCN may communicate with various endpoints, which can be within the same subnet as the source compute instance, within a different subnet within the same VCN as the source compute instance, or outside of the VCN of the source compute instance. These communications are facilitated using the VNIC associated with the compute instance, the VCN VR, and the gateway associated with the VCN.

[0102] In the case of communication between two compute instances on the same subnet within the VCN, the communication is facilitated using the VNICs associated with the source compute instance and the destination compute instance. The source compute instance and the destination compute instance may be hosted by the same host machine or by different host machines. Packets originating from the source compute instance may be transferred from the host machine hosting the source compute instance to the NVD connected to that host machine. In the NVD, the packets are processed using a packet processing pipeline, which can include the execution of the VNIC associated with the source compute instance. Since the destination endpoint of the packet is within the same subnet, the execution of the VNIC associated with the source compute instance causes the packet to be transferred to the NVD that is executing the VNIC associated with the destination compute instance, and then this VNIC processes the packet and transfers it to the destination compute instance. The VNICs associated with the source compute instance and the destination compute instance may be executed on the same NVD (e.g., when the source compute instance and the destination compute instance are both hosted by the same host machine) or on different NVDs (e.g., when the source compute instance and the destination compute instance are hosted by different host machines connected to different NVDs). The VNIC may use the routing table / forwarding table stored by the NVD to determine the next hop of the packet.

[0103] In the case of a packet transmitted from a compute instance within a subnet to an endpoint within a different subnet within the same VCN, the packet originating from the source compute instance is transmitted from the host machine hosting the source compute instance to the NVD connected to that host machine. At the NVD, the packet is processed using a packet processing pipeline, which can include the execution of one or more VNICs and the VR associated with the VCN. For example, as part of the packet processing pipeline, the NVD executes or invokes (also called running the VNIC) the function corresponding to the VNIC associated with the source compute instance. The function executed by the VNIC can include examining the VLAN tag on the packet. Since the destination of the packet is outside the subnet, the VCN VR function is then called and executed by the NVD. Next, the VCN VR routes the packet to the NVD running the VNIC associated with the destination compute instance. Next, the VNIC associated with the destination compute instance processes the packet and forwards the packet to the destination compute instance. The VNICs associated with the source compute instance and the destination compute instance may be executed on the same NVD (e.g., if the source compute instance and the destination compute instance are both hosted by the same host machine), or on different NVDs (e.g., if the source compute instance and the destination compute instance are hosted by different host machines connected to different NVDs).

[0104] If the destination of a packet is outside the VCN of the source compute instance, the packet originating from the source compute instance is transmitted from the host machine hosting the source compute instance to the NVD connected to that host machine. The NVD runs the VNIC associated with the source compute instance. Since the destination endpoint of the packet is outside the VCN, the packet is then processed by the VCN VR of that VCN. The NVD may invoke the VCN VR function, causing the packet to be transferred to the NVD running the appropriate gateway associated with the VCN. For example, if the destination is an endpoint within the customer's on-premises network, the packet may be transferred by the VCN VR to the NVD running the DRG gateway configured for the VCN. The VCN VR may be run on the same NVD as the NVD running the VNIC associated with the source compute instance, or by a different NVD. The gateway may be run by an NVD that can be an implementation of a smart NIC, a host machine, or another NVD. The packet is then processed by the gateway and transferred to the next hop that facilitates the transmission of the packet to the intended destination endpoint. For example, in the embodiment shown in FIG. 2, a packet originating from compute instance 268 may be transmitted from host machine 202 to NVD 210 via link 220 (using NIC 232). At NVD 210, VNIC 276 is called because it is the VNIC associated with source compute instance 268. VNIC 276 examines the encapsulated information within the packet, determines the next hop for transferring the packet for the purpose of facilitating the transmission of the packet to the intended destination endpoint, and then is configured to transfer the packet to the determined next hop.

[0105] The compute instances deployed in the VCN can communicate with various endpoints. These endpoints may include the endpoints hosted by the CSPI200 and the endpoints outside the CSPI200. The endpoints hosted by the CSPI200 may include instances within the same VCN or other VCNs, and these VCNs may be the customer's VCN or VCNs not belonging to the customer. Communication between the endpoints hosted by the CSPI200 may be performed via the physical network 218. The compute instances may also communicate with endpoints that are not hosted by the CSPI200 or are outside the CSPI200. Examples of these endpoints include endpoints within the customer's on-premises network or data center, or public endpoints accessible via a public network such as the Internet. Communication with endpoints outside the CSPI200 may be performed using various communication protocols via a public network (e.g., the Internet) (not shown in FIG. 2) or a private network (not shown in FIG. 2).

[0106] The architecture of the CSPI200 shown in FIG. 2 is merely an example and is not intended to be limiting. In alternative embodiments, variations, alternative means, and modifications are possible. For example, in some implementations, the CSPI200 may include more or fewer systems or components than those shown in FIG. 2, two or more systems may be combined, or different configurations or arrangements of the systems may be included. The systems, subsystems, and other components shown in FIG. 2 may be implemented using hardware, software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of each system, or a combination of these. The software may be stored on a non-transitory storage medium (e.g., a memory device).

[0107] Figure 4 shows the connection between a host machine and an NVD to implement I / O virtualization for supporting a multi-tenancy function according to an embodiment. As shown in Figure 4, host machine 402 runs hypervisor 404 that provides a virtual environment. Host machine 402 runs two virtual machine instances, VM1 406 belonging to customer / tenant 1 and VM2 408 belonging to customer / tenant 2. Host machine 402 includes a physical NIC 410 connected to NVD 412 via link 414. Each of the compute instances is connected to a VNIC executed by NVD 412. In the embodiment of Figure 4, VM1 406 is connected to VNIC-VM1 420 and VM2 408 is connected to VNIC-VM2 422.

[0108] As shown in Figure 4, NIC 410 includes two logical NICs, logical NIC A 416 and logical NIC B 418. Each virtual machine is connected to its own logical NIC and is configured to function with that logical NIC. For example, VM1 406 is connected to logical NIC A 416 and VM2 408 is connected to logical NIC B 418. Even if host machine 402 includes only one physical NIC 410 that is shared by multiple tenants, due to the logical NICs, each tenant's virtual machine is convinced that it has its own host machine and NIC.

[0109] In one embodiment, each logical NIC is assigned its own VLAN ID. Thus, a particular VLAN ID is assigned to the logical NIC A 416 of tenant 1, and another VLAN ID is assigned to the logical NIC B 418 of tenant 2. When a packet is transmitted from VM1 406, the tag assigned to tenant 1 is attached to the packet by the hypervisor, and then the packet is transmitted from the host machine 402 to the NVD412 via the link 414. In a similar manner, when a packet is transmitted from VM2 408, the tag assigned to tenant 2 is attached to the packet by the hypervisor, and then the packet is transmitted from the host machine 402 to the NVD412 via the link 414. Thus, the packet 424 transmitted from the host machine 402 to the NVD412 has an associated tag 426 that identifies a particular tenant and the associated VM. At the NVD, for the packet 424 received from the host machine 402, the tag 426 associated with the packet is used to determine whether the packet should be processed by the VNIC-VM1 420 or by the VNIC-VM2 422. Next, the packet is processed by the corresponding VNIC. The configuration shown in FIG. 4 allows one to be confident that the computing instances of each tenant own their own host machines and NICs. The setup shown in FIG. 4 enables I / O virtualization to support the multi-tenant function.

[0110] Figure 5 shows a simplified block diagram of a physical network 500 according to an embodiment. The embodiment shown in FIG. 5 is structured as a Clos network. A Clos network is a particular type of network topology designed to provide connection redundancy while maintaining high bisection bandwidth and maximum resource utilization. A Clos network is a type of non-blocking multi-stage or multi-layer switching network, and the number of its stages or layers can be two, three, four, five, etc. The embodiment shown in FIG. 5 is a three-layer network including layers 1, 2, and 3. The TOR switch 504 represents a layer 0 switch within the Clos network. One or more NVDS are connected to the TOR switch. The layer 0 switch is also called an edge device of the physical network. The layer 0 switch is connected to a layer 1 switch, also called a leaf switch. In the embodiment shown in FIG. 5, a set of "n" layer 0 TOR switches is connected to a set of "n" layer 1 switches, together forming a pod. Each layer 0 switch within a pod is interconnected with all the layer 1 switches within the pod, but there is no switch connection between pods. In one implementation, two pods are called a block. Each block is served by or connected to a set of "n" layer 2 switches (sometimes called spine switches). There can be multiple blocks in the physical network topology. Next, the layer 2 switches are connected to "n" layer 3 switches (sometimes called super spine switches). Communication of packets through the physical network 500 is typically performed using one or more layer 3 communication protocols. Usually, all layers of the physical network except the TOR layer have n-way redundancy, thus enabling high availability. Policies may be specified for pods and blocks to control the visibility of switches to each other within the physical network to enable scaling of the physical network.

[0111] The characteristic of a Clos network is that the maximum number of hops required to reach from one layer 0 switch to another layer 0 switch (or from an NVD connected to a layer 0 switch to another NVD connected to a layer 0 switch) is fixed. For example, in a 3-layer Clos network, a packet requires a maximum of 7 hops to reach from one NVD to another NVD, and the source NVD and the target NVD are connected to the leaf layer of the Clos network. Similarly, in a 4-layer Clos network, a packet requires a maximum of 9 hops to reach from one NVD to another NVD, and the source NVD and the target NVD are connected to the leaf layer of the Clos network. Therefore, the Clos network architecture maintains a consistent latency throughout the network, which is important for communication within and between data centers. The Clos topology scales horizontally and is cost-effective. By adding more switches (e.g., more leaf switches and spine switches) to different layers and increasing the number of links between switches in adjacent layers, the bandwidth / throughput capacity of the network can be easily increased.

[0112] In one embodiment, a unique identifier called a cloud identifier (CID) is assigned to each resource within the CSPI. This identifier is included as part of the resource's information and can be used to manage the resource, for example, via a console or via an API. An exemplary syntax of the CID is as follows. ocid1.<type of resource>.<realm>[.<region>[.<future use>].<unique ID> Here, ocid1: A literal string indicating the version of the CID. Type of resource: The type of resource (e.g., instance, volume, VCN, subnet, user, group, etc.). Realm: The realm where the resource exists. Exemplary values are "c1" for a commercial realm, "c2" for a government cloud realm, or "c3" for a federal government cloud realm, etc. Each realm may have its own domain name. Region: The region where the resource exists. This part may be blank if the region is not applicable to the resource. Future Use: Reserved for future use. Unique ID: The unique part of the ID. This format may vary depending on the type of resource or service.

[0113] Generation and Provisioning of Compiled Rules Communication within the VCN passes between VNICs via the NVD. Specifically, a packet can be received by the NVD, and then the NVD can pass, forward, redirect, and / or direct that packet to or towards the destination VNIC. The NVD uses the information associated with the received packet and one or more rules to determine how to process that received packet. These rules are typically stored in an individual NVD such that when the NVD receives a packet, the NVD processes the packet based on the rules stored in that NVD. Each of these rules is small in itself and does not consume a large amount of memory, but these rules can cumulatively become very large. The cumulative size of these rules can, in some embodiments, limit the number of VNICs on the NVD, thereby potentially increasing the number of NVDs within the VCN. This increase in the number of NVDs within the VCN increases the complexity and cost associated with an increase in the amount of hardware used within the network.

[0114] The present disclosure relates to techniques, systems, and methods for enabling an increase in VNIC density on an NVD. Specifically, the present disclosure includes more efficiently utilizing memory on the NVD by limiting the number of rules stored in the NVD. This includes creating compiled rules, which are associated with the processing of multiple different packets and / or flows. In some embodiments, the efficiency of the rules installed on the NVD can be improved and / or optimized by making the rules installed on that NVD as general-purpose as possible; in other words, each of the installed rules is made to target the maximum possible number of flows. Minimizing the number of rules required in the NVD by efficient rule compression, or in other words, maximizing the applicability of the rules.

[0115] The compiled rules can be created by identifying possible rules for each of a plurality of stages. These stages can include, for example, a routing stage, a security stage, a throttle stage, and a transformation stage. A common portion of the possible rules for each of the plurality of stages is determined, and the compiled rules are formed from the common portion of the possible rules.

[0116] The routing stage includes information for resolving the destination of the next hop of the packet received by the NVD. This routing stage can be derived, for example, from the customer's routing rules, L2 forwarding within a VLAN, or L3 forwarding to a VNIC inside a subnet and / or inside a VCN. The security stage includes information for determining whether the packet is permitted. In some embodiments, the security stage can include, for example, one or more subnet security lists and / or network security groups that can be accessed to determine whether the packet is permitted. The throttle stage can include information for setting a maximum delivery speed and / or a minimum delivery speed. The transformation stage can include information for performing any transformation and / or rewriting before delivering the packet. The transformation stage can include information for performing network address translation (NAT), port address translation (PAT), MAC address resolution, etc.

[0117] By using compiled rules, the memory requirements for rules in a Smart NIC can be reduced. Specifically, a standard application includes one rule per flow, while a compiled rule can be applied to multiple flows, thus minimizing the total number of rules to target all possible flows in the network. In addition to this, the embodiments disclosed herein relate to providing a limited number of compiled rules to the Smart NIC. This limited number can result in the Smart NIC not including rules for all possible flows in the network. In other words, a Smart NIC so configured may not have rules for all possible flows and can have rules for a subset of all possible flows. In some embodiments, this subset can be a subset of the most frequently used rules, rules that have been used by the Smart NIC, and / or rules that are expected to be used by the Smart NIC.

[0118] When the Smart NIC receives a packet for processing, the Smart NIC determines whether it has an applicable rule. If the Smart NIC has an applicable rule, the Smart NIC processes the packet. If the Smart NIC does not have an applicable rule, the Smart NIC passes the packet to at least one server, and that at least one server can be configured to process the packet. The at least one server, which can include one or more servers and / or a fleet of one or more servers, can, in some embodiments, include rules for all possible flows in the network.

[0119] At least one server receives packets, identifies rules for processing the packets, and then processes the packets according to the identified rules. Next, at least one server forms a compiled rule that includes the rules used to process the packets. The formation of this compiled rule includes identifying possible rules for each of the multiple stages of the rule, identifying common portions between the possible rules, and forming the compiled rule from the common portions of the possible rules. Thereafter, the compiled rule can be provided to the smart NIC from which packets for processing by at least one server are received. In some embodiments, the compiled rule can be provided to the smart NIC from which packets for processing by at least one server are received, and the compiled rule can be provided to one or more similar smart NICs.

[0120] FIG. 6 shows an embodiment of the NVD210 / 212. The NVD210 / 212 includes packet processing components 286 / 288. These packet processing components 286 / 288 can be configured to process packets received by the NVD210 / 212. Specifically, the packet processor 602 of the packet processing components 286 / 288 can be configured to process packets received by the NVD210 / 212. Processing the received packets can include, for example, interacting with the ports and hardware interfaces of the NVD210 / 212 and monitoring all packets received by the NVD210 / 212 and packets transmitted using the NVD210 / 212. In some embodiments, processing the received packets can further include identifying one or more rules applicable to the received packets and applying the one or more rules to the packets. In some embodiments, processing the received packets can include transmitting the packets to a next hop, such as a destination VNIC, and / or a destination.

[0121] The packet processing components 286 / 288 can be further configured to store network information. Specifically, the packet processing components 286 / 288 can include a rule memory 604 that can be configured to store network information. The network information can include, for example, network flow information that identifies various network flows processed by the NVD and flow-by-flow information (e.g., flow-by-flow statistics). In certain embodiments, the network flow information can be stored per VNIC. Based on this network information, which can include one or more rules, also referred to herein as configurations, the packet processing components 286 / 288 can process received packets. In some embodiments, the rules stored by the rule memory 604 can include one or more rules related to routing, security, throttling, and / or packet transformation.

[0122] In addition to performing per-packet operations, the packet processing components 286 / 288 can implement stateful NAT and a Layer 4 firewall (FW: firewall). As another example, the packet processing component can include a replication agent 606 configured to replicate information stored by the NVD to one or more different replicated target stores. As yet another example, the packet processing components 286 / 288 may include a logging agent 608 configured to perform the logging function of the NVD 210 / 212. The packet processing components 286 / 288 may also include software for monitoring the performance and health of the NVD 210 / 212, shown as a health monitor 610, and, optionally, software for monitoring the state and health of other components connected to the NVD 210 / 212.

[0123] FIG. 7 is a schematic diagram of one embodiment of a system 700 for packet processing and for generating and distributing compiled rules. System 700 includes a server fleet 702 that includes at least one instance, processor, computer, and / or server that communicates with a plurality of NVDs 210. In some embodiments, at least one processor, computer, and / or server within server fleet 702 can be located in a single location or can be located in different locations. In some embodiments, server fleet 702 can be instantiated in physical hardware and / or can be instantiated in software.

[0124] The server fleet can include a VCN distribution service 704, a configuration storage fleet 706, and a transfer rule prefetcher 708, also referred to herein as a prefetch service 708. Each of the VCN distribution service 704, the configuration storage fleet 706, and the transfer rule prefetcher 708 can be instantiated in hardware or can be instantiated in software. In some embodiments, each of the VCN distribution service 704, the configuration storage fleet 706, and the transfer rule prefetcher 708 can be instantiated in separate hardware, including, for example, one or more computers, processors, and / or servers.

[0125] In some embodiments, the VCN distribution server 704 can be configured to distribute low-level components to the NVDs, to the configuration storage fleet 706, and / or to the transfer rule prefetcher 708. In some embodiments, this can include, for example, that the low-level components can include rules applicable to a single flow and / or a single endpoint within the network and, in particular, within a virtual network.

[0126] Rules can include flow keys which can include, for example, match criteria. This match criteria can be used to match the rule against a packet. For the rule to be identified as matching a packet, the match criteria of the rule can match the attributes of the packet, specifically, when the packet is received from a host port on the endpoint in question, it can match the match criteria. The flow key can be generic in some embodiments, such that a flow offload instruction can refer to a "flow class" as opposed to specific five-tuples. The flow key can include, for example, a VNIC identifier, protocol, source CIDR and destination CIDR, and / or source port range and destination port range.

[0127] Rules can include a local security policy. The local security policy can indicate both inbound security rules and outbound security rules. In some embodiments, the local security policy can include indicating whether inbound packets and / or outbound packets are permitted by stateful rules or stateless rules. In some embodiments, if stateful, the local NVD can perform state tracking to permit inbound return packets.

[0128] Rules can include an override of the egress mapping. The override of the egress mapping can indicate where to send the packet and, in some embodiments, can override local mappings and / or route rules.

[0129] The rule can include a remote security policy. The remote security policy can include one or more entries, and these entries can be stored and transmitted with any packet overridden using a flow-offload policy. In some embodiments, if the local security policy is stateful and thus there is a stateful security policy, the sender can include the local flow state. In some embodiments, the remote security policy can indicate the type of security rule to permit traffic and, in particular, to permit the processing of packets. In some embodiments, this rule can be, for example, stateful or stateless. In some embodiments, the remote security policy can include a Reverse Path Forwarding (RPF) key. The RPF key can include a signature for authenticating which entity provided the override instruction. The RPF key may be able to trace back to the original expected sender that matches the local RPF check.

[0130] The rule can include packet transformation information. The packet transformation information can include instructions regarding how to change the packet. These instructions can include, for example, instructions such as NAT, PAT, etc.

[0131] The rule can include throttle selection information. In some embodiments, the throttle selection information can indicate and / or identify any classified throttle to apply to the packet.

[0132] The rules can include information that identifies a configuration version. In some embodiments, this configuration version can be used to determine that the rule is still valid. In some embodiments, the rule should be checked based on its version to determine that the rule is valid.

[0133] In some embodiments, if the rule is a prefetch rule, the rule can include information indicating that it is a prefetch rule. As used herein, a prefetch rule is a rule that is provided to the NVD before a packet that requires the use of that rule is received by the NVD. In some embodiments, the identification of a rule as a prefetch rule can be used to collect information for prioritizing rules based on actual traffic demands in order to prevent cases where unused rules, and in particular unused prefetch rules, flood available memory. In such embodiments, unused rules, and in particular unused prefetch rules, can be removed from the configuration cache first.

[0134] The configuration storage fleet 706, also referred to herein as the configuration storage service 706, can hold the configuration of a set of VNICs. In some embodiments, the configuration storage fleet 706 can include configuration information that can be one or more rules regarding the set of VNICs, and the set of VNICs can, in some embodiments, exist within at least one virtual network. In some embodiments, for example, the configuration storage fleet 706 can hold a configuration, or put another way, for example, rules for packet delivery to some or all of the VNICs within a network, including an availability domain (AD). The configuration storage fleet 706 processes packets of NVDs that are missing the relevant configuration or, put another way, do not have rules for processing the packets, and then installs those previously missing rules into the NVDs, thereby enabling those NVDs to process similar packets in the future. When these rules are installed into the NVDs, the rules can also be streamed to the prefetch fleet 708 so that the prefetch fleet 708 can recognize the rules available for each VNIC.

[0135] The prefetch fleet 708 is configured to detect potential candidate forwarding rules and install them into the NVDs. This includes determining and / or predicting rules that may be applicable to the NVDs, or put another way, determining and / or predicting NVDs that may need rules.

[0136] Therefore, during operation, when the NVD receives a packet, the receiving NVD can determine whether it has rules and / or configurations for processing that packet. If the NVD has rules and / or configurations, the NVD can process the packet. If the NVD does not have rules and / or configurations, the NVD can provide the packet to the server fleet 702, and in particular, to the configuration storage fleet 706. The configuration storage service 706 can identify rules for processing the packet and can process the packet. The configuration storage service 706 can further install those rules in the NVD. In some embodiments, the installation of those rules can include the creation of compiled rules that can include information for processing multiple flows. Next, the configuration storage service 706 can notify the prefetch service 708 of the rules installed in the NVD. In some embodiments, this results in different NVDs having different rules, and the rules can be related to the traffic passing through the NVD, thereby limiting the wasted space used to store unused rules.

[0137] FIG. 8 is a flowchart illustrating one embodiment of a process 800 for the operation of the NVD. This process 800 can be executed by some or all of the NVD. The process 800 begins at block 802, where at least one rule is received at the NVD. In some embodiments, this rule can be provided by the server fleet, and in particular, can be provided by the VCN distribution server 704 and / or the configuration storage service 706. In some embodiments, this rule can be provided by the configuration storage service 706 when it is determined that the NVD does not have rules and / or configurations for processing the packets received by the NVD.

[0138] In some embodiments, this rule can include a compiled rule. The compiled rule can be applicable to the class of packets received by the NVD. In some embodiments, the compiled rule can be applicable to the class of packets received by the NVD for direct or indirect delivery to the VNIC. In some embodiments, this compiled rule can be received, for example, from the configuration storage fleet 706. In some embodiments, each of at least one rule received by the NVD can include a flow key, a security policy, and mapping information. In some embodiments, this security policy can include a local security policy and / or a remote security policy. In some embodiments, each of at least one rule can include packet conversion information and / or throttle information.

[0139] At block 804, a packet is received by the NVD. In some embodiments, this packet can be received by the NVD from another NVD and / or from an instance such as a compute instance, a bare metal instance, etc. In some embodiments, the packet can be received from another NVD and / or from an instance via an intermediate component such as a TOR switch. This packet can be for delivery to the VNIC in some embodiments. In some embodiments, such as when the VNIC is directly coupled to the NVD, the packet can be for direct delivery to the VNIC. The packet can be for indirect delivery to the VNIC in some embodiments, such as when the NVD sends the packet to another component, such as another NVD, before the packet is delivered to the VNIC.

[0140] As shown in block 806, the NVD that receives a packet can determine the attributes of the packet. The attributes of the packet can include at least one of a source address, such as a source IP address and / or a source IP identifier, and a destination address, such as a destination IP address and / or a destination port identifier.

[0141] As shown in block 808, the NVD evaluates the rules available in light of the attributes of the packet. In some embodiments, this evaluation can include evaluating the compiled rules available in light of the attributes of the packet. This evaluation can include, for example, comparing the attributes of the packet determined in block 806 with the available rules, and in particular, comparing with the flow key of each of the available rules. In decision step 810, it is determined whether there are any applicable rules available. In some embodiments, the available rules can be determined to be applicable based on the result of the evaluation in block 808. In some embodiments, the available rules can be determined to be applicable if the attributes of the packet match the flow key of one of the available rules.

[0142] If it is determined that at least one of the available rules is applicable, process 800 proceeds to block 812, where the packet is processed according to the applicable rule. In some embodiments, the packet can be processed according to the applicable rule and / or at least one attribute of the packet. In some embodiments, this processing can include resolving routing, security, throttling, and transformation according to the applicable rule. After the packet is processed, process 800 returns to block 804 if the next packet is received at the NVD, or returns to block 802 if the next rule is received at the NVD.

[0143] Returning again to decision step 810, if it is determined that there are no applicable rules, process 800 proceeds to block 814 and transfers the packet for processing to server fleet 702. After the packet is transferred to server fleet 702, process 800 returns to block 804 if the next packet is received at the NVD, and returns to block 802 if the next rule is received at the NVD.

[0144] FIG. 9 is a flowchart illustrating one embodiment of a process 900 for the operation of all or a portion of server fleet 702. In some embodiments, this process can include packet processing by server fleet 702, generation of compiled rules by server fleet 702, and / or distribution of compiled rules by server fleet 702 to the NVD. Process 900 begins at block 902 where one or more rules for packet processing are received and / or generated. In some embodiments, this can include, for example, the VCN distribution service 704 receiving one or more low-level rules and / or the VCN distribution service 704 providing the low-level rules to the configuration storage fleet 706.

[0145] At block 904, the packet is received by server fleet 702 and, in some embodiments, by configuration storage fleet 706. The packet can be received by server fleet 702 from the NVD. In some embodiments, the packet can include a packet for which the NVD did not have an applicable rule and / or a packet for which the NVD determined that there was no applicable rule.

[0146] As shown in block 906, the attributes of a packet can be determined. In some embodiments, these attributes of the packet can be determined by the server fleet 702, and in particular, can be determined by the configuration storage fleet 706. The attributes of the packet can include, for example, at least one of a source address such as a source IP address and / or a source IP identifier, and a destination address such as a destination IP address and / or a destination port identifier.

[0147] In block 908, rules for processing the packet received in block 904 are determined. In some embodiments, this rule can be determined by the configuration storage fleet 706. In some embodiments, this rule can be determined by evaluating the rules available in light of the attributes of the packet. These rules can be either compiled rules or uncompiled rules. In some embodiments, determining the rules for processing a packet can include determining the rules based on at least one attribute of the packet, as well as information related to the rules and / or the configuration.

[0148] In some embodiments, evaluating the rules available in light of the attributes of the packet can include, for example, comparing the attributes of the packet determined in block 806 with the available rules, and in particular, comparing with the flow key of each of the available rules. A rule including a flow key that matches the attributes of the packet is identified and can be selected for use in processing the packet. In some embodiments, the rule for processing a second packet can be determined, for example, by a destination address that can be at least one of a destination IP address and / or a destination port identifier, and at least one of the attributes of the packet including at least one of a source address that can be at least one of a source IP address and / or a source port identifier.

[0149] In block 910, the packet is processed according to the applicable rules determined in block 908. In some embodiments, the packet may be processed according to the applicable rules and / or at least one attribute of the packet. In some embodiments, this processing can include resolving routing, security, throttling, and translation according to the applicable rules.

[0150] In block 912, a compiled rule including the rule determined in block 910 is generated and / or identified. In some embodiments, generating and / or identifying the compiled rule can include, for example, identifying each possible rule for each of a plurality of stages. In some embodiments, these stages can include, for example, a routing stage, a security stage, a throttle stage, and a translation stage. In some embodiments, the plurality of stages can include two or more of, for example, a routing stage, a security stage, a throttle stage, and a translation stage. In some embodiments, each of the possible rules is associated with a plurality of packet flows. In some embodiments, each of these possible rules can be a rule that includes the most generalized rule determined in block 908 for that stage.

[0151] The generation of the compiled rule, also referred to herein as rule compression, minimizes the table space of the rules installed on the NVD. This is achieved by maximizing the coverage of each compiled rule installed on the NVD by maximizing the volume targeted by the compiled rules within the n-dimensional space of possible packet flows. By maximizing the volume of the coverage of the compiled rules, the number of rules required by the VNIC is reduced.

[0152] In some embodiments, identifying and / or generating the compiled rules can include, for example, forming new compiled rules from common portions among the possible rules for each of the multiple stages of the flow. In some embodiments, identifying the common portions among the possible rules for each of the multiple stages of the flow can result in identifying the most generalized rules common to all of the multiple stages. The compiled rules can be stored in the configuration storage fleet 706.

[0153] For example, rule compression can be performed across multiple stages including a routing stage, a security stage, a throttle stage, and a transformation stage. The routing stage can resolve the destination of the next hop of the packet. This resolution can be performed, for example, based on customer route rules, L2 forwarding within a VLAN, L3 forwarding to a VNIC within a subnet and / or VCN, etc. In some embodiments, the output of the routing stage can be the identifier of the destination of the next hop of the packet. In some embodiments, the output key CIDR, or in other words, the possible rule of the routing stage, is the largest CIDR that matches the packet such that specific routes pointing to different targets no longer overlap with the possible rules.

[0154] In some embodiments, the possible rules in the routing stage can be generated by the "Longest Prefix Match" (LPM) algorithm. In some embodiments, this longest prefix match algorithm can be executed using a trie tree, which can be a binary tree where each node in the tree corresponds to a bit in the IP address of the packet. The root of this tree can correspond to the first bit in the destination address being considered. Such an algorithm traverses to a child pointer based on the value of the current bit, i.e., moves to the left child if the current bit is 0 and moves to the right child if the current bit is 1. A node with a matching route rule is marked with a pointer to the next hop. It is possible to detect the LPM by traversing the trie tree until a certain node is reached, and this node has no further children that match the packet (i.e., the current bit is 0 and there is no left child). The last matching route rule above the end point is the LPM.

[0155] In some embodiments, the LPM algorithm can be modified to generate and / or determine the output key of the routing stage. This algorithm can traverse the trie tree until the LPM is detected. After the LPM is detected, the algorithm continues to traverse the trie tree by traversing the nodes of the LPM until the first end node is detected. This first end node can be the end node that is closest to the node of the LPM in the trie tree, or in other words, a leaf node. Next, this first end node along the lookup address is identified as the output key. In some embodiments, this algorithm can ignore nodes that have the same next hop as the current node.

[0156] An example embodiment of the application of this modified LPM algorithm is shown in FIG. 10. In the case of IP address 10.1.1.1, in the shown trie, 10.0.0.0 / 8 is the LPM and the output key for 10.1.1.1 is 10.1.1.0 / 24.

[0157] The security stage determines whether packet delivery is permitted. This determination includes searching for the packet within one or more security lists, which can be subnet security lists and / or network security lists. In some embodiments, the packet is searched within one or more security lists to determine whether the packet is permitted. The output of the security stage can be a decision and / or indication of whether the packet should be permitted or dropped. In some embodiments, the output key of the security stage is the largest block that matches the packet such that all other flows that could be subject to the rule have the same security properties.

[0158] In some embodiments, the selection of possible rules at the security stage can be performed by a Binary Decision Diagram (BDD) algorithm. The BDD algorithm provides fast search and powerful rule compression and can quickly process security rules for each packet. In some embodiments, the BDD algorithm can complicate the process of determining the amount of possible transfer rules because information about the individual rules that caused a packet to be permitted or rejected may be lost. To optimize rule compression while minimizing the impact on BDD performance, some embodiments include an additional step of tracking rule segments in BDD calculations. In such embodiments, each segment can correspond to the volume of space allocated to permit or reject. In some embodiments, such segments result in an optimal volume for possible rules, and in some embodiments, such segments result in a near-optimal volume for possible rules. Such segments may not result in an optimal volume, but result in a volume sufficient to produce efficient rule compression.

[0159] In some embodiments, the BDD algorithm has two terminal points, which can be, for example, a terminal endpoint for "permit" and a terminal endpoint for "deny". The BDD table is compressed by including only these two nodes, but important information may be lost due to this compression.

[0160] In some embodiments, the BDD algorithm can be compressed while retaining desirable information. In such embodiments, the rules are preprocessed to combine adjacent rules. In some embodiments, the rules can be preprocessed to simply combine any adjacent rules. For example, a 0 / 0 rule permitting source ports 300 - 400 to port 80 and a 0 / 0 rule permitting source ports 400 - 500 to port 80 can be combined into a 0 / 0 rule permitting source ports 300 - 500 to port 80. Nodes at the edges of the graph can be evaluated to determine whether each node is a permit node or a deny node. In some embodiments, an identifier indicating whether a node is a permit node or a deny node can be associated with each of the nodes at the edges of the graph. In some embodiments, this can include coloring the edges of the graph such that each permit node is colored green and each deny node is colored red.

[0161] Each edge node of the BDD algorithm can be related to the set of rules associated with that edge. If an edge node is related to multiple rules, one of the multiple rules can be selected to be the "primary rule" of that edge node. In some embodiments, the largest of the multiple nodes can be selected to be the primary node. In some embodiments, the rule related to the largest number of flows is the largest rule and can be designated as the primary rule. In some embodiments, a new terminal node can be created for each of the selected primary rules, and then BDD compression can continue.

[0162] The application of this compression may reduce the compression ability of the BDD algorithm because more nodes are used to track the rule information, but this compression enables each endpoint to be mapped to a contiguous region (defined by the primary rule) that can be used to determine rule compression.

[0163] The throttle stage includes information related to subjecting one or more packets to throttling. In some embodiments, this information can include information for determining whether the data transmission rate of the packet is within an acceptable limit. In some embodiments, the security stage can be used to determine and / or identify one or more throttles for application to the packet and / or for application to transfer rules applicable to the packet. In some embodiments, the output key of the throttle stage is the largest block of a flow having the same set of throttles.

[0164] The transformation stage can include information for performing any desired and / or necessary rewrites before delivering the packet. The transformation stage can include, for example, NAT of overlay packets, MAC address resolution when delivering to a host, PAT, etc. In some embodiments, the transformation of the packet can be a routing next hop and / or a sender function. In some embodiments, the possible rules of the transformation stage can be created such that the same rewrite is required for all packets to which a possible rule is applicable. In some embodiments, for cases such as an egress flow, the output key of the transformation stage can be the same key as the output key of the rules of a possible route stage. In such embodiments, the rewrite can be based on the routing destination. In some embodiments, for cases such as an ingress flow, the desired transformation can depend on the sender, and the output key can be limited to a particular VNIC or gateway that is sending the packet.

[0165] In block 914, one or more NVDs are identified for receipt of the compiled rules. These one or more NVDs can include the NVD that is the source of the packets of the received block 904. In some embodiments, the one or more NVDs can further include one or more NVDs similar to the NVD that is the source of the packets of the received block 904.

[0166] At block 916, the compiled rules from block 912 are provided to one or more NVDs identified at block 914. In some embodiments, the compiled rules may be provided to one or more NVDs by server fleet 702 and in particular by configuration storage fleet 706. In some embodiments, as part of providing the compiled rules, configuration storage fleet 706 can provide information, in particular, can provide identification information of one or more NVDs that receive the compiled rules, and can provide the compiled rules and / or identifiers of the compiled rules to prefetch fleet 708. In some embodiments, prefetch fleet 708 can update a database that identifies the rules available to each VNIC based on this information.

[0167] Example embodiments of packet processing In a first embodiment, as shown at block 802, one or more compiled rules can be received by an NVD from server fleet 702. As shown at block 804, a first packet can be received by the NVD, and the first packet can be for delivery to a first VNIC. As shown at block 806, the attributes of the first packet can be determined by the NVD, and as shown at blocks 808 and decision step 810, the NVD can determine and / or identify applicable rules based at least in part on the attributes of the first packet. As shown at block 812, next, the first packet can be processed by the NVD according to the applicable rules.

[0168] As shown in block 804, a second packet can be received by the NVD, and the second packet can be, for example, for delivery to a second VNIC. The NVD can determine the attributes of the second packet, as shown in block 806, and can determine whether the NVD contains or does not contain rules for processing the second packet, as shown in blocks 808 and decision step 810. As shown in block 814, the NVD can then transfer the second packet to the server fleet 702.

[0169] As shown in block 904, a second packet can be received by the server fleet 702, and the server fleet 702 can determine the attributes of the packet, as shown in block 906, and can determine rules for processing the packet, as shown in block 908. As shown in block 910, the packet can be processed by the server fleet 702, and the server fleet 702 can generate a compiled rule that includes the rules used to process the second packet. The server fleet 702 can identify one or more NVDs for receiving the compiled rule and can provide the compiled rule to those one or more NVDs. In some embodiments, the NVD that is the source of the second packet sent to the server fleet 702 can receive the compiled rule. In some embodiments, one or more NVDs can be identified as being similar to the NVD that is the source of the second packet sent to the server fleet 702, and the compiled rule can be sent to each of these similar NVDs.

[0170] Figure 11 shows a second embodiment of packet processing. As seen in Figure 11, process 900 after the packet has been processed. The packet is sent from VNIC A 1102 having a first IP address 10.0.0.1 to VNIC B 1104 having a second IP address 10.1.1.1. Each of VNICs 1102, 1104 includes a VNIC configuration 1106, 1108, and each configuration corresponds to a rule. Each of these configurations includes rules for a plurality of stages, including a routing stage, a security stage, a throttling stage, and a conversion (rewrite) stage.

[0171] Table 1 below shows the processing of VNIC A 1102. Specifically, Table 1 shows the VNIC configuration 1106 of VNIC A 1102 and explains the application of that configuration to the packets sent from VNIC A 1102 to VNIC B 1104. The output key of the VNIC configuration 1106 of VNIC A 1102 is as follows. Output key: Source IP: 0.0.0.0 / 0 Destination IP: 10.1.1.1 / 32 Source port: all Destination ports: 80.

[0172]

Table 1

[0173] Table 2 below shows the processing of VNIC B 1104. Specifically, Table 2 shows the VNIC configuration 1108 of VNIC B 1104 and explains the application of that configuration to the packets received by VNIC B 1104 from VNIC A 1102. The output key of the VNIC configuration 1108 of VNIC B 1104 is as follows. Output key: Source IP: 10.0.0.1 / 32 Destination IP: 10.1.1.1 / 32 Source port: all Destination ports: 80.

[0174]

Table 2

[0175] Exemplary implementation As described above, IaaS (infrastructure as a service) is a specific type of cloud computing. IaaS can be configured to provide virtualized computing resources via a public network (e.g., the Internet). In the IaaS model, a cloud computing provider can host infrastructure components (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer), etc.). In some cases, the IaaS provider may provide various services (e.g., billing, monitoring, logging, load balancing, and clustering, etc.) that arise in connection with those infrastructure components. Therefore, since these services can be policy-driven, IaaS users may be able to implement policies to drive load balancing to maintain application availability and performance.

[0176] In some cases, IaaS customers may access resources and services via a wide area network (WAN) such as the Internet and use the cloud provider's services to install the remaining elements of the application stack. For example, a user can log in to the IaaS platform, create virtual machines (VMs), install an operating system (OS) on each VM, deploy middleware such as a database, create storage buckets for workloads and backups, and install enterprise software on the VMs. Next, the customer can use the provider's services to perform various functions, including balancing network traffic, troubleshooting application problems, monitoring performance, managing disaster recovery, etc.

[0177] In most cases, the cloud computing model requires the participation of a cloud provider. The cloud provider may be a third-party service that specializes in providing (e.g., offering, lending, selling) IaaS, but this is not necessary. An entity may choose to deploy a private cloud and become its own provider of infrastructure services.

[0178] In some examples, the deployment of IaaS is the process of connecting a new application or a new version of an application to a prepared application server or the like. This process may include the process of preparing the server (e.g., installing libraries, daemons, etc.). This process is often managed by a cloud provider under a hypervisor layer (e.g., servers, storage, network hardware, and virtualization). Thus, a customer may play a role in handling the deployment of an OS, middleware, and / or application on top of, for example, a self-service virtual machine (e.g., that can be spun up on demand).

[0179] In some examples, the provisioning of IaaS may also refer to obtaining a computer or virtual host for use and installing the required libraries or services on those computers or virtual hosts. In most cases, deployment does not include provisioning, and provisioning may need to be performed first.

[0180] In some cases, there are two different challenges in IaaS provisioning. First, there is the initial challenge of provisioning an initial set of infrastructure before anything is executed. Second, after everything is provisioned, there is the challenge of evolving the existing infrastructure (e.g., adding new services, changing services, removing services, etc.). In some cases, these two challenges may be addressed by enabling the infrastructure configuration to be defined declaratively. In other words, the infrastructure (e.g., which components are required and how those components communicate with each other) can be defined by one or more configuration files. In this way, the entire infrastructure topology (e.g., which resources depend on which other resources and how they cooperate) can be described declaratively. In some cases, after the topology is defined, a workflow for creating and / or managing the various components described in the configuration file may be generated.

[0181] In some examples, the infrastructure may include many interconnected elements. For example, there may be one or more virtual private clouds (VPCs), also known as core networks (e.g., a pool of configurable and / or shared computing resources, sometimes on-demand). In some examples, there may be one or more inbound traffic / outbound traffic group rules provisioned to define how inbound and / or outbound network traffic is set up, and one or more virtual machines (VMs). Other infrastructure elements such as load balancers, databases, etc. may be provisioned. As more infrastructure elements are desired and / or added, the infrastructure can evolve gradually.

[0182] In some cases, continuous deployment techniques may be employed to enable the deployment of infrastructure code across various virtual computing environments. Further, the techniques described can enable infrastructure management within these environments. In some examples, a service team may write code that is desirably deployed to one or more, but in many cases a large number of, different production environments (e.g., across various geographical locations, sometimes worldwide). However, in some examples, the infrastructure to which the code is deployed must first be provisioned. In some cases, provisioning can be done manually, provisioning tools may be utilized to provision resources, and / or deployment tools may be utilized to deploy the code after the infrastructure has been provisioned.

[0183] FIG. 12 is a block diagram 1200 showing an exemplary pattern of an IaaS architecture according to at least one embodiment. A service operator 1202 can be communicatively coupled to a secure host tenancy 1204 that can include a virtual cloud network (VCN) 1206 and a secure host subnet 1208. In some examples, the service operator 1202 may use one or more client computing devices, which can be portable handheld devices (e.g., iPhone (registered trademark), mobile phone, iPad (registered trademark), computing tablet, personal digital assistant (PDA)) or wearable devices (e.g., Google Glass (registered trademark) head-mounted display) that are enabled with Internet, email, short message service (SMS), Blackberry (registered trademark), or other communication protocols and that run software such as Microsoft Windows Mobile (registered trademark) and / or various mobile operating systems such as iOS, Windows Phone, Android, BlackBerry 8, Palm OS. Alternatively, the client computing device can be a general-purpose personal computer, including, by way of example, personal computers and / or laptop computers that run various versions of Microsoft Windows (registered trademark), Apple Macintosh (registered trademark), and / or Linux (registered trademark) operating systems. The client computing device can be a workstation computer that runs any of various commercially available UNIX (registered trademark) or UNIX-like operating systems, including, but not limited to, various GNU / Linux operating systems such as Google Chrome OS.Alternatively or in addition, the client computing device can be any other electronic device, such as a thin client computer, an Internet-enabled gaming system (e.g., a Microsoft Xbox gaming console with or without a Kinect® gesture input device), and / or a personal messaging device, that can communicate via a network and / or the Internet that has access to the VCN 1206.

[0184] The VCN 1206 can include a Local Peering Gateway (LPG) 1210 that can be communicatively coupled to a Secure Shell (SSH) VCN 1212 via the LPG 1210 included in the SSH VCN 1212. The SSH VCN 1212 can include an SSH subnet 1214 and can be communicatively coupled to a control plane VCN 1216 via the LPG 1210 included in the control plane VCN 1216. Also, the SSH VCN 1212 can be communicatively coupled to a data plane VCN 1218 via the LPG 1210. The control plane VCN 1216 and the data plane VCN 1218 can be included in a service tenancy 1219 that can be owned and / or operated by an IaaS provider.

[0185] The control plane VCN 1216 can include a control plane demilitarized zone (DMZ) layer 1220 that functions as a border network (e.g., a part of the enterprise network between the enterprise intranet and the external network). Servers based on the DMZ can have limited responsibilities and can help contain intrusions. Further, the DMZ layer 1220 can include one or more load balancer (LB) subnets 1222, a control plane application layer 1224 that can include an application subnet 1226, and a control plane data layer 1228 that can include a database (DB) subnet 1230 (e.g., a front-end DB subnet and / or a back-end DB subnet). The LB subnet 1222 included in the control plane DMZ layer 1220 can be communicatively coupled to the application subnet 1226 included in the control plane application layer 1224 that can be included in the control plane VCN 1216 and the Internet gateway 1234, and the application subnet 1226 can be communicatively coupled to the DB subnet 1230 included in the control plane data layer 1228 as well as the service gateway 1236 and the network address translation (NAT) gateway 1238. The control plane VCN 1216 can include the service gateway 1236 and the NAT gateway 1238.

[0186] The control plane VCN 1216 can include a data plane mirror application layer 1240 that can include the application subnet 1226. The application subnet 1226 included in the data plane mirror application layer 1240 can include a virtual network interface controller (VNIC) 1242 that can execute a compute instance 1244. The compute instance 1244 can communicatively couple the application subnet 1226 of the data plane mirror application layer 1240 to the application subnet 1226 that can be included in the data plane application layer 1246.

[0187] The data plane VCN 1218 can include a data plane application layer 1246, a data plane DMZ layer 1248, and a data plane data layer 1250. The data plane DMZ layer 1248 can include an LB subnet 1222 communicatively coupled to the application subnet 1226 of the data plane application layer 1246 and the Internet gateway 1234 of the data plane VCN 1218. The application subnet 1226 can be communicatively coupled to the service gateway 1236 of the data plane VCN 1218 and the NAT gateway 1238 of the data plane VCN 1218. The data plane data layer 1250 can also include a DB subnet 1230 communicatively coupled to the application subnet 1226 of the data plane application layer 1246.

[0188] The Internet gateway 1234 of the control plane VCN 1216 and the data plane VCN 1218 can be communicatively coupled to a metadata management service 1252 communicatively coupled to the public Internet 1254. The public Internet 1254 can be communicatively coupled to the NAT gateway 1238 of the control plane VCN 1216 and the data plane VCN 1218. The service gateway 1236 of the control plane VCN 1216 and the data plane VCN 1218 can be communicatively coupled to a cloud service 1256.

[0189] In some examples, the service gateway 1236 of the control plane VCN1216 or the data plane VCN1218 can make application programming interface (API) calls to the cloud service 1256 without going through the public Internet 1254. The API calls from the service gateway 1236 to the cloud service 1256 can be one-way, and the service gateway 1236 can make API calls to the cloud service 1256, and the cloud service 1256 can send the requested data to the service gateway 1236. However, the cloud service 1256 does not have to initiate API calls to the service gateway 1236.

[0190] In some examples, the secure host tenancy 1204 can be directly connected to the service tenancy 1219, or otherwise, it can be separated. The secure host subnet 1208 can communicate with the SSH subnet 1214 via the LPG 1210, and the LPG 1210 can enable two-way communication on a separated system if not. Connecting the secure host subnet 1208 to the SSH subnet 1214 may give the secure host subnet 1208 access to other entities within the service tenancy 1219.

[0191] The control plane VCN 1216 may enable users of service tenancy 1219 to set or otherwise provision desired resources. Desired resources provisioned within the control plane VCN 1216 may be deployed or otherwise used in the data plane VCN 1218. In some examples, the control plane VCN 1216 may be separable from the data plane VCN 1218, and the data plane mirror app layer 1240 of the control plane VCN 1216 may communicate with the data plane app layer 1246 of the data plane VCN 1218 via VNICs 1242 that may be included in the data plane mirror app layer 1240 and the data plane app layer 1246.

[0192] In some examples, a user or customer of the system may perform requests, such as create, read, update, or delete (CRUD) operations, via the public Internet 1254 that can communicate requests to the metadata management service 1252. The metadata management service 1252 can communicate requests to the control plane VCN 1216 via the Internet gateway 1234. The requests may be received by the LB subnet 1222 included in the control plane DMZ layer 1220. The LB subnet 1222 may determine that the requests are valid, and in response, the LB subnet 1222 may send the requests to the app subnet 1226 included in the control plane app layer 1224. If the requests are authenticated and the requests require calls to the public Internet 1254, the calls to the public Internet 1254 may be sent to the NAT gateway 1238 that can make calls to the public Internet 1254. Memory that may desirably be stored by the requests may be stored within the DB subnet 1230.

[0193] In some examples, the data plane mirror application layer 1240 can facilitate direct communication between the control plane VCN 1216 and the data plane VCN 1218. For example, it may be desirable for changes, updates, or other appropriate modifications to the configuration to be applied to the resources included in the data plane VCN 1218. Through the VNIC 1242, the control plane VCN 1216 can communicate directly with the resources included in the data plane VCN 1218, thereby enabling changes, updates, or other appropriate modifications to the configuration of the resources.

[0194] In some embodiments, the control plane VCN 1216 and the data plane VCN 1218 may be included in the service tenancy 1219. In this case, the user or customer of the system does not have to own or operate either the control plane VCN 1216 or the data plane VCN 1218. Instead, the IaaS provider may own or operate both the control plane VCN 1216 and the data plane VCN 1218, which may both be included in the service tenancy 1219. This embodiment can enable network isolation that can prevent a user or customer from exchanging information with the resources of other users or other customers. Also, this embodiment can allow the user or customer of the system to privately store the database without having to rely on the public Internet 1254, which may not have the desired level of threat prevention for storage.

[0195] In other embodiments, the LB subnet 1222 included in the control plane VCN 1216 may be configured to receive signals from the service gateway 1236. In this embodiment, the control plane VCN 1216 and the data plane VCN 1218 may be configured to be invoked by a customer of the IaaS provider without invoking the public Internet 1254. A customer of the IaaS provider may desire this embodiment because the databases used by the customer may be controlled by the IaaS provider and may be stored in a service tenancy 1219 that can be isolated from the public Internet 1254.

[0196] FIG. 13 is a block diagram 1300 showing another exemplary pattern of an IaaS architecture according to at least one embodiment. A service operator 1302 (e.g., service operator 1202 of FIG. 12) can be communicatively coupled to a secure host tenancy 1304 (e.g., secure host tenancy 1204 of FIG. 12) that can include a virtual cloud network (VCN) 1306 (e.g., VCN 1206 of FIG. 12) and a secure host subnet 1308 (e.g., secure host subnet 1208 of FIG. 12). The VCN 1306 can include an LPG 1310 (e.g., LPG 1210 of FIG. 12) that can be communicatively coupled to an SSH VCN 1312 (e.g., SSH VCN 1212 of FIG. 12) via an LPG 1210 included in the secure shell (SSH) VCN 1312. The SSH VCN 1312 can include an SSH subnet 1314 (e.g., SSH subnet 1214 of FIG. 12), and the SSH VCN 1312 can be communicatively coupled to a control plane VCN 1316 (e.g., control plane VCN 1216 of FIG. 12) via an LPG 1310 included in the control plane VCN 1316. The control plane VCN 1316 can be included in a service tenancy 1319 (e.g., service tenancy 1219 of FIG. 12), and the data plane VCN 1318 (e.g., data plane VCN 1218 of FIG. 12) can be included in a customer tenancy 1321 that can be owned or operated by a user or customer of the system.

[0197] The control plane VCN1316 can include a control plane DMZ layer 1320 (e.g., the control plane DMZ layer 1220 of FIG. 12) that can include an LB subnet 1322 (e.g., the LB subnet 1222 of FIG. 12), a control plane application layer 1324 (e.g., the control plane application layer 1224 of FIG. 12) that can include an application subnet 1326 (e.g., the application subnet 1226 of FIG. 12), and a control plane data layer 1328 (e.g., the control plane data layer 1228 of FIG. 12) that can include a database (DB) subnet 1330 (e.g., similar to the database (DB) subnet 1230 of FIG. 12). The LB subnet 1322 included in the control plane DMZ layer 1320 can be communicatively coupled to the application subnet 1326 and the Internet gateway 1334 (e.g., the Internet gateway 1234 of FIG. 12) included in the control plane application layer 1324 that can be included in the control plane VCN1316, and the application subnet 1326 can be communicatively coupled to the DB subnet 1330 included in the control plane data layer 1328 as well as the service gateway 1336 (e.g., the service gateway 1236 of FIG. 12) and the network address translation (NAT) gateway 1338 (e.g., the NAT gateway 1238 of FIG. 12). The control plane VCN1316 can include the service gateway 1336 and the NAT gateway 1338.

[0198] The control plane VCN 1316 can include a data plane mirror application layer 1340 (e.g., the data plane mirror application layer 1240 of FIG. 12) that can include an application subnet 1326. The application subnet 1326 included in the data plane mirror application layer 1340 can include a virtual network interface controller (VNIC) 1342 (e.g., the VNIC 1242) that can execute a compute instance 1344 (e.g., similar to the compute instance 1244 of FIG. 12). The compute instance 1344 can facilitate communication between the application subnet 1326 of the data plane mirror application layer 1340 and an application subnet 1326 that can be included in the data plane application layer 1346 (e.g., the data plane application layer 1246 of FIG. 12) via the VNIC 1342 included in the data plane mirror application layer 1340 and the VNIC 1342 included in the data plane application layer 1346.

[0199] The internet gateway 1334 included in the control plane VCN 1316 can be communicatively coupled to a metadata management service 1352 (e.g., the metadata management service 1252 of FIG. 12) that can be communicatively coupled to the public internet 1354 (e.g., the public internet 1254 of FIG. 12). The public internet 1354 can be communicatively coupled to the NAT gateway 1338 included in the control plane VCN 1316. The service gateway 1336 included in the control plane VCN 1316 can be communicatively coupled to a cloud service 1356 (e.g., the cloud service 1256 of FIG. 12).

[0200] In some examples, the data plane VCN 1318 can be included in the customer's tenancy 1321. In this case, the IaaS provider may provide a control plane VCN 1316 for each customer, and the IaaS provider may configure the specific compute instances 1344 included in the service tenancy 1319 for each customer. Each compute instance 1344 may enable communication between the control plane VCN 1316 included in the service tenancy 1319 and the data plane VCN 1318 included in the customer's tenancy 1321. The compute instance 1344 may enable the resources provisioned within the control plane VCN 1316 included in the service tenancy 1319 to be deployed or otherwise used in the data plane VCN 1318 included in the customer's tenancy 1321.

[0201] In other examples, a customer of an IaaS provider may have a database that persists in the customer's tenancy 1321. In this example, the control plane VCN 1316 can include a data plane mirror app layer 1340 that can include an app subnet 1326. The data plane mirror app layer 1340 can exist in the data plane VCN 1318, but the data plane mirror app layer 1340 need not persist in the data plane VCN 1318. That is, the data plane mirror app layer 1340 may have access rights to the customer's tenancy 1321, but the data plane mirror app layer 1340 need not exist in the data plane VCN 1318 and need not be owned or operated by the customer of the IaaS provider. The data plane mirror app layer 1340 may be configured to make calls to the data plane VCN 1318, but need not be configured to make calls to any entity included in the control plane VCN 1316. The customer may wish to deploy or otherwise use resources within the data plane VCN 1318 that are provisioned within the control plane VCN 1316, and the data plane mirror app layer 1340 can facilitate the desired deployment or other use of the customer's resources.

[0202] In some embodiments, a customer of an IaaS provider can apply a filter to the data plane VCN 1318. In this embodiment, the customer can determine which data plane VCNs 1318 are accessible, and the customer may restrict access from the data plane VCN 1318 to the public Internet 1354. The IaaS provider need not be able to apply a filter or otherwise control access of the data plane VCN 1318 to any external network or database. Applying filters and controls by the customer to the data plane VCN 1318 included in the customer's tenancy 1321 can help to isolate the data plane VCN 1318 from other customers and from the public Internet 1354.

[0203] In some embodiments, cloud service 1356 may be invoked by service gateway 1336 to access services that may not exist on public Internet 1354, control plane VCN 1316, or data plane VCN 1318. The connection between cloud service 1356 and control plane VCN 1316 or data plane VCN 1318 need not be active or continuous. Cloud service 1356 may exist on a different network owned or operated by an IaaS provider. Cloud service 1356 may be configured to receive calls from service gateway 1336 and may be configured not to receive calls from public Internet 1354. Some cloud services 1356 may be isolated from other cloud services 1356, and control plane VCN 1316 may be isolated from cloud services 1356 that may not exist in the same region as control plane VCN 1316. For example, control plane VCN 1316 may be located in "Region 1", and "Deployment 12" of the cloud service may be located in Region 1 and "Region 2". When a call to Deployment 12 is made by service gateway 1336 included in control plane VCN 1316 located in Region 1, this call may be sent to Deployment 12 within Region 1. In this example, control plane VCN 1316, or Deployment 12 within Region 1, need not be communicatively coupled to Deployment 12 within Region 2 or, otherwise, communicate with Deployment 12 within Region 2.

[0204] FIG. 14 is a block diagram 1400 showing another exemplary pattern of an IaaS architecture according to at least one embodiment. A service operator 1402 (e.g., service operator 1202 of FIG. 12) can be communicatively coupled to a secure host tenancy 1404 (e.g., secure host tenancy 1204 of FIG. 12) that can include a virtual cloud network (VCN) 1406 (e.g., VCN 1206 of FIG. 12) and a secure host subnet 1408 (e.g., secure host subnet 1208 of FIG. 12). The VCN 1406 can be communicatively coupled to an SSH VCN 1412 (e.g., SSH VCN 1212 of FIG. 12) via an LPG 1410 (e.g., LPG 1210 of FIG. 12) included in the SSH VCN 1412. The SSH VCN 1412 can include an SSH subnet 1414 (e.g., SSH subnet 1214 of FIG. 12), and the SSH VCN 1412 can be communicatively coupled to a control plane VCN 1416 (e.g., control plane VCN 1216 of FIG. 12) via an LPG 1410 included in the control plane VCN 1416 and to a data plane VCN 1418 (e.g., data plane 1218 of FIG. 12) via an LPG 1410 included in the data plane VCN 1418. The control plane VCN 1416 and the data plane VCN 1418 can be included in a service tenancy 1419 (e.g., service tenancy 1219 of FIG. 12).

[0205] The control plane VCN 1416 can include a control plane DMZ layer 1420 (e.g., the control plane DMZ layer 1220 of FIG. 12) that can include a load balancer (LB) subnet 1422 (e.g., the LB subnet 1222 of FIG. 12), a control plane application layer 1424 (e.g., the control plane application layer 1224 of FIG. 12) that can include an application subnet 1426 (similar to the application subnet 1226 of FIG. 12), and a control plane data layer 1428 (e.g., the control plane data layer 1228 of FIG. 12) that can include a DB subnet 1430. The LB subnet 1422 included in the control plane DMZ layer 1420 can be communicatively coupled to the application subnet 1426 included in the control plane application layer 1424 that can be included in the control plane VCN 1416, and to an Internet gateway 1434 (e.g., the Internet gateway 1234 of FIG. 12). The application subnet 1426 can be communicatively coupled to the DB subnet 1430 included in the control plane data layer 1428, as well as to a service gateway 1436 (e.g., the service gateway of FIG. 12) and a network address translation (NAT) gateway 1438 (e.g., the NAT gateway 1238 of FIG. 12). The control plane VCN 1416 can include the service gateway 1436 and the NAT gateway 1438.

[0206] The data plane VCN 1418 can include a data plane application layer 1446 (e.g., the data plane application layer 1246 of FIG. 12), a data plane DMZ layer 1448 (e.g., the data plane DMZ layer 1248 of FIG. 12), and a data plane data layer 1450 (e.g., the data plane data layer 1250 of FIG. 12). The data plane DMZ layer 1448 can include a reliable application subnet 1460 and an unreliable application subnet 1462 of the data plane application layer 1446 and an LB subnet 1422 communicatively coupled to an Internet gateway 1434 included in the data plane VCN 1418. The reliable application subnet 1460 can be communicatively coupled to a service gateway 1436 included in the data plane VCN 1418, a NAT gateway 1438 included in the data plane VCN 1418, and a DB subnet 1430 included in the data plane data layer 1450. The unreliable application subnet 1462 can be communicatively coupled to a service gateway 1436 included in the data plane VCN 1418 and a DB subnet 1430 included in the data plane data layer 1450. The data plane data layer 1450 can include a DB subnet 1430 communicatively coupled to a service gateway 1436 included in the data plane VCN 1418.

[0207] The untrusted application subnet 1462 can include one or more primary VNICs 1464(1)-(N) communicatively coupled to tenant virtual machines (VMs) 1466(1)-(N). Each tenant VM 1466(1)-(N) can be communicatively coupled to respective application subnets 1467(1)-(N) that can be included in respective container egress VCNs 1468(1)-(N) that can be included in respective customer tenancies 1470(1)-(N). Each secondary VNIC 1472(1)-(N) can facilitate communication between the untrusted application subnet 1462 included in the data plane VCN 1418 and the application subnets included in the container egress VCNs 1468(1)-(N). Each container egress VCN 1468(1)-(N) can include a NAT gateway 1438 communicatively coupled to the public internet 1454 (e.g., the public internet 1254 of FIG. 12).

[0208] The internet gateway 1434 included in the control plane VCN 1416 and the data plane VCN 1418 can be communicatively coupled to a metadata management service 1452 (e.g., the metadata management system 1252 of FIG. 12) communicatively coupled to the public internet 1454. The public internet 1454 can be communicatively coupled to the NAT gateway 1438 included in the control plane VCN 1416 and the data plane VCN 1418. The service gateway 1436 included in the control plane VCN 1416 and the data plane VCN 1418 can be communicatively coupled to cloud services 1456.

[0209] In some embodiments, the data plane VCN 1418 can be integrated with the customer's tenancy 1470. This integration can be useful or desirable for the IaaS provider's customers in some cases, such as when they may want support when running code. The customer may provide code to execute that can be disruptive, communicate with other customers' resources, or otherwise cause undesirable effects. In response, the IaaS provider may determine whether to execute code provided to the IaaS provider by the customer.

[0210] In some examples, a customer of an IaaS provider may grant the IaaS provider temporary network access rights and request a function that connects to the data plane application layer 1446. The code for performing this function may be executed in the VMs 1466(1) to (N), and this code need not be configured to execute elsewhere on the data plane VCN 1418. Each of the VMs 1466(1) to (N) may be connected to the tenancy 1470 of one customer. Each of the containers 1471(1) to (N) included in the VMs 1466(1) to (N) may be configured to execute the code. In this case, there can be a double separation (e.g., the containers 1471(1) to (N) that execute the code, the containers 1471(1) to (N) may be included in at least the VMs 1466(1) to (N) included in the untrusted application subnet 1462), which can help prevent incorrect or otherwise undesirable code from damaging the IaaS provider's network or damaging the networks of different customers. The containers 1471(1) to (N) may be communicatively coupled to the customer's tenancy 1470 and may be configured to send or receive data with the customer's tenancy 1470. The containers 1471(1) to (N) need not be configured to send or receive data with any other entity within the data plane VCN 1418. Upon completion of the execution of the code, the IaaS provider may force the containers 1471(1) to (N) to terminate or otherwise discard them.

[0211] In some embodiments, the trusted application subnet 1460 may execute code owned or operated by an IaaS provider. In this embodiment, the trusted application subnet 1460 may be communicatively coupled to the DB subnet 1430 and may be configured to perform CRUD operations within the DB subnet 1430. The untrusted application subnet 1462 may be communicatively coupled to the DB subnet 1430, but in this embodiment, the untrusted application subnet may be configured to perform read operations within the DB subnet 1430. The containers 1471(1)-(N) that can execute customer code, which may be included in each customer's VMs 1466(1)-(N), may not be communicatively coupled to the DB subnet 1430.

[0212] [[ID=Z3]] In other embodiments, the control plane VCN 1416 and the data plane VCN 1418 may not be directly communicatively coupled. In this embodiment, there may be no direct communication between the control plane VCN 1416 and the data plane VCN 1418. However, communication can occur indirectly by at least one method. The LPG 1410 may be established by an IaaS provider and can facilitate communication between the control plane VCN 1416 and the data plane VCN 1418. In another example, the control plane VCN 1416 or the data plane VCN 1418 can make calls to cloud services 145^ via the service gateway 1436. For example, a call from the control plane VCN 1416 to the cloud services 1456 can include a request for a service that can communicate with the data plane VCN 1418.

[0213] FIG. 15 is a block diagram 1500 showing another exemplary pattern of an IaaS architecture according to at least one embodiment. A service operator 1502 (e.g., service operator 1202 of FIG. 12) can be communicatively coupled to a secure host tenancy 1504 (e.g., secure host tenancy 1204 of FIG. 12) that can include a virtual cloud network (VCN) 1506 (e.g., VCN 1206 of FIG. 12) and a secure host subnet 1508 (e.g., secure host subnet 1208 of FIG. 12). The VCN 1506 can be communicatively coupled to an SSH VCN 1512 (e.g., SSH VCN 1212 of FIG. 12) via an LPG 1510 (e.g., LPG 1210 of FIG. 12) included in the SSH VCN 1512. The SSH VCN 1512 can include an SSH subnet 1514 (e.g., SSH subnet 1214 of FIG. 12), and the SSH VCN 1512 can be communicatively coupled to a control plane VCN 1516 (e.g., control plane VCN 1216 of FIG. 12) via an LPG 1510 included in the control plane VCN 1516 and to a data plane VCN 1518 (e.g., data plane 1218 of FIG. 12) via an LPG 1510 included in the data plane VCN 1518. The control plane VCN 1516 and the data plane VCN 1518 can be included in a service tenancy 1519 (e.g., service tenancy 1219 of FIG. 12).

[0214] The control plane VCN 1516 can include a control plane DMZ layer 1520 (e.g., the control plane DMZ layer 1220 in FIG. 12) that can include an LB subnet 1522 (e.g., the LB subnet 1222 in FIG. 12), a control plane application layer 1524 (e.g., the control plane application layer 1224 in FIG. 12) that can include an application subnet 1526 (e.g., the application subnet 1226 in FIG. 12), and a control plane data layer 1528 (e.g., the control plane data layer 1228 in FIG. 12) that can include a DB subnet 1530 (e.g., the DB subnet 1430 in FIG. 14). The LB subnet 1522 included in the control plane DMZ layer 1520 can be communicatively coupled to the application subnet 1526 included in the control plane application layer 1524 that can be included in the control plane VCN 1516, and to an Internet gateway 1534 (e.g., the Internet gateway 1234 in FIG. 12). The application subnet 1526 can be communicatively coupled to the DB subnet 1530 included in the control plane data layer 1528, and to a service gateway 1536 (e.g., the service gateway in FIG. 12) and a network address translation (NAT) gateway 1538 (e.g., the NAT gateway 1238 in FIG. 12). The control plane VCN 1516 can include the service gateway 1536 and the NAT gateway 1538.

[0215] The data plane VCN 1518 can include a data plane application layer 1546 (e.g., the data plane application layer 1246 of FIG. 12), a data plane DMZ layer 1548 (e.g., the data plane DMZ layer 1248 of FIG. 12), and a data plane data layer 1550 (e.g., the data plane data layer 1250 of FIG. 12). The data plane DMZ layer 1548 can include a trusted application subnet 1560 (e.g., the trusted application subnet 1460 of FIG. 14) and an untrusted application subnet 1562 (e.g., the untrusted application subnet 1462 of FIG. 14) of the data plane application layer 1546, and an LB subnet 1522 communicatively coupled to an Internet gateway 1534 included in the data plane VCN 1518. The trusted application subnet 1560 can be communicatively coupled to a service gateway 1536 included in the data plane VCN 1518, a NAT gateway 1538 included in the data plane VCN 1518, and a DB subnet 1530 included in the data plane data layer 1550. The untrusted application subnet 1562 can be communicatively coupled to a service gateway 1536 included in the data plane VCN 1518, and a DB subnet 1530 included in the data plane data layer 1550. The data plane data layer 1550 can include a DB subnet 1530 communicatively coupled to a service gateway 1536 included in the data plane VCN 1518.

[0216] The untrusted application subnet 1562 can include primary VNICs 1564(1) to (N) communicatively coupled to tenant virtual machines (VMs) 1566(1) to (N) existing within the untrusted application subnet 1562. Each tenant VM 1�66(1) to (N) can execute code within respective containers 1567(1) to (N) and can be communicatively coupled to an application subnet 1526 that can be included in a data plane application layer 1546 that can be included in a container egress VCN 1568. Each secondary VNIC 1572(1) to (N) can facilitate communication between the untrusted application subnet 1562 included in the data plane VCN 1518 and the application subnet included in the container egress VCN 1568. The container egress VCN can include a NAT gateway 1538 communicatively coupled to a public internet 1554 (e.g., the public internet 1254 of FIG. 12).

[0217] The internet gateway 1534 included in the control plane VCN 1516 and in the data plane VCN 1518 can be communicatively coupled to a metadata management service 1552 (e.g., the metadata management system 1252 of FIG. 12) communicatively coupled to a public internet 1554. The public internet 1554 can be communicatively coupled to a NAT gateway 1538 included in the control plane VCN 1516 and in the data plane VCN 1518. The service gateway 1536 included in the control plane VCN 1516 and in the data plane VCN 1518 can be communicatively coupled to a cloud service 1556.

[0218] In some examples, the pattern shown by the architecture of block diagram 1500 in FIG. 15 may be considered an exception to the pattern shown by the architecture of block diagram 1400 in FIG. 14, which may be desirable for an IaaS provider's customers when the IaaS provider cannot communicate directly with a customer (e.g., a disconnected region). Each container 1567(1)-(N) included in VM1566(1)-(N) for each customer may be accessed in real time by the customer. Containers 1567(1)-(N) may be configured to make calls to respective secondary VNICs 1572(1)-(N) included in the app subnet 1526 of the data plane app layer 1546 that may be included in the container egress VCN 1568. Secondary VNICs 1572(1)-(N) may be able to send the calls to the NAT gateway 1538, and the NAT gateway 1538 may send the calls to the public internet 1554. In this example, containers 1567(1)-(N) that may be accessed in real time by the customer can be separated from the control plane VCN 1516 and may be separated from other entities included in the data plane VCN 1518. Containers 1567(1)-(N) may also be separated from the resources of other customers.

[0219] In other examples, a customer can call cloud service 1556 using containers 1567(1) to (N). In this example, the customer may execute the code within containers 1567(1) to (N) that requests a service from cloud service 1556. Containers 1567(1) to (N) can send this request to secondary VNICs 1572(1) to (N), which can send this request to a NAT gateway, which can send this request to public internet 1554. Public internet 1554 can send this request to the LB subnet 1522 included in control plane VCN 1516 via internet gateway 1534. In response to determining that this request is valid, the LB subnet can send this request to app subnet 1526, which can send this request to cloud service 1556 via service gateway 1536.

[0220] It should be understood that the IaaS architectures 1200, 1300, 1400, 1500 shown in the figures may include components other than those shown. Further, the embodiments shown in the figures are merely examples of a cloud infrastructure system that can incorporate embodiments of the present disclosure. In some other embodiments, the IaaS system may include more or fewer components than those shown in the figures, combine two or more components, or have a different configuration or arrangement of components.

[0221] In one embodiment, the IaaS system described herein may include the provision of a series of application, middleware, and database services that are delivered to customers in a self-service, subscription-based, elastically scalable, reliable, highly available, and secure manner. An example of such an IaaS system is the Oracle Cloud Infrastructure (OCI) provided by the present assignee.

[0222] FIG. 16 shows an exemplary computer system 1600 in which various embodiments may be implemented. System 1600 may be used to implement any of the computer systems described above. As shown in the figure, computer system 1600 includes a processing unit 1604 that communicates with a plurality of peripheral subsystems via a bus subsystem 1602. These peripheral subsystems may include a processing acceleration unit 1606, an I / O subsystem 1608, a storage subsystem 1618, and a communication subsystem 1624. Storage subsystem 1618 includes tangible computer-readable storage media 1622 and system memory 1610.

[0223] The bus subsystem 1602 provides a mechanism for the various components and subsystems of the computer system 1600 to communicate with each other as intended. Although the bus subsystem 1602 is schematically shown as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses. The bus subsystem 1602 may be any of a plurality of types of bus structures, including a memory bus or memory controller, a peripheral bus, and a local bus that uses any of a variety of bus architectures. For example, such architectures may include an ISA (Industry Standard Architecture) bus, an MCA (Micro Channel Architecture) bus, an EISA (Enhanced ISA) bus, a VESA (Video Electronics Standards Association) local bus, and a PCI (Peripheral Component Interconnect) bus implemented as a mezzanine bus manufactured to the IEEE P1386.1 standard.

[0224] The processing unit 1604, which may be implemented as one or more integrated circuits (e.g., conventional microprocessors or microcontrollers), controls the operation of the computer system 1600. One or more processors may be included in the processing unit 1604. These processors may include single-core processors or multi-core processors. In certain embodiments, the processing unit 1604 may be implemented as one or more independent processing units 1632 and / or 1634, with a single-core processor or multi-core processor included in each processing unit. In other embodiments, the processing unit 1604 may be implemented as a quad-core processing unit formed by integrating two dual-core processors on a single chip.

[0225] In various embodiments, the processing unit 1604 can execute various programs according to program code and can maintain a plurality of programs or processes to be executed simultaneously. At any given time, some or all of the program code to be executed can be present in the processor 1604 and / or the storage subsystem 1618. With appropriate programming, the processor 1604 can provide the various functions described above. The computer system 1600 may further include a processing acceleration unit 1606 that can include a digital signal processor (DSP), an application specific processor, and / or the like.

[0226] The I / O subsystem 1608 may include user interface input devices and user interface output devices. The user interface input devices may include a keyboard, a pointing device such as a mouse or trackball, a touchpad or touch screen incorporated in a display, a scroll wheel, a click wheel, a dial, a button, a switch, a keypad, a voice input device with a voice command recognition system, a microphone, and other types of input devices. The user interface input devices may enable a user to control input devices such as a Microsoft Xbox (registered trademark) 360 game controller and exchange information via a natural user interface using gestures and spoken commands, and may include motion detection devices and / or gesture recognition devices such as a Microsoft Kinect (registered trademark) motion sensor. The user interface input devices may include gesture recognition devices such as a Google Glass (registered trademark) blink detector that detects a user's eye activity (e.g., a "blink" when taking a photo and / or selecting a menu) and converts the eye gesture into an input to the input device (e.g., Google Glass (registered trademark)). Further, the user interface input devices may include a voice recognition detection device that enables a user to interact with a voice recognition system (e.g., a Siri (registered trademark) navigator) via voice commands.

[0227] The user interface input device may include, but is not limited to, a three-dimensional (3D) mouse, joystick or pointing stick, game pad, and graphic tablet, as well as audio / visual devices such as speakers, digital cameras, digital video cameras, portable media players, webcams, image scanners, fingerprint scanners, barcode readers, 3D scanners, 3D printers, laser distance meters, and eye-tracking devices. Further, the user interface input device may include, for example, medical image input devices such as computed tomography, magnetic resonance imaging, positron emission tomography, and medical ultrasonic examination devices. The user interface input device may include, for example, audio input devices such as MIDI keyboards and digital musical instruments.

[0228] The user interface output device may include, among others, visual displays other than display subsystems, indicator lights, or audio output devices. The display subsystem may be a flat panel device such as a flat panel device using a cathode ray tube (CRT), liquid crystal display (LCD), or plasma display, a projection device, a touch screen, and the like. Generally, the use of the term "output device" is intended to include all possible types of devices and mechanisms for outputting information from the computer system 1600 to the user or other computers. For example, the user interface output device may include, but is not limited to, various display devices for visually transmitting text information, graphics information, and audio / video information, such as monitors, printers, speakers, headphones, car navigation systems, plotters, audio output devices, and modems.

[0229] The computer system 1600 may comprise a storage subsystem 1618 that includes software elements as shown currently within system memory 1610. The system memory 1610 may store data generated during the execution of these programs in addition to program instructions that are readable and executable by the processing unit 1604.

[0230] Depending on the configuration and type of the computer system 1600, the system memory 1610 may be volatile (such as random-access memory (RAM)) and / or non-volatile (such as read-only memory (ROM), flash memory, etc.). The RAM is typically directly accessible by the processing unit 1604 and / or contains data and / or program modules that are currently being operated on and executed. In some implementations, the system memory 1610 may include multiple different types of memory, such as static random-access memory (SRAM) or dynamic random-access memory (DRAM). In some implementations, the basic input / output system (BIOS), which contains basic routines that help transfer information between elements within the computer system 1600, such as during startup, may typically be stored in the ROM. By way of example and not limitation, the system memory 1610 also shows application programs 1612, which may include client applications, web browsers, mid-tier applications, relational database management systems (RDBMS), etc., program data 1614, and an operating system 1616.Examples of the operating system 1616 may include Microsoft Windows®, Apple Macintosh®, and / or Linux operating systems, various commercially available UNIX® or UNIX-like operating systems (including, but not limited to, various GNU / Linux operating systems, Google Chrome® OS, etc.), and / or various versions of mobile operating systems such as iOS, Windows® Phone, Android® OS, BlackBerry® OS, and Palm® OS.

[0231] The storage subsystem 1618 may provide a tangible computer-readable storage medium for storing the basic programming and data configurations that provide the functionality of some embodiments. Software (programs, code modules, instructions) that provides the aforementioned functionality when executed by a processor may be stored in the storage subsystem 1618. These software modules or instructions may be executed by the processing unit 1604. The storage subsystem 1618 may provide a repository for storing data used in accordance with the present disclosure.

[0232] The storage subsystem 1600 may include a computer-readable storage medium reader 1620 that may be further connected to a computer-readable storage medium 1622. In combination with the system memory 1610, optionally, the computer-readable storage medium 1622 may comprehensively represent a storage medium for temporarily and / or more persistently containing, storing, transmitting, and retrieving computer-readable information, in addition to remote, local, fixed, and / or removable storage media.

[0233] A computer-readable storage medium 1622 that includes code or a portion of code can include any suitable medium known in or used in the art, including storage media and communication media, such as volatile and non-volatile, removable and non-removable media implemented in any method or technology, but not limited to these. The computer-readable storage medium 1622 can include tangible computer-readable storage media such as RAM, ROM, electronically erasable programmable ROM (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile disk (DVD), or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage, or other magnetic storage devices, or other tangible computer-readable media. The computer-readable storage medium 1622 can also include non-tangible computer-readable media such as data signals, data transmissions, or any other medium that can be used to transmit desired information and can be accessed by the computing system 1600.

[0234] As an example, computer-readable storage medium 1622 may include a hard disk drive that reads from or writes to a removable non-volatile magnetic medium, a magnetic disk drive that reads from or writes to a removable non-volatile magnetic disk, and an optical disk drive that reads from or writes to a removable non-volatile optical disk such as a CD ROM, DVD, and Blu-ray (registered trademark) disk, or other optical medium. Computer-readable storage medium 1622 may include, but is not limited to, a Zip (registered trademark) drive, a flash memory card, a universal serial bus (USB) flash drive, a secure digital (SD) card, a DVD disk, a digital video tape, etc. Computer-readable storage medium 1622 may include semiconductor drives (SSDs) based on non-volatile memory such as flash memory-based SSDs, enterprise flash drives, semiconductor ROMs, SSDs based on volatile memory such as semiconductor RAM, dynamic RAM, static RAM, DRAM-based SSDs, magneto resistive RAM (MRAM) SSDs, and hybrid SSDs that use a combination of DRAM and flash memory-based SSDs. Disk drives and associated computer-readable media may provide non-volatile storage of computer-readable instructions, data structures, program modules, and other data of computer system 1600.

[0235] The communication subsystem 1624 provides an interface to other computer systems and networks. The communication subsystem 1624 functions as an interface for receiving data from other systems of the computer system 1600 and for transmitting data to other systems. For example, the communication subsystem 1624 may enable the computer system 1600 to connect to one or more devices via the Internet. In some embodiments, the communication subsystem 1624 can include components of a radio frequency (RF) transceiver for accessing wireless voice and / or data networks (such as cellular phone technology, advanced data network technologies such as 3G, 4G, or EDGE (enhanced data rates for global evolution), WiFi (IEEE 802.11 family of standards, or other mobile communication technologies, or any combination thereof), components of a global positioning system (GPS) receiver, and / or other components. In some embodiments, the communication subsystem 1624 can provide a wired network connection (such as Ethernet) in addition to, or instead of, a wireless interface.

[0236] In some embodiments, the communication subsystem 1624 may receive input communications in the form of structured and / or unstructured data feeds 1626, event streams 1628, event updates 1630, etc., on behalf of one or more users who may use the computer system 1600.

[0237] As an example, the communication subsystem 1624 may be configured to receive in real time a data feed 1626 from social networks such as Twitter (registered trademark) feeds, Facebook (registered trademark) updates, and / or other communication services, web feeds such as Rich Site Summary (RSS) feeds, and / or real-time updates from one or more third-party information sources.

[0238] Furthermore, the communication subsystem 1624 may be configured to receive data in the form of a continuous data stream, which may include an event stream 1628 and / or event updates 1630 of real-time events that have no explicit end and are essentially continuous or boundaryless. Examples of applications that generate continuous data may include, for example, sensor data applications, financial tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automotive traffic monitoring, and the like.

[0239] The communication subsystem 1624 may be configured to output structured and / or unstructured data feeds 1626, event streams 1628, event updates 1630, etc. to one or more databases that can communicate with one or more streaming data source computers coupled to the computer system 1600.

[0240] The computer system 1600 can be one of various types, including handheld portable devices (e.g., iPhone (registered trademark) mobile phones, iPad (registered trademark) computing tablets, PDAs), wearable devices (e.g., Google Glass (registered trademark) head-mounted displays), PCs, workstations, mainframes, ticket vending machines, server racks, or any other data processing system.

[0241] Due to the constantly changing nature of computers and networks, the description of the computer system 1600 shown in the figures is merely intended to be a specific example. Many other configurations are possible that include more or fewer components than the system shown in the figures. For example, customized hardware may be used and / or certain elements may be implemented in hardware, firmware, software (including applets), or combinations thereof. Additionally, connections to other computing devices such as network input / output devices may be employed. Based on the disclosure and teachings provided herein, those skilled in the art will understand other methods and / or ways to implement various embodiments.

[0242] Although specific embodiments have been described, various modifications, changes, alternative structures, and equivalents are also included within the scope of the present disclosure. Embodiments are not limited to operating within a particular data processing environment and can operate freely within multiple data processing environments. Further, although embodiments have been described using a particular series of transactions and steps, it should be apparent to those skilled in the art that the scope of the present disclosure is not limited to the series of transactions and steps described. The various features and aspects of the foregoing embodiments may be used individually or together.

[0243] Furthermore, although embodiments have been described using specific combinations of hardware and software, it should be recognized that other combinations of hardware and software are within the scope of the present disclosure. Embodiments may be implemented using only hardware, only software, or a combination of these. The various processes described herein may be implemented on the same processor or different processors in any combination. Thus, when a component or module is described as being configured to perform an operation, such a configuration may be realized, for example, by designing an electronic circuit to perform this operation, by programming a programmable electronic circuit (such as a microprocessor) to perform this operation, or by any combination of these. Processes can communicate using a variety of techniques including, but not limited to, prior art techniques for interprocess communication, and different pairs of processes may use different techniques, or the same pair of processes may use different techniques at different times.

[0244] Accordingly, this specification and the drawings are to be regarded as illustrative rather than limiting. However, it is clear that additions, subtractions, deletions, and other modifications and changes may be made to this specification and the drawings without departing from the broader ideas and scope as set forth in the claims. Thus, while specific embodiments of the disclosure have been described, these are not intended to be limiting. Various changes and equivalents are within the scope of the appended claims.

[0245] In the context of describing the disclosed embodiments (in particular, in the context of the appended claims), the use of the terms "a," "an," and "the" and similar referents should be construed to cover both the singular and the plural, unless specifically indicated otherwise herein or clearly contradicted by the context. The terms "comprising," "having," "including," and "containing" should be construed as open-ended terms (i.e., meaning "including, but not limited to") unless otherwise noted. The term "connected" should be construed to mean either internally or partially or completely contained within, connected to, or joined together with, even in the presence of intervening elements. The recitation of a range of values herein is merely intended to provide a convenient way of referring individually to each separate value falling within the range, and each separate value is incorporated herein as if it were individually recited herein. All methods described herein can be performed in any suitable order, unless otherwise indicated herein or clearly contradicted by the context. The use of any examples, or exemplary language (e.g., "such as") provided herein is merely intended to better illuminate the embodiments and does not impose a limitation on the scope of the disclosure unless otherwise claimed. No language in this specification should be construed as indicating any non-claimed element as essential to the practice of the disclosure.

[0246] Disjunctive language, such as the phrase "at least one of X, Y, or Z," is generally intended, unless otherwise explicitly stated, to convey that an item, condition, etc. may be any one of X, Y, or Z, or any combination thereof (e.g., X, Y, and / or Z) within the context in which it is used. Thus, such disjunctive language is generally not intended, and should not be taken, to mean that a particular embodiment requires the presence of at least one of each of at least one of X, at least one of Y, or at least one of Z.

[0247] In this specification, preferred embodiments of the present disclosure are described, including the best mode known to the applicant for carrying out the present disclosure. Variations of such preferred embodiments may become apparent to those skilled in the art upon reading the foregoing description. Those skilled in the art should be able to adopt such variations as appropriate, and the present disclosure may be practiced otherwise than as specifically described herein. Accordingly, the present disclosure includes all modifications and equivalents of the subject matter recited in the claims appended hereto as permitted by applicable law. Further, any combination of the foregoing elements in all possible variations of the embodiments is included in the present disclosure unless otherwise specifically indicated herein.

[0248] All references, including publications, patent applications, and patents, cited herein are hereby incorporated by reference in their entirety, to the same extent as if each reference had been individually and specifically indicated to be incorporated by reference and were set forth in its entirety herein.

[0249] In the foregoing specification, aspects of the present disclosure have been described with reference to specific embodiments of the present specification. However, those skilled in the art will recognize that the present disclosure is not limited thereto. The various features and aspects of the foregoing disclosure may be used individually or together. Further, embodiments may be utilized in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of the present specification. Accordingly, the present specification and drawings should be regarded as illustrative rather than restrictive.

Claims

1. A method comprising: receiving, at a first network virtualization device (“NVD”), at least one compiled rule, each of the at least one compiled rules being applicable to a class of packets received by the first NVD for delivery to a virtual network interface card (“VNIC”); receiving, at the first NVD, a first packet for delivery to a first VNIC; determining, using the first NVD, that a first rule of the at least one compiled rules is applicable to the first packet; and processing, using the first NVD, the first packet according to the first rule.

2. The method of claim 1, wherein the first packet is processed according to the first rule and according to at least one attribute of the first packet.

3. The method of claim 2, wherein the at least one attribute of the packet includes at least one of a destination address and a source address.

4. The method of claim 3, wherein the source address includes at least one of a source IP address and a source port identifier.

5. The method of claim 3, wherein the destination address includes at least one of a destination IP address and a destination port identifier.

6. The method of claim 1, wherein each of the at least one compiled rules includes a flow key, a security policy, and mapping information.

7. The method of claim 6, wherein the security policy includes a local security policy and a remote security policy.

8. The method of claim 6, wherein each of the at least one compiled rules further includes packet transformation information and throttle information.

9. receiving, at the first NVD, a second packet for delivery to a second VNIC; and determining, using the first NVD, that the first NVD does not have a rule for processing the second packet. Further including transferring the second packet to at least one server, wherein the at least one server is configured to process the second packet, the method according to claim 1.

10. The method according to claim 9, wherein the at least one server includes configuration information of a set of VNICs in at least one virtual network.

11. The method further includes Determining rules for processing the second packet using the at least one server, the rules being determined based on the configuration information and at least one attribute of the second packet, The method according to claim 10, further including processing the second packet according to the determined rules using the at least one server.

12. The method according to claim 11, wherein the at least one attribute of the packet includes at least one of a destination address and a source address.

13. The method according to claim 12, wherein the source address includes at least one of a source IP address and a source port identifier.

14. The method according to claim 12, wherein the destination address includes at least one of a destination IP address and a destination port identifier.

15. The method further includes generating a new compiled rule, the new compiled rule including the second rule, and generating the new compiled rule includes Identifying possible rules for each of a plurality of stages, each of the possible rules being related to a plurality of packet flows, Further including forming the new compiled rule from a common portion among the possible rules for each of the plurality of stages of the flow, The method according to claim 11, further including providing the new compiled rule to the first NVD.

16. The method according to claim 15, further including identifying a plurality of NVDs as being similar to the first NVD, and providing the new compiled rule to the plurality of NVDs similar to the first NVD.

17. The method according to claim 15, wherein the plurality of stages includes two or more of a routing stage, a security stage, a throttle stage, and a conversion stage.

18. At least one server including a plurality of compiled rules, and a first network virtualization device ("NVD"), wherein the NVD is configured to receive at least one compiled rule, and each of the at least one compiled rule is applicable to a class of packets received by the first NVD for delivery to a virtual network interface card ("VNIC"), receive a first packet for delivery to a first VNIC, determine that a first rule among the at least one compiled rule is applicable to the first packet, and a system configured to process the first packet according to the first rule.

19. The first NVD receives a second packet for delivery to a second VNIC, determines that the first NVD does not have a rule for processing the second packet, and is further configured to transfer the second packet to the at least one server, wherein the at least one server includes configuration information of a set of VNICs in at least one virtual network, and the at least one server is configured to determine a rule for processing the second packet, the rule being determined based on the configuration information and at least one attribute of the second packet, The system according to claim 18, further configured to process the second packet according to the determined rule.

20. A non-transitory computer-readable storage medium storing a plurality of instructions executable by one or more processors of a first network virtualization device ("NVD"), wherein the plurality of instructions, when executed by the one or more processors of the first NVD, cause the one or more processors of the first NVD to Causing to execute receiving at least one compiled rule, each of the at least one compiled rule being applicable to a class of packets received by the first NVD for delivery to a virtual network interface card (“VNIC”), receiving a first packet for delivery to a first VNIC, determining that a first rule of the at least one compiled rule is applicable to the first packet, and further causing to execute processing the first packet according to the first rule, a non-transitory computer-readable storage medium.

Images