Invalidating cached flow information in cloud infrastructure

Abstract

Techniques are described for managing distribution of configuration information supporting flow of packets in a cloud environment. In one example, a virtual network interface card (VNIC) hosted on a network virtualization device NVD receives a first packet from a compute instance associated with the VNIC. The VNIC determines that flow information for transmitting the first packet on a virtual network is unavailable from a memory of the NVD. The VNIC transmits the first packet to a network interface service via the NVD, the network interface service maintaining configuration information for transmitting the packet on a substrate network and configured to transmit the first packet on the substrate network based on the configuration information. The NVD receives the flow information from the network interface service, the flow information being a subset of the configuration information. The NVD stores the flow information in memory.

Description

Technical Field

[0001] Reference to Related Applications This international patent application claims priority to U.S. Patent Application No. 17 / 237,750, titled "INVALIDATING CACHED FLOW INFORMATION IN A CLOUD INFRASTRUCTURE," filed on April 22, 2021, which claims the benefit of U.S. Provisional Application No. 63 / 149,276, filed on February 13, 2021, the entire content of which is incorporated by reference for all purposes.

Background Art

[0002] Background Cloud infrastructures such as Oracle Cloud Infrastructure (OCI) can provide a set of cloud services that enable entities (e.g., enterprises) subscribing to cloud services to build and run a wide range of applications and services in a highly available cloud hosting environment. The subscribing entity is called a customer of the cloud service provider. The cloud infrastructure operates on a physical underlying network and can provide high-performance computing, storage, and networking capabilities in a flexible overlay virtual network that is securely accessible from an enterprise's on-premises network. Cloud infrastructures such as OCI generally enable customers to manage their cloud-based workloads in the same way that they manage their on-premises workloads. Thus, an organization can obtain all the benefits of the cloud with the same control, isolation, security, and predictable performance as its on-premises network.

[0003] Virtual networking is fundamental to cloud infrastructure and cloud applications because it enables the ability to access, connect to, secure, and modify cloud resources. It allows communication between multiple computers, virtual machines (VMs), virtual servers, or other devices across different physical locations. While physical networking connects computer systems via cabling and other hardware, virtual networking uses software management to connect computers and servers at different physical locations over the internet. Virtual networks utilize virtualized versions of traditional network components such as network switches, routers, and adapters, enabling more efficient routing and easier network configuration and reconfiguration. [Overview of the project] [Means for solving the problem]

[0004] overview This disclosure generally relates to managing the distribution of configuration information that supports the flow of packets between endpoints in a cloud environment. Configuration information generally includes policies, rules, mappings, routing, and other types of information for sending and / or receiving packets. Distribution can optimize the balance between memory usage and bandwidth. In particular, configuration information may be stored by a network interface service that centrally supports connectivity between compute instances within a virtual network. A relevant subset of the configuration information (referred to herein as flow information) may also be stored locally on the network virtualization device that provides interfaces for compute instances to the network interface service and the virtual network. If flow information does not reside locally on the network virtualization device to support the packet flow to an endpoint, the packet may be sent to the network interface service to process it and send it to the endpoint. Otherwise, the flow information is used in the network virtualization device to process the packet and send it to the endpoint without sending this packet through the network interface service. In this way, instead of storing the entire set of configuration information on the network virtualization device (which uses a relatively large amount of memory and limits the density of interfaces that the network virtualization device can support), only a relevant subset is stored and used locally for packet flow, thereby saving memory space (which then enables a relatively high interface density on the network virtualization device) and saving bandwidth (for example, by not sending packets to endpoints via network interface services).

[0005] The terms and expressions used are for illustrative purposes only, not limitation, and in using such terms and expressions, there is no intention to exclude any equivalents of the illustrated and described features or parts thereof. However, it is recognized that various modifications are possible within the scope of the claimed system and method. Therefore, although the system and method are specifically disclosed by examples and optional features, modifications and variations of the concepts disclosed herein should be recognizable to those skilled in the art, and such modifications and variations should be considered within the scope of the system and method as defined by the claims.

[0006] This summary is not intended to identify any major or essential features of the claimed subject matter, nor is it intended to be used alone to determine the scope of the claimed subject matter. The subject matter should be understood by referring to the entire specification of this disclosure, any or all of the drawings, and the appropriate portions of each claim.

[0007] The matters described above, along with other features and embodiments, will become more apparent upon reference to the following specification, claims, and accompanying drawings.

[0008] Brief explanation of the drawing An example for illustrative purposes is described in detail below with reference to the following diagram. [Brief explanation of the drawing]

[0009] [Figure 1] This is a top-level diagram of a distributed environment showing a virtual or overlay cloud network hosted by a cloud service provider infrastructure, according to one embodiment. [Figure 2] A simplified architectural diagram of the physical components in the physical network within the CSPI, according to one embodiment, is shown. [Figure 3]This illustrates an exemplary configuration within a CSPI in which a host machine is connected to multiple network virtualization devices (NVDs) according to one embodiment. [Figure 4] This figure shows a connection between a host machine and an NVD to provide I / O virtualization to support multi-tenancy, according to one embodiment. [Figure 5] A simplified block diagram of the physical network provided by CSPI according to one embodiment is shown. [Figure 6] This figure shows an example of a network architecture that supports packet flow based on cached configuration information, according to one embodiment. [Figure 7] This figure shows an example of a network architecture that supports packet flow based on uncached configuration information, according to one embodiment. [Figure 8] This figure shows an example of a network architecture that supports packet flow based on cached flow information and uncached configuration information, according to one embodiment. [Figure 9] This figure shows an example of an outbound packet flow that results in cached flow information according to one embodiment. [Figure 10] This figure shows an example of an outbound packet flow based on cached flow information according to one embodiment. [Figure 11] This figure shows an example of an inbound packet flow that results in cached flow information according to one embodiment. [Figure 12] This figure shows an example of an inbound packet flow based on cached flow information according to one embodiment. [Figure 13] This example illustrates how to invalidate and update cached flow information according to one embodiment. [Figure 14] Another example of invalidating and updating cached flow information according to one embodiment is shown. [Figure 15]FIG. showing an example of a method for transmitting a packet that results in cached flow information according to an embodiment. [Figure 16] FIG. showing an example of a method for receiving a packet that results in cached flow information according to an embodiment. [Figure 17] FIG. showing an example of a method for generating and transmitting flow information according to an embodiment. [Figure 18] FIG. showing an example of a method for invalidating and updating cached flow information according to an embodiment. [Figure 19] FIG. showing another example of a method for invalidating and updating cached flow information according to an embodiment. [Figure 20] FIG. showing another example of a method for invalidating and updating cached flow information according to an embodiment. [Figure 21] FIG. showing a block diagram of one pattern for implementing a cloud infrastructure as a service system according to at least one embodiment. [Figure 22] FIG. showing a block diagram of another pattern for implementing a cloud infrastructure as a service system according to at least one embodiment. [Figure 23] FIG. showing a block diagram of another pattern for implementing a cloud infrastructure as a service system according to at least one embodiment. [Figure 24] FIG. showing a block diagram of another pattern for implementing a cloud infrastructure as a service system according to at least one embodiment. [Figure 25] FIG. showing a block diagram of an exemplary computer system according to at least one embodiment. DETAILED DESCRIPTION

[0010] Detailed Description Exemplary virtual networking architecture The term "cloud service" generally refers to services that a cloud service provider (CSP) offers to users or customers on demand (e.g., via a subscription model), using their systems and infrastructure (cloud infrastructure). Typically, the servers and systems that make up the CSP's infrastructure are separate from the customer's own on-premises servers and systems. Therefore, customers can use cloud services provided by the CSP without having to separately purchase hardware and software resources for the service. Cloud services are designed to provide subscribers with easy and scalable access to applications and computing resources without requiring them to invest in procuring the infrastructure used to provide the service.

[0011] There are several cloud service providers that offer various types of cloud services. Cloud services include various different types or models such as SaaS (Software-as-a-Service), PaaS (Platform-as-a-Service), and IaaS (Infrastructure-as-a-Service).

[0012] A customer can subscribe to one or more cloud services provided by a CSP. A customer may be any entity, such as an individual, organization, or company. When a customer subscribes to or registers for a service provided by a CSP, a tenant or account is created for that customer. The customer can then access one or more subscribed cloud resources associated with this account.

[0013] As mentioned above, IaaS (Infrastructure as a Service) is a specific type of cloud computing service. In the IaaS model, the CSP provides the infrastructure (called Cloud Service Provider Infrastructure or CSPI) that customers can use to build their own customizable networks and deploy their customer resources. Thus, the customer's resources and network are hosted in a distributed environment by the infrastructure provided by the CSP. This differs from traditional computing where the customer's infrastructure hosts the customer's resources and network.

[0014] A CSPI can comprise interconnected high-performance computing resources, including various host machines, memory resources, and network resources, forming a physical network also known as an underlay network. Resources within a CSPI can span one or more data centers that are geographically spread across one or more regions. Virtualization software can run on these physical resources to provide a virtualized distributed environment. Virtualization creates an overlay network (also known as a software-based network, software-defined network, or virtual network) on top of a physical network. The CSPI physical network provides the foundation for creating one or more overlay or virtual networks on top of the physical network. The physical network (or underlay network) includes physical network devices such as physical switches, routers, computers, and host machines. An overlay network is a logical (or virtual) network that operates on top of the physical underlay network. A given physical network can support one or more overlay networks. Overlay networks typically use encapsulation techniques to distinguish between traffic belonging to different overlay networks. A virtual or overlay network is also known as a virtual cloud network (VCN). Virtual networks are implemented using software virtualization technologies (e.g., hypervisors, virtualization functions implemented by network virtualization devices (NVDs) (e.g., smart NICs), top-of-rack (TOR) switches, smart TORs that implement one or more functions performed by NVDs, and other mechanisms) to create a layer of network abstraction that can run on top of the physical network. Virtual networks can take many forms, including peer-to-peer networks, IP networks, and others. Virtual networks are typically either Layer 3 IP networks or Layer 2 VLANs. This method of virtual or overlay networking is often called virtual or overlay Layer 3 networking.Examples of protocols developed for virtual networks include IP-in-IP (or General Purpose Routing Encapsulation (GRE)), Virtual Extensible LAN (VXLAN-IETF RFC7348), Virtual Private Networks (VPNs) (e.g., MPLS Layer 3 Virtual Private Network (RFC4364)), VMware's NSX, and Geneve (General Purpose Network Virtualization Encapsulation).

[0015] In the case of IaaS, the infrastructure provided by the CSP (CSPI) may be configured to deliver virtualized computing resources over a public network (e.g., the internet). In the IaaS model, the cloud computing service provider can host infrastructure elements (e.g., servers, storage, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer)). In some cases, the IaaS provider can provide various services associated with these infrastructure elements (e.g., billing, monitoring, logging, security, load balancing, and clustering). Because these services are policy-driven, IaaS users can maintain application availability and performance by implementing policies to drive load balancing. The CSPI provides infrastructure and a set of complementary cloud services. This allows customers to build and run a wide range of applications and services in a highly available hosted distributed environment. The CSPI provides high-performance computing resources and capabilities, as well as storage capacity, on a flexible virtual network that can be securely accessed from various network locations, such as the customer's on-premises network. When a customer subscribes to or registers for an IaaS service provided by a CSP, the tenancy created for that customer becomes a securely isolated partition from the CSP, allowing the customer to create, organize, and manage cloud resources.

[0016] Customers can build their own virtual networks using the computing, memory, and networking resources provided by CSPI. They can deploy one or more customer resources or workloads, such as compute instances, on these virtual networks. For example, a customer can build one or more customizable private virtual networks called Virtual Cloud Networks (VCNs) using the resources provided by CSPI. On a customer VCN, a customer can deploy one or more customer resources, such as compute instances. Compute instances may be virtual machines, bare-metal instances, etc. Thus, CSPI provides the infrastructure and a set of complementary cloud services that enable customers to build and run a variety of applications and services in a highly available virtual host environment. Customers do not manage or control the underlying physical resources provided by CSPI, but they control the operating system, memory, and deployed applications, and, in some cases, have limited control over certain networking components (e.g., firewalls).

[0017] A CSP can provide a console that enables customers and network administrators to configure, access, and manage resources deployed to the cloud using CSPI resources. In certain embodiments, the console provides a web-based user interface that can be used to utilize and manage CSPI. In some embodiments, the console is a web-based application provided by the CSP.

[0018] CSPI can support single-tenancy or multi-tenancy architectures. In a single-tenancy architecture, software (e.g., applications, databases) or hardware elements (e.g., host machines or servers) serve a single customer or tenant. In a multi-tenancy architecture, software or hardware elements serve multiple customers or tenants. Therefore, in a multi-tenancy architecture, CSPI resources are shared among multiple customers or tenants. In a multi-tenancy environment, CSPI implements precautions and safeguards to isolate each tenant's data and prevent it from being visible to other tenants.

[0019] In a physical network, a network endpoint (endpoint) refers to a computing device or system that is connected to the physical network and communicates bidirectionally with the connected network. A network endpoint in a physical network may be connected to a local area network (LAN), a wide area network (WAN), or other types of physical networks. Examples of traditional endpoints in a physical network include modems, hubs, bridges, switches, routers, and other networking devices, as well as physical computers (or host machines). Each physical device in a physical network has a fixed network address that can be used to communicate with that device. This fixed network address may be a Layer 2 address (e.g., a MAC address), a fixed Layer 3 address (e.g., an IP address), etc. In a virtualized environment or virtual network, endpoints can include various virtual endpoints, such as virtual machines hosted by elements of the physical network (e.g., hosted by a physical host machine). These endpoints in a virtual network are addressed by overlay addresses, such as overlay Layer 2 addresses (e.g., overlay MAC addresses) and overlay Layer 3 addresses (e.g., overlay IP addresses). Network overlays provide flexibility by allowing network administrators to move overlay addresses associated with network endpoints using software management (e.g., through software implementing the control plane of the virtual network). Therefore, unlike physical networks, in virtual networks, network management software can be used to move overlay addresses (e.g., overlay IP addresses) from one endpoint to another. Because virtual networks are built on top of physical networks, both the virtual network and the underlying physical network are involved in communication between elements of the virtual network.To facilitate such communication, each element of the CSPI is configured to learn and store mappings that map the overlay address of the virtual network to the actual physical address of the underlying network, or vice versa. These mappings are used to facilitate communication. To facilitate routing within the virtual network, customer traffic is encapsulated.

[0020] Therefore, physical addresses (e.g., physical IP addresses) are associated with components within a physical network, while overlay addresses (e.g., overlay IP addresses) are associated with entities within a virtual or overlay network. A physical IP address is an IP address associated with a physical device (e.g., a network device) within the infrastructure or physical network. For example, each NVD has an associated physical IP address. An overlay IP address is an overlay address associated with an entity within an overlay network, such as a compute instance within a customer's virtual cloud network (VCN). Two different customers or tenants, each with their own private VCN, could potentially use the same overlay IP address within their VCNs without each other's knowledge. Both physical IP addresses and overlay IP addresses are types of real IP addresses. They are distinct from virtual IP addresses. A virtual IP address is typically a single IP address that represents or maps to multiple real IP addresses. A virtual IP address provides a one-to-many mapping between a virtual IP address and multiple real IP addresses. For example, a load balancer may use VIPs to map or represent multiple servers, each having its own actual IP address.

[0021] The cloud infrastructure, or CSPI, is physically hosted in one or more data centers in one or more regions around the world. The CSPI may include elements of a physical network or underlying network and virtualization elements of a virtual network built on top of the physical network elements (e.g., virtual networks, compute instances, virtual machines). In certain embodiments, the CSPI is organized and hosted in realms, regions, and available domains. A region is typically a local geographical area containing one or more data centers. Regions are generally independent of each other and may be separated by vast distances, for example, across countries or continents. For example, one region may be in Australia, another in Japan, and yet another in India. CSPI resources are divided among these regions such that each region has an independent subset of CSPI resources. Each region can provide a set of core infrastructure services and resources, such as computing resources (e.g., bare metal servers, virtual machines, containers, and related infrastructure), storage resources (e.g., block volume storage, file storage, object storage, archive storage), networking resources (e.g., virtual cloud networks (VCNs), load balancing resources, connectivity to on-premises networks), database resources, edge networking resources (e.g., DNS), and access management and monitoring resources. Each region generally has multiple routes for connecting to other regions within the realm.

[0022] Generally, applications are deployed in the region where they are most frequently used (i.e., on infrastructure relevant to that region) because using nearby resources is faster than using distant resources. Applications may also be deployed in different regions for various reasons, such as redundancy to mitigate the risks of large-scale weather systems or region-wide events like earthquakes, or redundancy to meet various requirements for legal jurisdictions, tax domains, and other business or social standards.

[0023] Data centers within a region may be further organized and subdivided into availability domains (ADs). An availability domain may correspond to one or more data centers located in a region. A region may consist of one or more availability domains. In such a distributed environment, CSPI resources are region-specific, such as virtual cloud networks (VCNs), or availability domain-specific, such as compute instances.

[0024] ADs within a single region are configured to be fault-tolerant, isolated from one another, and configured to be highly unlikely to fail simultaneously. This is achieved by configuring ADs so that a failure in one AD within a region has little impact on the availability of other ADs within the same region, by not sharing critical infrastructure resources such as networking, physical cabling, cabling routes, and cabling entry points. Connecting ADs within the same region with a low-latency, high-bandwidth network provides highly available connectivity to other networks (e.g., the internet, customer on-premises networks), and a replication system for both high availability and disaster recovery can be built across multiple ADs. CloudSense utilizes multiple ADs to ensure high availability and protect against resource failures. As the infrastructure provided by the IaaS provider grows, more regions and ADs may be added along with additional capacity. Traffic between available domains is typically encrypted.

[0025] In certain embodiments, regions are grouped into realms. A realm is a logical collection of regions. Realms are isolated from each other and do not share any data. Regions within the same realm can communicate with each other, but regions in different realms cannot. A CSP customer's tenancy or account may reside in a single realm and span one or more regions belonging to that single realm. Typically, when a customer subscribes to an IaaS service, their tenancy or account is created in a customer-designated region within a realm (referred to as the "home" region). The customer can extend their tenancy to one or more other regions within a realm. A customer cannot access regions that do not reside in the realm where their tenancy resides.

[0026] An IaaS provider can offer multiple realms, each corresponding to a specific set of customers or users. For example, a commercial realm may be offered for commercial customers. Another example is that a realm may be offered for a specific country or for customers in that country. Yet another example is that a government realm may be offered for a government, for example. For example, a government realm may be created for a specific government and may have a higher security level than a commercial realm. For example, Oracle Cloud Infrastructure (OCI) currently offers a realm for the commercial domain and two realms for the government cloud domain (e.g., FedRAMP accreditation and IL5 accreditation).

[0027] In certain embodiments, an Active Directory (AD) can be subdivided into one or more fault domains. A fault domain is a grouping of infrastructure resources within the AD to provide anti-affinity. Fault domains can distribute compute instances so that they are not located on the same physical hardware within a single AD. This is known as anti-affinity. A fault domain refers to a collection of hardware elements (computers, switches, etc.) that share a single point of failure. The compute pool is logically divided into fault domains. Therefore, a hardware failure or compute hardware maintenance event affecting one fault domain does not affect instances in other fault domains. Depending on the embodiment, the number of fault domains in each AD may vary. For example, in certain embodiments, each AD may contain three fault domains. Fault domains function as logical data centers within the AD.

[0028] When a customer subscribes to an IaaS service, resources from CSPI are provisioned to the customer and associated with the customer's tenancy. The customer can use these provisioned resources to build private networks and deploy resources on these networks. Customer networks hosted on the cloud by CSPI are called Virtual Cloud Networks (VCNs). A customer can configure one or more Virtual Cloud Networks (VCNs) using the CSPI resources allocated to them. A VCN is a virtual or software-defined private network. Customer resources deployed in a customer's VCN can include compute instances (e.g., virtual machines, bare metal instances) and other resources. These compute instances may represent various customer workloads such as applications, load balancers, and databases. Compute instances deployed on a VCN can communicate with publicly accessible endpoints (public endpoints) over public networks such as the internet, with other instances within the same VCN or other VCNs (e.g., other VCNs of the customer, or VCNs not belonging to the customer), with the customer's on-premises data center or network, with Sendi endpoints, and with other types of endpoints.

[0029] A CSP can provide a variety of services using a CSPI. In some cases, a CSPI customer can act like a service provider themselves and provide services using CSPI resources. A service provider can expose service endpoints characterized by identifying information (e.g., IP address, DNS name, and port). A customer's resources (e.g., compute instances) can consume a particular service by accessing the service endpoint of that particular service exposed by the service. These service endpoints are generally publicly accessible endpoints that users can access via public communication networks such as the internet using the public IP address associated with the endpoint. Publicly accessible network endpoints are sometimes called public endpoints.

[0030] In certain embodiments, a service provider may expose a service through an endpoint of the service (sometimes called a service endpoint). Customers of the service can access the service using this service endpoint. In certain embodiments, the service endpoint provided for a service may be accessible to multiple customers who wish to consume that service. In other implementations, a dedicated service endpoint may be provided to a customer. Thus, only that customer can access the service using that dedicated service endpoint.

[0031] In certain embodiments, a VCN, once created, is associated with a private overlay classless inter-domain routing (CIDR) address space, which is a private overlay IP address range (e.g., 10.0 / 16) assigned to that VCN. A VCN includes associated subnets, route tables, and gateways. While a VCN resides within a single region, it can extend to one or more or all available domains within a region. A gateway is a virtual interface configured for a VCN, enabling traffic communication between the VCN and one or more endpoints outside the VCN. By configuring one or more different types of gateways for a VCN, communication between different types of endpoints can be enabled.

[0032] A VCN may be subdivided into one or more subnets, such as one or more subnets. Thus, a subnet is a constituent unit or partition that can be created within a VCN. A VCN can have one or more subnets. Each subnet within a VCN does not overlap with other subnets within that VCN and is associated with a contiguous range of overlay IP addresses (e.g., 10.0.0.0 / 24 and 10.0.1.0 / 24) that represent an address space subset of the VCN's address space.

[0033] Each compute instance is associated with a virtual network interface card (VNIC). This allows each compute instance to join a subnet in a VCN. A VNIC is a logical representation of a physical network interface card (NIC). Generally, a VNIC is the interface between an entity (e.g., compute instance, service) and a virtual network. A VNIC resides in a subnet and has one or more associated IP addresses and associated security rules or policies. A VNIC is equivalent to a Layer 2 port on a switch. A VNIC connects a compute instance to a subnet within a VCN. The VNIC associated with a compute instance enables the compute instance to be part of a subnet in a VCN and allows the compute instance to communicate (e.g., send and receive packets) with endpoints on the same subnet as the compute instance, endpoints in different subnets within the VCN, or endpoints outside the VCN. Therefore, the VNIC associated with a compute instance determines how the compute instance connects to internal and external endpoints in the VCN. The VNIC for a compute instance is created and associated with that compute instance when the compute instance is created and added to a subnet within the VCN. If a subnet consists of a set of compute instances, it includes the VNICs corresponding to the set of compute instances, and each VNIC is connected to a compute instance within the set of computer instances.

[0034] Each compute instance is assigned a private overlay IP address via the VNIC associated with it. This private overlay IP address is assigned to the VNIC associated with the compute instance when the compute instance is created and is used to route the compute instance's traffic. All VNICs within a given subnet use the same route table, security lists, and DHCP options. As mentioned above, each subnet within a VCN does not overlap with other subnets within that VCN and is associated with a contiguous range of overlay IP addresses (e.g., 10.0.0.0 / 24 and 10.0.1.0 / 24) that represents a subset of the address space of that VCN. For a VNIC on a particular subnet of a VCN, the overlay IP address assigned to the VNIC is an address from the contiguous range of overlay IP addresses assigned to the subnet.

[0035] In certain embodiments, a compute instance may be assigned additional overlay IP addresses, such as one or more public IP addresses in the case of a public subnet, in addition to its private overlay IP address, as needed. These multiple addresses may be assigned to the same VNIC or to multiple VNICs associated with the compute instance. However, each instance has a primary VNIC that is created at instance launch and associated with the overlay private IP address assigned to the instance. This primary VNIC cannot be deleted. Additional VNICs, called secondary VNICs, can be added to an existing instance in the same available domain as the primary VNIC. All VNICs are in the same available domain as the instance. Secondary VNICs may be in the same VCN subnet as the primary VNIC, or they may be in the same VCN or different VCN subnets.

[0036] Compute instances can optionally be assigned a public IP address if they are located in a public subnet. When creating a subnet, you can specify that subnet as either a public or private subnet. A private subnet means that resources within that subnet (e.g., compute instances) and associated VNICs cannot have public overlay IP addresses. A public subnet means that resources within that subnet and associated VNICs can have public IP addresses. Customers can specify subnets that exist across a single available domain or multiple available domains within a region or realm.

[0037] As described above, a VCN may be subdivided into one or more subnets. In certain embodiments, a virtual router (referred to as a VCN VR or simply a VR) configured for the VCN enables communication between subnets within the VCN. For subnets within a VCN, the VR represents the logical gateway for that subnet, enabling communication between the subnet (i.e., compute instances on that subnet) and endpoints on other subnets within the VCN and other endpoints outside the VCN. A VCN VR is a logical entity configured to route traffic between VNICs within the VCN and virtual gateways (gateways) associated with the VCN. Gateways are described further below with reference to Figure 1. A VCN VR is a Layer 3 / IP layer concept. In one embodiment, there is one VCN VR for one VCN. This VCN VR potentially has an unlimited number of ports addressed by IP addresses, with one port for each subnet of the VCN. Thus, the VCN VR has a different IP address for each subnet of the VCN to which the VCN VR is connected. The VR is also connected to various gateways configured for the VCN. In certain embodiments, specific overlay IP addresses from a subnet's overlay IP address range are held on ports of the VCN VR for that subnet. For example, consider a VCN having two subnets, each having the associated address ranges 10.0 / 16 and 10.1 / 16. For the first subnet of the VCN having the address range 10.0 / 16, addresses from this range are held on ports of the VCN VR for that subnet. In some cases, the first IP address from this range may be held on the VCN VR. For example, for a subnet having the overlay IP address range 10.0 / 16, the IP address 10.0.0.1 may be held on ports of the VCN VR for that subnet. For a second subnet within the same VCN having the address range 10.1 / 16, the VCN VR may have ports for the second subnet having the IP address 10.1.0.1.A VCN VR has a different IP address for each subnet within the VCN.

[0038] In some other embodiments, each subnet within a VCN may have its own associated VR, which is addressable by the subnet using a reserved or default IP address associated with the VR. The reserved or default IP address may be, for example, a first IP address from a range of IP addresses associated with that subnet. A VNIC within a subnet can use this default or reserved IP address to communicate with the VR associated with the subnet (e.g., send and receive packets). In such embodiments, the VR is the incoming / outgoing point for that subnet. A VR associated with a subnet within a VCN can communicate with other VRs associated with other subnets within the VCN. A VR can also communicate with gateways associated with the VCN. The VR functionality of a subnet is performed on, or by, one or more NVDs that perform the VNIC functionality of the VNICs within the subnet.

[0039] Route tables, security rules, and DHCP options may be configured for the VCN. The route table is the VCN's virtual route table and contains rules for routing traffic from subnets within the VCN to destinations outside the VCN, via a gateway or specially configured instance. The VCN's route table can be customized to control packet forwarding / routing to and from the VCN. DHCP options refer to configuration information automatically provided to an instance when it is launched.

[0040] Security rules configured for a VCN represent the VCN's overlay firewall rules. Security rules can include inbound and outbound rules and can specify the types of traffic allowed to enter and exit instances within the VCN (e.g., based on protocol and port). Customers can choose whether certain rules are stateful or stateless. For example, a customer can allow incoming SSH traffic to a pair of instances from any location by configuring a stateful inbound rule with source CIDR 0.0.0.0 / 0 and destination TCP port 22. Security rules may be implemented using network security groups or security lists. A network security group consists of a set of security rules that apply only to resources within that group. A security list, on the other hand, contains rules that apply to all resources within a subnet that uses that security list. A VCN may include default security rules and default security lists. DHCP options configured for a VCN provide configuration information that is automatically provided when instances within the VCN start up.

[0041] In certain embodiments, VCN configuration information is determined and stored by the VCN control plane. VCN configuration information may include, for example, address ranges associated with the VCN, subnets and associated information within the VCN, one or more VRs associated with the VCN, compute instances and associated VNICs within the VCN, NVDs that perform various virtualized network functions associated with the VCN (e.g., VNICs, VRs, gateways), VCN status information, and other VCN-related information. In certain embodiments, the VCN distribution service exposes the configuration information or a portion thereof stored by the VCN control plane to the NVD. Using the distributed information, packets can be forwarded to and from compute instances within the VCN by updating information stored and used by the NVD (e.g., forwarding tables, routing tables, etc.).

[0042] In certain embodiments, the creation of VCNs and subnets is handled by the VCN control plane (CP), and the startup of compute instances is handled by the compute control plane. The compute control plane is configured to allocate physical resources for compute instances and then call the VCN control plane to create VNICs and connect to the compute instances. The VCN CP also sends VCN data mappings to the VCN data plane, which is configured to perform packet forwarding and routing functions. In certain embodiments, the VCN CP provides a distribution service configured to provide updates to the VCN data plane. Examples of VCN control planes are shown in Figures 21, 22, 23, and 24 (see reference numbers 2116, 2216, 2316, and 2416) and are described below.

[0043] Customers can create one or more VCNs using resources hosted by CSPI. Compute instances deployed on a customer VCN can communicate with different endpoints. These endpoints can include endpoints hosted by CSPI and endpoints outside of CSPL.

[0044] Various different architectures for implementing cloud-based services using CSPI are shown in Figures 1, 2, 3, 4, 5, 21, 22, 23, and 25, and are described below. Figure 1 is a high-level diagram of a distributed environment 100 showing an overlay VCN or customer VCN hosted by CSPI according to a particular embodiment. The distributed environment shown in Figure 1 includes multiple elements within the overlay network. The distributed environment 100 shown in Figure 1 is merely an example and is not intended to unduly limit the scope of the claimed embodiments. Many variations, alternatives, and modifications are possible. For example, in some implementations, the distributed environment shown in Figure 1 may have more or fewer systems or elements than those shown in Figure 1, may combine two or more systems, or may have different system configurations or arrangements.

[0045] As shown in the example in Figure 1, the distributed environment 100 includes a CSPI 101 that provides services and resources that customers can subscribe to and use to build a virtual cloud network (VCN). In a particular embodiment, the CSPI 101 provides IaaS services to subscriber customers. The data centers within the CSPI 101 may be organized into one or more regions. Figure 1 shows an example of a region, the “US region” 102. The customer has configured a customer VCN 104 for region 102. The customer can deploy various compute instances on the VCN 104, which may include virtual machines or bare metal instances. Examples of instances include applications, databases, load balancers, etc.

[0046] In the embodiment shown in Figure 1, customer VCN 104 includes two subnets, namely "Subnet-1" and "Subnet-2", each subnet having its own CIDR IP address range. In Figure 1, the overlay IP address range for subnet-1 is 10.0 / 16, and the address range for subnet-2 is 10.1 / 16. The VCN virtual router 105 represents the logical gateway of the VCN, enabling communication between subnets of VCN 104 and communication with other endpoints outside the VCN. The VCN VR 105 is configured to route traffic between VNICs within VCN 104 and gateways associated with VCN 104. The VCN VR 105 provides ports to each subnet of VCN 104. For example, the VR 105 can provide a port with IP address 10.0.0.1 to subnet-1 and a port with IP address 10.1.0.1 to subnet-2.

[0047] Multiple compute instances can be deployed on each subnet. In this case, compute instances may be virtual machine instances and / or bare metal instances. Compute instances within a subnet may be hosted by one or more host machines within CSPI101. Compute instances join the subnet via the VNIC associated with them. For example, as shown in Figure 1, compute instance C1 is part of subnet-1 via the VNIC associated with it. Similarly, compute instance C2 is part of subnet-1 via the VNIC associated with C2. Similarly, multiple compute instances, which may be virtual machine instances or bare metal instances, may be part of subnet-1. Each compute instance is assigned a private overlay IP address and MAC address via the associated VNIC. For example, in Figure 1, compute instance C1 has the overlay IP address 10.0.0.2 and MAC address M1, and compute instance C2 has the private overlay IP address 10.0.0.3 and MAC address M2. Each compute instance in subnet-1, including compute instances C1 and C2, has a default route to VCN VR105 using IP address 10.0.0.1, which is the IP address of the port of VCN VR105 in subnet-1.

[0048] Multiple compute instances, including virtual machine instances and / or bare metal instances, can be deployed in subnet-2. For example, as shown in Figure 1, compute instances Dl and D2 are part of subnet-2 via the VNIC associated with each compute instance. In the embodiment shown in Figure 1, compute instance D1 has the overlay IP address 10.1.0.2 and MAC address MM1, and compute instance D2 has the private overlay IP address 10.1.0.3 and MAC address MM2. Each compute instance in subnet-2, including compute instances D1 and D2, has a default route to VCN VR105 using IP address 10.1.0.1, which is the IP address of the port of VCN VR105 in subnet-2.

[0049] Furthermore, VCN A104 may include one or more load balancers. For example, a load balancer may be provided for a subnet and configured to load balance traffic among multiple compute instances on the subnet. Alternatively, a load balancer may be provided to load balance traffic among subnets within the VCN.

[0050] A specific compute instance deployed on VCN104 can communicate with various different endpoints. These endpoints may include endpoints hosted by CSPI200 and endpoints outside of CSPI200. Endpoints hosted by CSPI101 may include endpoints on the same subnet as a particular compute instance (e.g., communication between two compute instances in subnet-1), endpoints on different subnets but within the same VCN (e.g., communication between a compute instance in subnet-1 and a compute instance in subnet-2), endpoints in different VCNs within the same region (e.g., communication between a compute instance in subnet-1 and an endpoint in a VCN in the same region 106 or 110, or between a compute instance in subnet-1 and an endpoint in service network 110 in the same region), or endpoints in VCNs in different regions (e.g., communication between a compute instance in subnet-1 and an endpoint in a VCN in a different region 108). In addition, compute instances in subnets hosted by CSPI101 can communicate with endpoints not hosted by CSPI101 (i.e., outside of CSPI101). These external endpoints include endpoints within the customer's on-premises network 116, endpoints within other remote cloud host networks 118, public endpoints 114 accessible via public networks such as the internet, and other endpoints.

[0051] Communication between compute instances on the same subnet is facilitated using VNICs associated with the source and destination compute instances. For example, compute instance C1 in subnet-1 may want to send a packet to compute instance C2, also in subnet-1. For a packet sent from the source compute instance, whose destination is another compute instance on the same subnet, this packet is first processed by the VNIC associated with the source compute instance. The processing performed by the VNIC associated with the source compute instance may include determining the packet's destination information from the packet header, identifying any policies (e.g., security lists) configured for the VNIC associated with the source compute instance, determining the packet's next hop, performing any packet encapsulation / decapsulation functions as needed, and forwarding / routing the packet to the next hop to facilitate communication to its intended destination. If the destination compute instance is on the same subnet as the source compute instance, the VNIC associated with the source compute instance is configured to identify the VNIC associated with the destination compute instance and forward the packet to that VNIC for processing. Next, the VNIC associated with the destination compute instance is executed and forwards the packets to the destination compute instance.

[0052] When a packet is transmitted from a compute instance within a subnet to an endpoint in a different subnet of the same VCN, the communication is facilitated by the VNICs associated with the source and destination compute instances, and the VCN VR. For example, if compute instance C1 in subnet-1 in Figure 1 wants to send a packet to compute instance D1 in subnet-2, the packet is first processed by the VNIC associated with compute instance C1. The VNIC associated with compute instance C1 is configured to route the packet to VCN VR105 using the VCN VR's default route or port 10.0.0.1. VCN VR105 is configured to route the packet to subnet-2 using port 10.1.0.1. The packet is then received and processed by the VNIC associated with D1, and the VNIC forwards the packet to compute instance D1.

[0053] To transmit packets from compute instances within VCN104 to endpoints outside VCN104, communication is facilitated by a VNIC associated with the source compute instance, VCN VR105, and a gateway associated with VCN104. One or more types of gateways can be associated with VCN104. A gateway is an interface between the VCN and another endpoint, which is outside the VCN. A gateway is a Layer 3 / IP layer concept that enables communication between the VCN and endpoints outside the VCN. Therefore, gateways facilitate traffic flow between the VCN and other VCNs or networks. Different types of gateways can be configured in the VCN to facilitate different types of communication with different types of endpoints. Through gateways, communication may take place over a public network (e.g., the internet) or a private network. Various communication protocols may be used for these communications.

[0054] For example, compute instance C1 may want to communicate with an endpoint outside of VCN104. The packet may first be processed by the VNIC associated with source compute instance C1. The VNIC processing determines that the packet's destination is outside subnet-1 of Cl. The VNIC associated with C1 can then forward the packet to VCN VR105 of VCN104. VCN VR105 then processes the packet and, as part of the processing, determines a specific gateway associated with VCN104 as the packet's next hop based on the packet's destination. VCN VR105 can then forward the packet to the specific gateway. For example, if the destination is an endpoint within the customer's operation-premise network, the packet may be forwarded by VCN VR105 to a dynamic routing gateway (DRG) 122 configured for VCN104. The packet is then forwarded from the gateway to the next hop, facilitating communication of the packet to its intended final destination.

[0055] Various different types of gateways may be configured for the VCN. Examples of gateways that may be configured for a VCN are shown in Figure 1 and described below. Examples of gateways associated with a VCN are also shown in Figures 21, 22, 23 and 24 (for example, gateways shown by reference numbers 2134, 2136, 2138, 2234, 2236, 2238, 2334, 2336, 2338, 2434, 2436, and 2438) and described below. As shown in the embodiment shown in Figure 1, a dynamic routing gateway (DRG) 122 may be added to or associated with the customer VCN 104. The DRG 122 provides a path for private network traffic communication between the customer VCN 104 and another endpoint. The other endpoint may be the customer on-premises network 116, VCN 108 in a different region of CSPI 101, or another remote cloud network 118 not hosted by CSPI 101. The customer on-premises network 116 may be a customer network or customer data center built using the customer's resources. Access to the customer on-premises network 116 is generally strictly restricted. For a customer that has both the customer on-premises network 116 and one or more VCNs 104 deployed or hosted in the cloud by CSPI 101, the customer may want the on-premises network 116 and the cloud-based VCNs 104 to be able to communicate with each other. This would allow the customer to build an enhanced hybrid environment that includes the customer's VCNs 104 hosted by CSPI 101 and the on-premises network 116. DRG 122 enables such communication. To enable such communication, a communication channel 124 is configured. In this case, one endpoint of the communication channel is on the customer on-premises network 116, and the other endpoint is on CSPI 101 and connected to the customer VCN 104. The communication channel 124 can be via a public communication network such as the internet, or a private communication network.Various different communication protocols can be used, such as IPsec VPN technology on public communication networks like the Internet, and Oracle®'s FastConnect technology which uses a private network instead of a public network. Devices or equipment within the customer on-premises network 116 that form one endpoint of communication channel 124 are called customer premises equipment (CPE), such as CPE126 shown in Figure 1. The endpoint on the CSPI101 side may be a host machine running DRG122.

[0056] In certain embodiments, a Remote Peering Connection (RPC) can be added to the DRG. This allows a customer to peer one VCN with another VCN in a different region. Using such an RPC, a customer VCN 104 can connect to a VCN 108 in a different region using the DRG 122. The DRG 122 may also be used to communicate with other remote cloud networks 118 not hosted by the CSPI 101, such as the Microsoft® Azure cloud or the Amazon® AWS cloud.

[0057] As shown in Figure 1, an Internet Gateway (IGW) 120 can be configured on the customer VCN 104 to enable compute instances on the customer VCN 104 to communicate with a public endpoint 114 accessible via a public network such as the Internet. The IGW 120 is a gateway for connecting the VCN to a public network such as the Internet. The IGW 120 enables public subnets within a VCN, such as VCN 104 (resources within public subnets have public overlay IP addresses), to directly access a public endpoint 112 on a public network such as the Internet 114. Connections can be initiated from subnets within VCN 104 or from the Internet using the IGW 120.

[0058] A Network Address Translation (NAT) gateway 128 can be configured in customer VCN 104. The NAT gateway 128 enables cloud resources within the customer VCN that do not have dedicated public overlay IP addresses to access the internet without directly exposing them to incoming internet connectivity (e.g., L4-L7 connectivity). This allows private subnets within the VCN, such as private subnet-1 of VCN 104, to have private access to public endpoints on the internet. With the NAT gateway, private subnets can initiate connections to the public internet, but connections cannot be initiated from the internet to the private subnets.

[0059] In certain embodiments, a service gateway (SGW) 126 can be configured in a customer VCN 104. The SGW 126 provides a route for private network traffic between VCN 104 and service endpoints supported by a service network 110. In certain embodiments, the service network 110 may be provided by a CSP and can provide a variety of services. An example of such a service network is the Oracle® service network, which provides a variety of services that customers can use. For example, compute instances (e.g., database systems) in a private subnet of customer VCN 104 can back up data to service endpoints (e.g., object storage devices) without requiring a public IP address or access to the internet. In some embodiments, a VCN may have only one SGW, and connections can only be initiated from subnets within the VCN, and not from the service network 110. When peering a VCN with another VCN, resources in the other VCN typically cannot access the SGW. Resources in an on-premises network connected to a VCN via FastConnect or VPN Connect can also use the service gateway configured for that VCN.

[0060] In some implementations, SGW126 uses service-classless inter-domain routing (CIDR) labels. A CIDR label is a string representing all regionally exposed IP address ranges for a service or group of services of interest. Customers use service CIDR labels to control traffic to services when configuring SGW and associated routing rules. Customers can optionally use service CIDR labels when configuring security rules without having to adjust security rules if the public IP addresses of services change in the future.

[0061] The Local Peering Gateway (LPG) 132 is an addable gateway to the customer VCN 104 that enables the VCN 104 to peer with other VCNs within the same region. Peering means that VCNs communicate using private IP addresses without traffic traversing a public network such as the internet or routing traffic through the customer's on-premises network 116. In a preferred embodiment, the VCN has a separate LPG for each established peering. Local peering, or VCN peering, is a common practice used to establish network connectivity between different applications or infrastructure management functions.

[0062] Service providers, such as service providers on service network 110, can provide access to their services using different access models. According to the public access model, a service may be exposed as a public endpoint accessible publicly by compute instances within the customer VCN via a public network such as the internet, or it may be accessed privately via SGW126. According to a specific private access model, a service may be accessed as a private IP endpoint within a private subnet within the customer VCN. This is called private endpoint (PE) access and allows service providers to expose their services as instances within the customer's private network. A private endpoint resource represents a service within the customer VCN. Each PE appears as a VNIC (referred to as a PE-VNIC, having one or more private IPs) selected by the customer from a subnet within the customer VCN. Thus, a PE provides a way to provide services within the customer's private VCN subnet using a VNIC. Because the endpoint is exposed as a VNIC, the PE VNIC can utilize all the features associated with a VNIC, such as routing rules and security lists.

[0063] Service providers enable access via PE by registering their services. Providers can associate policies with services that restrict their visibility to customer tenants. Providers can register multiple services under a single virtual IP address (VIP), especially in the case of multi-tenant services. Multiple private endpoints may exist representing the same service (across multiple VCNs).

[0064] Subsequently, compute instances within the private subnet can access the service using the PE VNIC's private IP address or service DNS name. Compute instances within the customer VCN can access the service by sending traffic to the PE's private IP address within the customer VCN. The Private Access Gateway (PAGW) 130 is a gateway resource that can connect to a service provider VCN (e.g., a VCN within service network 110) and act as the receiving / transmitting point for all traffic to and from the customer subnet private endpoint. The PAGW 130 allows the provider to scale the number of PE connections without utilizing internal IP address resources. The provider only needs to configure one PAGW for any number of services registered in a single VCN. The provider can present a service as a private endpoint in multiple VCNs for one or more customers. From the customer's perspective, the PE VNIC appears to be connected to the service the customer wants to interact with, rather than to the customer's instances. Traffic directed to the private endpoint is routed to the service via the PAGW 130. These are called customer-to-service private connections (C2S connections).

[0065] Furthermore, by using the PE concept, private access to the service can be extended to the customer's on-premises network and data center by enabling traffic to flow through FastConnect / IPsec links and private endpoints within the customer's VCN. Private access to the service can also be extended to the customer's peering VCN by enabling traffic to flow between LPG132 and PEs within the customer's VCN.

[0066] Customers can control VCN routing at the subnet level, allowing them to specify which subnets use which gateways within their VCN, such as VCN104. The VCN's route table can be used to determine whether traffic can be routed outside the VCN through a particular gateway. For example, in a specific case, the route table for a public subnet within customer VCN104 might allow non-local traffic to be sent via IGW120. The route table for a private subnet within the same customer VCN104 might allow traffic to CSP services via SGW126. All remaining traffic could be sent via NAT gateway 128. The route table only controls traffic leaving the VCN.

[0067] Security lists associated with a VCN are used to control inbound connections and traffic entering the VCN via gateways. All resources within a subnet use the same mute table and security lists. Security lists may be used to control specific types of traffic entering and leaving instances within a VCN subnet. Security list rules may include inbound and outbound rules. For example, inbound rules may specify allowed source address ranges, and outbound rules may specify allowed destination address ranges. Security rules may specify specific protocols (e.g., TCP, ICMP), specific ports (e.g., port 22 for SSH, port 3389 for Windows® RDP), etc. In certain implementations, the instance's operating system may enforce its own firewall rules that match the security list rules. Rules may be stateful (e.g., connections are tracked and responses are automatically allowed without explicit security list rules for response traffic) or stateless.

[0068] Access from a customer VCN (i.e., resources or compute instances deployed on VCN104) can be classified as public access, private access, or dedicated access. Public access refers to an access model for accessing public endpoints using public IP addresses or NAT. Private access allows customer workloads within VCN104 with private IP addresses (e.g., resources in a private subnet) to access services without traversing a public network such as the internet. In certain embodiments, CSPI101 allows customer VCN workloads with private IP addresses to access the public service endpoint of a service using a service gateway. Thus, the service gateway provides a private access model by establishing a virtual link between the customer VCN and the public endpoint of a service that resides outside the customer's private network.

[0069] Furthermore, CSPI can provide dedicated public access using technologies such as FastConnect public peering. In this case, customer on-premises instances can access one or more services within the customer VCN using FastConnect connectivity without going through a public network such as the internet. CSPI can also provide dedicated private access using FastConnect private peering. In this case, customer on-premises instances with private IP addresses can access customer VCN workloads using FastConnect connectivity. FastConnect is a network connectivity used as an alternative to connecting customer on-premises networks to CSPI and its services using the public internet. FastConnect provides a simple, flexible, and economical way to create dedicated private connectivity with higher bandwidth options and a reliable, consistent networking experience compared to internet-based connectivity.

[0070] Figure 1 and the accompanying description above illustrate various virtualization elements in an exemplary virtual network. As mentioned above, the virtual network is built on an underlying physical network or infrastructure network. Figure 2 is a simplified architectural diagram showing the physical elements within the physical network within the CSPI200 that provide the foundation for the virtual network, according to a particular embodiment. As shown, the CSPI200 provides a distributed environment including elements and resources (e.g., compute resources, memory resources, and networking resources) provided by a Cloud Service Provider (CSP). These elements and resources are used to provide cloud services (e.g., IaaS services) to subscribers, i.e., customers who subscribe to one or more services provided by the CSP. Based on the services a customer subscribes to, the CSPI200 provides some resources (e.g., compute resources, memory resources, and networking resources) to the customer. The customer can then use the physical compute resources, memory resources, and networking resources provided by the CSPI200 to build their own cloud-based (i.e., CSPI-hosted) customizable private virtual network. As mentioned above, these customer networks are called virtual cloud networks (VCNs). Customers can deploy one or more customer resources, such as compute instances, to these customer VCNs. Compute instances may be virtual machines, bare metal instances, etc. CSPI200 provides infrastructure and a suite of complementary cloud services that enable customers to build and run a wide range of applications and services in a highly available host environment.

[0071] In the exemplary embodiment shown in Figure 2, the physical elements of CSPI200 include one or more physical host machines or physical servers (e.g., 202, 206, 208), network virtualization devices (NVDs) (e.g., 210, 212), top-of-rack (TOR) switches (e.g., 214, 216), a physical network (e.g., 218), and switches within physical network 218. The physical host machines or servers can host and run various compute instances participating in one or more subnets of the VCN. Compute instances may include virtual machine instances and bare metal instances. For example, the various compute instances shown in Figure 1 may be hosted by the physical host machines shown in Figure 2. Virtual machine compute instances in the VCN may run on one host machine or on several different host machines. The physical host machines can also host virtual host machines, container-based hosts or functions, etc. The VIC and VCN VR shown in Figure 1 may run on the FTVD shown in Figure 2. The gateway shown in Figure 1 may be run by the host machine and / or NVD shown in Figure 2.

[0072] A host machine or server can run a hypervisor (also known as a virtual machine monitor or VMM) that creates and enables a virtualized environment on the host machine. Virtualization or a virtualized environment facilitates cloud-based computing. One or more compute instances may be created, run, and managed on the host machine by a hypervisor on the host machine. The hypervisor on the host machine can share the host machine's physical compute resources (e.g., compute resources, memory resources, and networking resources) among various compute instances running on the host machine.

[0073] For example, as shown in Figure 2, host machines 202 and 208 run hypervisors 260 and 266, respectively. These hypervisors may be implemented using software, firmware, hardware, or a combination thereof. Typically, a hypervisor is a process or software layer residing in the host machine's operating system (OS), which runs on the host machine's hardware processors. A hypervisor provides a virtualization environment that allows the host machine's physical computing resources (e.g., processing resources such as processors / cores, memory resources, and networking resources) to be shared among various virtual machine computing instances running on the host machine. For example, in Figure 2, hypervisor 260 resides in the OS of host machine 202 and allows the host machine 202's computing resources (e.g., processing resources, memory resources, and networking resources) to be shared among computing instances (e.g., virtual machines) running on host machine 202. A virtual machine can have its own OS (called a guest OS). This guest OS may be the same as or different from the host machine's OS. The operating system (OS) of a virtual machine running on a host machine may be the same as, or different from, the operating systems of other virtual machines running on the same host machine. Therefore, a hypervisor can run multiple OSs in parallel while sharing the same computing resources of the host machine. The host machines shown in Figure 2 may have the same type of hypervisor or different types of hypervisors.

[0074] Compute instances may be virtual machine instances or bare metal instances. In Figure 2, compute instance 268 on host machine 202 and compute instance 274 on host machine 208 are examples of virtual machine instances. Host machine 206 is an example of a bare metal instance provided to a customer.

[0075] In certain examples, the entire host machine may be provided to a single customer, and one or more compute instances (either virtual machines or bare metal instances) hosted by that host machine may all belong to the same customer. In other examples, the host machine may be shared among multiple customers (i.e., multiple tenants). In such a multi-tenant scenario, the host machine can host virtual machine compute instances belonging to different customers. These compute instances may be members of different VCNs of different customers. In certain embodiments, bare metal compute instances are hosted by bare metal servers without a hypervisor. When bare metal compute instances are provided, a single customer or tenant maintains control of the physical CPU, memory, and network interfaces of the host machine hosting the bare metal instances, and the host machine is not shared with other customers or tenants.

[0076] As mentioned above, each compute instance that is part of a VCN is associated with a VNIC that enables the compute instance to be a member of the VCN's subnet. The VNIC associated with a compute instance facilitates the communication of packets or frames to and from the compute instance. The VNIC is associated with the compute instance when it is created. In certain embodiments, for a compute instance run by a host machine, the VNIC associated with that compute instance is run by an NVD connected to the host machine. For example, in Figure 2, host machine 202 runs virtual machine compute instance 268 associated with VNIC 276, and VNIC 276 is run by an NVD 210 connected to host machine 202. In another example, bare metal instance 272 hosted by host machine 206 is associated with VNIC 280, which is run by an NVD 212 connected to host machine 206. In yet another example, VNIC 284 is associated with compute instance 274 run by host machine 208, and VNIC 284 is run by an NVD 212 connected to host machine 208.

[0077] For compute instances hosted by a host machine, an NVD connected to that host machine executes a VCN VR corresponding to the VCN of which the compute instance is a member. For example, in the embodiment shown in Figure 2, NVD210 executes VCN VR277 corresponding to the VCN of which compute instance 268 is a member. Additionally, NVD212 can execute one or more VCN VR283 corresponding to the VCNs of compute instances hosted by host machines 206 and 208.

[0078] A host machine may include one or more network interface cards (NICs) for connecting it to other devices. The NICs on the host machine may provide one or more ports (or interfaces) for communicating with another device. For example, a host machine can be connected to an NVD using one or more ports (or interfaces) provided on the host machine and the NVD. Alternatively, a host machine can be connected to other devices, such as another host machine.

[0079] For example, in Figure 2, host machine 202 is connected to NVD210 using a link 220 that extends between port 234 provided by NIC 232 of host machine 202 and port 236 of NVD210. Host machine 206 is connected to NVD212 using a link 224 that extends between port 246 provided by NIC 244 of host machine 206 and port 248 of NVD212. Host machine 208 is connected to NVD212 using a link 226 that extends between port 252 provided by NIC 250 of host machine 208 and port 254 of NVD212.

[0080] Similarly, the NVDs are connected via communication links to top-of-rack (TOR) switches connected to a physical network 218 (also called a switch fabric). In certain embodiments, the links between the host machines and the NVDs, and between the NVDs and the TOR switches, are Ethernet® links. For example, in Figure 2, NVDs 210 and 212 are connected to TOR switches 214 and 216, respectively, via links 228 and 230. In certain embodiments, links 220, 224, 226, 228, and 230 are Ethernet® links. The collection of host machines and NVDs connected to the TOR is sometimes referred to as a rack.

[0081] The physical network 218 provides a communication fabric that enables communication between TOR switches. The physical network 218 may be a multi-layer network. In a particular implementation, the physical network 218 is a multi-layer Clos network of switches, and TOR switches 214 and 216 represent leaf-level nodes of the multi-layer and multi-node physical switching network 218. Different Clos network configurations are possible, including but not limited to 2-layer, 3-layer, 4-layer, 5-layer networks, and generally "n"-layer networks. An example of a Clos network is shown in Figure 5 and described below.

[0082] Various connection configurations are possible between the host machine and N VDs, including one-to-one, many-to-one, and one-to-many configurations. In an example of a one-to-one configuration, each host machine is connected to its own separate NVD. For example, in Figure 2, host machine 202 is connected to NVD210 via host machine 202's NIC232. In a many-to-one configuration, multiple host machines are connected to a single NVD. For example, in Figure 2, host machines 206 and 208 are connected to the same NVD212 via NIC244 and 250, respectively.

[0083] In a one-to-many configuration, one host machine is connected to multiple NVDs. Figure 3 shows an example within CSPI300 where a host machine is connected to multiple NVDs. As shown in Figure 3, the host machine 302 has a network interface card (NIC) 304 that includes multiple ports 306 and 30S. The host machine 300 is connected to the first NVD 310 via port 306 and link 320, and to the second NVD 312 via port 308 and link 322. Ports 306 and 308 may be Ethernet® ports, and links 320 and 322 between the host machine 302 and the NVDs 310 and 312 may be Ethernet® links. The NVD 310 is connected to the first TOR switch 314, and the NVD 312 is connected to the second TOR switch 316. The links between the NVDs 310 and 312 and the TOR switches 314 and 316 may be Ethernet® links. TOR switches 314 and 316 represent layer-0 switching devices within a multilayer physical network 318.

[0084] The configuration shown in Figure 3 provides two separate physical network paths from the physical switch network 318 to the host machine 302: a first path from TOR switch 314 through NVD 310 to the host machine 302, and a second path from TOR switch 316 through NVD 312 to the host machine 302. These separate paths provide enhanced availability (referred to as high availability) for the host machine 302. If one path experiences a problem (e.g., one link in the path fails) or if there is a problem with a device (e.g., a particular NVD is not functioning), the other path can be used for communication with the host machine 302.

[0085] In the configuration shown in Figure 3, the host machine is connected to two different NVDs using two different ports provided by the host machine's NIC. In other embodiments, the host machine may include multiple NICs that enable connections between the host machine and multiple NVDs.

[0086] Referring again to Figure 2, the NVD is a physical device or element that performs one or more network virtualization functions and / or memory virtualization functions. The NVD may be any device having one or more processing units (e.g., CPU, network processing unit (NPU), FPGA, packet processing pipeline), memory including a cache, and ports. Various virtualization functions may be performed by software / firmware executed by one or more processing units of the NVD.

[0087] NVDs may be implemented in various different forms. For example, in certain embodiments, an NVD may be implemented as an interface card called a smart NIC or intelligent NIC with an integrated processor. A smart NIC is a separate device from the NIC on the host machine. In Figure 2, NVD210 may be implemented as a smart NIC connected to host machine 202, and NVD212 may be implemented as smart NICs connected to host machines 206 and 208.

[0088] However, the smart NIC is just one example of an NVD implementation. Various other implementations are possible. For example, in some other implementations, the NVD or one or more functions performed by the NVD may be incorporated into or performed by one or more host machines, one or more TOR switches, and other elements of the CSPI200. For example, the NVD may be integrated into the host machine. In this case, the functions performed by the NVD are performed by the host machine. As another example, the NVD may be part of a TOR switch, or the TOR switch may be configured to perform functions performed by the NVD, enabling the TOR switch to perform various complex packet translations used in public clouds. A TOR that performs the functions of the NVD is sometimes called a smart TOR. In yet another implementation that provides customers with virtual machine (VM) instances rather than bare metal (BM) instances, the functions provided by the NVD may be implemented within the hypervisor of the host machine. In some other implementations, some of the functions of the NVD may be offloaded to a centralized service running on a set of host machines.

[0089] As shown in Figure 2, in certain embodiments, such as when implemented as a smart NIC, the NVD may have multiple physical ports that enable it to connect to one or more host machines and one or more TOR switches. The ports on the NVD can be classified as host-facing ports (also called "south ports") or network-facing or TOR-facing ports (also called "north ports"). Host-facing ports on the NVD are the ports used to connect the NVD to a host machine. Examples of host-facing ports in Figure 2 include port 236 on the NVD210 and ports 248 and 254 on the NVD212. Network-facing ports on the NVD are the ports used to connect the NVD to a TOR switch. Examples of network-facing ports in Figure 2 include port 256 on the NVD210 and port 258 on the NVD212. As shown in Figure 2, the NVD210 is connected to the TOR switch 214 via a link 228 extending from port 256 on the NVD210 to the TOR switch 214. Similarly, the NVD212 is connected to the TOR switch 216 via a link 230 that extends from port 258 of the NVD212 to the TOR switch 216.

[0090] The NVD can receive packets and frames from the host machine (for example, packets and frames generated by compute instances hosted by the host machine) via its host-facing port, perform the necessary packet processing, and then forward the packets and frames to the TOR switch via its network-facing port. The NVD can also receive packets and frames from the TOR switch via its network-facing port, perform the necessary packet processing, and then forward the packets and frames to the host machine via its host-facing port.

[0091] In certain embodiments, multiple ports and associated links may be provided between the NVD and the TOR switch. By aggregating these ports and links, a link aggregator group (LAG) of multiple ports or links can be formed. Link aggregation allows multiple physical links between two endpoints (e.g., between the NVD and the TOR switch) to be treated as a single logical link. All physical links within a given LAG can operate in full-duplex mode at the same speed. LAGs help to increase the bandwidth and reliability of the connection between the two endpoints. If one of the physical links in the LAG fails, traffic is dynamically and transparently reassigned to another physical link within the LAG. Aggregated physical links provide higher bandwidth than individual links. Multiple ports associated with a LAG are treated as a single logical port. Traffic can be load-balanced across the multiple physical links of the LAG. One or more LAGs can be configured between two endpoints. The two endpoints may be, for example, between the NVD and the TOR switch, or between a host machine and the NVD.

[0092] NVD implements or performs network virtualization functions. These functions are performed by software / firmware run by NVD. Examples of network virtualization functions, but not limited to, include packet encapsulation and decapsulation functions, functions for creating VCN networks, functions for implementing network policies such as VCN security list (firewall) functions, and functions for facilitating the routing and forwarding of packets to and from compute instances within the VCN. In certain embodiments, upon receiving a packet, NVD is configured to run a packet processing pipeline that processes the packet and determines how to forward or route it. As part of this packet processing pipeline, NVD provides the execution of VNICs related to compute instances within the VCN, the execution of virtual routers (VRs) related to the VCN, packet encapsulation and decapsulation to facilitate forwarding or routing within the virtual network, the execution of specific gateways (e.g., local peering gateways), the implementation of security lists, network security groups, network address translation (NAT) functions (e.g., translation from public IP to private IP on a per-host basis), throttling functions, and other functions.

[0093] In some embodiments, the packet processing data path in the NVD may include multiple packet pipelines. Each packet pipeline consists of a set of packet translation stages. In some implementations, upon receiving a packet, it is parsed and classified into a single pipeline. The packet is then processed linearly, stage by stage, until it is discarded or sent out through the NVD's interface. These stages provide the basic functional packet processing building blocks (e.g., header validation, throttling, insertion of new Layer 2 headers, L4 firewall execution, VCN encapsulation / decapsulation), and as a result, new pipelines can be constructed by assembling existing stages, and new functionality can be added by creating new stages and inserting them into existing pipelines.

[0094] The NVD can perform both control plane and data plane functions corresponding to the VCN's control plane and data plane. An example of the VCN control plane is shown in Figures 21, 22, 23, and 24 (see references 2116, 2216, 2316, and 2416) and is described below. An example of the VCN data plane is shown in Figures 21, 22, 23, and 24 (see references 2118, 2218, 2318, and 2418) and is described below. Control plane functions include functions used to configure the network to control how data is forwarded (e.g., setting routes and route tables, configuring VNICs). In certain embodiments, a VCN control plane is provided that centrally calculates the mapping of all overlays to the substrate and exposes it to the NVD and virtual network edge devices (e.g., various gateways such as DRG, SGW, IGW). Firewall rules can also be exposed using the same mechanism. In certain embodiments, the NVD retrieves only the mappings relevant to that NVD. The data plane function includes the ability to perform the actual routing / forwarding of packets based on the configuration set using the control plane. The VCN data plane is implemented by encapsulating customer network packets before they pass through the backbone network. The encapsulation / decapsulation function is implemented in the NVD. In certain embodiments, the NVD is configured to intercept all network packets entering and leaving the host machine and to perform network virtualization functions.

[0095] As described above, NVD performs various virtualization functions, including VNICs and VCN VRs. An NVD can run VNICs associated with compute instances hosted by one or more host machines connected to a VNIC. For example, as shown in Figure 2, NVD210 runs the functions of VNIC276 associated with compute instance 268 hosted by host machine 202 connected to NVD210. As another example, NVD212 runs VNIC280 associated with bare-metal compute instance 272 hosted by host machine 206 and VNIC284 associated with compute instance 274 hosted by host machine 208. A host machine can host compute instances belonging to different VCNs belonging to different customers. An NVD connected to a host machine can run VNICs corresponding to compute instances (i.e., perform functions associated with VNICs).

[0096] Furthermore, the NVD runs a VCN virtual router corresponding to the VCN of the compute instance. For example, in the embodiment shown in Figure 2, NVD210 runs VCN VR277 corresponding to the VCN to which compute instance 268 belongs. NVD212 runs one or more VCN VR283 corresponding to one or more VCNs to which compute instances hosted on host machines 206 and 208 belong. In a particular embodiment, a VCN VR corresponding to a VCN is run by all NVDs connected to a host machine that hosts at least one compute instance belonging to that VCN. If a host machine hosts compute instances belonging to a different VCN, the NVDs connected to that host machine can run VCN VRs corresponding to different VCNs.

[0097] In addition to VNICs and VCN VRs, an NVD may include one or more hardware elements that run various software (e.g., daemons) and facilitate various network virtualization functions performed by the NVD. For simplicity, these various elements are grouped as “packet processing elements” as shown in Figure 2. For example, NVD210 includes packet processing element 286, and NVD212 includes packet processing element 288. For example, a packet processing element of an NVD may include a packet processor configured to monitor all packets received and communicated using the NVD and to store network information by interacting with the NVD’s ports and hardware interfaces. Network information may include, for example, network flow information to identify different network flows processed by the NVD and information about each flow (e.g., statistics for each flow). In certain embodiments, network flow information may be stored on a per-VNIC basis. As another example, a packet processing element may include a replication agent configured to replicate the information stored by the NVD to one or more different replication target stores. As yet another example, the packet processing element may include a logging agent configured to perform the NVD's logging function. The packet processing element may also include software to monitor the performance and health of the NVD, and optionally the status and health of other elements connected to the NVD.

[0098] Figure 1 shows the elements of an exemplary virtual or overlay network, including a VCN, subnets within the VCN, compute instances deployed on the subnets, VNICs associated with the compute instances, a VR for the VCN, and a set of gateways configured for the VCN. The overlay elements shown in Figure 1 may be run or hosted by one or more of the physical elements shown in Figure 2. For example, compute instances within a VCN may be run or hosted by one or more host machines shown in Figure 2. In the case of compute instances hosted by host machines, the VNICs associated with those compute instances are typically run by NVDs connected to that host machine (i.e., VNIC functionality is provided by NVDs connected to that host machine). The VCN VR functionality of the VCN is run by all NVDs connected to the host machines that host or run the compute instances that are part of that VCN. Gateways associated with the VCN may be run by one or more different types of NVDs. For example, some gateways may be run by smart NICs, and others may be run by one or more host machines or other implementations of NVDs.

[0099] As described above, compute instances within a customer VCN can communicate with a variety of different endpoints. These endpoints may be on the same subnet as the source compute instance, on a different subnet but still within the same VCN, or may include endpoints outside the source compute instance's VCN. These communications are facilitated using the VNIC associated with the compute instance, the VCN VR, and the gateway associated with the VCN.

[0100] Communication between two compute instances on the same subnet within a VCN is facilitated using VNICs associated with the source and destination compute instances. The source and destination compute instances may be hosted on the same host machine or on different host machines. Packets originating from the source compute instance may be forwarded from the host machine hosting the source compute instance to an NVD connected to that host machine. In the NVD, packets are processed using a packet processing pipeline, which may include the execution of the VNIC associated with the source compute instance. Because the destination endpoint of the packets is on the same subnet, the execution of the VNIC associated with the source compute instance forwards the packets to the NVD running the VNIC associated with the destination compute instance, where the NVD processes the packets and forwards them to the destination compute instance. The VNICs associated with the source and destination compute instances may run on the same NVD (for example, if both the source and destination compute instances are hosted on the same host machine) or on different NVDs (for example, if the source and destination compute instances are hosted on different host machines connected to different NVDs). The VNIC can use the routing / forwarding table stored by the NVD to determine the next hop of a packet.

[0101] When a packet is communicated from a compute instance within a subnet to an endpoint in a different subnet within the same VCN, the packet originating from the source compute instance is communicated from the host machine hosting the source compute instance to the NVD connected to that host machine. In the NVD, the packet is processed using a packet processing pipeline and a VR associated with the VCN, which may include the execution of one or more VNICs. For example, as part of the packet processing pipeline, the NVD executes or invokes a function corresponding to the VNIC associated with the source compute instance (also called executing the VNIC). The function executed by the VNIC may include examining the VLAN tag on the packet. Because the packet's destination is outside the subnet, a VCN VR function is invoked and executed by the NVD. The VCN VR then routes the packet to the NVD executing the VNIC associated with the destination compute instance. The VNIC associated with the destination compute instance then processes the packet and forwards it to the destination compute instance. The VNICs associated with the source compute instance and the destination compute instance may run on the same NVD (for example, if both the source compute instance and the destination compute instance are hosted by the same host machine), or they may run on different NVDs (for example, if the source compute instance and the destination compute instance are hosted by different host machines connected to different NVDs).

[0102] If the packet's destination is outside the VCN of the source compute instance, the packet originating from the source compute instance is communicated from the host machine hosting the source compute instance to the NVD connected to that host machine. The NVD runs the VNIC associated with the source compute instance. Because the packet's destination endpoint is outside the VCN, the packet is processed by the VCN VR of that VCN. The NVD invokes VCN VR functionality, which may result in the packet being forwarded to an NVD running the appropriate gateway associated with the VCN. For example, if the destination is an endpoint within the customer's on-premises network, the packet may be forwarded by the VCN VR to an NVD running the DRG gateway configured for the VCN. The VCN VR may run on the same NVD as the NVD running the VNIC associated with the source compute instance, or it may run on a different NVD. The gateway may run on an NVD that is a smart NIC, a host machine, or another NVD implementation. The packet is then processed by the gateway and forwarded to the next hop to facilitate communication of the packet to its intended destination endpoint. For example, in the embodiment shown in Figure 2, a packet originating from compute instance 268 may be communicated from host machine 202 to NVD210 via link 220 (using NIC 232). VNIC 276 on NVD210 is invoked because it is the VNIC associated with source compute instance 268. VNIC 276 is configured to examine the encapsulated information in the packet, determine the next hop for forwarding the packet to facilitate communication of the packet to its intended destination endpoint, and forward the packet to the determined next hop.

[0103] Compute instances deployed on a VCN can communicate with various different endpoints. These endpoints may include endpoints hosted by CSPI200 and endpoints outside of CSPI200. Endpoints hosted by CSPI200 may include instances within the same VCN or other VCNs (which may be customer VCNs or VCNs not belonging to a customer). Communication between endpoints hosted by CSPI200 may be performed over the physical network 218. Compute instances can also communicate with endpoints not hosted by CSPI200 or located outside of CSPI200. Examples of these endpoints include endpoints within the customer's on-premises network or data center, or public endpoints accessible over a public network such as the Internet. Communication with endpoints outside of CSPI200 may be performed over a public network (e.g., the Internet) (not shown in Figure 2) or a private network (not shown in Figure 2) using various communication protocols.

[0104] The architecture of the CSPI200 shown in Figure 2 is merely an example and is not intended to be limiting. Alternative embodiments are possible, and variations, substitutions, and modifications are possible. For example, in some implementations, the CSPI200 may have more or fewer systems or elements than those shown in Figure 2, may combine two or more systems, or may have different system configurations or arrangements. The systems, subsystems, and other elements shown in Figure 2 may be implemented as software (e.g., code, instructions, programs), hardware, or a combination thereof, executed by one or more processing units (e.g., processors, cores) of each system. The software may be stored in a non-temporary storage medium (e.g., a memory device).

[0105] Figure 4 shows a connection between a host machine and an NVD to provide I / O virtualization to support multi-tenancy functionality, according to a particular embodiment. As shown in Figure 4, the host machine 402 runs a hypervisor 404 that provides the virtualization environment. The host machine 402 runs two virtual machine instances, namely VM1 406 belonging to customer / tenant #1 and VM2 408 belonging to customer / tenant #2. The host machine 402 includes a physical NIC 410 connected to the NVD 412 via link 414. Each compute instance is connected to a VNIC run by the NVD 412. In the embodiment of Figure 4, VM1 406 is connected to VNIC-VM1 420 and VM2 408 is connected to VNIC-VM2 422.

[0106] As shown in Figure 4, NIC410 includes two logical NICs, namely logical NIC A 416 and logical NIC B 418. Each virtual machine is connected to its own logical NIC and configured to operate with its own logical NIC. For example, VM1 406 is connected to logical NIC A 416, and VM2 408 is connected to logical NIC B 418. Although the host machine 402 consists of only one physical NIC 410 shared by multiple tenants, the logical NICs allow each tenant's virtual machine to believe that it owns its own host machine and NIC.

[0107] In a particular embodiment, each logical NIC is assigned its own VLAN ID. Thus, logical NIC A 416 for tenant #1 is assigned a specific VLAN ID, and logical NIC B 418 for tenant #2 is assigned a different VLAN ID. When a packet is communicated from VM1 406, the hypervisor attaches the tag assigned to tenant #1 to the packet and then communicates the packet from host machine 402 to NVD412 via link 414. Similarly, when a packet is communicated from VM2 408, the hypervisor attaches the tag assigned to tenant #2 to the packet and then communicates the packet from host machine 402 to NVD412 via link 414. Thus, the packet 424 communicated from host machine 402 to NVD412 has an associated tag 426 that identifies a specific tenant and associated VM. When packet 424 is received from host machine 402 on NVD, the tag 426 associated with the packet is used to determine whether the packet should be processed by VNIC-VM1 420 or VNIC-VM2 422. The packet is then processed by the corresponding VNIC. The configuration shown in Figure 4 allows each tenant's compute instance to believe that it owns its own host machine and NIC. The configuration shown in Figure 4 provides I / O virtualization to support multi-tenancy functionality.

[0108] Figure 5 is a schematic block diagram showing a physical network 500 according to a particular embodiment. The embodiment shown in Figure 5 is constructed as a Clos network. A Clos network is a specific type of network topology designed to provide connectivity redundancy while maintaining high bimodal bandwidth and maximum resource utilization. A Clos network is a type of non-blocking, multi-stage or multi-layer switching network, where the number of stages or layers may be 2, 3, 4, 5, etc. The embodiment shown in Figure 5 is a 3-layer network including layers 1, 2, and 3. A TOR switch 504 represents a layer-0 switch in the Clos network. One or more NVDs are connected to the TOR switch. Layer-0 switches are also called edge devices in the physical network. Layer-0 switches are connected to layer-1 switches, also called leaf switches. In the embodiment shown in Figure 5, "n" layer-0 TOR switches are connected to "n" layer-1 switches to form pods. Each layer-0 switch in a pod is interconnected to all layer-1 switches in the pod, but switches between pods are not connected. In a particular implementation, two pods are referred to as a block. Each block is serviced by or connected to n Layer-2 switches (also called spine switches). The physical network topology may contain multiple blocks. Similarly, the Layer-2 switches are connected to n Layer-3 switches (also called superspine switches). Packet communication over the physical network 500 is typically performed using one or more Layer 3 communication protocols. Typically, all layers of the physical network except the TOR layer are n-way redundant, thus achieving high availability. The physical network can be extended by specifying policies for pods and blocks to control the mutual visibility of switches in the physical network.

[0109] A key feature of Clos networks is that the maximum hop count required to reach one Layer-0 switch from one Layer-0 switch to another (or from an NVD connected to a Layer-0 switch to another NVD connected to a Layer-0 switch) remains constant. For example, in a Layer 3 Clos network, a packet requires a maximum of 7 hops to reach one NVD from another. In this case, the source NVD and target NVD are connected to the leaf layer of the Clos network. Similarly, in a Layer 4 Clos network, a packet requires a maximum of 9 hops to reach one NVD from another. In this case, the source NVD and target NVD are connected to the leaf layer of the Clos network. Therefore, the Clos network architecture maintains a constant overall network latency, which is crucial for communication within and between data centers. Clos topologies are horizontally scalable and cost-effective. Network bandwidth / throughput capacity can be easily increased by adding more switches to each layer (e.g., more leaf and spine switches) and increasing the number of links between switches in adjacent layers.

[0110] In certain embodiments, each resource within the CSPI is assigned a unique identifier called a Cloud Identifier (CID). This identifier is included as part of the resource's information. This identifier can be used to manage the resource, for example, via a console or API. An example syntax for a CID is as follows:

[0111] ocid1.<RESOURCE TYPE> . <realm>[REGION][FUTURE USE]<UNIQUE ID> In the formula, "ocid1" is a string that indicates the CID version.

[0112] "RESOURCE TYPE" represents the type of resource (e.g., instance, volume, VCN, subnet, user, group).

[0113] "REALM" represents the region where the resources reside. Exemplary values ​​include "c1" representing a commercial region, "c2" representing a government cloud region, or "c3" representing a federal government cloud region. Each region can have its own domain name.

[0114] "REGION" represents the region to which the resource belongs. If no region applies to the resource, this section may be left blank.

[0115] "FUTURE USE" indicates that it is reserved for future use. The "UNIQUE ID" is the unique identifier portion. This format may vary depending on the type of resource or service.

[0116] Packet flow based on cached and uncached configuration information Figure 6 shows an example of a network architecture that supports packet flow based on cached configuration information according to one embodiment. This network architecture represents, at least in part, the cloud infrastructure of the CSP. As shown in the figure, the network architecture includes host machine 660, NVD670, board network 680, and host machine 690.

[0117] Generally, host machine 660 hosts one or more compute instances, such as customer compute instance 662, and connects to NVD670. The connection can be an Ethernet® link. NVD670 hosts one or more VNICs, including VNIC672 attached to compute instance 662, among other virtual functions. Typically, a VNIC can be associated with a single compute instance and can provide virtual network interface functionality to the compute instance. Virtual network interface functionality includes providing an interface to a virtual network (e.g., compute instance 662's interface to virtual network 682). NVD670 also connects to the onboard network 680 via a switch (e.g., a top-of-rack (TOR) switch supporting Ethernet® links). The onboard network 680 is also connected to host machine 690, and this connection can be via another NVD (not shown in Figure 6 for brevity). Host machine 690 can host one or more compute instances, including endpoint 692. Compute instance 662 can communicate with endpoint 692 by sending and receiving packets over virtual network 682, for example, via VNIC 672 (and similarly, via a VNIC attached to endpoint 692, also not shown in Figure 6 for simplification). Virtual network 682 may be a software-based or software-defined network overlaid on the underlying network 680. As shown by the dashed box, a customer's VCN600 may include compute instance 662, VNIC 672, and endpoint 692 (although endpoint 692 does not necessarily have to be part of VCN600).

[0118] In one example, the flow of packets to and from a compute instance on host machine 660 may depend on configuration information 674 stored in the memory of NVD670. Configuration information 674 may include one or all of the following: security policies (e.g., firewall rule-based policies for inbound and / or outbound traffic), routing rules, overlay-to-board IP address mappings, routing tables, and other types of information for sending and / or receiving packets. In one example, configuration information 674 may contain multiple sets, each cached for a particular VNIC, associated with a corresponding compute instance, and specific to the information required for the flow of outbound or inbound packets to that compute instance. In this example, when compute instance 662 sends a packet, host machine 660 forwards this packet to NVD670. VNIC 672 then looks up the applicable configuration information to process the packet (e.g., apply rules, update the packet header, etc.) and sends the processed packet over virtual network 682. Sending a packet over virtual network 682 involves NVD670 further processing the packet in a manner transparent to VNIC672 and compute instance 662, and then transmitting it over the underlying network 680. Finally, the packet is received by endpoint 692. Conversely, a packet destined for compute instance 662 is first received by NVD670, processed by VNIC672 based on applicable configuration information, and then sent from NVD670 to host machine 660, where it reaches compute instance 662.

[0119] As described above in this specification, the network architecture in Figure 6 may require a large amount of configuration information to be stored and maintained on the NVD670. Furthermore, the VNICs hosted by the NVD670 also use the processing and memory resources of the NVD670. Therefore, the density of VNICs on the NVD670 (e.g., the number of VNICs that the NVD670 can host) may be relatively limited.

[0120] Figure 7 shows an example of a network architecture that supports packet flow based on uncached configuration information according to one embodiment. This network architecture can generally increase the VNIC density per NVD, but may impact network bandwidth. Instead of the NVD storing the configuration information, the NVD hosts stripped-down versions of VNICs (called micro-VNICs), and the configuration information is stored by a network interface service that provides a centralized network interface service. As shown in the figure, the network architecture includes a host machine 710, an NVD 720, a server fleet 730, an onboard network 740, and a host machine 750.

[0121] Generally, the host machine 710 hosts one or more compute instances, such as the customer's compute instance 712, and is connected to the NVD720. The connection can be an Ethernet® link. The NVD720 hosts one or more VNICs, including VNIC722 which is attached to compute instance 712, among other virtual functions. Unlike the VNICs hosted on the NVD670 in Figure 6, the VNICs hosted on the NVD670 may be micro-VNICs. Typically, micro-VNICs are simpler to implement than VNICs, do not need to use configuration information to attach to compute instances and process packets, and instead may be configured to send outbound packets from compute instances directly to the network interface service 732 on the server fleet 730 via the NVD720, or to send inbound packets directly to compute instances via the NVD720. The NVD720 is connected to the server fleet 730 (e.g., via a tunnel or some other peering mechanism).

[0122] A server fleet 730 may be a set or enclave of servers or host machines that host a network interface service 732 for one or more customers. The network interface service 732 provides network interface functionality for connectivity between compute instances. For example, the network interface service 732 may be a VNIC as a service (VNICaaS). For instance, the network interface service 732 may include multiple VNICs, including VNIC 734 associated with micro VNIC 722. For clarity, these VNICs may be called service VNICs (as opposed to micro VNICs). Typically, a service VNIC may be associated with multiple micro VNICs, each of which is then associated with a single compute instance. In other words, a service VNIC may be associated with multiple compute instances through their corresponding micro VNICs, and may provide virtual network interface functionality to these compute instances. Virtual network interface functionality includes providing interfaces for compute instances to virtual networks (e.g., an interface for compute instance 712 to virtual network 742).

[0123] The server fleet 730 is connected to the board network 740 via a set of switches. The board network 740 is also connected to the host machine 750, and the connection may be via another NVD, the server fleet 730, and / or another server fleet (not shown in Figure 7 for simplification). The host machine 750 may host one or more compute instances, including endpoint 752. Compute instance 712 can communicate with endpoint 752 by sending and receiving packets over virtual network 742, for example, via micro VNIC 722 and service VNIC 734 (and similarly, via a VNIC attached to endpoint 752 or a pair of micro VNIC and service VNIC attached to endpoint 752, also not shown in Figure 7 for simplification). The virtual network 742 may be a software-based or software-defined network that is overlaid on top of the board network 740. As indicated by the dashed box, a customer's VCN760 can include compute instance 712, micro VNIC 722, service VNIC 734, and endpoint 752 (however, endpoint 752 does not necessarily have to be part of VCN760).

[0124] In one example, the flow of packets between the host machine 710 and the compute instance may depend on configuration information 736 stored by the network interface service 732. The configuration information 736 may include any or all of the following: security policies (e.g., firewall rule-based policies for inbound and / or outbound traffic), routing rules, overlay-to-board IP address mappings, routing tables, and other types of information for sending and / or receiving packets. In one example, the configuration information 736 may contain multiple sets, each cached for a service VNIC, a micro VNIC, and / or a compute instance. In this example, when compute instance 712 sends a packet, the host machine 710 forwards this packet to NVD720. Micro VNIC 722 then forwards the packet to the network interface service 732 via NVD720. Server fleet 730 receives the packet from NVD670 via the tunnel and forwards it to the network interface service 732. The network interface service 732 selects service VNIC 734 to further process the packet. Next, the service VNIC 734 looks up applicable configuration information, processes the packet (e.g., applies rules, updates the packet header, etc.), and transmits the processed packet over virtual network 742. Transmitting the packet over virtual network 742 involves the server fleet 730 transmitting the packet over the underlying network 740 in a manner transparent to the service VNIC 734, micro VNIC 722, and compute instance 712. Finally, the packet is received by endpoint 752. Conversely, packets destined for compute instance 712 are processed by the service VNIC 734 based on applicable configuration information, transmitted to micro VNIC 722 via server fleet 730 and NVD 720, and then transmitted from micro VNIC 722 to compute instance 712 via NVD 720 and host machine 710.

[0125] As described above in this specification, the network architecture in Figure 7 may require packets to flow through the network interface service 732 to and from the host machine 710. This flow can affect network bandwidth, especially as the volume of packets increases.

[0126] Figure 8 shows an example of a network architecture that supports packet flow based on cached flow information and uncached configuration information according to one embodiment. Here, the network architecture can optimize the balance between memory usage (and VNIC density on the NVD) and network bandwidth. In one example, a network interface service can maintain configuration information that is available to manage the flow of packets. The NVD also hosts the VNICs and can store only a relevant subset of the configuration information, referred to herein as flow information. This subset may be the minimum or reduced amount of configuration information required to send and / or receive packets directly from the virtual network without requiring the use of the network interface service. Only if this subset is unavailable or outdated does the packet flow need to be accompanied by the network interface service. As shown in the figure, the network architecture includes a host machine 810, an NVD 820, a server fleet 830, an underlay network 840, and a host machine 850.

[0127] Generally, the host machine 810 hosts one or more compute instances, such as the customer's compute instance 812, and connects to the NVD820. The connection can be an Ethernet® link. The NVD820 hosts one or more VNICs, including VNIC822 attached to compute instance 812, among other virtual functions. The NVD820 also stores flow information 824 that can be used by such VNICs to send and / or receive packets on one or more virtual networks, such as virtual network 842. The NVD820 is connected to the server fleet 830 (e.g., via a tunnel or some other peering mechanism). The NVD820 is also connected to the board network 840 (e.g., via a TOR switch).

[0128] The server fleet 830 may be a set or enclave of servers or host machines that host the network interface service 832 for one or more customers. The network interface service 832 provides network interface functionality for connectivity between compute instances. For example, the network interface service 832 may be a VNICaaS. For instance, the network interface service 832 includes multiple service VNICs, including VNIC 822 and service VNIC 834 associated with compute instance 812. These service VNICs rely on configuration information 836 stored by the network interface service 832 to process and transmit packets.

[0129] Server fleet 830 is connected to the board network 840 via a set of switches. Board network 840 is also connected to host machine 850, and the connection may be via another NVD, server fleet 830, and / or another server fleet (not shown in Figure 8 for simplification). Host machine 850 may host one or more compute instances, including endpoint 852. Compute instance 812 can communicate with endpoint 852, for example, by sending and receiving packets over virtual network 842 via VNIC 822 and / or service VNIC 834 (and similarly, via VNICs attached to endpoint 852 or other service VNICs, also not shown in Figure 8 for simplification). Virtual network 842 may be a software-based or software-defined network overlaid on top of board network 840. As indicated by the dashed box, a customer's VCN860 can include compute instance 812, VNIC 822, service VNIC 834, and endpoint 852 (although endpoint 852 may not be part of VCN860).

[0130] In one example, the flow of packets to and from the compute instance of host machine 810 may depend on configuration information 836 stored by network interface service 832 and flow information 824 stored by NVD820. Configuration information 836 may include any or all of security policies (e.g., firewall rule-based policies for inbound and / or outbound traffic), routing rules, overlay-to-board IP address mappings, routing tables, and other types of information for sending and / or receiving packets. In comparison, flow information 824 may contain a subset of configuration information 836, which is the minimum or reduced amount of configuration information required to process and manage the flow of packets through NVD820. For example, flow information 824 may include one or more security policies, one or more overlay-to-board IP address mappings, one or more routing tables, and / or one or more aggregations thereof, or a combination thereof. Flow information 824 can be granular to a specific resolution, such as packet type (for example, it can indicate rewrite rules applicable to a particular packet type) or it can be comprehensive to a higher resolution level (for example, it can apply to multiple packet types and indicate how packets should be processed, for example, given a firewall rule). Generally, because flow information 824 is specific to the flow of packets to and from one or more compute instances on a host machine 810 connected to the NVD820, the amount of flow information 824 is smaller than the amount of configuration information 836, while configuration information 836 is applicable to packet flows to and / or from a much larger number of compute instances hosted on a large number of host machines. In one example, configuration information 836 may contain multiple sets, each set cached for a service VNIC, a VNIC attached to a compute instance, and / or that compute instance.In this example, the flow information 824 can contain multiple sets, each set being cached for a VNIC attached to a particular compute instance and / or for that compute instance.

[0131] For illustrative purposes, consider two scenarios. In the first scenario, compute instance 812 sends a packet to endpoint 852, and flow information 824 is available to support this transmission. In this scenario, the packet is sent directly over virtual network 842 by VNIC 822 (as shown by the upper dotted line), and correspondingly sent directly over the underlying network 840 by NVD 820 (as shown by the lower solid line). In the second alternative scenario, flow information 824 is unavailable. In this case, the packet is processed by service VNIC 834 (as shown by the lower dotted line) and sent over virtual network 842, and correspondingly sent over the underlying network 840 by server fleet 830 (as shown by the upper solid line).

[0132] In the first scenario, when compute instance 812 sends a packet, host machine 810 forwards this packet to NVD820. VNIC822 then processes the packet by looking up applicable flow information (e.g., applying rules and updating the packet header) and sends the processed packet over virtual network 842. Sending the packet over virtual network 842 involves NVD820 further processing the packet in a manner transparent to VNIC822 and compute instance 812 and sending it over the underlying network 840. Finally, the packet is received by endpoint 852.

[0133] In the second scenario, when compute instance 812 sends a packet, host machine 810 forwards this packet to NVD820. VNIC822 then determines that there is no available flow information to process this packet and instead sends the packet to network interface service 832 via NVD820. Server fleet 830 receives the packet from NVD820 via the tunnel and forwards it to network interface service 832. Network interface service 832 selects service VNIC834 to further process the packet. Service VNIC834 then processes the packet by looking up applicable configuration information (e.g., applying rules and updating the packet header) and sends the processed packet over virtual network 842. Sending the packet over virtual network 842 involves server fleet 830 sending the packet over the underlying network 840 in a manner transparent to service VNIC834, VNIC822, and compute instance 812. Finally, the packet is received by endpoint 852.

[0134] As described above in this specification, with respect to the network architecture of Figure 6, the NVD820 stores less configuration information (e.g., flow information 824), thereby enabling a higher density of VNICs. With respect to the network architecture of Figure 7, packet flows are accompanied only by network interface services 832 as needed (e.g., when flow information is not available), thereby reducing the bandwidth burden. The aspects of generating, distributing, updating, and managing packet flows of flow information are further illustrated in the following figures.

[0135] As described above in this specification, the packet flow on the virtual network includes not only the overlay resources but also the corresponding packet flow on the underlying network via the hardware resources hosting the overlay resources. In the following diagram, and for clarity and brevity, details regarding the packet flow on the underlying network may be omitted when describing the packet flow on the virtual network, and such details may be implied by referring to the above description.

[0136] Figure 9 shows an example of an outbound packet flow resulting in cached flow information according to one embodiment. NVD910, hosting VNIC912, is connected to server fleet 920, hosting network interface service 922. Initially, NVD910 does not store flow information that would allow VNIC912 to directly transmit packets over the virtual network. Instead, packets are transmitted indirectly over the virtual network via network interface service 922 (e.g., via a specific service VNIC). NVD910 then receives and stores applicable flow information from network interface service 922, thereby enabling VNIC912 to transmit packets directly over the virtual network from its point of origin, as further illustrated in Figure 10. NVD910, VNIC912, server fleet 920, and network interface service 922 are examples of NVD820, VNIC822, server fleet 830, and network interface service 832, respectively. In Figure 9, the numbers shown in circles refer to sequences of events.

[0137] In one example, VNIC912 receives a packet from, for example, the compute instance to which VNIC912 is attached, and this packet is sent over the virtual network (for example, an outbound packet). Given the header information in the packet, VNIC912 looks up the cache 914 of NVD910 to determine if cached flow information is available to process the packet. This lookup can use n-tuple matching of source information, destination information, source subnet, destination subnet, packet type, etc. The result of the lookup is a cache miss. Therefore, VNIC912 sends the packet to the network interface service 922. Then, given configuration information 924, the network interface service 922 processes the packet (for example, by applying rules, rewriting header information, performing encapsulation, etc.) and sends the processed packet over the underlying network. Furthermore, the network interface service 922 generates a cache entry containing the flow information and the version information associated with that flow information. Flow information can be a subset of specific configuration information used to transmit packets on a board number. For example, flow information may include at least one of a security policy, an overlay-to-board IP address mapping, or a route rule. Version information may indicate the current version of the flow information or the current version of a part of the flow information. For example, version information may be an indicator of the current version of the flow information, or an indicator of the current version of each part of the flow information (e.g., the current version of the security policy, the current version of the overlay-to-board IP address mapping, and the current version of the route rule). Version information can be used to invalidate cache entries, as further illustrated in the following diagram. Network interface service 922 sends the cache entry to NVD910. NVD910 then stores the flow information 916 and version information 918 from the received cache entry as entries in cache 914.

[0138] Figure 10 shows an example of an outbound packet flow based on cached flow information according to one embodiment. The example in Figure 10 follows the example in Figure 9. NVD1010, which hosts VNIC1012, is connected to server fleet 1020, which hosts network interface service 1022. NVD1010 stores flow information 1016 and version information 1018 in cache 1014. In comparison, network interface service 1022 stores configuration information 1024. NVD1010, VNIC1012, cache 1014, flow information 1016, version information 1018, server fleet 1020, network interface service 1022, and configuration information 1024 are examples of NVD910, VNIC912, cache 914, flow information 916, version information 918, server fleet 920, network interface service 922, and configuration information 924, respectively.

[0139] In one example, VNIC1012 receives a packet from, for example, the compute instance to which VNIC1012 is attached, and this packet is sent over the virtual network (for example, it is an outbound packet). Given the header information in the packet, VNIC1012 looks up cache 1014 to determine if cached flow information is available to process the packet. The lookup results in a cache hit with flow information 1016. Version information 1018 indicates that flow information 1016 is valid. Therefore, VNIC1012 sends the packet directly over the virtual network without having to send the packet via network interface service 1022.

[0140] Figure 11 shows an example of an inbound packet flow resulting in cached flow information according to one embodiment. While Figure 9 illustrates generating and caching flow information based on outbound traffic flows, the example in Figure 11 illustrates this process based on inbound traffic flows. NVD1110, hosting VNIC1112, is connected to server fleet 1120, hosting network interface service 1122. Initially, NVD1110 does not store flow information that would allow VNIC1112 to receive packets directly from the virtual network, as further illustrated in Figure 12. Instead, packets are further processed by network interface service 1122 (e.g., via a specific service VNIC). Subsequently, NVD1110 receives the processed packets from network interface service 1122, stores applicable flow information, and thereby enables VNIC1112 to send packets directly over the virtual network. NVD1110, VNIC1112, Server Fleet 1120, and Network Interface Service 1122 are examples of NVD820, VNIC822, Server Fleet 830, and Network Interface Service 832, respectively.

[0141] In one example, VNIC1112 receives an inbound packet on a virtual network. Given the header information in the packet, VNIC1112 looks up the cache 1114 on NVD1110 to determine if cached flow information is available to process the packet. The result of the lookup is a cache miss. Therefore, VNIC1112 sends the packet to network interface service 1122. Then, given configuration information 1124, network interface service 1122 processes the packet (e.g., applies rules, rewrites header information, performs encapsulation, etc.), sends the processed packet back to VNIC1112, and VNIC1112 then sends the packet to the compute instance to which VNIC1112 is attached. In addition, network interface service 1122 generates a cache entry containing flow information and version information associated with that flow information. Flow information may be a subset of specific configuration information used to send packets on the underlying network. For example, flow information includes at least one of a security policy, an overlay-to-board IP address mapping, or a route rule. Version information can indicate the current version of the flow information or the current version of a part of the flow information. For example, version information is an indicator of the current version of the flow information, or an indicator of the current version of each part of the flow information (e.g., the current version of the security policy, the current version of the overlay-to-board IP address mapping, and the current version of the route rule). Version information can be used to invalidate cache entries, as further illustrated in the following diagram. The network interface service 1122 sends the cache entry to the NVD 1110. The NVD 1110 then stores the flow information 1116 and version information 1118 from the received cache entry as entries in the cache 1114.

[0142] As explained above in relation to Figure 11, the receiving NVD (e.g., NVD1110) caches flow information based on inbound traffic. Variations may exist. For example, upon receiving a packet from a VNIC, the network interface service sends the relevant flow information to both the receiving and transmitting NVDs for caching there. For example, referring again to Figures 9 and 11, suppose the transmitting NVD is NVD910 and the receiving NVD is NVD1110. When VNIC912 sends a packet (e.g., via transmitting NVD910), the network interface service (e.g., network interface service 922 or 1122) updates both transmitting NVD910 and receiving NVD1110.

[0143] Furthermore, as described in relation to Figures 9 and 11, in the event of a cache miss, the packet triggers the network interface service to send a cache entry containing the necessary flow information. Variations of this can also exist. For example, the flow information may be included in the header of a packet sent to the NVD by the network interface service, or in the header of a packet sent from the sending NVD to the receiving NVD. In a scenario where the network interface service updates both the receiving and sending NVDs, different flow information may be included in different packets (e.g., flow information in a packet sent to the receiving NVD, and flow information in a response packet sent to the sending NVD). In one exemplary variation, the sending NVD includes the flow information in the receiving NVD's header for subsequent use by the receiving NVD. In this example, the receiving NVD may not have an incoming security policy. Instead, a signature may be included in the header and used by the receiving NVD to authenticate that the flow information was sent by an authorized entity.

[0144] Figure 12 shows an example of an inbound packet flow based on cached flow information according to one embodiment. The example in Figure 12 follows the example in Figure 11. NVD1210, which hosts VNIC1212, is connected to server fleet 1220, which hosts network interface service 1222. NVD1210 stores flow information 1216 and version information 1218 in cache 1214. In comparison, network interface service 1222 stores configuration information 1224. NVD1210, VNIC1212, cache1214, flow information1216, version information1218, server fleet1220, network interface service1222, and configuration information1224 are examples of NVD1110, VNIC1112, cache1114, flow information1116, version information1118, server fleet1120, network interface service1122, and configuration information1124, respectively.

[0145] In one example, VNIC1212 receives an inbound packet on the virtual network. Given the header information in the packet, VNIC1212 looks up cache 1214 to determine if cached flow information is available to process the packet. The lookup results in a cache hit with flow information 1216. Version information 1218 indicates that flow information 1216 is valid. Therefore, VNIC1212 sends the packet to the compute instance to which VNIC1212 is attached, without needing to send this packet to network interface service 1222 for further processing.

[0146] Figure 13 shows an example of invalidating and updating cached flow information according to one embodiment. In this example, invalidation can be managed by a network interface service, and updates can follow a proactive strategy (e.g., by implementing a push mechanism, the updated flow information is sent to the relevant VNIC). Alternatively, updates can follow a passive strategy (e.g., by implementing a pull mechanism, the updated flow information is sent to the VNIC upon an update request from the VNIC). NVD 1310, hosting VNIC 1312, is connected to server fleet 1320, which hosts network interface service 1322. Network interface service 1322 is connected to control plane 1330. A customer's computing device (not shown in Figure 13) can connect to control plane 1330 (e.g., via an Application Programming Interface (API) call) to update the customer's VCN configuration. The update can result in changes to configuration information available for managing packet flow.

[0147] In one example, the network interface service 1322 stores configuration information 1324 and version information 1326 associated with configuration information 1324. Configuration information 1324 may contain multiple parts (e.g., security policies, overlay-to-board IP address mappings, route rules). Version information 1326 may contain an indicator of the current version of the entire configuration information 1324 and / or an indicator of the current version of each part of the configuration information 1324 (e.g., the current version of the security policy, the current version of the overlay-to-board IP address mappings, and the current version of the route rules). To clarify the explanation, the following paragraph will describe the use of version indicators for each part of the configuration information 1324. However, this process also applies when a single version indicator is used for the entire configuration information 1324.

[0148] The network interface service 1322 sends an update request to the control plane 1330 requesting an update to the configuration information 1324. For example, the update request may include a vector clock with a popularized version indicator. The control plane 1330 determines which of the version indicators is outdated. If none of them are outdated, no update to the configuration information 1324 is required. If, however, one of the version indicators is outdated, the corresponding portion of the configuration information 1324 is outdated, and the control plane 1330 sends the replacement configuration information portion along with the updated version indicator. For example, if the version indicator for a security policy indicates that it is still the latest version, but the version indicator for an overlay-to-board IP address mapping is outdated, the control plane 1330 simply sends the updated overlay-to-board IP address mapping and its updated version indicator.

[0149] Next, the network interface service 1322 receives the update from the control plane 1330. The network interface service 1322 tracks the VNICs using the flow information portion corresponding to the old configuration information portion, generates the updated flow information portion and the corresponding version indicator, and pushes the updated flow information portion and the corresponding version indicator to the NVD hosting the VNIC. For example, the NVD 1310 receives the updated flow information 1316 and the updated version information 1318 and stores them in its cache 1314 for use by the VNIC 1312.

[0150] As described above in this specification, a passive strategy can be used instead. Here, instead of pushing updates to the NVD, the network interface service 1322 notifies the affected VNICs that their flow information (or parts thereof) is outdated and sends updates only upon request. For example, upon receiving notification that flow information 1316 (or parts thereof) is outdated, VNIC 1312 can request an update for it from the network interface service 1322. In response, the network interface service 1322 sends the updated flow information (or parts thereof) and corresponding version information.

[0151] Figure 14 shows another example of invalidating and updating cached flow information according to one embodiment. In this example, invalidation can be managed by the NVD, and updates can follow a proactive strategy (e.g., by requesting an update before receiving a packet to which the flow information applies). Alternatively, updates can follow a passive strategy (e.g., by sending a packet to a network interface service for further processing and then receiving the updated flow information). The NVD 1410, hosting the VNIC 1412, is connected to a server fleet 1420, which hosts a network interface service 1422. The network interface service 1422 is connected to a control plane 1430. A customer's computing device (not shown in Figure 14) can connect to the control plane 1430 and update the configuration of the customer's VCN. The update can result in changes to configuration information available for managing the flow of packets.

[0152] In one example, the NVD1410 stores flow information 1416 and version information 1418 associated with flow information 1416 in cache 1414. Flow information 1416 can include multiple parts (e.g., security policy, overlay-to-board IP address mapping, route rule). Version information 1418 can include an indicator of the current version of the entire flow information 1416 and / or an indicator of the current version of each part of the flow information 1416 (e.g., the current version of the security policy, the current version of the overlay-to-board IP address mapping, and the current version of the route rule). To clarify the explanation, the following paragraph will describe the use of a version indicator for each part of the flow information 1416. However, this process also applies when a single version indicator is used for the entire flow information 1416.

[0153] The NVD1410 sends an update request to the control plane 1430 requesting an update to the flow information 1416. For example, the update request may include a vector clock with a popularized version indicator. The control plane 1430 determines which of the version indicators is outdated. If none of them are outdated, no update to the flow information 1416 is required. If, however, one of the version indicators is outdated, the corresponding portion of the flow information 1416 is outdated, and the control plane 1430 sends a notification to the NVD1410 indicating that an update is available. This notification may indicate which portion of the flow information 1416 is outdated.

[0154] In the passive strategy, NVD1410 does not immediately update flow information 1416. Instead, VNIC1412 receives the packet and determines that flow information 1416 is applicable to it (for example, based on a cache hit). The VNIC also determines that version information 1418 is outdated (for example, part of flow information 1416 is outdated according to a notification from control plane 1430). Alternatively, upon receiving the notification, NVD1410 invalidates the cached flow information 1416 so that if a cache lookup is performed, the result of the lookup will be a cache miss. In both cases, VNIC1412 sends the packet to network interface service 1422 for further processing. This packet transmission represents a passive update request. Network interface service 1422 processes the packet using configuration information 1424 with the latest version information 1426 (which can be updated from control plane 1430 in a similar manner to Figure 13). Furthermore, the network interface service 1422 sends an update to the NVD 1410, which includes the updated flow information (e.g., the entire flow information, not just a portion of it) and the corresponding version indicator. The NVD 1410 then replaces the existing flow information 1416 (or its older portion) and version information 1418 (or the older version indicator) with the updated flow information portion and the corresponding version indicator.

[0155] For example, only the overlay-to-board IP mapping is outdated in flow information 1416. NVD1410 receives a notification from control plane 1430 indicating that the corresponding version indicator is outdated. VNIC1412 then sends the packet, which would otherwise have been sent directly over the virtual network, to network interface service 1422. In response, network interface service 1422 sends the updated flow information and the corresponding version indicator. NVD1410 replaces the outdated overlay-to-board IP address mapping in flow information 1416 and the corresponding version indicator in version information 1418 with the updated overlay-to-board IP address mapping and the corresponding version indicator, respectively.

[0156] As described above in this specification, a proactive strategy can be used instead. Here, instead of sending a packet to the network interface service 1422 and then receiving the update back, the NVD 1410 can automatically send an update request to the network interface service 1422. This request may indicate which part of the flow information 1416 is outdated, and the network interface service 1422 may send only the updated portion. Alternatively, no such instruction may be given, and the network interface service 1422 may send the entire flow information.

[0157] Figures 15–20 illustrate examples of methods for distributing configuration information and managing packet flow. The operation of these methods may be performed by a system including an NVD and / or network service interface. Some or all of the instructions for performing the operations may be implemented as hardware circuits and / or stored as computer-readable instructions on a non-temporary computer-readable medium of the system. When implemented, the instructions represent a module containing code executable by the circuit or the system's processor. The use of such instructions configures the system to perform the specific operations described herein. Each circuit or code, combined with the relevant processor, represents a means for performing its respective operation. While the operations are presented in a specific order, it should be understood that this order is not required, and one or more operations may be omitted, skipped, executed in parallel, and / or reordered.

[0158] Figure 15 shows an example of a method for sending a packet that yields cached flow information, according to one embodiment. Here, the packet is an outbound packet sent over a virtual network. In one example, the method includes operation 1502, in which a VNIC hosted on an NVD receives the packet. The packet may be sent from a compute instance to which the VNIC is attached. In operation 1504, the VNIC determines whether the flow information is available from the NVD's memory to process the packet. For example, the VNIC performs a cache lookup based on information from the packet's header. If the lookup yields a cache hit, the flow information is available. In this case, operation 1510 follows operation 1504. Otherwise, operation 1520 follows operation 1504. In operation 1510, the VNIC directly sends the packet over the virtual network. For example, the VNIC processes the packet according to the flow information and then sends the processed packet over the virtual network (for example, with updated header information). Transmitting over a virtual network may involve the NVD further processing the packet for transmission over the underlying infrastructure network. In operation 1520, there is no flow information available for the NVD to process the packet locally (e.g., not cached). Instead, the VNIC sends the packet to a network interface service that stores configuration information available for processing the packet. In operation 1522, the NVD receives the flow information. This flow information may be a subset of the configuration information used by the network interface service (e.g., by the service VNIC hosted there) to process the packet and transmit it over the virtual network. In addition, version information associated with the flow information may be received. In operation 1524, the NVD stores the flow information and, if applicable, the version information in memory (e.g., as a cache entry). The flow may be repeated to transmit additional packets over the virtual network, which are received from compute instances.

[0159] Figure 16 shows an example of a method for receiving a packet that yields cached flow information, according to one embodiment. Here, the packet is an inbound packet received on a virtual network. In one example, the method includes operation 1602, in which a VNIC hosted on an NVD receives the packet. The packet can be sent from an endpoint on the virtual network and is destined for a compute instance to which the VNIC is attached. In operation 1604, the VNIC determines whether the flow information is available from the NVD's memory to process the packet. For example, the VNIC performs a cache lookup based on information from the packet's header. If the lookup yields a cache hit, the flow information is available. In this case, operation 1610 follows operation 1604. Otherwise, operation 1620 follows operation 1604. In operation 1610, the VNIC sends the packet directly on the compute instance. For example, the VNIC processes the packet according to the flow information and then sends the processed packet (along with updated header information, for example) to the compute instance via the NVD and the compute instance's host machine. In operation 1620, there is no flow information available to the NVD for processing the packet locally (e.g., it is not cached). Instead, the VNIC sends the packet to a network interface service that stores configuration information available for processing the packet. In operation 1622, the NVD receives the packet processed by the network interface service along with the flow information. This flow information may be a subset of the configuration information used by the network interface service (e.g., by the service VNIC hosted there) to process the packet. In addition, version information associated with the flow information may be received. In operation 1624, the NVD stores the flow information and, if applicable, the version information in memory (e.g., as a cache entry). The received packet is also sent directly to the compute instance, as shown in the loop from operation 1624 to operation 1610.The flow is repeated to send additional packets to the compute instance, and these additional packets are received on the virtual network.

[0160] Figure 17 shows an example of a method for generating and transmitting flow information according to one embodiment. Here, the method can be implemented by a network interface service. In one example, the method includes operation 1702 in which the network interface service receives a packet. In one example, the packet is an outbound packet originating from a compute instance and transmitted over the virtual network. In another example, the packet is an inbound packet received on the virtual network and destined for the compute instance. In both examples, the network interface service can receive the packet from the VNIC via the NVD hosting the VNIC, and the NVD does not store (cached) flow information that enables the VNIC to process the packet. In operation 1704, the network interface service (e.g., the service VNIC hosted therein) determines the configuration information to process the packet. This determination can be based on the packet's header information. In operation 1706, the network interface service (e.g., the service VNIC) transmits the packet based on the configuration information. Referring back to the first example, the service VNIC processes the packet according to the configuration information and transmits the processed packet over the virtual network. In the second example, the service VNIC processes packets according to the configuration information but sends the processed packets to the NVD's VNIC. In operation 1708, the network interface service generates flow information. For example, flow information is a subset of the configuration information that allows the NVD's VNIC to process packets of the same type or similar packets locally. This flow information can correspond to the latest configuration information and can be associated with version information. In operation 1710, the network interface service sends the flow information to the NVD. Furthermore, version information can be sent to the NVD.

[0161] Figure 18 shows an example of a method for invalidating and updating cached flow information according to one embodiment, where the method can be implemented by an NVD hosting a VNIC. In one example, the method includes operation 1802 in which the VNIC receives a packet. In one example, the packet is an outbound packet originating from a compute instance and sent over the virtual network. In another example, the packet is an inbound packet received over the virtual network and destined for the compute instance. In operation 1804, the VNIC determines whether the flow information is old, where the flow information is available from the NVD's memory to process the packet. For example, the VNIC determines whether the version information of the flow information is old (or some version indicators of the flow information are old). This determination may include sending a vector clock to the control plane and receiving a notification about updated version information. Although operation 1804 is shown following operation 1802, operation 1804 can be performed independently of operation 1804, and the cached flow information is invalidated before the packet is received. If the flow information is not outdated, operation 1810 follows operation 1804. Otherwise, operation 1820 follows operation 1804. In operation 1810, the VNIC sends the packet directly on the virtual network for outbound traffic, or directly to the compute instance for inbound traffic. In operation 1820, no flow information is available that is valid for processing the packet locally at the NVD. Instead, the VNIC sends the packet to a network interface service that stores configuration information available for processing the packet. In operation 1822, the NVD receives the flow information (and, in the case of inbound traffic, the packet). This flow information may be a subset of the configuration information used by the network interface service (e.g., by the service VNIC hosted there) to process the packet and send it back to the VNIC on the overlay board. In addition, version information associated with the flow information may be received.In operation 1824, NVD stores flow information and, if applicable, version information in memory (for example, as cache entries). The flow may be repeated to send additional packets, which are received from compute instances or virtual networks.

[0162] Figure 19 shows another example of a method for invalidating and updating cached flow information according to one embodiment, where this method can be implemented by a network interface service. In one example, the method includes operation 1902 in which the network interface service determines that configuration information is outdated. For example, the network interface service determines that the version information of the configuration information (or some version indicators of the flow information) is outdated. This determination may include sending a vector clock to the control plane, receiving notification of updated version information, and / or receiving the updated configuration information (or the updated portion thereof). In operation 1904, the network interface service determines which VNICs need to be updated based on the update from the control plane. For example, the network interface service tracks the corresponding configuration information and its associated version information for each VNIC on the NVD. If the update affects the configuration information used for the VNIC, the network interface service determines that this VNIC should be updated. In operation 1906, the network interface service sends the updated flow information, and optionally the version information, to the affected VNIC. Updated flow information can be generated as a subset of updated configuration information. The entire updated flow information can be pushed, or alternatively, only the updated portion can be pushed along with the latest version indicator.

[0163] Figure 20 shows another example of a method for invalidating and updating cached flow information according to one embodiment. Here, the method can be implemented by a network interface service and completes the update according to a passive strategy. In one example, the method includes operation 2002, in which the network interface service determines that the configuration information is outdated, similar to operation 1902. In operation 2004, the network interface service determines the VNIC that needs to be updated, similar to operation 1904. In operation 2006, the network interface service indicates to the VNIC that its flow information is outdated (or part of it is outdated). For example, it may send a notification that results in the invalidation of the corresponding cached entry. In operation 2008, the network interface service receives a packet from the VNIC. The packet may be an inbound or outbound packet. In operation 2010, the network interface service processes and transmits the packet using the updated configuration information. For outbound traffic, the packet is transmitted over the virtual network (for example, via the service VNIC). For inbound traffic, the packet is sent back to the VNIC. In Operation 2012, the Network Interface Service sends updated flow information and, optionally, version information to the VNIC. The updated flow information may be generated as a subset of the updated configuration information. The entire updated flow information can be sent, or alternatively, only the updated portion can be sent along with the latest version indicator.

[0164] Exemplary Infrastructure as a Service Architecture As mentioned above, IaaS (Infrastructure as a Service) is a specific type of cloud computing. IaaS may be configured to provide virtualized computing resources over a public network (e.g., the internet). In the IaaS model, a cloud computing provider can host infrastructure elements (e.g., servers, storage, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer)). In some cases, the IaaS provider can provide various services associated with the infrastructure elements (e.g., billing, monitoring, logging, security, load balancing, and clustering). Therefore, since these services can be policy-driven, IaaS users can implement policies to drive load balancing in order to maintain application availability and performance.

[0165] In some cases, IaaS customers can access resources and services over a wide area network (WAN), such as the internet, and install the rest of their application stack using the cloud provider's services. For example, a user can log into an IaaS platform, create virtual machines (VMs), install an operating system (OS) on each VM, deploy middleware such as databases, create storage buckets for workloads and backups, and install enterprise software on the VMs. Customers can use the provider's services to perform a variety of functions, including balancing network traffic, troubleshooting applications, monitoring performance, and managing disaster recovery.

[0166] In most cases, the cloud computing model requires the participation of a cloud provider. This cloud provider may or may not be a third-party service specializing in IaaS provision (e.g., offering, renting, or selling). Alternatively, a company can become a provider of private clouds and infrastructure services.

[0167] In some cases, IaaS deployment is the process of deploying a new application or a new version of an application to a pre-configured application server. IaaS deployment may also include the process of preparing the server (e.g., installing libraries, daemons, etc.). IaaS deployment is often managed by the cloud provider under the hypervisor layer (e.g., servers, storage, network hardware, and virtualization). Therefore, customers can deploy the OS, middleware, and / or applications (e.g., self-service virtual machines, which can be spun up on demand).

[0168] In some cases, IaaS provisioning may include acquiring the computers or virtual hosts to be used and installing the necessary libraries or services on those computers or virtual hosts. In most cases, deployment does not include provisioning, and provisioning must be performed first.

[0169] In some cases, IaaS provisioning presents two distinct challenges. First, there's the challenge of provisioning an initial set of infrastructure before doing anything. Second, there's the challenge of evolving existing infrastructure after everything has been provisioned (e.g., adding new services, modifying services, removing services). In some cases, these two challenges can be addressed by enabling the declarative definition of infrastructure configuration. In other words, the infrastructure (e.g., what elements are needed and how these elements interact) may be defined by one or more configuration files. Thus, the overall topology of the infrastructure (e.g., which resources depend on which and how they work together) can be described declaratively. In some examples, once the topology is defined, workflows can be generated to create and / or manage the different elements described in the configuration files.

[0170] In some examples, infrastructure can include many interconnected elements. For example, there may be one or more virtual private clouds (VPCs), also known as core networks (e.g., configurable compute resources and / or potential on-demand pools of shared compute resources). In some examples, there may be one or more security group rules provisioned to define how network security is configured, and one or more virtual machines (VMs). Other infrastructure elements such as load balancers and databases may also be provisioned. As more and more infrastructure elements are desired and / or added, infrastructure can evolve incrementally.

[0171] In some examples, sequential deployment techniques may be employed to enable the deployment of infrastructure code across various virtual computing environments. Furthermore, the techniques described can enable infrastructure management within these environments. In some examples, a service team may write code that is intended to be deployed to one or more different production environments, typically many different geographical locations, sometimes across the globe. However, in some examples, the infrastructure for deploying the code must first be configured. In some examples, provisioning can be done manually, resources can be provisioned using provisioning tools, and / or the code can be deployed using deployment tools after the infrastructure has been provisioned.

[0172] Figure 21 is a block diagram 2100 illustrating an exemplary pattern of an IaaS architecture according to at least one embodiment. The service operator 2102 may be communicably coupled to a secure host tenancy 2104 which may include a virtual cloud network (VCN) 2106 and a secure host subnet 2108. In some examples, the service operator 2102 may use one or more client computing devices. One or more client computing devices may be handheld mobile devices (e.g., iPhone®, mobile phones, iPad®, tablets, personal digital assistants (PDAs) or wearable devices (Google® Glass® head-mounted displays)) with Internet, email, short message service (SMS), BlackBerry®, or other communication protocols enabled, and which can run software such as Microsoft Windows Mobile® and / or various mobile operating systems such as iOS, Windows Phone, Android®, BlackBerry 8, and Palm OS. The client computing devices may also be general-purpose personal computers, including, exemplarily, personal computers and / or laptop computers, which run various versions of the Microsoft Windows® operating system, Apple Macintosh® operating system, and / or Linux® operating system. Alternatively, the client computing devices may be workstation computers running various commercially available UNIX® or UNIX-like operating systems, including, but not limited to, various GNU / Linux operating systems and, for example, Google Chrome® OS.Alternatively or additionally, the client computing device may be other electronic devices that can communicate via a network that has access to the VCN2106 and / or the Internet, such as a thin client computer, an Internet-enabled game system (e.g., a Microsoft Xbox® game console with or without a Kinect® gesture input device), and / or a personal messaging device.

[0173] VCN2106 may include a local peering gateway (LPG) 2110 that can communicately connect to Secure Shell (SSH) VCN2112 via LPG2110 included in SSH VCN2112. SSH VCN2112 may include an SSH subnet 2114, and SSH VCN2112 can communicately connect to control plane VCN2116 via LPG2110 included in control plane VCN2116. Furthermore, SSH VCN2112 can communicately connect to data plane VCN2118 via LPG2110. Control plane VCN2116 and data plane VCN2118 may be included in a service tenancy 2119, which may be owned and / or operated by an IaaS provider.

[0174] The control plane VCN2116 may include a control plane DMZ (demilitarized zone) layer 2120 that functions as a perimeter network (e.g., the portion of the corporate network between the corporate intranet and the external network). DMZ-based servers have a certain level of reliability and can contain security breaches. Furthermore, the DMZ layer 2120 may include a control plane application layer 2124 that may include one or more load balancer (LB) subnets 2122 and application subnets 2126, and a control plane data layer 2128 that may include database (DB) subnets 2130 (e.g., a front-end DB subnet and / or a back-end DB subnet). The LB subnet 2122 included in the control plane DMZ layer 2120 may be communicatively coupled to the application subnet 2126 included in the control plane application layer 2124 and the internet gateway 2134 which may be included in the control plane VCN 2116. The application subnet 2126 may be communicatively coupled to the DB subnet 2130 included in the control plane data layer 2128, the service gateway 2136, and the network address translation (NAT) gateway 2138. The control plane VCN 2116 may include the service gateway 2136 and the NAT gateway 2138.

[0175] The control plane VCN 2116 may include a data plane mirror application layer 2140, which may include an application subnet 2126. The application subnet 2126 included in the data plane mirror application layer 2140 may include a virtual network interface controller (VNIC) 2142 on which compute instance 2144 can run. Compute instance 2144 can communicatively connect the application subnet 2126 of the data plane mirror application layer 2140 to an application subnet 2126 that may be included in the data plane application layer 2146.

[0176] The data plane VCN2118 may include a data plane application layer 2146, a data plane DMZ layer 2148, and a data plane data layer 2150. The data plane DMZ layer 2148 may include an LB subnet 2122 that can be communicatively coupled to the application subnet 2126 of the data plane application layer 2146 and the internet gateway 2134 of the data plane VCN2118. The application subnet 2126 may also be communicatively coupled to the service gateway 2136 of the data plane VCN2118 and the NAT gateway 2138 of the data plane VCN2118. Additionally, the data plane data layer 2150 may include a DB subnet 2130 that can be communicatively coupled to the application subnet 2126 of the data plane application layer 2146.

[0177] The Internet gateway 2134 of the control plane VCN2116 and the Internet gateway 2134 of the data plane VCN2118 may be communicatively coupled to a metadata management service 2152 which can be communicatively coupled to the public internet 2154. The public internet 2154 may be communicatively coupled to the NAT gateway 2138 of the control plane VCN2116 and the NAT gateway 2138 of the data plane VCN2118. The service gateway 2136 of the control plane VCN2116 and the service gateway 2136 of the data plane VCN2118 may be communicatively coupled to a cloud service 2156.

[0178] In some cases, a service gateway 2136 of the control plane VCN2116 or data plane VCN2118 can make application programming interface (API) calls to a cloud service 2156 without going through the public internet 2154. API calls from the service gateway 2136 to the cloud service 2156 can be one-way. The service gateway 2136 can make API calls to the cloud service 2156, and the cloud service 2156 can send request data to the service gateway 2136. However, the cloud service 2156 may not initiate an API call to the service gateway 2136.

[0179] In some examples, a secure host tenancy 2104 may be directly connected to a potentially isolated service tenancy 2119. A secure host subnet 2108 can communicate with an SSH subnet 2114 via an LPG 2110, which enables bidirectional communication with the isolated system. By connecting the secure host subnet 2108 to the SSH subnet 2114, the secure host subnet 2108 can access other entities within the service tenancy 2119.

[0180] The control plane VCN2116 allows users of service tenancy 2119 to configure or provision desired resources. Desired resources provisioned in the control plane VCN2116 may be deployed or used in the data plane VCN2118. In some examples, the control plane VCN2116 may be isolated from the data plane VCN2118, and the data plane mirror application layer 2140 of the control plane VCN2116 can communicate with the data plane application layer 2146 of the data plane VCN2118 via a VNIC 2142 which may be included in the data plane mirror application layer 2140 and the data plane application layer 2146.

[0181] In some examples, a user or customer of the system may make requests, such as create, read, update, or delete (CRUD) operations, via the public internet 2154, which can communicate the requests to the metadata management service 2152. The metadata management service 2152 can communicate the requests to the control plane VCN 2116 via the internet gateway 2134. The requests may also be received by an LB subnet 2122 included in the control plane DMZ layer 2120. The LB subnet 2122 may determine that the request is valid, and in response to this determination, it may send the request to an application subnet 2126 included in the control plane application layer 2124. If the request is validated and requires a call to the public internet 2154, the call to the public internet 2154 can be sent to a NAT gateway 2138, which can make calls to the public internet 2154. Memory for storing the requests may be stored in the DB subnet 2130.

[0182] In some cases, the data plane mirror application layer 2140 can facilitate direct communication between the control plane VCN2116 and the data plane VCN2118. For example, it may be desirable that changes, updates, or other appropriate modifications to the configuration be applied to the resources contained in the data plane VCN2118. Since the control plane VCN2116 can communicate directly with the resources contained in the data plane VCN2118 via VNIC2142, it can perform changes, updates, or other appropriate modifications to the configuration.

[0183] In some embodiments, the control plane VCN2116 and the data plane VCN2118 may be included in the service tenancy 2119. In this case, the system user or customer does not have to own or operate either the control plane VCN2116 or the data plane VCN2118. Instead, the IaaS provider may own or operate the control plane VCN2116 and the data plane VCN2118, and both may be included in the service tenancy 2119. This embodiment can prevent a user or customer from interacting with other users' resources or other customers' resources by enabling network isolation. This embodiment can also enable a system user or customer to store databases privately without having to rely on the public internet 2154, which may not have the desired level of security for storage.

[0184] In another embodiment, the LB subnet 2122 included in the control plane VCN2116 may be configured to receive signals from the service gateway 2136. In this embodiment, the control plane VCN2116 and the data plane VCN2118 may be configured to be invoked by the IaaS provider's customers without calling the public internet 2154. The IaaS provider's customers may prefer this embodiment because the database used by the customer may be stored in a service tenancy 2119 that is controlled by the IaaS provider and can be isolated from the public internet 2154.

[0185] Figure 22 is a block diagram 2200 showing another exemplary parameter of an IaaS architecture according to at least one embodiment. A service operator 2202 (e.g., service operator 2102 in Figure 21) may be communicably coupled to a secure host tenancy 2204 (e.g., secure host tenancy 2104 in Figure 21), which may include a virtual cloud network (VCN) 2206 (e.g., VCN2106 in Figure 21) and a secure host subnet 2208 (e.g., secure host subnet 2108 in Figure 21). VCN2206 may include a local peering gateway (LPG) 2210 (e.g., LPG2110 in Figure 21), which may be communicably coupled to a secure shell (SSH) VCN2212 (e.g., SSH VCN2112 in Figure 21) via an LPG 2110 contained in an SSH VCN2212. SSH VCN2212 may include SSH subnet 2214 (e.g., SSH subnet 2114 in Figure 21), and SSH VCN2212 may be communicably coupled to control plane VCN2216 (e.g., control plane VCN2116 in Figure 21) via LPG2210 included in control plane VCN2216. Control plane VCN2216 may be included in service tenancy 2219 (e.g., service tenancy 2119 in Figure 21), and data plane VCN2218 (e.g., data plane VCN2118 in Figure 21) may be included in customer tenancy 2221, which may be owned or operated by a user or customer of the system.

[0186] The control plane VCN2216 may include a control plane DMZ tier 2220 (e.g., control plane DMZ tier 2120 in Figure 21) which may include an LB subnet 2222 (e.g., LB subnet 2122 in Figure 21), a control plane application tier 2224 (e.g., control plane application tier 2124 in Figure 21) which may include an application subnet 2226 (e.g., application subnet 2126 in Figure 21), and a control plane data tier 2228 (e.g., control plane data tier 2128 in Figure 21) which may include a database (DB) subnet 2230 (e.g., similar to DB subnet 2130 in Figure 21). The LB subnet 2222 included in the control plane DMZ tier 2220 may be coupled to communicate with the application subnet 2226 included in the control plane application tier 2224 and with an internet gateway 2234 (e.g., internet gateway 2134 in Figure 21) which may be included in the control plane VCN2216. The application subnet 2226 may be communicatively coupled to the DB subnet 2230, service gateway 2236 (e.g., the service gateway in Figure 21), and network address translation (NAT) gateway 2238 (e.g., the NAT gateway 2138 in Figure 21), which are included in the control plane data layer 2228. The control plane VCN 2216 may include the service gateway 2236 and the NAT gateway 2238.

[0187] The control plane VCN2216 may include a data plane mirror application layer 2240 (e.g., the data plane mirror application layer 2140 in Figure 21) which may include an application subnet 2226. The application subnet 2226 included in the data plane mirror application layer 2240 may include a virtual network interface controller (VNIC) 2242 (e.g., VNIC2142) which can run a compute instance 2244 (e.g., similar to compute instance 2144 in Figure 21). The compute instance 2244 can facilitate communication between the application subnet 2226 of the data plane mirror application layer 2240 and the application subnet 2226 that may be included in the data plane application layer 2246 (e.g., the data plane application layer 2146 in Figure 21) via the VNIC2242 included in the data plane mirror application layer 2240 and the VNIC2242 included in the data plane application layer 2246.

[0188] The Internet gateway 2234 included in the control plane VCN2216 may be communicably coupled to a metadata management service 2252 (e.g., metadata management service 2152 in Figure 21), which can be communicably coupled to the public internet 2254 (e.g., public internet 2154 in Figure 21). The public internet 2254 may be communicably coupled to a NAT gateway 2238 included in the control plane VCN2216. The service gateway 2236 included in the control plane VCN2216 may be communicably coupled to a cloud service 2256 (e.g., cloud service 2156 in Figure 21).

[0189] In some examples, the data plane VCN2218 may be included in customer tenancy 2221. In this case, the IaaS provider can provide a control plane VCN2216 for each customer, and the IaaS provider can configure a unique compute instance 2244 for each customer, included in service tenancy 2219. Each compute instance 2244 can allow communication between the control plane VCN2216 included in service tenancy 2219 and the data plane VCN2218 included in customer tenancy 2221. The compute instance 2244 can allow resources provisioned in the control plane VCN2216 included in service tenancy 2219 to be deployed to or used in the data plane VCN2218 included in customer tenancy 2221.

[0190] In another example, an IaaS provider's customer may have a database residing in customer tenancy 2221. In this example, control plane VCN 2216 may include a data plane minor application tier 2240 that can include application subnet 2226. A data plane mirror application tier 2240 may reside in data plane VCN 2218, but does not have to reside in data plane VCN 2218. That is, a data plane mirror application tier 2240 can access customer tenancy 2221, but does not have to reside in data plane VCN 2218 and does not have to be owned or operated by an IaaS provider's customer. A data plane mirror application tier 2240 may be configured to make calls to data plane VCN 2218, but does not have to be configured to make calls to any entity contained in control plane VCN 2216. The customer may wish to deploy or use resources within the data plane VCN2218 provisioned to the control plane VCN2216, and the data plane mirror application layer 2240 can facilitate the customer's desired deployment or other use of resources.

[0191] In some embodiments, a customer of the IaaS provider can apply filters to the data plane VCN2218. In this embodiment, the customer can determine what the data plane VCN2218 can access and can restrict access from the data plane VCN2218 to the public internet 2254. The IaaS provider may not be able to apply filters or control access from the data plane VCN2218 to any external network or database. Applying filters and controls to the data plane VCN2218 included in the customer tenancy 2221 can help isolate the data plane VCN2218 from other customers and the public internet 2254.

[0192] In some embodiments, the cloud service 2256 can be invoked by the service gateway 2236 to access services that may not reside on the public internet 2254, on the control plane VCN 2216, or on the data plane VCN 2218. The connection between the cloud service 2256 and the control plane VCN 2216 or data plane VCN 2218 does not have to be live or continuous. The cloud service 2256 may reside on a separate network owned or operated by the IaaS provider. The cloud service 2256 may be configured to receive calls from the service gateway 2236 and not to receive calls from the public internet 2254. Some cloud services 2256 may be isolated from other cloud services 2256, and the control plane VCN 2216 may be isolated from cloud services 2256 that may not be located in the same region as the control plane VCN 2216. For example, the control plane VCN 2216 may be located in "Region 1", and the cloud service "Deployment 21" may be located in "Region 1" and "Region 2". If a call to deployment 21 is made by a service gateway 2236 included in the control plane VCN2216 located in region 1, this call may be sent to deployment 21 in region 1. In this example, the control plane VCN2216 or deployment 21 in region 1 does not need to be communicatively coupled with deployment 21 in region 2.

[0193] Figure 23 is a block diagram 2300 showing another exemplary pattern of an IaaS architecture according to at least one embodiment. A service operator 2302 (e.g., service operator 2102 in Figure 21) may be communicably coupled to a secure host tenancy 2304 (e.g., secure host tenancy 2104 in Figure 21), which may include a virtual cloud network (VCN) 2306 (e.g., VCN2106 in Figure 21) and a secure host subnet 2308 (e.g., secure host subnet 2108 in Figure 21). VCN2306 may include an LPG2310 (e.g., LPG2110 in Figure 21), which may be communicably coupled to an SSH VCN2312 (e.g., SSH VCN2112 in Figure 21) via an LPG2310 contained in the SSH VCN2312. SSH VCN2312 may include SSH subnet 2314 (e.g., SSH subnet 2114 in Figure 21), and SSH VCN2312 may be communicatively coupled to control plane VCN2316 (e.g., control plane VCN2116 in Figure 21) via LPG2310 included in control plane VCN2316, and may be communicatively coupled to data plane VCN2318 (e.g., data plane 2118 in Figure 21) via LPG2310 included in data plane VCN2318. Control plane VCN2316 and data plane VCN2318 may be included in service tenancy 2319 (e.g., service tenant 2119 in Figure 21).

[0194] The control plane VCN2316 may include a control plane DMZ layer 2320 (e.g., control plane DMZ layer 2120 in Figure 21) which may include a load balancer (LB) subnet 2322 (e.g., LB subnet 2122 in Figure 21), a control plane application layer 2324 (e.g., control plane application layer 2124 in Figure 21) which may include an application subnet 2326 (e.g., similar to application subnet 2126 in Figure 21), and a control plane data layer 2328 (e.g., control plane data layer 2128 in Figure 21) which may include a DB subnet 2330. The LB subnet 2322 included in the control plane DMZ layer 2320 may be communicably coupled to the application subnet 2326 included in the control plane application layer 2324 and to an internet gateway 2334 (e.g., internet gateway 2134 in Figure 21) which may be included in the control plane VCN2316. The application subnet 2326 may be communicatively coupled to the DB subnet 2330 included in the control plane data layer 2328, and to the service gateway 2336 (e.g., the service gateway in Figure 21) and the network address translation (NAT) gateway 2338 (e.g., the NAT gateway 2138 in Figure 21). The control plane VCN 2316 may include the service gateway 2336 and the NAT gateway 2338.

[0195] The data plane VCN2318 may include a data plane application layer 2346 (e.g., data plane application layer 2146 in Figure 21), a data plane DMZ layer 2348 (e.g., data plane DMZ layer 2148 in Figure 21), and a data plane data layer 2350 (e.g., data plane data layer 2150 in Figure 21). The data plane DMZ layer 2348 may include an LB subnet 2322 that can be communicatively coupled to the trusted application subnet 2360 and untrusted application subnet 2362 of the data plane application layer 2346 and Internet gateway 2334 included in the data plane VCN2318. The trusted application subnet 2360 may be communicatively coupled to the service gateway 2336 included in the data plane VCN2318, the NAT gateway 2338 included in the data plane VCN2318, and the DB subnet 2330 included in the data plane data layer 2350. The untrusted application subnet 2362 may be communicatively coupled to the service gateway 2336 included in the data plane VCN 2318 and to the DB subnet 2330 included in the data plane data layer 2350. The data plane data layer 2350 may include the DB subnet 2330, which can be communicatively coupled to the service gateway 2336 included in the data plane VCN 2318.

[0196] An untrusted application subnet 2362 may include one or more primary VNICs 2364(1)-(N) that can be communicatively coupled to tenant virtual machines (VMs) 2366(1)-(N). Each tenant VM 2366(1)-(N) may be communicatively coupled to each application subnet 2367(1)-(N) that can be included in each container transmission VCN 2368(1)-(N) that can be included in each customer tenancy 2370(1)-(N). Each secondary VNIC 2372(1)-(N) can facilitate communication between the untrusted application subnet 2362 included in the data plane VCN 2318 and the application subnets included in the container transmission VCN 2368(1)-(N). Each container transmission VCN 2368(1)-(N) may include a NAT gateway 2338 that can be communicatively coupled to the public internet 2354 (e.g., public internet 2154 in Figure 21).

[0197] The Internet gateway 2334 included in the control plane VCN2316 and the Internet gateway 2334 included in the data plane VCN2318 may be communicatively coupled to a metadata management service 2352 (e.g., the metadata management system 2152 in Figure 21), which can be communicatively coupled to the public internet 2354. The public internet 2354 may be communicatively coupled to the NAT gateway 2338 included in the control plane VCN2316 and the NAT gateway 2338 included in the data plane VCN2318. The service gateway 2336 included in the control plane VCN2316 and the service gateway 2336 included in the data plane VCN2318 may be communicatively coupled to a cloud service 2356.

[0198] In some embodiments, the data plane VCN2318 may be integrated with the customer tenancy 2370. This integration may be useful or desirable for the IaaS provider's customer in some cases, such as when they may want support when executing code. The customer may provide code that, when executed, could be destructive, could communicate with other customer resources, or could cause undesirable effects. Thus, the IaaS provider can determine whether or not to execute the code that the customer has provided to the IaaS provider.

[0199] In some examples, an IaaS provider's customer may grant the IaaS provider temporary network access and request functionality to be added to the data plane application layer 2346. The code to perform the functionality may run on VMs 2366(1)-(N), but cannot be configured to run elsewhere on the data plane VCN 2318. Each VM 2366(1)-(N) may be connected to one customer tenancy 2370. Each container 2371(1)-(N) contained within VMs 2366(1)-(N) may be configured to run the code. In this case, a double isolation may exist (for example, containers 2371(1)-(N) run the code, and containers 2371(1)-(N) may be contained within VMs 2366(1)-(N) that are at least in an untrusted application subnet 2362), which can help prevent erroneous or undesirable code from damaging the IaaS provider's network or the networks of different customers. Containers 2371(1)-(N) may be communicatively coupled to customer tenancy 2370 and may be configured to send or receive data from customer tenancy 2370. Containers 2371(1)-(N) do not have to be configured to send or receive data from any other entities in the data plane VCN2318. Once code execution is complete, the IaaS provider may kill or discard containers 2371(I)-(N).

[0200] In some embodiments, a trusted application subnet 2360 may execute code that may be owned or operated by the IaaS provider. In this embodiment, the trusted application subnet 2360 may be communicatively coupled to the DB subnet 2330 and configured to perform CRUD operations in the DB subnet 2330. An untrusted application subnet 2362 may be communicatively coupled to the DB subnet 2330, but in this embodiment, the untrusted application subnet may be configured to perform read operations within the DB subnet 2330. Containers 2371(1)~(N) contained in each customer's VM2366(1)~(N) and capable of executing code from the customer do not need to be communicatively coupled to the DB subnet 2330.

[0201] In other embodiments, the control plane VCN2316 and the data plane VCN2318 do not have to be directly coupled in a communicative manner. In this embodiment, direct communication between the control plane VCN2316 and the data plane VCN2318 may not exist. However, indirect communication by at least one method may exist. An LPG2310 that facilitates communication between the control plane VCN2316 and the data plane VCN2318 may be established by the IaaS provider. In another example, the control plane VCN2316 or the data plane VCN2318 can make a call to the cloud service 2356 via the service gateway 2336. For example, a call from the control plane VCN2316 to the cloud service 2356 may include a request for a service that can communicate with the data plane VCN2318.

[0202] Figure 24 is a block diagram 2400 showing another exemplary parameter of an IaaS architecture according to at least one embodiment. A service operator 2402 (e.g., service operator 2102 in Figure 21) may be communicatively coupled to a secure host tenancy 2404 (e.g., secure host tenancy 2104 in Figure 21), which may include a virtual cloud network (VCN) 2406 (e.g., VCN2106 in Figure 21) and a secure host subnet 2408 (e.g., secure host subnet 2108 in Figure 21). VCN2406 may include an LPG2410 (e.g., LPG2110 in Figure 21), which may be communicatively coupled to an SSH VCN2412 (e.g., SSH VCN2112 in Figure 21) via an LPG2410 contained in an SSH VCN2412. SSH VCN2412 may include SSH subnet-2414 (e.g., SSH subnet 2114 in Figure 21), and SSH VCN2412 may be communicatively coupled to control plane VCN2416 (e.g., control plane VCN2116 in Figure 21) via LPG2410 included in control plane VCN2416, and may be communicatively coupled to data plane VCN2418 (e.g., data plane 2118 in Figure 21) via LPG2410 included in data plane VCN2418. Control plane VCN2416 and data plane VCN2418 may be included in service tenancy 2419 (e.g., service tenancy 2119 in Figure 21).

[0203] The control plane VCN2416 may include a control plane DMZ layer 2420 (e.g., control plane DMZ layer 2120 in Figure 21) which may include an LB subnet 2422 (e.g., LB subnet 2122 in Figure 21), a control plane application layer 2424 (e.g., control plane application layer 2124 in Figure 21) which may include an application subnet 2426 (e.g., application subnet 2126 in Figure 21), and a control plane data tier 2428 (e.g., control plane data tier 2128 in Figure 21) which may include a DB subnet 2430 (e.g., DB subnet 2330 in Figure 23). The LB subnet 2422 included in the control plane DMZ layer 2420 may be communicably coupled to the application subnet 2426 included in the control plane application layer 2424 and to an internet gateway 2434 (e.g., internet gateway 2134 in Figure 21) which may be included in the control plane VCN2416. The application subnet 2426 may be communicatively coupled to the DB subnet 2430, which is included in the control plane data layer 2428, and to the service gateway 2436 (e.g., the service gateway in Figure 21) and the network address translation (NAT) gateway 2438 (e.g., the NAT gateway 2138 in Figure 21). The control plane VCN 2416 may include the service gateway 2436 and the NAT gateway 2438.

[0204] The data plane VCN2418 may include a data plane application layer 2446 (e.g., data plane application layer 2146 in Figure 21), a data plane DMZ layer 2448 (e.g., data plane DMZ layer 2148 in Figure 21), and a data plane data layer 2450 (e.g., data plane data layer 2150 in Figure 21). The data plane DMZ layer 2448 may include trusted application subnets 2460 (e.g., trusted application subnet 2360 in Figure 23) and untrusted application subnets 2462 (e.g., untrusted application subnet 2362 in Figure 23) of the data plane application layer 2446, and an LB subnet 2422 that can be communicatively coupled to the internet gateway 2434 included in the data plane VCN2418. A trusted application subnet 2460 may be communicatively coupled to a service gateway 2436 included in data plane VCN2418, a NAT gateway 2438 also included in data plane VCN2418, and a DB subnet 2430 included in data plane data layer 2450. An untrusted application subnet 2462 may be communicatively coupled to a service gateway 2436 included in data plane VCN2418, and a DB subnet 2430 included in data plane data layer 2450. The data plane data layer 2450 may include a DB subnet 2430 that can be communicatively coupled to a service gateway 2436 included in data plane VCN2418.

[0205] An untrusted application subnet 2462 may include primary YNICs 2464(1)-(N) that can be communicatively coupled to tenant virtual machines (VMs) 2466(1)-(N) residing in the untrusted application subnet 2462. Each tenant VM 2466(1)-(N) may execute code in its respective container 2467(1)-(N) and may be communicatively coupled to an application subnet 2426 that may be included in a data plane application layer 2446 that may be included in a container-transmitting VCN 2468. Each secondary VNIC 2472(1)-(N) can facilitate communication between the untrusted application subnet 2462 included in a data plane VCN 2418 and the application subnet included in a container-transmitting VCN 2468. The container-transmitting VCN may include a NAT gateway 2438 that can be communicatively coupled to the public internet 2454 (e.g., public internet 2154 in Figure 21).

[0206] The Internet gateway 2434 included in the control plane VCN2416 and data plane VCN2418 may be communicatively coupled to a metadata management service 2452 (e.g., the metadata management system 2152 in Figure 21), which can be communicatively coupled to the public internet 2454. The public internet 2454 may be communicatively coupled to a NAT gateway 2438 included in the control plane VCN2416 and data plane VCN2418. The service gateway 2436 included in the control plane VCN2416 and data plane VCN2418 may be communicatively coupled to a cloud service 2456.

[0207] In some examples, the pattern shown by the architecture in block diagram 2400 of Figure 24 can be considered an exception to the pattern shown by the architecture in block diagram 2300 of Figure 23, and may be desirable for the IaaS provider's customers when the IaaS provider cannot communicate directly with the customers (e.g., in a disconnected area). Customers can access in real time each container 2467(1)~(N) contained within each customer's VM2466(1)~(N). Containers 2467(1)~(N) may be configured to call each secondary VNIC 2472(1)~(N) contained within the application subnet 2426 of the data plane application layer 2446, which may be contained within the container sending VCN 2468. Secondary VNICs 2472(1)~(N) can send calls to a NAT gateway 2438 which can send calls to the public internet 2454. In this example, the containers 2467(1)-(N) that customers can access in real time may be isolated from the control plane VCN2416 and from other entities included in the data plane VCN2418. Furthermore, the containers 2467(1)-(N) may be isolated from other customers' resources.

[0208] In another example, a customer can use containers 2467(1)-(N) to invoke cloud service 2456. In this example, the customer can execute code in containers 2467(1)-(N) to request a service from cloud service 2456. Containers 2467(1)-(N) can then send this request to secondary VNICs 2472(1)-(N), which can send the request to a NAT gateway that can send the request to the public internet 2454. The public internet 2454 can then send this request to LB subnet 2422, which is included in control plane VCN 2416, via internet gateway 2434. In response to determining that the request is valid, the LB subnet can send this request to application subnet 2426, which can then send this request to cloud service 2456 via service gateway 2436.

[0209] The illustrated IaaS architectures 2100, 2200, 2300, and 2400 may include elements other than those shown. Furthermore, the illustrated embodiments are only examples of some cloud infrastructure systems that may incorporate embodiments of this disclosure. In some other embodiments, the IaaS system may have more or fewer elements than those shown, may combine two or more elements, or may have different configurations or arrangements of elements.

[0210] In certain embodiments, the IaaS systems described herein may include a suite of applications, middleware, and database services delivered to customers in a self-service, subscription-based, flexibly scalable, reliable, highly available, and secure manner. An example of such an IaaS system is Oracle® Cloud Infrastructure (OCI), provided by the applicant.

[0211] Figure 25 shows an exemplary computer system 2500 in which various embodiments may be implemented. System 2500 may be used to implement any of the computer systems described above. As shown, computer system 2500 includes a processing unit 2504 that communicates with a number of peripheral subsystems via a bus subsystem 2502. These peripheral subsystems may include a processing acceleration unit 2506, an I / O subsystem 2508, a storage subsystem 2518, and a communication subsystem 2524. The storage subsystem 2518 includes a tangible computer-readable storage medium 2522 and system memory 2510.

[0212] The bus subsystem 2502 provides a mechanism for various components and subsystems of the computer system 2500 to communicate with each other as intended. Although the bus subsystem 2502 is schematically shown as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses. The bus subsystem 2502 may be one of several types of bus structures, including a memory bus or memory controller, a peripheral bus, and a local bus using any of the various bus architectures. For example, such architectures may include the Industry Standard Architecture (ISA) bus, Microchannel Architecture (MCA), Bus Extension ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Interconnect (PCI) bus, which can be implemented as a mezzanine bus manufactured in accordance with the IEEE P1386.1 standard.

[0213] A processing unit 2504, which can be implemented as one or more integrated circuits (e.g., conventional microprocessors or microcontrollers), controls the operation of the computer system 2500. The processing unit 2504 may include one or more processors. These processors may include single-core or multi-core processors. In some embodiments, the processing unit 2504 may be implemented as one or more independent processing units 2532 and / or 2534, each containing a single-core or multi-core processor. In other embodiments, the processing unit 2504 may be implemented as a quad-core processing unit formed by integrating two dual-core processors onto a single chip.

[0214] In various embodiments, the processing unit 2504 can execute various programs in response to program code and can maintain multiple programs or processes running simultaneously. At any given time, some or all of the program code being executed can reside in the processor 2504 and / or the memory subsystem 2518. The processor 2504 can provide the various functionalities described above through appropriate programming. The computer system 2500 may further include a processing acceleration unit 2506 which may include a digital signal processor (DSP), a dedicated processor and / or the same.

[0215] The I / O subsystem 2508 may include user interface input devices and user interface output devices. User interface input devices may include pointing devices such as keyboards, mice or trackballs, touchpads or touchscreens integrated into displays, scroll wheels, click wheels, dials, buttons, switches, keypads, voice input devices with voice command recognition systems, microphones, and other types of input devices. User interface input devices may also include motion detection and / or gesture recognition devices, such as the Microsoft Kinect® motion sensor. The Microsoft Kinect® motion sensor can control and interact with input devices such as Microsoft Xbox® 360 game controllers via a natural user interface (NUI) that utilizes gestures and voice commands. User interface input devices may also include eye gesture recognition devices, such as the Google Glass® blink detector. The Google Glass® blink detector detects the user's eye activity (e.g., blinking when taking a picture and / or selecting a menu) and converts the eye activity into input to an input device (e.g., Google Glass®). Furthermore, the user interface input device may include a voice recognition detection device that enables interaction between the user and a voice recognition system (e.g., Siri® Navigator) via voice commands.

[0216] Furthermore, user interface input devices include, but are not limited to, three-dimensional (3D) mice, joysticks or pointing sticks, gamepads, graphic tablets, audio / visual devices such as speakers, digital cameras, digital video cameras, portable media players, webcams, image scanners, fingerprint scanners, barcode readers, 3D scanners, 3D printers, laser rangefinders, and eye-tracking devices. In addition, user interface input devices may include medical image input devices such as computed tomography scanners, magnetic resonance imaging scanners, ultrasound imaging scanners, or medical ultrasound devices. Furthermore, user interface input devices may include audio input devices such as MIDI keyboards and electronic musical instruments.

[0217] The user interface output device may include non-visual displays such as display subsystems, indicator lights, or audio output devices. The display subsystem may be, for example, a flat-panel display using a cathode ray tube (CRT), liquid crystal display (LCD), or plasma display, a projection device, or a touchscreen. Generally, when the term “output device” is used, it is intended to include all possible types of devices and mechanisms for outputting information from the computer system 2500 to a user or another computer. For example, the user interface output device includes, but is not limited to, various display devices that visually convey text, images, and audio / video information, such as monitors, printers, speakers, headphones, car navigation systems, plotters, audio output devices, and modems.

[0218] The computer system 2500 may include a storage subsystem 2518. The storage subsystem 2518 comprises software elements, which, in the illustration, are located in the system memory 2510. The system memory 2510 can store program instructions that can be loaded and executed by the processing unit 2504, and data generated by the execution of these programs.

[0219] Depending on the configuration and type of the computer system 2500, the system memory 2510 may be volatile memory (e.g., random access memory: RAM) and / or non-volatile memory (e.g., read-only memory: ROM, flash memory). Generally, RAM contains data and / or program modules that are readily accessible to the processing unit 2504, and / or data and / or program modules currently being operated and executed by the processing unit 2504. In some implementations, the system memory 2510 may include several different types of memory, such as static random access memory (SRAM) or dynamic random access memory (DRAM). In some implementations, a basic input / output system (BIOS), which includes basic routines that help transfer information between elements within the computer system 2500 during startup and other times, may generally be stored in ROM. As an example and not limited thereto, system memory 2510 also represents application programs 2512, program data 2514, and operating systems 2516, which may include client applications, web browsers, middle-tier applications, relational database management systems (RDBMS), etc.For example, Operating System 2516 may include various versions of Microsoft Windows®, Apple Macintosh®, and / or Linux® operating systems, various commercially available UNIX® or UNIX-like operating systems (including, but not limited to, various GNU / Linux operating systems, Google Chrome® OS, etc.), and / or mobile operating systems such as iOS, Windows® Phone, Android® OS, BlackBerry® OS, and Palm® OS.

[0220] Furthermore, the storage subsystem 2518 can provide a tangible, computer-readable storage medium for storing basic programming and data structures that provide the functionality of several embodiments. Software (programs, code modules, instructions) that provides the above functionality when executed by the processor may be stored in the storage subsystem 2518. These software modules or instructions may be executed by the processing unit 2504. The storage subsystem 2518 can also provide a repository for storing data used in accordance with this disclosure.

[0221] Furthermore, the storage subsystem 2500 may include a computer-readable storage medium reader 2520 that can be further connected to the computer-readable storage medium 2522. The computer-readable storage medium 2522, together with the system memory 2510, or in combination with the system memory 2510 as needed, can comprehensively represent remote storage devices, local storage devices, fixed storage devices, and / or removable storage devices, in addition to storage media for temporarily and / or permanently storing, storing, transmitting, and retrieving computer-readable information.

[0222] Furthermore, the computer-readable storage medium 2522 containing code or a portion of code may include any suitable medium known or used in the art. Such medium includes, but is not limited to, volatile and non-volatile, removable and non-removable media, which are implemented in any way or technique for storing and / or transmitting information. This may include tangible computer-readable storage media such as RAM, ROM, electronically erasable programmable ROM (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile disk (DVD), or other optical storage devices, magnetic cassettes, magnetic tapes, magnetic disk storage devices or other magnetic storage devices, or other tangible computer-readable media. It may also include intangible computer-readable media such as data signals, data transmissions, or other media that can be used to transmit desired information and are accessible by the computer system 2500.

[0223] For example, the computer-readable storage medium 2522 may include a hard disk drive that reads from or writes to a non-removable non-volatile magnetic medium, a magnetic disk drive that reads from or writes to a removable non-volatile magnetic disk, and an optical disk drive that reads from or writes to a removable non-volatile optical disk such as a CD-ROM, DVD, or Blu-ray® disc or other optical medium. The computer-readable storage medium 2522 may include, but is not limited to, a Zip® drive, a flash memory card, a Universal Serial Bus (USB) flash drive, a Secure Digital (SD) card, a DVD disc, a digital videotape, and the like. Furthermore, the computer-readable storage medium 2522 may include flash memory-based SSDs, enterprise flash drives, solid-state drives (SSDs) based on non-volatile memory such as solid-state ROM, SSDs based on volatile memory such as solid-state RAM, dynamic RAM, and static RAM, DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs that use a combination of DRAM and flash memory-based SSDs. Disk drives and their associated computer-readable media can provide computer-readable instructions, data structures, program modules, and other data as non-volatile storage devices to the computer system 2500.

[0224] The communication subsystem 2524 provides interfaces with other computer systems and networks. It acts as an interface for receiving data from other systems and transmitting data from computer system 2500 to other systems. For example, the communication subsystem 2524 may enable computer system 2500 to connect to one or more devices via the Internet. In some embodiments, the communication subsystem 2524 may include radio frequency (RF) transceiver components for accessing wireless voice and / or data networks (using, for example, cellular technologies such as 3G, 4G, or EDGE (enhanced data rates for global evolution), advanced data network technologies), WiFi (IEEE 802.11 family standards or other mobile communication technologies or any combination thereof), global positioning system (GPS) receiver components, and / or other components. In some embodiments, the communication subsystem 2524 may provide wired network connectivity (e.g., Ethernet) in addition to, or instead of, the wireless interface.

[0225] In addition, in some embodiments, the communication subsystem 2524 can receive input communications on behalf of one or more users who may be using the computer system 2500, in the form of structured and / or unstructured data feeds 2526, event streams 2528, event updates 2530, etc.

[0226] For example, the communications subsystem 2524 may be configured to receive data feeds 2526 in real time from users of social networks and / or other communications services, such as Twitter® feeds, Facebook® updates, and Rich Site Summary (RSS) feeds, and / or to receive real-time updates from one or more third-party sources.

[0227] Furthermore, the communication subsystem 2524 may be configured to receive data in the form of a continuous data stream, which may include an event stream 2528 and / or event update 2530 of real-time events, which may be continuous or have no boundaries in an essentially definite-end state. Examples of applications that generate continuous data may include, for example, sensor data applications, financial tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, and automotive traffic monitoring.

[0228] The communication subsystem 2524 may also be configured to output structured and / or unstructured data feeds 2526, event streams 2528, event updates 2530, etc., to one or more databases that can communicate with one or more streaming data source computers coupled to the computer system 2500.

[0229] The computer system 2500 may be one of a variety of devices, including handheld portable devices (e.g., iPhone® mobile phones, iPad® computing tablets, PDAs), wearable devices (e.g., Google Glass® head-mounted displays), PCs, workstations, mainframes, kiosks, server racks, or other data processing systems.

[0230] In the preceding description, specific details are provided for illustrative purposes to enable a full understanding of the embodiments of this disclosure. However, it will be apparent that various embodiments can be carried out without these specific details. The following description provides examples only and is not intended to limit the scope, applicability, or configuration of this disclosure. Rather, the following description of embodiments provides a possible description for carrying out the embodiments to those skilled in the art. It should be understood that various modifications can be made to the function and arrangement of elements without departing from the spirit and scope of this disclosure as set forth in the appended claims. The drawings and descriptions are not intended to be restrictive. Circuits, systems, networks, processes, and other components may be shown as components in the form of block diagrams so as not to obscure the embodiments with unnecessary details. In other embodiments, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary details so as not to obscure the embodiments. The teachings of this disclosure can also be applied to various types of applications, such as mobile applications, non-mobile applications, desktop applications, web applications, and enterprise applications. Furthermore, the teachings in this disclosure are not limited to a specific operating environment (e.g., operating system, device, platform, etc.) but can be applied to multiple different operating environments.

[0231] It should also be noted that each embodiment is described as a process shown as a flowchart, flow diagram, data flow diagram, structure diagram, or block diagram. While flowcharts describe operations as sequential processes, many operations can be performed in parallel or simultaneously. Furthermore, the order of operations may be rearranged. A process terminates when its operations are completed, but it may include additional steps not shown in the diagram. A process can correspond to a method, function, procedure, subroutine, subprogram, etc. If a process corresponds to a function, its termination can correspond to the return of the calling function or the main function.

[0232] The terms “example” and “exemplary” are used herein to mean “serving as an example, case, or illustration.” Any embodiment or design described herein as “exemplary” or “example” should not necessarily be construed as being preferable or advantageous to other embodiments or designs.

[0233] The terms “machine-readable storage medium” or “computer-readable storage medium” include, but are not limited to, portable or non-portable storage devices, optical storage devices, and various other media that can store, store, or transport instructions and / or data. Machine-readable storage medium or computer-readable storage medium may also include non-transient media that can store data and do not contain carrier waves and / or transient electronic signals that propagate over wireless or wired connections. Examples of non-transient media may include, but are not limited to, magnetic disks or tapes, optical storage media such as compact discs (CDs) or digital multipurpose discs (DVDs), flash memory, memory, or memory devices. Computer program products may include code and / or machine-executable instructions that can represent any combination of procedures, functions, subprograms, programs, routines, subroutines, modules, software packages, classes, instructions, data structures, or program statements. Code segments may be coupled to other code segments or hardware circuits by transferring and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, and data may be transmitted, transferred, or sent via any suitable means, such as memory sharing, message transfer, token transfer, or network transmission.

[0234] Furthermore, embodiments may be implemented by hardware, software, firmware, middleware, microcode, hardware description language, or any combination thereof. When implemented by software, firmware, middleware, or microcode, the program code or code segment (e.g., a computer program product) that performs the required work may be stored in a machine-readable medium. The processor can then perform the required work. Systems shown in some drawings may be provided in various configurations. In some examples, the system may be configured as a distributed system in which one or more elements of the system are distributed across one or more networks within a cloud computing system. Where an element is described as "configured" to perform a particular operation, such a configuration may be achieved, for example, by designing electronics or other hardware to perform the operation, by programming or controlling electronics (e.g., a microprocessor or other suitable electronics) to perform the operation, or by any combination thereof.

[0235] While specific embodiments of the Disclosure have been described, various changes, modifications, alternative configurations, and equivalents are also included within the scope of the Disclosure. The embodiments of the Disclosure are not limited to operating within a specific data processing environment, but can freely operate within multiple data processing environments. Furthermore, while the embodiments of the Disclosure have been described using a set of specific procedures and steps, it will be apparent to those skilled in the art that the scope of the Disclosure is not limited to the procedures and steps described. The various features and aspects of the embodiments described above can be used individually or in combination.

[0236] Furthermore, while embodiments of the Disclosure have been described using specific combinations of hardware and software, it should be recognized that other combinations of hardware and software are also included within the scope of the Disclosure. Embodiments of the Disclosure can be implemented using hardware alone, software alone, or a combination thereof. The various processes described in the Disclosure can be run on the same processor or on different processors in any combination. Thus, where it is described that a component or module is configured to perform a particular process, that configuration can be implemented, for example, by designing an electronic circuit to perform that process, by programming a programmable electronic circuit (such as a microprocessor) to perform that process, or by a combination thereof. Processes can communicate using a variety of technologies, including but not limited to prior art for communication between processes. Different pairs of processes may use different technologies, or the same pair of processes may use different technologies at different times.

[0237] Therefore, the specification and drawings should be considered illustrative rather than restrictive. However, it will be apparent that additions, reductions, deletions, and other modifications and alterations may be made without departing from the broad spirit and scope defined by the claims. Thus, although specific embodiments of this disclosure have been described, these embodiments are not intended to be limiting. Various modifications and their equivalents are included in the appended claims.

[0238] The indefinite articles “a” / “an”, the definite article “the”, and similar references used in the context describing this disclosure (particularly in the context of the claims) should be interpreted as including both singular and plural unless otherwise specifically stated herein or unless the meaning is clearly indicated otherwise. The terms “comprising”, “having”, “including”, and “containing” should be interpreted as non-restrictive terms (i.e., “including but not limited to”) unless otherwise specifically stated. The term “connected” should be interpreted as some or all of it being contained, attached, or joined together, even if something is intervening. In this specification, enumerations of value ranges are intended simply as a shorthand way of referring to each individual value that falls within that range, and unless otherwise specifically stated herein, each individual value is incorporated herein as it is described separately herein. Unless otherwise specifically stated herein or unless the meaning is clearly indicated otherwise, all methods described herein may be performed in any appropriate order. In this specification, the use of any and all examples or exemplary language (e.g., "like") is intended to clarify embodiments of the Disclosure and, unless otherwise specified, does not limit the scope of the Disclosure. Terms in the specification should not be construed as indicating any non-claimed elements essential to the implementation of the Disclosure.

[0239] Disjunctive language, such as the phrase "at least one of X, Y, or Z," is intended to be understood in context as commonly used to indicate that an item, term, etc., may be X, Y, Z, or any combination thereof (e.g., X, Y, and / or Z), unless otherwise specified. Therefore, such disjunctive language is not generally intended, nor implies, that a particular embodiment requires the presence of at least one X, at least one Y, or at least one Z.

[0240] Examples of embodiments of this disclosure can be described in light of the following sections: Item 1. A system comprising: a set of servers connected to an infrastructure network and hosting network interface services; a network virtualization device connected to the infrastructure network and hosting virtual network interface cards; and a host machine connected to the network virtualization device and hosting compute instances from a customer's virtual network, wherein the network interface services are configured to store configuration information relating to the configuration of the virtual network, to store version information relating to the configuration information, to determine that the virtual network interface cards are associated with at least a portion of the configuration information, to determine, based on the version information, that at least a portion of the configuration information is outdated, and to send an instruction to the network virtualization device that at least a portion is outdated.

[0241] Item 2. The system as described in Item 1, wherein the above-mentioned portion of the configuration information includes flow information for transmitting packets by the virtual network interface card on the virtual network, and the virtual network interface card is configured to store the flow information.

[0242] Item 3. The system described in Item 2, wherein the configuration information includes security policies, overlay-to-onboard Internet Protocol (IP) address mappings, and route rules for one or more packet flows, and the flow information includes at least one of a security policy, an overlay-to-onboard IP address mapping, or a route rule.

[0243] Item 4. The system described in any one of Items 1 to 3, wherein the above configuration information includes multiple parts, and the network interface service is further configured to store an indicator of the version of each of the multiple parts, and is configured to determine, based on the stored indicator of the version, that the version of the part associated with the virtual network interface card is outdated.

[0244] 5. The network interface service described above is further configured to send the updated version and the indicator for the updated version to the network virtualization device, the instructions including the updated version and the indicator for the updated version, as described in Section 4 of the system.

[0245] 6. The system as described in Section 4, wherein the network interface service is further configured to receive requests from the network virtualization device for the updated versions associated with the virtual network interface cards, the instructions notify the network virtualization device that the updated versions are available, and the network interface service is further configured to send the updated versions to the network virtualization device in response to the requests.

[0246] 7. The system as described in Section 4, wherein the network interface service is further configured to transmit the version information to the control plane and to receive from the control plane the updated version of the part associated with the virtual network interface card and an indicator of the updated version.

[0247] Item 8. The system described in Item 7, wherein transmitting the above version information includes transmitting a vector clock with a populated indicator of the version of each of the multiple parts that make up the above configuration information.

[0248] Item 9. A method comprising a network virtualization device storing at least some configuration information relating to the configuration of a customer's virtual network, wherein the network virtualization device hosts a virtual network interface card for the customer's compute instances to the virtual network and is connected to an underlying network, and the method further comprises the network virtualization device storing version information relating to the some configuration information, the network virtualization device determining that the some is outdated based on communication with a control plane regarding the version information, and the network virtualization device receiving an updated version of the some from the control plane or a network interface service, wherein the network interface service is hosted on a set of servers connected to the underlying network, and the method further comprises the network virtualization device replacing the some with the updated version.

[0249] Item 10. The method of Item 9, further comprising the network virtualization device transmitting the version information to the control plane, and the network virtualization device receiving an instruction from the control plane that the part is outdated.

[0250] Item 11. The method of Item 10, wherein transmitting the above version information includes transmitting a vector clock including the version indicator of the above part and the version indicator of the other part of the above configuration information, the above part and the above other part forming flow information for the virtual network interface card to transmit packets on the virtual network, and receiving the above instruction includes receiving the updated version indicator of the above part.

[0251] Item 12. The method according to any one of items 9 to 11, further comprising the network virtualization device sending a request to the network interface service to receive the updated version, the network virtualization device receiving the updated version from the network interface service based on the request, and the network virtualization device sending packets based on the updated version.

[0252] Item 13. The method of Item 12, wherein the above request is sent to the network interface service in response to receiving an instruction from the control plane that the above portion of the configuration information is outdated, and before receiving the above packet from the compute instance.

[0253] Item 14. The method according to any one of items 9 to 13, further comprising the network virtualization device receiving a packet from the host machine of the compute instance, the network virtualization device sending the packet to the network interface service on the basis that part of the configuration information is outdated, and the network virtualization device receiving the updated version from the network interface service on the basis of the packet.

[0254] Item 15. A non-temporary computer-readable instruction, when executed on a network virtualization device, causes the network virtualization device to perform an operation, the operation comprising storing at least some configuration information relating to the configuration of a customer's virtual network, the network virtualization device hosts a virtual network interface card to the virtual network for the customer's compute instances and is connected to an underlying network, the operation further comprising storing version information relating to the some configuration information, determining that the some is outdated based on communication with a control plane regarding the version information, and receiving an updated version of the some from the control plane or a network interface service, the network interface service being hosted on a set of servers connected to the underlying network, and the operation further comprising replacing the some with the updated version.

[0255] Item 16. The above operation further includes receiving the above-mentioned partial configuration information and the above-mentioned version information from the above-mentioned network interface service, wherein the above-mentioned partial and the above-mentioned version information are stored in a cache associated with the above-mentioned virtual network interface card, as described in Item 15, a non-temporary computer-readable instruction.

[0256] Item 17. A non-temporary computer-readable instruction as described in Item 15 or 16, wherein the configuration information includes a plurality of parts that form flow information for transmitting packets by the virtual network interface card, and the operation further includes receiving and storing each of the plurality of parts and the corresponding version indicator from the network interface service.

[0257] Item 18. The non-temporary computer-readable instruction described in Item 17, wherein the operation further includes receiving a packet from the host machine of the compute instance, determining, based on communication with the control plane, that the flow information contains at least one old portion, and, based on the fact that the flow information contains at least one old portion, sending the packet to the virtual network interface card.

[0258] Item 19. The operation described above further includes receiving an updated version of the at least one older portion of the non-temporary computer-readable instruction described in Item 18 in response to sending the packet from the virtual network interface card.

[0259] Item 20. The configuration information described above includes security policies, overlay-to-onboard Internet Protocol (IP) address mappings, and route rules for one or more packet flows, and the flow information includes at least one of a security policy, an overlay-to-onboard IP address mapping, or a route rule, as described in Item 17.

[0260] Preferred embodiments of the Disclosure are described herein, including the best known mode for carrying out the Disclosure. Variations of these preferred embodiments will become apparent to those skilled in the art by reading the foregoing description. Those skilled in the art may, as appropriate, adopt such modifications, and the Disclosure may be carried out in ways other than those specifically described herein. Accordingly, the Disclosure includes all variations and equivalents of the subject matter described in the claims appended herein, as permitted by applicable law. Furthermore, any combination of the above elements in all possible variations is incorporated herein unless otherwise indicated herein.

[0261] All references cited herein, including publications, patent applications, and patents, shall be incorporated by reference to the same extent as if they were included herein, provided that each reference is individually and clearly indicated as being incorporated by reference.

[0262] While aspects of the disclosure described above are explained with reference to specific embodiments, those skilled in the art will recognize that the disclosure is not limited thereto. The various features and aspects of the disclosure described above may be used individually or in combination. Furthermore, embodiments may be used in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of this specification. Accordingly, the specification and drawings should be considered illustrative rather than restrictive.< / realm>

Claims

1. It is a system, A set of servers connected to the board network and hosting network interface services, A network virtualization device that hosts a virtual network interface card and is connected to the aforementioned board network, The system comprises a host machine connected to the aforementioned network virtualization device and hosting compute instances from the customer's virtual network, The aforementioned network interface service is It is configured to store configuration information relating to the configuration of the virtual network, It is configured to store version information relating to the aforementioned configuration information, The virtual network interface card is configured to determine that it is associated with at least a portion of the configuration information, Based on the aforementioned version information, it is configured to determine that at least a portion of the aforementioned configuration information is outdated. The system is configured to send an instruction to the network virtualization device that at least a portion of the above is outdated. A portion of the configuration information includes flow information for transmitting packets by the virtual network interface card on the virtual network, and the virtual network interface card is configured to store the flow information. The aforementioned configuration information includes multiple parts, The network interface service is further configured to store an indicator of the version of each of the plurality of parts, The network interface service is further configured to determine, based on the stored indicator of the version, that some of the versions associated with the virtual network interface card are outdated. The network interface service is further configured to receive requests from the network virtualization device for the updated versions associated with the virtual network interface card, and the instructions notify the network virtualization device that the updated versions are available. The network interface service is further configured to send the updated version to the network virtualization device in response to the request.

2. The system according to claim 1, wherein the configuration information includes security policies, overlay-to-onboard Internet Protocol (IP) address mappings, and route rules for one or more packet flows, and the flow information includes at least one of a security policy, an overlay-to-onboard IP address mapping, or a route rule.

3. The aforementioned network interface service further, The system according to claim 1, configured to send the partial updated version and an indicator of the updated version to the network virtualization device, wherein the instruction includes the updated version and an indicator of the updated version.

4. The aforementioned network interface service further, The aforementioned version information is configured to be transmitted to the control plane, The system according to claim 1, configured to receive from the control plane a portion of the updated version associated with the virtual network interface card and an indicator of the updated version.

5. The system according to claim 4, wherein transmitting the version information includes transmitting a vector clock with a version indicator for each of the plurality of parts forming the configuration information.

6. It is a method, The network virtualization device includes storing at least some configuration information relating to the configuration of the customer's virtual network, the network virtualization device hosts a virtual network interface card for the customer's compute instances to the virtual network, and is connected to the underlying network, and the method further includes The network virtualization device stores version information relating to some of the configuration information, The network virtualization device determines that the part is outdated based on communication with the control plane regarding the version information, The network virtualization device includes receiving the updated version of the portion from the control plane or network interface service, the network interface service being hosted on a set of servers connected to the board network, and the method further includes: The network virtualization device replaces a portion of the flow information with the updated version, The network virtualization device includes transmitting the version information to the control plane, the transmission of the version information includes transmitting a vector clock including the version indicators for part of the configuration information and the version indicators for other parts of the configuration information, and the method further includes A method comprising the network virtualization device receiving an instruction from the control plane that the part is outdated.

7. The method according to claim 6, wherein the part and the other part form flow information for transmitting packets by the virtual network interface card on the virtual network, and receiving the instruction includes receiving an updated version indicator of the part.

8. The network virtualization device sends a request to the network interface service to receive the updated version, The network virtualization device receives the updated version from the network interface service based on the request, The method according to claim 6, further comprising the network virtualization device transmitting packets based on the updated version.

9. The method of claim 8, wherein the request is sent to the network interface service in response to receiving an instruction from the control plane that the portion of the configuration information is outdated, and before receiving the packet from the compute instance.

10. The network virtualization device receives packets from the host machine of the compute instance, The network virtualization device sends the packet to the network interface service based on the fact that part of the configuration information is outdated. The method according to claim 6, further comprising the network virtualization device receiving the updated version from the network interface service based on the packet.

11. A computer-readable program that, when executed on a network virtualization device, causes the network virtualization device to perform an action, wherein the action is: The operation includes storing at least some configuration information relating to the configuration of the customer's virtual network, the network virtualization device hosts a virtual network interface card for the customer's compute instances to the virtual network, is connected to the underlying network, and the operation further includes: The system stores version information related to some of the aforementioned configuration information, Based on communication with the control plane regarding the aforementioned version information, it is determined that the aforementioned part is outdated. This includes receiving the updated version of the control plane or network interface service, the network interface service being hosted on a set of servers connected to the board network, and the operation further includes: This includes replacing the aforementioned portion with the updated version of the flow information, The configuration information includes a plurality of parts that form flow information for transmitting packets by the virtual network interface card, and the operation further includes receiving and storing each of the plurality of parts and corresponding version indicators from the network interface service, a computer-readable program.

12. The computer-readable program according to claim 11, wherein the operation further includes receiving the partial configuration information and the version information from the network interface service, the partial and the version information being stored in a cache associated with the virtual network interface card.

13. The aforementioned operation further, Receiving packets from the host machine of the aforementioned computing instance, Based on communication with the control plane, it is determined that the flow information includes at least one old portion. A computer-readable program according to claim 11, comprising transmitting the packet to the virtual network interface card based on the fact that the flow information includes at least one old portion.

14. The aforementioned operation further, The computer-readable program according to claim 13, comprising receiving an updated version of the at least one old portion in response to transmitting the packet from the virtual network interface card.

15. The computer-readable program according to any one of claims 11 to 14, wherein the configuration information includes security policies, overlay-to-onboard Internet Protocol (IP) address mappings, and route rules for one or more packet flows, and the flow information includes at least one of a security policy, an overlay-to-onboard IP address mapping, or a route rule.

Images