Implementation of Communication in a Container Environment

Abstract

Techniques for implementing a container environment in which each internal pod is provided with a unique IP address and virtual communication devices such as an IPvlan device are described. Without the need for bridging and encapsulation, communication from a source pod is directly routed to a destination pod within the container environment by one or more virtualized network interface cards (VNICs) that utilize the unique IP address of the destination pod. This reduces the size of the data to be transmitted and eliminates the computational cost required to perform encapsulation of the data during transmission.

Description

Technical Field

[0001] Cross - reference to Related Applications This application claims the benefit of the filing dates of U.S. Provisional Patent Application No. 63 / 353,462, filed on June 17, 2022, and U.S. Patent Application No. 17 / 964,768, filed on October 12, 2022, and incorporates by reference herein in their entirety each of the above - mentioned applications.

Background Art

[0002] Background Today, container environments, including orchestration systems such as Kubernetes, are well - known means for implementing and deploying container - based applications. However, applications and pods implemented within conventional systems communicate with each other using only bridge devices that encapsulate the communication. This encapsulation not only adds computational and data overhead to such communication but also makes it impossible to identify specific sender - receiver information in the communication flow logs.

Summary of the Invention

[0003] Brief Summary Embodiments of the present disclosure relate to facilitating direct communication between pods within a container environment. A container environment is disclosed that includes one or more nodes, each including one or more pods. In the container environment, neither the use of a bridge nor encapsulation is required, and each pod includes a virtual communication device used for communication with other pods. Each node including a pod is accompanied by one or more virtualized network interface cards (VNICs), and these VNICs facilitate pod communication by receiving and forwarding data for transmission to and from each pod via their respective virtual communication devices.

[0004] At least one embodiment is directed to a method performed by a computer. The method may include configuring a node by one or more operating system level virtualization instances. The method may further include configuring communication via a virtual communication device from one or more operating system level virtualization instances.

[0005] At least one embodiment is directed to a method performed by a computer. The method may include configuring a first node that includes a first container in a container environment. The method may further include configuring communication via a first virtual communication device from the first container. The method may further include receiving, at a first virtual network interface card (VNIC) connected to the first node within the container environment, data that is configured to be transmitted from a first virtual communication device assigned to a first container within the first node to a destination device. The method may further include the first VNIC routing data that includes an address identifying at least the first container to the destination device.

[0006] Another embodiment is directed to a computing device comprising one or more processors and instructions that, when executed by the one or more processors, cause the computing device to perform any suitable combination of the methods disclosed herein.

[0007] Yet another embodiment is directed to a non-transitory computer-readable medium storing computer-executable instructions that, when executed by one or more processors of a computing cluster, cause the computing cluster to perform any suitable combination of the methods disclosed herein.

[0008] Yet another embodiment is directed to an apparatus for implementing communication in a container environment, the apparatus comprising means for provisioning a first node in the container environment that includes a first pod representing a group of one or more containers having shared resources; means for provisioning communication from the first pod via a first virtual communication device; means for receiving, at a first virtualized network interface card (VNIC) connected to the first node within the container environment, data configured to be transmitted from a first virtual communication device assigned to the first pod within the first node to a destination device; and means for routing the data, which includes an address identifying at least the first pod, to the destination device via the first virtualized network interface card.

[0009] Yet another embodiment is directed to a computer program product that includes computer instructions which, when executed by a processor, perform any suitable combination of the methods disclosed herein.

[0010] For ease of readily identifying the discussion of any particular element or operation, the most significant one or more digits of a reference number represent the figure number in which the element was first introduced.

Brief Description of the Drawings

[0011]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

[0012] **DETAILED DESCRIPTION** **Exemplary Virtual Networking Architecture** The term "cloud service" is generally used to represent services that are made available on demand (e.g., through a subscription model) to users or customers by a cloud service provider (CSP) through the use of systems and infrastructure (cloud infrastructure) provided by the CSP. Usually, the servers and systems that make up the CSP's infrastructure are separate from the customer's own on-premises servers and systems. As a result, customers can utilize the cloud services provided by the CSP without having to purchase separate hardware and software resources for the services. Cloud services are designed to provide easy and scalable access to applications and computing resources to registered customers without the customer having to invest in the procurement of the infrastructure used to provide the services.

[0013] There are multiple cloud service providers that offer various types of cloud services. There are various different types or models of cloud services, such as software as a service (SaaS), platform as a service (PaaS), infrastructure as a service (IaaS), etc.

[0014] Customers can register for one or more cloud services provided by the CSP. Any entity can be a customer, such as an individual, an organization, a company, etc. When a customer registers or subscribes to the services provided by the CSP, a tenancy or account is created for that customer. And the customer can then access one or more registered cloud resources associated with that account using this account.

[0015] As described above, Infrastructure as a Service (IaaS) is a specific type of cloud computing service. In the IaaS model, the CSP provides the infrastructure (referred to as cloud service provider infrastructure or CSPI) that can be used by customers to build their respective customizable networks and deploy customer resources. Thus, the customer's resources and networks are hosted in a distributed environment by the infrastructure provided by the CSP. This is different from traditional computing where the customer's resources and networks are hosted by the infrastructure provided by the customer.

[0016] CSPI may include interconnected high-performance computing resources that comprise various host machines, memory resources, and network resources that make up a physical network, also referred to as the underlying network or basic network. The resources of CSPI may be geographically dispersed across one or more data centers that may span one or more geographical regions. These physical resources may execute virtualization software to provide a virtualized distributed environment. Through virtualization, an overlay network (also known as a software-based network, software-defined network, or virtual network) is generated on the physical network. The CSPI physical network serves as the basis for generating one or more overlays or virtual networks on the physical network. The physical network (or underlying network or basic network) comprises physical network devices such as physical switches, routers, computers, and host machines. An overlay network is a logical (or virtual) network that operates on the physical underlying network. A given physical network may support one or more overlay networks. Overlay networks typically use encapsulation techniques to distinguish traffic belonging to different overlay networks. The virtual or overlay network is also referred to as a virtual cloud network (VCN). The virtual network is realized through the use of software virtualization techniques (such as hypervisors, virtualization functions executed by network virtualization devices (NVDs) (such as smart NICs), top-of-rack (TOR) switches, smart TORs that implement one or more functions executed by NVDs, and other mechanisms) to generate a network abstraction layer that can operate on the physical network. The virtual network can take many forms, including peer-to-peer networks, IP networks, etc. The virtual network is typically a layer 3 IP network or a layer 2 VLAN. This method of virtual or overlay networking is often referred to as virtual or overlay layer 3 networking.Examples of protocols developed for virtual networks include IP-in-IP (or GRE (Generic Routing Encapsulation)), VXLAN-IETF RFC 7348 (Virtual Extensible LAN), VPN (Virtual Private Network) (e.g., MPLS Layer-3 Virtual Private Networks (RFC 4364)), VMware's NSX, GENEVE (Generic Network Virtualization Encapsulation), and the like.

[0017] In the case of IaaS, the infrastructure provided by the CSP (CSPI) can be configured to provide virtualized computing resources via a public network (e.g., the Internet). In the IaaS model, a cloud computing service provider can host infrastructure components (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer), etc.). Also, optionally, the IaaS provider may supply various services associated with these infrastructure components (e.g., billing, monitoring, logging, security, load balancing, and clustering, etc.). Therefore, since these services can be policy-driven, IaaS users may be able to maintain the availability and performance of applications by implementing policies that promote load balancing. CSPI provides the infrastructure and a set of complementary cloud services that enable customers to build and operate a wide range of applications and services in a hosted highly available distributed environment. CSPI provides high-performance computing resources, functions, and storage capacity in a flexible virtual network that can be securely accessed from various network locations such as the customer's on-premises network. When a customer registers or subscribes to the IaaS services provided by the CSP, the tenant generated for the customer is an isolated secure partition within the CSPI where the customer can create, orchestrate, and manage each cloud resource.

[0018] Customers can build their respective virtual networks using the compute, memory, and network resources provided by the CSPI. One or more customer resources or workloads, such as compute instances, can be deployed on these virtual networks. For example, customers can use the resources provided by the CSPI to build one or more customizable private virtual networks (referred to as virtual cloud networks (VCNs)). Customers can deploy one or more customer resources, such as compute instances, to the customer VCN. Compute instances can be in the form of virtual machines, bare metal instances, etc. In this way, the CSPI provides the infrastructure and a set of complementary cloud services that enable customers to build and operate a wide range of applications and services in a hosted highly available virtual environment. While customers do not manage or control the underlying physical resources provided by the CSPI, they do control the operating system, storage, and deployed applications. For some network components (e.g., firewalls), control may also be restricted.

[0019] The CSP may provide a console that enables customers and network administrators to configure, access, and manage resources deployed in the cloud using CSPI resources. In one embodiment, the console provides a web-based user interface that can be used for access and management of the CSPI. In some embodiments, the console is a web-based application provided by the CSP.

[0020] CSPI can support single-tenancy or multi-tenancy architectures. In a single-tenancy architecture, software (e.g., applications, databases) or hardware components (e.g., host machines or servers) serve a single customer or tenant. In a multi-tenancy architecture, software or hardware components serve multiple customers or tenants. Thus, in a multi-tenancy architecture, CSPI resources are shared among multiple customers or tenants. In a multi-tenancy situation, preventive measures and protective measures are taken within CSPI so that the data of each tenant is isolated and not visible to other tenants.

[0021] In a physical network, a network endpoint (the "endpoint") represents a computing device or system that is connected to the physical network and communicates with the destination network. The network endpoints of a physical network may be connected to a local area network (LAN), a wide area network (WAN), or other types of physical networks. Examples of conventional endpoints in a physical network include modems, hubs, bridges, switches, routers, and other network devices, physical computers (or host machines), and the like. Each physical device in a physical network has a fixed network address that can be used for communication with the device. This fixed network address can be a layer 2 address (e.g., MAC address), a fixed layer 3 address (e.g., IP address), and the like. In a virtualized environment or virtual network, endpoints can include various virtual endpoints such as virtual machines hosted by components of the physical network (e.g., hosted by a physical host machine). The endpoints of these virtual networks are addressed by overlay addresses such as an overlay layer 2 address (e.g., overlay MAC address) and an overlay layer 3 address (e.g., overlay IP address). According to the network overlay, flexibility is provided by enabling network administrators to move the overlay addresses associated with network endpoints using software management (e.g., via software implementing the control plane of the virtual network). Thus, unlike physical networks, in virtual networks, the use of network management software allows an overlay address (e.g., overlay IP address) to move from one endpoint to another. Since the virtual network is built on the physical network, communication between components of the virtual network involves both the virtual network and the underlying physical network.To facilitate such communication, the components of the CSPI are configured to learn and store mappings that perform the mapping from the overlay address in the virtual network to the actual physical address in the underlying network and vice versa, and to facilitate communication by using these mappings. Also, encapsulation of customer traffic facilitates routing in the virtual network.

[0022] Accordingly, a physical address (e.g., a physical IP address) is associated with a component of the physical network, and an overlay address (e.g., an overlay IP address) is associated with an entity of the virtual or overlay network. A physical IP address is an IP address associated with a physical device (e.g., a network device) of the underlying or physical network. For example, each NVD has an associated physical IP address. An overlay IP address is an overlay address associated with an entity of the overlay network, such as a compute instance of a customer's virtual cloud network (VCN). Two different customers or tenants, each having their own private VCN, can potentially use the same overlay IP address in their respective VCNs without having any knowledge of each other. Both the physical IP address and the overlay IP address are a type of real IP address. These are distinct from virtual IP addresses. A virtual IP address is typically a single IP address that represents or is mapped to a plurality of real IP addresses, or a single IP address that maps to a plurality of real IP addresses. A virtual IP address provides a one-to-many mapping between the virtual IP address and the plurality of real IP addresses. For example, a load balancer may use a VIP for mapping to or representing a plurality of servers, each having its own real IP address.

[0023] Cloud infrastructure or CSPI is physically hosted in one or more data centers in one or more regions around the world. CSPI can include physical or underlying network components as well as virtualized components (e.g., virtual networks, compute instances, virtual machines, etc.) in virtual networks built on top of physical network components. In certain embodiments, CSPI is orchestrated and hosted within realms, regions, and availability domains. A region is typically a local geographic area that includes one or more data centers. Regions are generally independent of each other and can be separated by long distances, for example spanning a country or continent. For example, a first region could be in Australia, another region could be in Japan, and yet another region could be in India. CSPI resources are divided across regions such that each region has a subset of the independent CSPI resources. Each region can provide a set of core infrastructure services and resources such as compute resources (e.g., bare metal servers, virtual machines, containers, and related infrastructure, etc.), storage resources (e.g., block volume storage, file storage, object storage, archive storage), network resources (e.g., virtual cloud network (VCN), load balancing resources, connection to on-premises networks), database resources, edge network resources (e.g., DNS), as well as access management and monitoring resources. Each region generally has multiple paths connecting it to other regions of the realm.

[0024] Generally, an application is deployed in the region where it is most frequently used (i.e., deployed on the infrastructure associated with that region). This is because using nearby resources is faster than using distant resources. Also, an application can be deployed in different regions for various reasons, such as redundancy to reduce the risk of region-wide events like large-scale weather systems or earthquakes, jurisdiction, tax, and redundancy to meet various requirements such as other business or social standards.

[0025] Data centers within a region can be further organized and subdivided into Availability Domains (ADs). An Availability Domain can correspond to one or more data centers located within a region. A region can be composed of one or more Fault Domains. In such a distributed environment, CSPI resources are specific to a region such as a Virtual Cloud Network (VCN) or specific to an Availability Domain such as a compute instance.

[0026] ADs within a region are isolated from each other, fault-tolerant, and configured such that the probability of simultaneous failures is very low. This is achieved by ADs not sharing important infrastructure resources such as networks, physical cables, cable paths, cable entry points, etc. Therefore, even if a failure occurs in one AD within a region, the availability of other ADs within the same region is less likely to be affected. ADs within the same region can be connected to each other by a low-latency and high-bandwidth network, enabling high-availability connections to other networks (e.g., the Internet, the customer's on-premises network, etc.), and allowing replication systems to be built across multiple ADs for both high availability and disaster recovery. In cloud services, multiple ADs are used to ensure high availability and protect against resource failures. As the infrastructure provided by an IaaS provider grows, capacity can be added to more regions and ADs. Also, traffic between Availability Domains is typically encrypted.

[0027] In one embodiment, regions are grouped as realms. A realm is a logical collection of regions. Realms are isolated from each other and do not share any data. Regions within the same realm can communicate with each other, but regions in different realms cannot. A customer's tenancy or account using CSP exists in a single realm and can be distributed across one or more regions belonging to that realm. Typically, when a customer registers for an IaaS service, the customer's tenancy or account is created in a customer-specific region (referred to as the "home" region) within the realm. A customer can extend the customer's tenancy across one or more other regions within the realm. A customer cannot access regions that are not in the realm in which the customer's tenancy exists.

[0028] An IaaS provider can provide multiple realms, each corresponding to a specific set of customers or users. For example, a commercial realm can be provided for commercial customers. As another example, for a specific country, a realm can be provided for customers within that country. As yet another example, a government realm can be provided for the government, and so on. For example, a government realm can correspond to a specific government and can have a higher security level than a commercial realm. For example, OCI (Oracle Cloud Infrastructure) currently provides one realm for commercial regions and two realms (e.g., FedRAMP certified and IL5 certified) for government cloud regions.

[0029] In one embodiment, the AD can be subdivided into one or more fault domains. A fault domain is a group of infrastructure resources within the AD that provides anti-affinity. The fault domain can disperse compute instances so that they do not reside on the same physical hardware within a single AD. This is known as anti-affinity. The fault domain represents a set of hardware components (computers, switches, etc.) that share a single point of failure. The compute pool is logically divided into fault domains. Thus, a hardware failure or compute hardware maintenance event that affects one fault domain does not affect instances in other fault domains. Depending on the embodiment, the number of fault domains per AD can vary. For example, in one embodiment, each AD includes three fault domains. The fault domain acts as a logical data center within the AD.

[0030] When a customer registers for an IaaS service, resources are provisioned from the CSPI for the customer and associated with the customer's tenancy. The customer can use these provisioned resources to build a private network and deploy resources on these networks. The customer network hosted in the cloud by the CSPI is referred to as a Virtual Cloud Network (VCN). The customer can use the allocated CSPI resources to set up one or more Virtual Cloud Networks (VCNs). A VCN is a virtual or software-defined private network. Customer resources deployed in the customer's VCN can include compute instances (e.g., virtual machines, bare metal instances) and other resources. These compute instances can represent various customer workloads such as applications, load balancers, databases, etc. Compute instances deployed in a VCN can communicate with public endpoints (the "public endpoints") accessible via a public network such as the Internet, other instances in the same VCN or other VCNs (e.g., other VCNs of the customer or VCNs not belonging to the customer), the customer's on-premises data center or network, as well as service endpoints and other types of endpoints.

[0031] CSP can provide various services using CSPI. In some cases, the customers of CSPI themselves can act like service providers and provide services using CSPI resources. A service provider may expose service endpoints characterized by identification information (such as IP addresses, DNS names, and ports). A customer's resources (such as compute instances) can consume a particular service by accessing the service endpoint exposed by that service for that particular service. These service endpoints are generally endpoints that can be publicly accessed by a user through the use of a public IP address associated with the endpoint via a public communication network such as the Internet. Also, a network endpoint that is publicly accessible may sometimes be referred to as a public endpoint.

[0032] In one embodiment, a service provider can expose a service through a service endpoint (which may also be referred to as a service endpoint). And a customer of the service can use this service endpoint to access the service. In one implementation, multiple customers intending to consume the service can access a given service endpoint for the service. In other embodiments, a dedicated service endpoint may be provided for a customer so that only that customer can access the service using that dedicated service endpoint.

[0033] In one embodiment, when a VCN is created, it is associated with a private overlay CIDR (Classless Inter-Domain Routing) address space that is a broad range of private overlay IP addresses assigned to the VCN (e.g., 10.0 / 16). The VCN includes associated subnets, route tables, and gateways. The VCN exists within a single region, but may span one or more or all of the availability domains of a region. A gateway is a virtual interface configured for the VCN that enables communication of traffic between the VCN and one or more endpoints outside the VCN. Different types of gateways may be configured for the VCN to enable communication between different types of endpoints.

[0034] The VCN can be subdivided into one or more subnetworks such as one or more subnets. Thus, a subnet is a configuration unit or subdivision that can be created within the VCN. The VCN can have one or more subnets. Each subnet within the VCN does not overlap with other subnets of the VCN and is associated with a continuous range of overlay IP addresses (e.g., 10.0.0.0 / 24 and 10.0.1.0 / 24) that represent a subset of the address space within the VCN's address space.

[0035] Each compute instance is associated with a virtual network interface card (VNIC) that enables the compute instance to participate in a subnet of the VCN. A VNIC is a logical representation of a physical network interface card (NIC). Generally, a VNIC is an interface between an entity (e.g., a compute instance, a service) and a virtual network. A VNIC exists within a subnet, has one or more associated IP addresses, and is associated with security rules or policies. A VNIC corresponds to a Layer 2 port on a switch. A VNIC is associated with a compute instance and a subnet within the VCN. According to the VNIC associated with a compute instance, the compute instance can become part of a subnet of the VCN, and the compute instance can communicate (e.g., send and receive packets) with endpoints on the same subnet as the compute instance, endpoints on different subnets of the VCN, or endpoints outside the VCN. Therefore, the VNIC associated with a compute instance determines how the compute instance connects to endpoints inside and outside the VCN. A VNIC for a compute instance is created and associated with the compute instance when the compute instance is created and added to a subnet within the VCN. In the case of a subnet containing a set of compute instances, this subnet contains VNICs corresponding to the set of compute instances, and each VNIC is associated with a certain compute instance among the set of compute instances.

[0036] Each compute instance is assigned a private overlay IP address via the VNIC associated with that compute instance. This private overlay IP address is assigned to the VNIC associated with the compute instance when the compute instance is created and is used for routing traffic to the compute instance. In a given subnet, all VNICs use the same route table, security list, and DHCP options. As described above, each subnet within a VCN does not overlap with other subnets of the VCN and is associated with a continuous range of overlay IP addresses (e.g., 10.0.0.0 / 24 and 10.0.1.0 / 24) that represent a subset of the address space within the VCN's address space. For a VNIC on a particular subnet of a VCN, the private overlay IP address assigned to the VNIC is an address within the continuous range of overlay IP addresses assigned to the subnet.

[0037] In certain embodiments, in addition to the private overlay IP address, additional overlay IP addresses, such as one or more public IP addresses in the case of a public subnet, may optionally be assigned to the compute instance. These multiple addresses are assigned to the same VNIC or to multiple VNICs associated with the compute instance. However, each instance has a primary VNIC that is created at instance startup and is associated with the overlay private IP address assigned to the instance (this primary VNIC need not be deleted). In the same availability domain as the primary VNIC, additional VNICs, referred to as secondary VNICs, can be added to an existing instance. All VNICs are in the same availability domain as this instance. The secondary VNIC can be in the same subnet of the same VCN or in different subnets of the same or different VCNs.

[0038] If a compute instance exists in a public subnet, it can optionally be assigned a public IP address. A subnet can be specified as a public subnet or a private subnet at the time of creation. A private subnet means that resources in the subnet (e.g., compute instances) and associated VNICs cannot have public overlay IP addresses. A public subnet means that resources in the subnet and associated VNICs can have public IP addresses. A customer can specify a subnet to exist in a single availability domain or multiple availability domains of a region or realm.

[0039] As described above, a VCN can be subdivided into one or more subnets. In some embodiments, a virtual router (VR) (referred to as a VCN VR or simply a VR) configured for a VCN enables communication between the subnets of the VCN. For a subnet within a VCN, the VR represents a logical gateway for the subnet that enables the compute instances on that subnet to communicate with endpoints on other subnets within the VCN and other endpoints outside the VCN. The VCN VR is a logical entity configured to route traffic between VNICs in the VCN and virtual gateways (the "gateways") associated with the VCN. The gateways are described separately below with respect to FIG. 1. The VCN VR is a Layer 3 / IP layer concept. In one embodiment, there is one VCN VR for a VCN, and the VCN VR potentially has a limited number of ports addressed by IP addresses, with one port per subnet of the VCN. Thus, the VCN VR has a different IP address for each subnet of the VCN to which the VCN VR pertains. Also, the VR is connected to various gateways configured for the VCN. In some embodiments, a particular overlay IP address from an overlay IP address range for a subnet is reserved for the port of the VCN VR for that subnet. For example, assume a VCN has two subnets, each associated with address ranges 10.0 / 16 and 10.1 / 16, respectively. For the first subnet within the VCN with an address range of 10.0 / 16, the addresses in this range are reserved for the port of the VCN VR for that subnet. Optionally, the first IP address in this range may be reserved for the VCN VR. For example, for a subnet with an overlay IP address range of 10.0 / 16, the IP address 10.0.0.1 may be reserved for the port of the VCN VR for that subnet. For the second subnet within the same VCN with an address range of 10.1 / 16, the VCN VR may have a port with an IP address of 10.1.0.1 for that second subnet.The VCN VR has a different IP address for each of the subnets of the VCN.

[0040] In some other embodiments, each subnet within a VCN may have its own associated VR, which can be addressed by the subnet using a reserved or default IP address associated with that VR. The reserved or default IP address may be, for example, the first IP address in the range of IP addresses associated with the subnet. VNICs within the subnet can communicate (e.g., send and receive packets) with the VR associated with the subnet using this default or reserved IP address. In such embodiments, the VR is the input / output point of the subnet. The VRs associated with subnets within a VCN can communicate with other VRs associated with other subnets within the VCN. Also, the VR can communicate with the gateway associated with the VCN. The VR functionality of a subnet operates on one or more NVDs that execute the VNIC functionality of the VNICs within the subnet, or is executed by one or more NVDs.

[0041] For a VCN, a route table, security rules, and DHCP options can be set. The route table is the virtual route table of the VCN and includes rules for routing traffic from subnets within the VCN to destinations outside the VCN via a gateway or a specially configured instance. The route table of the VCN can be customized to control how packets are forwarded / routed for the VCN. The DHCP options represent configuration information that is automatically provided to an instance at startup.

[0042] The security rules set for a VCN represent the overlay firewall rules of the VCN. Security rules include ingress and egress rules and can specify the types of traffic (e.g., based on protocol and port) that can enter and leave instances within the VCN. A customer can select whether a given rule is stateful or stateless. For example, a customer can allow incoming SSH traffic to a set of instances from anywhere by setting a stateful ingress rule for source CIDR 0.0.0.0 / 0 and destination TCP port 22. Security rules can be implemented by using network security groups or security lists. A network security group consists of a set of security rules that apply only to the resources of that group. On the other hand, a security list contains rules that apply to all resources of any subnet that uses that security list. A VCN can be provided with a default security list that includes default security rules. The DHCP options set for a VCN provide the configuration information that is automatically given to the instances of the VCN at startup.

[0043] In one embodiment, the configuration information of a VCN is determined and stored by the VCN control plane. The configuration information of a VCN can include information such as the address range associated with the VCN, subnets and related information within the VCN, one or more VRs associated with the VCN, compute instances and related VNICs within the VCN, NVDs that execute various virtualized network functions (e.g., VNIC, VR, gateway) associated with the VCN, status information of the VCN, as well as information regarding other VCN-related information. In one embodiment, the VCN distribution service publishes the configuration information stored by the VCN control plane or a part thereof to the NVD. The distributed information can be stored by the NVD and used to update information (e.g., forwarding tables, routing tables, etc.) used for packet forwarding to the compute instances of the VCN.

[0044] In one embodiment, the creation of the VCN and subnets is processed by the VCN control plane (CP), and the launch of compute instances is processed by the compute control plane. After the compute control plane is responsible for allocating the physical resources of the compute instance, it calls the VCN control plane to generate a VNIC and associate it with the compute instance. Also, the VCN CP sends a VCN data mapping to the VCN data plane that is configured to perform packet transfer and routing functions. In one embodiment, the VCN VP provides a distribution service responsible for supplying updates to the VCN data plane. Examples of the VCN control plane are also shown in FIGS. 16, 17, 18, and 19 (see reference numerals 1616, 1716, 1816, and 1916), and will be described later.

[0045] A customer can create one or more VCNs using the resources hosted by the CSPI. Compute instances deployed in the customer VCN can communicate with various endpoints. These endpoints can include endpoints hosted by the CSPI and endpoints outside the CSPI.

[0046] Various different architectures for implementing cloud-based services using CSPI are shown and described below in FIGS. 1, 2, 3, 4, 5, 16, 17, 18, and 20. FIG. 1 is a high-level diagram of a distributed environment 100 showing an overlay or customer VCN hosted by CSPI according to an embodiment. The distributed environment shown in FIG. 1 includes a plurality of components of an overlay network. The distributed environment 100 shown in FIG. 1 is merely an example and is not intended to unduly limit the scope of the embodiments according to the claims. Many variations, alternatives, and improvements are possible. For example, in some embodiments, the distributed environment shown in FIG. 1 may have more or fewer systems or components than those shown in FIG. 1, may combine two or more systems, or may have different system configurations or arrangements.

[0047] As shown in the example of FIG. 1, the distributed environment 100 includes a CSPI 101 that provides services and resources that customers can register to use to build their respective virtual cloud networks (VCNs). In one embodiment, CSPI 101 provides IaaS services to registered customers. The data centers within CSPI 101 may be organized as one or more regions. FIG. 1 shows one exemplary region, "Region US" 102. The customer has already set up a customer VCN 104 for region 102. The customer may deploy various compute instances to VCN 104, and these compute instances may include virtual machines or bare metal instances. Examples of instances include applications, databases, load balancers, etc.

[0048] In the embodiment shown in FIG. 1, customer VCN 104 includes two subnets, namely "Subnet 1" and "Subnet 2", and each subnet has its own CIDR IP address range. In FIG. 1, the overlay IP address range of Subnet 1 is 10.0 / 16, and the address range of Subnet 2 is 10.1 / 16. VCN virtual router 105 represents the logical gateway of the VCN that enables communication between the subnets of VCN 104 and other endpoints outside the VCN. VCN VR 105 is configured to route traffic between the VNICs in VCN 104 and the gateways associated with VCN 104. VCN VR 105 provides ports to each subnet of VCN 104. For example, VR 105 may provide a port with IP address 10.0.0.1 to Subnet 1 and a port with IP address 10.1.0.1 to Subnet 2.

[0049] Multiple compute instances may be deployed to each subnet, and the compute instances can be virtual machine instances and / or bare metal instances. The compute instances in a subnet may be hosted by one or more host machines within CSPI101. A compute instance participates in a subnet via a VNIC associated with the compute instance. For example, as shown in FIG. 1, compute instance C1 is part of subnet 1 via a VNIC associated with the compute instance. Similarly, compute instance C2 is part of subnet 1 via a VNIC associated with C2. Similarly, multiple compute instances (which may be virtual machine instances or bare metal instances) may be part of subnet 1. Each compute instance is assigned a private overlay IP address and a MAC address via its associated VNIC. For example, in FIG. 1, compute instance C1 has an overlay IP address of 10.0.0.2 and a MAC address of M1, while compute instance C2 has a private overlay IP address of 10.0.0.3 and a MAC address of M2. Each compute instance (including compute instances C1 and C2) in subnet 1 has a default route to VCN VR105 that uses IP address 10.0.0.1, which is the IP address of the port of VCN VR105 for subnet 1.

[0050] Subnet 2 can have multiple compute instances, including virtual machine instances and / or bare metal instances, deployed thereon. For example, as shown in FIG. 1, compute instances D1 and D2 are part of Subnet 2 via VNICs associated with each of the compute instances. In the embodiment shown in FIG. 1, compute instance D1 has an overlay IP address of 10.1.0.2 and a MAC address of MM1, while compute instance D2 has a private overlay IP address of 10.1.0.3 and a MAC address of MM2. Each compute instance (including compute instances D1 and D2) in Subnet 2 has a default route to VCN VR105 that uses IP address 10.1.0.1, which is the IP address of the port of VCN VR105 for Subnet 2.

[0051] Also, VCN A104 may include one or more load balancers. For example, a load balancer may be provided for a subnet and configured to load balance traffic among multiple compute instances on the subnet. Also, a load balancer may be provided to load balance traffic between subnets of a VCN.

[0052] A specific compute instance deployed on VCN104 can communicate with various different endpoints. These endpoints can include the endpoints hosted by CSPI200 and the endpoints outside CSPI200. The endpoints hosted by CSPI101 can include endpoints on the same subnet as the specific compute instance (e.g., communication between two compute instances on subnet 1), endpoints on different subnets within the same VCN (e.g., communication between a compute instance on subnet 1 and a compute instance on subnet 2), endpoints in different VCNs within the same region (e.g., communication between a compute instance on subnet 1 and an endpoint in a VCN in the same region 106 or 110, communication between a compute instance on subnet 1 and an endpoint in service network 110 in the same region), or endpoints in VCNs in different regions (e.g., communication between a compute instance on subnet 1 and an endpoint in a VCN in a different region 108). Also, a compute instance in the subnet hosted by CSPI101 can communicate with endpoints that are not hosted by CSPI101 (i.e., outside CSPI101). These outside endpoints can include endpoints in the customer's on-premises network 116, endpoints within another hosted remote cloud network 118, public endpoints 114 accessible via a public network such as the Internet, and other endpoints.

[0053] Communication between compute instances on the same subnet is facilitated by the use of the VNICs associated with the source compute instance and the destination compute instance. For example, a compute instance C1 on subnet 1 may want to send a packet to a compute instance C2 on subnet 1. For a packet that originates from a source compute instance and has a destination that is another compute instance within the same subnet, this packet is first processed by the VNIC associated with the source compute instance. The processing performed by the VNIC associated with the source compute instance may include determining the destination information of the packet by the packet header, identifying any policies (such as security lists) set for the VNIC associated with the source compute instance, determining the next hop of the packet, performing the packet encapsulation / decapsulation function as required, and transferring / routing the packet to the next hop to facilitate communication to the intended destination of the packet. If the destination compute instance is in the same subnet as the source compute instance, the VNIC associated with the source compute instance is configured to identify the VNIC associated with the destination compute instance and transfer the packet to that VNIC for processing. Subsequently, the execution of the VNIC associated with the destination compute instance transfers the packet to the destination compute instance.

[0054] When a packet is sent from a compute instance in one subnet to an endpoint in a different subnet of the same VCN, this communication is facilitated by the VNICs associated with the source and destination compute instances, as well as the VCN VR. For example, if compute instance C1 in subnet 1 in Figure 1 wants to send a packet to compute instance D1 in subnet 2, the packet is first processed by the VNIC associated with compute instance C1. The VNIC associated with compute instance C1 is configured to route the packet to the VCN VR using the default route or port 10.0.0.1 of VCN VR105. VCN VR105 is configured to route the packet to subnet 2 using port 10.1.0.1. Then, when the packet is received and processed by the VNIC associated with D1, this VNIC transfers the packet to compute instance D1.

[0055] When a packet is sent from a compute instance in VCN104 to an endpoint outside VCN104, this communication is facilitated by the VNIC associated with the source compute instance, VCN VR105, and the gateway associated with VCN104. VCN104 may be associated with one or more types of gateways. A gateway is an interface between a VCN and another endpoint, where the other endpoint is outside the VCN. A gateway is a Layer 3 / IP layer concept that enables a VCN to communicate with endpoints outside the VCN. Thus, a gateway facilitates the flow of traffic between a VCN and other VCNs or networks. For a VCN, different types of gateways can be configured to facilitate different types of communication with different types of endpoints. Depending on the gateway, the communication may be via a public network (e.g., the Internet) or a private network. Various communication protocols can be used for these communications.

[0056] For example, compute instance C1 may want to communicate with an endpoint outside of VCN104. The packet may first be processed by the VNIC associated with the source compute instance C1. By the processing of the VNIC, it is determined that the destination of the packet is outside of subnet 1 of C1. The VNIC associated with C1 may transfer the packet to VCN VR105 of VCN104. Thereafter, VCN VR105 processes the packet, and as part of the processing, based on the destination of the packet, determines a specific gateway associated with VCN104 as the next hop of the packet. Then, VCN VR105 may transfer the packet to the identified specific gateway. For example, if the destination is an endpoint within the customer's on-premises network, the packet may be transferred by VCN VR105 to DRG (Dynamic Routing Gateway) gateway 122 configured for VCN104. And by transferring the packet from the gateway to the next hop, the delivery of the packet to the final destination can be facilitated.

[0057] For a VCN, various different types of gateways can be set. Examples of gateways that can be set for a VCN are shown in FIG. 1 and will be described later. Examples of gateways associated with a VCN are also shown in FIGS. 16, 17, 18, and 19 (for example, the gateways referenced by reference numerals 1634, 1636, 1638, 1734, 1736, 1738, 1834, 1836, 1838, 1934, 1936, and 1938), and will be described later. As shown in the embodiment of FIG. 1, a DRG (Dynamic Routing Gateway) 122 is added or associated with the customer VCN 104 and can provide a path for private network traffic communication between the customer VCN 104 and another endpoint. As the other endpoint, the customer's on-premises network 116, a VCN 108 in a different region of the CSPI 101, or another remote cloud network 118 not hosted by the CSPI 101 is possible. The customer on-premises network 116 may be a customer's network constructed using the customer's resources or may be a customer's data center. Access to the customer on-premises network 116 is generally severely restricted. If the customer has both the on-premises network 116 and one or more VCNs 104 deployed or hosted in the cloud by the CSPI 101, the customer may want to enable communication between the on-premises network 116 and the cloud-based VCN 104. Thereby, the customer can construct an extended hybrid environment including the customer's VCN 104 hosted by the CSPI 101 and the on-premises network 116. It is the DRG 122 that enables such communication. To enable such communication, a communication channel 124 is set such that one endpoint of the channel exists in the customer on-premises network 116 and the other endpoint exists in the CSPI 101 and is connected to the customer VCN 104. The communication channel 124 can pass through a public communication network such as the Internet or a private communication network.Various different communication protocols may be used, such as IPsec VPN technology on a public communication network such as the Internet, Oracle's FastConnect technology that uses a private network instead of a public network, and so on. The device or equipment of the customer on-premises network 116 that constitutes one endpoint of the communication channel 124 is called a customer premise equipment (CPE), such as the CPE 126 shown in FIG. 1. On the CSPI 101 side, the endpoint may be a host machine that executes the DRG 122.

[0058] In certain embodiments, a remote peering connection (RPC) can be added to the DRG, and a customer can peer one VCN with another VCN in a different region. By using such an RPC, the customer VCN 104 can be connected to the VCN 108 in another region using the DRG 122. Also, by using the DRG 122, communication is possible with other remote cloud networks 118 that CSPI 101 does not host, such as the Microsoft Azure cloud, the Amazon AWS cloud, and so on.

[0059] As shown in FIG. 1, an Internet gateway (IGW) 120 may be configured for the customer VCN 104, and compute instances on the VCN 104 can communicate with a public endpoint 114 that is accessible via a public network such as the Internet. The IGW 120 is a gateway that connects the VCN to a public network such as the Internet. According to the IGW 120, a public subnet within the VCN such as the VCN 104 (when the resources in the public subnet have a public overlay IP address) can be directly accessible to a public endpoint 112 on the public network 114 such as the Internet. By using the IGW 120, a connection can be initiated from within a subnet in the VCN 104 or from the Internet.

[0060] The configuration of the Network Address Translation (NAT) gateway 128 for the customer's VCN 104 enables access to the Internet by cloud resources in the customer's VCN that do not have a dedicated public overlay IP address. However, in this case, the NAT gateway 128 does not expose these resources to a direct Internet connection (e.g., L4-L7 connection). As a result, the private subnets within the VCN (such as private subnet 1 of VCN 104) can have private access to public endpoints on the Internet. In the NAT gateway, connections are only initiated from the private subnet towards the public Internet and cannot be initiated from the Internet towards the private subnet.

[0061] In one embodiment, a service gateway (SGW) 126 can be configured for customer VCN 104, which provides a path for private network traffic between VCN 104 and service endpoints supported in service network 110. In one embodiment, service network 110 may be provided by a CSP and can provide various services. An example of such a service network is Oracle's Services Network, which provides various services that can be used by customers. For example, a compute instance (e.g., a database system) in a private subnet of customer VCN 104 can back up data to a service endpoint (e.g., Object Storage) without the need for a public IP address or access to the Internet. In one embodiment, a VCN can have only one SGW, and the connection can only be initiated from a subnet within the VCN, not from service network 110. When a VCN is peered with another VCN, resources in the other VCN typically cannot access the SGW. Also, resources in an on-premises network connected to a VCN via FastConnect or VPN Connect can use the service gateway configured for that VCN.

[0062] In one embodiment, SGW 126 uses the concept of a service CIDR (Classless Inter-Domain Routing) label, which is a string representing the regional public IP address range of the service or group of services of interest. Customers use the service CIDR label when configuring the SGW and related routing rules to control traffic to the service. Customers can optionally use the service CIDR label when setting security rules without the need for adjustment even if the public IP address of the service changes in the future.

[0063] The Local Peering Gateway (LPG) 132 is a gateway that can be added to the customer VCN 104, enabling the VCN 104 to peer with another VCN in the same region. Peering means that the VCNs communicate using private IP addresses, but traffic does not pass through a public network such as the Internet or route traffic to the customer's on-premises network 116. In a preferred embodiment, the VCN has a separate LPG for each peering established. Local peering or VCN peering is a common method used to establish network connections between different applications or infrastructure management functions.

[0064] Service providers (such as providers of services in the service network 110) can provide access to services using various access models. According to the public access model, the service may be made publicly available as a public endpoint that can be accessed publicly by compute instances in the customer VCN via a public network such as the Internet and / or may be privately accessible via the SGW 126. According to a specific private access model, the service can be accessed as a private IP endpoint in the private subnet of the customer's VCN. This is referred to as private endpoint (PE) access, and the service provider can expose each service as an instance in the customer's private network. The resources of the private endpoint represent the service within the customer VCN. Each PE is recognized as a VNIC of the subnet selected by the customer in the customer VCN (referred to as a PE-VNIC and having one or more private IPs). Therefore, the PE provides a way to present the service within the private customer VCN subnet using the VNIC. Since the endpoint is exposed as a VNIC, all functions associated with the VNIC, such as routing rules and security lists, are available to the PE VNIC.

[0065] Service providers can enable access through the PE by registering each service. The provider can associate a policy that restricts the visibility of the service to the customer tenancy with the service. The provider can register multiple services under a single virtual IP address (VIP), especially in the case of multi-tenant services. There may be multiple such private endpoints (across multiple VCNs) representing the same service.

[0066] And compute instances in the private subnet can access the service using the private IP address of the PE VNIC or the service DNS name. Compute instances in the customer VCN can access the service by sending traffic to the private IP address of the PE in the customer VCN. The Private Access Gateway (PAGW) 130 is a gateway resource that can be associated with a service provider VCN (such as the VCN in the service network 110) that acts as the input / output point for all traffic to the customer subnet private endpoint. According to the PAGW 130, the provider can scale the number of PE connections without using its internal IP address resource. The provider only needs to set up one PAGW for any number of services registered in a single VCN. The provider can represent a service as a private endpoint in one or more VCNs of multiple customers. From the customer's perspective, the PE VNIC appears to be associated with the service the customer wishes to interact with rather than with the customer's instance. Traffic destined for the private endpoint is routed to the service via the PAGW 130. These are referred to as customer-service private connections (C2S connections).

[0067] Also, by using the concept of PE, traffic can flow through the FastConnect / IPsec links and the private endpoints in the customer VCN, enabling the private access of the service to be extended to the customer's on-premises network and data center. Also, traffic can flow between the LPG132 and the PE in the customer VCN, enabling the private access of the service to be extended to the customer's peering VCN.

[0068] Since the customer can control the routing of the VCN at the subnet level, the customer can specify the subnets in the customer's VCN (such as VCN104) that use each gateway. To determine whether traffic through a specific gateway from the VCN is permitted, the route table of the VCN is used. For example, in a specific instance, the route table of the public subnet in the customer VCN104 may be configured to send non-local traffic through the IGW120. The route table of the private subnet in the same customer VCN104 may be configured to send traffic destined for the CSP service through the SGW126. All other traffic may be configured to be sent through the NAT gateway 128. The route table only controls the traffic going out of the VCN.

[0069] To control traffic entering a VCN via a gateway over an inbound connection, the security list associated with the VCN is used. All resources in a subnet use the same route table and security list. By using a security list, you can control the types of traffic that can enter and exit instances in a subnet of a VCN. Security list rules can include inbound (input) and outbound (output) rules. For example, an inbound rule can specify the permitted source address range, while an outbound rule can specify the permitted destination address range. A security rule can specify a particular protocol (such as TCP, ICMP), a particular port (such as 22 for SSH, 3389 for Windows RDP), and so on. In one embodiment, the operating system of an instance can enforce its own firewall rules in accordance with the security list rules. The rules can be stateful (for example, the connection is tracked and responses are automatically permitted without an explicit security list rule for the response traffic) or stateless.

[0070] Access from a customer VCN (i.e., by resources or compute instances deployed to VCN 104) can be classified as public access, private access, or dedicated access. Public access represents an access model in which a public IP address or NAT is used for access to a public endpoint. According to private access, a customer workload in VCN 104 having a private IP address (e.g., a resource in a private subnet) can access the service without passing through a public network such as the Internet. In certain embodiments, CSPI 101 enables a customer VCN workload having a private IP address to access the service (at its public service endpoint) using a service gateway. Thus, the service gateway provides a private access model by establishing a virtual link between the customer's VCN and the public endpoint of the service existing outside the customer's private network.

[0071] Also, CSPI can provide dedicated public access by using technologies such as FastConnect public peering, whereby a customer on-premises instance can access one or more services in the customer VCN without passing through a public network such as the Internet using a FastConnect connection. Further, CSPI can provide dedicated private access by using FastConnect private peering, whereby a customer on-premises instance having a private IP address can access the customer's VCN workload using a FastConnect connection. FastConnect is an alternative network connection to using the public Internet to connect a customer's on-premises network to CSPI and its services. FastConnect provides a simple, flexible, and cost-effective way to create a dedicated private connection with higher bandwidth options and reliability and a consistent networking experience compared to Internet-based connections.

[0072] FIG. 1 and the accompanying description describe various virtualized components in an exemplary virtual network. As described above, a virtual network is built on a basic physical network or substrate network. FIG. 2 is a simplified architecture diagram of the physical components in the physical network within the CSPI 200 that forms the basis of a virtual network according to an embodiment. As shown, the CSPI 200 provides a distributed environment that includes components and resources (e.g., compute, memory, and network resources) provided by a cloud service provider (CSP). These components and resources are used to provide cloud services (e.g., IaaS services) to registered customers, i.e., customers registered for one or more services provided by the CSP. For a customer, a subset of the resources of the CSPI 200 (e.g., compute, memory, and network resources) is provisioned based on the services registered by the customer. And the customer can use the physical compute, memory, and network resources provided by the CSPI 200 to build their respective cloud-based (i.e., hosted by the CSPI) customizable private virtual networks. As described above, these customer networks are referred to as virtual cloud networks (VCNs). A customer can deploy one or more customer resources, such as compute instances, to these customer VCNs. The compute instances can be in the form of virtual machines, bare metal instances, etc. The CSPI 200 provides the infrastructure and a set of complementary cloud services that enable customers to build and operate a wide range of applications and services in a hosted highly available environment.

[0073] In the exemplary embodiment shown in FIG. 2, the physical components of CSPI200 include one or more physical host machines or physical servers (e.g., 202, 206, 208), network virtualization devices (NVDs) (e.g., 210, 212), top-of-rack (TOR) switches (e.g., 214, 216), and a physical network (e.g., 218) and switches of the physical network 218. The physical host machine or server can host and execute various compute instances that participate in one or more subnets of the VCN. The compute instances can include virtual machine instances and bare metal instances. For example, the various compute instances shown in FIG. 1 may be hosted by the physical host machines shown in FIG. 2. The virtual machine compute instances in the VCN may be configured to be executed by one host machine or may be configured to be executed by multiple different host machines. Also, the physical host machine can host a virtual host machine, a container-based host, or functions, etc. The VNIC and VCN VR shown in FIG. 1 may be configured to be executed by the NVD shown in FIG. 2. The gateway shown in FIG. 1 may be configured to be executed by the host machine and / or NVD shown in FIG. 2.

[0074] The host machine or server may be configured to execute a hypervisor (also referred to as a virtual machine monitor or VMM) that creates and makes available a virtualized environment on the host machine. The virtualization or virtualized environment facilitates cloud-based computing. By the hypervisor on the host machine, one or more compute instances can be created, executed, and managed on the host machine. The hypervisor on the host machine enables sharing of the physical computing resources (e.g., compute, memory, and network resources) of the host machine among the various compute instances executed by the host machine.

[0075] For example, as shown in FIG. 2, host machines 202 and 208 each execute hypervisors 260 and 266. These hypervisors may be implemented by software, firmware, hardware, or a combination thereof. Usually, a hypervisor is placed on the operating system (OS) of a host machine and is a process or software layer that runs on the hardware processor of the host machine. The hypervisor provides a virtualized environment by enabling the sharing of the physical computing resources of the host machine (e.g., processing resources such as processors / cores, memory resources, network resources, etc.) among various virtual machine computing instances executed by the host machine. For example, in FIG. 2, hypervisor 260 may be placed on the OS of host machine 202 and enables the sharing of the computing resources (e.g., processing, memory, and network resources) of host machine 202 among the computing instances (e.g., virtual machines) executed by host machine 202. A virtual machine may have its own operating system (referred to as a guest operating system), which may be the same as or different from the OS of the host machine. The operating systems of the virtual machines executed by a host machine may be the same as or different from the operating systems of other virtual machines executed by the same host machine. Thus, the hypervisor enables multiple operating systems to run in parallel with each other while sharing the same computing resources of the host machine. The host machines shown in FIG. 2 may have the same type of hypervisor or different types of hypervisors.

[0076] As compute instances, virtual machine instances or bare metal instances are possible. In FIG. 2, compute instance 268 on host machine 202 and compute instance 274 on host machine 208 are examples of virtual machine instances. Host machine 206 is an example of a bare metal instance provided to a customer.

[0077] In some cases, an entire host machine may be provisioned for a single customer, and all one or more compute instances (virtual machines or bare metal instances) hosted by the host machine belong to the same customer. In other cases, a host machine may be shared among multiple customers (i.e., multiple tenants). In such a multi-tenant scenario, the host machine may host virtual machine compute instances belonging to different customers. These compute instances may be elements of different VCNs of different customers. In some embodiments, a bare metal compute instance is hosted by a bare metal server without a hypervisor. When a bare metal compute instance is provisioned, the host machine hosting the bare metal instance is not shared with other customers or tenants because a single customer or tenant maintains control of the physical CPUs, memory, and network interfaces of the host machine.

[0078] As described above, each compute instance that is part of the VCN is associated with a VNIC that can make the compute instance an element of the VCN's subnet. The VNIC associated with the compute instance facilitates the transmission of packets or frames to the compute instance. The VNIC is associated with the compute instance at the time the compute instance is created. In certain embodiments, in the case of a compute instance executed by a host machine, the VNIC associated with the compute instance is executed by the NVD connected to the host machine. For example, in FIG. 2, host machine 202 executes virtual machine compute instance 268 associated with VNIC 276, and VNIC 276 is executed by NVD 210 connected to host machine 202. In another example, bare metal instance 272 hosted by host machine 206 is associated with VNIC 280 executed by NVD 212 connected to host machine 206. In yet another example, VNIC 284 is associated with compute instance 274 executed by host machine 208, and VNIC 284 is executed by NVD 212 connected to host machine 208.

[0079] In the case of a compute instance hosted by a host machine, the NVD connected to the host machine also executes the VCN VR corresponding to the VCN of which the compute instance is an element. For example, in the embodiment shown in FIG. 2, NVD 210 executes VCN VR 277 corresponding to the VCN of which compute instance 268 is an element. Also, NVD 212 may execute one or more VCN VRs 283 corresponding to the VCNs corresponding to the compute instances hosted by host machines 206 and 208.

[0080] The host machine may include one or more network interface cards (NICs) that enable connection to other devices of the host machine. The NIC on the host machine may provide one or more ports (or interfaces) that enable a communication connection to another device of the host machine. For example, the host machine may be connected to the NVD by one or more ports (or interfaces) provided on the host machine and the NVD. Also, the host machine may be connected to other devices such as another host machine.

[0081] For example, in FIG. 2, the host machine 202 is connected to the NVD 210 by using the link 220 extending between the port 234 provided by the NIC 232 of the host machine 202 and the port 236 of the NVD 210. Also, the host machine 206 is connected to the NVD 212 by using the link 224 extending between the port 246 provided by the NIC 244 of the host machine 206 and the port 248 of the NVD 212. Also, the host machine 208 is connected to the NVD 212 by using the link 226 extending between the port 252 provided by the NIC 250 of the host machine 208 and the port 254 of the NVD 212.

[0082] On the other hand, the NVD is connected to a top-of-rack (TOR) switch via a communication link, which is connected to a physical network 218 (also referred to as a switch fabric). In certain embodiments, the links between the host machine and the NVD and between the NVD and the TOR switch are Ethernet links. For example, in FIG. 2, the NVDs 210 and 212 are connected to the TOR switches 214 and 216, respectively, by using the links 228 and 230. In certain embodiments, the links 220, 224, 226, 228, and 230 are Ethernet links. The set of host machines and NVDs connected to the TOR may sometimes be referred to as a rack.

[0083] The physical network 218 provides a communication fabric that enables communication between the TOR switches. As the physical network 218, a multi-layer network is possible. In one embodiment, the physical network 218 is a multi-layer Clos network, and the TOR switches 214 and 216 represent the leaf-level nodes of the multi-layer multi-node physical switching network 218. Various Clos network configurations are possible, including, but not limited to, 2-layer networks, 3-layer networks, 4-layer networks, 5-layer networks, and generally "n" layer networks. An example of a Clos network is shown in and described below with reference to FIG. 5.

[0084] Various different connection configurations are possible between the host machine and the NVD, such as a one-to-one configuration, a many-to-one configuration, and a one-to-many configuration. In one embodiment of the one-to-one configuration, each host machine is connected to its own separate NVD. For example, in FIG. 2, the host machine 202 is connected to the NVD 210 via the NIC 232 of the host machine 202. In the many-to-one configuration, a plurality of host machines are connected to one NVD. For example, in FIG. 2, the host machines 206 and 208 are each connected to the same NVD 212 via the NICs 244 and 250.

[0085] In a one-to-many configuration, one host machine is connected to multiple NVDs. Figure 3 shows an example within CSPI300 where a host machine is connected to multiple NVDs. As shown in Figure 3, host machine 302 is equipped with a network interface card (NIC) 304 that includes multiple ports 306 and 308. Host machine 300 is connected to a first NVD 310 via port 306 and link 320, and is also connected to a second NVD 312 via port 308 and link 322. Ports 306 and 308 may be Ethernet ports, and links 320 and 322 between host machine 302 and NVDs 310 and 312 may be Ethernet links. And, NVD 310 is connected to a first TOR switch 314, and NVD 312 is connected to a second TOR switch 316. The links between NVDs 310 and 312 and TOR switches 314 and 316 may be Ethernet links. TOR switches 314 and 316 represent layer 0 switching devices of a multi-layer physical network 318.

[0086] The configuration shown in Figure 3 provides two separate physical network paths from the physical switch network 318 to host machine 302 (a first path passing through TOR switch 314 to NVD 310 and then to host machine 302, and a second path passing through TOR switch 316 to NVD 312 and then to host machine 302). These separate paths can enhance the availability of host machine 302 (referred to as high availability). If there is a problem with one of the paths (for example, if one of the links in the path is down) or if there is a problem with one of the devices (for example, if a specific NVD is not functioning), the other path may be used for communication with host machine 302.

[0087] In the configuration shown in FIG. 3, the host machine is connected to two different NVDs by using two different ports provided by the NIC of the host machine. In other embodiments, the host machine may include multiple NICs that enable connections to multiple NVDs.

[0088] Referring again to FIG. 2, the NVD is a physical device or component that executes one or more network and / or storage virtualization functions. The NVD can be any device having one or more processing units (e.g., CPU, network processing unit (NPU), FPGA, packet processing pipeline, etc.), memory including a cache, and ports. The various virtualization functions may be executed by software / firmware executed by one or more processing units of the NVD.

[0089] The NVD may be implemented in various different forms. For example, in one embodiment, the NVD is implemented as an interface card referred to as a smart NIC or intelligent NIC with a built-in processor. The smart NIC is a device separate from the NIC on the host machine. In FIG. 2, NVDs 210 and 212 may be implemented as smart NICs connected to host machine 202 and host machines 206 and 208, respectively.

[0090] However, the Smart NIC is only an example of an implementation of the NVD. Various other implementations are possible. For example, in some other implementations, the NVD or one or more functions performed by the NVD may be incorporated into one or more host machines, one or more TOR switches, and other components of the CSPI200, or may be performed thereby. For example, the NVD may be embodied in a host machine, and the functions performed by the NVD may be performed by the host machine. As another example, the NVD may be part of a TOR switch, or the TOR switch may be configured to perform the functions performed by the NVD that enable various complex packet conversions used in a public cloud. The TOR that performs the functions of the NVD may sometimes be referred to as a Smart TOR. Further, in some other implementations, when a virtual machine (VM) instance rather than a bare metal (BM) instance is provided to a customer, the functions performed by the NVD may be realized inside the hypervisor of the host machine. In some other implementations, some of the functions of the NVD may be offloaded to a centralized service operating on a group of host machines.

[0091] In some embodiments, such as when implemented as a Smart NIC as shown in FIG. 2, the NVD may include a plurality of physical ports that enable the NVD to connect to one or more host machines and one or more TOR switches. The ports on the NVD can be classified as host-side ports (also referred to as "south ports") or network-side or TOR-side ports (also referred to as "north ports"). The host-side ports of the NVD are the ports used to connect the NVD to the host machine. Examples of host-side ports in FIG. 2 include port 236 on NVD210 and ports 248 and 254 on NVD212. The network-side ports of the NVD are the ports used to connect the NVD to the TOR switch. Examples of network-side ports in FIG. 2 include port 256 on NVD210 and port 258 on NVD212. As shown in FIG. 2, NVD210 is connected to TOR switch 214 by a link 228 extending from port 256 of the NVD210 to the TOR switch 214. Similarly, NVD212 is connected to TOR switch 216 by a link 230 extending from port 258 of the NVD212 to the TOR switch 216.

[0092] The NVD can receive packets and frames from the host machine (e.g., packets and frames generated by a compute instance hosted by the host machine) via the host-side port, perform the necessary packet processing, and then transfer the packets and frames to the TOR switch via the network-side port of the NVD. The NVD can receive packets and frames from the TOR switch via the network-side port of the NVD, perform the necessary packet processing, and then transfer the packets and frames to the host machine via the host-side port of the NVD.

[0093] In some embodiments, there may be multiple ports and associated links between the NVD and the TOR switch. These ports and links may be aggregated to form a link aggregator group (referred to as a LAG) consisting of multiple ports or links. According to the aggregation of the links, multiple physical links between two endpoints (for example, between the NVD and the TOR switch) can be processed as a single logical link. All of the physical links in a given LAG may be configured to operate in full-duplex mode at the same speed. The LAG helps to increase the bandwidth and reliability of the connection between two endpoints. If one of the physical links of the LAG goes down, traffic is dynamically and transparently reallocated to one of the other physical links of the LAG. The aggregated physical links provide a wider bandwidth than each individual link. The multiple ports associated with the LAG are processed as a single logical port. Traffic can be load-balanced across the multiple physical links of the LAG. One or more LAGs may be set up between two endpoints. The two endpoints may be, for example, between the NVD and the TOR switch, or between the host machine and the NVD.

[0094] The NVD implements or executes network virtualization functions. These functions are executed by the software / firmware that the NVD runs. Examples of network virtualization functions include, but are not limited to, packet encapsulation and decapsulation functions, functions for generating a VCN network, functions for implementing network policies such as VCN security list (firewall) functions, functions for facilitating packet routing and forwarding to compute instances in a VCN, etc. In one embodiment, when receiving a packet, the NVD is configured to execute a packet processing pipeline for processing the packet and determining a method for packet forwarding or routing. As part of this packet processing pipeline, the NVD may execute the VNIC associated with a compute instance in the VCN, execute the virtual router (VR) associated with the VCN, perform packet encapsulation and decapsulation to facilitate transfer or routing in the virtual network, execute a certain gateway (e.g., a local peering gateway), implement security lists, network security groups, network address translation (NAT) functions (e.g., conversion from public IP to private IP at the host level), throttling functions, and other functions, and may execute one or more virtual functions associated with the overlay network.

[0095] In one embodiment, the packet processing data path of the NVD may comprise a plurality of packet pipelines each configured in a series of packet transformation stages. In one embodiment, upon receiving a packet, the packet is parsed and classified into a single pipeline. Thereafter, the packet is processed linearly, one stage at a time, until it is either discarded or sent out via the NVD interface. These stages provide basic functional packet processing building blocks (e.g., header verification, throttle enforcement, insertion of a new layer 2 header, L4 firewall enforcement, VCN encapsulation / decapsulation, etc.), so new pipelines can be constructed by assembling existing stages and new functionality can be added by generating new stages and inserting them into existing pipelines.

[0096] The NVD can execute both control plane functions and data plane functions corresponding to the control plane and data plane of the VCN. Examples of the VCN control plane are also shown in FIGS. 16, 17, 18, and 19 (see reference numbers 1616, 1716, 1816, and 1916), and will be described later. Examples of the VCN data plane are shown in FIGS. 16, 17, 18, and 19 (see reference numbers 1618, 1718, 1818, and 1918), and will be described later. The control plane functions include functions used for network configuration (for example, routing and route table configuration, VNIC configuration, etc.) and functions for controlling the data transfer method. In one embodiment, a VCN control plane is provided that centrally calculates all overlay-substrate mappings and publishes them to the NVD and virtual network edge devices (various gateways such as DRG, SGW, IGW, etc.). Also, firewall rules can be published by the same mechanism. In one embodiment, the NVD obtains only the mappings related to the NVD. The data plane functions include functions for the actual routing / transfer of packets based on the configuration set using the control plane. The VCN data plane is realized by encapsulating the customer's network packets before passing through the underlying network. The encapsulation / decapsulation function is realized on the NVD. In one embodiment, the NVD is configured to intercept all network packets input / output to / from the host machine and execute network virtualization functions.

[0097] As described above, the NVD executes various virtualization functions including VNIC and VCN VR. The NVD can execute a VNIC associated with a compute instance hosted by one or more host machines connected to the VNIC. For example, as shown in FIG. 2, the NVD 210 executes the functions of the VNIC 276 associated with the compute instance 268 hosted by the host machine 202 connected to the NVD 210. As another example, the NVD 212 executes the VNIC 280 associated with the bare metal compute instance 272 hosted by the host machine 206 and also executes the VNIC 284 associated with the compute instance 274 hosted by the host machine 208. A host machine can host compute instances belonging to different VCNs belonging to different customers, and the NVD connected to the host machine can execute the VNICs corresponding to these compute instances (i.e., can execute VNIC-related functions).

[0098] Also, the NVD executes a VCN virtual router corresponding to the VCN of the compute instance. For example, in the embodiment shown in FIG. 2, the NVD 210 executes the VCN VR 277 corresponding to the VCN to which the compute instance 268 belongs. The NVD 212 executes one or more VCN VRs 283 corresponding to one or more VCNs to which the compute instances hosted by the host machines 206 and 208 belong. In an embodiment, the VCN VR corresponding to the VCN is executed by all NVDs connected to the host machine hosting at least one compute instance belonging to the VCN. When a host machine hosts compute instances belonging to different VCNs, the NVD connected to the host machine can execute the VCN VRs corresponding to the different VCNs.

[0099] In addition to VNIC and VCN VR, the NVD may also be made to execute various software (e.g., daemons), and may also include one or more hardware components that facilitate the various network virtualization functions executed by the NVD. For simplicity, these various components are grouped together as the "packet processing components" shown in FIG. 2. For example, NVD210 includes packet processing component 286, and NVD212 includes packet processing component 288. For example, the packet processing components of the NVD may include a packet processor configured to interact with the ports and hardware interfaces of the NVD to monitor all packets received and communicated by the use of the NVD, and to store network information. The network information may include, for example, network flow information for identifying various network flows processed by the NVD and flow-by-flow information (e.g., flow-by-flow statistics). In certain embodiments, the network flow information may be stored per VNIC. The packet processor may perform packet-level operations and may also implement stateful NAT and L4 firewall (FW). As another example, the packet processing component may include a replication agent configured to replicate the information stored by the NVD to one or more different replicated target stores. As yet another example, the packet processing component may include a logging agent configured to perform a logging function for the NVD. Also, the packet processing component may include software for monitoring the performance and health of the NVD and, in some cases, the status and health of other components connected to the NVD.

[0100] FIG. 1 shows the components of an exemplary virtual or overlay network that includes a VCN, subnets within the VCN, compute instances deployed in the subnets, VNICs associated with the compute instances, the VR of the VCN, and a set of gateways configured for the VCN. The overlay components shown in FIG. 1 may be executed or hosted by one or more of the physical components shown in FIG. 2. For example, a compute instance in a VCN may be executed or hosted by one or more of the host machines shown in FIG. 2. In the case of a compute instance hosted by a host machine, the VNIC associated with the compute instance is typically executed by an NVD connected to the host machine (i.e., the VNIC functionality is provided by an NVD connected to the host machine). The VCN VR functionality of the VCN is executed by all NVDs connected to the host machine that hosts or executes a compute instance that is part of the VCN. The gateway associated with the VCN may be executed by one or more different types of NVDs. For example, one gateway may be executed by a smart NIC, while other gateways may be executed by one or more host machines or other implementations of NVDs.

[0101] As described above, a compute instance in a customer VCN can communicate with various different endpoints, which can be endpoints within the same subnet as the source compute instance, endpoints in a different subnet but within the same VCN as the source compute instance, or endpoints outside the VCN of the source compute instance. These communications are facilitated by the use of the VNIC associated with the compute instance, the VCN VR, and the gateway associated with the VCN.

[0102] Regarding communication between two compute instances on the same subnet in a VCN, this communication is facilitated by the use of VNICs associated with the source and destination compute instances. The source and destination compute instances may be hosted by the same host machine or by different host machines. Packets originating from the source compute instance can be transferred from the host machine hosting the source compute instance to the NVD connected to that host machine. On the NVD, the packets are processed by using a packet processing pipeline, which may include the execution of the VNIC associated with the source compute instance. Since the destination endpoint of the packet is within the same subnet, the execution of the VNIC associated with the source compute instance transfers the packet to the NVD that executes the VNIC associated with the destination compute instance. Then, the NVD processes the packet and transfers it to the destination compute instance. The VNICs associated with the source and destination compute instances may be configured to execute on the same NVD (e.g., if both the source and destination compute instances are hosted by the same host machine) or on different NVDs (e.g., if the source and destination compute instances are hosted by different host machines connected to different NVDs). The VNIC may use the routing / forwarding table stored by the NVD to determine the next hop of the packet.

[0103] When a packet is sent from a compute instance in a subnet to an endpoint in a different subnet of the same VCN, the packet originating from the source compute instance is sent from the host machine hosting the source compute instance to the NVD connected to that host machine. On the NVD, the packet is processed by using a packet processing pipeline, which may include the execution of one or more VNICs as well as VRs associated with the VCN. For example, as part of the packet processing pipeline, the NVD performs the execution or invocation of functions corresponding to the VNIC associated with the source compute instance (also referred to as VNIC execution). The functions performed by the VNIC may include verifying the VLAN tag on the packet. Since the destination of the packet is outside the subnet, next, the VCN VR function is called and executed by the NVD. Then, the VCN VR routes the packet to the NVD that executes the VNIC associated with the destination compute instance. Thereafter, the VNIC associated with the destination compute instance processes the packet and transfers it to the destination compute instance. The VNICs associated with the source and destination compute instances may be executed on the same NVD (for example, when both the source and destination compute instances are hosted by the same host machine) or may be executed on different NVDs (for example, when the source and destination compute instances are hosted by different host machines connected to different NVDs).

[0104] If the destination of a packet is outside the VCN of the source compute instance, the packet originating from the source compute instance is sent from the host machine hosting the source compute instance to the NVD connected to that host machine. The NVD runs the VNIC associated with the source compute instance. Since the destination endpoint of the packet is outside the VCN, the packet is then processed by the VCN VR of that VCN. The NVD calls the VCN VR function, as a result of which the packet is transferred to the NVD running the appropriate gateway associated with the VCN. For example, if the destination is an endpoint within the customer's on-premises network, the packet can be transferred by the VCN VR to the NVD running the DRG gateway configured for the VCN. The VCN VR may be run on the same NVD as the NVD running the VNIC associated with the source compute instance, or it may be run by a different NVD. The gateway can be run by the NVD, which can be a smart NIC, a host machine, or other embodiments of the NVD. Thereafter, the packet is processed by the gateway and transferred to the next hop that facilitates the transmission of the packet to the intended destination endpoint. For example, in the embodiment shown in FIG. 2, a packet originating from compute instance 268 may be sent from host machine 202 to NVD 210 via link 220 (by using NIC 232). In NVD 210, VNIC 276 is called because it is associated with the source compute instance 268. VNIC 276 examines the information encapsulated in the packet and, after determining the next hop to which the packet is to be transferred to facilitate the transmission of the packet to the intended destination endpoint, is configured to transfer the packet to the determined next hop.

[0105] Compute instances deployed in a VCN can communicate with a variety of different endpoints. These endpoints can include endpoints hosted by CSPI200 and endpoints outside of CSPI200. Endpoints hosted by CSPI200 can include instances in the same VCN or other VCNs (which may be the customer's VCN or a VCN not belonging to the customer). Communication between endpoints hosted by CSPI200 may be configured to occur via the physical network 218. Also, a compute instance can communicate with endpoints that CSPI200 does not host (i.e., outside of CSPI200). Examples of such endpoints include endpoints within a customer's on-premises network or data center or public endpoints accessible via a public network such as the Internet. Communication with endpoints outside of CSPI200 may be configured to occur via a public network (e.g., the Internet) (not shown in FIG. 2) or via a private network (not shown in FIG. 2) using various communication protocols.

[0106] The architecture of CSPI200 shown in FIG. 2 is merely an example and is not intended to be limiting in any way. In alternative embodiments, variations, alternatives, and improvements are possible. For example, in some embodiments, CSPI200 may have more or fewer systems or components than shown in FIG. 2, may combine two or more systems, or may have a different system configuration or arrangement. The systems, subsystems, and other components shown in FIG. 2 may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of each system, in hardware, or in a combination of these. The software may be stored on a non-transitory storage medium (e.g., a memory device).

[0107] Figure 4 shows the connection between a host machine and an NVD for providing I / O virtualization that supports multi-tenancy according to an embodiment. As shown in Figure 4, host machine 402 runs hypervisor 404 that provides a virtualized environment. Host machine 402 runs two virtual machine instances (VM1 406 belonging to customer / tenant #1 and VM2 408 belonging to customer / tenant #2). Host machine 402 includes a physical NIC 410 connected to NVD 412 via link 414. Each compute instance is associated with a VNIC executed by NVD 412. In the embodiment of Figure 4, VM1 406 is associated with VNIC-VM1 420, and VM2 408 is associated with VNIC-VM2 422.

[0108] As shown in Figure 4, NIC 410 includes two logical NICs (logical NIC A 416 and logical NIC B 418). Each virtual machine is configured to cooperate in association with its own logical NIC. For example, VM1 406 is associated with logical NIC A 416, and VM2 408 is associated with logical NIC B 418. Although host machine 402 includes only one physical NIC 410 shared by multiple tenants, due to the logical NICs, each tenant's virtual machine recognizes itself as having a host machine and a NIC.

[0109] In one embodiment, each logical NIC is assigned its own VLAN ID. Thus, for tenant #1, a specific VLAN ID is assigned to logical NIC A416, and for tenant #2, a separate VLAN ID is assigned to logical NIC B418. When a packet is sent from VM1 406, the hypervisor attaches the tag assigned to tenant #1 to the packet, and the packet is then sent from host machine 402 to NVD412 via link 414. Similarly, when a packet is sent from VM2 408, the hypervisor attaches the tag assigned to tenant #2 to the packet, and the packet is then sent from host machine 402 to NVD412 via link 414. Therefore, the packet 424 sent from host machine 402 to NVD412 is associated with a tag 426 that identifies the specific tenant and the associated VM. On the NVD, for the packet 424 received from host machine 402, the use of the tag 426 associated with the packet determines whether the packet should be processed by VNIC-VM1 420 or VNIC-VM2 422. Then, the packet is processed by the corresponding VNIC. According to the configuration shown in FIG. 4, the compute instances of each tenant can be recognized as each having a host machine and a NIC. The setting shown in FIG. 4 provides I / O virtualization to support multi-tenancy.

[0110] FIG. 5 is a simplified block diagram of a physical network 500 according to an embodiment. The embodiment shown in FIG. 5 is structured as a Clos network. A Clos network is a particular type of network topology designed to provide connection redundancy while maintaining a wide two-way bandwidth and maximum utilization of resources. A Clos network is a type of non-blocking multi-stage or multi-layer switching network, and the number of stages or layers can be two, three, four, five, etc. The embodiment shown in FIG. 5 is a three-layer network including layers 1, 2, and 3. The TOR switch 504 represents a layer 0 switch in the Clos network. One or more NVDs are connected to the TOR switch. The layer 0 switch is also referred to as an edge device of the physical network. The layer 0 switch is connected to a layer 1 switch, also referred to as a leaf switch. In the embodiment shown in FIG. 5, a set of "n" layer 0 TOR switches is connected to a set of "n" layer 1 switches, forming a pod integrally. Each layer 0 switch in a pod is interconnected with all the layer 1 switches in that pod, but there is no connection between switches in different pods. In one embodiment, two pods are referred to as a block. Each block is served or connected to by a set of "n" layer 2 switches (sometimes referred to as spine switches). There can be multiple blocks in the physical network topology. And the layer 2 switches are connected to "n" layer 3 switches (sometimes referred to as super spine switches). Communication of packets through the physical network 500 is typically performed by using one or more layer 3 communication protocols. Usually, since all layers of the physical network except the TOR layer have n-way redundancy, high availability can be achieved. To enable scaling of the physical network, the visibility between switches in the physical network may be controlled by specifying policies for pods and blocks.

[0111] As a characteristic of the Clos network, the maximum hop count reaching from one layer-0 switch to another layer-0 switch (or reaching from an NVD connected to a layer-0 switch to another NVD connected to a layer-0 switch) is fixed. For example, in a three-layer Clos network, the maximum number of hops required for a packet to reach from one NVD to another NVD is at most 7, provided that the source and destination NVDs are connected to the leaf layer of the Clos network. Similarly, in a four-layer Clos network, the maximum number of hops required for a packet to reach from one NVD to another NVD is at most 9, provided that the source and destination NVDs are connected to the leaf layer of the Clos network. Therefore, the Clos network architecture maintains consistent latency across the entire network, which is important for communication within and between data centers. The Clos topology is horizontally scalable and cost-effective. The bandwidth / throughput capacity of the network can be easily increased by adding more switches (e.g., more leaf and spine switches) in different layers and increasing the number of links between adjacent layer switches.

[0112] In one embodiment, each resource within the CSPI is assigned a unique identifier called a cloud identifier (CID). This identifier is included as part of the resource's information and can be used, for example, when managing the resource via a console or API. An exemplary syntax of the CID is as follows. ocid1.<RESOURCE TYPE>. <realm>.[REGION].[FUTURE USE].<UNIQUE ID> Here, ocid1 is a string indicating the version of the CID, RESOURCE TYPE is the type of resource (e.g., instance, volume, VCN, subnet, user, group, etc.), REALM is the realm where the resource exists (exemplary values are "c1" for the commercial realm, "c2" for the government cloud realm, or "c3" for the federal government cloud realm, etc., and each realm may have its own domain name), REGION is the region where the resource exists (this part may be blank if the region cannot be applied to the resource), FUTURE USE is a reservation for future use, UNIQUE ID is the unique part of the ID (the format may vary depending on the type of resource or service).

[0113] In a container environment (e.g., a container orchestration system such as Kubernetes), a pod is an abstract concept representing a group of one or more containers that have shared resources including a namespace, network resources (such as an IP address), and storage components (such as a storage volume, a file system volume, etc.). Also, a pod includes instructions (instructions including available ports, the version of the container image, etc.) that detail how the containers are to be run. The contents of a pod are always placed in the same location and are scheduled in the same way.

[0114] In the current method for performing pod communication, bridging and encapsulation are required. For example, when pods located on different nodes (also known as hosts) within a container environment attempt to communicate with each other, such communication needs to be routed between hosts using an overlay network by encapsulating such communication with a bridge device. As a result, these pods cannot be directly identified / distinguished in the flow log for encapsulation. In addition, the execution of encapsulation incurs a disadvantage in execution because, in addition to additional processing resources for execution, it is necessary to add encapsulation data to the packets to be transmitted.

[0115] The present disclosure provides a technique for implementing a container environment in which each internal pod is provided with a unique IP address and a virtual communication device such as an IPvlan device. For example, the IPvlan device may include a Linux (registered trademark) networking device. The IPvlan device may be generated in a network namespace, and another device may be designated as the parent device of the IPvlan. Each time a network request is sent to the IPvlan device, the parent device is operable as a bridge / router and will route the traffic to an appropriate location (the parent device knows all the connected child IPvlan devices). Without the need for bridging and encapsulation, communication from the source pod is directly routed to the destination pod within the container environment by one or more virtualized network interface cards (VNICs) that utilize the unique IP address of the destination pod. Each VNIC may include an interface device used by a virtual machine to connect to a virtual cloud network.

[0116] In this way, by using the IP address assigned to the pod, the VNIC can route all traffic to and from the pod through the virtual communication device assigned to the pod. Subsequently, while this traffic maintains the IP address associated with the pod, rule execution within the pod remains possible. Furthermore, such traffic can be routed by the VNIC without the need to perform encapsulation or pass through a bridge device.

[0117] As a result, (for example, by eliminating the overhead required during encapsulation, which is additional to the transmission of network packets) the size of the data to be transmitted becomes smaller, and (for example, by eliminating the computational cost required to perform encapsulation of the data during transmission) the amount of computation performed by the hardware implementing the container environment decreases. Also, (for example, by eliminating the need to perform encapsulation during data transmission) the latency of data transmission within the container environment is reduced, and the performance of the hardware implementing communication within the container environment is improved.

[0118] Container orchestration systems such as Kubernetes enable the implementation and management of software. Kubernetes provides an open-ended interface for cloud providers to define how pod communication should function (including communication from pod to pod, from pod to host, from pod to the Internet, and from pod to itself). This "open-ended interface" is called the Container Network Interface (CNI) and may include Kubernetes specifications for managing network resources on the cluster. According to this specification, Kubernetes can interact with a wide range of CNI-based software solutions. Also, according to the CNI plugin, the Kubernetes cluster can be integrated with a specific network technology to comply with the CNI specification. Additionally, Kubernetes has the concept of Kubernetes Services, which is an abstract way of referring to a group of pods (for example, a service may be defined such that when a network request is sent to the service, the request is routed to multiple predetermined pods).

[0119] This disclosure describes a technique for clearly defining how pod communication functions within a Kubernetes environment and a Virtual Cloud Network (VCN). Here, communication between pods and with the VCN is performed efficiently, and the path of network traffic can be easily traced.

[0120] In one exemplary solution, for each pod, an IPvlan device and a point-to-point device (ptp) are provided. The ptp device is only used for transmission to the node / host. For example, the ptp device may include a Linux networking device that connects two network namespaces and enables communication from the pod to the host. In other cases, all other traffic passes through the IPvlan device. IPvlan can operate in L3S mode, which means that the host IPtable is processed. The IPtable may include filtering software on the compute instance that filters network packets and modifies the packet source / destination.

[0121] In some embodiments, the traffic flow for a pod may proceed as follows. That is, a pod sends a network request to another pod. The network request originates from the IPvlan device of the sending pod. This network packet will be converted by the host IPtable as needed (e.g., when the request is sent to the Kubernetes service instead of directly to the pod). After converting the packet, the traffic exits from the parent device of the IPvlan (e.g., the VLAN assigned to the IPvlan device) and is routed to the next router. This router will decide whether to (1) route the traffic back to the pod on the original host (also known as the use of hairpin) or, (2) transfer the traffic to the destination if the packet is destined for another destination. The pod that sends the network response will also execute a traffic flow similar to the above steps.

[0122] FIG. 6 is a block diagram of a container environment 600 for facilitating direct pod - to - pod communication according to at least one embodiment. The illustrated exemplary container environment 600 may include a data plane of a container orchestration system (such as Kubernetes), and such a data plane executes a workload in response to instructions received from a separate control plane (not shown). As shown, the exemplary container environment 600 includes a first node 602 and a second node 604. Of course, in other exemplary embodiments, more nodes may be included or fewer nodes may be included.

[0123] Also, each of the first node 602 and the second node 604 may include a virtual or physical machine that includes one or more services required for the implementation of one or more pods. Each node (also known as a host) may include a physical or virtual machine that hosts and operates pods in a Kubernetes cluster. In one example, components on a node include a Kubelet (an agent on each node that enables containers to operate in a pod), a container runtime, and a Kube proxy (a network proxy that operates on each node). As shown, the first node 602 implements a first pod 606, a second pod 608, and a third pod 610, while the second node 604 implements a fourth pod 612. Each of the pods 606 - 612 includes containers 614 - 620, and each container includes one or more applications to be executed, as well as related libraries and dependencies. Each of the pods 606 - 612 further includes shared storage / network resources and specifications regarding how the containers 614 - 620 operate as described above.

[0124] Furthermore, each of the pods 606 - 612 includes a virtual communication device 622 - 628 respectively, and each virtual communication device 622 - 628 is configured for its respective pod 606 - 612. In one embodiment, each virtual communication device 622 - 628 may include an Internet Protocol virtual local area network (IPvlan) device. For example, each virtual communication device 622 - 628 can facilitate communication between its respective pod 606 - 612 and other pods within the container environment 600, as well as external networks such as virtual network interface cards (VNICs) 638 - 642, virtual cloud networks 644, etc. Also, each pod 606 - 612 is assigned a unique Internet Protocol (IP) address for direct routing of data communication to the pod via its respective virtual communication device 622 - 628 and one or more VNICs 638 - 642.

[0125] Furthermore, one of the VNICs 638 - 642 is assigned as the parent device to each virtual communication device 622 - 628. For example, the second VNIC 632 is assigned as the parent device to the virtual communication device 622 of the first pod 606, the fourth VNIC 636 is assigned as the parent device to the virtual communication devices 624 of the second pod 608 and 626 of the third pod 610, and the seventh VNIC 642 is assigned as the parent device to the virtual communication device 628 of the fourth pod 612. Each VNIC 638 - 642 stores the IP address of each pod 606 - 612 to which the VNIC 638 - 642 is assigned. Also, each of the pods 606 - 612 includes a veth pair 646A - B - 652A - B respectively. Each veth pair 646A - B - 652A - B includes a network connection that connects the container network within each pod 606 - 612 to the respective node network of each node 602 - 604. For example, the first veth 646A - 652A of the veth pair can be used for output traffic, and the second veth 646B - 652B can be used for input traffic.

[0126] Also, a first VNIC 630, a second VNIC 632, a third VNIC 634, a fourth VNIC 636, and a fifth VNIC 638 are connected to the first node 602. And a sixth VNIC 640 and a seventh VNIC 642 are connected to the second node 604. Each of these VNICs 630 to 642 can implement a networking communication function for their respective nodes 602 and 604. In one embodiment, the first VNIC 630 may include a primary parent VNIC for the first node 602, while the second, third, fourth, and fifth VNICs 632 to 638 may include secondary parent VNICs for the first node 602. Similarly, the sixth VNIC 640 may include a primary parent VNIC for the second node 604, while the seventh VNIC 642 may include a secondary parent VNIC for the second node 604. The first / primary parent VNIC 630 for the first node 602 communicates with the sixth / primary parent VNIC 640 for the second node 604, while the second to fifth / secondary parent VNICs 632 to 638 for the first node 602 communicate with the seventh / secondary parent VNIC 642 for the second node 604.

[0127] As described herein, FIGS. 7 to 9 show various exemplary data flows within the exemplary container environment 600 shown above.

[0128] More specifically, FIG. 7 is a block diagram showing an exemplary request flow 700 between two pods sharing a single VNIC within the exemplary container environment 600 of FIG. 6 according to at least one embodiment. More specifically, the third pod 610 and the second pod 608 communicate with the fourth VNIC 636 via a connection. When a request for data addressed to the second pod 608 (e.g., a request including the IP address of the second pod 608) is sent by the container 618 disposed within the third pod 610, this request is first sent through the veth pair 650A - B of the third pod 610 such that one or more third - party interfaces can perform one or more evaluation operations on the request. Thereafter, the request is sent from the virtual communication device 626 of the third pod 610 to the fourth VNIC 636.

[0129] If the request is received at the fourth VNIC 636, it is routed through the fourth VNIC 636 by utilizing another hairpin operation such that the information associated with the request is stored in one or more flow logs. This information may include the destination IP address (e.g., the IP address of the second pod 608), the source IP address (e.g., the IP address of the third pod 610), etc. After performing the hairpin operation, the fourth VNIC 636 routes the request to the virtual communication device 624 of the second pod 608 by utilizing the IP address of the second pod 608 included in the request. Thereafter, the request is sent from the virtual communication device 624 of the second pod 608 through the veth pair 648A - B of the second pod 608 by the container 616 disposed within the second pod 608 such that one or more third - party interfaces can perform one or more evaluation operations on the request, and then is received by the container 616 disposed within the second pod 608.

[0130] FIG. 8 is a block diagram illustrating an exemplary flow of responses 800 between two pods sharing a single VNIC within the exemplary container environment 600 of FIG. 6, according to at least one embodiment. For example, a third pod 610 and a second pod 608 communicate with a fourth VNIC 636 via a connection. In response to receiving a data request in container 616 disposed within the second pod 608 from container 618 of the third pod 610, container 616 of the second pod 608 generates a response (including the IP address of the third pod 610) to the request addressed to the third pod 610. This response is sent by container 616 to the fourth VNIC 636 via the virtual communication device 624 of the second pod 608.

[0131] If the response is received by the fourth VNIC 636, it is routed through the fourth VNIC 636 by utilizing another hairpin operation such that the information associated with the response is stored in one or more flow logs. This information may include the destination IP address (e.g., the IP address of the third pod 610), the source IP address (e.g., the IP address of the second pod 608), and the like. After performing the hairpin operation, the fourth VNIC 636 routes the response to the virtual communication device 626 of the third pod 610 by utilizing the IP address of the third pod 610 included in the response. Thereafter, the response is sent from the virtual communication device 626 of the third pod 610 through the veth pair 650A - B of the third pod 610 such that one or more third - party interfaces may perform one or more evaluation operations on the response, and then is received by container 618 disposed within the third pod 610.

[0132] Furthermore, FIG. 9 is a block diagram showing an exemplary request flow 900 between two pods communicating via different VNICs within the same node 602 of the exemplary container environment 600 of FIG. 6, according to at least one embodiment. More specifically, the first pod 606 communicates with the second VNIC 632 of the first node 602 via a connection, and the second pod 608 communicates with the fourth VNIC 636 of the first node 602 via a connection. When a request for data addressed to the first pod 606 (e.g., a request including the IP address of the first pod 606) is sent by a container 616 disposed within the second pod 608, this request is first sent through the veth pair 648A-B of the second pod 608 such that one or more third-party interfaces can perform one or more evaluation operations on the request. Thereafter, the request is sent from the virtual communication device 624 of the second pod 608 to the fourth VNIC 636.

[0133] If the request is received at the fourth VNIC 636, it is routed to the second VNIC 632 using the IP address of the first pod 606 included in the request. After receiving the request at the second VNIC 632, the second VNIC 632 routes the request to the virtual communication device 622 of the first pod 606 using the IP address of the first pod 606 included in the request. Thereafter, the request is sent through the veth pair 646A-B of the first pod 606 from the virtual communication device 622 of the first pod 606 such that one or more third-party interfaces can perform one or more evaluation operations on the request, and then received by a container 614 disposed within the first pod 606.

[0134] Furthermore, FIG. 10 is a block diagram showing an exemplary response flow 1000 between two pods communicating via different VNICs within the same node 602 of the exemplary container environment 600 of FIG. 6 according to at least one embodiment. For example, the first pod 606 communicates with the second VNIC 632 of the first node 602 via a connection, and the second pod 608 communicates with the fourth VNIC 636 of the first node 602 via a connection. In response to receiving a request for data in the container 614 located within the first pod 608 from the container 616 in the second pod 608, the container 614 in the first pod 606 generates a response (including the IP address of the second pod 608) to the request addressed to the second pod 608. This response is sent by the container 614 to the second VNIC 632 via the virtual communication device 622 of the first pod 608.

[0135] If the response is received by the second VNIC 632, it is routed to the fourth VNIC 636. After receiving the response at the fourth VNIC 636, the fourth VNIC 636 uses the IP address of the second pod 608 included in the response to route the response to the virtual communication device 624 of the second pod 608. Thereafter, the response is sent from the virtual communication device 624 of the second pod 608 through the veth pair 648A - B of the second pod 608 so that one or more third - party interfaces can perform one or more evaluation operations on the response, and then is received by the container 616 located within the second pod 608.

[0136] FIG. 11 is a block diagram showing an exemplary request flow 1100 between two pods communicating via different VNICs in different nodes of the exemplary container environment 600 according to at least one embodiment. More specifically, a third pod 610 communicates with a fourth VNIC 636 of a first node 602 via a connection, and a fourth pod 612 communicates with a seventh VNIC 642 of a second node 604 via a connection. When a request for data addressed to the fourth pod 612 (e.g., a request including the IP address of the fourth pod 612) is sent by a container 618 disposed within the third pod 610, the request is first sent through the veth pair 650A - B of the third pod so that one or more third - party interfaces can perform one or more evaluation operations on the request. Thereafter, the request is sent from the virtual communication device 626 of the third pod 610 to the fourth VNIC 636.

[0137] If the request is received at the fourth VNIC 636, it is routed to the seventh VNIC 642. After receiving the request at the seventh VNIC 642, the seventh VNIC 642 uses the IP address of the fourth pod 612 included in the request to route the request to the virtual communication device 628 of the fourth pod 612. Thereafter, the request is sent through the veth pair 652A - B of the fourth pod 612 from the virtual communication device 628 of the fourth pod 612 by utilizing a hairpin operation so that one or more third - party interfaces can perform one or more evaluation operations on the request, and then is received by a container 620 disposed within the fourth pod 612.

[0138] FIG. 12 is a block diagram showing an exemplary flow of responses 1200 between two pods communicating via different VNICs in different nodes of the exemplary container environment 600 of FIG. 6, according to at least one embodiment. For example, a third pod 610 communicates with a fourth VNIC 636 of a first node 602 via a connection, and a fourth pod 612 communicates with a seventh VNIC 642 of a second node 604 via a connection. In response to receiving a request for data in a container 620 disposed within the fourth pod 612 from a container 618 within the third pod 610, the container 620 of the fourth pod 612 generates a response (including the IP address of the third pod 610) to the request addressed to the third pod 610. This response is sent by the container 620 to the seventh VNIC 642 via the virtual communication device 628 of the fourth pod 612.

[0139] If the response is received by the seventh VNIC 642, it is routed to the fourth VNIC 636. After receiving the response at the fourth VNIC 636, the fourth VNIC 636 uses the IP address of the third pod 610 included in the response to route the response to the virtual communication device 626 of the third pod 610. Thereafter, the response is sent from the virtual communication device 626 of the third pod 610 through the veth pair 650A - B of the third pod 610 so that one or more third - party interfaces can perform one or more evaluation operations on the response, and then is received by the container 618 disposed within the third pod 610.

[0140] Further, FIG. 13 is a block diagram showing an exemplary request flow 1300 between a pod and a virtual cloud network (VCN) 644 within the exemplary container environment 600 of FIG. 6, according to at least one embodiment. When a request for data addressed to the VCN 644 (e.g., a request including the IP address of the VCN 644) is sent by a container 614 disposed within the first pod 606, this request is first sent through the veth pair 646A-B of the first pod 606 by utilizing a hairpin operation such that one or more third-party interfaces can perform one or more evaluation operations on the request. Thereafter, the request is sent from the virtual communication device 622 of the first pod 606 to the second VNIC 632. If the request is received at the second VNIC 632, it is routed to the VCN 644 using the IP address of the VCN 644. Thereafter, the request may be directed to an appropriate destination within the VCN 644.

[0141] Further, FIG. 14 is a block diagram showing an exemplary response flow 1400 between a pod and a virtual cloud network (VCN) 644 within the exemplary container environment 600 of FIG. 6, according to at least one embodiment. In response to receiving a request for data in the VCN 644 from a container 614 in the first pod 606, an entity within the VCN 644 generates a response to the request addressed to the first pod 606 (including the IP address of the first pod 606). This response is sent through the VCN 644 to the second VNIC 632.

[0142] When the response is received by the second VNIC 632, it is routed to the virtual communication device 622 of the first pod 606 using the IP address of the first pod 606 included in the response. Thereafter, the response is sent from the virtual communication device 622 of the first pod 606 through the veth pair 646A - B of the first pod 606 so that one or more third - party interfaces can perform one or more evaluation operations on the response, and then is received by the container 614 arranged within the first pod 606.

[0143] FIG. 15 shows an exemplary method 1500 for implementing communication in a container environment according to at least one embodiment. The method 1500 may be adapted to be executed by one or more components of the container environment 600 of FIG. 6. A computer - readable storage medium includes computer - readable instructions that, when executed by one or more processors of a computing device, cause the computing device to execute the method 1500. The method 1500 may be adapted to be executed in any suitable order. Of course, the method 1500 may include more steps than those shown in FIG. 15 or fewer steps.

[0144] The method 1500 starts at 1502 and may set a first node including a first pod in a container environment. In some embodiments, each node (including the first node) within the container environment may comprise a virtual or physical machine including one or more services required for the implementation of a pod (such as the first pod). In some embodiments, the first pod may include operating system (OS) - level virtualization. In some embodiments, the first node of the container environment may include a host set within the container environment.

[0145] Also, in some embodiments, the container environment comprises a system (e.g., a Kubernetes container orchestration system, etc.) for managing one or more pods. Further, in some embodiments, the first pod may include one or more containers, and each container may include one or more applications to be executed, as well as related libraries and dependencies. Further, in some embodiments, a plurality of different pods (including the first pod) may be implemented on the first node. Also, in some embodiments, each pod within the first node may have its own namespace. In some embodiments, one or more pods may be located in the same location within the first node. Further, in some embodiments, a unique IP address may be assigned to each pod within the first node.

[0146] In 1504, communication via a first virtual communication device from the first pod is configured. In some embodiments, the first virtual communication device may include an Internet Protocol Virtual Local Area Network (IPvlan) device. In some embodiments, the virtual communication device may be configured for each pod within each node of the container environment. In some embodiments, the virtual communication device may facilitate communication between the assigned pod and other pods within the container environment, as well as between a virtual network interface card (VNIC), a network external to the container environment (e.g., a virtual cloud network (VCN)), etc. For example, the virtual communication device of the first pod may communicate with the VNIC to send data requests (and responses to data requests) to a destination pod within the container environment, and such requests and responses may be directed to these destination pods using the unique IP address assigned to the destination pod.

[0147] In 1506, in a first virtual network interface card (VNIC) connected to a first node within a container environment, data configured to be transmitted to a destination device is received from a first virtual communication device assigned to a first pod within the first node. In some embodiments, the first VNIC may include a secondary parent device associated with the first node in order to implement networking functions within the first node and the container environment. In some embodiments, the data may include a request for additional data (such as from a second pod within the first node, a second pod within a second node separate from the first node, etc.).

[0148] Also, in some embodiments, the data may include a response to a request received by the first pod. In some embodiments, the data may include one or more network packets. In some embodiments, the data may be configured to be processed by one or more rules (such as host rules) implemented by the first virtual communication device. In some embodiments, the data may be configured to be instantiated by a local host within the first pod.

[0149] In 1508, the first VNIC routes data including an address identifying at least the first pod to the destination device. In some embodiments, the destination may include a second pod different from the first pod that was the origin of the request. For example, the first pod and the second pod may share the first VNIC within the first node. For example, both the first pod and the second pod may be assigned an IP address by the first VNIC.

[0150] Furthermore, in some embodiments, data may be directly routed from a first VNIC to a second pod. In some embodiments, the first VNIC may directly route data to the second pod by utilizing a hairpin operation. For example, in a hairpin operation, data may be routed through the first VNIC such that the transmitted information is stored in one or more logs (e.g., flow logs, etc.). In another example, the hairpin may be performed by using the vepa mode selected by utilizing a mode flag. In some embodiments, the first VNIC may directly route data to the second pod by utilizing a bridging operation.

[0151] Furthermore, in some embodiments, the destination may include a second VNIC different from the first VNIC that performs routing. For example, both the second VNIC and the first VNIC may be associated with a first node in a container environment. In another example, the second VNIC may be associated with a second node in a container environment different from the first node. In yet another example, data may be directly routed from the first VNIC to the second VNIC.

[0152] Also, in one example, the second VNIC may receive data and route it to a second pod different from the first pod. For example, the second VNIC may be assigned to a second pod within its corresponding node. In this way, data may be directly sent between VNICs within a container environment without the need to pass through a bridge and encapsulate the data.

[0153] Also, in some embodiments, the destination may include a Virtual Cloud Network (VCN). For example, data may be directly routed from a first VNIC to the VCN. In some embodiments, the address identifying the first pod may include a first Internet Protocol (IP) address assigned to the first pod. For example, the first VNIC may assign the first IP address from a pool of available IP addresses to a first virtual communication device of the first pod. In this way, the first IP address of the first virtual communication device may be recorded in one or more flow logs when data is transmitted.

[0154] Furthermore, in some embodiments, the data may further include a second Internet Protocol (IP) address assigned to the recipient pod. For example, the recipient pod may include the intended recipient of the data. In another example, the recipient pod may be located within a node, and the VNIC associated with the node may assign the second IP address to a second virtual communication device assigned to the recipient pod within the node.

[0155] Furthermore, in some embodiments, the data may be processed by a second communication device (e.g., a point-to-point device) assigned to the first pod before being sent from the first communication device to the first VNIC. For example, the second communication device may include a virtual Ethernet device (veth) (e.g., one or more veth pairs) that connects a first network (e.g., a container network, etc.) to a second network (e.g., a host network, etc.). In another example, each of the one or more veth pairs may include a first portion existing in the pod namespace of the pod environment and a second portion existing in the host namespace of the pod environment.

[0156] Also, in one example, the first pod may send data through the second communication device by utilizing the hairpin operation. In another example, the second communication device may be configured such that a third-party container networking interface (such as Calico) can perform one or more evaluation operations.

[0157] Also, in some embodiments, in response to receiving data at the second pod, the data may be processed by a third communication device (e.g., a point-to-point device) assigned to the second pod. For example, the third communication device may include a virtual Ethernet device (veth) (e.g., one or more veth pairs) that connects a first network (e.g., a container network, etc.) to a second network (e.g., a host network, etc.). In another example, one or more veth pairs may include a first portion present in the pod namespace of the container environment and a second portion present in the host namespace of the container environment. In yet another example, the second pod may send data through the third communication device by utilizing the hairpin operation. In one example, the hairpin operation may include a process of sending network traffic from a network device, and the network traffic may be intended to be re-routed back to the same network device. Thus, a third-party container networking interface (such as Calico) may be configured to perform one or more evaluation operations.

[0158] Furthermore, in some embodiments, the first VNIC may be configured to receive a response from a destination. For example, the data that the first VNIC first receives / routes may include a request for data. In another example, the first VNIC may be configured to receive a response to the request from the initial destination. For example, the response may include the first Internet Protocol (IP) address assigned to the first pod. In yet another example, the first VNIC may be configured to route the response to the first virtual communication device assigned to the first pod within the first node.

[0159] Thus, by utilizing the address identifying the first pod, the first VNIC can route all traffic to and from the first pod via the first virtual communication device (e.g., an IPvlan device). In this way, while this traffic can maintain the IP address associated with the first pod, rule enforcement within the first node by the first virtual communication device remains possible. Further, such traffic can be routed by the first VNIC without the need to perform encapsulation or pass through a bridge device. This reduces the size of the data to be transmitted (e.g., by eliminating the overhead required for encapsulation added to the transmission of network packets), and can reduce the amount of computation performed by the hardware implementing the container environment (e.g., by eliminating the computational cost required to perform encapsulation of the data during transmission). Also, the latency of data transmission within the container environment is reduced (e.g., by eliminating the need to perform encapsulation during data transmission), and the performance of the hardware implementing the container environment can be improved.

[0160] As described above, service - type infrastructure (IaaS) is a specific type of cloud computing. IaaS can be configured to provide virtualized computing resources via a public network (e.g., the Internet). In the IaaS model, a cloud computing provider can host infrastructure components (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer), etc.). Also, optionally, the IaaS provider may supply various services associated with these infrastructure components (e.g., billing, monitoring, logging, load balancing, and clustering, etc.). Therefore, since these services can be policy - driven, IaaS users can maintain the availability and performance of applications by implementing policies that promote load balancing.

[0161] Optionally, IaaS customers can access resources and services through a wide - area network (WAN) such as the Internet and use the cloud provider's services to install other elements of the application stack. For example, a user can log in to the IaaS platform to create virtual machines (VMs), install an operating system (OS) on each VM, deploy middleware such as a database, create storage buckets for workloads and backups, and even install enterprise - specific software on the VMs. And the customer can use the provider's services to perform various functions such as distributing network traffic, troubleshooting application problems, monitoring performance, and managing disaster recovery.

[0162] In most cases, cloud computing models require the participation of a cloud provider. The cloud provider may be a third-party service specialized in providing IaaS (e.g., offering, renting, selling), but this is not necessary. Also, an entity may consider deploying a private cloud and becoming its own infrastructure service provider.

[0163] In some examples, an IaaS deployment is the process of placing a new application or a new version of an application on a prepared application server, etc. It may also include the process of preparing the server (e.g., installing libraries, daemons, etc.). This is often managed by a cloud provider under the hypervisor layer (e.g., servers, storage, network hardware, and virtualization). For this reason, a customer may be responsible for the deployment of handling (OS), middleware, and / or applications (e.g., on a self-service virtual machine that can be spun up on demand).

[0164] In some examples, IaaS provisioning represents obtaining the computers or virtual hosts to be used and may even represent installing the necessary libraries or services on them. In most cases, deployment does not include provisioning, and provisioning may need to be performed first.

[0165] In some cases, there are two different challenges in IaaS provisioning. First, as the initial challenge, there is the provisioning of the initial set of infrastructure before anything operates. Second, after all provisioning, there are challenges in evolving the existing infrastructure (e.g., adding new services, changing services, deleting services, etc.). In some cases, these two challenges can be addressed by enabling the definition of the infrastructure settings through declarations. In other words, the infrastructure (e.g., the necessary components and how the components interact) can be defined by one or more configuration files. Thus, the overall topology of the infrastructure (e.g., the resource dependencies and how each resource functions integrally) can be described by declarations. In some cases, once the topology is defined, a workflow can be generated to form and / or manage the various components described in the configuration files.

[0166] In some examples, the infrastructure can have many interconnected elements. For example, there can be one or more virtual private clouds (VPCs), also known as core networks (e.g., a potential on-demand pool of configurable and / or shared computing resources). Also, in some examples, there can be one or more inbound / outbound traffic group rules such that provisioning is done to define the manner in which the network's inbound and / or outbound traffic is set up and one or more virtual machines (VMs). Provisioning of other infrastructure elements such as load balancers, databases, etc. can also be done. If there are more requests and / or additions of infrastructure elements, the infrastructure can evolve little by little.

[0167] In some cases, the adoption of continuous deployment techniques can enable the deployment of infrastructure code across various virtual computing environments. Further, the described techniques can enable infrastructure management within these environments. In some examples, a service team can write code that is desirable for deployment to one or more (but often multiple) different production environments (e.g., various geographically different locations, possibly worldwide). However, in some examples, the infrastructure to which the code is deployed needs to be set up first. In some cases, it is also possible to perform provisioning manually, use a provisioning tool for resource provisioning, and / or use a deployment tool for code deployment after infrastructure provisioning.

[0168] FIG. 16 is a block diagram 1600 showing an exemplary pattern of an IaaS architecture according to at least one embodiment. A service operator 1602 may be communicatively coupled to a secure host tenancy 1604 that may include a virtual cloud network (VCN) 1606 and a secure host subnet 1608. In some examples, the service operator 1602 may use one or more client computing devices, which may run software such as Microsoft Windows Mobile® and / or various mobile operating systems such as iOS, Windows Phone, Android, BlackBerry 8, Palm OS, etc., and may be portable handheld devices (e.g., iPhone®, mobile phone, iPad®, computing tablet, personal digital assistant (PDA)) or wearable devices (e.g., Google Glass® head-mounted display, etc.) that can use the Internet, email, short message service (SMS), BlackBerry®, or other communication protocols. Alternatively, the client computing device may be a general-purpose personal computer, such as a personal computer and / or laptop computer running various versions of Microsoft Windows®, Apple Macintosh®, and / or Linux® operating systems. The client computing device may be a workstation computer running any of various commercially available UNIX® or UNIX-like operating systems, such as various GNU / Linux operating systems such as Google Chrome OS, but is not limited thereto.As an alternative or in addition, the client computing device may be any other electronic device such as a sink client computer capable of communicating via a network that can access VCN 1606 and / or the Internet, an Internet-enabled gaming system (e.g., a Microsoft Xbox gaming console with or without a Kinect (registered trademark) gesture input device), and / or a personal messaging device.

[0169] VCN 1606 can include an LPG 1610 that can be communicatively coupled to an SSH VCN 1612 via a local peering gateway (LPG) 1610 included in a secure shell (SSH) VCN 1612. The SSH VCN 1612 can include an SSH subnet 1614, and the SSH VCN 1612 can be communicatively coupled to a control plane VCN 1616 via an LPG 1610 included in the control plane VCN 1616. Also, the SSH VCN 1612 can be communicatively coupled to a data plane VCN 1618 via an LPG 1610. The control plane VCN 1616 and the data plane VCN 1618 can be included in a service tenancy 1619 that can be owned and / or operated by an IaaS provider.

[0170] The control plane VCN1616 can include a control plane buffer area (DMZ) layer 1620 that functions as a border network (e.g., a part of the enterprise network between the enterprise intranet and the external network). DMZ-based servers have limited responsibility and can help suppress intrusions. Also, the DMZ layer 1620 can include one or more load balancer (LB) subnets 1622, a control plane application layer 1624 that can include an app subnet 1626, and a control plane data layer 1628 that can include a database (DB) subnet 1630 (e.g., a front-end DB subnet and / or a back-end DB subnet). The LB subnet 1622 included in the control plane DMZ layer 1620 is communicatively coupled to the app subnet 1626 included in the control plane application layer 1624 and the internet gateway 1634 that can be included in the control plane VCN1616, and the app subnet 1626 can be communicatively coupled to the DB subnet 1630, service gateway 1636, and network address translation (NAT) gateway 1638 included in the control plane data layer 1628. The control plane VCN1616 can include a service gateway 1636 and a NAT gateway 1638.

[0171] The control plane VCN1616 can include a data plane mirror app layer 1640 that can include an app subnet 1626. The app subnet 1626 included in the data plane mirror app layer 1640 can include a virtual network interface controller (VNIC) 1642 that can execute a compute instance 1644. The compute instance 1644 can communicatively couple the app subnet 1626 of the data plane mirror app layer 1640 to the app subnet 1626 that can be included in the data plane app layer 1646.

[0172] The data plane VCN 1618 can include a data plane application layer 1646, a data plane DMZ layer 1648, and a data plane data layer 1650. The data plane DMZ layer 1648 can include an LB subnet 1622 communicatively coupled to the application subnet 1626 of the data plane application layer 1646 and the Internet gateway 1634 of the data plane VCN 1618. The application subnet 1626 can be communicatively coupled to the service gateway 1636 of the data plane VCN 1618 and the NAT gateway 1638 of the data plane VCN 1618. Also, the data plane data layer 1650 can include a DB subnet 1630 communicatively coupled to the application subnet 1626 of the data plane application layer 1646.

[0173] The Internet gateway 1634 of the control plane VCN 1616 and the data plane VCN 1618 can be communicatively coupled to a metadata management service 1652 communicatively coupled to the public Internet 1654. The public Internet 1654 can be communicatively coupled to the NAT gateway 1638 of the control plane VCN 1616 and the data plane VCN 1618. The service gateway 1636 of the control plane VCN 1616 and the data plane VCN 1618 can be communicatively coupled to a cloud service 1656.

[0174] In some examples, the service gateways 1636 of the control plane VCN 1616 and the data plane VCN 1618 can make application programming interface (API) calls to cloud services 1656 without going through the public Internet 1654. API calls from the service gateway 1636 to the cloud service 1656 can be unidirectional. The service gateway 1636 can make API calls to the cloud service 1656, and the cloud service 1656 can send the requested data to the service gateway 1636. However, the cloud service 1656 may not need to initiate an API call to the service gateway 1636.

[0175] In some examples, the secure host tenancy 1604 can be directly connected to or otherwise isolated from the service tenancy 1619. The secure host subnet 1608 can communicate with the SSH subnet 1614 through the LPG 1610 that can enable bidirectional communication through a system that is isolated otherwise. By connecting the secure host subnet 1608 to the SSH subnet 1614, the secure host subnet 1608 can access other entities within the service tenancy 1619.

[0176] The control plane VCN 1616 can enable the setting or provisioning of desired resources by the users of the service tenancy 1619. The desired resources provisioned in the control plane VCN 1616 may be deployed or used in the data plane VCN 1618. In some examples, the control plane VCN 1616 can be isolated from the data plane VCN 1618, and the data plane mirror app layer 1640 of the control plane VCN 1616 can communicate with the data plane app layer 1646 of the data plane VCN 1618 through the VNIC 1642 that may be included in the data plane mirror app layer 1640 and the data plane app layer 1646.

[0177] In some examples, a user or customer of the system can make requests (e.g., perform create, read, update, or delete (CRUD) operations) over the public Internet 1654, and the public Internet 1654 can send the requests to the metadata management service 1652. The metadata management service 1652 can send the requests through the Internet gateway 1634 to the control plane VCN 1616. The requests can be received by the LB subnet 1622 included in the control plane DMZ layer 1620. The LB subnet 1622 may be configured to determine that the requests are valid, and in response to this determination, the LB subnet 1622 can send the requests to the app subnet 1626 included in the control plane app layer 1624. If the requests are verified and a call to the public Internet 1654 is required, the call to the public Internet 1654 may be sent to the NAT gateway 1638 that can make the call to the public Internet 1654. Metadata that is considered desirable to be stored by the requests can be stored in the DB subnet 1630.

[0178] In some examples, the data plane mirror app layer 1640 can facilitate direct communication between the control plane VCN 1616 and the data plane VCN 1618. For example, it may be desirable to apply configuration changes, updates, or other suitable modifications to resources included in the data plane VCN 1618. The control plane VCN 1616 can execute configuration changes, updates, or other suitable modifications to the resources by directly communicating with the resources included in the data plane VCN 1618 via the VNIC 1642.

[0179] In some embodiments, the control plane VCN 1616 and the data plane VCN 1618 may be included in the service tenancy 1619. In this case, the user or customer of the system may not own either the control plane VCN 1616 or the data plane VCN 1618, or may disable either of them. Instead, the IaaS provider may own both the control plane VCN 1616 and the data plane VCN 1618, may operate both of them, and both may be included in the service tenancy 1619. This embodiment may enable network isolation that can prevent interaction with resources of other users or other customers of the user or customer. Also, this embodiment may enable private storage of databases by the users or customers of the system without relying on the public Internet 1654, which may not have the desired level of threat prevention for storage.

[0180] In other embodiments, the LB subnet 1622 included in the control plane VCN 1616 can be configured to receive signals from the service gateway 1636. In this embodiment, the control plane VCN 1616 and the data plane VCN 1618 may be configured to be invoked by the customers of the IaaS provider without invoking the public Internet 1654. The customers of the IaaS provider may desire this embodiment because the databases used by the customers can be stored in the service tenancy 1619, which can be controlled by the IaaS provider and isolated from the public Internet 1654.

[0181] FIG. 17 is a block diagram 1700 showing another exemplary pattern of an IaaS architecture according to at least one embodiment. A service operator 1702 (e.g., the service operator 1602 of FIG. 16) may be communicatively coupled to a secure host tenancy 1704 (e.g., the secure host tenancy 1604 of FIG. 16) that may include a virtual cloud network (VCN) 1706 (e.g., the VCN 1606 of FIG. 16) and a secure host subnet 1708 (e.g., the secure host subnet 1608 of FIG. 16). The VCN 1706 can include an LPG 1710 that can be communicatively coupled to an SSH VCN 1712 (e.g., the SSH VCN 1612 of FIG. 16) via an LPG 1610 included in the secure shell (SSH) VCN 1712 (e.g., the LPG 1610 of FIG. 16). The SSH VCN 1712 can include an SSH subnet 1714 (e.g., the SSH subnet 1614 of FIG. 16), and the SSH VCN 1712 can be communicatively coupled to a control plane VCN 1716 (e.g., the control plane VCN 1616 of FIG. 16) via an LPG 1710 included in the control plane VCN 1716. The control plane VCN 1716 is included in a service tenancy 1719 (e.g., the service tenancy 1619 of FIG. 16), and a data plane VCN 1718 (e.g., the data plane VCN 1618 of FIG. 16) can be included in a customer tenancy 1721 that can be owned or operated by a user or customer of the system.

[0182] The control plane VCN 1716 can include a control plane DMZ layer 1720 (e.g., the control plane DMZ layer 1620 in FIG. 16) that can include an LB subnet 1722 (e.g., the LB subnet 1622 in FIG. 16), a control plane application layer 1724 (e.g., the control plane application layer 1624 in FIG. 16) that can include an application subnet 1726 (e.g., the application subnet 1626 in FIG. 16), and a control plane data layer 1728 (e.g., the control plane data layer 1628 in FIG. 16) that can include a DB subnet 1730 (similar to the database (DB) subnet 1630 in FIG. 16). The LB subnet 1722 included in the control plane DMZ layer 1720 is communicatively coupled to the application subnet 1726 included in the control plane application layer 1724 and an Internet gateway 1734 (e.g., the Internet gateway 1634 in FIG. 16) that can be included in the control plane VCN 1716. The application subnet 1726 can be communicatively coupled to the DB subnet 1730 included in the control plane data layer 1728, a service gateway 1736 (e.g., the service gateway 1636 in FIG. 16), and a network address translation (NAT) gateway 1738 (e.g., the NAT gateway 1638 in FIG. 16). The control plane VCN 1716 can include the service gateway 1736 and the NAT gateway 1738.

[0183] The control plane VCN 1716 can include a data plane mirror app layer 1740 (e.g., the data plane mirror app layer 1640 of FIG. 16) that can include an app subnet 1726. The app subnet 1726 included in the data plane mirror app layer 1740 can include a virtual network interface controller (VNIC) 1742 (e.g., VNIC 1642) that can execute a compute instance 1744 (similar to the compute instance 1644 of FIG. 16). The compute instance 1744 can facilitate communication between the app subnet 1726 of the data plane mirror app layer 1740 and an app subnet 1726 that can be included in the data plane app layer 1746 (e.g., the data plane app layer 1646 of FIG. 16) via the VNIC 1742 included in the data plane mirror app layer 1740 and the VNIC 1742 included in the data plane app layer 1746.

[0184] The internet gateway 1734 included in the control plane VCN 1716 can be communicatively coupled to a metadata management service 1752 (e.g., the metadata management service 1652 of FIG. 16) that can be communicatively coupled to the public internet 1754 (e.g., the public internet 1654 of FIG. 16). The public internet 1754 can be communicatively coupled to the NAT gateway 1738 included in the control plane VCN 1716. The service gateway 1736 included in the control plane VCN 1716 can be communicatively coupled to a cloud service 1756 (e.g., the cloud service 1656 of FIG. 16).

[0185] In some examples, the data plane VCN 1718 can be included in the customer tenancy 1721. In this case, the IaaS provider may provide a control plane VCN 1716 for each customer, and the IaaS provider may also configure a unique compute instance 1744 included in the service tenancy 1719 for each customer. Each compute instance 1744 can enable communication between the control plane VCN 1716 included in the service tenancy 1719 and the data plane VCN 1718 included in the customer tenancy 1721. According to the compute instance 1744, in the data plane VCN 1718 included in the customer tenancy 1721, the deployment or use of resources provisioned in the control plane VCN 1716 included in the service tenancy 1719 may be made possible.

[0186] In other examples, a customer of the IaaS provider may have a database existing in the customer tenancy 1721. In this example, the control plane VCN 1716 can include a data plane mirror app layer 1740 that may include an app subnet 1726. The data plane mirror app layer 1740 may exist in the data plane VCN 1718, but may not exist in the data plane VCN 1718. That is, the data plane mirror app layer 1740 may be accessible to the customer tenancy 1721, but may not exist in the data plane VCN 1718, and may not be owned or operated by a customer of the IaaS provider. The data plane mirror app layer 1740 may be configured to make calls to the data plane VCN 1718, but may also be configured not to make calls to any entity included in the control plane VCN 1716. The customer may desire to deploy or use resources in the data plane VCN 1718 that are provisioned in the control plane VCN 1716, and the data plane mirror app layer 1740 can facilitate the deployment or other use of the resources desired by the customer.

[0187] In some embodiments, a customer of an IaaS provider can apply filters to the data plane VCN 1718. In this embodiment, the customer can determine the targets that the data plane VCN 1718 can access, and the customer may also restrict access from the data plane VCN 1718 to the public Internet 1754. The IaaS provider may not be able to apply filters, or may not be able to control access of the data plane VCN 1718 to any external network or database. The application of customer filters and controls to the data plane VCN 1718 included in the customer tenancy 1721 can help isolate the data plane VCN 1718 from other customers and the public Internet 1754.

[0188] In some embodiments, cloud service 1756 can be made accessible to services that may not exist on either the public Internet 1754, the control plane VCN 1716, or the data plane VCN 1718 by calls made by service gateway 1736. The connection between cloud service 1756 and the control plane VCN 1716 or the data plane VCN 1718 may not be live or continuous. Cloud service 1756 may exist on different networks owned or operated by an IaaS provider. Cloud service 1756 may be configured to receive calls from service gateway 1736 and may also be configured not to receive calls from the public Internet 1754. Some cloud services 1756 may be isolated from other cloud services 1756, and the control plane VCN 1716 may be isolated from cloud services 1756 that may not be in the same region as the control plane VCN 1716. For example, the control plane VCN 1716 may be located in "Region 1", and cloud service "Deployment 16" may be located in Region 1 and "Region 2". If a call to Deployment 16 is made by a service gateway 1736 included in the control plane VCN 1716 located in Region 1, the call may be configured to be sent to Deployment 16 in Region 1. In this example, the control plane VCN 1716 or the Deployment 16 in Region 1 may not be communicatively coupled to the Deployment 16 in Region 2 and may not communicate with the Deployment 16 in Region 2.

[0189] FIG. 18 is a block diagram 1800 showing another exemplary pattern of an IaaS architecture according to at least one embodiment. A service operator 1802 (e.g., the service operator 1602 of FIG. 16) may be communicatively coupled to a secure host tenancy 1804 (e.g., the secure host tenancy 1604 of FIG. 16) that may include a virtual cloud network (VCN) 1806 (e.g., the VCN 1606 of FIG. 16) and a secure host subnet 1808 (e.g., the secure host subnet 1608 of FIG. 16). The VCN 1806 can comprise an LPG 1810 that can be communicatively coupled to an SSH VCN 1812 (e.g., the SSH VCN 1612 of FIG. 16) via the LPG 1810 included in the SSH VCN 1812. The SSH VCN 1812 can comprise an SSH subnet 1814 (e.g., the SSH subnet 1614 of FIG. 16), and the SSH VCN 1812 can be communicatively coupled to a control plane VCN 1816 (e.g., the control plane VCN 1616 of FIG. 16) via the LPG 1810 included in the control plane VCN 1816 and can be communicatively coupled to a data plane VCN 1818 (e.g., the data plane VCN 1618 of FIG. 16) via the LPG 1810 included in the data plane VCN 1818. The control plane VCN 1816 and the data plane VCN 1818 may be included in a service tenancy 1819 (e.g., the service tenancy 1619 of FIG. 16).

[0190] The control plane VCN 1816 can include a control plane DMZ layer 1820 (e.g., the control plane DMZ layer 1620 in FIG. 16) that can include a load balancer (LB) subnet 1822 (e.g., the LB subnet 1622 in FIG. 16), a control plane application layer 1824 (e.g., the control plane application layer 1624 in FIG. 16) that can include an application subnet 1826 (similar to the application subnet 1626 in FIG. 16), and a control plane data layer 1828 (e.g., the control plane data layer 1628 in FIG. 16) that can include a DB subnet 1830. The LB subnet 1822 included in the control plane DMZ layer 1820 is communicatively coupled to the application subnet 1826 included in the control plane application layer 1824 and an Internet gateway 1834 (e.g., the Internet gateway 1634 in FIG. 16) that can be included in the control plane VCN 1816. The application subnet 1826 can be communicatively coupled to the DB subnet 1830 included in the control plane data layer 1828, a service gateway 1836 (e.g., the service gateway in FIG. 16), and a network address translation (NAT) gateway 1838 (e.g., the NAT gateway 1638 in FIG. 16). The control plane VCN 1816 can include the service gateway 1836 and the NAT gateway 1838.

[0191] The data plane VCN 1818 can include a data plane application layer 1846 (e.g., the data plane application layer 1646 in FIG. 16), a data plane DMZ layer 1848 (e.g., the data plane DMZ layer 1648 in FIG. 16), and a data plane data layer 1850 (e.g., the data plane data layer 1650 in FIG. 16). The data plane DMZ layer 1848 can include an LB subnet 1822 communicatively coupled to the trusted app subnet 1860 and the untrusted app subnet 1862 of the data plane application layer 1846 and the Internet gateway 1834 included in the data plane VCN 1818. The trusted app subnet 1860 can be communicatively coupled to the service gateway 1836 included in the data plane VCN 1818, the NAT gateway 1838 included in the data plane VCN 1818, and the DB subnet 1830 included in the data plane data layer 1850. The untrusted app subnet 1862 can be communicatively coupled to the service gateway 1836 included in the data plane VCN 1818 and the DB subnet 1830 included in the data plane data layer 1850. The data plane data layer 1850 can include a DB subnet 1830 communicatively coupled to the service gateway 1836 included in the data plane VCN 1818.

[0192] The untrusted application subnet 1862 can include one or more primary VNICs 1864(1) - 1864(N) that can be communicatively coupled to tenant virtual machines (VMs) 1866(1) - 1866(N). Each tenant VM 1866(1) - 1866(N) can be communicatively coupled to an application subnet 1867(1) - 1867(N) respectively, which can be included in each of the container output VCNs 1868(1) - 1868(N) that can be included in respective customer tenancies 1870(1) - 1870(N). The secondary VNICs 1872(1) - 1872(N) can each facilitate communication between the untrusted application subnet 1862 included in the data plane VCN 1818 and the application subnets included in the container output VCNs 1868(1) - 1868(N). Each container output VCN 1868(1) - 1868(N) can include a NAT gateway 1838 that can be communicatively coupled to the public internet 1854 (e.g., the public internet 1654 of FIG. 16).

[0193] The internet gateways 1834 included in the control plane VCN 1816 and the data plane VCN 1818 can be communicatively coupled to a metadata management service 1852 (e.g., the metadata management system 1652 of FIG. 16) that can be communicatively coupled to the public internet 1854. The public internet 1854 can be communicatively coupled to the NAT gateways 1838 included in the control plane VCN 1816 and the data plane VCN 1818. The service gateways 1836 included in the control plane VCN 1816 and the data plane VCN 1818 can be communicatively coupled to cloud services 1856.

[0194] In some embodiments, the data plane VCN 1818 can be integrated with the customer tenancy 1870. This integration can be useful or desirable to the customers of the IaaS provider, such as when they may wish to have support for code execution. The customer may provide the code to execute, which may be disruptive, may communicate with other customer resources, or may have undesirable effects. In response, the IaaS provider may decide whether to execute the code provided by the customer to the IaaS provider.

[0195] In some examples, a customer of an IaaS provider may permit temporary network access to the IaaS provider and request features imparted to the data plane application layer 1846. The code that executes this feature may be configured to execute in the VMs 1866(1) - 1866(N), and may not be configured to execute elsewhere on the data plane VCN 1818. Each of the VMs 1866(1) - 1866(N) may be connected to one customer tenancy 1870. The containers 1871(1) - 1871(N) included in the VMs 1866(1) - 1866(N) may each be configured to execute code. In this case, there may be double isolation (e.g., the containers 1871(1) - 1871(N) that execute code may be included in the VMs 1866(1) - 1866(N) included in at least the untrusted app subnet 1862), which can help prevent damage to the IaaS provider's network or the networks of different customers by incorrect or unwanted code. The containers 1871(1) - 1871(N) may be communicatively coupled to the customer tenancy 1870 and may be configured to send or receive data to or from the customer tenancy 1870. The containers 1871(1) - 1871(N) may be configured not to send or receive data to or from any other entity in the data plane VCN 1818. Upon completion of the execution of the code, the IaaS provider may deactivate or discard the containers 1871(1) - 1871(N).

[0196] In some embodiments, the trusted application subnet 1860 may be configured to execute code that may be owned or operated by an IaaS provider. In this embodiment, the trusted application subnet 1860 may be communicatively coupled to the DB subnet 1830 and may be configured to perform CRUD operations in the DB subnet 1830. The untrusted application subnet 1862 may be communicatively coupled to the DB subnet 1830, but in this embodiment, may be configured to perform read operations in the DB subnet 1830. The containers 1871(1) to 1871(N) included in each customer's VMs 1866(1) to 1866(N) and capable of executing code from the customer may not be communicatively coupled to the DB subnet 1830.

[0197] In other embodiments, the control plane VCN 1816 and the data plane VCN 1818 may not be directly communicatively coupled. In this embodiment, there may not be a direct communication between the control plane VCN 1816 and the data plane VCN 1818. However, communication can be performed indirectly by at least one method. An LPG 1810 that can facilitate communication between the control plane VCN 1816 and the data plane VCN 1818 may be established by the IaaS provider. In another example, the control plane VCN 1816 or the data plane VCN 1818 can make calls to the cloud service 1856 via the service gateway 1836. For example, a call from the control plane VCN 1816 to the cloud service 1856 may include a request for a service that can communicate with the data plane VCN 1818.

[0198] FIG. 19 is a block diagram 1900 showing another exemplary pattern of an IaaS architecture according to at least one embodiment. A service operator 1902 (e.g., service operator 1602 of FIG. 16) may be communicatively coupled to a secure host tenancy 1904 (e.g., secure host tenancy 1604 of FIG. 16) that may include a virtual cloud network (VCN) 1906 (e.g., VCN 1606 of FIG. 16) and a secure host subnet 1908 (e.g., secure host subnet 1608 of FIG. 16). The VCN 1906 can include an LPG 1910 that can be communicatively coupled to an SSH VCN 1912 (e.g., SSH VCN 1612 of FIG. 16) via the LPG 1910 included in the SSH VCN 1912. The SSH VCN 1912 can include an SSH subnet 1914 (e.g., SSH subnet 1614 of FIG. 16), and the SSH VCN 1912 can be communicatively coupled to a control plane VCN 1916 (e.g., control plane VCN 1616 of FIG. 16) via the LPG 1910 included in the control plane VCN 1916 and can be communicatively coupled to a data plane VCN 1918 (e.g., data plane VCN 1618 of FIG. 16) via the LPG 1910 included in the data plane VCN 1918. The control plane VCN 1916 and the data plane VCN 1918 can be included in a service tenancy 1919 (e.g., service tenancy 1619 of FIG. 16).

[0199] The control plane VCN 1916 can include a control plane DMZ layer 1920 (e.g., the control plane DMZ layer 1620 in FIG. 16) that can include an LB subnet 1922 (e.g., the LB subnet 1622 in FIG. 16), a control plane application layer 1924 (e.g., the control plane application layer 1624 in FIG. 16) that can include an application subnet 1926 (e.g., the application subnet 1626 in FIG. 16), and a control plane data layer 1928 (e.g., the control plane data layer 1628 in FIG. 16) that can include a DB subnet 1930 (e.g., the DB subnet 1830 in FIG. 18). The LB subnet 1922 included in the control plane DMZ layer 1920 is communicatively coupled to the application subnet 1926 included in the control plane application layer 1924 and an Internet gateway 1934 (e.g., the Internet gateway 1634 in FIG. 16) that can be included in the control plane VCN 1916. The application subnet 1926 can be communicatively coupled to the DB subnet 1930 included in the control plane data layer 1928, a service gateway 1936 (e.g., the service gateway in FIG. 16), and a network address translation (NAT) gateway 1938 (e.g., the NAT gateway 1638 in FIG. 16). The control plane VCN 1916 can include the service gateway 1936 and the NAT gateway 1938.

[0200] The data plane VCN 1918 can include a data plane application layer 1946 (e.g., the data plane application layer 1646 in FIG. 16), a data plane DMZ layer 1948 (e.g., the data plane DMZ layer 1648 in FIG. 16), and a data plane data layer 1950 (e.g., the data plane data layer 1650 in FIG. 16). The data plane DMZ layer 1948 can include a trusted application subnet 1960 (e.g., the trusted application subnet 1860 in FIG. 18) and an untrusted application subnet 1962 (e.g., the untrusted application subnet 1862 in FIG. 18) of the data plane application layer 1946, and an LB subnet 1922 communicatively coupled to an Internet gateway 1934 included in the data plane VCN 1918. The trusted application subnet 1960 can be communicatively coupled to a service gateway 1936 included in the data plane VCN 1918, a NAT gateway 1938 included in the data plane VCN 1918, and a DB subnet 1930 included in the data plane data layer 1950. The untrusted application subnet 1962 can be communicatively coupled to the service gateway 1936 included in the data plane VCN 1918 and the DB subnet 1930 included in the data plane data layer 1950. The data plane data layer 1950 can include a DB subnet 1930 communicatively coupled to the service gateway 1936 included in the data plane VCN 1918.

[0201] The untrusted application subnet 1962 can include primary VNICs 1964(1) to 1964(N) communicatively coupled to tenant virtual machines (VMs) 1966(1) to 1966(N) existing within the untrusted application subnet 1962. Each tenant VM 1966(1) to 1966(N) can execute code in respective containers 1967(1) to 1967(N) and can be communicatively coupled to an application subnet 1926 that can be included in a data plane application layer 1946 that can be included in a container output VCN 1968. Secondary VNICs 1972(1) to 1972(N) can each facilitate communication between the untrusted application subnet 1962 included in the data plane VCN 1918 and the application subnet included in the container output VCN 1968. The container output VCN can include a NAT gateway 1938 communicatively coupled to a public internet 1954 (e.g., the public internet 1654 of FIG. 16).

[0202] The internet gateways 1934 included in the control plane VCN 1916 and the data plane VCN 1918 can be communicatively coupled to a metadata management service 1952 (e.g., the metadata management system 1652 of FIG. 16) communicatively coupled to the public internet 1954. The public internet 1954 can be communicatively coupled to the NAT gateway 1938 included in the control plane VCN 1916 and the data plane VCN 1918. The service gateways 1936 included in the control plane VCN 1916 and the data plane VCN 1918 can be communicatively coupled to a cloud service 1956.

[0203] In some cases, the pattern shown by the architecture of block diagram 1900 in FIG. 19 is considered an exception to the pattern shown by the architecture of block diagram 1800 in FIG. 18 and is considered desirable for the customers of the IaaS provider when the IaaS provider cannot communicate directly with the customers (e.g., in a non-connected region). Containers 1967(1) to 1967(N) included in VM1966(1) to 1966(N) for each customer are each accessible in real time by the customer. Containers 1967(1) to 1967(N) may be configured to make calls to respective secondary VNICs 1972(1) to 1972(N) included in the application subnet 1926 of the data plane application layer 1946 that may be included in the container output VCN1968. Secondary VNICs 1972(1) to 1972(N) may be able to send the calls to the NAT gateway 1938, and the NAT gateway 1938 may send the calls to the public Internet 1954. In this example, containers 1967(1) to 1967(N) accessible in real time by the customer may be isolated from the control plane VCN1916 and may also be isolated from other entities included in the data plane VCN1918. Also, containers 1967(1) to 1967(N) may be isolated from the resources of other customers.

[0204] In other examples, a customer can call cloud service 1956 using containers 1967(1) to 1967(N). In this example, the customer can execute code in containers 1967(1) to 1967(N) that requests services from cloud service 1956. Containers 1967(1) to 1967(N) can send this request to secondary VNICs 1972(1) to 1972(N), and secondary VNICs 1972(1) to 1972(N) can send the request to a NAT gateway, which can send the request to public internet 1954. Public internet 1954 can send the request to the LB subnet 1922 included in control plane VCN 1916 via internet gateway 1934. In response to a determination that the request is valid, the LB subnet can send the request to app subnet 1926, and app subnet 1926 can send the request to cloud service 1956 via service gateway 1936.

[0205] Of course, the IaaS architectures 1600, 1700, 1800, 1900 shown in the drawings may have components other than those shown. Furthermore, the embodiments shown in the drawings are merely examples of a cloud infrastructure system that may include one embodiment of the present disclosure. In some other embodiments, the IaaS system may have more or fewer components than shown, may combine two or more components, or may have different component configurations or arrangements.

[0206] In one embodiment, the IaaS system described herein may include a series of application, middleware, and database services that are delivered to customers in a self-service and subscription-based, elastically scalable, reliable, highly available, and secure manner. OCI (Oracle Cloud Infrastructure) provided by the present assignee is an example of such an IaaS system.

[0207] Figure 20 shows an exemplary computer system 2000 in which various embodiments can be implemented. The system 2000 may be adapted to be used in any implementation of the computer systems described above. As shown in the drawings, the computer system 2000 includes a processing unit 2004 that communicates with a number of peripheral subsystems via a bus subsystem 2002. The peripheral subsystems may include a processing acceleration unit 2006, an I / O subsystem 2008, a storage subsystem 2018, and a communication subsystem 2024. The storage subsystem 2018 includes a tangible computer-readable storage medium 2022 and a system memory 2010.

[0208] The bus subsystem 2002 provides a mechanism for the various components and subsystems of the computer system 2000 to communicate with each other as intended. The bus subsystem 2002 is shown schematically as a single bus, but alternative embodiments of the bus subsystem may utilize multiple buses. The bus subsystem 2002 may be any of a variety of bus architectures, including a memory bus or memory controller, a peripheral bus, and a local bus that uses any of a variety of bus architectures. For example, such architectures may include an Industry Standard Architecture (ISA) bus, which may be implemented as a mezzanine bus manufactured in accordance with the IEEE P1386.1 standard, a Micro Channel Architecture (MCA) bus, an Extended ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0209] A processing unit 2004, which may be implemented as one or more integrated circuits (e.g., a conventional microprocessor or microcontroller), controls the operation of computer system 2000. The processing unit 2004 may include one or more processors. These processors may include single-core or multi-core processors. In certain embodiments, the processing unit 2004 may be configured to be implemented as one or more independent processing units 2032 and / or 2034, each including a single-core or multi-core processor. Also, in other embodiments, the processing unit 2004 may be implemented as a quad-core processing unit formed by integrating two dual-core processors on a single chip.

[0210] In various embodiments, the processing unit 2004 can execute various programs in response to program code and maintain multiple concurrently executing programs or processes. At any given time, some or all of the program code being executed may reside in the processor 2004 and / or the storage subsystem 2018. With suitable programming, the processor 2004 can provide the various functions described above. Also, the computer system 2000 may include a processing acceleration unit 2006 that can include a digital signal processor (DSP), an application-specific processor, and / or the like.

[0211] The I / O subsystem 2008 may include a user interface input device and a user interface output device. The user interface input device may include a keyboard, a pointing device such as a mouse or trackball, a touchpad or touch screen incorporated in a display, a scroll wheel, a click wheel, a dial, a button, a switch, a keypad, an audio input device having a voice command recognition system, a microphone, and other types of input devices. The user interface input device may include a motion detection and / or gesture recognition device such as a Microsoft Kinect (registered trademark) motion sensor that enables user control and interaction with input devices such as a Microsoft Xbox (registered trademark) 360 game controller through a natural user interface using gestures and voice commands. Further, the user interface input device may include an eye gesture recognition device such as a Google Glass (registered trademark) blink detector that detects the activity of a user's eyes (e.g., "blinks" during shooting and / or menu selection) and converts the eye gesture into an input to the input device (e.g., Google Glass (registered trademark)). Also, the user interface input device may include a voice recognition detection device that enables interaction of the user with a voice recognition system (e.g., Siri (registered trademark) navigator) via voice commands.

[0212] In addition, the user interface input device may include, but is not limited to, a 3D mouse, a joystick or a pointing stick, a game pad, and a graphic tablet, as well as audio / visual devices such as a speaker, a digital camera, a digital video camera, a portable media player, a web camera, an image scanner, a fingerprint scanner, a barcode reader, a 3D scanner, a 3D printer, a laser range finder, and a gaze tracking device. Further, the user interface input device may include medical imaging input devices such as computer tomography, magnetic resonance imaging, positron emission tomography, and medical ultrasonic examination devices. Additionally, the user interface input device may include audio input devices such as a MIDI keyboard and digital musical instruments.

[0213] The user interface output device may include a non-visual display such as a display subsystem, an indicator light, or an audio output device. The display subsystem may be a flat panel device using a cathode ray tube (CRT), a liquid crystal display (LCD), or a plasma display, a projection device, a touch screen, or the like. Generally, the use of the term "output device" is intended to include all possible types of devices and mechanisms for outputting information from the computer system 2000 to the user or another computer. For example, the user interface output device may include, but is not limited to, a monitor, a printer, a speaker, headphones, an automotive navigation system, a plotter, an audio output device, and a modem, as well as various display devices for visually transmitting text, graphics, and audio / video information.

[0214] The computer system 2000 may include a storage subsystem 2018 that provides a tangible, non-transitory computer-readable storage medium for storing software and data constructs that provide the functionality of the embodiments described in this disclosure. The software may include programs, code modules, instructions, scripts, etc. that, when executed by one or more cores or processors of the processing unit 2004, provide the above-described functionality. Additionally, the storage subsystem 2018 may be configured to provide a repository for storing data used in accordance with this disclosure.

[0215] As shown in the example of FIG. 20, the storage subsystem 2018 may include various components including a system memory 2010, a computer-readable storage medium 2022, and a computer-readable storage medium reader 2020. The system memory 2010 may store program instructions that are loadable and executable by the processing unit 2004. Additionally, the system memory 2010 may be configured to store data used during the execution of the instructions and / or data generated during the execution of the program instructions. Various different types of programs may be loadable into the system memory 2010, including but not limited to client applications, web browsers, middle-tier applications, relational database management systems (RDBMS), virtual machines, containers, etc.

[0216] Also, the system memory 2010 may be configured to store the operating system 2016. Examples of the operating system 2016 include various versions of Microsoft Windows (registered trademark), Apple Macintosh (registered trademark), and / or Linux operating systems, various commercially available UNIX (registered trademark) or UNIX-like operating systems (including, but not limited to, various GNU / Linux operating systems, Google Chrome (registered trademark) OS, etc.), and / or mobile operating systems such as iOS, Windows (registered trademark) Phone, Android (registered trademark) OS, BlackBerry (registered trademark) OS, and Palm (registered trademark) OS. In an embodiment where the computer system 2000 executes one or more virtual machines, the virtual machines may be loaded into the system memory 2010 in conjunction with their respective guest operating systems (GOS) and may be configured to be executed by one or more processors or cores of the processing unit 2004.

[0217] The system memory 2010 can have various configurations depending on the type of the computer system 2000. For example, the system memory 2010 may be volatile memory (such as random access memory (RAM)) and / or non-volatile memory (such as read-only memory (ROM), flash memory, etc.). Also, different types of RAM configurations including static random access memory (SRAM), dynamic random access memory (DRAM), and others may be provided. In some embodiments, the system memory 2010 may include a basic input / output system (BIOS) that contains basic routines useful for transferring information between elements within the computer system 2000 at startup and the like.

[0218] The computer-readable storage medium 2022 can represent a storage medium for temporarily and / or permanently containing, storing computer-readable information (including instructions executable by the processing unit 2004 of the computer system 2000) used by the computer system 2000, in addition to remote, local, fixed, and / or removable storage devices.

[0219] The computer-readable storage medium 2022 can include any suitable medium known or used in the art (including storage media and communication media), and can include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing and / or transmitting information, but is not limited thereto. This can include tangible computer-readable storage media such as memories technologies such as RAM, ROM, electronically erasable programmable ROM (EEPROM), flash memory, optical storage such as CD-ROM, digital versatile disk (DVD), magnetic storage devices such as magnetic cassettes, magnetic tapes, magnetic disk storage, or other tangible computer-readable media.

[0220] As an example, the computer-readable storage medium 2022 may include a hard disk drive that reads and writes to a non-removable non-volatile magnetic medium, a magnetic disk drive that reads and writes to a removable non-volatile magnetic disk, and an optical disk drive that reads and writes to a removable non-volatile optical disk such as a CD-ROM, DVD, Blu-Ray (registered trademark) disk, or other optical medium. The computer-readable storage medium 2022 may include, but is not limited to, a Zip (registered trademark) drive, a flash memory card, a Universal Serial Bus (USB) flash drive, a Secure Digital (SD) card, a DVD disk, a digital video tape, etc. Further, the computer-readable storage medium 2022 may include a semiconductor drive (SSD) based on non-volatile memory such as a flash memory-based SSD, an enterprise flash drive, a semiconductor ROM, a semiconductor RAM, a dynamic RAM, a static RAM, a DRAM-based SSD, a magnetoresistive RAM (MRAM) SSD, and a hybrid SSD that uses a combination of DRAM and flash memory-based SSDs, etc., SSDs based on volatile memory. The disk drive and each associated computer-readable medium may provide non-volatile storage of computer-readable instructions, data structures, program modules, and other data for the computer system 2000.

[0221] Machine-readable instructions executable by one or more processors or cores of the processing unit 2004 may be stored in a non-transitory computer-readable storage medium. The non-transitory computer-readable storage medium may include a physically tangible memory or storage device that includes a volatile memory storage device and / or a non-volatile storage device. Examples of non-transitory computer-readable storage media include magnetic storage media (e.g., disks or tapes), optical storage media (e.g., DVDs, CDs), various types of RAM, ROM, or flash memory, hard drives, floppy (registered trademark) drives, removable memory drives (e.g., USB drives), or other types of storage devices.

[0222] The communication subsystem 2024 provides an interface to other computer systems and networks. The communication subsystem 2024 functions as an interface for sending and receiving data between the computer system 2000 and other systems. For example, the communication subsystem 2024 may enable the computer system 2000 to connect to one or more devices via the Internet. In some embodiments, the communication subsystem 2024 may comprise a wireless voice and / or data network (using advanced data network technologies such as, for example, cellular phone technology, 3G, 4G, or EDGE (Enhanced Data Rates for Global Evolution), WiFi (IEEE 802.11 family standards, or other mobile communication technologies, or any combination thereof)), a Global Positioning System (GPS) receiver component, and / or a radio frequency (RF) transceiver component for accessing other components. In some embodiments, the communication subsystem 2024 may provide a wired network connection (such as Ethernet) as an addition or alternative to the wireless interface.

[0223] Also, in some embodiments, the communication subsystem 2024 can receive input information in the form of structured and / or unstructured data feeds 2026, event streams 2028, event updates 2030, etc., on behalf of one or more users who may use the computer system 2000.

[0224] As an example, the communication subsystem 2024 may be configured to receive the data feed 2026 in real time from users of other communication services, such as social networks and / or web feeds such as Twitter (registered trademark) feeds, Facebook (registered trademark) updates, RSS (Rich Site Summary) feeds, and / or real-time updates from one or more third-party information sources.

[0225] Also, communication subsystem 2024 may be configured to receive data in the form of a continuous data stream, which may include an event stream 2028 of real-time events and / or event updates 2030, which may be continuous or effectively infinite with no explicit end. Examples of applications that generate continuous data include, for example, sensor data applications, financial tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automotive traffic monitoring, and the like.

[0226] Also, communication subsystem 2024 may be configured to output structured and / or unstructured data feeds 2026, event streams 2028, event updates 2030, etc. to one or more databases that may communicate with one or more streaming data source computers coupled to computer system 2000.

[0227] Computer system 2000 can be one of a variety of types including portable handheld devices (e.g., iPhone (registered trademark) cell phones, iPad (registered trademark) computing tablets, PDAs, etc.), wearable devices (e.g., Google Glass (registered trademark) head-mounted displays, etc.), PCs, workstations, mainframes, kiosks, server racks, or any other data processing system.

[0228] Due to the constantly changing nature of computers and networks, the description of the illustrated computer system 2000 is intended as a specific example only. Many other configurations are possible, whether they have more or fewer components than the illustrated system. For example, the use of customized hardware and / or the implementation of specific elements in hardware, firmware, software (including applets), or combinations is also conceivable. Further, connections to other computing devices such as network input / output devices may be employed. Based on the disclosure and teachings described herein, those skilled in the art will recognize other techniques and / or methods for implementing various embodiments.

[0229] Although specific embodiments have been described above, various improvements, changes, alternative configurations, and equivalents are also included within the scope of the present disclosure. Embodiments are not limited to operating within a particular data processing environment and can operate freely within multiple data processing environments. Also, although embodiments have been described using a specific series of transactions and steps, it will be apparent to those skilled in the art that the scope of the present disclosure is not limited to the described series of transactions and steps. The various features and aspects of the above-described embodiments may be used individually or in combination.

[0230] Furthermore, although embodiments have been described using specific combinations of hardware and software, it will be recognized that other combinations of hardware and software are also within the scope of the present disclosure. Embodiments may be implemented only in hardware, only in software, or by using combinations thereof. The various processes described herein can be implemented on the same processor or on any combination of different processors. Thus, when a component or module is described as being configured to perform a certain operation, such a configuration can be achieved, for example, by designing an electronic circuit to perform this operation, programming a programmable electronic circuit (such as a microprocessor) to perform this operation, or by any combination thereof. Processes can communicate using a variety of techniques, including but not limited to conventional techniques for inter-process communication. Also, different sets of processes may use different techniques, or the same set of processes may use different techniques at different times.

[0231] Accordingly, the present specification and drawings are to be regarded in an illustrative rather than a restrictive sense. However, it is obvious that additions, differences, deletions, and other improvements and changes can be made without departing from the broad ideas and scope described in the claims. Thus, while specific embodiments of the present disclosure have been described, there is no intention to limit them thereto. Various improvements and equivalents are included within the scope of the following claims.

[0232] In the context of describing the disclosed embodiments (particularly, in the context of the following claims), the use of the terms "a", "an", and "the", and similar referents shall be construed to cover both the singular and plural forms, unless otherwise indicated herein or there is an obvious contradiction in the context. The terms "comprising", "having", "including", and "containing" shall be construed as open-ended terms (i.e., meaning "including but not limited to") unless otherwise stated. The term "connected" shall be construed to mean attached, mounted, or integrally coupled, even if something intervenes and even if part or all is included. The description of a range of values herein is intended merely to serve as a concise way of referring individually to each separate value falling within the range, and each separate value is incorporated herein as if it were individually described herein, unless otherwise indicated herein. Unless otherwise indicated herein or there is an obvious contradiction in the context, all methods described herein may be performed in any suitable order. The use of any example or exemplary language (e.g., "such as") herein is intended only to facilitate understanding of the embodiments and does not limit the scope of the present disclosure unless otherwise claimed. No representation herein shall be construed as indicating that any unclaimed element is essential to the practice of the present disclosure.

[0233] Unless otherwise specified, disjunctive language such as the phrase "at least one of X, Y, or Z" is intended to be understood within the context in which it is commonly used to present that a listed item, term, etc. can be any one of X, Y, or Z, or any combination thereof (e.g., X, Y, and / or Z). Thus, such disjunctive language is generally not intended, and shall not be construed, to imply that any particular embodiment requires the presence of at least one of each of X, at least one of Y, or at least one of Z.

[0234] This specification describes preferred embodiments of the disclosure, including the best mode known as of the filing date for carrying out the disclosure. Variations of these preferred embodiments will be apparent to those skilled in the art upon reading the above description. Those skilled in the art can adopt such variations as needed, and the disclosure can be practiced otherwise than as specifically described herein. Accordingly, the disclosure includes all improvements and equivalents of the subject matter recited in the claims appended hereto as permitted by applicable law. Further, unless otherwise indicated herein, any combination of the possible variations of the above-described elements is included in the disclosure.

[0235] All references, including publications, patent applications, and patents cited herein, are hereby incorporated by reference to the same extent as if each reference had been individually and specifically indicated to be incorporated by reference and were set forth in its entirety herein.

[0236] In the above specification, aspects of the present disclosure are described with reference to specific embodiments, but those skilled in the art will recognize that the present disclosure is not limited thereto. The various features and aspects of the present disclosure described above may be used individually or integrally. Further, the embodiments can be utilized in any number of environments and applications beyond those described herein and the uses contemplated herein without departing from the broad spirit and scope of this specification. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a limiting sense.< / realm>

Claims

1. 1. A computer-implemented method for communication in a container environment, comprising: A computer system configuring a first node in a container environment, the first node including a first pod representing a group of one or more containers having shared resources; the computer system establishing communication from the first pod through a first virtual communication device; The computer system receives, at a first virtualized network interface card (VNIC) connected to the first node in the container environment, data configured to be transmitted to a destination device from the first virtual communication device assigned to the first pod in the first node; the computer system routing the data, including an address identifying at least the first pod, to the destination device via the first virtualized network interface card; A computer-implemented method, including:

2. the destination device includes a second pod different from the first pod; The computer-implemented method of claim 1 , wherein the first pod and the second pod share the first virtualized network interface card.

3. the destination device includes a second virtualized network interface card different from the first virtualized network interface card; The computer-implemented method of claim 1 , wherein the second virtualized network interface card and the first virtualized network interface card are both associated with the first node of the container environment.

4. the destination device includes a second virtualized network interface card different from the first virtualized network interface card; 10. The computer-implemented method of claim 1, wherein the second virtualized network interface card is associated with a second node of the container environment that is different from the first node.

5. The computer-implemented method of claim 1 , wherein the destination device comprises a virtual cloud network (VCN).

6. 10. The computer-implemented method of claim 1, wherein the first virtual communication device comprises an Internet Protocol Virtual Local Area Network (IPvlan) device.

7. receiving a response at the first virtual communication device of the first pod by the computer system; the computer system sending the response over one or more virtual Ethernet (veth) pairs of the first pod; the computer system sending the response to a container located within the first pod; 2. The computer-implemented method of claim 1, comprising:

8. In response to the computer system determining that the destination device includes a second pod that shares the first virtualized network interface card with the first pod, the computer system routing the data through the first virtualized network interface card using a hairpin operation; the computer system routing the data from the first virtualized network interface card to a second virtual communication device of the second pod; 2. The computer-implemented method of claim 1, comprising:

9. In response to the computer system determining that the destination device includes a second pod of the first node connected to a second virtualized network interface card that is separate from the first virtualized network interface card, the computer system routing the data from the first virtualized network interface card to the second virtualized network interface card; the computer system routing the data from the second virtualized network interface card to a second virtual communication device of the second pod within the first node; 2. The computer-implemented method of claim 1, comprising:

10. In response to the computer system determining that the destination device includes a second pod of a second node distinct from the first node connected to a second virtualized network interface card distinct from the first virtualized network interface card, the computer system routing the data from the first virtualized network interface card to the second virtualized network interface card; the computer system routing the data from the second virtualized network interface card to a second virtual communication device of the second pod in the second node; 2. The computer-implemented method of claim 1, comprising:

11. A system comprising one or more processors configured to perform the method of any one of claims 1 to 10.

12. A computer program causing one or more processors to carry out the method according to any one of claims 1 to 10.

13. In a container environment, means for configuring a first node including a first pod representing a group of one or more containers having shared resources; means for establishing communications from the first pod through a first virtual communications device; means for receiving data configured to be transmitted to a destination device from the first virtual communication device assigned to the first pod in the first node, in a first virtualized network interface card (VNIC) connected to the first node in the container environment; means for routing the data, including an address identifying at least the first pod, to the destination device via the first virtualized network interface card; An apparatus comprising:

14. A computer program product comprising computer instructions that, when executed by a processor, perform the operations of the method according to any one of claims 1 to 10.

Images