Synchronization of communication channel state information for highly available flow

Abstract

To provide a method, system, program, and non-transitory computer-readable storage medium for maintaining and replicating state information relating to communication channels between multiple host machines.SOLUTION: A primary host machine determines changes in communication channel state information for a communication channel having a first endpoint in a customer on-premises network and a second endpoint on a primary host machine in a cloud service provider infrastructure, identifies a backup host machine for the communication channel, and replicates the changes in state information to the backup host machine. The changes in state information are replicated to the backup host machine when the backup host machine becomes the second endpoint of the communication channel due to fail-over. The stored state information is usable by the backup host machine.SELECTED DRAWING: Figure 11

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of and priority under 35 U.S.C. §119(e) to U.S. Provisional Application No. 63 / 132,036, filed December 30, 2020, and U.S. Nonprovisional Application No. 17 / 556,540, filed December 20, 2021. The entire contents of the foregoing applications are incorporated herein by reference for all purposes. [Background technology]

[0002] background Demand for cloud-based services continues to grow rapidly. The term cloud service generally refers to services provided to users or customers on demand (e.g., via a subscription model) using systems and infrastructure (cloud infrastructure) provided by a cloud service provider. Typically, the servers and systems that make up the cloud service provider's infrastructure are separate from the customer's own on-premises servers and systems. Therefore, customers can use cloud services provided by cloud service providers without having to separately purchase hardware and software resources for the services. Cloud services include various different types of services, such as Software-as-a-Service (SaaS), Platform-as-a-Service (PaaS), and Infrastructure-as-a-Service (IaaS).

[0003] In the IaaS model, a cloud service provider provides infrastructure resources (e.g., compute resources, memory resources, networking resources such as servers, and storage devices) that customers can use to build their own resources and virtual networks. The infrastructure provided by the IaaS cloud service provider, including interconnected high-performance computers, memory, and network resources or components, forms a physical network (also called a substrate network or underlay network). The physical network provides the basis for creating customer virtual networks (also called overlay networks) on top of the physical network.

[0004] Cloud service providers typically offer a wide range of applications, services, and APIs that enable customers (e.g., enterprises) to efficiently create virtual networks and deploy and manage various virtual workloads (e.g., virtual machines, containers, and applications) on these virtual networks in a highly available, distributed environment. Customers can generally manage their cloud-based workloads in the same way they manage on-premises workloads, gaining all the benefits of high-performance computing and networking capabilities with the same control, isolation, security, and predictable performance as on-premises networks.

[0005] A customer typically has one or more customer on-premises networks and one or more virtual networks deployed in the cloud using infrastructure provided by the cloud service provider. The customer on-premises networks can communicate with the customer virtual cloud network by providing a variety of different communication mechanisms. To establish communication, devices or equipment within the customer on-premises network must be configured to participate in these communications. For example, a customer on-premises network and a customer virtual network hosted on a cloud service infrastructure (CSPI) can be connected to each other. You must properly configure the customer-premises equipment (CPE), which is one endpoint of the communication channel between the CPE and the host machine, which is another endpoint in the network. You must configure an Internet Protocol Security (IPsec) tunnel to securely handle traffic between the CPE and CSPI endpoints.

[0006] Customers expect high availability of cloud services, with minimal downtime even if there is a problem with the underlying infrastructure (e.g., a host machine goes down or there is a problem with the cabling). To achieve this, cloud service providers build redundancy into their infrastructure. For example, a host machine that processes network packets has one or more backup systems (backup host machines) designated. This ensures that if a problem occurs with a host machine and it is no longer able to process packets, one of the backup host machines will take over processing the packets. This minimizes any disruption to customers. Typically, when a host machine goes down, the communication channel related to that host machine also goes down, causing all traffic in both directions to be dropped.

[0007] When an endpoint or host machine in a customer virtual network goes down, it can take a long time (e.g., several seconds to several minutes) for the backup system to negotiate a new communication channel. If the customer virtual cloud network is equipped with a backup communication channel to handle packets in the event of an endpoint failure, the solution of establishing the backup channel and dynamically routing packets to the backup channel during failover can be very complex and time-consuming, especially when the solution is deployed across a distributed environment. Summary of the Invention

[0008] overview

[0001] The present disclosure relates generally to techniques for improving the availability of cloud services, and more particularly to techniques for maintaining and replicating communication channel-related state information between multiple host machines. This specification describes various embodiments, including methods, systems, and non-transitory computer-readable storage media that store programs, code, or instructions executable by one or more processors.

[0009] An aspect of the present disclosure provides a method, for a communication channel having a first endpoint in a customer on-premises network and a second endpoint on a primary host machine in a cloud service provider infrastructure, the method including: the primary host machine determining changes in state information for the communication channel; the primary host machine identifying a backup host machine for the communication channel; and the primary host machine replicating the changes in state information to the backup host machine, wherein the state information replicated and stored on the backup host machine is available to the backup host machine when the backup host machine becomes the second endpoint of the communication channel due to a failover.

[0010] Another aspect of the present disclosure provides a non-transitory computer-readable storage medium storing computer-executable instructions that, when executed, cause a processor of a computer system to perform a method, the method including, for a communication channel having a first endpoint in a customer on-premises network and a second endpoint on a primary host machine in a cloud service provider infrastructure, the primary host machine determining a change in state information for the communication channel, the primary host machine identifying a backup host machine for the communication channel, and the primary host machine replicating the change in state information to the backup host machine, thereby providing a failover to the backup host machine. The state information replicated and stored on the backup host machine is available for use by the backup host machine when the backup host machine becomes the second endpoint of the communication channel.

[0011] This specification describes various embodiments, including methods, systems, and non-transitory computer-readable storage media that store programs, code, or instructions executable by one or more processors. These exemplary embodiments are not described to limit or define the disclosure, but to provide examples to aid in its understanding. Additional embodiments are further described and discussed in the detailed description.

[0012] The features, embodiments, and advantages of the present disclosure will be better understood from the following detailed description when read in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a high-level diagram of a distributed environment illustrating a virtual or overlay cloud network hosted by a cloud service provider infrastructure, according to certain embodiments. [Figure 2] 1 is an architectural schematic diagram illustrating physical elements of a physical network within a CSPI, according to certain embodiments. [Figure 3]FIG. 1 illustrates an exemplary arrangement of CSPI in which a host machine is connected to multiple network virtualization devices (NVDs), according to certain embodiments. [Figure 4] FIG. 1 illustrates a connection between a host machine and an NVD that provides I / O virtualization to support multi-tenancy, according to certain embodiments. [Figure 5] 1 is a schematic block diagram illustrating a physical network provided by CSPI in accordance with certain embodiments. [Figure 6] FIG. 1 is a schematic block diagram illustrating a distributed environment incorporating an exemplary embodiment for replicating communication channel-related state information, in accordance with certain embodiments. [Figure 7A] FIG. 2 is an exemplary block diagram illustrating multiple host machines in a replication chain, according to certain embodiments. [Figure 7B] FIG. 2 is another exemplary block diagram illustrating multiple host machines in a replication chain, according to certain embodiments. [Figure 8] 1 is a schematic flow chart illustrating a method for replicating communication channel related state information to a backup host machine, according to certain embodiments. [Figure 9] 1 is a schematic flow diagram illustrating a replication channel replicating state information related to a communication channel, according to certain embodiments; [Figure 10] 10 is a schematic flow chart illustrating detailed processing performed by a primary host machine to replicate communication channel-related state information in accordance with certain embodiments. [Figure 11] 1 is a schematic flow chart illustrating steps performed to handle packet processing during a failover of a primary host machine, in accordance with certain embodiments. [Figure 12] FIG. 1 is a block diagram illustrating one pattern for implementing a cloud infrastructure as a service system in accordance with at least one embodiment. [Figure 13]FIG. 1 is a block diagram illustrating another pattern for implementing a cloud infrastructure as a service system in accordance with at least one embodiment. [Figure 14] FIG. 1 is a block diagram illustrating another pattern for implementing a cloud infrastructure as a service system in accordance with at least one embodiment. [Figure 15] FIG. 1 is a block diagram illustrating another pattern for implementing a cloud infrastructure as a service system in accordance with at least one embodiment. [Figure 16] FIG. 1 is a block diagram illustrating an exemplary computer system in accordance with at least one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0014] Detailed Description In the following description, for purposes of explanation, specific details are set forth in order to provide a thorough understanding of particular embodiments. It will be apparent, however, that various embodiments may be practiced without these specific details. The drawings and description are not intended to be limiting. The term "exemplary" is used in this disclosure to mean "serving as an example, instance, or illustration." Any embodiment or design described in this disclosure as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments or designs.

[0015] This disclosure describes techniques for synchronizing state information for communication channels of multiple host machines hosted by a Cloud Service Provider Infrastructure (CSPI) for highly available flows. More specifically, techniques are described for synchronizing state information for communication channels handling traffic between endpoints in a customer on-premises network and endpoints in a virtual network hosted on the Cloud Service Provider Infrastructure (CSPI).

[0016] A customer subscribes to one or more cloud services offered by a cloud service provider (CSP) using a CSPI. A communication channel (e.g., an IPsec tunnel) may be configured to securely transmit packets between a customer premise equipment (CPE) in the customer's on-premises network, which is the communication endpoint, and a host machine in the cloud service infrastructure's (CSPI's) customer virtual network, which is the primary endpoint. The two endpoints establish the communication channel by negotiating with each other. Specifically, the two endpoints can negotiate an agreement regarding encryption and authentication-related parameters to be used for communication by sending and receiving a series of encryption- and authentication-related messages between them. This negotiation determines a security association (SA) between the two endpoints, which specifies the security characteristics agreed upon and approved by the communicating endpoints. A single SA can protect data in one direction. The SA provides both endpoints with details about how to encrypt and decrypt communication packets between them.

[0017] Typically, if the endpoint receiving traffic goes down for any reason (e.g., failover or maintenance), the communication channel or tunnel also goes down, causing all traffic in both directions to be dropped. If an endpoint within the cloud infrastructure (CSPI) goes down, the infrastructure can provide a backup system to take over handling traffic between the CPE endpoint and the CSPI. However, this can be complex and time-consuming because the backup system must negotiate a new tunnel with the CPE endpoint to send and receive traffic on the backup system.

[0018] Establishing a new tunnel in the event of a failover can take several minutes due to a series of negotiations and configurations that must occur between the endpoints. While the backup tunnel is being established, applications using certain types of connections, such as TCP connections, may time out, which is unacceptable to customers. For example, if an online shopper is shopping on a website and the website's database is located on a cloud network provided by a service provider, a detected failure of the endpoint in the cloud network hosting the database could result in the shopping transaction failing or the shopping cart timing out. This is unacceptable to cloud service customers. Certain service providers offer backup tunnels to handle traffic between CPE endpoints and backup systems in the event that the primary endpoint hosted on the CSPI fails. However, handling traffic using backup tunnels to backup systems requires establishing backup tunnels and complex, dynamic routing of traffic to the backup tunnels whenever the primary endpoint goes down. These solutions are complex and time-consuming to implement in cloud computing environments where systems within the CSPI manage a large number of customers communicating using Layer 3 protocols.

[0019] This disclosure discusses a method for handling network traffic using the same communication channel to a backup endpoint that was used to handle packets between a CPE endpoint and a primary CSPI endpoint. This method provides a highly available tunnel that can handle traffic at a different endpoint, avoiding the complexity of having two tunnels and the risk of losing connectivity while negotiating a new connection for the backup tunnel.

[0020] The term cloud services generally refers to services provided on demand (e.g., via a subscription model) by a cloud service provider (CSP) using systems and infrastructure (cloud infrastructure). Typically, the servers and systems that make up the CSP's infrastructure are separate from the customer's own on-premise servers and systems. Therefore, customers can use cloud services provided by the CSP without separately purchasing hardware and software resources for the service. Cloud services are designed to provide subscribing customers with easy and scalable access to application and computing resources without requiring the customer to invest in procuring the infrastructure used to deliver the service.

[0021] Several cloud service providers (CSPs) offer various types of cloud services, including a variety of different types or models such as Software-as-a-Service (SaaS), Platform-as-a-Service (PaaS), and Infrastructure-as-a-Service (IaaS).

[0022] A customer can subscribe to one or more cloud services offered by a CSP. A customer can be any entity, such as an individual, an organization, or a business. When a customer subscribes or registers for a service offered by a CSP, a tenant or account is created for that customer. The customer can then access one or more subscribed cloud resources associated with the account through this account.

[0023] As mentioned above, IaaS (Infrastructure as a Service) is a service that provides services to a specific type of client. Cloud computing services. In the IaaS model, a CSP provides infrastructure (called cloud service provider infrastructure or CSPI) that customers can use to build their own customizable networks and deploy customer resources. Therefore, customer resources and networks are hosted in a distributed environment by the infrastructure provided by the CSP. This differs from traditional computing, where the customer's infrastructure hosts the customer's resources and networks.

[0024] CSPI is a network of various host machines, memory resources, and network resources that form the physical network, also known as the substrate network or underlay network. A CSPI physical network may include interconnected high-performance computing resources, including servers. CSPI resources may be distributed across one or more data centers, geographically dispersed across one or more geographic regions. Virtualization software can run on these physical resources to provide a virtualized distributed environment. Virtualization creates an overlay network (also called a software-based network, software-defined network, or virtual network) on top of the physical network. The CSPI physical network provides the basis for creating one or more overlay or virtual networks on top of the physical network. A virtual or overlay network can include one or more virtual cloud networks (VCNs). Virtual networks are implemented using software virtualization technologies (e.g., hypervisors, functions performed by network virtualization devices (NVDs) (e.g., smart NICs), top-of-rack (TOR) switches, smart TORs that implement one or more functions performed by NVDs, and other mechanisms) to create a network abstraction layer that can run on top of the physical network. Virtual networks can take various forms, such as peer-to-peer networks and IP networks. Virtual networks are typically either Layer 3 IP networks or Layer 2 VLANs. Such virtual networks or overlay networks are often called virtual Layer 3 networks or overlay Layer 3 networks. Examples of protocols developed for virtual networks include IP-in-IP (or Generic Routing Encapsulation (GRE)), Virtual Extensible LAN (VXLAN - IETF RFC7348), virtual private networks (VPNs) (e.g., MPLS Layer 3 Virtual Private Networks (RFC4364)), VMware NSX, Generic Network Virtualization Encapsulation (GENEVE), etc.

[0025] In IaaS, the infrastructure provided by the CSP (CSPI) may be configured to provide virtualized computing resources over a public network (e.g., the Internet). In the IaaS model, a cloud computing service provider may host infrastructure elements (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer), etc.). In some cases, the IaaS provider may offer various services (e.g., billing, monitoring, logging, security, load balancing, clustering, etc.) associated with those infrastructure elements. Because these services are policy-driven, IaaS users can maintain application availability and performance by implementing policies to drive load balancing. The CSPI provides infrastructure and a set of complementary cloud services, enabling customers to build and run a wide range of applications and services in a highly available, hosted, distributed environment. The CSPI provides high-performance computing resources and power, as well as storage capacity, over a flexible virtual network that can be securely accessed from various network locations, such as customer on-premises networks. When a customer subscribes or registers for an IaaS service offered by a CSP, the tenancy created for that customer is a secure, isolated partition from the CSP where the customer can create, organize, and manage their cloud resources.

[0026] Customers can build their own virtual networks using the compute, memory, and networking resources provided by CSPI. On these virtual networks, they can connect one or more customer resources or workloads, such as compute instances. For example, a customer can use resources provided by the CSP to build one or more customizable private virtual networks called Virtual Cloud Networks (VCNs). The customer can then deploy one or more customer resources, such as compute instances, on the customer VCN. The compute instances may be virtual machines, bare metal instances, etc. Thus, the CSP I provides infrastructure and a set of complementary cloud services that enable customers to build and run a variety of applications and services in a highly available, virtualized, hosted environment. Customers do not manage or control the underlying physical resources provided by CSPI, but they do control the operating systems, storage, and deployed applications, and in some cases have limited control over some networking components (e.g., firewalls).

[0027] The CSP may provide a console that enables customers and network administrators to configure, access, and manage resources deployed in the cloud using CSPI resources. In certain embodiments, the console provides a web-based user interface that can be used to utilize and manage CSPI. In some embodiments, the console is a web-based application provided by the CSP.

[0028] CSPI can support single-tenancy or multi-tenancy architectures. In a single-tenancy architecture, software (e.g., applications, databases) or hardware elements (e.g., host machines or servers) serve a single customer or tenant. In a multi-tenancy architecture, software or hardware elements serve multiple customers or tenants. Thus, in a multi-tenancy architecture, CSPI resources are shared among multiple customers or tenants. In a multi-tenancy environment, CSPI employs precautions and safeguards to ensure that each tenant's data is isolated and not visible to other tenants.

[0029] In a physical network, a network endpoint (endpoint) refers to a computing device or system that is connected to the physical network and communicates bidirectionally with the connected network. A network endpoint of a physical network may be connected to a local area network (LAN), a wide area network (WAN), or other types of physical networks. Examples of traditional endpoints of a physical network include modems, hubs, bridges, switches, routers, and other networking devices, physical computers (or host machines), etc. Each physical device of a physical network has a fixed network address that can be used to communicate with the device. This fixed network address may be a Layer 2 address (e.g., a MAC address), a fixed Layer 3 address (e.g., an IP address), etc. In a virtualized environment or virtual network, endpoints may include various virtual endpoints, such as virtual machines hosted by elements of the physical network (e.g., hosted by a physical host machine). These endpoints of the virtual network are addressed by overlay addresses, such as overlay Layer 2 addresses (e.g., an overlay MAC address) and overlay Layer 3 addresses (e.g., an overlay IP address). Network overlays achieve flexibility by allowing network administrators to move overlay addresses associated with network endpoints using software management (e.g., via software implementing the virtual network's control plane). Thus, unlike physical networks, in virtual networks, overlay addresses (e.g., overlay IP addresses) can be moved from one endpoint to another using network management software. Because virtual networks are built on physical networks, both the virtual network and the underlying physical network are involved in communications between elements of the virtual network.To facilitate such communication, each CSPI element maps its virtual network's overlay address to a real physical address on the substrate network, or vice versa. The virtual network is configured to learn and store mappings that map customer traffic to virtual addresses. These mappings are used to facilitate communications. To facilitate routing in the virtual network, customer traffic is encapsulated.

[0030] Thus, physical addresses (e.g., physical IP addresses) are associated with elements of a physical network, and overlay addresses (e.g., overlay IP addresses) are associated with entities of a virtual network. Both physical and overlay IP addresses are real IP addresses. They are distinct from virtual IP addresses. Virtual IP addresses map to multiple real IP addresses. Virtual IP addresses provide a one-to-many mapping between virtual IP addresses and multiple real IP addresses.

[0031] A cloud infrastructure or CSPI is physically hosted in one or more data centers in one or more regions around the world. A CSPI may include elements of a physical or substrate network and virtualized elements (e.g., virtual networks, compute instances, virtual machines) of a virtual network built on top of the physical network elements. In certain embodiments, a CSPI may be organized into realms, regions, and CSPI resources are organized and hosted in available domains. A region is typically a local geographic area containing one or more data centers. Regions are generally independent of one another and may be separated by vast distances, e.g., across countries or continents. For example, a first region may be in Australia, another region may be in Japan, and yet another region may be in India. CSPI resources are divided among these regions so that each region has an independent subset of CSPI resources. Each region can provide a set of core infrastructure services and resources, such as compute resources (e.g., bare metal servers, virtual machines, containers, and related infrastructure), storage resources (e.g., block volume storage, file storage, object storage, archive storage), networking resources (e.g., virtual cloud networks (VCNs), load balancing resources, connectivity to on-premises networks), database resources, edge networking resources (e.g., DNS), access management, and monitoring resources. Each region typically has multiple routes connecting it to other regions within the realm.

[0032] Typically, applications are deployed in the region where they will be used the most (i.e., on infrastructure associated with that region) because using nearby resources is faster than using resources that are farther away. Applications may also be deployed in different regions for a variety of reasons, such as redundancy to mitigate the risk of region-wide events such as large weather systems or earthquakes, or to meet various requirements for legal jurisdictions, tax domains, and other business or societal criteria.

[0033] Data centers within a region may be further organized and subdivided into availability domains (ADs). An availability domain is one or more data centers located in a region. A region may correspond to a data center in the cloud. A region may be composed of one or more availability domains. In such a distributed environment, CSPI resources may be region-specific, such as a virtual cloud network (VCN), or availability domain-specific, such as a compute instance.

[0034] ADs within a region are isolated from each other to be fault tolerant and configured to have a very low probability of simultaneous failure. This is achieved by configuring ADs to not share critical infrastructure resources such as networking, physical cables, cable routes, and cable entrances, so that the failure of one AD in a region has little or no impact on the availability of other ADs in the same region. ADs within the same region can be connected to each other via low-latency, high-bandwidth networks to provide highly available connections to other networks (e.g., the Internet, customer on-premises networks) and to create replicated systems across multiple ADs for both high availability and disaster recovery. Cloud services utilize multiple ADs to ensure high availability and protect against resource failures. As the infrastructure provided by an IaaS provider grows, more regions and ADs may be added along with additional capacity. Traffic between available domains is typically encrypted.

[0035] In certain embodiments, regions are grouped into realms. A realm is a logical collection of regions. Realms are isolated from each other and do not share any data. Regions within the same realm can communicate with each other, but regions within different realms cannot. A CSP's customer tenancy or account exists in a single realm and can span one or more regions within that single realm. Typically, when a customer subscribes to an IaaS service, their tenancy or account is created in a customer-specified region (called their "home" region) within a realm. The customer can extend their tenancy to one or more other regions within the realm. The customer cannot access regions that do not exist within the realm in which the customer's tenancy resides.

[0036] An IaaS provider may offer multiple realms, each corresponding to a particular set of customers or users. For example, a commercial realm may be offered for commercial customers. As another example, a realm may be offered for a particular country or for customers in that country. As yet another example, a government realm may be offered, for example, for a government. For example, a government realm may be created for a particular government and may have a higher security level than a commercial realm. For example, Oracle® Cloud Infrastructure (OCI) currently offers a realm for the commercial domain and two realms for the government cloud domain (e.g., FedRAMP-authorized and IL5-authorized).

[0037] In certain embodiments, an AD can be subdivided into one or more fault domains. A fault domain is a grouping of infrastructure resources within an AD to provide anti-affinity. Fault domains can distribute compute instances so that compute instances are not located on the same physical hardware within an AD. This is known as anti-affinity. A fault domain refers to a collection of hardware elements (computers, switches, etc.) that share a single point of failure. A compute pool is logically divided into fault domains. Thus, a hardware failure or compute hardware maintenance event that affects one fault domain does not affect instances in other fault domains. Depending on the embodiment, the number of fault domains in each AD may vary. For example, in certain embodiments, each AD includes three fault domains. Fault domains function as logical data centers within an AD.

[0038] When a customer subscribes to an IaaS service, resources from CSPI are provisioned to the customer and associated with the customer's tenancy. Customers can use these provisioned resources to build private networks and deploy resources on these networks. A customer network hosted on the cloud by CSPI is called a Virtual Cloud Network (VCN). Customers can configure one or more Virtual Cloud Networks (VCNs) using the CSPI resources allocated for the customer. A VCN is a virtual or software-defined private network. Customer resources deployed in a customer's VCN can include compute instances (e.g., virtual machines, bare metal instances) and other resources. These compute instances may represent various customer workloads, such as applications, load balancers, databases, etc. Compute instances deployed on a VCN can be used to connect to the Internet. It can communicate with publicly accessible endpoints (public endpoints) over a public network such as the internet, it can communicate with other instances in the same VCN or other VCNs (e.g., other VCNs of the customer or VCNs that do not belong to the customer), it can communicate with customer on-premises data centers or networks, it can communicate with service endpoints, and it can communicate with other types of endpoints.

[0039] CSPs can offer various services using CSPI. In some cases, customers of a CSPI themselves can act as service providers and provide services using CSPI resources. Service providers can expose service endpoints characterized by identifying information (e.g., IP addresses, DNS names, and ports). Customer resources (e.g., compute instances) can consume a particular service by accessing the service endpoint for that particular service exposed by the service. These service endpoints are generally publicly accessible over a public communications network, such as the Internet, with users using the public IP address associated with the endpoint. Publicly accessible network endpoints are sometimes referred to as public endpoints.

[0040] In certain embodiments, a service provider may expose a service through a service endpoint (sometimes referred to as a service endpoint). Customers of the service may access the service using this service endpoint. In certain embodiments, a service endpoint provided for a service may be accessed by multiple customers wishing to consume the service. In other implementations, a dedicated service endpoint may be provided to a customer. Thus, only that customer may access the service using that dedicated service endpoint.

[0041] In certain embodiments, when a VCN is created, it is assigned a private overlay IP address range (e.g., 10.0 / 16) called Private Overlay Classless Inter-Domain Routing (PIR). A VCN is associated with a CIDR (Communication Override) address space. It contains associated subnets, route tables, and gateways. A VCN exists within a single region but can extend to one or more or all available domains in the region. A gateway is a virtual interface configured for a VCN that enables traffic communication between the VCN and one or more endpoints outside the VCN. You can configure one or more different types of gateways for a VCN to enable communication between different types of endpoints.

[0042] A VCN may be subdivided into one or more subnetworks, such as one or more subnets. A subnet is thus a building block or division that can be created within a VCN. A VCN can have one or more subnets. Each subnet within a VCN is associated with a contiguous range of overlay IP addresses (e.g., 10.0.0.0 / 24 and 10.0.1.0 / 24) that does not overlap with other subnets within the VCN and represents a subset of the VCN's address space.

[0043] Each compute instance is associated with a virtual network interface card (VNIC), which allows it to participate in a subnet of a VCN. A VNIC is a logical representation of a physical network interface card (NIC). In general, a VNIC is a logical representation of an entity (e.g., compute instance, service) A VNIC is an interface between a compute instance and a virtual network. A VNIC resides in a subnet and has one or more associated IP addresses and associated security rules or policies. A VNIC is equivalent to a Layer 2 port on a switch. A VNIC is connected to a compute instance and a subnet within a VCN. A VNIC associated with a compute instance allows the compute instance to be part of a subnet in a VCN, allowing the compute instance to communicate (e.g., send and receive packets) with endpoints on the same subnet as the compute instance, endpoints in a different subnet within the VCN, or endpoints outside the VCN. Thus, a VNIC associated with a compute instance determines how the compute instance connects with endpoints inside and outside the VCN. A VNIC for a compute instance is created and associated with the compute instance when the compute instance is created and added to a subnet within a VCN. If a subnet consists of a set of compute instances, it contains VNICs corresponding to the set of compute instances, and each VNIC is connected to a compute instance in the set of compute instances.

[0044] Each compute instance is assigned a private overlay IP address via the VNIC associated with the compute instance. This private overlay IP address is assigned to the VNIC associated with the compute instance when the compute instance is created and is used to route traffic for the compute instance. All VNICs in a particular subnet use the same route table, security lists, and DHCP options. As described above, each subnet in a VCN is associated with a contiguous range of overlay IP addresses (e.g., 10.0.0.0 / 24 and 10.0.1.0 / 24) that do not overlap with other subnets in the VCN and represent an address space subset of the VCN's address space. For a VNIC on a particular subnet of a VCN, the overlay IP address assigned to the VNIC is an address from the contiguous range of overlay IP addresses assigned to the subnet.

[0045] In certain embodiments, if desired, a compute instance can be assigned additional overlay IP addresses in addition to the private overlay IP address, for example, one or more public IP addresses in the case of a public subnet. These multiple addresses are assigned to the same VNIC or multiple VNICs associated with the compute instance. However, each instance has a primary VNIC associated with the overlay private IP address that is created and assigned to the instance at instance launch. This primary VNIC cannot be deleted. Additional VNICs, called secondary VNICs, can be added to an existing instance within the same availability domain as the primary VNIC. All VNICs are in the same availability domain as the instance. Secondary VNICs can be in the same subnetwork in the same VCN as the primary VNIC, or in different subnetworks in the same or a different VCN.

[0046] Compute instances can optionally be assigned public IP addresses if they are in a public subnet. When creating a subnet, you can specify that the subnet is either a public or private subnet. A private subnet means that resources (e.g., compute instances) and associated VNICs within the subnet cannot have public overlay IP addresses. A public subnet means that resources and associated VNICs within the subnet can have public IP addresses. Customers can specify subnets that exist across a single available domain or multiple available domains within a region or realm.

[0047] As described above, a VCN may be subdivided into one or more subnets. In certain embodiments, a virtual router (referred to as a VCN VR or simply VR) configured for a VCN enables communication between subnets of the VCN. For a subnet within a VCN, the VR represents the logical gateway for that subnet, enabling communication between the subnet (i.e., the compute instances on that subnet) and endpoints on other subnets within the VCN and other endpoints outside the VCN. A VCN VR is a logical entity configured to route traffic between VNICs in a VCN and a virtual gateway (gateway) associated with the VCN. Gateways are further described below with respect to FIG. 1. A VCN VR is a Layer 3 / IP layer concept. In one embodiment, there is one VCN VR per VCN. This VCN VR potentially has an unlimited number of ports addressed by IP addresses, one port for each subnet of the VCN. In this way, a VCN VR has a different IP address for each subnet of the VCN to which the VCN VR is connected. VRs are also connected to various gateways configured for the VCN. In certain embodiments, a particular overlay IP address from a subnet's overlay IP address range is reserved in a port of the subnet's VCN VR. For example, consider a VCN with two subnets, each with associated address ranges 10.0 / 16 and 10.1 / 16. For a first subnet in the VCN with address range 10.0 / 16, addresses from this range are reserved in a port of the subnet's VCN VR. In some cases, a first IP address from this range may be reserved in a VCN VR. For example, for a subnet with overlay IP address range 10.0 / 16, IP address 10.0.0.1 may be reserved in a port of the subnet's VCN VR. For a second subnet in the same VCN with address range 10.1 / 16, the VCN VR may have a port of the second subnet with IP address 10.1.0.1.A VCN VR has a different IP address for each subnet in the VCN.

[0048] In some other embodiments, each subnet within a VCN may have its own associated VR that is addressable by the subnet using a reserved or default IP address associated with the VR. The reserved or default IP address may, for example, be the first IP address from a range of IP addresses associated with the subnet. VNICs within a subnet can use this default or reserved IP address to communicate (e.g., send and receive packets) with the VR associated with the subnet. In such embodiments, a VR is the ingress / egress point for that subnet. VRs associated with subnets within a VCN can communicate with other VRs associated with other subnets within the VCN. VRs can also communicate with gateways associated with the VCN. The VR functions for a subnet are performed on or by one or more NVDs that perform the VNIC functions for VNICs within the subnet.

[0049] Route tables, security rules, and DHCP options may be configured for a VCN. A route table is a virtual route table for a VCN and contains rules for routing traffic from subnets inside the VCN to destinations outside the VCN through gateways or specially configured instances. You can customize a VCN's route table to control the forwarding / routing of packets into and out of the VCN. DHCP options refer to configuration information that is automatically provided to an instance when it is launched.

[0050] The security rules configured for a VCN represent the overlay firewall rules for the VCN. Security rules can include inbound and outbound rules. A VCN can specify the type of traffic allowed in and out of instances in the VCN (for example, based on protocol and port). Customers can choose whether a particular rule is stateful or stateless. For example, a customer can allow incoming SSH traffic from anywhere to a set of instances by configuring a stateful inbound rule with source CIDR 0.0.0.0 / 0 and destination TCP port 22. Security rules may be implemented using network security groups or security lists. A network security group consists of a set of security rules that apply only to resources within that group. A security list, on the other hand, contains rules that apply to all resources in a subnet that uses that security list. A VCN may include default security rules and a default security list. DHCP options configured for a VCN provide configuration information that is automatically provided when instances in the VCN are launched.

[0051] In particular embodiments, configuration information for a VCN is determined and stored by a VCN control plane. The configuration information for a VCN may include, for example, address ranges associated with the VCN, subnets and associated information within the VCN, one or more VRs associated with the VCN, compute instances and associated VNICs within the VCN, NVDs that perform various virtualized network functions (e.g., VNICs, VRs, gateways) associated with the VCN, VCN state information, and other VCN-related information. In particular embodiments, a VCN distribution service publishes the configuration information stored by the VCN control plane, or portions thereof, to the NVD. The distributed information can be used to forward packets to and from compute instances within the VCN by updating information stored and used by the NVD (e.g., forwarding tables, routing tables, etc.).

[0052] In certain embodiments, VCN and subnet creation is handled by a VCN control plane (CP), and compute instance launch is handled by the compute control plane. The compute control plane is configured to allocate physical resources for the compute instance and then invoke the VCN control plane to create and attach VNICs to the compute instance. The VCN CP also sends VCN data mappings to a VCN data plane configured to perform packet forwarding and routing functions. In certain embodiments, the VCN CP provides a distribution service configured to provide updates to the VCN data plane. Examples of the VCN control plane are shown in Figures 12, 13, 14, and 15 (see reference numerals 1216, 1316, 1416, and 1516) and described below.

[0053] Customers can create one or more VCNs with resources hosted by CSPI. Compute instances deployed on a customer VCN can communicate with different endpoints. These endpoints can include endpoints hosted by CSPI and endpoints external to CSPL.

[0054] Various different architectures for implementing cloud-based services using CSPI are shown in Figures 1, 2, 3, 4, 5, 12, 13, 14, and 15 and described below. Figure 1 is a high-level diagram of a distributed environment 100 showing an overlay VCN or customer VCN hosted by CSPI, according to certain embodiments. The distributed environment shown in Figure 1 includes multiple elements in an overlay network. The distributed environment 100 shown in Figure 1 is merely an example and is not intended to unduly limit the scope of the claimed embodiments. Many variations, alternatives, and modifications are possible. For example, in some implementations, the distributed environment shown in Figure 1 may have more or fewer systems or elements than those shown in Figure 1, may combine two or more systems, or It may have different system configurations or arrangements.

[0055] As shown in the example of FIG. 1 , distributed environment 100 includes CSPI 101, which provides services and resources that customers can subscribe to and use to build a virtual cloud network (VCN). In a particular embodiment, CSPI 101 provides IaaS services to subscribing customers. Data centers within CSPI 101 may be organized into one or more regions. FIG. 1 shows an example region, "US Region" 102. A customer configures a customer VCN 104 for region 102. A customer can deploy various compute instances on VCN 104, which may include virtual machines or bare metal instances. Example instances include applications, databases, load balancers, etc.

[0056] In the embodiment shown in Figure 1, customer VCN 104 includes two subnets, "Subnet-1" and "Subnet-2," each with its own CIDR IP address range. In Figure 1, the overlay IP address range of Subnet-1 is 10.0 / 16, and the address range of Subnet-2 is 10.1 / 16. VCN virtual router 105 represents the logical gateway of the VCN, enabling communication between subnets of VCN 104 and with other endpoints outside the VCN. VCN VR105 is configured to route traffic between VNICs in VCN104 and gateways associated with VCN104. VCN VR105 provides a port for each subnet of VCN104. For example, VR105 may provide a port with IP address 10.0.0.1 for subnet-1 and a port with IP address 10.1.0.1 for subnet-2.

[0057] Multiple compute instances can be deployed on each subnet. In this case, the compute instances may be virtual machine instances and / or bare metal instances. The compute instances within a subnet may be hosted by one or more host machines within CSPI 101. A compute instance joins a subnet through a VNIC associated with the compute instance. For example, as shown in FIG. 1, compute instance C1 is part of subnet-1 through a VNIC associated with the compute instance. Similarly, compute instance C2 is part of subnet-1 through a VNIC associated with C2. Similarly, multiple compute instances, which may be virtual machine instances or bare metal instances, may be part of subnet-1. Each compute instance is assigned a private overlay IP address and MAC address through the associated VNIC. For example, in FIG. 1, compute instance C1 has an overlay IP address of 10.0.0.2 and a MAC address of M1, and compute instance C2 has a private overlay IP address of 10.0.0.3 and a MAC address of M2. Each compute instance in Subnet-1, including compute instances C1 and C2, has a default route to VCN VR105 using IP address 10.0.0.1, which is the IP address of a port in VCN VR105 in Subnet-1.

[0058] Subnet-2 may be deployed with multiple compute instances, including virtual machine instances and / or bare metal instances. For example, as shown in FIG. 1, compute instances D1 and D2 are part of subnet-2 via VNICs associated with the respective compute instances. In the embodiment shown in FIG. 1, compute instance D1 has an overlay IP address of 10.1.0.2 and a MAC address of MM1, and compute instance D2 has a private overlay IP address of 10.1.0.3 and a MAC address of MM2. Each compute instance in subnet-2, including compute instances D1 and D2, has a default connection to VCN VR105 using IP address 10.1.0.1, which is the IP address of a port in VCN VR105 in subnet-2. It has a root.

[0059] VCN A 104 may also include one or more load balancers. For example, a load balancer may be provided for a subnet and configured to load balance traffic among multiple compute instances on the subnet. A load balancer may also be provided to load balance traffic among subnets within the VCN.

[0060] A particular compute instance deployed on VCN 104 can communicate with a variety of different endpoints. These endpoints may include endpoints hosted by CSPI 200 and endpoints external to CSPI 200. Endpoints hosted by CSPI 101 may include endpoints on the same subnet as the particular compute instance (e.g., communication between two compute instances in Subnet-1), endpoints in a different subnet but within the same VCN (e.g., communication between a compute instance in Subnet-1 and a compute instance in Subnet-2), endpoints in a different VCN in the same region (e.g., communication between a compute instance in Subnet-1 and an endpoint in a VCN in the same region 106 or 110, or communication between a compute instance in Subnet-1 and an endpoint in the service network 110 in the same region), or endpoints in a VCN in a different region (e.g., communication between a compute instance in Subnet-1 and an endpoint in a VCN in a different region 108). Additionally, compute instances in a subnet hosted by CSPI 101 can communicate with endpoints not hosted by CSPI 101 (i.e., external to CSPI 101). These external endpoints include endpoints within customer on-premise networks 116, endpoints within other remote cloud host networks 118, public endpoints 114 accessible via a public network such as the Internet, and other endpoints.

[0061] Communication between compute instances on the same subnet is facilitated using VNICs associated with the source and destination compute instances. For example, compute instance C1 in Subnet-1 may want to send a packet to compute instance C2 in Subnet-1. For a packet sent from a source compute instance whose destination is another compute instance in the same subnet, the packet is first processed by the VNIC associated with the source compute instance. The processing performed by the VNIC associated with the source compute instance may include determining the packet's destination information from the packet header, identifying any policies (e.g., security lists) configured for the VNIC associated with the source compute instance, determining the packet's next hop, performing any packet encapsulation / decapsulation functions as needed, and forwarding / routing the packet to the next hop to facilitate communication of the packet to its intended destination. If the destination compute instance is in the same subnet as the source compute instance, the VNIC associated with the source compute instance is configured to identify the VNIC associated with the destination compute instance and forward the packet to that VNIC for processing. The VNIC associated with the destination compute instance then executes and forwards the packet to the destination compute instance.

[0062] When communicating a packet from a compute instance in a subnet to an endpoint in a different subnet of the same VCN, the communication is facilitated by the VNICs associated with the source and destination compute instances and the VCN VRs. For example, if compute instance C1 in Subnet-1 in Figure 1 wants to send a packet to compute instance D1 in Subnet-2, the packet is first routed by the VNIC associated with compute instance C1. The VNIC associated with compute instance C1 is configured to route the packet to VCN VR105 using the VCN VR's default route or port 10.0.0.1. VCN VR105 is configured to route the packet to subnet-2 using port 10.1.0.1. The packet is then received and processed by the VNIC associated with D1, which forwards the packet to compute instance D1.

[0063] To communicate packets from a compute instance within VCN 104 to an endpoint outside VCN 104, the communication is facilitated by a VNIC associated with the source compute instance, VCN VR 105, and a gateway associated with VCN 104. One or more types of gateways can be associated with VCN 104. A gateway is an interface between a VCN and another endpoint, where the other endpoint is outside the VCN. A gateway is a Layer 3 / IP layer concept that enables a VCN to communicate with endpoints outside the VCN. Thus, a gateway facilitates traffic flow between a VCN and other VCNs or networks. A variety of different types of gateways can be configured in a VCN to facilitate different types of communications with different types of endpoints. Through gateways, communications may occur over a public network (e.g., the Internet) or a private network. These communications may use various communication protocols.

[0064] For example, compute instance C1 may wish to communicate with an endpoint outside VCN 104. The packet may first be processed by a VNIC associated with source compute instance C1. The VNIC processing determines that the packet's destination is outside of Cl's subnet-1. The VNIC associated with C1 may forward the packet to VCN VR105 of VCN 104. VCN VR105 then processes the packet and, as part of the processing, determines a particular gateway associated with VCN 104 as the packet's next hop based on the packet's destination. VCN VR105 may then forward the packet to the particular gateway. For example, if the destination is an endpoint within a customer's operating premises network, the packet may be forwarded by VCN VR105 to dynamic routing gateway (DRG) 122 configured for VCN 104. The packet may then be forwarded from the gateway to the next hop to facilitate communication of the packet to its intended final destination.

[0065] Various different types of gateways may be configured for a VCN. Examples of gateways that may be configured for a VCN are shown in FIG. 1 and described below. Examples of gateways associated with VCNs are also shown in FIGS. 12, 13, 14, and 15 (e.g., gateways indicated by reference numbers 1234, 1236, 1238, 1334, 1336, 1338, 1434, 1436, 1438, 1534, 1536, and 1538) and described below. As shown in the embodiment shown in FIG. 1, a dynamic routing gateway (DRG) 122 may be added to or associated with the customer VCN 104. The DRG 122 provides a path for private network traffic communication between the customer VCN 104 and another endpoint. The other endpoint may be a customer on-premises network 116, a VCN 108 in a different region of the CSPI 101, or another remote cloud network 118 not hosted by the CSPI 101. The customer on-premise network 116 may be a customer network or customer data center built using customer resources. Access to the customer on-premise network 116 is typically highly restricted. For customers with both a customer on-premise network 116 and one or more VCNs 104 deployed or hosted in the cloud by CSPI 101, the customer may want the on-premise network 116 and the cloud-based VCNs 104 to be able to communicate with each other. This allows Thus, customers can build extended hybrid environments that include the customer's VCN 104 hosted by CSPI 101 and their on-premise network 116. DRG 122 enables such communications. To enable such communications, a communication channel 124 is established, where one endpoint of the communication channel is located in the customer's on-premise network 116 and the other endpoint is located in CSPI 101 and connected to the customer's VCN 104. The communication channel 124 can traverse a public communication network, such as the Internet, or a private communication network. A variety of different communication protocols can be used, such as IPsec VPN technology over a public communication network, such as the Internet, or Oracle's FastConnect technology, which uses a private network instead of a public network. The device or equipment in the customer on-premise network 116 that forms one endpoint of the communication channel 124 is called customer premises equipment (CPE), such as the CPE 126 shown in FIG. 1. The endpoint on the CSPI 101 side may be a host machine running the DRG 122. In certain embodiments, a remote peering connection (RPC) can be added to the DRG, allowing a customer to peer one VCN with another VCN in another region. Using such an RPC, the customer VCN 104 can connect to a VCN 108 in another region using the DRG 122. The DRG 122 can also connect to other remote cloud networks 118 not hosted by the CSPI 101, such as Microsoft® Azure Cloud, Amazon® It may be used to communicate with the AWS Cloud.

[0066] As shown in Figure 1, an Internet Gateway (IGW) 120 can be configured in a customer VCN 104 to enable compute instances on the customer VCN 104 to communicate with public endpoints 114 accessible over a public network, such as the Internet. The IGW 120 is a gateway for connecting a VCN to a public network, such as the Internet. The IGW 120 enables public subnets in a VCN, such as VCN 104 (resources in the public subnet have public overlay IP addresses) to directly access public endpoints 112 on the public network 114, such as the Internet. The IGW 120 can be used to initiate connections from subnets in the VCN 104 or from the Internet.

[0067] Customer VCN 104 can be configured with a network address translation (NAT) gateway 128. NAT gateway 128 allows cloud resources in the customer VCN that do not have dedicated public overlay IP addresses to access the Internet without exposing them to direct incoming Internet connections (e.g., L4-L7 connections). This allows private subnets in a VCN, such as private subnet-1 in VCN 104, to privately access public endpoints on the Internet. With a NAT gateway, connections can be initiated from the private subnet to the public Internet, but connections cannot be initiated from the Internet to the private subnet.

[0068] In certain embodiments, customer VCN 104 can be configured with a service gateway (SGW) 126. SGW 126 provides a path for private network traffic between VCN 104 and service endpoints supported by service network 110. In certain embodiments, service network 110 may be provided by a CSP and may offer a variety of services. An example of such a service network is the Oracle® Service Network, which offers a variety of services available to customers. For example, compute instances (e.g., database systems) in a private subnet of customer VCN 104 may be connected to a public IP address ( Or back up data to a service endpoint (e.g., object store) without requiring access to the Internet. In some embodiments, a VCN can have only one SGW, and connections can be initiated only from subnets within the VCN, not from the service network 110. When a VCN is peered with another VCN, resources in the other VCN typically do not have access to the SGW. Resources in an on-premises network connected to a VCN with FastConnect or VPN Connect can also use a service gateway configured in that VCN.

[0069] In some implementations, the SGW 126 uses service classless inter-domain routing (CIDR) labels. A CIDR label is a string that represents all regional public IP address ranges for a service or group of services of interest. Customers use service CIDR labels to control traffic to services when configuring the SGW and associated routing rules. Customers can optionally use service CIDR labels when configuring security rules without having to adjust the security rules if the service's public IP addresses change in the future.

[0070] A local peering gateway (LPG) 132 is a gateway that can be added to a customer VCN 104 to enable the VCN 104 to peer with another VCN in the same region. Peering means that the VCNs communicate using private IP addresses without the traffic going over a public network such as the Internet or routing the traffic through the customer on-premises network 116. In a preferred embodiment, a VCN has a separate LPG for each peering it establishes. Local peering or VCN peering is a common practice used to establish network connectivity between different applications or infrastructure management functions.

[0071] A service provider, such as a provider of a service in service network 110, can provide access to a service using different access models. According to a public access model, the service may be exposed as a public endpoint publicly accessible by a compute instance in the customer VCN over a public network such as the Internet, or may be accessed privately through SGW 126. According to a specific private access model, the service may be accessed as a private IP endpoint in a private subnet in the customer VCN. This is called private endpoint (PE) access and allows a service provider to expose its service as an instance in the customer's private network. A private endpoint resource represents a service in a customer VCN. Each PE appears as a VNIC (called a PE-VNIC, which has one or more private IPs) that the customer selects from a subnet in the customer VCN. Thus, the PE provides a way to provide a service within the customer's private VCN subnet using a VNIC. Because the endpoint is exposed as a VNIC, the PE VNIC can utilize all the functionality associated with a VNIC, such as routing rules and security lists.

[0072] Service providers register services to make them accessible through PEs. Providers can associate policies with services that regulate the visibility of the service to customer tenants. Providers can register multiple services under a single virtual IP address (VIP), especially for multi-tenant services. There can also be multiple private endpoints (in multiple VCNs) that represent the same service.

[0073] Compute instances in the private subnet can then access the service using the private IP address or service DNS name of the PE VNIC. Compute instances in the customer VCN can access the service by sending traffic to the private IP address of the PE in the customer VCN. The Private Access Gateway (PAGW) 130 is a gateway resource that can connect to a service provider VCN (e.g., a VCN in the service network 110) and serves as the ingress / egress point for all traffic from / to the customer subnet private endpoints. The PAGW 130 allows providers to scale the number of PE connections without utilizing internal IP address resources. A provider only needs to configure one PAGW for any number of services registered in a single VCN. A provider can present services as private endpoints in multiple VCNs for one or more customers. From the customer's perspective, the PE VNIC appears not to be connected to the customer's instance but to the service the customer wants to interact with. Traffic destined for the private endpoint is routed to the service through the PAGW 130. These are called customer-to-service private connections (C2S connections).

[0074] Also, the PE concept is used to ensure that traffic is routed between the FastConnect / IPsec link and the customer VCN. Private access for services can also be extended to customer on-premises networks and data centers by allowing traffic to flow through private endpoints within LPG 132. Private access for services can also be extended to customer peering VCNs by allowing traffic to flow between PEs in LPG 132 and the customer VCN.

[0075] Customers can control VCN routing at the subnet level, allowing them to specify which subnets in a customer VCN, such as VCN 104, use each gateway. A VCN's route tables can be used to determine whether traffic can be routed outside the VCN through a particular gateway. For example, in a particular case, the route table for a public subnet in customer VCN 104 can send non-local traffic through IGW 120. The route table for a private subnet in the same customer VCN 104 can send traffic to CSP services through SGW 126. All remaining traffic may be sent through NAT gateway 128. Route tables only control traffic that leaves the VCN.

[0076] Security lists associated with a VCN are used to control traffic entering the VCN through inbound connections and gateways. All resources within a subnet use the same mute tables and security lists. Security lists may be used to control specific types of traffic entering and leaving instances within a VCN's subnets. Security list rules may include inbound (inbound) rules and outbound (outbound) rules. For example, inbound rules may specify allowed source address ranges, and outbound rules may specify allowed destination address ranges. Security rules may specify specific protocols (e.g., TCP, ICMP), specific ports (e.g., port 22 for SSH, port 3389 for Windows RDP), etc. In certain implementations, the instance's operating system may enforce its own firewall rules that match security list rules. Rules may be stateful (e.g., connections are tracked and responses are automatically allowed without explicit security list rules for the response traffic) or stateless.

[0077] Access from a customer VCN (i.e., resources or compute instances deployed on VCN 104) may be categorized as public access, private access, or dedicated access. Public access refers to an access model for accessing public endpoints using public IP addresses or NATs. Private access enables customer workloads in VCN 104 with private IP addresses (e.g., resources in a private subnet) to access a service without traversing a public network such as the Internet. In particular embodiments, CSPI 101 enables customer VCN workloads with private IP addresses to access the service's public service endpoint using a service gateway. Thus, the service gateway provides a private access model by establishing a virtual link between the customer VCN and the service's public endpoint, which resides outside the customer's private network.

[0078] Additionally, CSPI is working to develop dedicated public peering services using technologies such as FastConnect public peering. It can provide brick access, where a customer's on-premises instance can access the FastConnect connection without going through a public network such as the Internet. You can access one or more services in a customer VCN using FastConnect. CSPI also provides dedicated private access using FastConnect private peering. In this case, customer on-premises instances with private IP addresses can access workloads in the customer VCN using the FastConnect connection. FastConnect connects customers' on-premises networks using the public internet. FastConnect is a network connection used instead of connecting your network to CSPI and its services. FastConnect offers higher bandwidth options and It provides an easy, flexible and economical way to create dedicated, private connections with a reliable and consistent networking experience.

[0079] FIG. 1 and the accompanying description above illustrate various virtualized elements in an exemplary virtual network. As noted above, a virtual network is built on an underlying physical network or substrate network. FIG. 2 is a simplified architecture diagram illustrating physical elements within a physical network within CSPI 200 that provides the foundation for the virtual network, according to certain embodiments. As shown, CSPI 200 provides a distributed environment including elements and resources (e.g., compute, memory, and networking resources) provided by a cloud service provider (CSP). These elements and resources are used to provide cloud services (e.g., IaaS services) to subscribing customers, i.e., customers who subscribe to one or more services offered by the CSP. Based on the services to which the customer subscribes, CSPI 200 provides some resources (e.g., compute, memory, and networking resources) to the customer. The customer can then build their own cloud-based (i.e., CSPI-hosted), customizable private virtual network using the physical compute, memory, and networking resources provided by CSPI 200. As previously mentioned, these customer networks are referred to as virtual cloud networks (VCNs). Customers can deploy one or more customer resources, such as compute instances, into these Customer VCNs. The compute instances may be virtual machines, bare metal instances, etc. CSPI200 provides infrastructure and a set of complementary cloud services that enable customers to build and run a wide range of applications and services in a highly available hosted environment.

[0080] In the exemplary embodiment shown in FIG. 2, the physical elements of CSPI 200 include one or more physical The VCN includes host machines or physical servers (e.g., 202, 206, 208), network virtualization devices (NVDs) (e.g., 210, 212), top-of-rack (TOR) switches (e.g., 214, 216), physical networks (e.g., 218), and switches within the physical network 218. The physical host machines or servers can host and execute various compute instances participating in one or more subnets of the VCN. The compute instances may include virtual machine instances and bare metal instances. For example, the various compute instances shown in FIG. 1 may be hosted by the physical host machines shown in FIG. 2. The virtual machine compute instances in the VCN may be executed by one host machine or by multiple different host machines. Also, the physical host machines can host virtual host machines, container-based hosts or functions, etc. The VIC and VCN VRs shown in FIG. 1 may be executed by the FTVDs shown in FIG. 2. The gateways shown in FIG. 1 may be executed by the host machines and / or NVDs shown in FIG. 2.

[0081] A host machine or server may run a hypervisor (also called a virtual machine monitor or VMM) that creates and enables a virtualized environment on the host machine. Virtualization or a virtualized environment facilitates cloud-based computing. One or more computing instances may be created, executed, and managed on the host machine by the hypervisor on the host machine. The hypervisor on the host machine enables the host machine's physical computing resources (e.g., computing resources, memory resources, and networking resources) to be shared among various computing instances running on the host machine.

[0082] For example, as shown in FIG. 2, host machines 202 and 208 execute hypervisors 260 and 266, respectively. These hypervisors may be implemented using software, firmware, hardware, or a combination thereof. Typically, a hypervisor is a process or software layer that resides in a host machine's operating system (OS), which executes on the host machine's hardware processor. The hypervisor provides a virtualized environment that allows the host machine's physical computing resources (e.g., processing resources such as processors / cores, memory resources, and networking resources) to be shared among various virtual machine computing instances executed by the host machine. For example, in FIG. 2, hypervisor 260 resides in the OS of host machine 202 and allows the host machine's computing resources (e.g., processing resources, memory resources, and networking resources) to be shared among computing instances (e.g., virtual machines) executed by host machine 202. A virtual machine can have its own OS (called a guest OS). This guest OS may be the same as or different from the host machine's OS. The OS of a virtual machine executed by a host machine may be the same as or different from the OS of other virtual machines executed by the same host machine. Thus, the hypervisor can run multiple OSs in parallel while sharing the same computing resources of the host machine. The host machines shown in Figure 2 may have the same type of hypervisor or different types of hypervisors.

[0083] A compute instance may be a virtual machine instance or a bare metal instance. In Figure 2, compute instance 268 on host machine 202 and compute instance 274 on host machine 208 are examples of virtual machine instances. Host machine 206 is an example of a bare metal instance provided to a customer.

[0084] In certain instances, an entire host machine may be provided to a single customer, with one or more compute instances (virtual machines or bare metal) hosted by that host machine. In some embodiments, bare metal compute instances (including any of the virtual machine compute instances) may all belong to the same customer. In other examples, a host machine may be shared among multiple customers (i.e., multiple tenants). In such a multi-tenant scenario, a host machine can host virtual machine compute instances belonging to different customers. These compute instances may be members of different VCNs for different customers. In certain embodiments, bare metal compute instances are hosted by bare metal servers that do not have a hypervisor. When bare metal compute instances are provided, a single customer or tenant maintains control of the physical CPU, memory, and network interfaces of the host machine that hosts the bare metal instance, and the host machine is not shared with other customers or tenants.

[0085] As previously described, each compute instance that is part of a VCN is associated with a VNIC that enables the compute instance to be a member of a subnet of the VCN. The VNIC associated with a compute instance facilitates communication of packets or frames to and from the compute instance. A VNIC is associated with the compute instance when the compute instance is created. In particular embodiments, for a compute instance executed by a host machine, the VNIC associated with the compute instance is executed by an NVD connected to the host machine. For example, in FIG. 2, host machine 202 executes virtual machine compute instance 268 associated with VNIC 276, which is executed by NVD 210 connected to host machine 202. As another example, bare metal instance 272 hosted by host machine 206 is associated with VNIC 280, which is executed by NVD 212 connected to host machine 206. As yet another example, VNIC 284 is associated with compute instance 274 executed by host machine 208, which is executed by NVD 212 connected to host machine 208.

[0086] For a compute instance hosted by a host machine, the NVD connected to that host machine executes a VCN VR corresponding to the VCN of which the compute instance is a member. For example, in the embodiment shown in Figure 2, NVD 210 executes VCN VR 277 corresponding to the VCN of which compute instance 268 is a member. NVD 212 may also execute one or more VCN VRs 283 corresponding to the VCNs corresponding to the compute instances hosted by host machines 206 and 208.

[0087] A host machine may include one or more network interface cards (NICs) for connecting the host machine to other devices. The NICs on a host machine may provide one or more ports (or interfaces) for communicatively connecting the host machine to another device. For example, one or more ports (or interfaces) on the host machine and the NVD may be used to connect the host machine to the NVD. The host machine may also be connected to other devices, such as other host machines.

[0088] 2, host machine 202 is connected to NVD 210 using link 220 extending between port 234 provided by NIC 232 of host machine 202 and port 236 of NVD 210. Host machine 206 is connected to NVD 212 using link 224 extending between port 246 provided by NIC 244 of host machine 206 and port 248 of NVD 212. Host machine 208 is connected to NVD 212 using link 226 extending between port 252 provided by NIC 250 of host machine 208 and port 254 of NVD 212.

[0089] Similarly, the NVDs are connected via communication links to top-of-rack (TOR) switches, which are connected to a physical network 218 (also called a switch fabric). In particular embodiments, the links between the host machines and the NVDs and the links between the NVDs and the TOR switches are Ethernet links. For example, in FIG. 2, NVDs 210 and 212 are connected to TOR switches 214 and 216, respectively, via links 228 and 230. In particular embodiments, links 220, 224, 226, 228, and 230 are Ethernet links. The collection of host machines and NVDs connected to a TOR may be referred to as a rack.

[0090] The physical network 218 provides a communications fabric that enables the TOR switches to communicate with each other. The physical network 218 may be a multi-tier network. In a particular implementation, the physical network 218 is a multi-tier Clos network of switches, with the TOR switches 214 and 216 representing leaf-level nodes of the multi-tier and multi-node physical switching network 218. Different Clos network configurations are possible, including, but not limited to, 2-tier networks, 3-tier networks, 4-tier networks, 5-tier networks, and generally "n"-tier networks. An example of a Clos network is shown in FIG. 5 and described below.

[0091] A variety of different connection configurations are possible between a host machine and the N virtual disks, including one-to-one, many-to-one, and one-to-many configurations. In a one-to-one implementation, each host machine is connected to its own separate virtual disk. For example, in FIG. 2, host machine 202 is connected to virtual disk 210 via NIC 232 of host machine 202. In a many-to-one configuration, multiple host machines are connected to a single virtual disk. For example, in FIG. 2, host machines 206 and 208 are connected to the same virtual disk 212 via NICs 244 and 250, respectively.

[0092] In a one-to-many configuration, one host machine is connected to multiple NVDs. Figure 3 shows an example of a CSPI 300 in which a host machine is connected to multiple NVDs. As shown in Figure 3, a host machine 302 includes a network interface card (NIC) 304 including multiple ports 306 and 30S. The host machine 300 is connected to a first NVD 310 via port 306 and link 320, and to a second NVD 312 via port 308 and link 322. Ports 306 and 308 may be Ethernet ports, and links 320 and 322 between the host machine 302 and the NVDs 310 and 312 may be Ethernet links. The NVD 310 is connected to a first TOR switch 314, and the NVD 312 is connected to a second TOR switch 316. The links between the NVDs 310 and 312 and the TOR switches 314 and 316 may be Ethernet links. TOR switches 314 and 316 represent Tier-0 switching devices within a multi-tier physical network 318 .

[0093] 3 provides two separate physical network paths from the physical switch network 318 to the host machine 302: a first path from the TOR switch 314 to the host machine 302 via the NVD 310, and a second path from the TOR switch 316 to the host machine 302 via the NVD 312. The separate paths provide enhanced availability (referred to as high availability) for the host machine 302. If there is a problem with one of the paths (e.g., a link on one of the paths fails) or if there is a problem with a device (e.g., a particular NVD is not functioning), the other path can be used for communications to and from the host machine 302.

[0094] In the configuration shown in Figure 3, the host machine uses the In another embodiment, the host machine may include multiple NICs, allowing the host machine to connect to multiple NVDs.

[0095] Referring again to Figure 2, an NVD is a physical device or element that performs one or more network virtualization and / or storage virtualization functions. An NVD may be any device that has one or more processing units (e.g., a CPU, a network processing unit (NPU), an FPGA, a packet processing pipeline), memory including cache, and ports. Various virtualization functions may be performed by software / firmware executed by one or more processing units of the NVD.

[0096] The NVD may be implemented in a variety of different ways. For example, in a particular embodiment, the NVD is implemented as an interface card with an embedded processor, called a smart NIC or intelligent NIC. The smart NIC is a separate device from the NIC on the host machine. In Figure 2, the NVD 210 may be implemented as a smart NIC connected to the host machine 202, and the NVD 212 may be implemented as a smart NIC connected to the host machines 206 and 208.

[0097] However, a smart NIC is only one example of an NVD implementation. Various other implementations are possible. For example, in some other implementations, the NVD or one or more functions performed by the NVD may be incorporated into or performed by one or more host machines, one or more TOR switches, and other elements of CSPI 200. For example, the NVD may be integrated into a host machine. In this case, the functions performed by the NVD are performed by the host machine. As another example, the NVD may be part of a TOR switch, or a TOR switch may be configured to perform the functions performed by the NVD that enable the TOR switch to perform various complex packet transformations used in public clouds. A TOR that performs the functions of an NVD may be referred to as a smart TOR. In yet other implementations that provide customers with virtual machine (VM) instances rather than bare metal (BM) instances, the functions provided by the NVD may be implemented inside the hypervisor of the host machine. In some other implementations, some of the NVD's functions may be offloaded to a centralized service running on a set of host machines.

[0098] In certain embodiments, such as when implemented as a smart NIC, as shown in FIG. 2, an NVD may include multiple physical ports that allow the NVD to connect to one or more host machines and one or more TOR switches. Ports on an NVD can be categorized as host-facing ports (also called "south ports") or network-facing or TOR-facing ports (also called "north ports"). A host-facing port of an NVD is a port used to connect the NVD to a host machine. Examples of host-facing ports in FIG. 2 include port 236 of NVD 210 and ports 248 and 254 of NVD 212. A network-facing port of an NVD is a port used to connect the NVD to a TOR switch. Examples of network-facing ports in FIG. 2 include port 256 of NVD 210 and port 258 of NVD 212. As shown in FIG. 2, the NVD 210 is connected to the TOR switch 214 via link 228 extending from port 256 of NVD 210 to the TOR switch 214. Similarly, the NVD 212 is connected to the TOR switch 216 via a link 230 that extends from a port 258 of the NVD 212 to the TOR switch 216 .

[0099] The NVD receives packets and frames (e.g., packets generated by compute instances hosted by the host machine) from the host machine via a host-facing port. The NVD can receive packets and frames from the TOR switch via its network-facing port, perform any necessary packet processing, and then forward the packets and frames to the TOR switch via its network-facing port. The NVD can receive packets and frames from the TOR switch via its network-facing port, perform any necessary packet processing, and then forward the packets and frames to the host machine via its host-facing port.

[0100] In certain embodiments, multiple ports and associated links may be provided between the NVD and the TOR switch. These ports and links can be aggregated to form a link aggregator group (LAG) of multiple ports or links. Link aggregation allows multiple physical links between two endpoints (e.g., between the NVD and the TOR switch) to be treated as a single logical link. All physical links within a given LAG can operate at the same speed in full-duplex mode. LAGs help increase the bandwidth and reliability of the connection between two endpoints. If one of the physical links in a LAG fails, traffic is dynamically and transparently reassigned to another physical link within the LAG. The aggregated physical link provides higher bandwidth than individual links. Multiple ports associated with a LAG are treated as a single logical port. Traffic can be load-balanced across the multiple physical links in the LAG. One or more LAGs can be configured between two endpoints. The two endpoints may be, for example, between the NVD and the TOR switch, or between a host machine and the NVD.

[0101] The NVD implements or performs network virtualization functions. These functions are performed by software / firmware executed by the NVD. Examples of network virtualization functions include, but are not limited to, packet encapsulation and decapsulation functions, functions for creating VCN networks, functions for implementing network policies such as VCN security list (firewall) functions, functions for facilitating routing and forwarding of packets to and from compute instances within a VCN, etc. In particular embodiments, upon receiving a packet, the NVD is configured to execute a packet processing pipeline that processes the packet and determines how to forward or route the packet. As part of this packet processing pipeline, the NVD provides execution of one or more virtual functions associated with the overlay network, such as execution of VNICs associated with compute instances in the VCN, execution of virtual routers (VRs) associated with the VCN, encapsulation and decapsulation of packets to facilitate forwarding or routing within the virtual network, execution of specific gateways (e.g., local peering gateways), implementation of security lists, network security groups, network address translation (NAT) functions (e.g., public IP to private IP translation per host), throttling functions, and other functions.

[0102] In some embodiments, the packet processing data path within the NVD may include multiple packet pipelines. Each packet pipeline consists of a series of packet transformation stages. In some implementations, upon receiving a packet, the packet is parsed and sorted into a single pipeline. The packet is then processed stage by stage in a linear fashion until it is discarded or sent out through an interface of the NVD. These stages provide packet processing building blocks of basic functionality (e.g., validating headers, performing throttling, inserting new Layer 2 headers, performing L4 firewalling, VCN encapsulation / decapsulation), such that new pipelines can be constructed by assembling existing stages, and new functionality can be added by creating and inserting new stages into existing pipelines.

[0103] The NVD provides control plane functions and data plane functions corresponding to the VCN control plane and data plane. The VCN can perform both control and data plane functions. Examples of the VCN control plane are shown in Figures 12, 13, 14, and 15 (see reference numbers 1216, 1316, 1416, and 1516) and described below. Examples of the VCN data plane are shown in Figures 12, 13, 14, and 15 (see reference numbers 1218, 1318, 1418, and 1518) and described below. The control plane functions include functions used to configure the network to control how data is forwarded (e.g., setting routes and route tables, configuring VNICs). In certain embodiments, a VCN control plane is provided that centrally computes and exposes all overlay-to-substrate mappings to the NVD and virtual network edge devices (e.g., various gateways such as DRGs, SGWs, and IGWs). Firewall rules can also be exposed using the same mechanism. In certain embodiments, the NVD retrieves only the mappings that are relevant to that NVD. The data plane functions include functions that perform the actual routing / forwarding of packets based on the configurations set using the control plane. The VCN data plane is implemented by encapsulating customer network packets before they traverse the backbone network. The encapsulation / decapsulation functionality is implemented in the NVD. In certain embodiments, the NVD is configured to intercept all network packets entering and leaving the host machine and perform network virtualization functions.

[0104] As described above, the NVD performs various virtualization functions, including VNICs and VCN VRs. The NVD can execute VNICs associated with compute instances hosted by one or more host machines connected to the VNICs. For example, as shown in FIG. 2, NVD 210 executes the functions of VNIC 276 associated with compute instance 268 hosted by host machine 202 connected to NVD 210. As another example, NVD 212 executes VNIC 280 associated with bare metal compute instance 272 hosted by host machine 206 and VNIC 284 associated with compute instance 274 hosted by host machine 208. The host machines can host compute instances that belong to different VCNs that belong to different customers. The NVDs connected to the host machines can execute VNICs (i.e., execute functions associated with the VNICs) corresponding to the compute instances.

[0105] NVDs also execute VCN virtual routers corresponding to the VCNs of the compute instances. For example, in the embodiment shown in FIG. 2, NVD 210 executes VCN VR 277 corresponding to the VCN to which compute instance 268 belongs. NVD 212 executes one or more VCN VRs 283 corresponding to one or more VCNs to which compute instances hosted on host machines 206 and 208 belong. In particular embodiments, a VCN VR corresponding to a VCN is executed by all NVDs connected to a host machine that hosts at least one compute instance belonging to that VCN. If a host machine hosts compute instances that belong to different VCNs, the NVDs connected to that host machine may execute VCN VRs corresponding to the different VCNs.

[0106] In addition to VNICs and VCN VRs, the NVD may include one or more hardware elements that run various software (e.g., daemons) and facilitate various network virtualization functions performed by the NVD. For simplicity, these various elements are grouped as "packet processing elements" shown in FIG. 2. For example, NVD 210 includes packet processing element 286, and NVD 212 includes packet processing element 288. For example, the packet processing element of the NVD may include a packet processor configured to monitor all packets received and communicated using the NVD by interacting with the NVD's ports and hardware working interfaces, and to store network information. The network information may include, for example, network flow information for identifying different network flows processed by the NVD and information for each flow (e.g., the timestamps for each flow). In certain embodiments, network flow information may be stored on a per-VNIC basis. As another example, the packet processing element may include a replication agent configured to replicate information stored by the NVD to one or more different replication target stores. As yet another example, The packet processing element may include a logging agent configured to perform logging functions for the NVD. The packet processing element may also monitor the performance and The NVD may include software for monitoring the status and health of the NVD and, in some cases, for monitoring the status and health of other elements connected to the NVD.

[0107] FIG. 1 illustrates elements of an exemplary virtual or overlay network, including a VCN, subnets within the VCN, compute instances deployed on the subnets, VNICs associated with the compute instances, VRs for the VCN, and a set of gateways configured for the VCN. The overlay elements illustrated in FIG. 1 may be executed or hosted by one or more of the physical elements illustrated in FIG. 2. For example, compute instances within a VCN may be executed or hosted by one or more host machines illustrated in FIG. 2. For compute instances hosted by a host machine, the VNICs associated with the compute instance are typically executed by an NVD connected to the host machine (i.e., the VNIC functionality is provided by an NVD connected to the host machine). The VCN VR functionality is performed by all NVDs connected to the host machines that host or execute compute instances that are part of the VCN. Gateways associated with a VCN may be executed by one or more different types of NVDs. For example, some gateways may be executed by smart NICs, and other gateways may be executed by one or more host machines or other implementations of NVDs.

[0108] As described above, compute instances within a customer VCN can communicate with a variety of different endpoints. These endpoints may be in the same subnet as the source compute instance, in a different subnet but in the same VCN as the source compute instance, or may include endpoints outside the VCN of the source compute instance. These communications are facilitated using the VNICs associated with the compute instances, the VCN VRs, and the gateways associated with the VCN.

[0109] Communication between two compute instances on the same subnet within a VCN is facilitated using VNICs associated with the source and destination compute instances. The source and destination compute instances may be hosted by the same host machine or different host machines. A packet originating from a source compute instance may be forwarded from the host machine hosting the source compute instance to an NVD connected to that host machine. In the NVD, the packet is processed using a packet processing pipeline, which may include the execution of a VNIC associated with the source compute instance. Because the packet's destination endpoint is in the same subnet, the execution of a VNIC associated with the source compute instance forwards the packet to an NVD running a VNIC associated with the destination compute instance, which processes the packet and forwards it to the destination compute instance. The VNICs associated with the source and destination compute instances may run on the same NVD (e.g., if both the source and destination compute instances are hosted by the same host machine) or on different NVDs (e.g., if the source and destination compute instances are hosted by different host machines connected to different NVDs). The VNIC can use the routing / forwarding tables stored by the NVD to determine the next hop for a packet.

[0110] When communicating a packet from a compute instance in a subnet to an endpoint in a different subnet within the same VCN, the packet originating from the source compute instance is communicated from the host machine hosting the source compute instance to the NVD connected to that host machine. In the NVD, the packet is processed using a packet processing pipeline that may include running one or more VNICs and VRs associated with the VCN. For example, the NVD executes or invokes a function corresponding to a VNIC associated with the source compute instance (also referred to as executing a VNIC) as part of the packet processing pipeline. The function executed by the VNIC may include searching for a VLAN tag on the packet. Because the packet's destination is outside the subnet, a VCN VR function is invoked and executed by the NVD. The VCN VR then routes the packet to the NVD executing the VNIC associated with the destination compute instance. The VNIC associated with the destination compute instance then processes the packet and forwards the packet to the destination compute instance. The VNICs associated with the source compute instance and the destination compute instance may run on the same NVD (e.g., if both the source compute instance and the destination compute instance are hosted by the same host machine) or may run on different NVDs (e.g., if the source compute instance and the destination compute instance are hosted by different host machines connected to different NVDs).

[0111] If the packet's destination is outside the VCN of the source compute instance, the packet originating from the source compute instance is communicated from the host machine hosting the source compute instance to the NVD connected to that host machine. The NVD runs the VNIC associated with the source compute instance. Because the packet's destination endpoint is outside the VCN, the packet is processed by the VCN VR for that VCN. The NVD invokes a VCN VR function, which may result in the packet being forwarded to an NVD running the appropriate gateway associated with the VCN. For example, if the destination is an endpoint in a customer's on-premises network, the packet may be forwarded by the VCN VR to an NVD running a DRG gateway configured for the VCN. The VCN VR may run on the same NVD as the NVD running the VNIC associated with the source compute instance, or it may be run by a different NVD. The gateway may be run by a smart NIC, a host machine, or another NVD implementation. The packet is then processed by the gateway and forwarded to the next hop that facilitates communication of the packet to the intended destination endpoint. 2, a packet originating from compute instance 268 may be communicated from host machine 202 to NVD 210 via link 220 (using NIC 232). VNIC 276 on NVD 210 is called out because it is the VNIC associated with source compute instance 268. VNIC 276 is configured to inspect encapsulation information in the packet, determine a next hop for forwarding the packet to facilitate communication of the packet to its intended destination endpoint, and forward the packet to the determined next hop.

[0112] Compute instances deployed on a VCN can communicate with a variety of different endpoints. These endpoints may include endpoints hosted by CSPI 200 and endpoints external to CSPI 200. Endpoints hosted by CSPI 200 may include instances in the same VCN or other VCNs (which may be customer VCNs or VCNs not belonging to the customer). Communication between endpoints hosted by CSPI 200 may be performed over physical network 218. Compute instances can also communicate with endpoints not hosted by or external to CSPI 200. Examples of these endpoints include endpoints within a customer on-premises network or data center, or public endpoints accessible over a public network such as the Internet. Communication with endpoints external to CSPI 200 may be performed over a public network (e.g., the Internet) (not shown in FIG. 2) or a private network (not shown in FIG. 2) using a variety of communication protocols.

[0113] The architecture of CSPI 200 shown in FIG. 2 is merely an example and is not intended to be limiting. Variations, substitutions, and modifications are possible in alternative embodiments. For example, in some implementations, CSPI 200 may have more or fewer systems or elements than those shown in FIG. 2, may combine two or more systems, or may have a different system configuration or arrangement. The systems, subsystems, and other elements shown in FIG. 2 may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective systems, hardware, or a combination thereof. Software may be stored in a non-transitory storage medium (e.g., a memory device).

[0114] FIG. 4 illustrates connections between host machines and an NVD to provide I / O virtualization to support multi-tenancy, according to certain embodiments. As shown in FIG. 4, a host machine 402 runs a hypervisor 404 that provides a virtualized environment. The host machine 402 runs two virtual machine instances: VM1 406, which belongs to customer / tenant #1, and VM2 408, which belongs to customer / tenant #2. The host machine 402 includes a physical NIC 410 connected to an NVD 412 via link 414. Each of the compute instances is connected to a VNIC run by the NVD 412. In the embodiment of FIG. 4, VM1 406 is connected to VNIC-VM1 420, and VM2 408 is connected to VNIC-VM2 422.

[0115] 4, NIC 410 includes two logical NICs: logical NIC A 416 and logical NIC B 418. Each virtual machine is connected to and configured to operate with its own logical NIC. For example, VM1 406 is connected to logical NIC A 416, and VM2 408 is connected to logical NIC B 418. The logical NICs allow each tenant's virtual machine to believe it owns its own host machine and NIC, even though host machine 402 consists of only one physical NIC 410 shared by multiple tenants.

[0116] In particular embodiments, each logical NIC is assigned its own VLAN ID. Thus, logical NIC A 416 for tenant #1 is assigned a particular VLAN ID, and logical NIC B 418 for tenant #2 is assigned a different VLAN ID. When a packet is communicated from VM1 406, the hypervisor attaches a tag assigned to tenant #1 to the packet before communicating the packet from host machine 402 to NVD 412 over link 414. Similarly, when a packet is communicated from VM2 408, the hypervisor attaches a tag assigned to tenant #2 to the packet before communicating the packet from host machine 402 to NVD 412 over link 414. Thus, a packet 424 communicated from host machine 402 to NVD 412 has an associated tag 426 that identifies the particular tenant and associated VM. When a packet 424 is received on the NVD from host machine 402, the tag 426 associated with the packet is used to determine whether the packet should be processed by VNIC-VM1 420 or VNIC-VM2 422. The packet is then processed by the corresponding VNIC. The configuration shown in Figure 4 allows each tenant's compute instance to believe that it owns its own host machine and NIC. The configuration shown in Figure 4 provides I / O virtualization to support multi-tenancy.

[0117] FIG. 5 is a schematic block diagram illustrating a physical network 500 according to a particular embodiment. The embodiment illustrated in FIG. 5 is constructed as a Clos network. A Clos network is a particular type of network topology designed to provide connection redundancy while maintaining high bisection bandwidth and maximum resource utilization. A Clos network is a type of non-blocking, multi-stage or multi-layer switching network, and the number of stages or layers may be 2, 3, 4, 5, etc. The embodiment illustrated in FIG. 5 is a three-layer network, including layers 1, 2, and 3. TOR switch 504 represents a layer-0 switch in the Clos network. One or more NVDs are connected to the TOR switch. The layer-0 switch is also referred to as an edge device of the physical network. The layer-0 switch is connected to a layer-1 switch, also referred to as a leaf switch. In the embodiment illustrated in FIG. 5, “n” layer-0 TOR switches are connected to “n” layer-1 switches to form a pod. Each layer-0 switch in a pod is interconnected to all layer-1 switches in the pod, but switches between pods are not connected. In a specific implementation, the two pods are referred to as blocks. Each block is served by or connected to "n" layer-2 switches (also referred to as spine switches). A physical network topology may include multiple blocks. Similarly, the layer-2 switches are connected to "n" layer-3 switches (also referred to as super-spine switches). Communication of packets through the physical network 500 is typically performed using one or more layer-3 communication protocols. Typically, all layers of the physical network, except for the TOR layer, are n-way redundant, thus achieving high availability. The physical network can be scaled by specifying policies on pods and blocks to control the mutual visibility of switches in the physical network.

[0118] A characteristic of Clos networks is that the maximum hop count from one tier-0 switch to another tier-0 switch (or from an NVD connected to a tier-0 switch to another NVD connected to a tier-0 switch) is constant. For example, in a three-tier Clos network, a packet requires a maximum of seven hops to travel from one NVD to another. In this case, the source and target NVDs are connected to the leaf layers of the Clos network. Similarly, in a four-tier Clos network, a packet requires a maximum of nine hops to travel from one NVD to another. In this case, the source and target NVDs are connected to the leaf layers of the Clos network. Therefore, the Clos network architecture maintains constant overall network latency, which is important for intra- and inter-datacenter communications. Clos topologies are horizontally scalable and cost-effective. The network bandwidth / throughput capacity can be easily increased by adding more switches (e.g., more leaf switches and spine switches) at each tier and by increasing the number of links between switches in adjacent tiers.

[0119] In certain embodiments, each resource in CSPI is assigned a unique identifier called a Cloud Identifier (CID). This identifier is included as part of the resource's information. This identifier can be used to manage the resource, for example, through a console or API. An exemplary syntax for a CID is as follows:

[0120] ocid1.<RESOURCE TYPE> . <realm>.[REGION].[FUTURE USE].<UNIQUE ID> is. During the ceremony, "ocid1" is a string that indicates the version of the CID.

[0121] "RESOURCE TYPE" is the type of resource (e.g., instance, volume, VCN) , subnet, user, group).

[0122] "REALM" represents the region where the resource resides. An example value is "c1" "c1" represents the government cloud region, "c2" represents the government cloud region, or "c3" represents the federal government cloud region. Each region can have its own domain name.

[0123] "REGION" represents the region that the resource belongs to. If a region does not apply to the resource, this part may be blank.

[0124] "FUTURE USE" indicates that the item is reserved for future use. "UNIQUE ID" is the unique ID part. This format is used for resources or servers. This may vary depending on the type of screw.

[0125] FIG. 6 is a schematic block diagram illustrating a distributed environment 600 incorporating an exemplary embodiment for replicating communication channel-related state information, in accordance with certain embodiments.

[0126] Distributed environment 600 may include multiple systems communicatively connected to each other via communication channel 616. For example, the embodiment of FIG. 6 illustrates various systems and networks, including a customer on-premises network 602 and a cloud service provider infrastructure (CSPI) 660. These systems may be communicatively connected to each other via communication channel 616. The distributed environment 600 illustrated in FIG. 6 is merely an example and is not intended to unduly limit the scope of the claimed embodiments. Many variations, alternatives, and modifications are possible. For example, in some implementations, distributed environment 600 may have more or fewer systems or components than those illustrated in FIG. 6, a combination of two or more systems, or systems in a different configuration or arrangement. The systems, subsystems, and other components illustrated in FIG. 6 may be implemented using software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective systems, hardware, or a combination thereof. The software may be stored on a non-transitory storage medium (e.g., a memory device).

[0127] 6, customer on-premises network 602 may include customer premises equipment (CPE) 608 and configuration information 607. Customer on-premises network 602 is a network of one or more customer resources, such as one or more customer data centers. Access to customer on-premises network 602 is typically highly restricted.

[0128] The device or equipment within the customer on-premises network 602 that forms one endpoint of the communication channel 616 is referred to as customer premises equipment (CPE), such as the CPE 608 shown in FIG. 6. The CPE 608 must be properly configured to establish the communication channel 616 using configuration information 607 and to enable communication using the communication channel 616 between the customer on-premises network 602 and a host machine or endpoint 620 within the CSPI 660. The specific configuration information applied to the CPE 608 also depends on various other factors, such as the particular CPE 608 device being used, the characteristics of the communication channel 616 being established (e.g., the type of channel, the protocol used for communication, etc.), the CSPI 660 equipment, the primary host machine 620 that forms the other endpoint of the communication channel, and other criteria. The CPEs may be from different vendors, have different CPE platforms, and have different versions.

[0129] In an exemplary embodiment, as shown in FIG. 6 , a cloud service provider infrastructure (CSPI) 660 may include a virtual cloud network (VCN) 640, a primary host machine 620, and a backup host machine 630. The cloud service provider infrastructure (CSPI) 660 may be provided by a cloud service provider (CSP) and used to provide one or more cloud services. A customer may subscribe to one or more of the cloud services provided by the CSP using CSPI 660. A customer is any entity that subscribes to cloud services provided by a CSP. A customer may subscribe to one or more cloud services, including different types of services, e.g., SaaS, PaaS, IaaS, and other types of cloud services. When a customer subscribes or registers for a service provided by a CSP, a tenancy or account is created for the customer. The customer can access and use this account to utilize cloud resources associated with the account.

[0130] In certain embodiments, a CSP may offer services based on an IaaS model. In this case, the CSP provides infrastructure, such as CSPI 660, over which customers can build their own networks and deploy customer resources. In such embodiments, CSPI 660 may include interconnected high-performance computing resources, including various host machines (primary host machine 620 and backup host machine 630), memory resources, and network resources that form a physical network called a substrate network or underlay network. CSPI 660 may include one or more computing systems. Resources within CSPI 660 may be distributed across one or more data centers, which may be geographically distributed across one or more regions.

[0131] A physical network provides the basis for creating one or more virtual or overlay networks on top of the physical network. These virtual or overlay networks (also known as software-based or software-defined networks) are implemented using software virtualization technology, creating a network abstraction layer that can run on top of the physical network. Overlay networks can take various forms. Overlay networks typically use Layer 3 IP addressing, which specifies endpoints with virtual IP addresses. This overlay networking method is often referred to as virtual Layer 3 networking. When a customer subscribes to or registers for an IaaS service offered by a CSP, the tenancy created for that customer becomes an isolated, secure partition in CSPI 660 in which the customer can create, organize, and manage their cloud resources.

[0132] Customers can build networks using resources provided by CSPI 660. These networks can contain one or more customer resources, such as compute instances. For example, customers can build one or more customizable private networks called virtual cloud networks (VCNs) using resources provided by CSPI 660. For example, as shown in Figure 6, a customer configures Customer VCN 640 using resources provided by CSPI 660. A customer can deploy one or more customer resources, such as compute instances, on the Customer VCN. The compute instances may be virtual machines, containers, bare metal instances, etc. Thus, CSPI 660 provides infrastructure and a set of complementary cloud services that enable customers to build and run various applications and services in a highly available host environment. While customers do not manage or control the underlying physical resources provided by CSPI 660, Control over the operating system, storage, and deployed applications, and possibly limited control over some networking elements (e.g., firewalls).

[0133] Typically, when a customer has both a customer on-premises network 602 and one or more VCNs deployed or hosted in the cloud by a CSPI 660, the customer desires that the on-premises network 602 and the cloud-based VCN network, e.g., VCN 640, be able to communicate with each other. To enable communication between the customer on-premises network 602 and the provider VCN 640, a communication channel 616 is established. In this case, one endpoint of the communication channel is located in the customer on-premises network 602, e.g., the CPE 108, and the other endpoint of the communication channel is located in the VCN 640, e.g., the primary host machine 620. The communication channel 616 can traverse one or more communication networks 614, which may include a public network such as the Internet. A variety of different communication protocols can be used to facilitate communication between the on-premises network 602 and the VCN 640 using the communication channel 616, e.g., an MPLS circuit, a virtual private network (VPN) (e.g., an Internet Protocol Security (IPSec) VPN), etc.

[0134] The two endpoints, i.e., the CPE 608 and the primary host machine 620, can establish a communication channel by negotiating with each other. Specifically, a series of encryption and authentication messages can be sent and received between the two endpoints 608 and 620 to negotiate an agreement involving encryption and authentication-related parameters to be used for communication. Through negotiation, a security association (SA) between the two endpoints specifies the security characteristics agreed upon and approved by the communicating endpoints 608 and 620. A single SA can protect data in one direction. The SA provides both endpoints with details on how to encrypt and decrypt data.

[0135] The particular configuration information 607 applied to a CPE 608 depends on various other factors, such as the particular CPE 608 device being used, the characteristics of the communication channel being established (e.g., type of channel, protocol used for communication, etc.), the CSPI equipment (primary host machine 620) that forms the other endpoint of the communication channel, and other criteria.

[0136] An Internet Key Exchange (IKE) daemon 650 within CSPI 660 manages cryptographic keys (e.g., encryption and decryption keys) for communication channels. The IKE daemon 650 can provide authenticated keying material for security associations (SAs) in a secure manner by negotiating with endpoints within customer on-premises network 602.

[0137] As shown, the distributed environment 600 may include a backup host machine 630 connected to a primary host machine 620 in a VCN 640. Both host machines are hosted in the VCN 640 by a CSPI 660. In some embodiments, the primary host machine 620, the backup host machine 630, and any additional backup host machines may form a replication chain (RC) 650. The replication chain may include a set of host machines for managing and storing state information for the communication channel 616 via a log-based storage system. Information identifying the replication chain and backup host machine assigned to a particular host machine 620 may be stored in a disk backup data storage device. The host machines in the replication chain are arranged linearly to form a chain. The primary host machine 620 of the chain is designated as the "head" or "master" of the chain. In the above example: , Replication Chain (RC) 650 includes two hosts (e.g., primary host machine 620 and backup host machine 630), with primary host machine 120 being the head of the chain and backup host machine 630 being the tail of RC 650.

[0138] The communication network 614 facilitates communication between the various systems and networks depicted in Figure 6. The communication network 614 can be of various types and can include one or more communication networks. Examples of the communication network 614 include, but are not limited to, a public network such as the Internet, a wide area network (WAN), a local area network (LAN), an Ethernet network, a public or private network, a wired network, a wireless network, and combinations thereof. Different communication protocols are used to communicate with the IEEE 802.XX suite of networks. The communications network 614 may facilitate communications including both wired and wireless protocols such as protocols, TCP / IP, IPX, SAN, AppleTalk®, Bluetooth®, and other protocols. In general, the communications network 614 may include any infrastructure that facilitates communications between the various systems depicted in FIG.

[0139] 7A is an exemplary block diagram illustrating multiple host machines in a replication chain, according to certain embodiments. In an exemplary embodiment, as shown in FIG. 7A, primary host machine 620 may include various components or subsystems, including replication system (RS) 622 and state information cache 624. Backup host machine 630 may include various components or subsystems, including replication system (RS) 632 and replicated state information cache 634.

[0140] 7A, RS 622 is a software application in primary host machine 620 that performs processing related to managing and replicating state information for communication channel 616 between CPE 608 and primary host machine 620 (as shown in FIG. 6). The state information for communication channel 616 may be stored in state information cache 624. State information cache 624 may be an in-memory key-value store for caching and managing state information for communication channel 616.

[0141] In some implementations, RS 622 can provide a caching application programming interface (API). RS clients (e.g., processors within host machines that process packets) can use this API to perform operations that manage state information for communication channel 616. For example, a packet processor within primary host machine 620 can receive packets sent to the Internet Protocol (IP) address of primary host machine 620. The packet processor can hash the packet's header and analyze the hash data to determine changes to state information for communication channel 616.

[0142] Hashing of the packet's header can identify tuple fields (e.g., packet identifier, source IP address, source port, destination IP address, destination port, and Layer 4 protocol). The packet processor can use this information to identify state information (e.g., packet number) associated with the communication channel. The processor can further determine whether this state information has changed relative to the communication channel state information stored in a cache associated with the primary host machine. Upon determining that the state information has changed, the packet processor can forward the changes to the communication channel's 616 state information for storage and replication to the backup host machine. Thus, any information associated with the communication channel 616 may be included in the packet before being received by the replication system 622. The signal may be received and analyzed by a network processor.

[0143] In an alternative implementation, RS 622 is a processor that receives and processes packets and performs operations to manage a cache responsible for the state information of communication channel 616. RS 622 in host machine 620 can receive packets, identify the state information of communication channel 616, and forward the state information to state information cache 624. After RS ​​622 obtains the state information from the received packets, it can perform the necessary processing to manage the state information on a backup host machine in the replication chain. In a particular embodiment, RS 622 can obtain information about backup host machines of communication channels adjacent to the primary host machine in replication chain 650 by communicating with a disk backup storage device.

[0144] In an exemplary embodiment, another host machine 630 may include a separate replication system (RS) 632 connected to a replication state information (RSI) cache 634. In some implementations, each of the host machines (primary host machine 620 and backup host machine 630) has a separate replication system (622 and 632) configured to initiate and / or perform replication-related functions. The replication system 622 of the primary host machine 620 communicates directly with the replication system 632 of the backup host machine, thereby enabling the replication system 632 to replicate the state information of the communication channel 616. Similarly, in some implementations, when a replication chain (RC) 650 includes three or more host machines, the replication system of one host machine can insert, update, or delete entries related to the state information of the communication channel 616 replicated by the chain by communicating with subsequent host machines in the chain.

[0145] A set of replication systems can work cooperatively to replicate state information. In the above embodiment, each replication system in the set is fully aware of the configuration of the chain. The configuration information can identify the host machines in the chain, the head of the chain, and the tail of the chain. Thus, each replication system knows where to send requests (e.g., requests to update entries in the communication channel state or requests to invalidate specific entries in the communication channel state). In some implementations, the primary host machine 620 in the RC 650 that is actively processing packets for the communication channel 616 is the head of the chain 650 and can request adjacent backup host machines (e.g., 630) in the communication channel to replicate changes to the communication channel's state information.

[0146] As shown in Figures 6 and 7A, the primary and backup host machines in chain RC 650 are configured to store state information for communication channel 616. Thus, primary host machine 620 is the head of chain RC 650, and backup host machine 630 is the tail of chain 650. The state information stored in primary host machine 620 is replicated in backup host machine 630. Specifically, RS 622 can forward changes to the state information to RS 632 of backup host machine 630. The replication steps are described below with reference to Figure 10. RS 632 can store the state information in a replicated state information (RSI) cache 634.

[0147] The state information for the communication channel 616 may include multiple types of state information related to the communication channel. The state information cache 624 may include a state information table, which may include a specific entry for a sequence number 701 of a packet received by the host machine 620, encryption and decryption key information 702 for the communication channel, Border Gateway Protocol (BGP) state information 703, and the internet address of the communication channel 616. IKE state information 704.

[0148] In certain scenarios, each time a primary host machine receives a packet, it may frequently update state information, such as the packet's sequence number. The packet's sequence number may be different, indicating a change in state information. For example, each time primary host machine 620 receives a packet, the sequence number in the packet's state information is incremented. Replication system 622 can identify the incremented packet's sequence number as a change in state information and transmit this change to its immediate successor in the chain (e.g., 630).

[0149] In other scenarios, certain state information, such as encryption and decryption keys for the communication channel, may be updated infrequently, typically every 20 minutes to an hour, when a communication channel connection needs to be established between two endpoints or when keys need to be renegotiated between the endpoints. The state involved in negotiating the communication channel is known as the IKE state. Because there is an IKE daemon on each side that negotiates all the parameters of the IPSEC SA, the associated state must also be replicated. Like the state related to encryption keys, the IKE state does not change frequently.

[0150] The replicated state information cache 634 may contain copies of state information from the state information cache 624, shown as sequence numbers 706 of packets received by the host machine 620, encryption and decryption key information 707 for communication channels, Border Gateway Protocol (BGP) state information 708, and Internet Key Exchange (IKE) state information 709.

[0151] In an exemplary embodiment, when a failover occurs to the primary host machine 620, the backup host machine that is the immediate successor of the primary host machine in the RC 650 can become the head of the chain. If the primary host machine 620 in the replication chain 650 fails and the head of the replication chain (RC) 650 needs to be reconfigured, access to the chain must be blocked until a new head is reconfigured. In some scenarios, all replication systems in other host machines in the chain will eventually learn of the head change. In other scenarios, the replication system 622 in the primary host machine 620 will only be aware of the immediate successor that is the backup for the host machine, without having additional knowledge of the chain topology and all nodes in the chain. Failover events are further explained in the discussion of FIG. 11.

[0152] The replication system 622 can support get and put operations (to support replication requests from the packet processor) to replicate and manage communication channel-related state information. The replication system 622 can operate on the same machine (e.g., 620) as the packet processor or on a different machine than the packet processor. The RS 622 can act as a gateway to other host machines in the RC 650. In an exemplary embodiment, state information table entries in the state information cache 624 may be processed by the RS 622 via three basic operations: 1) a read operation (the replication system can look up a particular key entry and return the associated value), 2) a write operation (the replication system can write an entry to the cache), and 3) a delete operation (the replication system can delete an entry in the cache).

[0153] In an exemplary embodiment, to provide high throughput of operations within the replication system, operational requests (e.g., requests to store and replicate changes in communication channel state information) If the request is successfully sent to the replication system, the replication system can provide a response to the request immediately or at a later time, depending on the type of operating mode (as further described with reference to FIG. 10). The packet processor can send a new request while the replication system is processing a previous replication request. Thus, the replication system can be fully pipelined with replication requests.

[0154] In an exemplary embodiment, for an operation that requires replication system 622 to forward a request to replicate state information to another replication system 632 in a backup host machine (e.g., 630), replication system 622 can queue the request in its own work queue and send it to peer replication system 632 to perform the operation. Peer-to-peer communication in the replication chain may occur over standard network protocols via messages. In an exemplary embodiment, TCP may be used for peer-to-peer communication of the replication systems to ensure order of communication.

[0155] In an exemplary embodiment, a packet processor or state monitoring system can request that entries managed by the replication system expire after a certain time. If such expiration is requested, the head replication system (e.g., 622) can store the expiration time and remove the mapping when the expiration time is reached. In the above embodiment, if the entry is also cached on other machines, the head replication system first has the other machines remove the entry before removing the expired entry from its own cache. In an alternative embodiment, the packet processor or state monitoring system can request notification when an entry expires. Thus, the head replication system can provide notification of expired entries to the processor so that clients can perform additional cleanup when the entries expire.

[0156] In an exemplary embodiment, replication system 622 may provide support for caching entries in state information cache 624 in a "shared" mode or a "read-only" mode. In an exemplary embodiment, entries related to communication channel-related state information may be cached in the form of key-value pairs in a key-value database (cache 624) associated with the head or primary host machine 620. In an exemplary embodiment, when the state of a communication channel is updated, the replication system updates the entry in itself (e.g., in-memory key-value storage) and then has other replication systems in the chain update the entry. In the above embodiment, writing this information to the chain ensures that data entries are preserved in the event of a head failure.

[0157] In an exemplary embodiment, replication system 622 can cache network state using shared caching. In a shared cache configuration, multiple replication systems (e.g., 622 and 632) in RC 650 maintain copies of the same entries but can only use these entries for reads. In this case, only replication system 622 in the head or primary host machine can edit state information entries. The head of RC 650 manages and replicates the state information and functions to pass the state information to other host machines in the chain (RC) 650. Therefore, the head is the only member that can propose changes related to state information in replication chain 650.

[0158] In an exemplary embodiment, when an entry needs to be updated, the replication system 622 of the primary host machine 620 can request that backup host machines (e.g., 630) that have cached different state information invalidate the entry associated with the state information before replicating the update within the replication chain. ,Every host machine in a replication chain is not aware of the ,configuration of the entire chain in the infrastructure, but only its ,immediate successor., ,Thus, every host machine knows where to send replication requests., ,In ,an ,alternative embodiment, ,a host machine ,can ,obtain ,information ,about ,its ,immediate ,successor ,from ,a ,disk ,backup ,storage.

[0159] It should be noted that the configuration illustrated in FIG. 7A is not intended to unduly limit the scope of the claimed embodiments. Many variations, alternatives, and modifications are possible. For example, FIG. 7B illustrates a centralized configuration of replication system 622. Specifically, replication system 622 may be a centralized entity that controls / manages replication operations associated with host machines included in replication chain 650. Replication system 622 may replicate and store state information data in caches associated with the host machines included in replication chain 650 (i.e., primary host machine 620, backup host machine 630, etc.).

[0160] FIG. 8 illustrates a schematic flowchart 800 of a method for replicating communication channel-related state information to a backup host machine, according to certain embodiments. The process illustrated in FIG. 8 may be implemented using software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of each system, hardware, or a combination thereof. The software may be stored in a non-transitory storage medium (e.g., a memory device). The method illustrated in FIG. 8 and described below is intended to be exemplary and non-limiting. While FIG. 8 depicts various process steps occurring in a particular sequence or order, this is not intended to be limiting. In certain alternative embodiments, the process may be performed in several different orders, or some steps may be performed in parallel. In certain embodiments, the steps illustrated in FIG. 8 may be performed by one or more components illustrated in FIGS. 7A and 7B.

[0161] In an exemplary embodiment, a communication channel 616 is established to securely process packets between a CPE 608 in a customer on-premises network and a primary host machine 620 in a cloud service provider infrastructure (CSPI) 660, as described in Figures 6, 7A, and 7B.

[0162] In step 810, a primary host machine and a backup host machine for the communication channel are configured. To configure a backup host machine 630 for the communication channel 616, the RS 622 in the primary host machine can identify the backup host machine for the communication channel 616 by communicating with a disk backup storage device. Thus, in some implementations, the primary host machine 620 can configure and assign the backup host machine 630 for storing state information for the communication channel 616 during the startup or bootstrap process of the primary host machine 620.

[0163] In step 820, an event that causes a change in the state information of the communication channel is detected. Specifically, a processor in the primary host machine 620 or a replication system 622 can detect an event that causes a change in the state information of the communication channel stored in the state information cache 624. For example, if a processor in the primary host machine 620 receives a packet, this packet may have a sequence number that is different from a previously stored sequence number. Thus, receiving a packet is an event that causes a change in the state information. Similarly, a security association (SA) between the primary host machine 620 and the CPE 608 for the communication channel 616 may be updated. ), i.e., when the SA between two endpoints needs to be renegotiated, an endpoint, e.g., primary host machine 620, can receive packets with encryption and decryption keys to renegotiate. Packets containing these encryption keys are a type of event that causes a change in state information for communication channel 616.

[0164] In step 830, a change in communication channel-related state information stored by the primary host machine is determined. Specifically, in some implementations, after detecting the event in step 820, the replication system 622 determines a change in state information after analyzing the packet received by the primary host machine 620 by comparing the state information from the packet with the state information stored in the cache 624. For example, the replication system 622 can inspect the packet received by the primary host machine 620. The replication system 622 can hash the header of the packet received by the primary host machine 620. The replication system 622 can determine the packet's sequence number from the hash of the packet's header. The packet's header can include a sequence or identification of the packet. When the new packet is received by the host machine 620, it has a different sequence number than previously received packets.

[0165] 7A, a packet processor or monitoring system may receive and analyze packets sent to the primary host machine 620 related to the state information on the communication channel 616. When the packet processor or monitoring system identifies a change in the state information, the processor may request the replication system 622 to update and replicate the change in the state information in the backup host machine 630.

[0166] In step 840, a backup host machine for storing the communication channel state information is identified. Specifically, the replication system 622 can identify the backup host machine by querying the disk backup data storage device. Alternatively, the replication system 622 can obtain the backup host machine information from the configuration data of the primary host machine 620. Specifically, the replication system 622 can obtain the IP address of the replication system (e.g., 632) in the backup host machine (e.g., 630).

[0167] In step 850, the state information changes of the communication are replicated to the identified backup host machine. Specifically, replication system 622 can send the state information changes to replication system 632 in backup host machine 630, and replication system 632 can store the state information changes received from replication system 622 in replication state information cache 634. Before storing the state information changes in cache 634, replication system 632 can delete or invalidate previously stored state information entries.

[0168] FIG. 9 is a flowchart 900 illustrating a replication channel replicating communication channel-related state information according to a particular embodiment. The process shown in FIG. 9 may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of each system, hardware, or a combination thereof. The software may be stored in a non-transitory storage medium (e.g., a memory device). The method shown in FIG. 9 and described below is intended to be exemplary and non-limiting. While FIG. 9 depicts various processing steps occurring in a particular sequence or order, it is not intended to be limiting.

[0169] In some alternative embodiments, the processes may be performed in several different orders, or some steps may be performed in parallel. In some embodiments, the processing steps illustrated in Figure 9 may be performed by one or more components (e.g., replication systems) within the host machines illustrated in Figures 6, 7A, and 7B. Specifically, Figure 9 is a flowchart illustrating steps performed by components (within the host machine receiving the packet) to replicate communication channel-related state information within a replication chain, according to particular embodiments. In an exemplary embodiment, as shown in Figure 6, a communication channel 616 is established to securely process packets between a CPE 608 within a customer on-premises network 602 and a primary host machine 620 within a cloud service provider infrastructure (CSPI) 660.

[0170] In step 910, a replication chain is configured to store state information related to a communication channel between a customer on-premises network and a customer virtual network hosted by the CSPI. The replication chain may include a set of host machines for storing communication channel-related state information. The host machines in the replication chain are arranged linearly to form a chain (e.g., RC 650). A primary host machine 620 of the chain is designated as the "head" or "master" of the chain, and the remaining nodes in the chain are designated as backup host machines for the communication channel 616.

[0171] In step 920, the node in the replication chain corresponding to the primary host machine is designated as the head node of the replication chain. Specifically, if the host machine 620 is an endpoint on one side of the communication channel 616 and receives traffic from a second endpoint (CPE 608) on the other side of the communication channel 616, the host machine 620 is designated as the primary host machine for the communication channel 616. Thus, in the replication chain, the primary host machine for the communication channel is designated as the head node of the chain.

[0172] In step 930, the remaining nodes in the replication chain, i.e., backup host machines, are configured as successors to the head of the replication chain. For communication channel 616, if a host machine in CSPI 660 does not primarily receive traffic for communication channel 616 but is assigned to store state information for the communication channel, the host machine is designated as a backup host machine 630 for communication channel 616. In some implementations, the replication chain can include a single backup host machine 630 for primary host machine 620, as shown in FIG. 6. In other implementations, multiple host machines may be designated as backup host machines for communication channel 616.

[0173] In step 940, a failure of the head node of the replication chain is detected. A mechanism including a chain log replication protocol can control and manage membership in the chain. This mechanism can also detect failures by monitoring the health of the head of the chain.

[0174] In step 950, upon detecting a failure of the head node of the replication chain, the closest successor of the head node in the replication chain is set as the new head node of the replication chain. Thus, the closest successor (e.g., 630) is promoted to head node. In some scenarios, when the primary host machine recovers from failure mode, it again becomes the head node of the chain. In other scenarios, when the primary host machine recovers from failure mode, it joins the end of the chain and may be promoted when other hosts are removed from the chain. In an exemplary implementation, there may be multiple chains containing multiple communication channels, and each node may be the head of more than one chain. Specifically, each chain may handle a different set of SAs, and one node may handle one SA. It can be the head node of a chain and a successor node of another chain.

[0175] 10 is a flowchart 1000 illustrating detailed processing performed by a primary host machine to replicate communication channel-related state information according to a particular embodiment. Specifically, the flowchart presents steps performed by a head host machine to replicate network state information when the head host machine receives a packet for a particular flow.

[0176] In step 1010, an event that causes a change in the state information of the communication channel is detected. Specifically, the replication system 622 in the primary host machine 620 can detect the event that causes a change in the state information of the communication channel stored in the state information cache 624. For example, the replication system 622 can analyze packets received by the primary host machine 620. The replication system 622 can hash the headers of the packets received by the primary host machine 620. The replication system 622 can determine the packet's sequence number from the hash of the packet's header. The header of the first packet may be a standard 5-tuple packet header that includes a source IP address, a source port, a destination IP address, a destination port, and information about the Layer 4 protocol (e.g., TCP, UDP, etc.). The packet header also includes a sequence number or identification number, i.e., an identifier associated with the packet. When the new packet is received by the host machine 620, it will have a different sequence number than previously received packets.

[0177] In step 1015, a change in communication channel-related state information stored by the primary host machine is determined. Specifically, after detecting the event in step 1010, the replication system 622 determines a change in state information after analyzing packets received by the primary host machine 620.

[0178] In step 1017, the determined communication channel-related state information changes are stored in a local cache of primary host machine 620. Specifically, a processor within host machine 620 can send the changes to replication system 622. Replication system 622 can store the state information changes in state information cache 624.

[0179] In step 1020, after storing the state information for the communication channel, the replication system 622 determines a mode for replicating the state information of the communication channel 616 in the backup host machine. In some embodiments, the operational mode for replicating the state information may be one of a safe operational mode and a best-effort operational mode. The operational mode may be determined based on one or more conditions. For example, if maintenance on the primary host machine 620 is scheduled within a threshold time after determining a change in the state information of the communication channel 616, the replication system 622 may determine the operational mode for replication to be the safe mode. Similarly, if the replication system 622 determines that the change in the state information is related to establishing communication for the communication channel 616, the operational mode is the safe mode for replicating the state information. Specifically, as described below, the safe operational mode ensures continued and successful operation of the communication channel, for example, in scenarios where the host machine may be offline for maintenance.

[0180] If, in step 1030, it is determined that a change in communication channel-related state information is being replicated in safe mode, it halts processing of the packet on the primary host machine 620. Specifically, the replication system 622 can alert the packet processor to temporarily halt the packet.

[0181] In step 1035, when host machine 620 stops processing the packet, replication system 622 can identify a replication chain configured for communication channel 616. The replication chain can be identified for communication channel 616 and one or more backup host machines in the replication chain by querying a disk backup data storage system. In some scenarios, information about the replication chain and one or more backup host machines for communication channel 616 is available to primary host machine 620 and may be configured at startup.

[0182] In step 1040, after identifying the replication chain for communication channel 616, one or more backup host machines are identified for communication channel 616. Specifically, replication system 622 in primary host 620 may identify another replication system (for communication channel 616), i.e., a replication system associated with a backup host machine.

[0183] In step 1045, the change in communication channel-related state information is replicated to the backup host machine. Specifically, the replication system 622 in the primary host machine 620 can send the change in communication channel-related state information to the backup host machine, which is a successor node in the replication chain of the primary host machine 620. The successor node as a backup host machine can have the state information replicated to the next backup host machine in the replication chain. Thus, each backup host machine in the replication chain replicates the change in state information. Specifically, the change in state information is replicated sequentially to each host machine in the replication chain. All replication systems in the chain can cache this entry, i.e., the change in state information, in their local caches. Once the entry has been replicated among all host machines in the chain, the head host machine can receive an acknowledgement or notification from the tail host machine of the chain.

[0184] According to one embodiment, as described above, in a mechanism for replicating state information changes executed on host machines in a replication chain, a head host machine can identify peer host machines in the replication chain by accessing disk-backed storage. In the above embodiment, a processor in the head host machine or host machine can send an entry insertion message with entry information to an immediate successor or peer host machine down the chain. The peer host machine can cache the entry in a local cache and send a similar entry insertion message to the next host machine in the replication chain for replication.

[0185] In step 1050, the head host machine receives an acknowledgement message indicating that the state information has been replicated by the backup host machine. Specifically, the replication system 622 in the head host machine receives an acknowledgement message from the tail host machine in the replication chain indicating that the change in the state information for the communication channel has been incorporated into all intermediate host machines. Note that the acknowledgement message may be sent from the tail host machine to the head host machine using any suitable communication protocol used for communication between host machines.

[0186] For example, according to some embodiments, a replication chain uses a log-structured mechanism (e.g., a chain log replication protocol) to perform replication among backup host machines. This type of replication protocol under a safe mode of operation provides an option to enhance safety by halting packet processing until it is guaranteed that changes to state information have replicated to all backup host machines.

[0187] According to some embodiments, safe mode can have different implementations: For example, in some implementations, in the safe mode of operation, as long as state changes are synchronized within a certain window, packets do not need to be aborted until the state is absolutely synchronized. The window may be defined by a number of packets. For example, the size of the window may be W=3 packets. This mode of operation may be referred to as the relaxed-safe mode of operation. The primary host machine may continue processing packets before all state changes are replicated for the entire window. According to some embodiments, another mode of operation for replicating state information may be the best-effort mode. In this mode of operation, similar to the relaxed-safe mode of operation, channel information is replicated without stopping or halting packet processing operations. However, the difference between the best-effort mode and the relaxed-safe mode of operation is that in the best-effort mode, the size of the window is very large, i.e., infinite. In other words, in the best-effort mode of operation, the size of the window is not a constraint in packet processing.

[0188] In step 1055, after receiving an acknowledgement indicating that the state information has been replicated by the backup host machine for the communication channel 616, processing of packets over the communication channel is resumed (eg, in safe mode).

[0189] Returning to step 1020, if it is determined that replication is not being performed in safe mode, for example, if replication is being performed in best effort mode, then there is no need to abort processing packets. If it is determined that replication is not being performed in safe mode, then the processing of packets is not aborted and state information is replicated essentially using steps 1035, 1040 and 1045 described above.

[0190] Note that if state information replication is not fast enough (e.g., replication speed cannot keep up with the rate at which the primary host machine receives packets, i.e., the rate at which packets arrive), one possible outcome is that in the event of a failover, i.e., when the primary host machine fails and the backup host becomes the new head of the replication chain, some packets may be discarded when re-establishing the communication channel with the backup host as the new head. Such a scenario can occur in the best-effort mode of data replication. Note that such a failover event does not compromise the integrity of customer data and therefore does not have a security impact on packet processing.

[0191] In other words, if the sequence of packets is not properly synchronized, it may discard the packets. If there is some failure in replication, i.e., if there is a failover, this means that the successor configured to take over will have outdated sequence numbers and will not be able to pick up the connection (in a fully successful manner) because the packets it is watching are not synchronized. The successor can interpret such a situation as a replay attack and can react by discarding some packets.

[0192] In certain scenarios, replication is performed without using a safe mode, e.g., best-effort mode or relaxed-safe mode, while addressing the sequence numbers of packets of communication channel-related state information. In some implementations, if replication of a large amount of state changes is delayed by a certain amount, e.g., a certain number of packets, then in the event of a failover, the backup host machine can predict how far it needs to jump (or overshoot the sequence numbers) and thereby discard packets that are behind the window. The protocol simply assumes that such packets (i.e., discarded packets) are missing, and the backup host machine can update its packet counters. Thus, the backup host machine always tries to match the highest authenticated sequence number it is monitoring. Therefore, in a mitigation-safe mode of operation, even if the replication of communication sequence number state information falls behind or becomes out of sync, the backup host machine can jump back into the window during failover by jumping forward a predicted amount of packets, e.g., a specific number of packets. This allows the originally established communication channel to continue operating with the backup host as the new head of the replication chain.

[0193] Therefore, when using the safe mode option, replication operations are performed such that packet processing is halted until replication is complete. For example, safe mode can be used when establishing a connection by sending encryption / decryption information between endpoints. However, after establishing a connection, replication can be performed in relaxed safe mode or best-effort mode rather than safe mode. Furthermore, if a state change involves a change in the cryptographic state information, replication mode may be switched to safe mode. Safe mode ensures continuous / complete replication of state information at the backup host machine. On the other hand, best-effort mode ensures a best effort to replicate state information and process packets during a failover scenario of the primary host machine.

[0194] FIG. 11 is a flowchart 1100 illustrating steps performed to handle packet processing during a failover of a primary host machine according to certain embodiments.

[0195] In step 1110, a data plane including the primary host machine receives a signal indicating that the primary host machine is unavailable. The unavailability of the primary host machine may be due to intentional or unintentional reasons. For example, the primary host machine may be intentionally taken down for maintenance. In other cases, the primary host machine may go down due to a sudden failover event within the host machine, which is an unintentional failover.

[0196] In step 1120, a backup host machine for the communication channel may be identified. Specifically, a data plane or processor in the VCN hosted on the CSPI identifies the backup host machine for the communication channel by querying a disk backup storage device for information about the communication channel. In some implementations, the backup host machine may be identified and configured within the data plane.

[0197] In step 1130, packets processed by the communication channel may be forwarded to the identified backup host machine. Specifically, the VCN's data plane can change the packet's destination IP address to the backup host machine's IP address. In this example, the customer terminal endpoint (CPE 608) still sends packets to the primary host machine's IP address, but these packets are redirected by the data plane processor to the backup host machine's IP address.

[0198] In step 1140, the backup host machine processes the packet using the communication channel state information previously synchronized from the primary host machine to the backup host machine. As explained, in the case of a failover due to maintenance scheduled to take the primary host machine offline, replication under safe mode (as explained in FIG. 10) ensures that all state information is synchronized. Therefore, replication under safe mode ensures that the backup host machine processes packets normally when the primary host machine is taken down for maintenance. This can be done.

[0199] While it is not guaranteed that all state information will be synchronized in the case of an unexpected failover scenario, using a replication method (as described in FIG. 10) still provides a high probability of successfully handling the failover scenario without compromising packet security. In the worst case scenario, connection parameters may be lost, requiring the parameters of the communication channel to be renegotiated. Aspects of the present disclosure allow a single highly available tunnel to be allocated from one host to another without bringing down the tunnel and patching the tunnel endpoints.

[0200] Exemplary Cloud Infrastructure Embodiments As mentioned above, IaaS (Infrastructure as a Service) is a specific type of Cloud computing. IaaS may be configured to provide virtualized computing resources over a public network (e.g., the Internet). In the IaaS model, a cloud computing provider may host infrastructure elements (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer), etc.). In some cases, the IaaS provider may offer various services (e.g., billing, monitoring, logging, security, load balancing, clustering, etc.) associated with the infrastructure elements. Therefore, because these services can be policy-driven, IaaS users can implement policies to drive load balancing to maintain application availability and performance.

[0201] In some examples, IaaS customers can access resources and services over a wide area network (WAN), such as the Internet, and use the cloud provider's services to install the remaining elements of their application stack. For example, a user can log into an IaaS platform to create virtual machines (VMs), install an operating system (OS) on each VM, deploy middleware such as databases, create storage buckets for workloads and backups, and install enterprise software on the VMs. Customers can use the provider's services to perform a variety of functions, including balancing network traffic, troubleshooting applications, monitoring performance, managing disaster recovery, and more.

[0202] In most cases, the cloud computing model requires the participation of a cloud provider, which can be, but does not have to be, a third-party service provider specializing in providing (e.g., offering, renting, or selling) IaaS. Alternatively, an enterprise can deploy a private cloud and become a provider of infrastructure services.

[0203] In some examples, IaaS deployment is the process of deploying a new application or a new version of an application onto a provisioned application server, etc. IaaS deployment may include the process of provisioning the server (e.g., installing libraries, daemons, etc.). IaaS deployment is often managed by the cloud provider below the hypervisor layer (e.g., servers, storage, network hardware, and virtualization). Thus, customers can perform OS, middleware, and / or application deployment (e.g., self-service virtual machines (e.g., that can be spun up on demand)).

[0204] In some instances, IaaS provisioning involves the use of computers or may involve obtaining a virtual host and installing the necessary libraries or services on the computer or virtual host. In most cases, deployment does not involve provisioning, which must be performed first.

[0205] In some cases, IaaS provisioning presents two distinct challenges. First, there is the challenge of provisioning an initial set of infrastructure before anything can be done. Second, there is the challenge of evolving the existing infrastructure (e.g., adding new services, modifying services, removing services) after everything has been provisioned. In some cases, these two challenges can be addressed by allowing the configuration of the infrastructure to be defined declaratively. In other words, the infrastructure (e.g., which elements are needed and how these elements interact) may be defined by one or more configuration files. Thus, the overall topology of the infrastructure (e.g., which resources depend on which and how they work together) can be described declaratively. In some instances, once the topology is defined, workflows can be generated to create and / or manage the different elements described in the configuration files.

[0206] In some examples, the infrastructure can include many interconnected elements. For example, there may be one or more virtual private clouds (VPCs) (e.g., potentially on-demand pools of configurable and / or shared computing resources), also known as a core network. In some examples, there may be one or more security group rules and one or more virtual machines (VMs) that are provisioned to define how the network is secured. Other infrastructure elements, such as load balancers, databases, etc., may also be provisioned. The infrastructure can evolve incrementally as more infrastructure elements are desired and / or added.

[0207] In some examples, continuous deployment techniques may be employed to enable deployment of infrastructure code across various virtual computing environments. The described techniques may also enable infrastructure management within these environments. In some examples, a service team may write code that is desired to be deployed to one or more, typically many, different production environments (e.g., across a variety of different geographic locations, sometimes spanning the entire world). However, in some examples, the infrastructure for deploying the code must first be set up. In some examples, provisioning may be done manually, with provisioning tools used to provision resources and / or deployment tools used to deploy the code after the infrastructure has been provisioned.

[0208] 12 is a block diagram 1200 illustrating an example pattern of an IaaS architecture, according to at least one embodiment. A service operator 1202 may be communicatively connected to a secure host tenancy 1204, which may include a virtual cloud network (VCN) 1206 and a secure host subnet 1208. In some examples, the service operator 1202 may use one or more client computing devices. The one or more client computing devices may run, for example, software such as Microsoft Windows Mobile® and / or various operating systems such as iOS, Windows Phone, Android®, BlackBerry 8, and Palm OS. A handheld mobile device (e.g., iPhone, mobile phone) capable of running a mobile operating system and enabled for Internet, email, Short Message Service (SMS), Blackberry, or other communication protocols. The device may be a smartphone, iPad (registered trademark), tablet, personal digital assistant (PDA), or wearable device (such as a Google (registered trademark) Glass (registered trademark) head-mounted display). The client computing device illustratively includes a Microsoft Windows Operating System, Apple Macintosh® Operating System and and / or running various versions of the Linux operating system. The client computing devices may be general-purpose personal computers, including personal computers and / or laptop computers running the VCN 1206. Alternatively, the client computing devices may be workstation computers running various commercially available UNIX or UNIX-like operating systems, including, but not limited to, various GNU / Linux operating systems, e.g., Google Chrome® OS. Alternatively or additionally, the client computing devices may be other electronic devices capable of communicating over VCN 1206 and / or a network with access to the Internet, such as thin-client computers, Internet-enabled gaming systems (e.g., Microsoft Xbox® game consoles with or without Kinect® gesture input devices), and / or personal messaging devices.

[0209] VCN 1206 may include a local peering gateway (LPG) 1210 that may be communicatively connected to a secure shell (SSH) VCN 1212 via an LPG 1210 included in SSH VCN 1212. SSH VCN 1212 may include an SSH subnet 1214, which may be communicatively connected to a control plane VCN 1216 via an LPG 1210 included in control plane VCN 1216. SSH VCN 1212 may also be communicatively connected to a data plane VCN 1218 via LPG 1210. Control plane VCN 1216 and data plane VCN 1218 may be included in a service tenancy 1219, which may be owned and / or operated by the IaaS provider.

[0210] The control plane VCN 1216 may include a control plane demilitarized zone (DMZ) tier 1220 that functions as a perimeter network (e.g., a portion of an enterprise network between the enterprise intranet and an external network). DMZ-based servers have a particular level of reliability and can contain security breaches. Additionally, the DMZ tier 1220 may include one or more load balancer (LB) subnets 1222, a control plane app tier 1224 that may include an app subnet 1226, and a control plane data tier 1228 that may include a database (DB) subnet 1230 (e.g., a front-end DB subnet and / or a back-end DB subnet). LB subnet 1222 included in control plane DMZ tier 1220 may be communicatively connected to app subnet 1226 included in control plane app tier 1224 and to an Internet gateway 1234 that may be included in control plane VCN 1216, and app subnet 1226 may be communicatively connected to DB subnet 1230, service gateway 1236, and network address translation (NAT) gateway 1238 included in control plane data tier 1228. Control plane VCN 1216 may include service gateway 1236 and NAT gateway 1238.

[0211] The control plane VCN 1216 can include a data plane mirror app layer 1240, which can include an app subnet 1226. The app subnet 1226 included in the data plane mirror app layer 1240 can include a virtual network interface controller (VNIC) 1242 on which a compute instance 1244 can run. The compute instance 1244 can communicatively connect the app subnet 1226 of the data plane mirror app layer 1240 to the app subnet 1226, which can be included in the data plane app layer 1246. can.

[0212] Data plane VCN 1218 may include a data plane app layer 1246, a data plane DMZ layer 1248, and a data plane data layer 1250. Data plane DMZ layer 1248 may include a LB subnet 1222 that may be communicatively connected to an app subnet 1226 of data plane app layer 1246 and an internet gateway 1234 of data plane VCN 1218. App subnet 1226 may be communicatively connected to a service gateway 1236 of data plane VCN 1218 and a NAT gateway 1238 of data plane VCN 1218. Data plane data layer 1250 may also include a DB subnet 1230 that may be communicatively connected to app subnet 1226 of data plane app layer 1246.

[0213] The internet gateway 1234 of the control plane VCN 1216 and the internet gateway 1234 of the data plane VCN 1218 may be communicatively connected to a metadata management service 1252, which may be communicatively connected to the public internet 1254. The public internet 1254 may be communicatively connected to a NAT gateway 1238 of the control plane VCN 1216 and the NAT gateway 1238 of the data plane VCN 1218. The service gateway 1236 of the control plane VCN 1216 and the service gateway 1236 of the data plane VCN 1218 may be communicatively connected to a cloud service 1256.

[0214] In some examples, the service gateway 1236 of the control plane VCN 1216 or the service gateway 1236 of the data plane VCN 1218 can make application programming interface (API) calls to the cloud service 1256 without traversing the public internet 1254. The API calls from the service gateway 1236 to the cloud service 1256 can be one-way. The service gateway 1236 can make API calls to the cloud service 1256, and the cloud service 1256 can send request data to the service gateway 1236. However, the cloud service 1256 cannot initiate API calls to the service gateway 1236.

[0215] In some examples, secure host tenancy 1204 may be directly connected to service tenancy 1219, which may be an orphan. Secure host subnet 1208 can communicate with SSH subnet 1214 through LPG 1210, which allows bidirectional communication with the orphan system. By connecting secure host subnet 1208 to SSH subnet 1214, secure host subnet 1208 can access other entities in service tenancy 1219.

[0216] Control plane VCN 1216 allows users of service tenancy 1219 to configure or provision desired resources. The desired resources provisioned in control plane VCN 1216 may be deployed or used in data plane VCN 1218. In some examples, control plane VCN 1216 may be isolated from data plane VCN 1218, and data plane mirror app layer 1240 of control plane VCN 1216 can communicate with data plane app layer 1246 of data plane VCN 1218 via VNIC 1242, which may be included in data plane mirror app layer 1240 and data plane app layer 1246.

[0217] In some examples, users or customers of the system can make requests, such as create, read, update, or delete (CRUD) operations, over the public internet 1254, which can communicate the requests to the metadata management service 1252. Metadata management service 1252 can communicate the request to control plane VCN 1216 via internet gateway 1234. The request may be received by LB subnet 1222 included in control plane DMZ tier 1220. LB subnet 1222 can determine that the request is valid, and in response to this determination, LB subnet 1222 can send the request to app subnet 1226 included in control plane app tier 1224. If the request is validated and requires a call to the public internet 1254, the call to the public internet 1254 can be sent to NAT gateway 1238, which can make the call to the public internet 1254. Memory for storing the request may be stored in DB subnet 1230.

[0218] In some examples, data plane mirror app layer 1240 can facilitate direct communication between control plane VCN 1216 and data plane VCN 1218. For example, it may be desirable for changes, updates, or other suitable modifications to a configuration to be applied to resources included in data plane VCN 1218. Because control plane VCN 1216 can communicate directly with resources included in data plane VCN 1218 via VNIC 1242, changes, updates, or other suitable modifications to the configuration can be made.

[0219] In some embodiments, the control plane VCN 1216 and the data plane VCN 1218 may be included in the service tenancy 1219. In this case, a user or customer of the system may not own or operate either the control plane VCN 1216 or the data plane VCN 1218. Instead, an IaaS provider may own or operate the control plane VCN 1216 and the data plane VCN 1218, both of which may be included in the service tenancy 1219. This embodiment can prevent users or customers from interacting with other users' or customers' resources by enabling network isolation. This embodiment can also enable users or customers of the system to store databases privately without having to rely on the public internet 1254, which may not have the desired level of security for storage.

[0220] In another embodiment, LB subnet 1222 included in control plane VCN 1216 may be configured to receive signals from service gateway 1236. In this embodiment, control plane VCN 1216 and data plane VCN 1218 may be configured to be called by customers of the IaaS provider without calling the public internet 1254. Customers of the IaaS provider may desire this embodiment because databases used by the customers may be stored in service tenancy 1219, which is controlled by the IaaS provider and may be isolated from the public internet 1254.

[0221] FIG. 13 is a block diagram 1300 illustrating further example parameters of an IaaS architecture, according to at least one embodiment. A service operator 1302 (e.g., service operator 1202 of FIG. 12 ) may be communicatively connected to a secure host tenancy 1304 (e.g., secure host tenancy 1204 of FIG. 12 ), which may include a virtual cloud network (VCN) 1306 (e.g., VCN 1206 of FIG. 12 ) and a secure host subnet 1308 (e.g., secure host subnet 1208 of FIG. 12 ). VCN 1306 may include a local peering gateway (LPG) 1310 (e.g., LPG 1210 of FIG. 12 ), which may be communicatively connected to a secure shell (SSH) VCN 1312 (e.g., SSH VCN 1212 of FIG. 12 ) via an LPG 1310 included in SSH VCN 1312. SSH VCN 1312 may include a local peering gateway (LPG) 1310 (e.g., LPG 1210 of FIG. 12 ), which may be communicatively connected to a secure shell (SSH) VCN 1312 (e.g., SSH VCN 1212 of FIG. 12 ). SSH VCN 1312 may include a local peering gateway (LPG) 1310 (e.g., LPG 1210 of FIG. 12 ). SSH VCN 1312 may include a local peering gateway (LPG) 1310 included in SSH VCN 1312. ... 12 ) and SSH VCN 1312 can be communicatively connected to control plane VCN 1316 (e.g., control plane VCN 1216 in FIG. 12 ) via LPG 1310 included in control plane VCN 1316. Control plane VCN 1316 can be included in service tenancy 1319 (e.g., service tenancy 1219 in FIG. 12 ), and data plane VCN 1318 (e.g., data plane VCN 1218 in FIG. 12 ) can be included in customer tenancy 1321, which can be owned or operated by a user or customer of the system.

[0222] The control plane VCN 1316 may include a control plane DMZ layer 1320 (e.g., the control plane DMZ layer 1220 in FIG. 12 ) that may include a LB subnet 1322 (e.g., the LB subnet 1222 in FIG. 12 ), a control plane app layer 1324 (e.g., the control plane app layer 1224 in FIG. 12 ) that may include an app subnet 1326 (e.g., the app subnet 1226 in FIG. 12 ), and a control plane data layer 1328 (e.g., the control plane data layer 1228 in FIG. 12 ) that may include a database (DB) subnet 1330 (e.g., similar to the DB subnet 1230 in FIG. 12 ). The LB subnet 1322 included in the control plane DMZ layer 1320 may be communicatively connected to the app subnet 1326 included in the control plane app layer 1324 and to an Internet gateway 1334 (e.g., the Internet gateway 1234 in FIG. 12 ), which may be included in the control plane VCN 1316. The app subnet 1326 may be communicatively connected to a DB subnet 1330, a service gateway 1336 (e.g., the service gateway in FIG. 12 ), and a network address translation (NAT) gateway 1338 (e.g., the NAT gateway 1238 in FIG. 12 ) included in the control plane data layer 1328. The control plane VCN 1316 may include the service gateway 1336 and the NAT gateway 1338.

[0223] The control plane VCN 1316 can include a data plane mirror app layer 1340 (e.g., data plane mirror app layer 1240 of FIG. 12 ), which can include an app subnet 1326. The app subnet 1326 included in the data plane mirror app layer 1340 can include a virtual network interface controller (VNIC) 1342 (e.g., VNIC 1242) on which a compute instance 1344 (e.g., similar to compute instance 1244 of FIG. 12 ) can run. The compute instance 1344 can facilitate communication between the app subnet 1326 of the data plane mirror app layer 1340 and the app subnet 1326, which can be included in the data plane app layer 1346 (e.g., data plane app layer 1246 of FIG. 12 ), via the VNIC 1342 included in the data plane mirror app layer 1340 and the VNIC 1342 included in the data plane app layer 1346.

[0224] An internet gateway 1334 included in the control plane VCN 1316 may be communicatively connected to a metadata management service 1352 (e.g., metadata management service 1252 of FIG. 12 ), which may be communicatively connected to a public internet 1354 (e.g., public internet 1254 of FIG. 12 ). The public internet 1354 may be communicatively connected to a NAT gateway 1338 included in the control plane VCN 1316. A service gateway 1336 included in the control plane VCN 1316 may be communicatively connected to a cloud service 1356 (e.g., cloud service 1256 of FIG. 12 ).

[0225] In some examples, the data plane VCN 1318 may be included in the customer tenancy 1321. In this case, the IaaS provider may provide a control plane VCN 1316 for each customer, and the IaaS provider may configure a unique compute instance 1344 for each customer that is included in the service tenancy 1319. Each compute instance The compute instance 1344 can allow communication between the control plane VCN 1316 included in the service tenancy 1319 and the data plane VCN 1318 included in the customer tenancy 1321. The compute instance 1344 can allow resources provisioned in the control plane VCN 1316 included in the service tenancy 1319 to be deployed or used in the data plane VCN 1318 included in the customer tenancy 1321.

[0226] In another example, a customer of the IaaS provider may have a database that resides in customer tenancy 1321. In this example, control plane VCN 1316 may include data plane minor app tier 1340, which may include app subnet 1326. Data plane mirror app tier 1340 may reside in data plane VCN 1318, but data plane mirror app tier 1340 may not reside in data plane VCN 1318. That is, data plane mirror app tier 1340 may have access to customer tenancy 1321, but data plane mirror app tier 1340 may not reside in data plane VCN 1318 and may not be owned or operated by the IaaS provider's customer. Data plane mirror app tier 1340 may be configured to make calls to data plane VCN 1318, but may not be configured to make calls to any entities included in control plane VCN 1316. A customer may desire to deploy or use resources in the data plane VCN 1318 provisioned to the control plane VCN 1316, and the data plane mirror application tier 1340 may facilitate the desired deployment or other use of the customer's resources.

[0227] In some embodiments, an IaaS provider's customer can apply filters to the data plane VCN 1318. In this embodiment, the customer can determine what the data plane VCN 1318 can access, and the customer can restrict access from the data plane VCN 1318 to the public internet 1354. The IaaS provider may not be able to apply filters or control access from the data plane VCN 1318 to any external network or database. Applying filters and controls to the data plane VCN 1318 included in the customer tenancy 1321 can help isolate the data plane VCN 1318 from other customers and the public internet 1354.

[0228] In some embodiments, cloud services 1356 can be called by service gateway 1336 to access services that may not reside on the public internet 1354, on the control plane VCN 1316, or on the data plane VCN 1318. The connection between cloud service 1356 and control plane VCN 1316 or data plane VCN 1318 may not be live or continuous. Cloud services 1356 may reside on a separate network owned or operated by the IaaS provider. Cloud services 1356 may be configured to receive calls from service gateway 1336 and may not be configured to receive calls from the public internet 1354. Some cloud services 1356 may be isolated from other cloud services 1356, and control plane VCN 1316 may be isolated from cloud services 1356 that may not be located in the same region as control plane VCN 1316. For example, control plane VCN 1316 may be located in “Region 1,” and cloud service “Deployment 12” may be located in “Region 1” and “Region 2.” If a call to a deployment 12 is made by a service gateway 1336 included in a control plane VCN 1316 located in Region 1, the call may be sent to the deployment 12 in Region 1. In this example, the control plane VCN 1316 or the deployment 12 in Region 1 can communicate with the deployment 12 in Region 2. It does not have to be connected.

[0229] 14 is a block diagram 1400 illustrating another example pattern of an IaaS architecture, according to at least one embodiment. A service operator 1402 (e.g., service operator 1202 of FIG. 12) may be communicatively connected to a secure host tenancy 1404 (e.g., secure host tenancy 1204 of FIG. 12), which may include a virtual cloud network (VCN) 1406 (e.g., VCN 1206 of FIG. 12) and a secure host subnet 1408 (e.g., secure host subnet 1208 of FIG. 12). VCN 1406 may include an LPG 1410 (e.g., LPG 1210 of FIG. 12), which may be communicatively connected to an SSH VCN 1412 (e.g., SSH VCN 1212 of FIG. 12) via an LPG 1410 included in SSH VCN 1412. VCN 1412 can include SSH subnet 1414 (e.g., SSH subnet 1214 in FIG. 12 ), which may be communicatively connected to control plane VCN 1416 (e.g., control plane VCN 1216 in FIG. 12 ) via LPG 1410 included in control plane VCN 1416, and may be communicatively connected to data plane VCN 1418 (e.g., data plane 1218 in FIG. 12 ) via LPG 1410 included in data plane VCN 1418. Control plane VCN 1416 and data plane VCN 1418 may be included in service tenancy 1419 (e.g., service tenant 1219 in FIG. 12 ).

[0230] The control plane VCN 1416 may include a control plane DMZ layer 1420 (e.g., the control plane DMZ layer 1220 in FIG. 12 ) that may include a load balancer (LB) subnet 1422 (e.g., the LB subnet 1222 in FIG. 12 ), a control plane app layer 1424 (e.g., the control plane app layer 1224 in FIG. 12 ) that may include an app subnet 1426 (e.g., similar to the app subnet 1226 in FIG. 12 ), and a control plane data layer 1428 (e.g., the control plane data layer 1228 in FIG. 12 ) that may include a DB subnet 1430. The LB subnet 1422 included in the control plane DMZ layer 1420 may be communicatively connected to the app subnet 1426 included in the control plane app layer 1424 and to an Internet gateway 1434 (e.g., the Internet gateway 1234 in FIG. 12 ), which may be included in the control plane VCN 1416. The app subnet 1426 may be communicatively connected to a DB subnet 1430 included in the control plane data layer 1428, a service gateway 1436 (e.g., the service gateway in FIG. 12 ) and a network address translation (NAT) gateway 1438 (e.g., the NAT gateway 1238 in FIG. 12 ). The control plane VCN 1416 may include the service gateway 1436 and the NAT gateway 1438.

[0231] Data plane VCN 1418 may include a data plane app layer 1446 (e.g., data plane app layer 1246 in FIG. 12 ), a data plane DMZ layer 1448 (e.g., data plane DMZ layer 1248 in FIG. 12 ), and a data plane data layer 1450 (e.g., data plane data layer 1250 in FIG. 12 ). Data plane DMZ layer 1448 may include LB subnet 1422, which may be communicatively connected to trusted app subnet 1460 and untrusted app subnet 1462 of data plane app layer 1446 and internet gateway 1434 included in data plane VCN 1418. Trusted app subnet 1460 may be communicatively connected to service gateway 1436 included in data plane VCN 1418, NAT gateway 1438 included in data plane VCN 1418, and DB subnet 1430 included in data plane data layer 1450. The untrusted app subnet 1462 is connected to the service gateway 1436 contained in the data plane VCN 1418 and the DB subnet 1430 contained in the data plane data tier 1450. The data plane data layer 1450 may include a DB subnet 1430 that may be communicatively connected to a service gateway 1436 included in the data plane VCN 1418.

[0232] The untrusted app subnet 1462 may include one or more primary VNICs 1464(1)-(N), which may be communicatively connected to tenant virtual machines (VMs) 1466(1)-(N). Each tenant VM 1466(1)-(N) may be communicatively connected to a respective app subnet 1467(1)-(N), which may be included in a respective container egress VCN 1468(1)-(N), which may be included in a respective customer tenancy 1470(1)-(N). Each secondary VNIC 1472(1)-(N) may facilitate communication between the untrusted app subnet 1462 included in the data plane VCN 1418 and the app subnet included in the container egress VCN 1468(1)-(N). Each container egress VCN 1468(1)-(N) may include a NAT gateway 1438, which may be communicatively connected to the public internet 1454 (e.g., public internet 1254 in FIG. 12 ).

[0233] The internet gateway 1434 included in the control plane VCN 1416 and the internet gateway 1434 included in the data plane VCN 1418 may be communicatively connected to a metadata management service 1452 (e.g., metadata management system 1252 of FIG. 12 ), which may be communicatively connected to the public internet 1454. The public internet 1454 may be communicatively connected to a NAT gateway 1438 included in the control plane VCN 1416 and the NAT gateway 1438 included in the data plane VCN 1418. The service gateway 1436 included in the control plane VCN 1416 and the service gateway 1436 included in the data plane VCN 1418 may be communicatively connected to cloud services 1456.

[0234] In some embodiments, data plane VCN 1418 may be consolidated into customer tenancy 1470. This consolidation may be useful or desirable for an IaaS provider's customer in some cases, such as when they may want support when running their code. Customers may provide code that, when run, may be disruptive, may communicate with other customer resources, or may cause undesirable effects. Thus, the IaaS provider can determine whether or not to run code that a customer has provided to the IaaS provider.

[0235] In some examples, an IaaS provider's customer can grant temporary network access to the IaaS provider and request a feature to be added to the data plane app layer 1446. The code to perform the feature may run in VMs 1466(1)-(N) but cannot be configured to run elsewhere on the data plane VCN 1418. Each VM 1466(1)-(N) may be connected to one customer tenancy 1470. Each container 1471(1)-(N) included in a VM 1466(1)-(N) may be configured to run code. In this case, double isolation (e.g., containers 1471(1)-(N) may run code, and containers 1471(1)-(N) may be included in at least one VM 1466(1)-(N) included in the untrusted app subnet 1462) may exist, which can help prevent erroneous or unwanted code from damaging the IaaS provider's network or from damaging a different customer's network. Containers 1471(1)-(N) may be communicatively connected to customer tenancy 1470 and may be configured to send or receive data from customer tenancy 1470. Containers 1471(1)-(N) may not be configured to send or receive data from any other entity in data plane VCN 1418. Code Execution Once this is complete, the IaaS provider can kill or discard the containers 1471(I)-(N).

[0236] In some embodiments, trusted app subnet 1460 can execute code that may be owned or operated by the IaaS provider. In this embodiment, trusted app subnet 1460 may be communicatively connected to DB subnet 1430 and configured to perform CRUD operations on DB subnet 1430. Untrusted app subnet 1462 may be communicatively connected to DB subnet 1430, but in this embodiment, the untrusted app subnet may be configured to perform read operations within DB subnet 1430. Containers 1471(1)-(N) included in each customer's VMs 1466(1)-(N) and capable of executing code from the customer may not be communicatively connected to DB subnet 1430.

[0237] In other embodiments, the control plane VCN 1416 and the data plane VCN 1418 may not be directly communicatively coupled. In this embodiment, there may not be direct communication between the control plane VCN 1416 and the data plane VCN 1418. However, there may be indirect communication by at least one method. An LPG 1410 may be established by an IaaS provider that can facilitate communication between the control plane VCN 1416 and the data plane VCN 1418. In another example, the control plane VCN 1416 or the data plane VCN 1418 can make a call to a cloud service 1456 through a service gateway 1436. For example, a call from the control plane VCN 1416 to the cloud service 1456 may include a request for a service that can communicate with the data plane VCN 1418.

[0238] 15 is a block diagram 1500 illustrating further example parameters of an IaaS architecture, according to at least one embodiment. A service operator 1502 (e.g., service operator 1202 of FIG. 12) may be communicatively connected to a secure host tenancy 1504 (e.g., secure host tenancy 1204 of FIG. 12), which may include a virtual cloud network (VCN) 1506 (e.g., VCN 1206 of FIG. 12) and a secure host subnet 1508 (e.g., secure host subnet 1208 of FIG. 12). VCN 1506 may include an LPG 1510 (e.g., LPG 1210 of FIG. 12), which may be communicatively connected to an SSH VCN 1512 (e.g., SSH VCN 1212 of FIG. 12) via an LPG 1510 included in SSH VCN 1512. VCN 1512 can include SSH subnet 1514 (e.g., SSH subnet 1214 in FIG. 12 ), which may be communicatively connected to control plane VCN 1516 (e.g., control plane VCN 1216 in FIG. 12 ) via LPG 1510 included in control plane VCN 1516, and may be communicatively connected to data plane VCN 1518 (e.g., data plane 1218 in FIG. 12 ) via LPG 1510 included in data plane VCN 1518. Control plane VCN 1516 and data plane VCN 1518 may be included in service tenancy 1519 (e.g., service tenancy 1219 in FIG. 12 ).

[0239] The control plane VCN 1516 may include a control plane DMZ layer 1520 (e.g., control plane DMZ layer 1220 of FIG. 12 ) that may include a LB subnet 1522 (e.g., LB subnet 1222 of FIG. 12 ), a control plane app layer 1524 (e.g., control plane app layer 1224 of FIG. 12 ) that may include an app subnet 1526 (e.g., app subnet 1226 of FIG. 12 ), and a control plane data layer 1528 (e.g., control plane data layer 1228 of FIG. 12 ) that may include a DB subnet 1530 (e.g., DB subnet 1430 of FIG. 14 ). The LB subnet included in the control plane DMZ layer 1520 may include a control plane app layer 1524 (e.g., control plane app layer 1224 of FIG. 12 ), a control plane data layer 1528 (e.g., control plane data layer 1228 ... The control plane VCN 1516 may include a service gateway 1536 and a network address translation (NAT) gateway 1538. ...

[0240] Data plane VCN 1518 may include a data plane app layer 1546 (e.g., data plane app layer 1246 in FIG. 12 ), a data plane DMZ layer 1548 (e.g., data plane DMZ layer 1248 in FIG. 12 ), and a data plane data layer 1550 (e.g., data plane data layer 1250 in FIG. 12 ). Data plane DMZ layer 1548 may include a trusted app subnet 1560 (e.g., trusted app subnet 1460 in FIG. 14 ) and an untrusted app subnet 1562 (e.g., untrusted app subnet 1462 in FIG. 14 ) of data plane app layer 1546 and an LB subnet 1522 that may be communicatively connected to an Internet gateway 1534 included in data plane VCN 1518. The trusted app subnet 1560 may be communicatively connected to a service gateway 1536 included in the data plane VCN 1518, a NAT gateway 1538 included in the data plane VCN 1518, and a DB subnet 1530 included in the data plane data layer 1550. The untrusted app subnet 1562 may be communicatively connected to the service gateway 1536 included in the data plane VCN 1518 and the DB subnet 1530 included in the data plane data layer 1550. The data plane data layer 1550 may include a DB subnet 1530 that may be communicatively connected to the service gateway 1536 included in the data plane VCN 1518.

[0241] The untrusted app subnet 1562 may include primary VNICs 1564(1)-(N) that may be communicatively connected to tenant virtual machines (VMs) 1566(1)-(N) that reside in the untrusted app subnet 1562. Each tenant VM 1566(1)-(N) may execute code in a respective container 1567(1)-(N) and may be communicatively connected to an app subnet 1526 that may be included in a data plane app layer 1546 that may be included in a container egress VCN 1568. Each secondary VNIC 1572(1)-(N) may facilitate communication between the untrusted app subnet 1562 included in the data plane VCN 1518 and the app subnet included in the container egress VCN 1568. The container egress VCN may include a NAT gateway 1538 that may be communicatively connected to the public internet 1554 (e.g., public internet 1254 in FIG. 24 ).

[0242] The internet gateway 1534 included in the control plane VCN 1516 and the internet gateway 1534 included in the data plane VCN 1518 may be communicatively connected to a metadata management service 1552 (e.g., metadata management system 1252 of FIG. 12 ), which may be communicatively connected to the public internet 1554. The public internet 1554 may be communicatively connected to a NAT gateway 1538 included in the control plane VCN 1516 and the NAT gateway 1538 included in the data plane VCN 1518. The service gateway 1536 included in the control plane VCN 1516 and the service gateway 1536 included in the data plane VCN 1518 may be communicatively connected to cloud services 1556.

[0243] In some examples, the pattern illustrated by the architecture of block diagram 1500 in FIG. 15 may be considered an exception to the pattern illustrated by the architecture of block diagram 1400 in FIG. 14 and may be desirable for an IaaS provider's customers when the IaaS provider cannot communicate directly with the customers (e.g., in unconnected regions). The customers may have real-time access to each of the containers 1567(1)-(N) contained in each customer's VMs 1566(1)-(N). The containers 1567(1)-(N) may be configured to call each of the secondary VNICs 1572(1)-(N) contained in the app subnet 1526 of the data plane app tier 1546, which may be included in the container egress VCN 1568. The secondary VNICs 1572(1)-(N) may send the call to a NAT gateway 1538, which may send the call to the public Internet 1554. In this example, containers 1567(1)-(N) that a customer can access in real time may be isolated from control plane VCN 1516 and may be isolated from other entities included in data plane VCN 1518. Containers 1567(1)-(N) may also be isolated from resources of other customers.

[0244] In another example, a customer can invoke cloud service 1556 using containers 1567(1)-(N). In this example, the customer can execute code in containers 1567(1)-(N) that requests a service from cloud service 1556. Containers 1567(1)-(N) can send the request to secondary VNICs 1572(1)-(N), which can send the request to a NAT gateway that can send the request to public internet 1554. Public internet 1554 can send the request to LB subnet 1522, which is included in control plane VCN 1516, via internet gateway 1534. In response to determining that the request is valid, LB subnet 1526 can send the request to app subnet 1526, which can send the request to cloud service 1556 via service gateway 1536.

[0245] It should be noted that the illustrated IaaS architectures 1200, 1300, 1400, and 1500 may include elements other than those shown. Furthermore, the illustrated embodiments are only examples of some of the cloud infrastructure systems that may incorporate embodiments of the present disclosure. In other embodiments, the IaaS system may have more or fewer elements than those shown, may combine two or more elements, or may have a different configuration or arrangement of elements.

[0246] In certain embodiments, the IaaS system described in this disclosure may include a suite of application, middleware, and database services that are delivered to customers in a self-service, subscription-based, elastically scalable, reliable, highly available, and secure manner. One example of such an IaaS system is the Oracle® Cloud Infrastructure (OCI) offered by the present applicant.

[0247] 16 illustrates an exemplary computer system 1600 upon which various embodiments may be implemented. System 1600 may be used to implement any of the computer systems described above. As shown, computer system 1600 includes a processing unit 1604 that communicates with a number of peripheral subsystems via a bus subsystem 1602. These peripheral subsystems may include a processing acceleration unit 1606, an I / O subsystem 1608, a storage subsystem 1618, and a communication subsystem 1624. Storage subsystem 1618 includes a tangible computer-readable storage medium 1622 and a system memory 1610.

[0248] The bus subsystem 1602 connects the various components of the computer system 1600 and The bus subsystem 1602 provides a mechanism for allowing the subsystems to communicate with each other as intended. While the bus subsystem 1602 is shown schematically as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses. The bus subsystem 1602 may be any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. For example, such architectures may include an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA), an Bus Extended ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus, which may be implemented as a mezzanine bus manufactured in accordance with the IEEE P1386.1 standard.

[0249] Processing unit 1604, which may be implemented as one or more integrated circuits (e.g., conventional microprocessors or microcontrollers), controls the operation of computer system 1600. Processing unit 1604 may include one or more processors. These processors may include single-core or multi-core processors. In some embodiments, processing unit 1604 may be implemented as one or more independent processing units 1632 and / or 1634, with each processing unit including a single-core or multi-core processor. In other embodiments, processing unit 1604 may be implemented as a quad-core processing unit formed by integrating two dual-core processors into a single chip.

[0250] In various embodiments, processing unit 1604 may execute various programs in response to program code and may maintain multiple programs or processes executing simultaneously. At any given time, some or all of the program code being executed may reside in processor 1604 and / or storage subsystem 1618. Processor 1604, through appropriate programming, may provide the various functionality discussed above. Computer system 1600 may further include a processing acceleration unit 1606, which may include a digital signal processor (DSP), a special purpose processor, and / or the like.

[0251] The I / O subsystem 1608 can include user interface input devices and user interface output devices. User interface input devices can include a keyboard, a pointing device such as a mouse or trackball, a touchpad or touchscreen integrated into a display, a scroll wheel, a click wheel, dials, buttons, switches, keypads, a voice input device with a voice command recognition system, a microphone, and other types of input devices. User interface input devices can also include motion sensing and / or gesture recognition devices, such as a Microsoft Kinect® motion sensor, which can provide input via a natural user interface (NUI) that utilizes gestures and voice commands, such as a Microsoft Xbox® 360 game controller. The user interface input device can control and interact with a force device. The user interface input device can also include an eye gesture recognition device, such as a Google Glass® blink detector. The Google Glass® blink detector detects a user's eye activity (e.g., "blinks" when taking a photo and / or selecting a menu) and converts the eye activity into input for input into the input device (e.g., Google Glass®). Furthermore, the user interface input device may include a voice recognition detection device that enables a user to interact with a voice recognition system (e.g., Siri® Navigator) via voice commands.

[0252] The user interface input device may also include a three-dimensional (3D) mouse, a joystick, or audio / visual devices such as pointing sticks, gamepads, graphics tablets, speakers, digital cameras, digital video cameras, portable media players, webcams, image scanners, fingerprint scanners, barcode readers, 3D scanners, 3D printers, laser range finders, and eye-tracking devices. Additionally, user interface input devices may include medical imaging input devices such as computed tomography (CT) scanners, magnetic resonance imaging (MRI) scanners, ultrasound emission tomography (EMC) scanners, and medical ultrasound scanners. User interface input devices may also include audio input devices such as MIDI keyboards and electronic musical instruments.

[0253] User interface output devices may include non-visual displays such as a display subsystem, indicator lights, or audio output devices. The display subsystem may be, for example, a flat-panel device using a cathode ray tube (CRT), liquid crystal display (LCD), or plasma display, a projection device, or a touchscreen. In general, when the term "output device" is used, it is intended to include all possible types of devices and mechanisms for outputting information from computer system 1600 to a user or to another computer. For example, user interface output devices include, but are not limited to, various display devices that visually convey text, images, and audio / video information, such as monitors, printers, speakers, headphones, car navigation systems, plotters, voice output devices, and modems.

[0254] Computer system 1600 may include a storage subsystem 1618. Storage subsystem 1618 comprises software elements, which are illustratively located in system memory 1610. System memory 1610 may store program instructions loadable and executable by processing unit 1604, as well as data generated by the execution of these programs.

[0255] Depending on the configuration and type of computer system 1600, system memory 1610 may be volatile memory (e.g., random access memory (RAM)) and / or non-volatile memory (e.g., read-only memory (ROM), flash memory). Generally, RAM contains data and / or program modules that are immediately accessible to and / or currently being operated on and executed by processing unit 1604. In some implementations, system memory 1610 may be static random access memory (SRAM). In some implementations, the computer system 1600 may include a number of different types of memory, such as a basic input / output system (BIOS), which contains the basic routines that help to transfer information between elements within the computer system 1600, such as during start-up. Typically stored in ROM. By way of example and not limitation, system memory 1610 may include client applications, web browsers, mid-tier applications, relational database management systems (RDBMs), and the like. Also shown are application programs 1612, program data 1614, and an operating system 1616, which may include applications such as Microsoft Windows, Apple Macintosh, and the like. and / or various versions of the Linux operating system, various commercially available UNIX or UNIX-like operating systems (including various GNU / Linux operating systems, Google Chrome ( (registered trademark) OS, etc.), and / or iOS, Windows Phone, Android OS, BlackBerry ( Palm OS and Palm OS operating systems The software may include a mobile operating system.

[0256] Additionally, the storage subsystem 1618 may provide a tangible, computer-readable storage medium for storing the basic programming and data constructs that provide the functionality of some embodiments. Software (programs, code modules, instructions) that, when executed by a processor, provide the above-described functionality may be stored in the storage subsystem 1618. These software modules or instructions may be executed by the processing unit 1604. The storage subsystem 1618 may also provide a repository for storing data used in accordance with the present disclosure.

[0257] Storage subsystem 1610 may also include a computer-readable storage medium reader 1620 that may further be connected to a computer-readable storage medium 1622. Computer-readable storage medium 1622 may comprehensively represent remote, local, fixed, and / or removable storage devices, as well as storage media for temporarily and / or permanently containing, storing, transmitting, and retrieving computer-readable information together with, or in combination with, system memory 1610 as needed.

[0258] Additionally, the computer-readable storage medium 1622 containing the code or portions of code may include any suitable medium known or used in the art, including, but not limited to, storage and communication media, such as volatile and nonvolatile, removable and non-removable media, implemented in any method or technology for storing and / or transmitting information. This may include tangible computer-readable storage media, such as RAM, ROM, electronically erasable programmable ROM (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disk (DVD), or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage, or other magnetic storage, or other tangible computer-readable media. This may also include intangible computer-readable media, such as a data signal, data transmission, or other medium usable to transmit the desired information and accessible by computer system 1600.

[0259] By way of example, computer-readable storage medium 1622 may include a hard disk drive that reads from or writes to non-removable, non-volatile magnetic media, a magnetic disk drive that reads from or writes to a removable, non-volatile magnetic disk, and a CD The computer-readable storage medium 1622 may include an optical disk drive that reads from or writes to removable, non-volatile optical disks such as ROM, DVD and Blu-ray disks or other optical media. The computer-readable storage medium 1622 may include, but is not limited to, Zip drives, flash memory cards, universal serial bus (USB) flash drives, secure digital (SD) cards, DVD disks, digital video tapes, etc. The computer-readable storage medium 1622 may also include flash memory-based SSDs, enterprise flash drives, solid-state drives (SSDs) based on non-volatile memory such as solid-state ROM, solid-state RAM, dynamic RAM, etc. The disk drives and their associated computer-readable media may provide non-volatile storage of computer-readable instructions, data structures, program modules, and other data for the computer system 1600.

[0260] The communications subsystem 1624 provides an interface with other computer systems and networks. The communications subsystem 1624 serves as an interface for receiving data from other systems and transmitting data from the computer system 1600 to other systems. For example, the communications subsystem 1624 may enable the computer system 1600 to connect to one or more devices via the Internet. In some embodiments, the communications subsystem 1624 may include a radio frequency (RF) transceiver component for accessing wireless voice and / or data networks (e.g., using cellular technologies, advanced data network technologies such as 3G, 4G, or EDGE (enhanced data rates for global evolution)), Wi-Fi (IEEE 802.11a), or other wireless technologies. .11 family standards or other mobile communications technologies or any combination thereof), a global positioning system (GPS) receiver component, and / or In some embodiments, the communications subsystem 1624 may provide a wired network connection (e.g., Ethernet) in addition to or instead of a wireless interface.

[0261] Additionally, in some embodiments, the communications subsystem 1624 may receive incoming communications in the form of structured and / or unstructured data feeds 1626, event streams 1628, event updates 1630, etc., on behalf of one or more users who may be using the computer system 1600.

[0262] As an example, the communications subsystem 1624 may provide Twitter feeds, Facebook updates, Rich Site Summary (RSS) feeds, and other similar services. The system may be configured to receive data feeds 1626, such as web feeds, in real time from users of social networks and / or other communication services, and / or to receive real-time updates from one or more third-party sources.

[0263] The communications subsystem 1624 may also be configured to receive data in the form of a continuous data stream, which may include an event stream 1628 of real-time events that may be continuous or may be essentially unbounded with no clear ends and / or event updates 1630. Examples of applications that generate continuous data may include, for example, sensor data applications, financial tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automobile traffic monitoring, etc.

[0264] The communications subsystem 1624 may also be configured to output structured and / or unstructured data feeds 1626, event streams 1628, event updates 1630, etc. to one or more databases that may communicate with one or more streaming data source computers connected to the computer system 1600.

[0265] The computer system 1600 may be one of a variety of types, including a handheld mobile device (e.g., an iPhone® mobile phone, an iPad® computing tablet, a PDA), a wearable device (e.g., a Google Glass® head-mounted display), a PC, a workstation, a mainframe, a kiosk, a server rack, or other data processing system.

[0266] Due to the ever-evolving nature of computers and networks, the description of the illustrated computer system 1600 is intended only as a specific example. Many other configurations are possible, having more or fewer components than the illustrated system. For example, customized hardware may also be used and / or particular elements may be implemented in hardware, firmware, software (including applets), or a combination. Additionally, connections to other computing devices, such as network input / output devices, may be utilized. Based on the disclosure and teachings provided in this disclosure, one of ordinary skill in the art will appreciate other means and / or methods for implementing various embodiments.

[0267] While specific embodiments of the present disclosure have been described, various modifications, variations, alternative configurations, and equivalents are encompassed within the scope of the present disclosure. The embodiments of the present disclosure are not limited to operating in a particular data processing environment, but can freely operate in multiple data processing environments. Furthermore, while embodiments of the present disclosure have been described using a particular series of actions and steps, it will be apparent to those skilled in the art that the scope of the present disclosure is not limited to the series of actions and steps described. Various features and aspects of the above-described embodiments can be used individually or jointly.

[0268] Furthermore, while embodiments of the present disclosure have been described using a particular combination of hardware and software, it should be appreciated that other combinations of hardware and software are within the scope of the present disclosure. Embodiments of the present disclosure may be implemented using only hardware, only software, or a combination thereof. The various processes described in this disclosure may run on the same processor or any combination of different processors. Thus, when a component or module is described as being configured to perform a particular process, that configuration may be achieved, for example, by designing electronic circuitry to perform the process, by programming a programmable electronic circuit (such as a microprocessor) to perform the process, or a combination thereof. Processes may communicate using a variety of techniques, including, but not limited to, conventional techniques for inter-process communication. Different pairs of processes may use different techniques, or the same pair of processes may use different techniques at different times.

[0269] Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense. It will be apparent, however, that additions, subtractions, deletions, and other modifications and changes may be made without departing from the broad spirit and scope defined by the appended claims. Accordingly, while specific embodiments of the present disclosure have been described, these embodiments are not intended to be limiting. Various modifications and equivalents thereof are intended to be encompassed within the scope of the appended claims.

[0270] The indefinite article "a" / used in the context of describing this disclosure (particularly in the context of the claims) "An," the definite article "the," and similar references are used in this disclosure unless otherwise stated or in context. Unless clearly indicated otherwise, the terms "comprising," "having," "including," and "including" are to be construed as including both the singular and the plural. and "containing" should be construed as open-ended terms (i.e., meaning "including, but not limited to") unless otherwise specified. The term "connected" should be construed as partly or wholly contained within, attached to, or joined together, even if there is something intervening. In this disclosure, the recitation of ranges of values ​​is intended merely as a shorthand method of referring to each individual value contained within the range, and unless otherwise specified in this disclosure, each individual value is incorporated into this disclosure as if set forth individually in this disclosure. Unless otherwise specified in this disclosure or unless the context clearly indicates otherwise, all methods described in this disclosure can be performed in any suitable order. In this disclosure, the use of any and all examples or exemplary language (e.g., "such as") is intended to further clarify embodiments of the disclosure, Unless otherwise specified, no limitation on the scope of the disclosure is intended. No language in the specification should be construed as indicating any non-claimed element essential to the practice of the disclosure.

[0271] Disjunctive language, such as the phrase "at least one of X, Y, or Z," is intended to be understood in context as generally used to indicate that an item, term, etc. may be either X, Y, or Z, or any combination thereof (e.g., X, Y, and / or Z), unless otherwise specified. Thus, such disjunctive language is not generally intended to, and does not imply, that a particular embodiment requires that at least one of X, at least one of Y, or at least one of Z be present.

[0272] Preferred embodiments of the present disclosure are described herein, including the best mode known for carrying out the disclosure. Variations of these preferred embodiments will become apparent to those skilled in the art upon reading the foregoing description. Those skilled in the art can employ such variations as appropriate, and the present disclosure may be practiced otherwise than as specifically described herein. Accordingly, this disclosure includes all modifications and equivalents of the subject matter recited in the claims appended hereto as permitted by applicable law. Moreover, any combination of the above-described elements in all possible variations thereof is encompassed by this disclosure unless otherwise indicated herein.

[0273] All references cited herein, including publications, patent applications, and patents, are incorporated by reference to the same extent as if each reference were individually and specifically indicated to be incorporated by reference and were set forth in its entirety herein. In the foregoing specification, aspects of the present disclosure have been described with reference to specific embodiments thereof, but those skilled in the art will recognize that the disclosure is not limited thereto. Various features and aspects of the above-described disclosure may be used individually or jointly. Moreover, embodiments can be utilized in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of the specification. Accordingly, the specification and drawings should be regarded as illustrative rather than restrictive.< / realm>

Claims

1. 1. A method comprising: determining, by a primary host machine, a change in state information for a communication channel having a first endpoint in a customer on-premises network and a second endpoint on a primary host machine in a cloud service provider infrastructure; the primary host machine identifying a backup host machine for the communication channel; The method includes the primary host machine replicating the changes to the state information to the backup host machine, and when the backup host machine becomes the second endpoint of the communication channel through a failover, the state information replicated and stored on the backup host machine is usable by the backup host machine.

2. replicating the changes in the state information of the communication channel to the backup host machine; determining whether the replication of the change in the state information is performed in a safe mode by the primary host machine, the change in the state information being identified by analyzing packets received by the primary host machine; If it is determined that the replication of the changes to the state information is performed in the safe mode, the primary host machine suspends processing of the packet; the primary host machine communicating the change in the state information to the backup host machine, the backup host machine being a successor host machine to the primary host machine in a replication chain; the primary host machine receives an acknowledgment indicating that the backup host machine has replicated the state information; 2. The method of claim 1, further comprising: in response to receiving the acknowledgement, the primary host machine resuming processing of the packet.

3. If it is determined that the replication of the changes to the state information is not performed in the safe mode, the primary host machine communicating the changes in the state information to the backup host machine; The method of claim 2 further comprising the primary host machine processing the packet.

4. 2. The method of claim 1, wherein the state information for the communication channel includes at least one of a sequence number, cryptographic state information, Internet Key Exchange (IKE) state information, and Border Gateway Protocol (BGP) state information.

5. 3. The method of claim 2, wherein determining that the replication of the changes to the state information is being performed in the safe mode further comprises the primary host machine determining that downtime for the primary host machine is scheduled within a threshold period of time.

6. Determining the change in the state information of the communication channel comprises: the primary host machine detecting an event that triggers the change in the state information of the communication channel; 2. The method of claim 1, further comprising: the primary host machine determining the change in the state information by analyzing a packet associated with the event.

7. The event that triggers the change of the state information is receiving a packet by the primary host machine; receiving a change in encryption or decryption information associated with said communication channel connection; and receiving a change in Border Gateway Protocol (BGP) state information associated with the connection.

8. Identifying the backup host machine for the communication channel includes: the primary host machine identifying the replication chain for the communication channel by querying a disk backup storage server; the primary host machine determining the backup host machine for the communication channel in the replication chain; the primary host machine is the head of the replication chain; The method of claim 2 , wherein the backup host machine is a successor host machine to the primary host machine in the replication chain.

9. Analyzing packets received by the primary host machine includes: the primary host machine generating a hash result by hashing a portion of the packet; The method of claim 2 , further comprising: the primary host machine determining the state information for the communication channel based on the hash result.

10. The method of claim 1 , wherein the communication channel is an Internet Protocol Security (IPSec) tunnel.

11. A non-transitory computer-readable storage medium storing computer-executable instructions that, when executed, cause a processor of a computer system to perform a method, comprising: The method comprises: determining, by a primary host machine, a change in state information for a communication channel having a first endpoint in a customer on-premises network and a second endpoint on a primary host machine in a cloud service provider infrastructure; the primary host machine identifying a backup host machine for the communication channel; a non-transitory computer-readable storage medium, the non-transitory computer-readable storage medium including the primary host machine replicating the changes to the state information to the backup host machine, wherein when the backup host machine becomes the second endpoint of the communication channel through a failover, the state information replicated and stored on the backup host machine is usable by the backup host machine.

12. replicating the changes in the state information of the communication channel to the backup host machine; determining whether the replication of the change in the state information is performed in a safe mode by the primary host machine, the change in the state information being identified by analyzing packets received by the primary host machine; If it is determined that the replication of the changes to the state information is performed in safe mode, If, the primary host machine suspends processing of the packet; the primary host machine communicating the change in the state information to the backup host machine, the backup host machine being a successor host machine to the primary host machine in a replication chain; the primary host machine receives an acknowledgment indicating that the backup host machine has replicated the state information; 12. The non-transitory computer-readable storage medium of claim 11, further comprising: in response to receiving the acknowledgment, the primary host machine resuming processing of the packet.

13. If it is determined that the replication of the changes to the state information is not performed in the safe mode, the primary host machine communicating the changes in the state information to the backup host machine; The non-transitory computer-readable storage medium of claim 12 , further comprising the primary host machine processing the packet.

14. 12. The non-transitory computer-readable storage medium of claim 11, wherein the state information for the communication channel includes at least one of a sequence number, cryptographic state information, Internet Key Exchange (IKE) state information, and Border Gateway Protocol (BGP) state information.

15. 13. The non-transitory computer-readable storage medium of claim 12, wherein determining that the replication of the changes to the state information is being performed in the safe mode further comprises the primary host machine determining that downtime for the primary host machine is scheduled within a threshold period of time.

16. Determining the change in the state information of the communication channel comprises: the primary host machine detecting an event that triggers the change in the state information of the communication channel; and the primary host machine determining the change in the state information by analyzing a packet associated with the event.

17. The event that triggers the change of the state information is receiving a packet by the primary host machine; receiving a change in encryption or decryption information associated with said communication channel connection; and receiving a change in Border Gateway Protocol (BGP) state information associated with the connection.

18. Identifying the backup host machine for the communication channel includes: the primary host machine identifying the replication chain for the communication channel by querying a disk backup storage server; the primary host machine determining the backup host machine for the communication channel in the replication chain; the primary host machine is the head of the replication chain; The backup host machine is the primary host machine in the replication chain.

13. The non-transitory computer-readable storage medium of claim 12, wherein the host machine is a successor of the host machine.

19. Analyzing packets received by the primary host machine includes: the primary host machine generating a hash result by hashing a portion of the packet; and the primary host machine determining the state information for the communication channel based on the hash result.

20. 12. The non-transitory computer-readable storage medium of claim 11, wherein the communication channel is an Internet Protocol Security (IPSec) tunnel.

Images