Connectivity for Virtual Private Label Clouds

Abstract

A technique for facilitating connectivity to a vPLC created within a CSP-provided infrastructure within a region. When a packet's destination is determined to be an endpoint associated with a particular vPLC within the CSP-provided infrastructure within a region, the packet is tagged with information related to the particular vPLC. The vPLC-related information for the particular vPLC may include, for example, a vPLC identifier that identifies the particular vPLC, an identifier that identifies a customer associated with the endpoint, a virtual cloud network identifier that identifies a virtual cloud network (VCN) that belongs to the particular vPLC (where the endpoint is part of the VCN), and other vPLC-related information. The packet is then routed or communicated within the CSP-provided infrastructure within the region with the tagged vPLC-related information. The vPLC-related information is used to route the packet within the CSP-provided infrastructure within the region as part of the connectivity.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a nonprovisional application of U.S. Provisional Patent Application No. 63 / 407,571, filed September 16, 2022, entitled "VIRTUAL PRIVATE LABEL CLOUD," and claims the benefit of and priority under 35 U.S.C. § 119(e), which is incorporated herein by reference in its entirety for all purposes.

[0002] This disclosure is also related to the following patent applications, the entire contents of each of which are incorporated herein by reference for all purposes: (1) Non-provisional application Ser. No. 18 / 368,881 (Attorney Docket No. 088325-1311130 (344800US)), entitled "Virtual Private Label Cloud," filed concurrently with this application; (2) Non-provisional application Ser. No. 18 / 368,863 (Attorney Docket No. 088325-1311132 (344900US)), entitled "CONSOLE CUSTOMIZATION FOR VIRTUAL PRIVATE LABEL CLOUDS," filed concurrently herewith; (3) Non-provisional application Ser. No. 18 / 368,870 (Attorney Docket No. 088325-1311140 (US 345000)), entitled "RESOURCE ALLOCATION FOR VIRTUAL PRIVATE LABEL CLOUDS," filed concurrently herewith; (4) Non-provisional application Ser. No. 18 / 368,877, entitled "ENDPOINTS FOR VIRTUAL PRIVATE LABEL CLOUDS," filed concurrently herewith (Attorney Docket No. 088325-1311138 (345100US)); (5) Non-provisional application Ser. No. 18 / 368,884, entitled "IDENTITY MANAGEMENT FOR VIRTUAL PRIVATE LABEL CLOUDS," filed concurrently herewith (Attorney Docket No. 088325-1311139 (345200US)); (6) Non-provisional application Ser. No. 18 / 468,024 (Attorney Docket No. 088325-1311136 (345300US)), entitled "METADATA CUSTOMIZATION FOR VIRTUAL PRIVATE LABEL CLOUDS," filed concurrently herewith; (7) Non-provisional application Ser. No. 18 / 468,037 (Attorney Docket No. 088325-1311141 (US 345400)), entitled "REMOTE DATA PLANES FOR VIRTUAL PRIVATE LABEL CLOUDS," filed concurrently herewith; (8) Non-provisional application Ser. No. 18 / 468,047 (Attorney Docket No. 088325-1315390 (346900US)), entitled "RESOURCE USAGE MONITORING, BILLING AND ENFORCEMENT FOR VIRTUAL PRIVATE LABEL CLOUDS," filed concurrently herewith; and (9) Non-provisional application Ser. No. 18 / 468,044 (Attorney Docket No. 088325-1315384 (346800US)), entitled "CLOUD INFRASTRUCTURE-BASED ONLINE PUBLISHING PLATFORMS FOR VIRTUAL PRIVATE LABEL CLOUDS," filed concurrently herewith.

[0003] Field The present disclosure generally relates to techniques for providing cloud services. More particularly, novel techniques are disclosed that enable cloud infrastructure in a region provided by a cloud service provider (CSP) to be used to create one or more virtual private clouds (referred to herein as virtual private label clouds or "vPLCs") for providing cloud services supplied by the CSP to customers of the CSP. A vPLC can be created for a reseller and used to provide one or more reseller-supplied cloud services to the reseller's customers. [Background technology]

[0004] background In recent years, cloud services offered by cloud service providers (CSPs) have become increasingly popular. Cloud services can include various types of services, including software as a service (SaaS), platform as a service (PaaS), infrastructure as a service (IaaS), database as a service (DBaaS), and so on. Consumers receive certain benefits from using cloud services. Here, consumers can be individuals or businesses. Unlike in the past, consumers no longer need to invest in procuring and maintaining the hardware and software resources associated with an enterprise installation. CSPs are responsible for providing, maintaining, and updating the infrastructure used to deliver cloud services to one or more of their customers. CSPs are also responsible for managing the security associated with the infrastructure. Consumers can also scale their use of services based on their needs. Additionally, consumers can access their subscribed services and their cloud-deployed workloads or data from anywhere with an Internet connection. This "anywhere access" feature enables better collaboration between teams within a company. All of these and other factors provide consumers who use cloud services with a significant competitive advantage. For example, consumers who use infrastructure-as-a-service (IaaS) services offered by CSPs can build applications faster and more efficiently and bring them to market faster than their competitors. As a result, demand for cloud services is growing significantly.

[0005] Due to the growing demand for cloud services, more and more business entities and enterprises want to become CSPs and provide cloud services to their customers. However, the barriers to becoming a CSP are very high. Becoming a CSP requires very high upfront and ongoing capital expenditures to procure, maintain, and update the infrastructure necessary to provide cloud services. The expertise to provide cloud services (e.g., to configure the infrastructure to provide the services, to ensure the security of the infrastructure, etc.) is also very limited. As a result, there are only a few companies in the CSP industry. Summary of the Invention [Means for solving the problem]

[0006] A brief overview

[0001] The present disclosure generally relates to techniques for providing cloud services. More particularly, novel techniques are disclosed that enable cloud infrastructure in a region provided by a cloud service provider (CSP) to be used to create one or more virtual private clouds (referred to herein as virtual private label clouds or "vPLCs") for providing cloud services supplied by the CSP to customers of the CSP. Various embodiments are described herein, including methods, systems, non-transitory computer-readable storage media storing programs, code, or instructions executable by one or more processors, and the like.

[0007] A vPLC created in accordance with the various techniques described herein can be used for a variety of purposes. For example, in one use case, a vPLC may be created for a reseller who is a customer of a CSP and used by the reseller to provide one or more reseller-supplied cloud services to the reseller's customers. As another use case, a vPLC may be used as a virtual data center and associated with a realm different from the realm associated with cloud infrastructure in a region provided by the CSP.

[0008] In certain embodiments, a vPLC represents a virtual cloud that includes a set(s) of resources that have been allocated to the vPLC from a CSP-provided infrastructure within a region. One or more vPLCs can be created using the CSP-provided infrastructure within a region.

[0009] In certain embodiments, regional infrastructure provided by a CSP in a region can be used both to provide cloud services to the CSP's customers and to provide reseller-supplied cloud services to customers of resellers who are customers of the CSP. This is achieved by creating one or more vPLCs for one or more resellers from the CSP-provided regional infrastructure, where a vPLC created for a particular reseller can be used to provide reseller-supplied and reseller-branded cloud services to that particular reseller's customers. This allows the reseller to become a provider of reseller-branded cloud services to its customers without having to invest in the infrastructure required to provide those cloud services. Instead, the reseller uses the infrastructure provided by the CSP to provide reseller-branded cloud services to the reseller's customers. In this manner, a first portion of the infrastructure provided by the CSP within the region may be used to provide CSP-supplied cloud services to customers of the CSP, a second portion of the infrastructure provided by the CSP within the region and allocated to a first vPLC created for a first reseller may be used to provide first reseller-supplied cloud services to customers of the first reseller, a third portion of the infrastructure provided by the CSP within the region and allocated to a second vPLC created for a second reseller may be used to provide second reseller-supplied cloud services to customers of the second reseller, and so on.

[0010] The present disclosure relates to facilitating connectivity to vPLCs created within a CSP-provided infrastructure within a region. When a packet is destined for an endpoint within the CSP-provided infrastructure within a region that is associated with a particular vPLC, the packet is tagged with information related to that particular vPLC. The vPLC-related information for that particular vPLC may include, for example, a vPLC identifier that identifies that particular vPLC, an identifier that identifies a customer associated with the endpoint, a virtual cloud network identifier that identifies a virtual cloud network (VCN) that belongs to that particular vPLC (where the endpoint is part of the VCN), and other vPLC-related information.

[0011] The packets are then routed or communicated within the CSP-provided infrastructure within the region with the tagged vPLC-related information. vPLC-related traffic is tagged with the vPLC-related information to clearly identify the traffic as related to the vPLC. In this manner, the vPLC-related information (e.g., vPLC ID) is used to route packets within the CSP-provided infrastructure within the region as part of connectivity.

[0012] Within a compute instance, one or more vPLCs may be created within a CSP-provided infrastructure within a region. The CSP-provided infrastructure within a region may include multiple physical and logical components. A first component within the CSP-provided infrastructure within the region may receive a packet. The first component may determine that the destination of the packet is an endpoint associated with a first virtual private label cloud (vPLC), the first vPLC being created for a first reseller using one or more resources from a cloud service provider (CSP)-provided infrastructure within the region, the first vPLC being created to provide a set of cloud services provided by the one or more first resellers to one or more customers of the first reseller. The first component may tag the received packet with the first vPLC-related information to create a tagged packet. The tagged packet with the first vPLC-related information may then be communicated from the first component to a second component within the CSP-provided infrastructure within the region, the second component being associated with the endpoint. The first vPLC-related information may include a first vPLC identifier that identifies the first vPLC. The first vPLC-related information may include an identifier that identifies a virtual cloud network (VCN) associated with the first vPLC, where the endpoint is part of the VCN.

[0013] In addition to including a first vPLC for providing services supplied by the first reseller to one or more customers of the first reseller, a first portion of the CSP-provided infrastructure in the region may be for providing one or more CSP-supplied cloud services to one or more customers of the CSP. Additional vPLCs may be created in the CSP-provided infrastructure in the region, each vPLC created for a corresponding reseller and used to provide reseller-supplied cloud services to the reseller's customers.

[0014] Various techniques may be used by the first component to tag the packet with the vPLC-related information. In certain embodiments, the tagging includes: creating an encapsulated packet by adding an encapsulation header to the packet, by the first component; and including, by the first component, the first vPLC-related information in the encapsulation header.

[0015] In certain embodiments, the processing performed by the first component to determine that the packet is destined for an endpoint associated with the first vPLC can include determining a destination address included in a header of the packet, the destination address being associated with the endpoint. The first component can then determine that the destination address falls within an address range allocated to the first vPLC.

[0016] In certain embodiments, first information may be stored for a set of vPLCs, including a first vPLC created using the CSP-provided infrastructure in a region. This first information may then be used to determine that a packet's destination belongs to the first vPLC. The first information may include, for each vPLC in the set of vPLCs, information identifying an address range allocated to the vPLC. The first component may use the first information to identify a particular address range within which the destination address falls and use the first information to determine that the particular address range is allocated to the first vPLC.

[0017] In certain embodiments, the first component can determine that the endpoint is associated with a first customer of the first reseller. The first vPLC-related information tagged to the packet can include an identifier that identifies the first customer. For example, the first component can determine a destination address included in a header of the packet, the destination address being associated with the endpoint. The first component can then determine that the destination address falls within an address subrange allocated to the first customer of the first reseller.

[0018] In some embodiments, first information may be stored for a set of vPLCs created using the CSP-provided infrastructure in a region, the set of vPLCs including a first vPLC. The first information may include, for each vPLC in the set of vPLCs, information identifying an address range allocated to the vPLC. The first information may further include, for each of one or more customers of the first reseller, an address subrange allocated to the customer from the address range allocated to the first vPLC. The first component may use the first information to identify a particular address subrange within which the destination address falls and use the first information to determine that the particular address subrange is allocated to the first customer.

[0019] The first component and the second component may refer to various different components within the CSP-provided infrastructure within the region. For example, in one use case, the endpoint is a destination compute instance within a first vPLC, the first component is a gateway within the CSP-provided infrastructure within the region, and the second component is a network virtualization device (NVD) that implements a virtual network interface controller (VNIC) associated with the destination compute instance. The gateway can receive packets from a source outside the CSP-provided infrastructure within the region.

[0020] As another example, the packet originates from a source compute instance in a CSP-provided infrastructure within a region, the endpoint is a destination compute instance associated with a first vPLC, the first component is an NVD implementing a VNIC associated with the source compute instance, and the second component is an NVD implementing a VNIC associated with the destination compute instance. In one scenario, the source compute instance may be associated with a second vPLC created for a second reseller using one or more resources from a cloud service provider (CSP)-provided infrastructure within the region, the second vPLC being created to provide a set of cloud services provided by one or more second resellers to one or more customers of the second reseller. In another scenario, the source compute instance may be associated with a first customer of the first reseller, and the destination compute instance may be associated with a second customer of the first reseller, where the second customer is different from the first customer. As another example, the source computing instance may be associated with a first customer of a first reseller, and the destination computing instance is associated with a first customer of the first reseller. In certain embodiments, the first component may be a component dedicated to handling traffic for the first vPLC.

[0021] A vPLC can be used for a variety of other purposes that do not involve a reseller. As an example, a vPLC can be created and associated with a realm that is different from the realm associated with the CSP-provided regional infrastructure used to create the vPLC. For example, the CSP-provided regional infrastructure may be associated with a first realm, and the vPLC may be associated with a second realm that is different from the first realm. In such a scenario, the infrastructure corresponding to the vPLC is hosted by the first realm but is considered to belong to the second realm. This can be used for a variety of different purposes. For example, because the vPLC is associated with the second realm, it can communicate with other infrastructure (e.g., other data centers) in the second realm. Thus, data centers in the second realm can communicate with the vPLC because both the data centers and the vPLC are in the same realm and share the trust and identity profile of the second realm.

[0022] In various embodiments, a system is provided that includes one or more data processors and a non-transitory computer-readable medium that contains instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more of the methods disclosed herein.

[0023] In various embodiments, a non-transitory computer-readable medium storing computer-executable instructions that, when executed by one or more processors, cause one or more processors of a computer system to perform one or more methods disclosed herein.

[0024] In various embodiments, a computer program product is disclosed herein that includes a computer program / instructions that, when executed by a processor, cause the processor to perform any of the methods described above. Embodiments can be implemented by using a computer program product that includes a computer program / instructions that, when executed by a processor, cause the processor to perform any of the methods described in the present disclosure.

[0025] The techniques described above and below can be implemented in multiple ways and in multiple contexts. Some exemplary implementations and contexts are provided with reference to the accompanying drawings, as described in more detail below. However, the following implementations and contexts are only a few of many. [Brief explanation of the drawings]

[0026] [Figure 1] FIG. 1 is a high-level diagram of a distributed environment illustrating a virtual or overlay cloud network hosted by a cloud service provider infrastructure according to an embodiment. [Figure 2] FIG. 2 is a simplified architectural diagram of physical components within a physical network within CSPI according to one embodiment. [Figure 3] FIG. 1 illustrates an exemplary configuration within CSPI in which a host machine is connected to multiple network virtualization devices (NVDs), according to one embodiment. [Figure 4] A diagram illustrating connectivity between host machines and NVDs to provide I / O virtualization to support multi-tenancy functionality in one embodiment. [Figure 5] FIG. 2 is a simplified block diagram of a physical network provided by CSPI according to one embodiment. [Figure 6]1 is a block diagram of a distributed environment illustrating an example of a virtual private label cloud (vPLC) hosted by a CSP-provided infrastructure within a region, according to an embodiment. [Figure 7] FIG. 1 illustrates relationships between a CSP, the CSP's direct and reseller customers and their associated users, and individual reseller customers and their associated users, according to an embodiment. [Figure 8] FIG. 1 illustrates the relationship between resources allocated to tenancies for a CSP's direct customers, tenancies for the CSP's resellers, and tenancies for the individual resellers' customers, according to an embodiment. [Figure 9A] FIG. 1 illustrates a mechanism for storing information within a tenancy for a customer of an individual reseller of a CSP, according to an embodiment. [Figure 9B] FIG. 1 illustrates a mechanism for storing information within a tenancy for a customer of an individual reseller of a CSP, according to an embodiment. [Figure 10] FIG. 1 is a flow diagram illustrating an example of a vPLC setup process for a CSP reseller, according to an embodiment. [Figure 11] 1 is a block diagram of a distributed environment illustrating an example of a virtual private label cloud (vPLC) hosted by a CSP-provided infrastructure within a region, according to an embodiment. [Figure 12] FIG. 1 is a block diagram illustrating a vPLC segmentation model for resource partitioning, according to an embodiment. [Figure 13] FIG. 1 is a block diagram illustrating a vPLC nested model for resource partitioning, according to an embodiment. [Figure 14] FIG. 1 is a flow diagram illustrating a sign-up process for customers of a reseller associated with a vPLC, according to an embodiment. [Figure 15] FIG. 10 is a flow diagram illustrating a login process for a user associated with a customer of a reseller associated with a vPLC, according to an embodiment. [Figure 16] FIG. 1 is a flow diagram illustrating a resource allocation and provisioning process for a customer of a reseller associated with a vPLC, according to an embodiment. [Figure 17] FIG. 1 is a flow diagram illustrating a resource allocation and provisioning process for a customer of a reseller associated with a vPLC, according to an embodiment. [Figure 18] FIG. 1 is a simplified diagram illustrating a use case in which a vPLC is hosted by infrastructure in a first realm but associated with a second, different realm, according to an embodiment. [Figure 19] FIG. 1 is a flow diagram illustrating an exemplary method for creating a vPLC, where the vPLC is hosted by a CSP-provided infrastructure in a particular region in a particular realm, but is virtually associated with different regions in different realms, according to an embodiment. [Figure 20] FIG. 1 is a simplified block diagram of a distributed environment including a CSP-provided infrastructure in a region including one or more vPLCs, according to an embodiment. [Figure 21] FIG. 10 is an exemplary flow diagram depicting a method for routing packets originating from a source outside a regional CSP-provided infrastructure and destined for an endpoint belonging to a vPLC, according to an embodiment. [Figure 22] FIG. 10 is an exemplary flow diagram depicting a method for routing packets originating from within a vPLC within a source outside a regional CSP-provided infrastructure and destined for an endpoint outside the CSP-provided infrastructure within a region, according to an embodiment. [Figure 23] FIG. 1 is an exemplary flow diagram depicting a method for routing packets between vPLCs in a CSP-provided infrastructure within a region, according to an embodiment. [Figure 24]1 is an exemplary flow diagram depicting a method for routing packets between compute instances within the same vPLC within a CSP-provided infrastructure within a region (e.g., intra-vPLC communication), according to an embodiment. [Figure 25] 25 is an exemplary flow diagram 2500 depicting a generalized method for tagging packets destined for a vPLC in a CSP-provided infrastructure in regional routing, according to an embodiment. [Figure 26] FIG. 1 is a block diagram illustrating one pattern for implementing a cloud infrastructure as a service system, according to at least one embodiment. [Figure 27] FIG. 1 is a block diagram illustrating another pattern for implementing a cloud infrastructure as a service system, according to at least one embodiment. [Figure 28] FIG. 1 is a block diagram illustrating another pattern for implementing a cloud infrastructure as a service system, according to at least one embodiment. [Figure 29] FIG. 1 is a block diagram illustrating another pattern for implementing a cloud infrastructure as a service system, according to at least one embodiment. [Figure 30] FIG. 1 is a block diagram illustrating an exemplary computer system according to at least one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0027] Detailed Description In the following description, for purposes of explanation, specific details are set forth in order to provide a thorough understanding of aspects of certain embodiments. It will be understood, however, that various embodiments may be practiced without these specific details. The drawings and descriptions are not intended to be limiting. The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment or design described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments or designs.

[0028]

[0001] The present disclosure generally relates to techniques for providing cloud services. More particularly, novel techniques are disclosed that enable cloud infrastructure in a region provided by a cloud service provider (CSP) to be used to create one or more virtual private clouds (referred to herein as virtual private label clouds or "vPLCs") for providing cloud services supplied by the CSP to customers of the CSP. Various embodiments are described herein, including methods, systems, non-transitory computer-readable storage media storing programs, code, or instructions executable by one or more processors, and the like.

[0029] A vPLC created in accordance with the various techniques described herein can be used for a variety of purposes. For example, in one use case, a vPLC may be created for a reseller who is a customer of a CSP and used by the reseller to provide one or more reseller-supplied cloud services to the reseller's customers. As another use case, a vPLC may be used as a virtual data center and associated with a realm different from the realm associated with cloud infrastructure in a region provided by the CSP.

[0030] In certain embodiments, a vPLC represents a virtual cloud that includes a set(s) of resources that are allocated to the vPLC from a CSP-provided infrastructure within a region. One or more vPLCs can be created using the CSP-provided infrastructure within a region. In some embodiments, a virtual private label cloud (vPLC) refers to a model and architecture provided by a CSP within a CSP-provided infrastructure within a region that enables and facilitates reseller-supplied and reseller-branded cloud services.

[0031] Just as a CSP-provided infrastructure in a region provides cloud resources that can be accessed by the CSP's customers, a vPLC created for a reseller provides a set of resources that can be accessed by the reseller's customers. From the perspective of the reseller's customers, a vPLC is like a reseller-provided infrastructure in a region that provides resources that can be accessed by the reseller's customers. A vPLC is similar to a data center in a region provided by a reseller that delivers reseller-supplied cloud services to the reseller's customers.

[0032] In certain embodiments, regional infrastructure provided by a CSP in a region can be used both to provide cloud services to the CSP's customers and to provide reseller-supplied cloud services to customers of resellers who are customers of the CSP. This is achieved by creating one or more vPLCs for one or more resellers from the CSP-provided regional infrastructure, where a vPLC created for a particular reseller can be used to provide reseller-supplied and reseller-branded cloud services to that particular reseller's customers. This allows the reseller to become a provider of reseller-branded cloud services to its customers without having to invest in the infrastructure required to provide those cloud services. Instead, the reseller uses the infrastructure provided by the CSP to provide reseller-branded cloud services to the reseller's customers.

[0033] In a reseller use case, a vPLC is created for a reseller using cloud infrastructure within a region provided by a cloud service provider (CSP). The vPLC created for the reseller can be used by the reseller to provide one or more reseller-supplied cloud services to the reseller's customers. In one embodiment, the CSP-provided regional infrastructure providing the services used by the reseller to provide one or more reseller-supplied services is partitioned into securely isolated portions, and the isolated portions (or partitions) are allocated to different resellers, such that the partitions allocated to one reseller are isolated from other resellers. In this manner, a first portion of the infrastructure provided by the CSP within a region may be used to provide CSP-supplied cloud services to the CSP's customers, a second portion of the infrastructure provided by the CSP within its region and allocated to a first vPLC created for a first reseller may be used to provide first reseller-supplied cloud services to the first reseller's customers, a third portion of the infrastructure provided by the CSP within its region and allocated to a second vPLC created for a second reseller may be used to provide second reseller-supplied cloud services to the second reseller's customers, and so on. Each portion of the infrastructure allocated to a reseller may also be further divided into a second level of securely isolated portions for providing multi-tenant cloud infrastructure services to the reseller's customers.

[0034] A reseller can be an entity such as a company or business entity or an individual. Using the techniques described herein enables a reseller entity to quickly become a cloud service provider while avoiding all of the barriers to entry discussed in the background section of this disclosure. For example, a reseller does not need to procure the infrastructure used to provide the cloud services because the offered cloud services use infrastructure provided by a CSP. To the reseller's customers, this appears as if the reseller is providing reseller-supplied cloud services. Thus, from the perspective of the reseller's customers, the reseller is a cloud service provider whose services the customers subscribe to. The reseller's customers may not even know or recognize the CSP whose infrastructure the reseller supplies and uses to provide the reseller's branded cloud services.

[0035] The ability to provide vPLC may be offered as a service by a CSP, to which one or more customers can subscribe. For example, an entity that wants to provide cloud services to its customers but does not want to invest in procuring the infrastructure required to provide the service can subscribe to this CSP-supplied vPLC service. Upon subscribing to the vPLC service, the entity is a customer of the CSP, but a “special” customer because the CSP-provided infrastructure allocated to the entity, in the form of a vPLC, is used to provide the entity-supplied and branded services to its customers. Thus, this customer is both a customer of the CSP and a reseller of cloud services that use the vPLC created for the entity. Therefore, this entity is referred to as a “reseller” to distinguish it from the CSP's actual direct customers. A CSP's direct customer is an entity that subscribes to and consumes one or more CSP-supplied and CSP-branded services, but does not use the CSP infrastructure to sell cloud services to its customers. For purposes of this disclosure, a CSP's direct customer is also referred to as a “non-reseller customer” to distinguish it from a reseller. With respect to the reseller, as part of the vPLC service, the CSP will provide infrastructure to the reseller in the form of a vPLC, which will be used by the reseller to deliver the reseller's branded cloud services to the reseller's customers.

[0036] A vPLC can be used for a variety of other purposes that do not involve a reseller. As an example, a vPLC can be created and associated with a realm that is different from the realm associated with the CSP-provided regional infrastructure used to create the vPLC. For example, the CSP-provided regional infrastructure may be associated with a first realm, and the vPLC may be associated with a second realm that is different from the first realm. In such a scenario, the infrastructure corresponding to the vPLC is hosted by the first realm but is considered to belong to the second realm. This can be used for a variety of different purposes. For example, because the vPLC is associated with the second realm, it can communicate with other infrastructure (e.g., other data centers) in the second realm. Thus, data centers in the second realm can communicate with the vPLC because both the data centers and the vPLC are in the same realm and share the trust and identity profile of the second realm.

[0037] Creating and managing a vPLC is performed using infrastructure and services provided by the CSP and is a technically very complex task. When a vPLC is created for a reseller, technical functionality is provided to enable the reseller to use the vPLC to offer reseller-supplied cloud services to the reseller's customers. This includes the ability to segregate traffic between resellers, segregate traffic of the reseller's different customers, dynamically manage the allocation of CSP-provided resources to the vPLC, manage the allocation of resources allocated to the vPLC among the vPLC's different customers, meter usage of the vPLC allocated to the reseller and perform related billing functions, meter usage of the vPLC allocated among the vPLC's different customers and perform related billing functions, enable a marketplace for different resellers, and perform identity management functions for resellers and their customers. This disclosure describes various embodiments depicting various architectures and corresponding methods for implementing and enabling vPLC-related functions.

[0038] Resellers can use the vPLC created for them to offer reseller-supplied and reseller-branded cloud services and provide specialized services in the reseller's areas of expertise. Reseller-supplied services can be tailored to various segments of customers. In some embodiments, reseller-supplied cloud services may be different from cloud services supplied by the CSP. In some other embodiments, reseller-supplied cloud services may be based on cloud services supplied by the CSP; for example, the reseller-supplied services may be customized versions of CSP-supplied cloud services.

[0039] For example, a first vPLC may be created from a CSP-provided infrastructure in a first region for a first reseller that specializes in providing financial services. The first reseller's customers may be banks and other financial institutions. Thus, the first reseller can use the first vPLC to provide customized, first reseller-branded financial cloud services to its customers. A second vPLC may be created from the same CSP-provided infrastructure in the first region for a second reseller that specializes in providing telecommunications services. The second reseller's customers may be users of telecommunications services. The second reseller can use the second vPLC to provide customized, second reseller-branded telecommunications cloud services to its customers. In addition to the first and second vPLCs, the CSP-provided infrastructure in the first region may be used by the CSP to provide CSP-supplied and CSP-branded cloud services to one or more of the CSP's direct (non-reseller) customers. In this way, the same CSP-provided infrastructure within a region is used to provide CSP-supplied and CSP-branded cloud services to one or more direct (non-reseller) customers of the CSP, to provide customized first reseller-branded financial cloud services to the first reseller's customers, and to provide customized second reseller-branded telecommunications cloud services to the second reseller's customers.

[0040] The present disclosure relates to facilitating connectivity to vPLCs created within a CSP-provided infrastructure within a region. When a packet is destined for an endpoint within the CSP-provided infrastructure within a region that is associated with a particular vPLC, the packet is tagged with information related to that particular vPLC. The vPLC-related information for that particular vPLC may include, for example, a vPLC identifier that identifies that particular vPLC, an identifier that identifies a customer associated with the endpoint, a virtual cloud network identifier that identifies a virtual cloud network (VCN) that belongs to that particular vPLC (where the endpoint is part of the VCN), and other vPLC-related information.

[0041] The packets are then routed or communicated within the CSP-provided infrastructure within the region with the tagged vPLC-related information. vPLC-related traffic is tagged with the vPLC-related information to clearly identify the traffic as related to the vPLC. In this manner, the vPLC-related information (e.g., vPLC ID) is used to route packets within the CSP-provided infrastructure within the region as part of connectivity.

[0042] Tagging vPLC-related traffic with vPLC-related information has several technical advantages. For example, it allows components within the CSP-provided infrastructure within the region, particularly physical, overlay, or virtual components responsible for routing traffic within the CSP-provided infrastructure within the region, to distinguish between vPLC-related and non-vPLC-related traffic and identify the vPLC context. This context can be used to ensure that packets are correctly routed within the CSP-provided infrastructure within the region. This eliminates the chance that traffic destined for a vPLC will be routed to some incorrect endpoint. It also eliminates the chance that traffic destined for an endpoint associated with a particular customer of a reseller will be destined to a wrong endpoint belonging to another customer, to another vPLC, or even to a non-vPLC endpoint. This increases the security and reliability of the CSP-provided infrastructure within the region. In this way, the vPLC-related information can be used for efficient traffic separation within the CSP-provided infrastructure within the region. This also enables ultra-fast packet processing using existing CSP-provided resources (e.g., using existing NVDs) without requiring changes at the data layer.

[0043] In embodiments where the vPLC-related information includes a vPLC ID that identifies a particular vPLC, the vPLC ID can be used for a quick single lookup to identify any policies or other metadata associated with the vPLC. Packets can then be routed according to these policy and metadata information. The single lookup results in faster traffic processing and faster communication, as opposed to multiple lookups when the vPLC ID is not included. By tagging packets with vPLC-specific information, vPLC-specific transit connectivity is provided for the vPLC. The transit connectivity can provide Internet, VPN, Fast Connect, and other types of connectivity to resources within the vPLC.

[0044] Figures 1-5 and the associated description provided in the "Exemplary Virtual Networking Architecture" section below describe networking concepts including network virtualization, foundation networks, overlay networks, VNICs, etc., and provide example environments in which certain embodiments described in this disclosure may be implemented. Figures 6-25 describe examples and embodiments related to virtual private label clouds (vPLCs) described in this disclosure. Figures 26-29 depict example architectures for implementing a cloud infrastructure for providing one or more cloud services, which infrastructure may incorporate teachings described herein. Figure 30 depicts a block diagram showing an example computer system or device, according to at least one embodiment.

[0045] Exemplary Virtual Networking Architecture The term cloud services is generally used to refer to services made available by a cloud service provider (CSP) to users or customers on demand (e.g., via a subscription model) using systems and infrastructure (cloud infrastructure) provided by the CSP. Typically, the servers and systems that make up the CSP's infrastructure are separate from the customer's own on-premises servers and systems. Thus, customers can use cloud services provided by the CSP without having to purchase separate hardware and software resources for the service. Cloud services are designed to provide subscribing customers with easy and scalable access to applications and computing resources without the customer having to invest in procuring the infrastructure used to provide the service.

[0046] There are several cloud service providers that offer different types of cloud services. There are various different types or models of cloud services, including Software as a Service (SaaS), Platform as a Service (PaaS), Infrastructure as a Service (IaaS), etc.

[0047] A customer can subscribe to one or more cloud services offered by a CSP. A customer can be any entity, such as an individual, an organization, a business, etc. When a customer subscribes or registers for a service offered by a CSP, a tenancy or account is created for the customer. The customer can then access one or more subscribed cloud resources associated with the account through this account.

[0048] As mentioned above, Infrastructure as a Service (IaaS) is one specific type of cloud computing service. In the IaaS model, a CSP provides infrastructure (referred to as Cloud Service Provider Infrastructure or CSPI) that customers can use to build their own customizable networks and deploy customer resources. Therefore, customer resources and networks are hosted in a distributed environment by the infrastructure provided by the CSP. This differs from traditional computing, where customer resources and networks are hosted by the infrastructure provided by the customer.

[0049] CSPI can include interconnected high-performance computing resources, including various host machines, memory resources, and network resources, forming a physical network, also referred to as an infrastructure network or underlay network. Resources within a CSPI can be distributed across one or more data centers, which can be geographically distributed across one or more geographic regions. These physical resources can run virtualization software to provide a virtualized distributed environment. Virtualization creates an overlay network (also known as a software-based network, software-defined network, or virtual network) on top of the physical network. The CSPI physical network provides the underlying infrastructure for creating one or more overlay or virtual networks on top of the physical network. The physical network (or infrastructure network or underlay network) includes physical network devices such as physical switches, routers, computers, and host machines. An overlay network is a logical (or virtual) network that operates on top of the physical infrastructure network. A given physical network can support one or more overlay networks. Overlay networks typically use encapsulation techniques to distinguish traffic belonging to different overlay networks. A virtual or overlay network is also referred to as a virtual cloud network (VCN). Virtual networks are implemented using software virtualization technologies (e.g., hypervisors, virtualization functions implemented by network virtualization devices (NVDs) (e.g., smartNICs), top-of-rack (TOR) switches, smart TORs that implement one or more functions performed by NVDs, and other mechanisms) to create a layer of network abstraction that can operate on top of a physical network. Virtual networks can take many forms, including peer-to-peer networks, IP networks, etc. Virtual networks are typically either layer-3 IP networks or layer-2 VLANs.This virtual or overlay networking method is often referred to as virtual or overlay Layer 3 networking. Examples of protocols deployed for virtual networks include IP-in-IP (or Generic Routing Encapsulation (GRE)), Virtual Extensible LAN (VXLAN - IETF RFC7348), Virtual Private Networks (VANs) (e.g., MPLS Layer 3 Virtual Private Networks (RFC4364)), VMware's NSX, GENEVE (Generic Network Virtualization Encapsulation), etc.

[0050] In IaaS, the infrastructure (CSPI) provided by the CSP can be configured to provide virtualized computing resources over a public network (e.g., the Internet). In the IaaS model, a cloud computing service provider can host infrastructure components (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer), etc.). In some cases, the IaaS provider can also provide various services (e.g., billing, monitoring, logging, security, load balancing, clustering, etc.) to accompany those infrastructure components. Accordingly, these services can be policy-driven, allowing IaaS users to implement policies to drive load balancing to maintain application availability and performance. The CSPI provides infrastructure and a set of complementary cloud services that enable customers to build and run diverse applications and services in a highly available, hosted, distributed environment. The CSPI delivers high-performance compute resources and capabilities, as well as storage capacity, within a flexible virtual network that is securely accessible from various networked locations, such as the customer's on-premises network. When a customer subscribes or registers for an IaaS service offered by a CSP, the tenancy created for that customer is a secure, isolated division within the CSP where the customer can create, organize, and manage their cloud resources.

[0051] Customers can build their own virtual networks using the compute, memory, and networking resources provided by CSPI. They can deploy one or more customer resources or workloads, such as compute instances, on these virtual networks. For example, customers can build one or more customizable private virtual networks, referred to as virtual cloud networks (VCNs), using resources provided by CSPI. Customers can deploy one or more customer resources, such as compute instances, on the customer VCN. Compute instances can take the form of virtual machines, bare metal instances, and so on. CSPI therefore provides an infrastructure and a set of complementary cloud services that enable customers to build and run diverse applications and services within a highly available, virtual host environment. While customers do not manage or control the underlying physical resources provided by CSPI, they can control the operating system, storage, and deployed applications, and in some cases, have limited control over the selection of networking components (e.g., firewalls).

[0052] The CSP can provide a console that enables customers and network administrators to configure, access, and manage resources deployed in the cloud using CSPI resources. In some embodiments, the console provides a web-based user interface that can be used to access and manage the CSPI. In some embodiments, the console is a web-based application provided by the CSP.

[0053] CSPI can support single-tenancy or multi-tenancy architectures. In a single-tenancy architecture, software (e.g., applications, databases) or hardware components (e.g., host machines or servers) serve a single customer or tenant. In a multi-tenancy architecture, software or hardware components serve multiple customers or tenants. Thus, in a multi-tenancy architecture, CSPI resources are shared among multiple customers or tenants. In a multi-tenancy situation, precautions are taken and safeguards are implemented within CSPI to ensure that each tenant's data remains isolated and invisible to other tenants.

[0054] In a physical network, a network endpoint (“endpoint”) refers to a computing device or system that is connected to the physical network and communicates to and from the connected network. Network endpoints in a physical network may be connected to a local area network (LAN), a wide area network (WAN), or other types of physical networks. Examples of traditional endpoints in a physical network include modems, hubs, bridges, switches, routers, and other networking devices, physical computers (or host machines), etc. Each physical device in a physical network has a fixed network address that can be used to communicate with the device. This fixed network address can be a Layer 2 address (e.g., a MAC address), a fixed Layer 3 address (e.g., an IP address), etc. In a virtualized environment or virtual network, endpoints can include various virtual endpoints, such as virtual machines hosted by components of the physical network (e.g., hosted by a physical host machine). These endpoints in the virtual network are addressed by overlay addresses, such as overlay Layer 2 addresses (e.g., an overlay MAC address) and overlay Layer 3 addresses (e.g., an overlay IP address). Network overlays enable flexibility by allowing network managers to move overlay addresses associated with network endpoints around using software management (e.g., via software implementing the virtual network's control plane). Thus, unlike physical networks, in virtual networks, overlay addresses (e.g., overlay IP addresses) can be moved from one endpoint to another using network management software. Because virtual networks are built on top of physical networks, communication between components in a virtual network involves both the virtual network and the underlying physical network.To facilitate such communications, CSPI components are configured to learn and store mappings that map overlay addresses in the virtual network to actual physical addresses in the underlying network, and vice versa. These mappings are then used to facilitate communications. Customer traffic is encapsulated to facilitate routing within the virtual network.

[0055] Thus, a physical address (e.g., a physical IP address) is associated with an element in a physical network, and an overlay address (e.g., an overlay IP address) is associated with an entity in a virtual or overlay network. A physical IP address is an IP address associated with a physical device (e.g., a network device) in an underlying or physical network. For example, each NVD has an associated physical IP address. An overlay IP address is an overlay address associated with an entity in an overlay network, such as a compute instance in a customer's virtual cloud network (VCN). Two different customers or tenants, each with their own private VCN, can potentially use the same overlay IP address in that VCN without knowing anything about each other. Both physical IP addresses and overlay IP addresses are types of real IP addresses. They are separate from virtual IP addresses. A virtual IP address is typically a single IP address that represents or maps to multiple real IP addresses. A virtual IP address provides a one-to-many mapping between a virtual IP address and multiple real IP addresses. For example, a load balancer may use a VIP to map to or represent multiple servers, each with its own real IP address.

[0056] A cloud infrastructure or CSPI is physically hosted in one or more data centers within one or more regions around the world. The CSPI can include components within a physical or underlying network and virtualized components (e.g., virtual networks, compute instances, virtual machines, etc.) within a virtual network built on top of the physical network components. In one embodiment, the CSPI is organized and hosted in realms, regions, and availability domains. A region is typically a local geographic area that includes one or more data centers. Regions are generally independent of each other and can be separated by large distances, for example, across countries or even continents. For example, a first region may be in Australia, another region may be in Japan, yet another region may be in India, etc. CSPI resources are divided among multiple regions, so that each region has its own independent subset of CSPI resources. Each region can provide a set of essential infrastructure services and resources, such as compute resources (e.g., bare metal servers, virtual machines, containers and related infrastructure, etc.), storage resources (e.g., block volume storage, file storage, object storage, archive storage), networking resources (e.g., virtual cloud networks (VCNs), load balancing resources, connectivity to on-premises networks), database resources, edge networking resources (e.g., DNS), and access management and monitoring resources, etc. Each region typically has multiple paths connecting it to other regions within the realm.

[0057] Because using nearby resources is faster than using distant resources, applications are typically deployed in the region where they are most heavily used (i.e., on the infrastructure associated with that region). Applications may also be deployed in different regions for a variety of reasons, such as redundancy to mitigate the risk of regional events such as large-scale weather systems or earthquakes, to meet the varying requirements of jurisdictions, tax areas, and other business or societal standards, etc.

[0058] Data centers within a region can also be further organized and subdivided into availability domains (ADs). An availability domain can correspond to one or more data centers located within a region. A region can consist of one or more availability domains. In such a distributed environment, CSPI resources are either region-specific, such as a virtual cloud network (VCN), or availability domain-specific, such as a compute instance.

[0059] ADs within a region are isolated from each other, fault-tolerant, and configured to be highly unlikely to fail simultaneously. This is achieved by ADs not sharing critical infrastructure resources such as networking, physical cables, cable routes, cable entry points, etc., so that a failure in one AD within a region is unlikely to affect the availability of other ADs in the same region. ADs within the same region can be connected to each other by low-latency, high-bandwidth networks, which provides highly available connectivity to other networks (e.g., the Internet, customer on-premises networks, etc.) and makes it possible to build replicated systems within multiple ADs for both high availability and disaster recovery. Cloud services use multiple ADs to ensure high availability and protect against resource failures. As the infrastructure provided by IaaS grows, more regions and ADs are added to add capacity. Traffic between availability domains is typically encrypted.

[0060] In one embodiment, regions are grouped into realms. A realm is a logical collection of multiple regions. Realms are isolated from each other and do not share any data. Regions within the same realm can communicate with each other, but regions in different realms cannot communicate with each other. A customer's tenancy and account with a CSP exist within a single realm and can be distributed across one or more regions belonging to that realm. Typically, when a customer subscribes to an IaaS service, their tenancy and account are created in a customer-specified region within the realm (referred to as the "home" region). The customer can extend their tenancy across one or more other regions within the realm. A customer cannot access regions that are not within the realm in which their tenancy resides.

[0061] An IaaS provider may offer multiple realms, each serving a particular set of customers or users. For example, a commercial realm may be offered to commercial customers. As another example, a realm for a particular country may be offered to customers within that country. As yet another example, a government realm may be offered to a government, and so on. For example, a government realm may be offered to a particular government and may have a higher level of security than a commercial realm. For example, Oracle Cloud Infrastructure (OCI) currently offers one realm in a commercial region and two realms in a government cloud region (e.g., FedRAMP-authorized and IL5-authorized).

[0062] In one embodiment, an AD can be subdivided into one or more fault domains. A fault domain is a grouping of infrastructure resources within an AD to provide anti-affinity. Fault domains allow for the distribution of compute instances so that instances are not on the same physical hardware within a single AD. This is known as anti-affinity. A fault domain refers to a set of hardware components (computers, switches, etc.) that share a single point of failure. A compute pool is logically divided into multiple fault domains. Due to this, a hardware failure or compute hardware maintenance event affecting one fault domain does not affect instances in other fault domains. Depending on the embodiment, the number of fault domains in each AD may vary. For example, in one embodiment, each AD includes three fault domains. Fault domains act as logical data centers within an AD.

[0063] When a customer subscribes to an IaaS service, resources from CSPI are provisioned to the customer and associated with the customer's tenancy. The customer can use these provisioned resources to build private networks and deploy resources on these networks. A customer network hosted in the cloud by CSPI is referred to as a virtual cloud network (VCN). A customer can set up one or more virtual cloud networks (VCNs) using the CSPI resources allocated to the customer. A VCN is a virtual or software-defined private network. Customer resources deployed within a customer's VCN can include compute instances (e.g., virtual machines, bare metal instances) and other resources. These compute instances can represent various customer workloads such as applications, load balancers, databases, etc. Compute instances deployed on a VCN can communicate with publicly accessible endpoints ("public endpoints") on public networks such as the Internet, other instances in the same VCN or other VCNs (e.g., other VCNs of the customer or VCNs not belonging to the customer), the customer's on-premises data center or network, and service endpoints and other types of endpoints.

[0064] CSPs can offer various services using CSPI. In some cases, customers of a CSPI themselves may act as service providers and offer services using CSPI resources. Service providers can expose service endpoints characterized by identifying information (e.g., IP addresses, DNS names, and ports). Customer resources (e.g., compute instances) can consume a particular service by accessing the service endpoint exposed by the service for that particular service. These service endpoints are generally publicly accessible by users via a public communications network, such as the Internet, using a public IP address associated with the endpoint. Publicly accessible network endpoints are sometimes referred to as public endpoints.

[0065] In some embodiments, a service provider may expose a service through an endpoint for that service (sometimes referred to as a service endpoint). Customers of the service may then access the service using this service endpoint. In some embodiments, a service endpoint provided for a service may be accessed by multiple customers who intend to consume the service. In other embodiments, a dedicated service endpoint may be provided for a customer, so that only that customer can access the service using that dedicated service endpoint.

[0066] In one embodiment, when a VCN is created, it is associated with a private overlay Classless Inter-Domain Routing (CIDR) address space, which is a range of private overlay IP addresses (e.g., 10.0 / 16) that are assigned to the VCN. A VCN includes associated subnets, routing tables, and gateways. A VCN exists within a single region but can span one, more, or all of the region's availability domains. Gateways are virtual interfaces configured for a VCN that enable traffic to and from the VCN to one or more endpoints outside the VCN. One or more different types of gateways may be configured for a VCN to enable communication to and from different types of endpoints.

[0067] A VCN can be subdivided into one or more subnetworks, such as one or more subnets. A subnet is thus a building block or subdivision that can be created within a VCN. A VCN can have one or more subnets. Each subnet in a VCN is associated with a contiguous range of overlay IP addresses (e.g., 10.0.0.0 / 24 and 10.0.1.0 / 24) that do not overlap with other subnets in that VCN and represent a subset of address space within the VCN's address space.

[0068] Each compute instance is associated with a virtual network interface card (VNIC) that enables the compute instance to participate in a subnet of a VCN. A VNIC is a logical representation of a physical network interface card (NIC). Generally, a VNIC is an interface between an entity (e.g., a compute instance, a service) and a virtual network. A VNIC exists within a subnet and has one or more associated IP addresses and associated security rules or policies. A VNIC is equivalent to a Layer 2 port on a switch. A VNIC is attached to a compute instance and a subnet within a VCN. A VNIC associated with a compute instance enables the compute instance to be part of a subnet of a VCN and enables the compute instance to communicate (e.g., send and receive packets) with endpoints on the same subnet as the compute instance, endpoints in a different subnet within the VCN, or endpoints outside the VCN. Thus, a VNIC associated with a compute instance determines how the compute instance connects with endpoints inside and outside the VCN. When a compute instance is created and added to a subnet within a VCN, a VNIC for the compute instance is created and associated with the compute instance. For a subnet that includes a set of compute instances, the subnet includes VNICs that correspond to the set of compute instances, with each VNIC attached to a compute instance in the set of compute instances.

[0069] Each compute instance is assigned a private overlay IP address via the VNIC associated with the compute instance. This private overlay IP address is assigned to the VNIC associated with the compute instance when the compute instance is created and is used to route traffic to and from the compute instance. All VNICs within a given subnet use the same routing table, security lists, and DHCP options. As described above, each subnet within a VCN is associated with a contiguous range of overlay IP addresses (e.g., 10.0.0.0 / 24 and 10.0.1.0 / 24) that do not overlap with other subnets within that VCN and represent a subset of address space within the VCN's address space. For a VNIC on a particular subnet of a VCN, the private overlay IP address assigned to the VNIC is from the contiguous range of overlay IP addresses allocated to the subnet.

[0070] In some embodiments, a compute instance may optionally be assigned additional overlay IP addresses in addition to its private overlay IP address, such as one or more public IP addresses if it is in a public subnet. These multiple addresses are assigned on the same VNIC or across multiple VNICs associated with the compute instance. However, each instance has a primary VNIC created during instance launch and associated with the overlay private IP address assigned to the instance, and this primary VNIC cannot be deleted. Additional VNICs, referred to as secondary VNICs, can be added to an existing instance in the same availability domain as the primary VNIC. All VNICs are in the same availability domain as the instance. The secondary VNIC can be in a subnet in the same VCN as the primary VNIC or in a different subnet, either in the same VCN or a different VCN.

[0071] Compute instances may optionally be assigned public IP addresses if they are in a public subnet. Subnets can be designed as either public or private subnets when the subnet is created. A private subnet means that resources (e.g., compute instances) and associated VNICs within the subnet cannot have public overlay IP addresses. A public subnet means that resources and associated VNICs within the subnet can have public IP addresses. Customers can specify a subnet to exist within a single availability domain or across multiple availability domains within a region or realm.

[0072] As described above, a VCN can be subdivided into one or more subnets. In one embodiment, a virtual router (VR) configured for a VCN (referred to as a VCN VR or simply VR) enables communication between subnets in the VCN. For a subnet in a VCN, the VR represents the logical gateway for that subnet, allowing the subnet (i.e., the compute instances on that subnet) to communicate with endpoints on other subnets in the VCN and with other endpoints outside the VCN. A VCN VR is a logical entity configured to route traffic between VNICs in a VCN and a virtual gateway ("gateway") associated with the VCN. Gateways are further described below with respect to FIG. 1. A VCN VR is a Layer 3 / IP layer concept. In one embodiment, there is one VCN VR for a VCN, and the VCN VR has a potentially infinite number of ports addressed by IP addresses, one port for each subnet in the VCN. In this way, the VCN VR has a different IP address for each subnet in the VCN to which the VCN VR is attached. The VRs are also connected to various gateways configured for the VCN. In some embodiments, a specific overlay IP address from a subnet's overlay IP address range is reserved for ports in that subnet's VCN VR. For example, consider a VCN with two subnets, each with associated address ranges 10.0 / 16 and 10.1 / 16. For the first subnet in the VCN with address range 10.0 / 16, an address from this range is reserved for ports in that subnet's VCN VR. In some cases, the first IP address from that range may be reserved for a VCN VR. For example, for a subnet with overlay IP address range 10.0 / 16, IP address 10.0.0.1 may be reserved for ports in that subnet's VCN VR.For a second subnet in the same VCN with address range 10.1 / 16, the VCN VR may have a port in that second subnet with IP address 10.1.0.1. The VCN VR has a different IP address for each of the subnets in the VCN.

[0073] In some other embodiments, each subnet within a VCN can have its own associated VR that is addressable by the subnet using a reserved or default IP address associated with the VR. The reserved or default IP address may, for example, be the first IP address from a range of IP addresses associated with the subnet. VNICs within a subnet can use this default or reserved IP address to communicate (e.g., send and receive packets) with the VR associated with the subnet. In such embodiments, a VR is the ingress / egress point for that subnet. A VR associated with a subnet within a VCN can communicate with other VRs associated with other subnets in the VCN. The VR can also communicate with a gateway associated with the VCN. The VR functions of a subnet are running on or performed by one or more NVDs that are performing the VNIC functions of VNICs within the subnet.

[0074] Routing tables, security rules, and DHCP options can be configured for a VCN. A routing table is a virtual routing table for a VCN that contains rules for routing traffic from subnets within the VCN to destinations outside the VCN by gateways or specially configured instances. A VCN's routing table can be customized to control how packets are forwarded / routed into and out of the VCN. DHCP options are configuration information that is automatically provided to instances when they launch.

[0075] Security rules configured for a VCN represent the overlay firewall rules for the VCN. Security rules can include ingress and egress rules and can specify the type of traffic (based on protocol and port) that is allowed in and out of instances within the VCN. Customers can choose whether a given rule is stateful or stateless. For example, a customer can allow SSH traffic from anywhere to enter a set of instances by setting up a stateful ingress rule with source CIDR 0.0.0.0 / 0 and destination TCP port 22. Security rules can be implemented using network security groups or security lists. A network security group consists of a set of security rules that apply only to resources within that group. A security list, on the other hand, contains rules that apply to all resources within any subnet that uses the security list. A VCN can be provided with a default security list with default security rules. DHCP options configured for a VCN provide configuration information that is automatically provided to instances within the VCN when the instances launch.

[0076] In one embodiment, configuration information for a VCN is determined and stored by a VCN control plane. The VCN configuration information may include, for example, information about address ranges associated with the VCN, subnets and associated information within the VCN, one or more VRs associated with the VCN, compute instances and associated VNICs within the VCN, NVDs performing various virtualized network functions (e.g., VNICs, VRs, gateways) associated with the VCN, VCN state information, and other VCN-related information. In one embodiment, a VCN distribution service publishes the configuration information stored by the VCN control plane, or portions thereof, to the NVD. The distributed information can be used to update information (e.g., forwarding tables, routing tables, etc.) stored and used by the NVD to forward packets to and from compute instances within the VCN.

[0077] In one embodiment, VCN and subnet creation is handled by a VCN control plane (CP), and compute instance launching is handled by the compute control plane. The compute control plane is responsible for allocating physical resources for the compute instances and then calls the VCN control plane to create and attach VNICs to the compute instances. The VCN CP also sends VCN data mappings to the VCN data plane, which is configured to perform packet forwarding and routing functions. In one embodiment, the VCN CP provides a distribution service that is responsible for providing updates to the VCN data plane. Examples of the VCN control plane are also shown in Figures 26, 27, 28, and 29 (see reference numbers 2616, 2716, 2816, and 2916) and described below.

[0078] Customers can create one or more VCNs using resources hosted by CSPI. Compute instances deployed on a customer VCN can communicate with multiple different endpoints. These endpoints can include endpoints hosted by CSPI and endpoints external to CSPI.

[0079] Various different architectures for implementing cloud-based services using CSPI are shown in Figures 1, 2, 3, 4, 5, 26, 27, 28, and 29 and described below. Figure 1 is a high-level diagram of a distributed environment 100 illustrating an overlay or customer VCN hosted by CSPI according to one embodiment. The distributed environment shown in Figure 1 includes multiple components within an overlay network. The distributed environment 100 shown in Figure 1 is illustrative only and is not intended to unduly limit the scope of the claimed embodiments. Many variations, alternatives, and modifications are possible. For example, in some other embodiments, the distributed environment shown in Figure 1 may have more or fewer systems or components than those shown in Figure 1, may combine two or more systems, or may have a different configuration or arrangement of systems.

[0080] As shown in the example depicted in FIG. 1 , a distributed environment 100 includes a CSPI 101 that provides services and resources that customers can subscribe to and use to build their virtual cloud networks (VCNs). In one embodiment, CSPI 101 provides IaaS services to subscribing customers. Data centers within CSPI 101 can be organized into one or more regions. One exemplary region, “Region US” 102, is shown in FIG. 1 . A customer configures a customer VCN 104 in region 102. A customer can deploy various compute instances on VCN 104, where the compute instances may include virtual machines or bare metal instances. Example instances include applications, databases, load balancers, etc.

[0081] In the embodiment shown in FIG. 1 , customer VCN 104 includes two subnets, "Subnet 1" and "Subnet 2," each with its own CIDR IP address range. In FIG. 1 , Subnet 1's overlay IP address range is 10.0 / 16, and Subnet 2's address range is 10.1 / 16. VCN virtual router 105 represents the VCN's logical gateway, enabling communication between subnets in VCN 104 and with other endpoints outside the VCN. VCN VR 105 is configured to route traffic between VNICs in VCN 104 and the gateway associated with VCN 104. VCN VR 105 provides a port for each subnet in VCN 104. For example, VR 105 may provide a port with IP address 10.0.0.1 for Subnet 1 and a port with IP address 10.1.0.1 for Subnet 2.

[0082] Multiple compute instances may be deployed on each subnet, where the compute instances may be virtual machine instances and / or bare metal instances. The compute instances within a subnet may be hosted by one or more host machines within CSPI 101. A compute instance joins a subnet via a VNIC associated with the compute instance. For example, as shown in FIG. 1, compute instance C1 is part of subnet 1 via a VNIC associated with the compute instance. Similarly, compute instance C2 is part of subnet 1 via a VNIC associated with C2. Similarly, multiple compute instances, which may be virtual machine instances or bare metal instances, may be part of subnet 1. Through its associated VNIC, each compute instance is assigned a private overlay IP address and MAC address. For example, in FIG. 1, compute instance C1 has an overlay IP address of 10.0.0.2 and a MAC address of M1, while compute instance C2 has a private overlay IP address of 10.0.0.3 and a MAC address of M2. Each compute instance in Subnet 1, including compute instances C1 and C2, has a default routing to VCN VR 105 using IP address 10.0.0.1, which is the IP address of a port in VCN VR 105 in Subnet 1.

[0083] Subnet2 may have multiple compute instances deployed thereon, including virtual machine instances and / or bare metal instances. For example, as shown in FIG. 1, compute instances D1 and D2 are part of Subnet2 via VNICs associated with the respective compute instances. In the embodiment shown in FIG. 1, compute instance D1 has an overlay IP address of 10.1.0.2 and a MAC address of MM1, while compute instance D2 has a private overlay IP address of 10.1.0.3 and a MAC address of MM2. Each compute instance in Subnet2, including compute instances D1 and D2, has default routing to VCN VR 105 using IP address 10.1.0.1, which is the IP address of a port in VCN VR 105 in Subnet2.

[0084] VCN A 104 may also include one or more load balancers. For example, a load balancer may be provided for a subnet and configured to load balance traffic across multiple compute instances on the subnet. A load balancer may also be provided to load balance traffic across subnets within the VCN.

[0085] A particular compute instance deployed on VCN 104 can communicate with a variety of different endpoints. These endpoints may include endpoints hosted by CSPI 200 and endpoints outside of CSPI 200. Endpoints hosted by CSPI 101 may include endpoints on the same subnet as the particular compute instance (e.g., communication between two compute instances in Subnet 1), endpoints on a different subnet but within the same VCN (e.g., communication between a compute instance in Subnet 1 and a compute instance in Subnet 2), endpoints in a different VCN within the same region (e.g., communication between a compute instance in Subnet 1 and an endpoint in a VCN in the same region 106 or 110, communication between a compute instance in Subnet 1 and an endpoint in a service network 110 in the same region), or endpoints in a VCN in a different region (e.g., communication between a compute instance in Subnet 1 and an endpoint in a VCN in a different region 108). Compute instances in a subnet hosted by CSPI 101 may also communicate with endpoints not hosted by CSPI 101 (i.e., outside of CSPI 101). These external endpoints include endpoints within the customer's on-premise network 116, endpoints within other remote cloud host networks 118, public endpoints 114 accessible via public networks such as the Internet, and other endpoints.

[0086] Communication between compute instances on the same subnet is facilitated using VNICs associated with the source and destination compute instances. For example, compute instance C1 in Subnet 1 may desire to send a packet to compute instance C2 in Subnet 1. For a packet originating from a source compute instance and destined for another compute instance in the same subnet, the packet is first processed by the VNIC associated with the source compute instance. The processing performed by the VNIC associated with the source compute instance may include determining the packet's destination information from the packet header, identifying any policies (e.g., security lists) configured for the VNIC associated with the source compute instance, determining the packet's next hop, performing any encapsulation / decapsulation as needed, and then forwarding / routing the packet to the next hop with the goal of facilitating communication of the packet to its intended destination. When the destination compute instance is in the same subnet as the source compute instance, the VNIC associated with the source compute instance is configured to identify the VNIC associated with the destination compute instance and forward the packet to that VNIC for processing. At this time, the VNIC associated with the destination compute instance is executed and forwards the packet to the destination compute instance.

[0087] For packets to be communicated from a compute instance in a subnet to an endpoint in a different subnet within the same VCN, communication is facilitated by VNICs and VCN VRs associated with the source and destination compute instances. For example, if compute instance C1 in Subnet 1 of FIG. 1 wants to send a packet to compute instance D1 in Subnet 2, the packet is first processed by the VNIC associated with compute instance C1. The VNIC associated with compute instance C1 is configured to route the packet to VCN VR 105 using the VCN VR's default routing or port 10.0.0.1. VCN VR 105 is configured to route the packet to Subnet 2 using port 10.1.0.1. The packet is then received and processed by the VNIC associated with D1, which forwards the packet to compute instance D1.

[0088] For packets to be communicated from a compute instance within VCN 104 to an endpoint outside VCN 104, the communication is facilitated by a VNIC associated with the source compute instance, VCN VR 105, and a gateway associated with VCN 104. One or more types of gateways may be associated with VCN 104. A gateway is an interface between a VCN and another endpoint, where the other endpoint is external to the VCN. A gateway is a Layer 3 / IP layer concept that allows a VCN to communicate with endpoints outside the VCN. Thus, a gateway facilitates traffic flow between a VCN and other VCNs or networks. Various different types of gateways may be configured for a VCN to facilitate different types of communication with different types of endpoints. Depending on the gateway, the communication may be over a public network (e.g., the Internet) or a private network. Various communication protocols may be used for these communications.

[0089] For example, compute instance C1 may desire to communicate with an endpoint outside VCN 104. The packet may first be processed by a VNIC associated with source compute instance C1. The VNIC processing determines that the packet's destination is outside Subnet 1 of C1. The VNIC associated with C1 may forward the packet to VCN VR 105 of VCN 104. VCN VR 105 then processes the packet and, as part of the processing, determines a particular gateway associated with VCN 104 as the packet's next hop based on the packet's destination. VCN VR 105 may then forward the packet to the particular identified gateway. For example, if the destination is an endpoint within a customer's on-premises network, the packet may be forwarded by VCN VR 105 to a dynamic routing gateway (DRG) gateway 122 configured for VCN 104. The packet may then be forwarded from the gateway to the next hop to facilitate communication of the packet to its final intended destination.

[0090] Various different types of gateways may be configured for a VCN. Examples of gateways that may be configured for a VCN are shown in FIG. 1 and described below. Examples of gateways associated with VCNs are also shown in FIGS. 26, 27, 28, and 29 (e.g., gateways referenced by reference numbers 2634, 2636, 2638, 2734, 2736, 2738, 2834, 2836, 2838, 2934, 2936, and 2938) and described below. As shown in the embodiment shown in FIG. 1, a dynamic routing gateway (DRG) 122 can be added to or associated with a customer VCN 104 to provide a path for private network traffic communication between the customer VCN 104 and another endpoint, where the other endpoint can be the customer's on-premises network 116, a VCN 108 in a different region of CSPI 101, or another remote cloud network 118 not hosted by CSPI 101. The customer on-premises network 116 may be a customer network or customer data center built using the customer's resources. Access to the customer on-premises network 116 is generally very restricted. For a customer that has both a customer on-premises network 116 and one or more VCNs 104 deployed or hosted in the cloud by CSPI 101, the customer may want their on-premises network 116 and their cloud-based VCNs 104 to be able to communicate with each other. This allows the customer to build an extended hybrid environment that encompasses the customer's VCN 104 hosted by CSPI 101 and their on-premises network 116. The DRG 122 enables this communication. To enable such communication, a communication channel 124 is set up, where one endpoint of the channel resides in the customer on-premises network 116 and the other endpoint resides in CSPI 101 and is connected to the customer VCN 104.The communication channel 124 can be over a public communication network, such as the Internet, or a private communication network. A variety of different communication protocols may be used, such as IPsec VPN technology over a public communication network, such as the Internet, or Oracle's FastConnect technology, which uses a private network instead of a public network. A device or equipment in the customer on-premises network 116 that forms one endpoint of the communication channel 124 is referred to as customer premises equipment (CPE), such as CPE 126 shown in FIG. 1. On the CSPI 101 side, the endpoint may be a host machine running DRG 122.

[0091] In some embodiments, remote peering connections (RPCs) can be added to a DRG, allowing customers to peer one VCN with another VCN in a different region. Using such RPCs, a customer VCN 104 can connect with a VCN 108 in another region using the DRG 122. The DRG 122 can also be used to communicate with other remote cloud networks 118 not hosted by CSPI 101, such as the Microsoft Azure cloud, the Amazon AWS cloud, etc.

[0092] 1, an internet gateway (IGW) 120 can be configured for a customer VCN 104 that enables compute instances on VCN 104 to communicate with public endpoints 114 accessible over a public network, such as the Internet. The IGW 120 is a gateway that connects a VCN to a public network, such as the Internet. The IGW 120 enables public subnets in a VCN, such as VCN 104 (where resources in the public subnet have public overlay IP addresses) to directly access public endpoints 112 on the public network 114, such as the Internet. The IGW 120 can be used to initiate connections from subnets in VCN 104 or from the Internet.

[0093] A network address translation (NAT) gateway 128 can be configured for a customer's VCN 104 to enable cloud resources in the customer's VCN that do not have dedicated public overlay IP addresses to access the Internet, without exposing those resources to a direct inbound Internet connection (e.g., an L4-L7 connection). This allows private subnets in the VCN, such as Private Subnet 1 in VCN 104, to have private access to public endpoints on the Internet. With a NAT gateway, connections can only be initiated from the private subnet to the public Internet, not from the Internet to the private subnet.

[0094] In one embodiment, a service gateway (SGW) 126 can be configured for customer VCN 104 and provides a pathway for private network traffic between VCN 104 and supported service endpoints in service network 110. In one embodiment, service network 110 can be provided by a CSP and can offer a variety of services. One example of such a service network is Oracle's service network, which offers a variety of services that can be used by customers. For example, a compute instance (e.g., a database system) in a private subnet of customer VCN 104 can back up data to a service endpoint (e.g., object storage) without requiring a public IP address or access to the Internet. In one embodiment, a VCN can have only one SGW, and connections can be initiated only from subnets within the VCN, not from service network 110. If a VCN is peered with another VCN, resources in the other VCN cannot access the SGW. Resources in an on-premises network connected to a VCN by FastConnect or VPN Connect can also use a service gateway configured for that VCN.

[0095] In one embodiment, SGW 126 uses the concept of a service classless inter-domain routing (CIDR) label, which is a string that represents all regional public IP address ranges for a service or group of services of interest. A customer uses the service CIDR label when configuring the SGW and associated routing rules to control traffic to the service. A customer can optionally utilize the service CIDR label when configuring security rules without having to adjust those security rules if the service's public IP addresses change in the future.

[0096] A local peering gateway (LPG) 132 is a gateway that can be added to a customer VCN 104, allowing the VCN 104 to peer with another VCN in the same region. Peering means that the VCNs communicate using private IP addresses without the traffic traversing a public network such as the Internet or routing the traffic through the customer's on-premises network 116. In a preferred embodiment, a VCN has a separate LPG for each peering it establishes. Local peering or VCN peering is a common practice used to establish network connectivity between different applications or infrastructure management functions.

[0097] A service provider, such as a provider of a service in service network 110, can provide access to the service using different access models. According to a public access model, the service may be exposed as a public endpoint publicly accessible by a compute instance in the customer VCN over a public network such as the Internet, and / or may be privately accessible through SGW 126. According to a specific private access model, the service is made accessible as a private IP endpoint in a private subnet in the customer's VCN. This is referred to as private endpoint (PE) access and allows service providers to expose their services as instances in the customer's private network. Private endpoint resources represent services in the customer's VCN. Each PE appears as a VNIC (referred to as a PE-VNIC with one or more private IPs) in a subnet chosen by the customer in the customer's VCN. Thus, the PE provides a way to present the service in the private customer VCN subnet using a VNIC. Because the endpoint is exposed as a VNIC, all features associated with the VNIC, such as routing rules, security lists, etc., are now made available to the PE VNIC.

[0098] Service providers can register their services to enable access through the PE. Providers can associate policies with services, which constrain the visibility of the service to a customer tenancy. Providers can register multiple services under a single virtual IP address (VIP), especially for multi-tenant services. There may be multiple such private endpoints (in multiple VCNs) representing the same service.

[0099] Compute instances in the private subnet can then access the service using the private IP address or service DNS name of the PE VNIC. Compute instances in the customer VCN can access the service by sending traffic to the private IP address of the PE in the customer VCN. A private access gateway (PAGW) 130 is a gateway resource that can be attached to a service provider VCN (e.g., a VCN in service network 110) that acts as the ingress / egress point for all traffic from / to customer subnet private endpoints. The PAGW 130 allows providers to scale the number of PE connections without utilizing their internal IP address resources. A provider only needs to configure one PAGW for any number of services registered in a single VCN. A provider can represent services as private endpoints in multiple VCNs for one or more customers. From the customer's perspective, the PE VNIC appears to be attached to the service with which the customer wants to interact, instead of being attached to the customer's instance. Traffic destined for the private endpoint is routed to the service through the PAGW 130. These are referred to as Customer-to-Service Private Connections (C2S Connections).

[0100] The PE concept can also be used to extend private access for services to a customer's on-premises network and data center by allowing traffic to flow through a FastConnect / IPsec link and a private endpoint within the customer VCN. Private access for services can also be extended to a customer's peered VCN by allowing traffic to flow between LPG 132 and a PE in the customer VCN.

[0101] Customers can control routing in their VCNs at the subnet level; thus, they can specify which subnets of their VCNs, such as VCN 104, use each gateway. A VCN's routing tables are used to determine whether traffic is allowed to exit a VCN through a particular gateway. For example, in a particular case, the routing table for a public subnet in customer VCN 104 can send non-local traffic through IGW 120. The routing table for a private subnet in the same customer VCN 104 can send traffic destined for CSP services through SGW 126. All remaining traffic can be sent through NAT gateway 128. The routing tables control only traffic that exits a VCN.

[0102] Security lists associated with a VCN are used to control traffic entering the VCN through the gateway via incoming connections. All resources within a subnet use the same routing table and security lists. Security lists can be used to control specific types of traffic allowed into and out of instances within a subnet of the VCN. Security list rules can include ingress (inbound) and egress (outbound) rules. For example, ingress rules can specify allowed source address ranges, while egress rules can specify allowed destination address ranges. Security rules can specify specific protocols (e.g., TCP, ICMP), specific ports (e.g., 22 for SSH, 3389 for Windows RDP), etc. In some embodiments, the instance's operating system can enforce its own firewall rules that are aligned with the security list rules. Rules can be stateful (e.g., connections are tracked and responses are automatically allowed without explicit security list rules for the response traffic) or stateless.

[0103] Access from a customer VCN (i.e., by resources or compute instances deployed on VCN 104) can be categorized as public access, private access, or private access. Public access refers to an access model in which public IP addresses or NATs are used to access public endpoints. Private access allows customer workloads in VCN 104 with private IP addresses (e.g., resources in a private subnet) to access services without traversing a public network such as the Internet. In one embodiment, CSPI 101 enables customer VCN workloads with private IP addresses to access services (public service endpoints) using a service gateway. Thus, the service gateway provides a private access model by establishing a virtual link between the customer VCN and the service's endpoints, which reside outside the customer's private network.

[0104] Additionally, CSPI can provide dedicated public access using technologies such as FastConnect public peering, which allows customer on-premises instances to use a FastConnect connection to access one or more services in the customer VCN without traversing a public network such as the Internet. CSPI can also provide dedicated private access using FastConnect private peering, which allows customer on-premises instances with private IP addresses to access customer VCN workloads using a FastConnect connection. FastConnect is a network connection alternative to using the public Internet to connect a customer's on-premises network to CSPI and its services. FastConnect provides an easy, elastic, and economical way to create a dedicated private connection, with higher bandwidth options and a more reliable and consistent networking experience compared to Internet-based connections.

[0105] FIG. 1 and the accompanying discussion above describe various virtualized components in an exemplary virtual network. As noted above, a virtual network is built on an underlying physical or infrastructure network. FIG. 2 shows a simplified architectural diagram of physical components in a physical network within CSPI 200 that provides the underlying layer of the virtual network, according to one embodiment. As shown, CSPI 200 provides a distributed environment including components and resources (e.g., compute, memory, and networking resources) provided by a cloud service provider (CSP). These components and resources are used to provide cloud services (e.g., IaaS services) to subscribing customers, i.e., customers who subscribe to one or more services offered by the CSP. Based on the services to which the customer subscribes, a subset of CSPI 200's resources (e.g., compute, memory, and networking resources) is provisioned to the customer. The customer can then build their own cloud-based (i.e., CSPI-hosted), customizable private virtual network using the physical compute, memory, and networking resources provided by CSPI 200. As previously indicated, these customer networks are referred to as virtual cloud networks (VCNs). Customers can deploy one or more customer resources, such as compute instances, on these customer VCNs. The compute instances can be virtual machines, bare metal instances, etc. CSPI 200 provides the infrastructure and a set of complementary cloud services that enable customers to build and run a wide variety of applications and services in a highly available hosted environment.

[0106] In the exemplary embodiment shown in FIG. 2, the physical components of CSPI 200 include one or more physical host machines or servers (e.g., 202, 206, 208), network virtualization devices (NVDs) (e.g., 210, 212), top-of-rack (TOR) switches (e.g., 214, 216), and a physical network (e.g., 218), as well as switches within physical network 218. The physical host machines or servers can host and execute various compute instances that participate in one or more subnets of a VCN. The compute instances may include virtual machine instances or bare metal instances. For example, the various compute instances shown in FIG. 1 may be hosted by the physical host machines shown in FIG. 2. The virtual machine compute instances in a VCN may be executed by a single host machine or by multiple different host machines. The physical host machines may also host virtual host machines, container-based hosts or functions, etc. The VNICs and VCN VRs shown in FIG. 1 may be executed by the NVDs shown in FIG. 2. The gateway shown in FIG. 1 may be implemented by a host machine and / or NVD shown in FIG.

[0107] A host machine or server may run a hypervisor (also referred to as a virtual machine monitor or VMM), which creates and enables a virtualized environment on the host machine. Virtualization or virtualized environments facilitate cloud-based computing. One or more compute instances may be created, run, and managed on the host machine by the hypervisor on the host machine. The hypervisor on the host machine enables the host machine's physical computing resources (e.g., compute, memory, and networking resources) to be shared among the various compute instances executed by the host machine.

[0108] For example, as shown in FIG. 2, host machines 202 and 208 execute hypervisors 260 and 266, respectively. These hypervisors may be implemented using software, firmware, or hardware, or a combination thereof. Typically, a hypervisor is a process or software layer that runs on the host machine's hardware processor and resides above the host machine's operating system (OS). A hypervisor provides a virtualized environment by allowing the host machine's physical computing resources (e.g., processing resources such as processors / cores, memory resources, and networking resources) to be shared among various virtual machine computing instances executed by the host machine. For example, in FIG. 2, hypervisor 260 can reside above the OS of host machine 202 and allow the host machine's computing resources (e.g., processing, memory, and networking resources) to be shared among computing instances (e.g., virtual machines) executed by host machine 202. A virtual machine can have its own operating system (referred to as a guest operating system), which may be the same as or different from the host machine's OS. The operating system of a virtual machine executed by a host machine may be the same as or different from the operating system of another virtual machine executed by the same host machine. Thus, the hypervisor allows multiple operating systems to run side by side with each other while sharing the same computing resources of the host machine. The host machines shown in Figure 2 may have the same or different types of hypervisors.

[0109] A compute instance can be a virtual machine instance or a bare metal instance. In Figure 2, compute instance 268 on host machine 202 and compute instance 274 on host machine 208 are examples of virtual machine instances. Host machine 206 is an example of a bare metal instance provided to a customer.

[0110] In some cases, an entire host machine may be provisioned to a single customer, and one or more compute instances (either virtual machines or bare metal instances) hosted by that host machine all belong to that same customer. In other cases, a host machine may be shared among multiple customers (i.e., multiple tenants). In such a multi-tenancy scenario, a host machine may host virtual machine compute instances belonging to different customers. These compute instances may be members of different VCNs for different customers. In some embodiments, bare metal compute instances are hosted by bare metal servers without a hypervisor. When a bare metal compute instance is provisioned, a single customer or tenant maintains control of the physical CPU, memory, and network interfaces of the host machine that hosts the bare metal instance, and the host machine is not shared with other customers or tenants.

[0111] As previously described, each compute instance that is part of a VCN is associated with a VNIC that enables the compute instance to be a member of a subnet of the VCN. A VNIC associated with a compute instance facilitates communication of packets or frames to and from the compute instance. When a compute instance is created, a VNIC is associated with the compute instance. In one embodiment, for a compute instance executed by a host machine, the VNIC associated with the compute instance is executed by an NVD connected to the host machine. For example, in FIG. 2, host machine 202 executes virtual machine compute instance 268 that is associated with VNIC 276, which is executed by NVD 210 connected to host machine 202. As another example, bare metal instance 272 hosted by host machine 206 is associated with VNIC 280, which is executed by NVD 212 connected to host machine 206. As yet another example, VNIC 284 is associated with compute instance 274 executed by host machine 208, which is executed by NVD 212 connected to host machine 208.

[0112] For compute instances hosted by a host machine, the NVD connected to that host machine also executes VCN VRs corresponding to the VCNs of which the compute instance is a member. For example, in the embodiment shown in Figure 2, NVD 210 executes VCN VR 277 corresponding to the VCN of which compute instance 268 is a member. NVD 212 may also execute one or more VCN VRs 283 corresponding to the VCNs corresponding to the compute instances hosted by host machines 206 and 208.

[0113] A host machine may include one or more network interface cards (NICs) that allow the host machine to be connected to other devices. The NICs on a host machine may provide one or more ports (or interfaces) that allow the host machine to be communicatively connected to another device. For example, a host machine may be connected to an NVD using one or more ports (or interfaces) provided on the host machine and the NVD. A host machine may also be connected to other devices, such as another host machine.

[0114] 2, host machine 202 is connected to NVD 210 using link 220 extending between port 234 provided by NIC 232 of host machine 202 and port 236 of NVD 210. Host machine 206 is connected to NVD 212 using link 224 extending between port 246 provided by NIC 244 of host machine 206 and port 248 of NVD 212. Host machine 208 is connected to NVD 212 using link 226 extending between port 252 provided by NIC 250 of host machine 208 and port 254 of NVD 212.

[0115] The NVDs are then connected via communication links to top-of-rack (TOR) switches, which are connected to a physical network 218 (also referred to as a switch fabric). In one embodiment, the links between the host machines and the NVDs and between the NVDs and the TOR switches are Ethernet links. For example, in Figure 2, NVDs 210 and 212 are connected to TOR switches 214 and 216 using links 228 and 230, respectively. In one embodiment, links 220, 224, 226, 228, and 230 are Ethernet links. The collection of host machines and NVDs connected to a TOR may be referred to as a rack.

[0116] The physical network 218 provides a communications fabric that allows the TOR switches to communicate with each other. The physical network 218 can be a multi-tiered network. In one embodiment, the physical network 218 is a multi-tiered Clos network of switches, with the TOR switches 214 and 216 representing leaf-level nodes of the multi-tiered, multi-node physical switching network 218. Different Clos network configurations are possible, including, but not limited to, two-tier networks, three-tier networks, four-tier networks, five-tier networks, and generally "n"-tier networks. An example of a Clos network is shown in FIG. 5 and described below.

[0117] A variety of different connection configurations between host machines and NVDs are possible, such as one-to-one configurations, many-to-one configurations, one-to-many configurations, etc. In a one-to-one configuration embodiment, each host machine is connected to its own separate NVD. For example, in Figure 2, host machine 202 is connected to NVD 210 via NIC 232 of host machine 202. In a many-to-one configuration, multiple host machines are connected to one NVD. For example, in Figure 2, host machines 206 and 208 are connected to the same NVD 212 via NICs 244 and 250, respectively.

[0118] In a one-to-many configuration, one host machine is connected to multiple NVDs. Figure 3 shows an example of a CSPI 300 in which a host machine is connected to multiple NVDs. As shown in Figure 3, host machine 302 includes a network interface card (NIC) 304 that includes multiple ports 306 and 308. Host machine 300 is connected to a first NVD 310 via port 306 and link 320, and to a second NVD 312 via port 308 and link 322. Ports 306 and 308 may be Ethernet ports, and links 320 and 322 between host machine 302 and NVDs 310 and 312 may be Ethernet links. NVD 310 is then connected to a first TOR switch 314, and NVD 312 is connected to a second TOR switch 316. The links between NVDs 310 and 312 and TOR switches 314 and 316 may be Ethernet links. TOR switches 314 and 316 represent tier 0 switching devices within a multi-tiered physical network 318 .

[0119] 3 provides two separate physical network paths from the physical switch network 318 to the host machine 302 and from the host machine 302 to the physical switch network 318: a first path traversing the TOR switch 314-NVD 310-host machine 302, and a second path traversing the TOR switch 316-NVD 312-host machine 302. These separate paths provide increased availability (referred to as high availability) of the host machine 302. If there is a problem with one of the paths (e.g., one link in the path fails) or one of the devices (e.g., a particular NVD is not functioning), the other path may be used for communications to / from the host machine 302.

[0120] In the configuration shown in Figure 3, the host machine is connected to two different NVDs using two different ports provided by the host machine's NIC. In other embodiments, the host machine may include multiple NICs that enable the host machine's connectivity to multiple NVDs.

[0121] Referring back to Figure 2, an NVD is a physical device or component that implements one or more network and / or storage virtualization functions. An NVD may be any device that has one or more processing units (e.g., a CPU, a network processing unit (NPU), an FPGA, a packet processing pipeline, etc.), memory including cache, and ports. Various virtualization functions may be implemented by software / firmware executed by the one or more processing units of the NVD.

[0122] The NVD may be implemented in a variety of different forms. For example, in one embodiment, the NVD is implemented as an interface card, referred to as a smartNIC or intelligent NIC, that has an on-board processor. A smartNIC is a separate device from the NIC on the host machine. In Figure 2, NVDs 210 and 212 may be implemented as smartNICs connected to host machine 202 and host machines 206 and 208, respectively.

[0123] However, smartNIC is merely one example of an NVD implementation. Various other implementations are possible. For example, in some other embodiments, the NVD or one or more functions performed by the NVD may be incorporated into or performed by one or more host machines, one or more TOR switches, and other components of CSPI 200. For example, the NVD may be embodied within a host machine, where the functions performed by the NVD are performed by the host machine. As another example, the NVD may be part of a TOR switch, or a TOR switch may be configured to perform the functions performed by the NVD that enable the TOR switch to perform various complex packet transformations used in public clouds. A TOR that performs the functions of an NVD may be referred to as a smart TOR. In yet other embodiments where customers are provided with virtual machine (VM) instances rather than bare metal (BM) instances, the functions performed by the NVD may be implemented inside the hypervisor of the host machine. In some other embodiments, some of the NVD's functions may be offloaded to a centralized service running on a fleet of host machines.

[0124] In some embodiments, such as when implemented as a smartNIC as shown in FIG. 2, an NVD may comprise multiple physical ports that allow it to be connected to one or more host machines and one or more TOR switches. Ports on an NVD can be classified as host-facing ports (also referred to as "south ports") or network-facing or TOR-facing ports (also referred to as "north ports"). A host-facing port of an NVD is a port used to connect the NVD to a host machine. Examples of host-facing ports in FIG. 2 include port 236 on NVD 210 and ports 248 and 254 on NVD 212. A network-facing port of an NVD is a port used to connect the NVD to a TOR switch. Examples of network-facing ports in FIG. 2 include port 256 on NVD 210 and port 258 on NVD 212. As shown in FIG. 2, NVD 210 is connected to TOR switch 214 using link 228 extending from port 256 of NVD 210 to TOR switch 214. Similarly, the NVD 212 is connected to the TOR switch 216 using a link 230 that extends from a port 258 of the NVD 212 to the TOR switch 216 .

[0125] The NVD can receive packets and frames from a host machine (e.g., packets and frames generated by a compute instance hosted by the host machine) via a host-side port, perform any necessary packet processing, and then forward the packets and frames to a TOR switch via a network-side port of the NVD. The NVD can receive packets and frames from a TOR switch via a network-side port of the NVD, and perform any necessary packet processing, and then forward the packets and frames to a host machine via a host-side port of the NVD.

[0126] In some embodiments, there may be multiple ports and associated links between the NVD and the TOR switch. These ports and links can be aggregated to form a link aggregator group (referred to as a LAG) of multiple ports or links. Link aggregation allows multiple physical links between two endpoints (e.g., between the NVD and the TOR switch) to be treated as a single logical link. All physical links within a given LAG may operate at the same speed and in full-duplex mode. LAGs help increase the bandwidth and reliability of the connection between two endpoints. If one of the physical links in the LAG fails, traffic is dynamically and transparently reassigned to one of the other physical links in the LAG. The aggregated physical link delivers higher bandwidth than each individual link. Multiple ports associated with a LAG are treated as a single logical port. Traffic can be load-balanced across the multiple physical links of a LAG. One or more LAGs can be configured between two endpoints. The two endpoints may be between the NVD and the TOR switch, between a host machine and the NVD, etc.

[0127] The NVD implements or performs network virtualization functions. These functions are performed by software / firmware executed by the NVD. Examples of network virtualization functions include, but are not limited to, packet encapsulation and decapsulation functions, functions for creating VCN networks, functions for implementing network policies such as VCN security list (firewall) functions, functions for facilitating routing and forwarding of packets to and from compute instances within the VCN, etc. In one embodiment, when a packet is received, the NVD is configured to execute a packet processing pipeline to process the packet and determine how the packet is to be forwarded or routed. As part of this packet processing pipeline, the NVD can execute one or more virtual functions associated with the overlay network, such as running VNICs associated with compute instances within the VCN, running virtual routers (VRs) associated with the VCN, encapsulating and decapsulating packets to facilitate forwarding or routing within the virtual network, running certain gateways (e.g., local peering gateways), implementing security lists, network security groups, network address translation (NAT) functions (e.g., translating per-host public IPs to private IPs), throttling functions, and other functions.

[0128] In one embodiment, the packet processing data path within the NVD may include multiple packet pipelines, each consisting of a series of packet transformation stages. In one embodiment, when a packet is received, it is parsed and classified into a single pipeline. The packet is then processed linearly from one stage to another until the packet is dropped or sent out on an interface of the NVD. These stages provide basic functional packet processing building blocks (e.g., header validation, throttling enforcement, insertion of new Layer 2 headers, L4 firewall enforcement, VCN encapsulation / decapsulation, etc.), such that new pipelines can be constructed by composing existing stages, and new functionality can be added by creating new stages and inserting them into existing pipelines.

[0129] The NVD can perform both control plane and data plane functions corresponding to the control and data planes of a VCN. Examples of a VCN control plane are also shown in Figures 26, 27, 28, and 29 (see reference numbers 2616, 2716, 2816, and 2916) and described below. Examples of a VCN data plane are also shown in Figures 26, 27, 28, and 29 (see reference numbers 2618, 2718, 2818, and 2918) and described below. Control plane functions include functions used to configure the network (e.g., setting up routes and routing tables, configuring VNICs, etc.) that control how data should be forwarded. In one embodiment, a VCN control plane is provided that centrally computes all overlay-to-foundation mappings and publishes them to virtual network edge devices such as the NVD and various gateways such as DRGs, SGWs, IGWs, etc. Firewall rules can also be published using the same mechanism. In one embodiment, the NVD obtains only mappings that are relevant to the NVD. The data plane functions include functionality for the actual routing / forwarding of packets based on the configuration set up using the control plane. The VCN data plane is implemented by encapsulating customer network packets before they traverse the underlying network. The encapsulation / decapsulation functions are implemented on the NVD. In one embodiment, the NVD is configured to intercept all network packets entering and leaving the host machine and perform network virtualization functions.

[0130] As indicated above, the NVD performs various virtualization functions, including VNICs and VCN VRs. The NVD can execute VNICs associated with compute instances hosted by one or more host machines connected to the VNICs. For example, as shown in FIG. 2, NVD 210 executes the functions of VNIC 276 associated with compute instance 268 hosted by host machine 202 connected to NVD 210. As another example, NVD 212 executes VNIC 280 associated with bare metal compute instance 272 hosted by host machine 206 and VNIC 284 associated with compute instance 274 hosted by host machine 208. The host machines can host compute instances that belong to different VCNs that belong to different customers, and the NVDs connected to the host machines can execute VNICs (i.e., perform VNIC-related functions) corresponding to the compute instances.

[0131] NVDs also run VCN virtual routers corresponding to the VCNs of the compute instances. For example, in the embodiment shown in FIG. 2, NVD 210 runs VCN VR 277 corresponding to the VCN to which compute instance 268 belongs. NVD 212 runs one or more VCN VRs 283 corresponding to one or more VCNs to which compute instances hosted by host machines 206 and 208 belong. In one embodiment, a VCN VR corresponding to that VCN is run by all NVDs connected to a host machine that hosts at least one compute instance belonging to that VCN. If a host machine hosts compute instances that belong to different VCNs, the NVDs connected to that host machine may run VCN VRs corresponding to those different VCNs.

[0132] In addition to VNICs and VCN VRs, an NVD may include one or more hardware components that run various software (e.g., daemons) and facilitate various network virtualization functions performed by the NVD. For simplicity, these various components are grouped together as a “packet processing component” shown in FIG. 2 . For example, NVD 210 includes packet processing component 286, and NVD 212 includes packet processing component 288. For example, the packet processing component of an NVD may include a packet processor configured to interact with the NVD's ports and hardware interfaces to monitor all packets received by and communicated using the NVD and store network information. The network information may include, for example, network flow information that identifies different network flows handled by the NVD per flow information (e.g., per flow statistics). In one embodiment, the network flow information may be stored per VNIC. The packet processor may perform per-packet operations and implement a stateful NAT and an L4 firewall (FW). As another example, the packet processing component may include a replication agent configured to replicate information stored by the NVD to one or more different replication target stores. As yet another example, the packet processing component may include a logging agent configured to perform logging functions for the NVD. The packet processing component may also include software for monitoring the performance and health of the NVD, and possibly also the status and health of other components connected to the NVD.

[0133] FIG. 1 illustrates components of an exemplary virtual or overlay network, including a VCN, subnets within the VCN, compute instances deployed on the subnets, VNICs associated with the compute instances, VRs for the VCN, and a set of gateways configured for the VCN. The overlay components illustrated in FIG. 1 may be executed or hosted by one or more of the physical components illustrated in FIG. 2. For example, compute instances in a VCN may be executed or hosted by one or more host machines illustrated in FIG. 2. For compute instances hosted by a host machine, the VNICs associated with the compute instance are typically executed by the NVD connected to the host machine (i.e., the VNIC functionality is provided by the NVD connected to the host machine). The VCN VR functionality for a VCN is performed by all NVDs connected to the host machines that host or execute compute instances that are part of the VCN. The gateways associated with a VCN may be executed by one or more different types of NVDs. For example, some gateways may be executed by smartNICs, while other gateways may be executed by one or more host machines or other implementations of NVDs.

[0134] As described above, a compute instance in a customer VCN can communicate with a variety of different endpoints, where the endpoints can be in the same subnet as the source compute instance, in a different subnet than the source compute instance but in the same VCN, or the endpoints can be outside the VCN of the source compute instance. These communications are facilitated using VNICs associated with the compute instance, VCN VRs, and gateways associated with the VCN.

[0135] For communication between two compute instances on the same subnet within a VCN, the communication is facilitated using VNICs associated with the source and destination compute instances. The source and destination compute instances may be hosted by the same host machine or may be hosted by different host machines. A packet originating from a source compute instance can be forwarded from the host machine hosting the source compute instance to an NVD connected to that host machine. On the NVD, the packet is processed using a packet processing pipeline, which may include running the VNIC associated with the source compute instance. Because the packet's destination endpoint is within the same subnet, running the VNIC associated with the source compute instance results in the packet being forwarded to an NVD running the VNIC associated with the destination compute instance, which then processes the packet and forwards it to the destination compute instance. The VNICs associated with the source and destination compute instances may run on the same NVD (e.g., when both the source and destination compute instances are hosted by the same host machine) or on different NVDs (e.g., when the source and destination compute instances are hosted by different host machines connected to different NVDs). The VNIC can use the routing / forwarding tables stored by the NVD to determine the next hop for the packet.

[0136] To allow packets to be communicated from a compute instance in a subnet to an endpoint in a different subnet within the same VCN, packets originating from a source compute instance are communicated from a host machine hosting the source compute instance to an NVD connected to that host machine. On the NVD, the packets are processed using a packet processing pipeline that may include the execution of one or more VNICs and VRs associated with the VCN. For example, as part of the packet processing pipeline, the NVD executes or invokes functions corresponding to the VNICs associated with the source compute instance (also referred to as executing VNICs). The functions performed by the VNICs may include looking at the VLAN tag on the packet. Because the packet destination is outside the subnet, VCN VR functions are then invoked and executed by the NVD. The VCN VR then routes the packet to an NVD that executes the VNIC associated with the destination compute instance. The VNIC associated with the destination compute instance then processes the packet and forwards it to the destination compute instance. The VNICs associated with the source and destination compute instances may run on the same NVD (e.g., when both the source and destination compute instances are hosted by the same host machine) or may run on different NVDs (e.g., when the source and destination compute instances are hosted by different host machines connected to different NVDs).

[0137] If the packet's destination is outside the VCN of the source compute instance, the packet originating from the source compute instance is communicated from the host machine hosting the source compute instance to the NVD connected to that host machine. The NVD runs a VNIC associated with the source compute instance. Because the packet's destination endpoint is outside the VCN, the packet is then processed by the VCN VR for that VCN. The NVD invokes a VCN VR function, which results in the packet being forwarded to an NVD running the appropriate gateway associated with the VCN. For example, if the destination is an endpoint in a customer's on-premises network, the packet can be forwarded by the VCN VR to an NVD running a DRG gateway configured for the VCN. The VCN VR may be run on the same NVD as the NVD running the VNIC associated with the source compute instance or by a different NVD. The gateway may be run by the NVD, which may be a smartNIC, a host machine, or another NVD implementation. The packet may then be processed by the gateway and forwarded to the next hop, which facilitates communication of the packet to its final intended destination. 2, a packet originating from compute instance 268 may be communicated from host machine 202 over link 220 (using NIC 232) to NVD 210. On NVD 210, VNIC 276 is called out because it is the VNIC associated with source compute instance 268. VNIC 276 is configured to examine encapsulation information within the packet, determine a next hop to forward the packet to with the goal of facilitating communication of the packet to its intended destination endpoint, and then forward the packet to the determined next hop.

[0138] Compute instances deployed on a VCN can communicate with a variety of different endpoints. These endpoints may include endpoints hosted by CSPI 200 and endpoints external to CSPI 200. Endpoints hosted by CSPI 200 may include instances in the same VCN or another VCN, which may be a customer VCN or a VCN not belonging to the customer. Communication between endpoints hosted by CSPI 200 may be conducted over physical network 218. Compute instances may also communicate with endpoints not hosted by CSPI 200 or external to CSPI 200. Examples of these endpoints include endpoints in a customer's on-premises network or data center, or public endpoints accessible over a public network such as the Internet. Communication with endpoints external to CSPI 200 may be conducted over a public network (e.g., the Internet) (not shown in FIG. 2) or a private network (not shown in FIG. 2) using various communication protocols.

[0139] The architecture of CSPI 200 shown in FIG. 2 is illustrative only and is not intended to be limiting. Variations, substitutions, and modifications are possible in alternative embodiments. For example, in some other embodiments, CSPI 200 may have more or fewer systems or components than those shown in FIG. 2, may combine two or more systems, or may have a different configuration or arrangement of systems. The systems, subsystems, and other components shown in FIG. 2 may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective systems, using hardware, or a combination thereof. The software may be stored on a non-transitory storage medium (e.g., on a memory device).

[0140] FIG. 4 illustrates connectivity between host machines and an NVD to provide I / O virtualization to support multitenancy functionality according to one embodiment. As shown in FIG. 4, a host machine 402 runs a hypervisor 404 that provides a virtualized environment. The host machine 402 runs two virtual machine instances: VM1 406 belonging to customer / tenant #1 and VM2 408 belonging to customer / tenant #2. The host machine 402 includes a physical NIC 410 connected to an NVD 412 via link 414. Each of the compute instances is attached to a VNIC implemented by the NVD 412. In the embodiment of FIG. 4, VM1 406 is attached to VNIC-VM1 420, and VM2 408 is attached to VNIC-VM2 422.

[0141] 4, NIC 410 includes two logical NICs: logical NIC A 416 and logical NIC B 418. Each virtual machine is attached to and configured to work with its own logical NIC. For example, VM1 406 is attached to logical NIC A 416, and VM2 408 is attached to NIC B 418. Although host machine 402 includes only one physical NIC 410 shared by multiple tenants, due to the logical NICs, each tenant's virtual machine believes it has its own host machine and NIC.

[0142] In one embodiment, each logical NIC is assigned its own VLAN ID. Thus, a particular VLAN ID is assigned to logical NIC A 416 for tenant #1, and a separate VLAN ID is assigned to logical NIC B 418 for tenant #2. When a packet is communicated from VM1 406, a tag assigned to tenant #1 is attached to the packet by the hypervisor, and the packet is then communicated from host machine 402 to NVD 412 over link 414. Similarly, when a packet is communicated from VM2 408, a tag assigned to tenant #2 is attached to the packet by the hypervisor, and the packet is then communicated from host machine 402 to NVD 412 over link 414. Thus, a packet 424 communicated from host machine 402 to NVD 412 has an associated tag 426 that identifies the particular tenant and associated VM. On the NVD, for a packet 424 received from host machine 402, a tag 426 associated with the packet is used to determine whether the packet should be processed by VNIC-VM1 420 or VNIC-VM2 422. The packet is then processed by the corresponding VNIC. The configuration shown in Figure 4 allows each tenant's compute instances to think that they own their own host machine and NIC. The setup shown in Figure 4 provides I / O virtualization to support multi-tenancy.

[0143] FIG. 5 illustrates a simplified block diagram of a physical network 500 according to one embodiment. The embodiment illustrated in FIG. 5 is structured as a Clos network. A Clos network is a specific type of network topology designed to provide connection redundancy while maintaining high bisection bandwidth and maximum resource utilization. A Clos network is a type of non-blocking, multi-stage or multi-tier switching network, and the number of stages or tiers can be 2, 3, 4, 5, etc. The embodiment illustrated in FIG. 5 is a three-tier network with tiers 1, 2, and 3. TOR switch 504 represents a tier 0 switch in the Clos network. One or more NVDs are connected to the TOR switch. A tier 0 switch is also referred to as an edge device of the physical network. A tier 0 switch is connected to a tier 1 switch, also referred to as a leaf switch. In the embodiment illustrated in FIG. 5, a set of “n” tier 0 TOR switches is connected to a set of “n” tier 1 switches, together forming a pod. Each tier 0 switch in a pod is interconnected to all tier 1 switches within the pod, but there is no switch connectivity between pods. In one embodiment, two pods are referred to as blocks. Each block is served by or connected to a set of “n” tier-2 switches (sometimes referred to as spine switches). There may be several blocks in a physical network topology. The tier-2 switches are then connected to “n” tier-3 switches (sometimes referred to as super-spine switches). Communication of packets over the physical network 500 is typically performed using one or more layer-3 communication protocols. Typically, all layers of the physical network except the TOR layer are n-way redundant, thus enabling high availability. Policies can be specified for pods and blocks to control the visibility of switches in the physical network to each other to enable scaling of the physical network.

[0144] A feature of Clos networks is that the maximum hop count required to reach from one tier-0 switch to another tier-0 switch (or from an NVD connected to a tier-0 switch to another NVD connected to a tier-0 switch) is fixed. For example, in a three-tier Clos network, a packet requires a maximum of seven hops to reach from one NVD to another, where the source and destination NVDs are connected to the leaf tiers of the Clos network. Similarly, in a four-tier Clos network, a packet requires a maximum of nine hops to reach from one NVD to another, where the source and destination NVDs are connected to the leaf tiers of the Clos network. Therefore, the Clos network architecture maintains consistent latency throughout the network, which is important for communication within and between data centers. Clos topologies are horizontally scalable and cost-efficient. The network's bandwidth / throughput capacity can be easily increased by adding more switches (e.g., more leaf and spine switches) to various tiers and by increasing the number of links between switches in adjacent tiers.

[0145] In one embodiment, each resource in CSPI is assigned a unique identifier called a Cloud Identifier (CID). This identifier is included as part of the resource's information and can be used to manage the resource, for example, via a console or through an API. An exemplary syntax for a CID is as follows:

[0146] ocid1.<RESOURCE TYPE> . <realm>.[REGION][.FUTURE USE].<UNIQUE ID> where: ocid1: A string indicating the version of the CID.

[0147] resource type: The type of resource (for example, instance, volume, VCN, subnet, user, group, etc.).

[0148] realm: The realm the resource is in. Example values ​​are "c1" for a commercial realm, "c2" for a government cloud realm, or "c3" for a federal cloud realm. Each realm can have its own domain name.

[0149] region: The region the resource is in. This part can be blank if no region is applicable to the resource.

[0150] future use: Reserved for future use. unique ID: The unique part of the ID. The format may vary depending on the type of resource or service.

[0151] Virtual Private Label Cloud (vPLC) Techniques are described herein that enable cloud infrastructure within a region provided by a cloud service provider (CSP) to be used to create one or more virtual private clouds (referred to herein as virtual private label clouds or "vPLCs") for providing cloud services supplied by the CSP to customers of the CSP. A vPLC created according to the various techniques described herein can be used for a variety of purposes. For example, in one use case, a vPLC may be created for a reseller who is a customer of the CSP and used by the reseller to provide one or more reseller-supplied cloud services to the reseller's customers. As another use case, a vPLC may be used as a virtual data center and associated with a realm different from the realm associated with cloud infrastructure within a region provided by the CSP.

[0152] A cloud service provider (CSP) provides infrastructure (referred to as cloud service provider infrastructure or CSPI or CSP-delivered infrastructure) used to deliver one or more cloud services to the CSP's customers. CSP customers can include the CSP's direct customers and / or resellers. CSPI can include physical infrastructure components and virtual infrastructure components. Physical CSPI can include interconnected high-performance computing resources, including racks, servers and host machines, memory resources, and network resources used to form a physical foundation or underlay network, network virtualization devices, and other resources. Virtual CSPI components may include components in an overlay network, such as virtual machines, virtual routers and gateways, and other virtual components.

[0153] A CSP may provide various types of cloud services. These may include various types of services, including software as a service (SaaS), platform as a service (PaaS), infrastructure as a service (IaaS), etc. In one embodiment, as described in this disclosure, a CSP may provide a "vPLC cloud service." A customer of a CSP may subscribe to one or more cloud services offered by the CSP. A customer may be any entity, such as an individual, an organization, a business, etc.

[0154] Infrastructure as a Service (IaaS) is a specific type of cloud computing service. In the IaaS model, a CSP provides infrastructure that customers can use to build their own customizable networks and deploy customer resources. Therefore, customer resources and networks are hosted in a distributed environment by infrastructure provided by the CSP. This differs from traditional computing, in which customer resources and networks are hosted by infrastructure provided by the customer. In one embodiment, a vPLC as a Service is a type of IaaS service in which infrastructure is allocated to create a virtual private label cloud for a customer. For example, if a CSP customer is a reseller, the vPLC created for the reseller can be used to provide a reseller-supplied cloud service.

[0155] When a customer subscribes or registers for a cloud service offered by a CSP, a tenancy (or account) is created for that customer. The customer can then access one or more subscribed cloud resources associated with the account / tenancy through this account / tenancy. Each tenancy is identified by a tenancy identifier (T_ID) that is assigned to the tenancy and uniquely identifies the tenancy. The tenancy identifier is generated and associated with the tenancy when the tenancy is created. For example, when a direct customer of a CSP subscribes to a service offered by the CSP, a tenancy is created for the customer, and a tenancy identifier that uniquely identifies the tenancy is created for the direct customer. When a reseller subscribes to a vPLC service, a tenancy is created for the reseller, and a tenancy identifier that uniquely identifies the tenancy is created for the reseller. A tenancy can be a logical and secure partition that contains all resources and services utilized by a customer associated with the tenancy.

[0156] Resources within a CSP-provided infrastructure are typically identified using a unique resource identifier (resource ID or RID). Because these resources are provided within the cloud and used to provide cloud services, resource IDs are also referred to as cloud resource identifiers (or cloud IDs or CIDs). For example, in the cloud environment provided by Oracle Corporation, resources are identified using an "ocid" or "Oracle cloud identifier." In some embodiments, a resource ID is a globally unique identifier string that identifies a cloud resource. In some embodiments, a tenancy ID is one type of cloud resource ID (or referred to as a cloud identifier (CID)).

[0157] A resource or cloud ID (CID) may be represented in the following format, as an example:

[0158] RID_version. <resourcetype> . <realm>.[REGION][.FUTURE USE].<UNIQUE ID> where: RID_version - Refers to the version of the resource ID.

[0159] "resource type" - Identifies the particular type of resource that the resource ID refers to. Examples of different resource types include instances, virtual network subnets, tenancies, etc. For example, an "instance RID" identifies a compute instance.

[0160] "realm" and "region" - Indicates the location (realm and region) of the resource.

[0161] "unique ID" - The unique part of the ID (e.g., a unique string to identify a specific resource). Thus, the tenancy ID can identify a unique ID within a CID's tenancy resource type.

[0162] CIDs are generated for resources when those resources are created. Virtual machines, accounts, tenancies, authentication policies, virtual network subnets, access control lists (ACLs), and load balancers are examples of cloud resources and can therefore be referenced by their respective CIDs. A tenancy is a logical container used to group other cloud resources that belong to the same cloud customer account.

[0163] When an entity becomes a customer of a CSP and subscribes to a vPLC service, a vPLC is created for the customer and associated with the customer's tenancy. For example, if the customer is a reseller, a vPLC is created for the reseller and associated with the reseller's tenancy. A vPLC created for a reseller is treated as a resource and assigned a unique resource identifier, referred to as a vPLC identifier (vPLC ID), and the vPLC ID is associated with the reseller's tenancy. As described herein, for a vPLC created for a reseller, the vPLC ID of the vPLC is used to identify all resources and requests related to the vPLC. Each vPLC ID uniquely identifies a vPLC.

[0164] For example, in one embodiment, all resources allocated to a vPLC are tagged with the vPLC identifier of that vPLC. In one embodiment, a vPLC ID field is inserted into the representation of the CID described above. When a resource is associated with a vPLC, the vPLC ID corresponding to that vPLC may be provided in this field of the CID; otherwise, if the resource is not associated with a vPLC, the field may be left blank.

[0165] For the purposes of this disclosure, the following terminology will be used for the sake of clarity. CSP.Cn (or simply Cn) - identifies a direct, non-reseller customer of the CSP. For example, "CSP.C1" refers to the CSP's non-reseller direct customer C1, and "CSP.C2" refers to the CSP's non-reseller direct customer C2. CSP.Rn (or simply Rn) - identifies a CSP's reseller customer. For example, "CSP.R1" refers to CSP's reseller customer R1, and "CSP.R2" refers to CSP's reseller customer R2. Rn.Cm - Resellers can have their own customers. This identifies a specific customer of a specific reseller. For example, "R1.C1" refers to customer C1 of reseller R1, and "R2.C1" refers to customer C1 of reseller R2. T.Cn - Identifies the tenancy for a CSP's reseller customer. For example, "T.C1" refers to the tenancy for direct non-reseller customer C1. T.Rn - Identifies the tenancy for the reseller. For example, "T.R1" refers to the tenancy for reseller R1. T.Rn.Cm - Each reseller's customer has their own tenancy. This identifies the tenancy associated with a particular customer of a particular reseller. For example, "T.R1.C1" refers to the tenancy associated with reseller R1's customer C1, and "T.R2.C1" refers to the tenancy associated with reseller R2's customer C1. T_ID.T.Cn - Identifies the tenancy identifier that identifies the tenancy for a CSP's non-reseller customer. For example, "T_ID.T.C1" refers to the tenancy ID of the tenancy for direct non-reseller customer C1. T_ID.T.Rn - Identifies the tenancy identifier of the tenancy for the reseller. For example, "T_ID.T.R1" refers to the tenancy identifier of the tenancy for reseller R1. T_ID.Rn.Cm - Identifies the tenancy identifier of the tenancy associated with a particular customer of a particular reseller. For example, "T_ID.T.R1.C1" refers to the tenancy identifier of the tenancy associated with customer C1 of reseller R1, and "T_ID.T.R2.C1" refers to the tenancy identifier of the tenancy associated with customer C1 of reseller R2. vPLC.Rn - Identifies a vPLC being created for reseller Rn. For example, "vPLC.R1" identifies the tenancy being created for and associated with reseller R1. This identifies a specific vPLC being created for and associated with the reseller's tenancy. vPLC_ID.vPLC.Rn - Identifies the vPLC identifier of a vPLC being created for a specific reseller. For example, "vPLC_ID.vPLC.R1" refers to the vPLC identifier of a vPLC being created for reseller R1.

[0166] As described herein, a vPLC is created for a reseller, who is a customer of a CSP. The vPLC is associated with the reseller's tenancy. In some embodiments, a vPLC ID, which identifies the vPLC, may be associated with a tenancy ID, which identifies the reseller's tenancy. Thus, given a vPLC ID, the tenancy and tenancy identifier for the reseller can be included to identify the corresponding vPLC and the reseller for whom the vPLC is being created.

[0167] CSPI or CSP-provided infrastructure may be organized into realms, regions, and data centers. A region refers to a local geographic area that includes one or more connected data centers. Regions are independent from other regions and can be separated by large distances, for example, across countries or even continents. A set of application programming interfaces (APIs) is provided for the CSP-provided infrastructure in a region, and the APIs are used to perform operations using the infrastructure in that region. Operations that can be performed using the infrastructure in a region may include creating resources in the infrastructure in that region, accessing resources in the infrastructure in the region, performing operations involving resources in the infrastructure in the region, deleting resources from the infrastructure in the region, and performing other operations involving the infrastructure in the region.

[0168] The set of APIs is region-specific. Thus, the CSP-provided infrastructure in a region is characterized by the set of APIs that can be used with the infrastructure in that region. CSP-provided infrastructure in different regions has different API sets provided for the different regions. Therefore, for purposes of this application, the phrase “CSP-provided infrastructure in a region” or “CSP-provided infrastructure in a region” means that the infrastructure is characterized by the specific API set provided for that region. Thus, a first API set may be provided for a first region, a second API set may be provided for a second region different from the first region, a third API set may be provided for a third region different from the first and second regions, and so on. Thus, a specific API set identifies the CSP-provided infrastructure in a particular region. In other words, CSP-provided infrastructure in two separate regions will have two separate API sets. Thus, a region is characterized by the infrastructure in the region and the set of APIs associated with the region for performing operations involving the region infrastructure. CSP-provided infrastructure within a region is also referred to as "region infrastructure" or "region-specific infrastructure."

[0169] For example, a CSP may provide a first infrastructure in Seattle and a second infrastructure in Portland. A first set of APIs may be provided for the first infrastructure in Seattle, and these APIs may be characterized by the following format:

[0170] "<service_identifier> .us-seattle.oci.Oraclecloud.com” A second set of APIs may be provided for a second infrastructure in Portland, and these APIs may be characterized by the following format:

[0171] "<service_identifier> .us-portland.oci.Oraclecloud.com” Note that Seattle and Portland are used here only as examples. The geographic location of the regions can vary. For example, it is possible to have two or more regions within the same geographic city. For example, it is possible to have a first region in a first building within a city and a second region in a second, separate building within the same city. As yet another example, it is possible to have a first region on one floor within a building and a second region on another floor of the building. As yet another example, for infrastructure provided by a CSP on a floor, a first portion of the infrastructure may belong to a first region associated with a first set of APIs and a second portion of the infrastructure may belong to a second region associated with a second set of APIs that is different from the first set of APIs.

[0172] CSP-provided infrastructure within a region may be organized into one or more datacenters. For example, in a particular region where a CSP may have two buildings, each hosting infrastructure used to provide cloud services, the infrastructure in the first building may be organized into a first datacenter and referred to as the first datacenter, and the infrastructure in the second building may be organized into a second datacenter and referred to as the second datacenter. As another example, in a particular region where a CSP may have a single building with multiple floors, each floor hosting infrastructure used to provide cloud services, the infrastructure on the first floor may be organized into a first datacenter and referred to as the first datacenter, and the infrastructure on the second floor may be organized into a second datacenter and referred to as the second datacenter. As yet another example, in a particular region where a CSP may have infrastructure used to provide cloud services, a first portion of the infrastructure may be organized into a first datacenter and referred to as the first datacenter, and a second separate portion of the infrastructure may be organized into a second datacenter and referred to as the second datacenter.

[0173] A region may have one or more data centers. In some embodiments, all regional infrastructure may be contained within a single data center, while in other embodiments, the infrastructure within a region may be organized into multiple data centers. Each data center may include infrastructure resources, such as compute, storage, and networking resources provided by a CSP. Within a region, the data centers within the region may be organized into one or more availability domains (ADs). Availability domains are isolated from each other, fault-tolerant, and highly unlikely to fail simultaneously. ADs are configured so that a failure in one AD within a region is unlikely to affect the availability of other ADs in the same region.

[0174] A realm refers to a logical collection of one or more regions. A realm can contain one or more regions. Realms are typically isolated from one another and do not share data. Each realm has its own identity and trust profile (e.g., passwords, credentials, etc.). The realm's identity and trust profile are configured so that cross-realm communication is not enabled. In this manner, each realm represents an isolated domain. For example, one realm may be created for a government agency that is a customer of the CSP, another realm may be created for a commercial customer of the CSP, yet another realm may be created for a private organization customer of the CSP, and so on. For regions within a realm, the region infrastructure can communicate with each other, but the region infrastructure associated with a first realm is not allowed to communicate with the region infrastructure associated with a second, different realm.

[0175] A cloud service reseller can be an entity such as a corporation (eg, a telecommunications company), a systems integrator, a government agency, an IT department of a large corporation, a school, and the like.

[0176] FIG. 6 is a simplified block diagram of a distributed environment 600 illustrating an example of a virtual private label cloud (vPLC) that includes a CSP-provided infrastructure within a region and is hosted by the CSP-provided infrastructure, according to an embodiment. The distributed environment 600 illustrated in FIG. 6 is merely exemplary and is not intended to unduly limit the scope of the claimed embodiments. Many variations, alternatives, and modifications are possible. For example, in some other embodiments, the distributed environment 600 may have more or fewer systems or components than those shown in FIG. 6, may combine two or more systems, or may have a different configuration or arrangement of systems. The systems, subsystems, and other components illustrated in FIG. 6 may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective systems, using hardware, or a combination thereof. The software may be stored on a non-transitory storage medium (e.g., on a memory device).

[0177] Figure 6 depicts a CSP-provided infrastructure within a region 601. The infrastructure 601 may be organized into one or more data centers. The infrastructure 601 may include memory or storage resources 622, computational resources 620, networking resources (not shown), and other physical and logical resources provided by the CSP. The storage resources 622 may include one or more volatile or non-volatile memory resources. The computational resources 620 may include one or more racks, each rack including one or more servers, each server hosting one or more operating systems.

[0178] The CSP-provided region infrastructure 601 may be used by a CSP to offer one or more CSP-supplied cloud services to one or more of the CSP's direct customers, and may also be used to create one or more vPLCs. In the example depicted in FIG. 6, two vPLCs, vPLC.R1 604 and vPLC.R2 606, have been created for resellers R1 and R2, respectively. vPLC.R1 604 is used to offer cloud services supplied by one or more resellers R1 and branded by R1 to one or more customers 642 of R1. vPLC.R2 606 is used to offer cloud services supplied by one or more resellers R2 and branded by R2 to one or more customers of R2 644.

[0179] As shown, infrastructure 601 can be communicatively coupled to computing devices used by users associated with direct customers 640 and users associated with resellers' customers (e.g., users associated with reseller R1's customers 642 and users associated with reseller R2's customers 644) via communications network 652. Communications network 652 can be of various types and can include one or more communications networks. Examples of communications network 652 include, without limitation, the Internet, a wide area network (WAN), a local area network (LAN), an Ethernet network, a public or private network, a wired network, a wireless network, etc., and combinations thereof. Different communications protocols may be used to facilitate communications, including both wired and wireless protocols such as the IEEE 802.XX protocol suite, TCP / IP, IPX, SAN, AppleTalk, Bluetooth, and other protocols. In general, communications network 652 may include any infrastructure that facilitates communications with infrastructure 601.

[0180] 6, infrastructure 601 is divided into three securely isolated portions: (1) a first portion 602 used to provide CSP-supplied and CSP-branded cloud services to one or more of the CSP's direct (or non-reseller) customers 640; (2) a second portion allocated to vPLC.R1 604 created for reseller R1; and (3) a third portion allocated to vPLC.R2 606 created for reseller R2. In one embodiment, these three portions are securely separated from each other.

[0181] The partitioning of resources includes partitioning of storage resources 622 and computational resources 620. For example, as shown in Figure 6, storage resources 622 include a memory portion 622a included in first portion 602 that is used to provide CSP-delivered cloud services to the CSP's direct customers. Storage resources 622 also include a memory portion 622b allocated to vPLC.R1 604 and a memory portion 622c allocated to vPLC.R2 606. The memory or memory portion allocated to a vPLC may include one or more volatile and / or non-volatile memory resources.

[0182] 6, the computational resources 620 are also partitioned. The computational resources 620 include a computational portion 620a included in the first portion 602 that is used to provide the CSP-delivered cloud services to the CSP's direct customers. The computational resources 620 also include a portion 620b allocated to vPLC.R1 604 and a portion 620c allocated to vPLC.R2 606.

[0183] As described above, a set of APIs are provided for accessing and implementing functions related to the CSP-provided infrastructure within the region. These APIs can be accessed by users associated with the CSP's direct customers 640 using a set of endpoints 610-1 provided by the CSP. For example, one or more of the endpoints 610-1 may be used by users to access and / or use compute resources 620a or memory resources 622a allocated to the infrastructure used to provide CSP-supplied cloud services to the CSP's customers. The 610-1 endpoints may point to URLs with associated fully qualified domain names (FQDNs) used to access cloud services provided by the infrastructure 601. Examples of these endpoints are:

[0184] "createVM.compute.us-seattle.oci.Oraclecloud.com" - for creating VMs, "deleteVM.compute.us-seattle.oci.Oraclecloud.com" - for deleting a VM, "storeObject.storage.us-seattle.oci.Oraclecloud.com" - for storing data, etc.

[0185] Users associated with the CSP's direct customers 640 and connected to the infrastructure 601 via a communications network may use one or more of the endpoints 610-1.

[0186] In one embodiment, a set of vPLC-specific endpoints is created for each vPLC. For example, as depicted in FIG. 6 , a set of endpoints 612-1 is created for vPLC.R1 and a set of endpoints 614-1 is created for vPLC.R2. Endpoint 612-1 can be used by users associated with customer 642 of reseller R1 to access resources associated with vPLC.R1 and to perform one or more functions or operations involving resources allocated to vPLC.R1. Endpoint 614-1 can be used by users associated with customer 644 of reseller R2 to access resources associated with vPLC.R2 and to perform one or more functions or operations involving resources allocated to vPLC.R2.

[0187] In certain embodiments, a set of vPLC-specific endpoints for a vPLC are created based on endpoints provided by the CSP for its direct customers. For example, in FIG. 6, endpoint 612-1 for vPLC.R1 is created based on endpoint 610-1. Endpoints created for a vPLC may include an identifier that specifically identifies the vPLC for which the endpoint is created. For example, given the exemplary endpoints provided by the CSP for its direct customers, endpoint 612-1 for vPLC.R1 is: "createVM.compute.vplcR1.us-seattle.oci.Oraclecloud.com" - for creating a VM, "deleteVM.compute.vplcR1.us-seattle.oci.Oraclecloud.com" - for deleting a VM, "storeObject.storage.vplcR1.us-seattle.oci.Oraclecloud.com" - for storing data, etc., where "vplcR1" uniquely identifies the vPLC corresponding to the endpoint.

[0188] An endpoint 614-1 for vPLC.R2 may also be created based on endpoint 610-1. For example, endpoint 614-1 for vPLC.R2 may be created based on endpoint 610-1. "createVM.compute.vplcR2.us-seattle.oci.Oraclecloud.com" - for creating VMs, "deleteVM.compute.vplcR2.us-seattle.oci.Oraclecloud.com" - for deleting a VM, "storeObject.storage.vplcR2.us-seattle.oci.Oraclecloud.com" - for storing data, etc., where "vplcR2" uniquely identifies the vPLC corresponding to the endpoint.

[0189] Because the endpoints associated with a vPLC include an identifier that identifies the particular vPLC, when an API call is received by infrastructure 601, the endpoint call can be analyzed to determine whether the call is for a direct customer of the CSP or for a reseller. Furthermore, because the endpoints use unique identifiers for different vPLCs, the particular vPLC and associated reseller can also be identified from the endpoint call. In this manner, based on the results of the analysis of the endpoint call, the endpoint request can be appropriately forwarded for processing. The CSP also provides a console 610-2 for use by its direct customers 640. Users associated with the CSP's direct customers 640 can use the console 610-2 to configure, access, and manage the first portion 602 of the CSP-provided region infrastructure. In some embodiments, the console 610-2 can provide a set of web-based graphical user interfaces (GUIs) that can be used to access and manage the CSP-provided cloud infrastructure in the region. In some embodiments, the console is a web-based application.

[0190] In one embodiment, a console is also provided for each vPLC. For example, as depicted in FIG. 6 , console 612-2 is provided for vPLC.R1 and console 614-2 is provided for vPLC.R2. The console 612-2 provided for vPLC.R1 can be used by a user associated with a customer 642 of reseller R1 to configure, access, and manage a second portion of the CSP-provided region infrastructure allocated to vPLC.R1 604 created for reseller R1. The console 614-2 provided for vPLC.R2 can be used by a user associated with a customer 644 of reseller R2 to configure, access, and manage a third portion of the CSP-provided region infrastructure allocated to vPLC.R2 606 created for reseller R2.

[0191] In some embodiments, a reseller may join two or more vPLCs, but each vPLC may still have an endpoint set and a console. For example, if a reseller joins two vPLCs, vPLC1 and vPLC2, vPLC1 may have an endpoint set (e.g., 612-1) and a console (e.g., 612-2), and vPLC2 (e.g., 614-1) may have another endpoint set and a console (e.g., 614-2). Both vPLC1 (e.g., 604) and vPLC2 (606) may be associated with the same reseller.

[0192] The infrastructure 601 may have networking resources that enable communication to and from the infrastructure 601. These network resources may include, for example, a network gateway 628 that enables traffic to and from the CSP provided region infrastructure 601 to be communicated to other destinations, which may be communicatively coupled to the infrastructure 601 via a public (e.g., the Internet) or private network, which may be within an on-premises network, within another cloud network, etc. The gateway 628 may be implemented as a physical network device (e.g., a router), a logical networking component (e.g., a virtual router), or some combination of physical and logical components.

[0193] The infrastructure 601 may include a control plane (CP) 624 and a management plane (MP) 626 provided by the CSP. In some embodiments, the CP 624 may be responsible for providing management, deployment, and orchestration functions within the cloud environment provided by the CSP. For example, the CP may receive requests from the CSP's direct customers, reseller R1's customers, and reseller R2's customers through endpoints and consoles, collaborate with other components in the infrastructure, such as resource managers and management planes, determine whether the requests can be fulfilled, and respond accordingly to the CSP's direct customers, reseller R1's customers, and reseller R2's customers. In other words, the CSP CP 624 may be responsible for performing vPLC-related functions. In some embodiments, the CSP may also provide a vPLC-specific control plane that is specifically configured to perform vPLC operations. In some embodiments, there may be one vPLC control plane that can perform processing for multiple vPLCs. In other embodiments, each vPLC may be provided with its own CP.

[0194] In some embodiments, MP 626 may be a collection of software components responsible for provisioning all the processes and workflows required for a requested operation. For example, for a service request to access a cloud service, the MP may provision a workflow with an identity check on the requestor that checks DNS records to obtain an IP address for accessing the service backend. The MP ensures that various tasks are performed appropriately and in the correct order by the data plane.

[0195] In one embodiment, the computing resources 620 may include one or more racks, each rack including one or more servers, each server hosting one or more operating systems. The division of the computing resources 620 into resources 620a allocated to infrastructure for serving the CSP's direct customers 640 and resources 620b and 620c allocated to vPLCs may occur along various levels or boundaries. The division of computing resources at a particular level may also be referred to herein as a segmentation level. In one embodiment, the division may be performed at the rack level (i.e., segmented at the rack level), with one or more individual racks allocated to 620a, 620b, and 620c. In this embodiment, the rack is allocated exclusively to a particular vPLC and is not used by other vPLCs or by the CSP to provide CSP-delivered services to its direct customers 640.

[0196] In another embodiment, the partitioning may be performed at the server level (i.e., segmentation may be performed at the server level), with one or more individual servers from the same or different racks being allocated to 620a, 620b, and 620c. In this embodiment, the servers are allocated exclusively to a particular vPLC and are not used by other vPLCs or by a CSP to provide CSP-supplied services to its direct customers 640. A rack may have multiple servers, so that one server on the rack may be allocated to 620a, another server to 620b, and a third server to 620c. Thus, a rack may be shared between two vPLCs, or may be shared between a vPLC and infrastructure used by a CSP to provide CSP-supplied services to its direct customers 640.

[0197] In some embodiments, the partitioning may be performed at the hypervisor level (i.e., segmentation may be performed at the hypervisor level), with one or more individual hypervisors from the same or different servers in the same or different racks being allocated to 620a, 620b, and 620c. In this embodiment, the hypervisors are allocated exclusively to a particular vPLC and are not used by other vPLCs or by a CSP to provide CSP-supplied services to its direct customers 640. Because a server can host multiple hypervisors, one hypervisor on a server may be allocated to 620a, another hypervisor on the same server may be allocated to 620b, and a third hypervisor on the same server may be allocated to 620c. Thus, a server may be shared between two vPLCs or may be shared between a vPLC and infrastructure used by a CSP to provide CSP-supplied services to its direct customers 640.

[0198] A hypervisor on a server on a rack can manage and run multiple virtual machines (VMs). In some embodiments, the division may be performed at the VM level (i.e., segmentation may be performed at the VM level), with VMs running by the same hypervisor being allocated to 620a, 620b, and 620c. In this embodiment, a VM is allocated exclusively to a particular vPLC and is not used by other vPLCs or by a CSP to provide CSP-supplied services to its direct customers 640. Because the hypervisor can support multiple VMs, one VM may be allocated to 620a, another VM may be allocated to 620b, and a third VM on the same hypervisor may be allocated to 620c. Thus, a hypervisor may be shared between two vPLCs or may be shared between a vPLC and infrastructure used by a CSP to provide CSP-supplied services to its direct customers 640.

[0199] In still other embodiments, a combination of the above partitioning techniques may be used. Regardless of the partitioning technique used, the partitioning is done in a secure and safe manner that ensures that traffic and storage intended for a particular vPLC for a particular reseller cannot be seen or accessed by other resellers or their customers, or other direct customers of the CSP. Also, traffic and storage intended for a particular customer of a reseller cannot be seen or accessed by other customers of that reseller.

[0200] Figure 7 illustrates the relationships between a CSP, the CSP's direct and reseller customers and their associated users, and individual reseller customers and their associated users, according to one embodiment. As shown in Figure 7, CSP 710 is at the top or root of the hierarchy. CSP 710 can supply one or more cloud services to which the CSP's customers may subscribe. In Figure 7, the customers of CSP 710 are represented by 702. A tenancy, identified by a tenancy ID, is created for each subscribing customer of CSP 710.

[0201] Customers 702 of a CSP 710 may include direct or non-reseller customers and reseller customers. A reseller customer is an entity that subscribes to a vPLC service offered by a CSP, and the vPLC created for the reseller as a result of subscribing to the vPLC service is used to provide reseller-supplied and reseller-branded cloud services to the reseller's customers. A reseller's customers may be different from the CSP's customers. A direct or non-reseller customer of a CSP is an entity that subscribes to one or more CSP-supplied cloud services, but does not utilize the CSP's infrastructure to provide any cloud services of its own to its customers.

[0202] 7, the direct or non-reseller customers of the CSP 710 include customer C1 726 with associated tenancy T.CSP.C1 and customer C2 728 with associated tenancy T.CSP.C2. There may be multiple such direct customers of the CSP 710. One or more users may be each direct customer of the CSP, and the users may use one or more of the CSP-supplied cloud services to which the corresponding direct customer subscribes. For example, a user associated with a customer of the CSP may be an employee or agent of the customer.

[0203] As shown in Figure 7, reseller customers of a CSP 710 include reseller R1 720 with associated tenancy T.CSP.R1 and reseller R2 722 with associated tenancy T.CSP.R2. There may be multiple such reseller customers of a CSP 710. One or more users (not shown in Figure 7) may be each reseller customer of the CSP.

[0204] A reseller may have its own customers who subscribe to one or more reseller-supplied and reseller-branded cloud services, which are delivered using a vPLC created by the CSP for the reseller. In Figure 7, customers of reseller R1 720 are illustrated using reference number 704, and customers of reseller R2 722 are illustrated using reference number 706.

[0205] A reseller's customers may have their own tenancies and associated sets of users. For example, as shown in FIG. 7 , reseller R1 may have multiple customers, including customer R1.C1 740 with tenancy T.R1.C1 and associated users, customer R1.C2 742 with tenancy T.R1.C2 and associated users, and so on. Reseller R1 may supply a set of R1-supplied and R1-branded cloud services, and R1's customers may subscribe to one or more of the R1-supplied services. The set of one or more R1-supplied services to which R1.C1 740 subscribes may be the same as or different from the set of one or more R1-supplied services to which R1.C2 742 subscribes. The cloud services supplied by R1 722 may be different from the cloud services supplied by reseller R2 722 and the cloud services supplied by CSP 710.

[0206] Reseller R2 722 may have multiple customers of its own who subscribe to R2-supplied and R2-branded cloud services provided using a vPLC created for R2. As shown in FIG. 7 , R2 722's customers include customer R2.C1 744 with tenancy T.R2.C1 and associated users, customer R2.C2 746 with tenancy T.R2.C2 and associated users, etc. The cloud services supplied by R2 722 may be different from the cloud services supplied by reseller R1 720 and the cloud services supplied by CSP 710. R2's customers may subscribe to one or more of the R2-supplied services. The set of one or more R2-supplied services to which R2.C1 744 subscribes may be the same as or different from the set of one or more R2-supplied services to which R2.C2 746 subscribes.

[0207] As shown in FIG. 7, there are two levels of tenancies that come into play when a vPLC is used by a reseller to sell reseller-supplied cloud services to the reseller's customers. The first level of tenancies corresponds to tenancies associated with the CSP's 710 customers, including direct or non-reseller customers and reseller customers. These tenancies may also be referred to as CSP customer tenancies (or CoCT). In FIG. 7, the CoCT includes tenancies associated with reseller R1 720, reseller R2 722, direct customer C1 726, and direct customer C2 728. The second level of tenancies includes tenancies associated with the reseller's customers. These tenancies may also be referred to as reseller customer tenancies (or CoRT). In Figure 7, CoRT includes tenancies associated with R1.C1 740, R1.C2 742, R2.C1 744, R2.C2 746, etc.

[0208] 8 is a diagram illustrating the relationship between resources allocated to tenancies for a CSP's direct customers, tenancies for the CSP's resellers, and tenancies for each reseller's customer, according to one embodiment. Referring to FIG. 8, in one embodiment, a CSP-provided infrastructure (e.g., data center) 810 within a region may be divided into several securely isolated portions for the CSP's resellers, such as resources allocated to vPLC.R1 820 created for reseller R1 and associated with reseller R1's tenancy (T.R1), resources allocated to vPLC.R2 822 created for reseller R2 and associated with reseller R2's tenancy (T.R2), etc. Resource partitions may also include securely isolated partitions for the CSP's non-reseller direct customers C1 and C2, such as resources allocated to non-reseller customer CSP.C1's tenancy (T.CSP.C1) 826 and resources allocated to non-reseller customer CSP.C2's tenancy (T.CSP.C2) 828. The partitions of the resource CSP-provided regional infrastructure allocated to reseller R1's tenancy 820, reseller R2's 822, customer CSP.C1's tenancy, and customer CSP.C2's tenancy may be referred to as first-level resource partitions.

[0209] Each first-level resource partition or securely isolated portion of resources associated with a reseller's tenancy (i.e., the R1 tenancy and the R2 tenancy) may be assigned or tagged with a vPLC ID (i.e., resource ID), such as vPLC_ID.vPLC.R1 820 and vPLC_ID.vPLC.R2 822. Each of these partitions for the CSP's non-reseller direct customers may be tagged with a resource ID. Each tenancy of the CSP's non-reseller direct customers can be accessed by their respective users, who can use one or more of the CSP-supplied cloud services to which their corresponding direct customers subscribe.

[0210] Within each first-level resource partition (e.g., or portion) for a reseller, there may be additional second-level resource partitions allocated to the tenancies of the reseller's customers. For example, securely isolated resource partition (vPLC.R1.C1) 840 may be allocated to the tenancy (T.R1.C1) associated with reseller R1's customer C1, and securely isolated resource partition (vPLC.R1.C2) 842 may be allocated to the tenancy (T.R1.C2) associated with reseller R1's customer C2. Isolation partition vPLC.R1.C1 may be tagged with a customer resource ID that identifies the resources allocated to T.R1.C1, and isolation partition vPLC.R1.C2 may be tagged with a customer resource ID that identifies the resources allocated to T.R1.C1.

[0211] Similarly, securely isolated resource partition (vPLC.R2.C1) 844 may be allocated to a tenancy (T.R2.C1) associated with reseller R2's customer C1, and securely isolated resource partition (vPLC.R2.C2) 846 may be allocated to a tenancy (T.R2.C2) associated with reseller R2's customer C2. Isolation partition vPLC.R2.C1 may be tagged with a customer resource ID that identifies the resources allocated to T.R2.C1, and isolation partition vPLC.R2.C2 may be tagged with a customer resource ID that identifies the resources allocated to T.R1.C1. Each tenancy associated with a reseller's customer (e.g., 840 for R1.C1, 842 for R1.C2, 844 for R2.C1, or 846 for R2.C2) can be accessed by one or more users associated with the customer, who can use one or more of the reseller-supplied cloud services to which the corresponding customer subscribes.

[0212] 9A and 9B illustrate a mechanism for storing information within tenancies for customers of individual resellers of a CSP, according to one embodiment. In some embodiments, the vPLC ID and tenancy ID (sometimes referred to as customer tenancy ID or CT ID) for the tenancy associated with the reseller may be associated in two ways: 1) a distributed manner (or record space per vPLC) as shown in FIG. 9A; and 7) a centralized manner (or vPLC ID as the partition key) as shown in FIG. 9B. While FIGS. 9A and 9B illustrate storing information in a table format, in some embodiments, the information may be stored in a different format, such as an array, a key-value store, XML, JSON, etc.

[0213] FIG. 9A illustrates the first approach, i.e., per-vPLC record space. In some embodiments, per-vPLC record space can be an approach of storing tenancy-related information in a vPLC-specific table based on each reseller's vPLC ID. In other words, the information can be stored in a distributed manner. Each table is assigned a vPLC ID. Within each table, one row can be designated for one customer of the reseller, another row can be designated for another customer of the reseller, and so on. For example, in FIG. 9A , there are two tables for storing information such as identity information (e.g., user logins, associated authentication information, and certificates) or billing information (e.g., resource usage, pricing). Table 910 is assigned a vPLC ID (vPLC_ID.vPLC.R1) for vPLC.R1 associated with reseller R1, and table 920 is assigned a vPLC ID (vPLC_ID.vPLC.R2) for vPLC.R2 associated with reseller R2. Within table 910, there are three records for three customers C1, C2, and C3 of reseller R1. The first record / row may include a tenancy ID (T_ID.T.R1.C1) for the tenancy associated with reseller R1's customer C1 and associated information for customer C1, e.g., identity, billing, etc. The second record / row may include a tenancy ID (T_ID.T.R1.C2) for the tenancy associated with reseller R1's customer C2 and associated information for customer C2. The third record / row may include a tenancy ID (T_ID.T.R1.C3) for the tenancy associated with reseller R1's customer C3 and associated information for customer C3.

[0214] The same structure may be used for table 920, storing three records for reseller R2's three customers C1, C2, and C3. The first record / row may include a tenancy ID (T_ID.T.R2.C1) for the tenancy associated with reseller R2's customer C1 and associated information for customer C1, e.g., identity, billing, etc. The second record / row may include a tenancy ID (T_ID.T.R2.C2) for the tenancy associated with reseller R2's customer C2 and associated information for customer C2. The third record / row may include a tenancy ID (T_ID.T.R2.C3) for the tenancy associated with reseller R2's customer C3 and associated information for customer C3.

[0215] FIG. 9B shows a second approach using the vPLC ID as a partition key. In FIG. 9B, the second approach stores information in a table with both the customer tenancy ID and the vPLC ID as two table columns for searching by using the vPLC ID as the partition key. In other words, information can be stored within a table in a centralized manner. For example, in FIG. 9B, reseller R1, who is associated with a vPLC ID (vPLC_ID.vPLC.R1), has three customers C1, C2, and C3, who are associated with tenancy IDs T_ID.T.R1.C1, T_ID.T.R1.C2, and T_ID.T.R1.C3, respectively, that identify the tenancies for these three customers. The first three rows of table 930, based on the vPLC ID column, can then be marked as vPLC_ID.vPLC.R1, which is the vPLC ID for vPLC.R1 created for reseller R1. The customer tenancy ID column of the first row can be marked as the tenancy ID for the tenancy associated with reseller R1's customer C1 (T_ID.T.R1.C1), while the tenancy ID column of the second row can be marked as the tenancy ID for the tenancy associated with reseller R1's customer C2 (T_ID.T.R1.C2), and the tenancy ID column of the third row can be marked as the tenancy ID for the tenancy associated with reseller R1's customer C3 (T_ID.T.R1.C3). The information stored in the first three rows belongs to reseller R1's customers C1, C2, and C3.

[0216] The same structure may also be used to store records for customers C1, C2, and C3 of reseller R2, e.g., in rows 4, 5, and 6, within the same table 930. For example, the last three rows of table 930 based on the vPLC ID column may be marked as vPLC_ID.vPLC.R2, which is the vPLC ID for vPLC.R2 created for reseller R2. The customer tenancy ID column of the fourth row can be marked as the tenancy ID for the tenancy associated with reseller R2's customer C1 (T_ID.T.R2.C1), while the tenancy ID column of the fifth row can be marked as the tenancy ID for the tenancy associated with reseller R2's customer C2 (T_ID.T.R2.C2), and the tenancy ID column of the sixth row can be marked as the tenancy ID for the tenancy associated with reseller R2's customer C3 (T_ID.T.R2.C3). The information stored in the last three rows belongs to reseller R2's customers C1, C2, and C3.

[0217] FIG. 10 is a flow diagram illustrating an example of a vPLC setup process, according to an embodiment. The process illustrated in FIG. 10 may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective systems, using hardware, or a combination thereof. The software may be stored on a non-transitory storage medium (e.g., on a memory device). The method presented in FIG. 10 and described below is intended to be exemplary and non-limiting. While FIG. 10 depicts various processing steps occurring in a particular sequence or order, this is not intended to be limiting. In an alternative embodiment, the process may be performed in some different order, or some steps may also be performed in parallel. It should be appreciated that in alternative embodiments, the process illustrated in FIG. 10 may include more or fewer steps than depicted in FIG. 10.

[0218] 10 may be performed by a CSP. Some steps may be performed by components within the CSP-provided region infrastructure (e.g., a control plane (CP), a resource manager (RM), and a management plane (MP), etc.) or one or more CSP-provided cloud services, e.g., an endpoint management service (EMS), an identity management service, etc.

[0219] As previously described, a vPLC may be created when a reseller subscribes to a CSP-provided vPLC service. In step 1002, a signal to create a vPLC may be received. In one embodiment, the signal may be a request received when a reseller subscribes to a CSP-provided vPLC service. The request may be an agreement between the CSP and the reseller that lists the virtual model (e.g., nested or segmented model), database, resource placement configuration, customization details, etc. In other embodiments, a signal may be received from some component indicating that a vPLC is being created.

[0220] In step 1004, configuration information for the vPLC to be created may be received. In some embodiments, the configuration information may include tenancy information (e.g., a tenancy identifier) ​​for which the vPLC is being created and information to identify the realm to be associated with the vPLC. As described above, a tenancy is created for the reseller, and a tenancy identifier that uniquely identifies the tenancy is also created for the reseller to be associated with the vPLC.

[0221] As previously mentioned, in use cases where a vPLC may be used as a virtual data center and associated with a different realm than the realm associated with the cloud infrastructure in the region provided by the CSP, identity and trust profile information (e.g., passwords, credentials, etc.) of the different realm needs to be associated with the vPLC. Association of identity and trust profile information of the different realm may be achieved by creating a mapping (or "shadow copy") of the identity and trust profile information in the vPLC. Further details are discussed in FIG. 18.

[0222] Step 1006, covering steps 1008-1018, may implement a process for creating a vPLC for a reseller. In step 1008, a vPLC identifier may be created for the vPLC. A vPLC created for a reseller is treated as a resource and assigned a unique resource identifier, referred to as a vPLC identifier (vPLC ID), which is associated with the reseller's tenancy.

[0223] In step 1010, a set of resources from the CSP-provided infrastructure in the region may be allocated to the vPLC. The set of resources allocated to the vPLC may be selected based on an agreement between the reseller and the CSP. For example, reseller R may subscribe to one or more cloud services offered by the CSP, such as compute services and object storage services. Thus, the set of resources may include compute resources and object storage resources.

[0224] In step 1012, a namespace may be reserved for vPLC. A CSP's resource allocation service (or resource manager (RM)) may create and reserve a namespace for grouping multiple different types of resources that can support vPLC. A namespace may refer to a logical grouping or virtual boundary that aids in the organization and management of resources and prevents naming conflicts when multiple instances of similar resources are created. Thus, a namespace provides a structured way to organize and group related resources. In some embodiments, namespace 0 is reserved for CSPs. Namespaces 1 and beyond are used for vPLC. In one embodiment, a range of Internet Protocol (IP) addresses may be reserved and associated with a namespace for vPLC for traffic separation purposes.

[0225] In step 1014, a console may be created for the vPLC that allows reseller R, the reseller's customers, and users associated with the reseller's customers to configure, access, and manage resources deployed within the vPLC and visualize, access, and interact with reseller-supplied cloud services. Additionally, reseller R may configure the console to provide different customized experiences to different customers of the reseller.

[0226] In step 1016, a set of endpoints for the vPLC may be created. In some embodiments, some of the set of endpoints may be default endpoints (e.g., identity endpoints) that are always available, and some endpoints are determined based on which cloud services the reseller subscribes to. For example, the reseller may subscribe to one or more CSP-supplied cloud services, such as compute, storage, virtual cloud network (VCN), database, etc. As a result, in addition to a default identity endpoint for accessing identity services, endpoints for accessing subscribed services may also be created.

[0227] In step 1018, vPLC-related information, such as the vPLC ID, the reseller's identity information, and other information facilitating the reseller-supplied cloud service, may be stored in a namespace for the vPLC. In step 1020, the CSP may send a response and other information to the reseller R indicating the created vPLC. The reseller can begin managing the vPLC for the reseller's customer, such as creating a customer account, configuring the reseller-supplied cloud service, etc.

[0228] Figure 11 is a block diagram of a distributed environment illustrating an example of a virtual private label cloud (vPLC) hosted by a CSP-provided infrastructure within a region, according to an embodiment. Figure 11 may be similar to Figure 6, and both figures include securely isolated portions (or partitions) of the CSP-provided region infrastructure for the CSP's direct customers (e.g., first portion 602, second portion 604, and third portion 606 of Figure 6), vPLC.R1 associated with reseller R1, and vPLC.R2 associated with reseller R2. The CSP's direct customers, customers of reseller R1, and customers of reseller R2 can access their respective partitions through one or more endpoints and consoles. The CSP-provided region infrastructure 602 may also have a control plane, management plane, and gateways, as described in Figure 6.

[0229] However, Figure 11 shows a more detailed view of the interconnections between resources within the CSP-provided region infrastructure, such as compute, storage, gateways, CPs, and the management plane. Additionally, the CSP's direct non-reseller customers, customers of reseller R1, and customers of reseller R2 may access the CSP-provided region infrastructure from either the Internet or a corporate network. For example, a gateway may be used to access the Internet or a corporate network from a real application cluster (RAC) within the CSP-provided region infrastructure.

[0230] FIG. 11 shows three cloud partitions running on the same CSP-provided region infrastructure 1102: the CSP's native cloud 1120 and two vPLCs for resellers R1 and R2: vPLC.R1 1122 and vPLC.R2 1124. The CSP-provided region infrastructure 1102 is operated by the CSP. The CSP-provided region infrastructure 1102 may also include a control plane 1104 and a management plane 1105. As mentioned above, the CP 1104 may be responsible for providing management, deployment, and orchestration functions with the cloud environment provided by the CSP. The MP 1105 may be a collection of software components responsible for provisioning all processes and workflows required for requested operation. Additionally, software implementing cloud services may originate from a deployment system 1106 and enter data centers within the region through the management plane 1105. Software updates (e.g., new services) for the control plane 1104 can be delivered by a deployment system 1106 through the management plane 1105. A CSP administrator or reseller operator 1107 deploying the software / services can also access the CSP-provided region infrastructure through the management plane 1105.

[0231] Computing resources, namely, three Real Application Clusters (RACs) 1140, 1142, and 1144, that share the same storage 1150 are partitioned and allocated to the CSP's native cloud 1120, vPLC.R1 1122, and vPLC.R2 1124, respectively. For example, RAC 1140 is dedicated to the CSP's direct customers, RAC 1142 is for vPLC.R1, and RAC 1144 is for vPLC.R2.

[0232] Network resources may also be divided or shared among the CSP's direct customers and resellers, allowing their computing resources to connect to the Internet or corporate networks. For example, each RAC 1140, 1142, and 1144 uses its network gateways 1160, 1162, and 1164 to access the Internet 1108 or corporate network 1109. In some embodiments, two RACs may share the same network interface. For example, RAC 1142 and RAC 1144 may share the same gateway 1162 to access the Internet 1108.

[0233] The storage in Figure 11 is simplified. Storage 1150 may include multiple different types of storage, such as global storage, dedicated storage, and block storage and object storage. In Figure 11, vPLC.R1 and vPLC.R2 have their dedicated storage 1152 and 1154, respectively. Global storage 1150 belongs to the CSP.

[0234] In FIG. 11 , a CSP's native cloud 1120 can provide CSP-supplied cloud services to the CSP's direct non-reseller customers (e.g., CSP.C1-n), such as corporations. Users associated with the reseller's customers can access the services through dedicated service endpoints that are part of their respective namespaces, including fully qualified domain names (FQDNs), IP addresses, console experiences, identities, etc. The CSP can provide individual consoles for each reseller, and the consoles provide the services through the endpoints. The endpoints associated with the consoles can differ based on which services are provided to that reseller's customers based on the reseller's subscription. For example, an individual reseller's customer may submit an access request through a dedicated endpoint or via the console.

[0235] Users can use service endpoints 1170 and 1172 exposed by the CSP (two endpoints are shown, but more are possible), which have DNS names. The native cloud 1120 may also include a console 1174 with a look and feel. The CSP's direct customers on the Internet 1108 can access the CSP-provided regional infrastructure 1102 through public endpoints 1170 and 1172 to request instance launch or termination. Reseller R1's customers on the Internet 1108 can also access the CSP-provided regional infrastructure 1102 through public endpoints 1180 and 1182 to request instance launch or termination. Reseller R2's customers on the corporate network 1109 can access the CSP-provided regional infrastructure 1102 through private endpoints 1190 and 1192 to request instance launch or termination. The control plane 1104 can then launch instances, terminate instances, configure connectivity for those instances, and orchestrate services within the data center. Network interfaces 1160 and 1162 (e.g., Internet Gateways (IGW)) for CSP and vPLC.R1, respectively, can provide network interconnection to the Internet 1108. Network interface 1164 (e.g., Dynamic Routing Gateway (DRG)) for vPLC.R2 can provide network interconnection to the enterprise network 1109.

[0236] In FIG. 11 , vPLC.R1 1122 and vPLC.R2 1124 share the same physical infrastructure. Their endpoints are specific to each reseller, such as endpoints 1180 and 1182 for reseller R1 and endpoints 1190 and 1192 for reseller R2. Consoles 1184 and 1194 are customized for reseller R1 and reseller R2, respectively. The vPLC endpoints and consoles can connect to the Internet or their corporate networks depending on their needs. For example, in FIG. 11 , vPLC.R1 connects to the Internet 1108 through a network gateway 1162 (e.g., IGW). However, vPLC.R2 does not connect to the Internet and therefore instead connects to the corporate network 1109 through a secured network 1164 (e.g., FastConnect). 11, gateway 1162 can be sliced ​​and shared by vPLC.R1 and vPLC.R2 and can be part of an orchestration performed by a CSP. In this example, some of vPLC.R2's data can go to the Internet 1108 and some can go to its corporate network 1109.

[0237] If the request comes in through console 1194, it can go to control plane 604. The control plane can determine that the request came from vPLC.R2 1124 and can find and provision an available resource (e.g., a computer) from the RAC specified for vPLC.R2 (i.e., 1144 in this case), then launch this resource. Control plane 604 can then respond to reseller R2's customer through console 1194 and provide an IP address for their network configuration (e.g., VCN). Control plane 1104 can orchestrate gateway 1164 depending on how it is configured toward either enterprise network 1109 or the Internet 1108. The control plane can recognize not only the request, but also the source of the request, whether it is a direct customer of the CSP, a customer of reseller R1, or a customer of reseller R2.

[0238] As mentioned above, the division of the CSP-provided region infrastructure can be implemented for different vPLCs at various predetermined infrastructure levels, such as the rack level, the server level, the hypervisor level, or the VM level. To further illustrate, for division at the rack level, a vPLC may have its own dedicated rack and resources therein. For division at the server level, multiple vPLCs may share a rack, but each vPLC has its dedicated server. In yet another example, for division at the hypervisor level, multiple vPLCs may share a server, but each vPLC has its dedicated hypervisor. For division at the VM level, multiple vPLCs may share a hypervisor, but each vPLC has its dedicated VM. Therefore, to achieve a multi-vPLC approach, there may be two main vPLC infrastructure division models (or opposite ends of a spectrum): a segmented model and a nested model. The segmented model may refer to resource division at the rack level. The nested model may refer to resource division at the VM level. However, other variations between them are possible, taking into account operational efficiency (e.g., control plane sharing) and fragmentation (i.e., resource division).

[0239] Figure 12 is a block diagram illustrating a vPLC segmentation model for resource partitioning, according to an embodiment. In this model, which represents one end of the partitioning spectrum, each vPLC may have its dedicated resources, including compute and network resources, at the rack level. In other words, a rack may be allocated exclusively to a particular vPLC and may not be used by other vPLCs or by a CSP to provide CSP-supplied services to its direct customers. The segmentation model implements resource and usage tracking at the physical resource level based on affinity tagging (e.g., vPLC ID and customer tenancy ID). There may be a default behavior for each segment, but each segment can be customized by the reseller associated with the vPLC in that segment.

[0240] In FIG. 12 , multiple vPLCs (e.g., vPLC.R1 1210, vPLC.R2 1220, and vPLC.Rn 1230) may exist within a CSP-provided infrastructure (e.g., a data center) within a region. In an embodiment, in a segmentation model, there may be physical and logical partitions, as shown in FIG. 12 . For example, for a physical partition, a vPLC is assigned its dedicated hardware resources, such as, for example, a rack containing one or more servers, with each server hosting one or more operating systems. In FIG. 12 , a first rack is assigned to the vPLC.R1 segment 1210, a second rack is assigned to the vPLC.R2 segment 1220, and a third rack is assigned to the vPLC.Rn segment 1230. Each vPLC may also have dedicated networking switches (also referred to as gateways), such as, for example, gateway 1219 for vPLC.R1 1210, gateway 1229 for vPLC.R2 1220, and gateway 1239 for vPLC.Rn 1230, that connect to the Internet / enterprise network 1260. Thus, in Figure 12, vPLC.R1 1210 has dedicated servers 1214a-b, a hypervisor 1216 running on server 1214b and hosting VMs 1218a-n, and a gateway (e.g., IGW) 1219. Similarly, vPLC.R2 1220 has dedicated servers 1224a-b, a hypervisor 1226 running on server 1224b and hosting VMs 1228a-n, and a gateway 1229. The vPLC.Rn 1230 includes dedicated servers 1234a-b, a hypervisor 1236 running on server 1234b and hosting VMs 1238a-n, and a gateway 1239.

[0241] For logical partitions, there is a common control plane 1250 and resource manager 1252 managed and operated by the CSP for all vPLCs. However, the data plane may be dedicated to each vPLC in a segmented model. For example, each vPLC.R1, vPLC.R2, and vPLC.Rn has its own data plane 1212, 1222, and 1232, respectively. In some embodiments, the resource manager 1252 may be part of the management plane. In other embodiments, the resource manager 1252 may be separate from the management plane.

[0242] In some embodiments, a user associated with a customer of reseller R1 1280 can send a request to launch a VM instance or a bare metal server through one of vPLC.R1 endpoints 1270. Assume the request is to launch a new VM instance. In that case, control plane 1250 can work with an identity service to check the appropriate credentials of the user of vPLC.R1, after which resource manager 1252 can identify all resources dedicated to vPLC.R1, such as an IP address range. A VM instance (e.g., 1218a) is then launched on hypervisor 1216 within vPLC.R1 segment 1210. A response is then sent by control plane 1250 back to the requesting user through vPLC.R1 endpoint 1270. If the request is for a bare metal server (e.g., 1214a), resources for vPLC.R1 can be provisioned with all necessary configurations and allocated to vPLC.R1 by resource manager 1252. Similar processes may be performed for requests from users associated with customers of reseller R2 1282, which are to be satisfied by vPLC.R2 segment 1220, and for requests from users associated with customers of reseller Rn 1284, which are to be satisfied by vPLC.Rn segment 1230. Each process for a vPLC may be performed independently of and concurrently with another vPLC.

[0243] The vPLC segmentation model may have several advantages, such as physical separation and sharing of a common control plane 1250 operated by the CSP. Additionally, a resource manager 1252 manages resources across the vPLCs 1210-1230. However, resources, including networking and storage, are duplicated to be designated for each vPLC, and therefore scaling challenges may be present. Small footprints, such as a single rack or server, are also difficult to achieve under this model. However, the vPLC segmentation model may still share power, cooling, etc. As mentioned above, the segmentation model is only one end of the spectrum, and there are various ways to partition based on resources and efficiency.

[0244] FIG. 13 is a block diagram illustrating a vPLC nested model for resource partitioning, according to an embodiment. The nested model of FIG. 13 illustrates the other end of the spectrum of resource partitioning at the VM level. In other words, a VM may be allocated exclusively to a particular vPLC and may not be used by other vPLCs or by a CSP to provide CSP-supplied services to its direct customers. However, a hypervisor, server, or rack may be shared between a CSP's customers and one or more vPLC customers. In some embodiments, the nested model may have a base layer, such as a hypervisor, as a shared foundation that is not visible to the customer; abstraction layers, such as VMs, may be built on top of the base layer, allowing each reseller to clone and customize the abstraction layers for their customers.

[0245] In the embodiment of FIG. 13 , all vPLCs (e.g., vPLC.R1, vPLC.R2, and vPLC.Rn) are hosted on a single segment with a shared infrastructure, shown as a “server CSP” 1320 (e.g., a bare-metal server) operated by a CSP, which may have control plane functionality. Server CSP 1320 may be configured to host a hypervisor 1330 that manages virtual machines (VMs) 1332a-n running on the hypervisor. In some embodiments, a segment may comprise multiple racks. Here, two VM instances for vPLC.R1 1332a-b, one VM instance for vPLC.R2 1332c, and one VM instance for vPLC.Rn 1332n reside on the shared hypervisor 1330. All vPLCs share the same data plane 1312 within segment 1310. In some embodiments, a segment may comprise multiple racks. Because the hypervisor, servers, and racks may be shared by multiple vPLCs, the servers for these vPLCs may be from different racks.

[0246] 13 further illustrates a variation of the split scenario, where a customer of reseller R2 may request an additional bare metal server, server vPLC.R2 1322, provisioned as a dedicated server. In the vPLC nesting model, a shared server pool can be located wherever space is available. This therefore provides a lower cost point for the reseller.

[0247] In FIG. 13 , a gateway 1340 (e.g., IGW or DRG) shared by the vPLCs has connectivity to the Internet or the corporate network, as needed. Incoming packets from the Internet / corporate network 1360 may be tagged (i.e., encapsulated in each packet header) with vPLC-related information (e.g., vPLC ID and customer tenancy ID) at the gateway 1340 for tracking purposes to the appropriate vPLC. Packet encapsulation helps separate traffic within a single segment 1310 shared by multiple vPLCs. As an example, a packet destined for VM 1332a belonging to a tenancy associated with customer C1 of reseller R1 associated with vPLC.R1 may be encapsulated in the packet header with vPLC_ID.vPLC.R1 and customer tenancy ID (T_ID.R1.C1). The packet can then be routed to VMs 1332a-b separated for vPLC.R1 based on the vPLC ID (vPLC_ID.vPLC.R1). A customer tenancy ID (T_ID.R1.C1) associated with reseller R1's customer C1 in the packet header can further assist in routing the packet to tenant C1 running on VM 1332a. The packet is decapsulated before being delivered to VM 1332a.

[0248] Similarly, a packet destined for VM 1332c belonging to a tenancy associated with customer C1 of reseller R2, which is associated with vPLC.R2, may be encapsulated in the packet header with vPLC_ID.vPLC.R2 and a customer tenancy ID (T_ID.R2.C1). The packet can then be routed to resources carved out for vPLC.R2 based on the vPLC ID (vPLC_ID.vPLC.R2). The customer tenancy ID (T_ID.R2.C1) associated with customer C1 of reseller R2 in the packet header can further assist in routing the packet to tenant C1 running on VM 1332c in vPLC.R2. The packet is decapsulated before being delivered to VM 1332c.

[0249] Outgoing packets can be performed in the reverse process. For example, an outgoing packet originating from VM 1332a in vPLC.R1 and destined for the Internet 1360 may be encapsulated in a packet header with vPLC_ID.vPLC.R1 and a customer tenancy ID (T_ID.R1.C1) associated with reseller R1's customer C1. When the packet reaches gateway 1340, the packet may be decapsulated before heading to the Internet 1360. Similarly, an outgoing packet originating from VM 1332c in vPLC.R2 and destined for the Internet 1360 may be encapsulated in a packet header with vPLC_ID.vPLC.R2 and a customer tenancy ID (T_ID.R2.C1) associated with reseller R2's customer C1. When the packet reaches gateway 1340, the packet may be decapsulated before heading to the Internet 1360. As a result, both incoming and outgoing traffic is segregated within a single segment 1310 shared by multiple vPLCs.

[0250] In some embodiments, in a nested model, one or more VMs may belong to a direct customer of the CSP and may share the same hypervisor with other vPLCs.

[0251] Reseller users accessing the vPLC infrastructure are authenticated in both segmented and nested models. A user associated with a particular reseller's customer can access an endpoint and be authenticated as a user of the vPLC associated with the particular reseller by using the user's credentials, such as a username and customer tenancy ID. For example, in some embodiments, in FIG. 13 , a user associated with reseller R1's customer C1 can send a request to launch a VM instance through one of the vPLC.R1 endpoints 1302. The particular vPLC.R1 endpoint 1302 can augment the request with the vPLC ID for vPLC.R1. The control plane 1350 can cooperate with the identity service to check the appropriate credentials for the user of vPLC.R1 (e.g., the user's username and customer C1's tenancy ID). Resource manager 1352 can then identify all resources allocated to vPLC.R1, such as server 1320 and hypervisor 1330 based on the vPLC ID, and VM 1332a based on the customer tenancy ID of customer C1. VM instance 1332a is then launched on hypervisor 1330 accordingly.

[0252] 14 is a flow diagram illustrating a sign-up process for a customer of a reseller associated with a vPLC, according to one embodiment. When a direct customer of a CSP subscribes or registers (i.e., signs up) for a cloud service offered by the CSP (i.e., a CSP-provided cloud service), a tenancy or account is created for that customer. Similarly, when a customer of an individual reseller subscribes or registers for a cloud service offered by that reseller (i.e., a reseller-provided cloud service), a tenancy or account is created for that customer.

[0253] FIG. 14 illustrates the sign-up process for a particular customer C1 of reseller R1. However, the sign-up process is applicable to customers of individual resellers. In FIG. 14, in step S1, reseller R1's customer C1 1401 can sign up to create a new account by entering user identification credentials, such as a username or account name (e.g., John), using a sign-up portal on the console of vPLC.R1 1402. In step S2, vPLC1.R1 console 1402 can augment the sign-up request by adding a vPLC ID (e.g., vPLC_ID.vPLC.R1) that identifies the vPLC being created for reseller R1 to the user's credentials and forward the sign-up request to CSP (control plane (CP) or management plane (MP)) 1404. In some embodiments, the creation of reseller R1's customer C1's account can be handled by reseller R1 with assistance from the CSP's CP or MP.

[0254] In step S3, the CSP then creates a customer tenancy (or account, T.R1.C1) for reseller R1's customer C1 1401. A tenancy identifier (T_ID.T.R1.C1) is assigned to the newly created tenancy (T.R1.C1) of customer C1. A vPLC ID (e.g., vPLC_ID.vPLC.R1) identifying vPLC.R1 for reseller R1 may be associated with customer C1's newly created customer tenancy ID (e.g., T_ID.T.R1.C1), shown at 1430, and may be stored in database 1420 along with other authentication information (e.g., account name and password). Other customers of reseller R1 and reseller R2 may also be stored in database 1420. For example, the customer tenancy ID of reseller R1's customer C2 (i.e., R1.C2) is associated with the vPLC ID of vPLC.R1, as shown in 1432 as T_ID.T.R1.C2 and vPLC_ID.vPLC.R1. The customer tenancy ID of reseller R2's customer C2 (i.e., R2.C1) is associated with the vPLC ID of vPLC.R2, as shown in 1434 as T_ID.T.R2.C1 and vPLC_ID.vPLC.R2.

[0255] The association between the vPLC ID and the tenancy IDs of individual resellers' customers can provide isolation between customer tenancies within a vPLC. For example, customer tenancy R1.C1 can be securely isolated from customer tenancy R1.C2 by using different customer tenancy IDs, i.e., T_ID.T.R1.C1 for customer C1 and T_ID.T.R1.C2 for customer C2, even though both customer tenancies are within the same vPLC.R1. The tenancies associated with reseller R1 and reseller R2 can be securely isolated using their respective vPLC IDs, i.e., vPLC_ID.vPLC.R1 for reseller R1 and vPLC_ID.vPLC.R2 for reseller R2.

[0256] In step S4, CSP 1404 can return a handle including the customer tenancy ID (e.g., T_ID.T.R1.C1), a temporary default password, and an account name to vPLC.R1 console 1402. In step S5, vPLC.R1 console can receive the temporary authentication information and forward the customer tenancy ID and password to reseller R1's customer C1 1401. As shown in FIG. 14 , reseller R1's customer C1 1401 can interact directly with reseller R1 through vPLC.R1 console 1402 to sign up and subscribe to one or more reseller-supplied cloud services. Similarly, the CSP's customers can interact separately with the CSP to sign up and subscribe to CSP-supplied cloud services at the CSP's console (not shown).

[0257] As described above in connection with tenancy creation, the CSP's database (e.g., 1420 of FIG. 14) can store vPLC IDs, customer tenancy IDs, and their corresponding authentication information. In an embodiment, multiple customer tenancy IDs may be associated with the same vPLC ID, as shown at 1430 (T_ID.T.R1.C1 is associated with vPLC_ID.vPLC.R1) and 1432 (T_ID.T.R1.C2 is associated with vPLC_ID.vPLC.R1). Tenancies may be defined at the realm level, and thus, a tenancy may have a realm identifier associated with it.

[0258] 15 is a flow diagram illustrating a login process for a user associated with a customer of a reseller associated with a vPLC, according to one embodiment. In FIG. 15, steps S1 and S2 comprise the login phase after an individual reseller customer completes the sign-up process described in FIG. 14.

[0259] In step S1, a user 1501 associated with a customer of reseller R1 can log in using a customer tenancy ID (or CT ID), username, and password at the vPLC.R1 endpoint 1510. At S2, the user 1501 and their customer tenancy can be verified at the vPLC.R1 endpoint 1510 by going through a verification process, which may include, but is not limited to, user authentication (e.g., verifying valid credentials such as username and password), session management (e.g., session tokens or timeouts), and access control enforcement (based on access rights, permissions, and policies).

[0260] In step S3, the vPLC.R1 endpoint 1510 can return a session handle. The session handle may refer to a token or unique identifier returned to the customer after successfully logging into their account. The session handle can be used for subsequent requests to access cloud services and resources (e.g., launch an instance) without having to re-authenticate for each individual request during the active login session (or API session).

[0261] Cloud resource IDs (CIDs), which include a customer tenancy ID, may be generated for resources when those resources are created. Virtual machines, accounts, tenancies, authentication policies, virtual network subnets, ACLs, and load balancers may be examples of cloud resources and therefore may be referenced by their respective CIDs.

[0262] 16 is a flow diagram illustrating a resource allocation and provisioning process for a reseller's customer associated with a vPLC, according to one embodiment. A resource manager (RM) manages the resources of the CSP-provided cloud infrastructure within a region and the resource partitions assigned to each vPLC.

[0263] As described above, a vPLC-specific endpoint may be used by a user associated with a customer of a reseller to access resources associated with the vPLC and to perform one or more functions or operations involving resources allocated to the vPLC. In Figure 16, in step S1, a user 1601 associated with customer C1 of reseller R1 may issue a request to launch a compute instance, such as a VM instance with 1 GB RAM, through a vPLC-specific endpoint, i.e., vPLC.R1 endpoint 1602.

[0264] The vPLC.R1 endpoint 1602, in step S2, can augment (or annotate) the request to indicate that the request is coming from a customer of reseller R1 by tagging the request with the vPLC.R1 endpoint's corresponding vPLC ID (e.g., vPLC_ID.vPLC.R1) and forwards the request to resource manager (RM) 1604. In other words, the annotated information (e.g., vPLC ID) enables the authorization, once the request is authenticated, to be processed using all policies associated with vPLC.R1. In some embodiments, the policies associated with the vPLC may include, for example, the type of resource, the amount of resource allowed for allocation (i.e., capacity limits), permissions, etc.

[0265] In step S3, RM 1604 queries its database (i.e., resource database) 1606 to determine whether the requested resources are available to vPLC.R1 based on the policy. Based on the query, in step S4, RM can perform resource allocation after confirming that the requested resources are available based on the policy. In other words, RM allocates a portion of the resources already allocated to vPLC.R1 of reseller R1 to a customer tenancy (T.R1.C1) associated with reseller R1's customer C1 (based on the customer tenancy ID when the user logs in). This resource allocation may be performed by allocating or assigning a portion of the available cloud resources (already allocated to vPLC.R1) to a user associated with customer C1 based on policies agreed upon between customer C1 and reseller R1, such as capacity limits or permissions. For example, reseller R1 and its customer C1 may have a contract policy to allow a maximum of 100 GB of available memory to be used. Currently, 80 GB is allocated (or used). If a user associated with customer C1 requests 10 GB, the RM can identify the CID to have the 10 GB provisioned by the CP. If a user associated with customer C1 requests 30 GB, which would exceed the total 100 GB limit, the request may be denied.

[0266] In step S5, the RM 1604 obtains from the resource database the cloud resource ID assigned to the resource (vPLC.R1.C1) allocated to the tenancy (T.R1.C1) of the customer C1 of the reseller R1. In step S6, the RM passes the resource ID to the vPLC.R1 endpoint 1602. In step S7, the vPLC.R1 endpoint 1602 makes a resource request to the control plane (CP) 1608 for a resource having a particular resource ID assigned to the resource allocated to the tenancy (T.R1.C1) of the customer C1 of the reseller R1. In other words, the resource manager 1604 can augment the user's request (e.g., step S6) with information related to the vPLC (e.g., the resource ID assigned to vPLC.R1.C1), which can be used to provision the resource by the control plane 1608 (e.g., step S8).

[0267] In step S8, the control plane can provision the requested resources (e.g., launch a specific VM instance and storage). The resources may include compute, networking, storage, etc. In step S9, the control plane 1608 can respond to the vPLC.R1 endpoint 1602 with a status and a resource handle. The resource handle may refer to a unique resource identifier assigned to the provisioned resource, such as a cloud resource ID assigned to the provisioned compute instance (e.g., VM), storage volume, virtual cloud network (VCN or subnet), database, etc. The resource handle can enable the customer to access and manage the provisioned resources through APIs or command line tools.

[0268] In step S10, the status and resource handle are forwarded to user 1601 associated with customer C1 of reseller R1. In step S11, the control plane may notify the RM regarding the resource provision status, e.g., success or failure. In step S12, the RM may update a record in resource DB 1606 that tracks resource usage for customer C1 associated with the user. Continuing with the above example, the record may be updated to indicate that the total amount of memory used by customer C1 is 90 GB after successfully provisioning 10 GB in addition to the previously provisioned 80 GB, or that it remains at 80 GB after the user's request was denied. Records maintained by the resource DB may aid in the enforcement of policies associated with the vPLC.

[0269] Figure 17 is a flow diagram illustrating a resource allocation and provisioning process for a reseller's customer associated with a vPLC, according to an embodiment. Figure 17 is an alternative embodiment to Figure 16. Instead of using a resource manager (RM) as a central role for communicating with all subsystems and receiving requests to coordinate the resource allocation and provisioning process, a control plane (CP) may also be used to play such a central role.

[0270] For example, in FIG. 17 , in step S1, a user 1701 associated with a customer C1 of reseller R1 can issue a request to a vPLC-specific endpoint, i.e., vPLC.R1 endpoint 1702. The vPLC.R1 endpoint 1702 can augment the request to indicate that the request is coming from a customer of reseller R1 by tagging it with a corresponding vPLC ID in step S2 and forwards the request to a control plane (CP) 1704. In step S3, the CP 1704 forwards the request to a resource manager (RM) 1706. In step S4, the RM 1706 queries its database (i.e., resource database) 1708 to determine whether the requested resources are available for vPLC.R1. Based on the query, in step S5, the RM performs resource allocation if the requested resources are available.

[0271] In step S6, the RM obtains from the resource database the resource ID assigned to the resource (vPLC.R1.C1) allocated to the tenancy (T.R1.C1) of reseller R1's customer C1. In step S7, the RM passes the resource ID to CP 1704. In step S8, the control plane can provision the requested resource. In step S9, the control plane 1704 can respond to vPLC.R1 endpoint 1702 with the status and resource handle. In step S10, the status and resource handle can be forwarded to user 1701. In step S11, the control plane can notify the RM regarding the resource provision status. In step S12, the RM can update a record in the resource DB that tracks resource usage of the customer associated with the user.

[0272] 17, the CP 1704 can play a central role by forwarding the resource request to the RM 1706 (e.g., step S3), receiving a resource ID from the RM (e.g., step S6), and provisioning the requested resources (e.g., step S8). Compared to FIG. 16, this embodiment avoids the need for the vPLC.R1 endpoint 1702 to perform extra steps, such as making a resource request to the CP, as shown in steps S7 and S8 of FIG. 16.

[0273] FIG. 18 is a simplified diagram illustrating a use case in which a vPLC is hosted by infrastructure in a first realm but associated with a second, different realm, according to one embodiment. In FIG. 18, there are two separate realms provided by a CSP: realm A 1802 and realm B 1812. Each realm is a logical collection of multiple regions. The realms are isolated from each other and do not share data. Thus, realm A 1802 is isolated from realm B 1812, and no data or communications are shared between the two realms. A tenancy created for a CSP's customer exists within a single realm and cannot be used to access another realm or region within another realm. Realms allow a CSP to provide a defined level of service across regions within that realm that meets specific organizational needs. For example, one realm may be created by a CSP for an enhanced level of security and isolation, such as a “government” realm created for a government agency that is a customer of the CSP. A second realm with a lower level of security and isolation may be created by the CSP, such as a "Commercial" realm for non-government commercial customers of the CSP. Examples of realms that may be created by a CSP include (a) a Commercial realm, (b) a U.S. Government FedRAMP-certified realm, (c) a U.S. Government IL5-certified realm, (d) a Country A Government realm, (e) a Country B Government realm, etc. For example, in FIG. 18, Realm A 1802 may have a higher security posture than Realm B 1812.

[0274] A realm can have one or more regions. Each realm has its own identity and trust profile (e.g., passwords, authentication information, etc.). This identity and trust profile information is shared between regions within the same realm, so regions within a realm can communicate with each other. The identity and trust profile of a realm is configured so that cross-realm communication is not enabled.

[0275] In the example depicted in Figure 18, realm A 1802 includes region A 1804. The infrastructure provided by the CSP within region A 1804 is organized as datacenter 1806. In Figure 18, datacenter 1806 is shown as a dedicated regional cloud datacenter (DRCC) hosted on the customer premises. A DRCC may refer to a fully managed cloud region built with CSP-specified high-performance infrastructure to help customers bring cloud primitives and services closer to their existing data and applications. A DRCC can be dedicated to a single customer and operates inside the customer's datacenter, but is fully managed by the CSP.

[0276] In FIG. 18 , realm B 1812 includes region B 1814. The infrastructure provided by the CSP in region B 1814 is organized as datacenter 1816. As shown in FIG. 18 , datacenter 1816 hosts vPLC 1818. Thus, the portion of the infrastructure provided by the CSP in datacenter 1816 in region B 1812 is allocated to vPLC 1818. Thus, vPLC 1818 is physically located within region B 1814 in realm B 1812.

[0277] As previously mentioned, when a vPLC is created, as part of configuring the vPLC, the vPLC is associated with a particular realm. Typically, a vPLC is associated with the same realm associated with the region in which the CSP-provided regional infrastructure hosting the vPLC is located. In some embodiments, when a vPLC is configured, it may be associated with a realm different from the realm associated with the regional infrastructure hosting the vPLC. For example, in the example depicted in FIG. 18 , vPLC 1818 may be associated with Realm A 1802 instead of Realm B 1812. The dashed box 1820 and associated arrow 1830 are intended to indicate that vPLC 1818 is physically hosted by datacenter 1816 in Realm B 1812, but is logically or virtually associated with and belongs to Realm A 1802. Realm B 1812 is referred to as the hosting realm or host realm of vPLC 1818. Realm A 1802 is referred to as the logical or virtual realm of the vPLC 1818.

[0278] This places vPLC 1818 and datacenter 1806 in the same realm, i.e., realm A 1802. Because infrastructure in the same realm shares the same identity and trust profile, vPLC 1818 is configured with the identity and trust profile of realm A 1802. Configuring identity and trust profile information for realm A 1802 can be achieved by creating a mapping in vPLC 1818. For example, an identity service API in the virtual realm (i.e., realm A 1802) may call a vPLC in the host realm (i.e., realm B 1812), which can accept the call and establish trust (i.e., share identity and trust profile information) between both realms. When an instance is created in a customer tenancy in a vPLC in the host realm, the resource ID and tenancy ID associated with the newly created instance in the host realm can be mapped to a new resource ID and tenancy ID in the virtual realm. Thus, a table of mappings (or a “shadow tenancy”) may be maintained within the host realm (i.e., Realm B 1812). The same process and mappings are also implemented within the virtual realm (i.e., Realm A 1802). Thus, both vPLC 1818 and Realm A 1802 (i.e., the virtual realm) share the same identity and trust profile information. As a result, because vPLC 1818 and datacenter 1806 are within Realm A 1802, they can communicate with each other even though vPLC 1818 is physically hosted by infrastructure in different Realm B 1812. vPLC 1818 may not be able to communicate with other infrastructure in Region B 1814 associated with Realm B 1812.

[0279] Additionally, vPLC 1818 is also logically associated with region A 1804, even if the vPLC is physically hosted by region B 1814. Region B 1814 is referred to as the hosting region or host region of vPLC 1818. Region A 1804 within realm A 1802 is referred to as the logical region or virtual region of vPLC 1818. The virtual realms and virtual regions of a vPLC may be configured when the vPLC is created.

[0280] As described with respect to the resource ID of a resource, it includes a portion that identifies the realm associated with the resource and a portion that identifies the region associated with the resource. A vPLC is considered a resource and is therefore assigned a resource ID. In one embodiment, the vPLC ID, which uniquely identifies a vPLC, is a type of resource ID. The vPLC ID includes a portion that identifies the virtual realm associated with the vPLC identified by the vPLC ID and a portion that identifies the virtual region associated with the vPLC. Thus, for purposes of managing identities in Realm A 1802, a vPLC is considered to be part of the realm and region identified by the vPLC ID corresponding to the vPLC.

[0281] In one embodiment, configuring a vPLC in the manner described above, where a vPLC is hosted by one region and one realm but associated with different virtual regions and different virtual realms, may be implemented using virtual bootstrap environment (ViBE) technology. A ViBE may refer to a virtual cloud network (VCN) provisioned within an overlay of an existing region (e.g., a “host region”). Once provisioned, the ViBE connects to the new region using a communication channel (e.g., an IPSec tunnel VPN). Certain required core services (or “seed” services), such as a deployment orchestrator, public key infrastructure (PKI) services, etc., may be provisioned within the ViBE. These services can provide the capabilities necessary to bring hardware online, establish a chain of trust within the new region, and deploy the remaining services within the new region. Using a virtual bootstrap environment can prevent circular dependencies between bootstrap resources by utilizing resources from the host region. Services can be staged and tested in the ViBE before the physical region (e.g., the target region) is available.

[0282] For example, in some embodiments, Realm B 1812 can create a ViBE within its realm, which is shared with identity and trust profile information of Realm A 1802. The ViBE within Realm B 1812 can then be used to bootstrap new regions within Realm A 1802. Further details related to ViBEs are described in the following patent applications, which are incorporated herein by reference for all purposes: (1) U.S. Nonprovisional Patent Application No. 18 / 105,779, filed February 3, 2023, entitled "VIRTUAL BOOTSTRAP ENVIRONMENT FOR BUILDING REGIONAL DATA CENTERS"; (2) U.S. Nonprovisional Patent Application No. 18 / 105,768, filed February 3, 2023, entitled "TECHNIQUES FOR A VIRTUAL BOOTSTRAP ENVIRONMENT IN A DISTRIBUTED VIRTUAL PRIVATE NETWORK," and (3) U.S. Nonprovisional Patent Application No. 18 / 105,779, filed February 3, 2023, entitled "TECHNIQUES FOR MIGRATING SERVICES FROM A VIRTUAL BOOTSTRAP ENVIRONMENT."

[0283] The architecture depicted in FIG. 18 , in which vPLCs are hosted by one region and one realm but associated with different virtual regions and different virtual realms, can be used for a variety of purposes. For example, in the example depicted in FIG. 18 , vPLC 1818 can serve as a backup for datacenter 1806. This makes it possible to have a backup for disaster recovery (DR) in a different geographic area. For example, if datacenter 1806 is a DRCC located on a customer's premises, the customer can ask a CSP to provide backup services for DRCC 1806. In response, the CSP can select vPLC 1818 to back up DRCC 1806, where vPLC 1818 is physically hosted in region B 1812, which is far away from region A 1802 that hosts DRCC 1806 and in a different geographic area. This can provide a more cost-effective solution for the CSP instead of having to build a separate datacenter for the customer in realm A 1802 solely for backup purposes. In some embodiments, the customer may not even know where the backup is performed, i.e., the existence and location of the vPLC 1818 may be invisible to the customer. The vPLC used for backup may be physically hosted in a CSP-provided cloud in a different realm and region.

[0284] Because the goal of the DR site is to introduce geographic separation between the primary and DR sites and minimize the occurrence of correlated failures by avoiding simultaneous changes to both the primary and secondary locations, the DRCC 1806 can enjoy DR guarantees given that the CSP can guarantee that the DRCC and its paired vPLC 1818 (and by extension, its hosting region in Realm B 1812) will not be changed at the same time.

[0285] 19 is a flow diagram 1900 illustrating an exemplary method for creating a vPLC, where the vPLC is hosted by a CSP-provided infrastructure in a particular region in a particular realm, but is virtually associated with a different region in a different realm, according to an embodiment. The process depicted in FIG. 19 may be performed as part of the process performed to create the vPLC. In an embodiment, the process depicted in FIG. 19 may be performed primarily by the CSP-provided infrastructure in the hosting region and realm.

[0286] Processing begins at 1902 when information is received identifying a host region within a host realm in which a vPLC to be created will be physically hosted. For example, for the embodiment depicted in Figure 18, information may be received that a vPLC will be created in region B 1814 within realm A 1812. The region and realm identified at 1902 represent the host region and host realm for the vPLC to be created.

[0287] At 1904, information may be received identifying a region within a particular realm, the particular realm being different from the host realm identified at 1902, and the region and particular realm identified at 1904 to be associated as a virtual region and virtual realm for the vPLC to be created. For example, for the embodiment depicted in FIG. 18, information may be received that the vPLC will be virtually associated with Region A 1804 within Realm A 1802.

[0288] At 1906, identity and trust profile information is obtained for the particular realm identified at 1904. The information obtained at 1906 may include identity authentication information, including passwords, certificates, etc., associated with the realm that will become the virtual realm for the vPLC.

[0289] At 1908, a vPLC ID is generated for the vPLC to be created, the vPLC ID identifying the region and particular realm identified in the information received at 1904.

[0290] At 1910, a vPLC is created in the host realm and resources are allocated to the vPLC, the allocated resources being selected from resources in the infrastructure provided by the CSP that are within the region identified in 1902 and within the realm identified in 1902. These resources may include one or more physical resources (e.g., racks, servers, routers, memory, or storage resources) and / or logical resources (e.g., hypervisors, virtual machines, virtual routers). For example, for the embodiment depicted in FIG. 18, resources are allocated to the vPLC from data center 1816.

[0291] At 1912, a vPLC is configured using the identity and trust profile information obtained at 1906. For example, for the embodiment depicted in Figure 18, the vPLC may be configured using the identity and trust profile information of Realm A 1802.

[0292] At 1914, the vPLC initiates communication with the CSP-provided infrastructure within the particular realm identified in 1904, which is the vPLC's virtual realm. For example, for the embodiment depicted in Figure 18, the vPLC 1818 communicates with the data center 1806. The communication may be initiated by the vPLC 1818 or the data center 1806.

[0293] Virtual Private Label Cloud (vPLC) - Connectivity At a high level, traffic (e.g., packets) received by the CSP-provided infrastructure within a region from traffic origins outside the infrastructure belongs to one or two categories: (1) traffic destined for one or more vPLCs created within the CSP-provided infrastructure, or (2) non-vPLC traffic (e.g., traffic associated with non-reseller customers of the CSP). vPLC traffic may be destined for a specific vPLC corresponding to a reseller. Furthermore, because a reseller may have multiple customers, traffic may be destined for a specific customer of a reseller corresponding to a specific vPLC. Therefore, processes are implemented to properly identify or segregate traffic entering the CSP-provided infrastructure within a region so that the traffic is forwarded to its intended destination within the CSP-provided infrastructure within the region. More specifically, processes are implemented to ensure that traffic destined for a specific customer of a reseller corresponding to a specific vPLC is properly distinguished from other traffic received by the CSP-provided infrastructure within the region. In one embodiment, this is done by tagging packets destined for a vPLC with vPLC-related information that identifies or can be used to identify the vPLC to which the packet is directed and the particular customer of the vPLC.

[0294] Thus, in one embodiment, for a packet received by the regional CSP-provided infrastructure from an external source (external to the regional CSP-provided infrastructure), routing the packet to its intended destination includes (a) determining whether the received packet is destined for a destination associated with a vPLC, (b) upon determining that the packet is destined for a vPLC, identifying the particular vPLC to which the packet is destined and the particular customer of the reseller corresponding to the vPLC, (c) tagging the packet with information related to the vPLC and customer identified in (b), and (d) communicating the packet within the CSP-provided infrastructure within the region with the tagged vPLC-related information and communicating the packet to its intended destination (e.g., compute instance destination). In this manner, the vPLC-related information (e.g., vPLC ID) is used to route the packet within the CSP-provided infrastructure within the region as part of connectivity.

[0295] The vPLC-related information for that particular vPLC may include, for example, a vPLC identifier that identifies that particular vPLC, an identifier that identifies the customer associated with the endpoint, a virtual cloud network identifier that identifies the virtual cloud network (VCN) that belongs to that particular vPLC (where the endpoint is part of the VCN), and other vPLC-related information.

[0296] FIG. 20 is a simplified block diagram of a distributed environment 2000 including a CSP-provided infrastructure in a region including one or more vPLCs, according to an embodiment. The distributed environment 2000 shown in FIG. 20 is merely exemplary and is not intended to unduly limit the scope of the claimed embodiments. Many variations, alternatives, and modifications are possible. For example, in some other embodiments, the distributed environment 2000 may have more or fewer systems or components than those shown in FIG. 20, may combine two or more systems, or may have a different configuration or arrangement of systems. The systems, subsystems, and other components shown in FIG. 20 may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective systems, using hardware, or a combination thereof. The software may be stored on a non-transitory storage medium (e.g., on a memory device).

[0297] As shown, the distributed environment 2000 includes a CSP-provided region infrastructure 2002 within a region that may be used by a CSP to provide one or more CSP-provided cloud services to the CSP's customers. The infrastructure 2002 also includes multiple vPLCs. A vPLC may be created by a CSP for a particular reseller and used to provide the reseller-provided cloud service to one or more customers of the reseller. Each vPLC includes a set of physical and virtual or overlay network resources used to provide the reseller-provided service. Physical resources may include NVDs (e.g., implemented in the form of smartNICs or sNICs), host machines or servers, memory and storage resources, networking resources, etc. Virtual or overlay resources may include VCNs, compute instances (e.g., virtual machines) running on one or more servers or host machines, VNICs, etc.

[0298] In the example depicted in Figure 20, the CSP-provided infrastructure 2002 in the region includes two vPLCs: a first vPLC 2020 created for reseller R1 and a second vPLC 2060 created for reseller R2. vPLC 2020 can be used to provide reseller R1-supplied cloud services to one or more customers of reseller R1. vPLC 2060 can be used to provide reseller R2-supplied cloud services to one or more customers of reseller R2.

[0299] A vPLC can include one or more virtual cloud networks (VCNs). Workloads (e.g., in the form of compute instances) belonging to or associated with a reseller's customers can be deployed on these VCNs. One or more compute instances can be deployed on each VCN. The compute instances (e.g., virtual machines) may be executed by one or more host machines or servers on a rack. In the example depicted in FIG. 20 , vPLC.R1 2020 created for reseller R1 includes two VCNs, VCN 1 2030 and VCN 2 2050. vPLC.R1 2060 created for reseller R2 includes a single VCN, VCN 1 2030. VCN 1 2030 includes multiple compute instances, including compute instances 2032 and 2034. VCN 2 2050 also includes multiple compute instances, including compute instances 2052 and 2054. VCN 2 2070 , which belongs to or is associated with vPLC.R2 2060 , also includes multiple compute instances, including compute instances 2072 and 2074 .

[0300] In some embodiments, a VCN is allocated exclusively to a specific vPLC within a CSP-provided infrastructure within a region. For a VCN allocated to a vPLC, the VCN is also allocated exclusively to the specific customer of the reseller corresponding to the vPLC to which the VCN is allocated. In such embodiments, VCNs are not shared among vPLCs or among the customers of the reseller for whom the vPLC is created. As shown in FIG. 20 , in a vPLC.R1 2020 created for reseller R1, VCN 1 2030 is created and allocated exclusively to customer C1, where C1 is a customer of reseller R1 and C1 subscribes to one or more R1-provided cloud services offered using vPLC.R1 2020. VCN 2 2050 is created and allocated exclusively to customer C2, where C2 is another customer of reseller R1 and C2 subscribes to one or more R1-provided cloud services offered using vPLC.R1 2020. For vPLC.R2 2060 created for reseller R2, VCN 1 2070 is created and allocated exclusively to customer C3, where C3 is a customer of reseller R2 and C3 subscribes to one or more R2-supplied cloud services offered using vPLC.R2 2060.

[0301] Each VCN is identified using a VCN identifier (VCN Id). Thus, in embodiments in which a VCN is allocated exclusively to a particular vPLC and a customer of the reseller corresponding to the vPLC, a given VCN identifier can identify the particular vPLC to which the VCN belongs and the particular customer of the reseller to which the VCN is allocated. In one embodiment, infrastructure 2002 stores VCN mapping information 2016, which includes information identifying a set of VCN identifiers that correspond to VCNs created in CSP-provided infrastructure 2002 within a region. For each VCN identifier identifying a VCN, VCN mapping information 2016 may include information identifying (1) a vPLC identifier that identifies the vPLC to which the VCN belongs, and (2) a customer identifier that identifies the customer to which the VCN belongs, where the customer is a customer of the reseller for whom the vPLC is created (i.e., the reseller corresponding to the vPLC).

[0302] VCN mapping information 2016 can be used for a variety of purposes: Given a VCN identifier, VCN mapping information 2016 can be used to determine (1) a vPLC identifier that maps to the VCN identifier (the vPLC identifier identifies the vPLC to which the VCN identified by the VCN identifier belongs), and (2) a customer identifier that maps to the VCN identifier (the customer identifier identifies the reseller's customer that corresponds to the vPLC to which the VCN belongs).

[0303] As described above, a VCN may include one or more compute instances deployed or connected to the VCN. Each of these compute instances has an associated overlay address that can be used to communicate with the compute instance. For a VCN created in a CSP-provided infrastructure within a region, information may be stored that maps the overlay address of the compute instance to the corresponding VCN to which the compute instance is connected. For an overlay address associated with a compute instance, the mapping information may include information identifying a VCN identifier that identifies the VCN to which the compute instance belongs. Given the overlay address of a compute instance, the mapping information can be used to determine the particular VCN to which the compute instance corresponding to the overlay address is connected.

[0304] In one embodiment, the overlay address-VCN mapping information may be stored as part of VCN mapping information 2016. As previously described, VCN mapping information 2016 stores information identifying a set of VCN identifiers corresponding to VCNs created in CSP-provided infrastructure 2002 within a region, and for each VCN identifier identifying a VCN, stores information identifying (1) a vPLC identifier identifying a vPLC to which the VCN belongs, and (2) a customer identifier identifying a customer to which the VCN belongs, where the customer is a customer of the reseller for which the vPLC is created (i.e., the reseller corresponding to the vPLC). Additionally, information VCN mapping information 2016 may include information associating a VCN identifier with one or more overlay addresses of compute instances belonging to the VCN identified by the VCN identifier. Given an overlay address corresponding to a compute instance, the VCN mapping information 2016 can be used to (1) identify the VCN that encompasses the compute instance, (2) identify the vPLC that includes the identified VCN (i.e., the vPLC to which the VCN belongs), and (3) identify the reseller's customer to whom the compute instance and VCN are allocated (i.e., the customer to which the compute instance and VCN belong).

[0305] A compute instance is associated with a VNIC, which provides a virtual network interface for the compute instance and enables the compute instance to be part of or connect to a VCN. The VNIC may be implemented or performed by an NVD. In FIG. 20 , for vPLC.R1 2020, compute instances 2032, 2034, 2052, and 2054 are associated with VNICs 2042, 2044, 2062, and 2064, respectively. VNICs 2042 and 2044 are implemented by NVD 2040, and VNICs 2062 and 2064 are implemented by NVD 2060. For vPLC.R2 2060, compute instances 2072 and 2074 are associated with VNICs 2082 and 2084, respectively. VNICs 2082 and 2084 are implemented by NVD 2080.

[0306] In the example depicted in FIG. 20 , for compute instances within a particular VCN belonging to a vPLC, the VNICs associated with the compute instance are implemented by the same NVD. For example, VNICs 2042 and 2044 associated with compute instances 2032 and 2034 in VCN 1 2030 belonging to vPLC.R1 2020 are implemented (or executed) by NVD 2040. However, this is not intended to be limiting. In some other embodiments, VNICs associated with compute instances within a VCN may be implemented by different NVDs. In some embodiments, an NVD may implement VNICs associated with compute instances belonging to different VCNs, or even different vPLCs.

[0307] In some embodiments, one or more NVDs are allocated and associated exclusively with a vPLC. In such embodiments, for an NVD allocated exclusively to a particular vPLC, the NVD implements only VNICs associated with compute instances belonging to that particular vPLC. The compute instances themselves may be deployed on one or more VCNs associated with a particular vPLC.

[0308] In yet other embodiments, an NVD may be allocated exclusively to a particular vPLC and to a particular customer of the reseller corresponding to that particular vPLC. In such embodiments, each customer of the reseller has its own NVD or NVDs. This is done to ensure traffic isolation between the reseller's customers. In a hybrid scenario, some customers of the reseller may have their own dedicated NVD, while other customers of the reseller may share an NVD.

[0309] In yet other embodiments, the NVD may be allocated exclusively to a particular vPLC and to a particular VCN within a particular vPLC. A combination of the various NVD allocation embodiments may also be used.

[0310] In yet other embodiments, the NVD may be allocated exclusively to a particular vPLC and to a particular VCN within that vPLC. In embodiments where a VCN within a vPLC is allocated exclusively to a particular customer of the reseller from which the vPLC is created, the allocation of the NVD exclusively to a particular vPLC and to a particular VCN within the vPLC translates to the allocation of the NVD exclusively to the customer exclusively associated with the VCN.

[0311] The CSP-provided infrastructure 2002 within a region may include one or more gateways 2004. The gateways 2004 represent ingress and egress points for traffic entering and leaving the CSP-provided infrastructure 2002 within a region. In the ingress direction, packets originating from a source outside the infrastructure 2002 and destined for a destination within the infrastructure 2002 are received by the ingress gateway 2004. The received packets are then routed within the CSP-provided infrastructure 2002 within the region so that the packets are received by their intended destination. In the egress direction, packets originating from a source within the infrastructure 2002 and destined for a destination outside the CSP-provided infrastructure 2002 within the region are routed internally within the infrastructure 2002 and received by the egress gateway 2004. The egress gateway then forwards the packets out of the CSP-provided infrastructure 2002 within the region, facilitating delivery of the packets to their intended destination outside the infrastructure 2002. The gateways may be configured as ingress gateways, egress gateways, or both ingress and egress gateways. The CSP provisioning infrastructure 2002 within the region may include a communications network 2014 to facilitate communication of packets between the gateways 2004 and the vPLCs.

[0312] The gateway 2004 can include various types of gateways. For example, the gateway may include a dynamic routing gateway (DRG) 2008 configured to facilitate communication of traffic between the CSP-provided infrastructure 2002 within a region and an endpoint within the customer on-premises network 2012. In one embodiment, a DRG can be added to or associated with a customer VCN belonging to a vPLC and provide a pathway for private network traffic communication between the customer VCN and another endpoint, where the other endpoint can be within the customer's on-premises network. The customer on-premises network may be a customer enterprise network or customer data center built using the customer's resources. This enables the customer to extend the customer's on-premises network to be able to communicate with a VCN implemented in the cloud using the CSP-provided infrastructure. For example, a DRG may be associated with VCN 2030 within vPLC.R1 2020 and enable communication between the on-premises or enterprise network of reseller R1's customer C1 and compute instances deployed on VCN 2030. To enable such communication, a communication channel is typically set up, where one endpoint of the channel is in the customer on-premises network and the other endpoint is in the CSP-provided infrastructure 2002 within the region. The communication channel can be over a public communication network such as the Internet or a private communication network. A variety of different communication protocols may be used, such as IPsec VPN technology over a public communication network such as the Internet, or FastConnect technology that uses a private network instead of a public network.

[0313] As another example, gateway 2004 may include an Internet gateway (IGW) 2006 configured to facilitate communications between infrastructure 2002 and public endpoints accessible via a public network 2010, such as the Internet. IGW 2008 may be used to initiate communications from points within the CSP-provided infrastructure within a region or from public endpoints accessible via the Internet.

[0314] Different techniques may be used to implement the gateway 2004. The gateway can be implemented using software only, using hardware, or using a combination of software and hardware. The gateway can be implemented as a logical or virtual or overlay network construct, such as a virtual router, run by a host machine or server, a Linux device, an NVD, etc. The gateway can also be implemented as a physical network device, such as a physical router. The gateway can be dynamically configured as needed.

[0315] In some embodiments, a single gateway can be configured to facilitate communications for multiple vPLCs. In some other embodiments, a gateway can be allocated to and dedicated to a particular vPLC and configured to facilitate communications to and from endpoints belonging to that vPLC. In such embodiments, only the gateway allocated to a vPLC handles communications to and from endpoints (e.g., compute instances) belonging to that vPLC. For example, in the embodiment depicted in FIG. 20, NVDs 2040 and 2060 are allocated exclusively to vPLC.R1 2020, and NVD 2080 is allocated exclusively to vPLC.R2 2060.

[0316] In yet other embodiments, gateways may be dedicated to a particular vPLC and to a particular customer of the reseller corresponding to that vPLC. In such embodiments, only the gateway allocated to a vPLC handles communications to and from endpoints (e.g., compute instances) belonging to that vPLC. For example, in the embodiment depicted in FIG. 20, NVD 2040 is allocated exclusively to customer C1 of reseller R1, NVD 2060 is allocated exclusively to customer C2 of reseller R1, and NVD 2080 is allocated exclusively to customer C3 of reseller R2.

[0317] In one embodiment, when a vPLC is created, it is allocated its own unique public address pool. This address pool corresponds to a range of public addresses allocated to the vPLC. Each vPLC is allocated a unique address range. In this way, no vPLC has any overlapping addresses. The gateway 2004 exposes these addresses to endpoints outside the CSP-provided infrastructure 2002 within the region, allowing endpoints outside the infrastructure 2002 to communicate with endpoints (e.g., compute instances) belonging to the vPLC. For example, a first vPLC may be allocated the address range 1.0.0.0 / 16 (i.e., "1.0.0.0" to "1.0.255.255"), a second vPLC may be allocated the address range 1.1.0.0 / 16 (i.e., "1.1.0.0" to "1.1.255.255"), and so on. Each vPLC thus gets a non-overlapping slice of addresses. As will be described below, this range information is used to separate traffic among different vPLCs.

[0318] Information 2018 identifying various vPLCs created using the CSP-provided infrastructure 2002 within the region (referred to as vPLC-address range information) and, for each vPLC, information identifying a range of public addresses allocated to the vPLC may be stored. Information 2018 may be in the form of a lookup table. vPLCs may be identified using their vPLC identifiers (vPLC IDs). When a packet is received by a gateway with a particular destination address corresponding to a destination within the CSP-provided infrastructure 2002 within the region, the gateway can use information 2018 to (1) determine whether the packet is addressed to an endpoint associated with a vPLC or an endpoint not associated with a vPLC by checking whether the destination address is within a range identified in information 2018, and (2) for an endpoint associated with a vPLC, determine the particular address range within which the destination address falls and the particular vPLC that corresponds to that address range. The gateway can then appropriately route the packet to its intended destination.

[0319] As indicated above, unique public address ranges can be allocated to individual vPLCs. In some embodiments, at the vPLC level from the address range allocated to the vPLC, unique address subranges may be allocated to customers of the reseller corresponding to the vPLC, with each customer being allocated a unique address subrange that does not overlap with subranges allocated to other customers of the reseller corresponding to the vPLC. In this way, each customer of the reseller obtains its own unique address range (or slice of addresses) from the addresses allocated to the vPLC. In such embodiments, customers of the reseller corresponding to the vPLC do not have any overlapping addresses. In such embodiments, information 2018 may additionally include, for each vPLC, a list of customers of the reseller corresponding to the vPLC and, for each customer, information identifying the address subrange allocated to that customer. Customers can be identified using a customer ID. When a packet is received by the gateway with a particular destination address corresponding to a destination in CSP-provided infrastructure 2002 within a region, the gateway can use information 2018 to (1) determine whether the packet is addressed to an endpoint associated with a vPLC or an endpoint not associated with a vPLC by checking whether the destination address is within a range identified in information 2018, and (2) for an endpoint associated with a vPLC, (a) determine the particular address range within which the destination address falls and determine the particular vPLC that corresponds to that address range, and (b) determine the particular customer of the reseller that corresponds to the identified vPLC based on the destination address. The gateway can then appropriately route the packet to its intended destination.

[0320] In some embodiments, a DDoS (Distributed Denial of Service) scrubber 2019 may be provided to detect DDoS attacks and take corrective or preventative action therefor. For example, the DDoS scrubber 2019 may be used to identify attacks against a vPLC. When an attack against a particular vPLC is detected, the scrubber 2019 may initiate one or more actions to mitigate the attack. In certain embodiments, security policies for a vPLC may be configured. For example, an attack detection policy may be defined for a vPLC that specifies or controls the detection of DDoS attacks against the vPLC. A mitigation policy may be defined for a vPLC that specifies how detected attacks should be handled for the vPLC. The mitigation policy may identify one or more mitigation actions that should be initiated when an attack is detected. Security policies for a vPLC, including any detection and mitigation policies, may be configurable and customizable by the reseller for which the vPLC is created. In some embodiments, the CSP may also specify several security policies targeted to the vPLC. These policies may also be customizable by the reseller for their individual vPLCs.

[0321] A packet may traverse a variety of different paths within a regional CSP-provided infrastructure, such as the regional CSP-provided infrastructure 2002 depicted in FIG. 20. The path a packet traverses may depend, among other things, on the source of the packet and the intended destination of the packet. Examples of these paths include: (1) a packet is received originating from a source outside infrastructure 2002 and its destination is a compute instance within a VCN belonging to a particular vPLC; (2) a packet originates at a compute instance within a VCN belonging to a particular vPLC and its destination is an endpoint outside of the regional CSP-provided infrastructure 2002; or (3) a packet originates at a compute instance within a VCN belonging to a particular vPLC and its destination is a compute instance within the same or a different vPLC. Further details describing the processes performed to route packets are provided below in FIGS. 21, 22, and 23 and the accompanying discussion.

[0322] FIG. 21 shows an example flow diagram 2100 depicting a method for routing packets originating from a source outside the regional CSP-provided infrastructure and destined for an endpoint belonging to a vPLC, according to an embodiment. The process shown in FIG. 21 may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective system, using hardware, or a combination thereof. The software may be stored on a non-transitory storage medium (e.g., on a memory device). The method presented in FIG. 21 and described below is intended to be exemplary and non-limiting. While FIG. 21 depicts various processing steps occurring in a particular sequence or order, this is not intended to be limiting. In an alternative embodiment, the process may be performed in some different order, or some steps may also be performed in parallel. It should be appreciated that in alternative embodiments, the process shown in FIG. 21 may include more or fewer steps than depicted in FIG. 21.

[0323] The process begins at 2102 when a packet originating from a source outside the CSP-provided infrastructure in the region is received at a gateway in the CSP-provided infrastructure in the region. For example, in the embodiment depicted in Figure 20, the IGW 2006 may receive a packet from an endpoint accessible via the public network 2010. As another example, the DRG 2008 may receive a packet from an endpoint in the on-premises network 2012.

[0324] At 2104, the gateway receiving the packet determines that the destination of the received packet is a compute instance that belongs to a specific vPLC. There are various ways in which the specific vPLC may be identified at 2104. According to a first technique, the processing performed at 2104 includes (1) determining the destination address from a header of the received packet, (2) determining a specific address range within which the destination address falls from among multiple address ranges associated with the vPLC, (3) determining a vPLC associated with the specific determined address range, and (4) identifying the vPLC as the specific vPLC at 2104. Thus, in this technique, a specific vPLC is identified based on the range of public addresses allocated to each vPLC.

[0325] The packet received at 2102 may include a header that includes several fields, including, for example, the following fields: (1) A source address field in which the address of the source of the packet (e.g., IP address) is entered. (2) A destination address field in which the address (e.g., IP address) of the packet's intended destination endpoint (e.g., destination compute instance) is entered. If the destination is an endpoint within an overlay network, the endpoint's overlay address (e.g., overlay IP address) is entered in this field. For example, if the destination is compute instance 2032 in FIG. 20, the overlay public IP address of compute instance 2032 is entered in this field. (3) A protocol field in which information identifying the communication protocol is entered (e.g., TCP). (4) a source port field that is populated with information identifying the port associated with the source of the packet; and (5) A destination port field that is populated with information identifying the port associated with the packet's destination endpoint.

[0326] In one embodiment, the vPLC-address range information facilitates the process performed in 2104 to identify a particular destination vPLC. For example, in FIG. 20 , when a packet is received from a public network endpoint, the IGW 2006 can determine the destination address of the received packet from a destination address field in the header of the received packet. The IGW 2006 can then use the vPLC-address range information 2018 to identify a particular address range within which the destination address resides. The IGW 2006 can then determine a particular vPLC associated with the identified particular address range.

[0327] As mentioned above, in some embodiments, separate dedicated gateways may be provided for separate vPLCs. In such embodiments, a separate technique is to use the Border Gateway Protocol (BGP) to route packets destined for endpoints belonging to a particular vPLC to a specific dedicated gateway allocated to that particular vPLC. The vPLC to which the receiving gateway is dedicated is identified as the specific vPLC in 2104.

[0328] At 2106, the gateway receiving the packet determines that the destination of the received packet is a compute instance belonging to a particular customer of the reseller that corresponds to the particular vPLC identified at 2104. There are various ways in which the particular customer may be identified at 2104. Once the destination address and the address range allocated to the corresponding vPLC are known at 2104, the process at 2106 involves identifying a particular sub-address range that encompasses the destination address from among the sub-ranges allocated to the reseller's customers. The customer that corresponds to the particular sub-address range is then identified as the particular customer at 2106. In one embodiment, vPLC-address range information 2018 may be used to identify the particular customer.

[0329] Although 2104 and 2106 are shown as two separate processing steps in flow diagram 2100 of Figure 21, this is not intended to be limiting. The processing performed in the two steps may be performed as one step, where the destination address in the header of the received packet is determined, and then vPLC-address-range information 2018 is used to determine the particular vPLC that contains the compute instance corresponding to the destination address and the particular customer to which the compute instance belongs.

[0330] At 2108, the gateway adds an encapsulation header to the received packet to create an encapsulated packet, where the encapsulation header contains, among other pieces of information, information related to the particular vPLC identified in 2104 and / or the particular customer identified in 2106. There are various ways this may be implemented. In one embodiment, the encapsulation header may include one or more fields, including a "vPLC field" for storing information identifying the particular vPLC determined in 2104 and a "customer field" for storing information identifying the particular customer identified in 2106.

[0331] For example, in one embodiment, the encapsulation header may include the following fields and information that may be entered into each field as part of the processing performed at 2108, as follows: (1) Source Substrate Address Field - The substrate IP address associated with the receiving gateway is entered in this field. If the gateway is implemented as a virtual router, the substrate IP address of the NVD implementing the VNIC associated with the virtual router is entered in this field. If the gateway is implemented as a physical router, the substrate IP address of the router is entered in this field. (2) Destination Substrate Address Field—The substrate IP address of the NVD implementing the VNIC associated with the compute instance corresponding to the destination address determined in 2104 is entered in this field. For example, if the destination of the received packet is compute instance 2032 belonging to vPLC.R1 2020, the substrate IP address of NVD 2040 is entered in this field. As another example, if the destination of the received packet is compute instance 2072 belonging to vPLC.R2 2060, the substrate IP address of NVD 2080 is entered in this field. (3) Destination VNIC Identifier Field—A VNIC identifier that identifies the VNIC associated with the compute instance that corresponds to the destination address determined in 2104 is entered in this field. For example, if the destination of the received packet is compute instance 2032 that belongs to vPLC.R1 2020, the VNIC identifier of VNIC 2042 is entered in this field. As another example, if the destination of the received packet is compute instance 2072 that belongs to vPLC.R2 2060, the VNIC identifier of VNIC 2082 is entered in this field. (4) One or more destination vPLC-related fields—fields that store information related to the destination vPLC. The information entered in these fields may specifically identify the destination vPLC (e.g., using a vPLC ID), the customer, etc., or may include information that can be used to determine the destination vPLC, the customer, and other information. Example fields include:

[0332] (a) A "vPLC field" may be entered in this field for storing information, such as a vPLC identifier, that identifies the particular vPLC determined in 2104; and (b) A "customer field" for storing information such as a customer identifier that identifies the particular customer determined in 2106. If the packet is not destined for an endpoint associated with a vPLC, these vPLC-related fields may be left blank or marked as not applicable.

[0333] In some other embodiments, instead of or in addition to the vPLC and customer fields, the vPLC-related fields in the encapsulation header may include a "VCN identifier field." When this field is present, a VCN identifier corresponding to the VCN containing the destination address is entered in this field. The receiving gateway can use the VCN mapping information to determine the VCN containing the destination address determined in 2104. As described above, the VCN mapping information can identify the VCN and, for each VCN, one or more overlay addresses of the compute instances that are part of the VCN. Thus, given a destination address, the VCN mapping information can be used by the gateway to identify a VCN identifier that identifies the VCN containing the destination address. The identified VCN identifier is then entered in the VCN identifier field. In embodiments in which the "VCN identifier field" is used instead of the "vPLC field" and "customer field" fields, the vPLC-related information can be efficiently communicated using a single field in the header instead of using multiple fields.

[0334] The fields of the encapsulation header described in this disclosure are examples only and are not intended to limit the scope of the claimed embodiments. In alternative embodiments, the encapsulation header may include more or fewer fields than those described in this disclosure. For example, in some implementations, the encapsulation header may also include a field for specifying a tenancy identifier associated with the reseller.

[0335] At 2110, using the encapsulation header, the encapsulated packet is communicated from the gateway to the NVD implementing the VNIC associated with the compute instance that is the packet's intended destination (i.e., the compute instance is associated with the destination address determined in 2104). A variety of different communication protocols may be used to facilitate this communication. In particular embodiments, the encapsulated packet is communicated from the receiving gateway using a tunneling protocol. A variety of different tunneling protocols may be used, such as GENEVE (Generic Network Virtual Encapsulation), Virtual Extensible Local Area Network (VXLAN), etc.

[0336] An NVD implementing a VNIC associated with the destination compute instance receives the encapsulated packet at 2112. For example, if the destination of the received packet is compute instance 2032 belonging to vPLC.R1 2020, then an NVD 2040 implementing or running VNIC 2042 associated with compute instance 2032 receives the encapsulated packet at 2112.

[0337] At 2114, the NVD receiving the encapsulated packet decapsulates the encapsulated packet by removing the encapsulation header added to the packet at 2110 to create a decapsulated packet. The VNIC information included in the encapsulation header is used to identify the VNIC that should be used to route the packet. The encapsulation header also includes information that identifies a particular vPLC and a particular customer. In embodiments where the encapsulation header includes VCN identification information, this information can be used to determine the identity of the customer that corresponds to the vPLC and VCN.

[0338] At 2116, using information contained in the header of the decapsulated packet, the packet is forwarded from the NVD to a destination compute instance belonging to the particular vPLC determined at 2104 and the particular customer determined at 2106. The processing at 2116 may include communicating the decapsulated packet from the NVD to a host machine or server running the destination compute instance, where the packet is then forwarded to the destination compute instance.

[0339] As described above, when a packet destined for an endpoint belonging to a vPLC is received, the packet is tagged with vPLC-related information. The tagging information may be in the form of vPLC-related information included in an encapsulation header added to the packet. The vPLC-related information may identify a particular vPLC and / or a particular customer of a reseller associated with the destination endpoint. In some other embodiments, the vPLC-related information may include a VCN identifier that identifies a VCN that includes the endpoint. Tagging vPLC-related traffic with vPLC-related information has several technical advantages, as described in more detail below.

[0340] The destination of a packet originating from various sources may be a compute instance that belongs to or is associated with a particular vPLC and that is associated with a particular customer of a reseller corresponding to the particular vPLC, which may include, for example, a source outside of the CSP-provided infrastructure in a region, a compute instance that is within the CSP-provided infrastructure in a region but is not associated with any vPLC, a compute instance in a different vPLC than the particular vPLC, a compute instance that is within the same vPLC but that belongs to a different customer of the reseller, a compute instance that belongs to the same particular vPLC and associated with the same particular customer but that is in a different VCN than the compute instance receiving the packet, a compute instance that belongs to the same particular vPLC and associated with the same particular customer and that is in the same VCN as the compute instance receiving the packet, etc.

[0341] FIG. 22 shows an example flow diagram 2200 depicting a method for routing a packet originating from within a vPLC within a source outside the regional CSP-provided infrastructure and destined for an endpoint outside the regional CSP-provided infrastructure, according to an embodiment. For example, in the embodiment depicted in FIG. 20, the packet may originate at compute instance 2032 belonging to vPLC.R1 2020 and be destined for an end endpoint outside the regional CSP-provided infrastructure 2002. The process depicted in FIG. 22 may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective systems, using hardware, or a combination thereof. The software may be stored on a non-transitory storage medium (e.g., on a memory device). The method presented in FIG. 22 and described below is intended to be exemplary and non-limiting. While FIG. 22 depicts various processing steps occurring in a particular sequence or order, this is not intended to be limiting. In some alternative embodiments, the process may be performed in some different order, or some steps may also be performed in parallel. It should be appreciated that in alternative embodiments, the process shown in Figure 22 may include more or fewer steps than depicted in Figure 22.

[0342] The process may be triggered at 2202 when a packet originates at a source compute instance belonging to a vPLC. The packet's destination may be set to an endpoint outside of the CSP-provided infrastructure within the region. For example, in FIG. 20, the packet may originate at compute instance 2032 and have a destination endpoint within public network 2010. The packet may have a header containing multiple fields, and information may be entered into these fields as follows: (1) Source Address Field—The address (e.g., IP address) of the packet source is entered here. If the source compute instance is in an overlay network, the overlay address (e.g., overlay IP address) of the source compute instance is entered in this field. For example, if the source compute instance is compute instance 2032 depicted in FIG. 20, the overlay public IP address of compute instance 2032 is entered in this field. (2) Destination Address field - The address (e.g., IP address) of the packet's intended destination endpoint is entered here. If the destination is an endpoint within an overlay network, the endpoint's overlay address (e.g., overlay IP address) is entered in this field. (3) Protocol field - Information identifying the communication protocol (e.g., TCP) is entered in this field. (4) Source Port field—Information identifying the port associated with the source of the packet is entered. For example, if the source of the packet is compute instance 2032, the port associated with compute instance 2032 is entered in this field. And, (5) Destination Port Field - Information identifying the port associated with the packet's destination endpoint is entered in this field.

[0343] An NVD implementing a VNIC associated with the source compute instance receives the packet at 2202. For example, if the source of the packet is compute instance 2032 belonging to vPLC.R1 2020, then an NVD 2040 implementing or running VNIC 2042 associated with compute instance 2032 receives the packet at 2202.

[0344] The NVD receiving the packet determines that the destination of the received packet is an endpoint outside of the CSP-provided infrastructure within the region at 2204. As part of the processing at 2204, the NVD can (1) determine the destination address inserted into the destination address field of the received packet's header and (2) determine that the destination address is for an endpoint outside of the CSP-provided infrastructure within the region.

[0345] At 2206, the NVD determines a gateway of the CSP-provided infrastructure within the region that should be used to egress the packet from the CSP-provided infrastructure.

[0346] At 2208, the NVD adds an encapsulation header to the received packet to create an encapsulated packet, the encapsulation header including, among other pieces of information, information for routing the encapsulated packet from the NVD to the gateway determined at 2206. In particular embodiments, the encapsulation header may include the following fields that are populated with the following information: (1) Source Substrate Address Field - The substrate IP address of the NVD receiving the packet at 2202 is entered in this field. For example, if the source compute instance is compute instance 2032 depicted in Figure 20, the substrate IP address of the NVD 2040 is entered in this field. (2) Destination Substrate Address field - The substrate IP address of the gateway determined in 2206 is entered here. If the gateway is implemented as a virtual router, the substrate IP address of the NVD implementing the VNIC associated with the virtual router is entered in this field. If the gateway is implemented as a physical router, the substrate IP address of the router is entered in this field. (3) Destination VNIC Identifier field - If the gateway is implemented as a virtual router, information identifying the VNIC associated with the virtual router is entered here. (4) One or more destination vPLC-related fields—Since the destination endpoint does not belong to any vPLC, these vPLC-related fields may be left blank or may be marked as not applicable.

[0347] At 2210, the encapsulated packet is communicated from the NVD to the gateway identified at 2206 using the encapsulation header added at 2208. A variety of different communication protocols may be used to facilitate this communication. In one embodiment, the encapsulated packet is communicated using a tunneling protocol such as GENEVE (Generic Network Virtual Encapsulation), Virtual Extensible Local Area Network (VXLAN), or the like.

[0348] The gateway receives the encapsulated packet at 2212. At 2114, the gateway decapsulates the encapsulated packet by removing the encapsulation header added to the packet at 2208 to create a decapsulated packet.

[0349] At 2216, using the information contained in the decapsulated packet's header (the packet's header is described above with respect to 2201), the gateway forwards the packet out of the CSP-provided infrastructure within the region to facilitate communication of the packet to its intended destination.

[0350] FIG. 23 shows an example flow diagram 2300 depicting a method for routing packets between vPLCs in a CSP-provided infrastructure within a region, according to an embodiment. For example, in the embodiment depicted in FIG. 20, a packet may originate at compute instance 2032 belonging to vPLC.R1 2020 and be destined for compute instance 2072 belonging to vPLC.R2 2060 in CSP-provided infrastructure 2002 within a region. The process shown in FIG. 23 may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective systems, using hardware, or a combination thereof. The software may be stored on a non-transitory storage medium (e.g., on a memory device). The method presented in FIG. 23 and described below is intended to be exemplary and non-limiting. While FIG. 23 depicts various processing steps occurring in a particular sequence or order, this is not intended to be limiting. In an alternative embodiment, the process may be performed in some different order, or some steps may also be performed in parallel. It should be appreciated that in alternative embodiments, the process shown in FIG. 23 may include more or fewer steps than depicted in FIG.

[0351] The process may be triggered at 2302 when a packet originates at a source compute instance belonging to a first vPLC. The destination of the packet may be set to a compute instance in a second vPLC within the CSP-provided infrastructure in the region, the second vPLC being different from the first vPLC. For example, in FIG. 20 , the packet may originate at compute instance 2032 belonging to vPLC.R1 2020, and the destination endpoint of the packet may be compute instance 2072 belonging to vPLC.R2 2060.

[0352] The packet may have a header containing several fields, and information may be entered into these fields as follows: (1) Source Address Field—The address (e.g., IP address) of the packet source is entered here. If the source compute instance is in an overlay network, the overlay address (e.g., overlay IP address) of the source compute instance is entered in this field. For example, if the source compute instance is compute instance 2032 depicted in FIG. 20, the overlay public IP address of compute instance 2032 is entered in this field. (2) Destination Address Field—The address (e.g., IP address) of the packet's intended destination endpoint is entered here. If the destination is an endpoint within an overlay network, the endpoint's overlay address (e.g., overlay IP address) is entered in this field. For example, if the destination compute instance is compute instance 2072 depicted in FIG. 20, the overlay public IP address of compute instance 2072 is entered in this field. (3) Protocol field - Information identifying the communication protocol (e.g., TCP) is entered in this field. (4) Source Port field—Information identifying the port associated with the source of the packet is entered. For example, if the source of the packet is compute instance 2032, the port associated with compute instance 2032 is entered in this field. And, (5) Destination Port field—Information identifying the port associated with the packet's destination endpoint is entered in this field. For example, if the packet's destination is compute instance 2072, then the port associated with compute instance 2072 is entered in this field.

[0353] An NVD implementing a VNIC associated with a source compute instance receives the packet at 2302. Because an NVD is associated with a source, it may be referred to as a source NVD. For example, if the source of the packet is compute instance 2032 belonging to vPLC.R1 2020, then an NVD 2040 implementing or running VNIC 2042 associated with compute instance 2032 receives the packet at 2302.

[0354] At 2304, the source NVD determines that the destination of the received packet is a compute instance belonging to a second vPLC in the CSP-provided infrastructure in the region. The processing performed by the source NVD at 2304 may include (1) determining the destination address entered in a destination address field of the header of the received packet, (2) determining a particular address range within which the destination address falls from among multiple address ranges associated with the vPLC, and (3) determining a vPLC associated with the particular determined address range.

[0355] In one embodiment, the vPLC-address range information facilitates the process for identifying the destination vPLC, which is performed at 2304. For example, in FIG. 20 , the source NVD 2040 can use the vPLC address range information 2018 to determine that the destination address of the received packet is within the address range allocated to vPLC.R2 2060. vPLC.R1 2060 is then identified as the second vPLC at 2304.

[0356] At 2306, the source NVD can determine the identity of a customer associated with the destination computing instance. The customer identified at 2306 is a customer of the reseller corresponding to the destination vPLC. There are various ways in which a particular customer can be identified at 2306. According to one technique, a particular sub-address range encompassing the destination address is identified from the address range allocated to the destination address and from among the sub-ranges allocated to customers of the reseller corresponding to the destination vPLC. The customer associated with this particular sub-address range is the customer associated with the destination computing instance. In one embodiment, vPLC-address-range information 2018, which identifies the sub-ranges allocated to different customers of the reseller, may be used to identify a particular customer associated with the destination computing instance.

[0357] Although 2304 and 2306 are shown as two separate processing steps in flow diagram 2300 of Figure 23, this is not intended to be limiting. The processing performed in two steps may be performed in a single step.

[0358] At 2308, the source NVD adds an encapsulation header to the received packet to create an encapsulated packet, the encapsulation header including, among other pieces of information, information related to the second vPLC identified at 2304. For example, in one embodiment, the encapsulation header may include the following fields and information that may be entered into each field as part of the processing performed at 2308, such as: (1) Source Substrate Address Field—The substrate IP address of the source NVD is entered in this field. For example, if the source compute instance is compute instance 2032 depicted in FIG. 20, the substrate IP address of the NVD 2040 is entered in this field. (2) Destination Substrate Address Field—The substrate IP address of the NVD (also referred to as the destination NVD because it is associated with the destination compute instance) that implements the VNIC associated with the destination compute instance is entered in this field. For example, if the destination of the received packet is the compute instance 2072 that belongs to vPLC.R2 2060, the substrate IP address of the NVD 2080 that implements the VNIC 2082 that is associated with the destination compute instance 2072 is entered in this field. (3) Destination VNIC Identifier Field—A VNIC identifier that identifies the VNIC associated with the destination compute instance is entered in this field. For example, if the destination of the received packet is compute instance 2072 belonging to vPLC.R2 2060, the VNIC identifier of VNIC 2082 is entered in this field. (4) One or more destination vPLC-related fields—fields that store information related to the destination vPLC. The information entered in these fields may specifically identify the destination vPLC (e.g., using a vPLC ID), the customer, etc., or may include information that can be used to determine the destination vPLC, the customer, and other information. Example fields include:

[0359] (a) "vPLC field" - the vPLC ID identifying the destination vPLC (i.e., the second vPLC identified in 2304) is entered here.

[0360] (b) "Customer Field" - Information identifying the customer identified in 2306 is entered here.

[0361] (c) "VCN Identifier Field" - In some embodiments, this "VCN Identifier Field" is provided to store information identifying the VCN to which the destination compute instance is connected, instead of or in addition to the "vPLC" and "Customer" fields. The source NVD can use the VCN mapping information to determine the VCN that contains the destination address read from the header of the received packet in 2304. The VCN identifier that identifies the VCN may be entered in this field. For example, if the destination of the received packet is compute instance 2072 belonging to vPLC.R2 2060, the VCN identifier of VCN 2070 is entered in this field. In embodiments in which the "VCN Identifier Field" is used instead of the "vPLC" and "Customer" fields, vPLC-related information can be efficiently communicated using a single field in the header instead of using multiple fields.

[0362] At 2310, the encapsulated packet created at 2308 is communicated from the source NVD to the destination NVD using the encapsulation header. A variety of different communication protocols may be used to facilitate this communication. In one embodiment, the encapsulated packet is communicated using a tunneling protocol such as GENEVE (Generic Network Virtual Encapsulation), Virtual Extensible Local Area Network (VXLAN), etc.

[0363] The destination NVD receives the encapsulated packet at 2312. For example, if the destination of the received packet is the compute instance 2072 belonging to vPLC.R2 2060, the encapsulated packet is communicated from the source NVD 2040 to the destination NVD 2080 at 2310 and received by the destination NVD 2080 at 2312.

[0364] At 2314, the destination NVD decapsulates the received encapsulated packet by removing the encapsulation header added to the packet at 2308 to create a decapsulated packet. At 2316, using the information contained in the decapsulated packet's header (the packet's header is described above with respect to 2301), the destination NVD forwards the packet to the destination compute instance.

[0365] As part of the processing at 2316, the VNIC associated with the destination compute instance is identified from information contained in the encapsulation header. Any policies and rules associated with the VNIC are executed. The packet is then forwarded to the destination compute instance based on the policies and rules. After the rules and policies have been satisfied, the packet is then forwarded from the destination NVD to the host machine or server running the destination compute instance, and then to the destination compute instance itself.

[0366] These policies may include, for example, security and firewall rules for a reseller associated with the vPLC that originated the packet (e.g., a first vPLC), as well as rules and policies associated with the destination vPLC (e.g., a second vPLC). These rules and policies can be used to control communications to and from the vPLC, such as controlling what traffic is allowed to be received or transmitted, who is allowed to communicate with the vPLC, etc.

[0367] In some embodiments, policies and rules associated with a VNIC may be configurable at the vPLC level, the customer level, the VCN level, or the level of an individual compute instance. At the vPLC level, the reseller corresponding to the vPLC (i.e., the reseller at which the vPLC is created) may specify or configure one or more rules and policies to be associated with the vPLC. These vPLC-level policies can then be associated with each VNIC associated with a compute instance belonging to the vPLC. This allows the reseller to control how traffic is communicated to and from the vPLCs created for the reseller.

[0368] At the customer level, a reseller corresponding to a vPLC may specify or configure one or more rules and policies to be associated with one or more individual customers of the vPLC. The customer-level policies are then associated with VNICs associated with compute instances within the vPLC that belong to that customer. The rules and policies configured for one customer may be the same or different from the rules and policies configured for another customer of the reseller. This allows the reseller to control how traffic is communicated to and from the individual customer level. In some embodiments, the customer may also be enabled to configure one or more rules and policies for the customer. This allows the individual customer to control how traffic is communicated to and from the compute instances belonging to the customer.

[0369] Rules and policies may also be configured at the VCN level. As previously described, a VCN may be allocated exclusively to a particular vPLC and, further, exclusively to a particular customer of a reseller corresponding to the vPLC. In such an embodiment, one or more rules and policies may be configured for a VCN associated with a customer of the reseller. These rules and policies may then be applied to the various compute instances that are part of that VCN to control communications to and from the VCN. The rules and policies may be configured, for example, by the customer or by the reseller.

[0370] Policies and rules may also be configured for individual compute instances or sets of compute instances. Rules or policies configured for a compute instance may be applicable to that compute instance and control communication of traffic to and from the compute instance. These policies and rules may be configurable by the reseller or by the reseller's customers.

[0371] For example, if the source compute instance is compute instance 2032 belonging to vPLC.R1 2020 and the destination compute instance is compute instance 2072 belonging to vPLC.R2 2060, the encapsulated packet is communicated from the NVD 2080 to the compute instance 2072. In the destination NVD 2080, the VNIC-related information included in the encapsulation header may be used to identify the VNIC 2082 associated with the destination compute instance 2072. As part of the processing at 2316, any rules and policies specified for the VNIC 2082 may then be identified and applied. The packet may then be communicated to the destination compute instance 2072 after executing these rules and policies. If any rules or policies prevent the packet from being communicated to the compute instance 2072, the packet may be dropped and not forwarded to the compute instance 2072.

[0372] 24 shows an example flow diagram 2400 depicting a method for routing packets between compute instances within the same vPLC (e.g., intra-vPLC communication) within a CSP-provided infrastructure within a region, according to an embodiment. For example, in the embodiment depicted in FIG. 20, a packet may originate at compute instance 2032 belonging to vPLC.R1 2020 and customer C1 and be destined for compute instance 2034 also belonging to vPLC.R1 2020 and customer C1. As another example, in the embodiment depicted in FIG. 20, a packet may originate at compute instance 2052 belonging to vPLC.R1 2020 belonging to customer C1 and be destined for compute instance 2052 also belonging to vPLC.R1 2020 but belonging to a different customer C2 of reseller R1.

[0373] The process shown in FIG. 24 may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective systems, using hardware, or a combination thereof. The software may be stored on a non-transitory storage medium (e.g., on a memory device). The method presented in FIG. 24 and described below is intended to be exemplary and non-limiting. While FIG. 24 depicts various processing steps occurring in a particular sequence or order, this is not intended to be limiting. In an alternative embodiment, the process may be performed in some different order, or some steps may also be performed in parallel. It should be appreciated that in alternative embodiments, the process shown in FIG. 24 may include more or fewer steps than depicted in FIG. 24.

[0374] The process may be triggered at 2402 when a packet originates at a source compute instance belonging to a vPLC. The destination of the packet may be set to a compute instance within the same vPLC as the source compute instance. For example, in the embodiment depicted in FIG. 20, a packet may originate at compute instance 2032 belonging to vPLC.R1 2020 and customer C1 and be destined for compute instance 2034 also belonging to vPLC.R1 2020 and customer C1. As another example, in the embodiment depicted in FIG. 20, a packet may originate at compute instance 2052 belonging to vPLC.R1 2020 belonging to customer C1 and be destined for compute instance 2052 also belonging to vPLC.R1 2020 but belonging to a different customer C2 of reseller R1.

[0375] The packet may have a header containing several fields, and information may be entered into these fields as follows: (1) Source Address Field—The address (e.g., IP address) of the packet source is entered here. If the source compute instance is in an overlay network, the overlay address (e.g., overlay IP address) of the source compute instance is entered in this field. For example, if the source compute instance is compute instance 2032 depicted in FIG. 20, the overlay public IP address of compute instance 2032 is entered in this field. (2) Destination Address Field - The address (e.g., IP address) of the packet's intended destination endpoint is entered here. If the destination is an endpoint within an overlay network, the endpoint's overlay address (e.g., overlay IP address) is entered in this field. For example, if the destination compute instance is compute instance 2034 depicted in FIG. 20, the overlay public IP address of compute instance 2072 is entered in this field. As another example, if the destination compute instance is compute instance 2052 depicted in FIG. 20, the overlay public IP address of compute instance 2052 is entered in this field. (3) Protocol field - Information identifying the communication protocol (e.g., TCP) is entered in this field. (4) Source Port field—Information identifying the port associated with the source of the packet is entered. For example, if the source of the packet is compute instance 2032, the port associated with compute instance 2032 is entered in this field. And, (5) Destination Port Field—Information identifying the port associated with the packet's destination endpoint is entered in this field. For example, if the packet's destination is compute instance 2034 (or compute instance 2052), the port associated with compute instance 2072 (or the port associated with compute instance 2052) is entered in this field.

[0376] An NVD implementing a VNIC associated with a source compute instance receives the packet at 2402. Because an NVD is associated with a source, it may be referred to as a source NVD. For example, if the source of the packet is compute instance 2032 belonging to vPLC.R1 2020, then an NVD 2040 implementing or running VNIC 2042 associated with compute instance 2032 receives the packet at 2402.

[0377] At 2404, the source NVD determines that the destination of the received packet is a compute instance that belongs to the same vPLC as the source compute instance. Thus, it is determined that the destination vPLC (i.e., the vPLC to which the destination compute instance belongs) is the same vPLC as the source vPLC (i.e., the vPLC to which the source compute instance belongs). The processing performed by the source NVD at 2404 may include (1) determining the destination address entered in a destination address field of the header of the received packet, (2) determining a particular address range within which the destination address falls from among multiple address ranges associated with the vPLC, and (3) identifying a vPLC associated with the particular determined address range, and determining that the identified vPLC is the same vPLC to which the source compute instance belongs.

[0378] In one embodiment, the vPLC-address range information facilitates the process for identifying the destination vPLC, performed at 2404. For example, in FIG. 20 , the source NVD 2040 can use the vPLC address range information 2018 to determine that the destination address of the received packet is within the address range allocated to vPLC.R1 2020.

[0379] At 2406, the source NVD can determine the identity of the customer associated with the destination computing instance. The customer identified at 2406 is a customer of the reseller corresponding to the destination vPLC. Here, two possibilities exist: (1) The customer identified in 2406 is the same as the customer associated with the source compute instance. This may occur, for example, when the source compute instance is compute instance 2032 in FIG. 20 and the destination compute instance is compute instance 2034 (both belonging to the same customer C1). (2) The customer identified in 2406 is different from the customer associated with the source compute instance. The packet is being communicated between compute instances belonging to two different customers belonging to a reseller corresponding to the vPLC. This may occur, for example, when the source compute instance is compute instance 2032 in FIG. 20 belonging to R1's customer C1 and the destination compute instance is compute instance 2052 (both belonging to R1's customer C2).

[0380] There are various ways in which a particular customer can be identified in 2406. According to one technique, a particular sub-address range encompassing the destination address is identified from the address range allocated to the destination address and from among the sub-ranges allocated to different customers of the reseller corresponding to the destination vPLC. The customer associated with this particular sub-address range is the customer associated with the destination compute instance. In one embodiment, vPLC-address-range information 2018, which identifies the sub-ranges allocated to different customers of the reseller, may be used to identify the particular customer associated with the destination compute instance.

[0381] Although 24...

Claims

1. It is a method, Receiving packets in the first component of the CSP provision infrastructure within the region, The first component includes determining that the destination of the packet is an endpoint associated with a first virtual private label cloud (vPLC), The first vPLC is created for a first reseller using one or more resources from the Cloud Service Provider (CSP) provided infrastructure within the region, and the first vPLC is created to provide a set of cloud services supplied by one or more first resellers to one or more customers of the first resellers. The aforementioned method, The first component creates a tagged packet by tagging the packet using the first vPLC-related information, The further includes communicating the tagged packet using the first vPLC-related information from the first component to a second component within the CSP provision infrastructure in the region, The second component is a method associated with the endpoint.

2. The method according to claim 1, further comprising using a first portion of the CSP-providing infrastructure within the region to provide one or more CSP-provided cloud services to one or more customers of the CSP.

3. The aforementioned tagging is, The first component creates an encapsulated packet by adding an encapsulation header to the packet, The method according to claim 1, wherein the first component includes the first vPLC-related information within the encapsulation header.

4. The method according to claim 1, wherein the first vPLC-related information includes a first vPLC identifier that identifies the first vPLC.

5. Determining that the endpoint associated with the first vPLC is the destination of the packet includes determining the destination address included in the header of the packet, wherein the destination address is associated with the endpoint, The method according to claim 1, wherein determining that the endpoint associated with the first vPLC is the destination of the packet further includes determining that the destination address falls within the address range allocated to the first vPLC.

6. The further includes storing first information for a set of vPLCs created using the CSP-providing infrastructure within the region, The set of vPLCs includes the first vPLC, and the first information includes, for each vPLC in the set of vPLCs, information identifying the address range allocated to the vPLC. Determining that the destination address falls within the address range allocated to the first vPLC means that Using the first information, identify a specific address range within which the destination address is located, The method according to claim 5, further comprising determining, using the first information, that the specific address range is allocated to the first vPLC.

7. The first component further includes determining that the endpoint is associated with the first customer of the first reseller, The method according to claim 1, wherein the first vPLC-related information includes an identifier that identifies the first customer.

8. Determining that the endpoint is associated with the first customer of the first reseller includes determining the destination address included in the header of the packet. The destination address is associated with the endpoint, The method according to claim 7, further comprising determining that the endpoint is associated with the first customer of the first reseller, determining that the destination address falls within an address portion range allocated to the first customer of the first reseller.

9. The further includes storing first information for a set of vPLCs created using the CSP-providing infrastructure within the region, The set of vPLCs includes the first vPLC, the first information includes, for each vPLC in the set of vPLCs, information identifying the address range allocated to the vPLC, and the first information further includes, for each of one or more customers of the first reseller, a sub-address range allocated to the customer from the address range allocated to the first vPLC. Determining that the destination address falls within the address range allocated to the first customer means Using the first information, identify a specific address subrange within the destination address, The method according to claim 8, further comprising determining, using the first information, that the particular address subrange has been allocated to the first customer.

10. The endpoint is a destination compute instance within the first vPLC, The first component is a gateway within the CSP provision infrastructure in the region, The method according to claim 1, wherein the second component is a network virtualization device (NVD) that implements a virtual network interface controller (VNIC) associated with the destination calculation instance.

11. The method according to claim 10, wherein the gateway receives the packets from a source outside the CSP provision infrastructure within the region.

12. The packet originates from a source compute instance within the CSP provision infrastructure in the region, The endpoint is a destination compute instance associated with the first vPLC, The first component is an NVD that implements a VNIC associated with the source computation instance, The method according to claim 1, wherein the second component is an NVD implementing a VNIC associated with the destination calculation instance.

13. The method according to claim 12, wherein the source compute instance is associated with a second vPLC created for a second reseller using one or more resources from the cloud service provider (CSP) provided infrastructure within the region, and the second vPLC is created to provide one or more customers of the second resellers a set of cloud services supplied by the one or more second resellers.

14. The method according to claim 12, wherein the source computation instance is associated with the first vPLC.

15. The method according to claim 14, wherein the source calculation instance is associated with a first customer of the first reseller, and the destination calculation instance is associated with a second customer of the first reseller, the second customer being different from the first customer.

16. The method according to claim 14, wherein the source calculation instance is associated with the first customer of the first reseller, and the destination calculation instance is associated with the first customer of the first reseller.

17. The method according to claim 1, wherein the first component is dedicated to handling traffic to the first vPLC.

18. The method according to claim 1, wherein the first vPLC-related information includes an identifier that identifies a virtual cloud network (VCN) associated with the first vPLC, and the endpoint is part of the VCN.

19. One or more processors, A system comprising: a memory storing a computer-readable program for causing one or more processors to perform the method according to any one of claims 1 to 18.

20. A computer-readable program that causes one or more processors to execute the method according to any one of claims 1 to 18.

Images