Virtual Private Label Cloud Remote Data Plane

Abstract

Novel techniques are disclosed for accessing resources in both a CSP-provided infrastructure in a region and a remote infrastructure through various control planes associated with a virtual private label cloud (vPLC). In some embodiments, the CSP-provided infrastructure in a region and the remote infrastructure are connected through a communication channel. In some embodiments, a control plane associated with the CSP-provided infrastructure in a region may provide access to both infrastructures (i.e., the CSP-provided infrastructure in a region and the remote infrastructure). In some embodiments, a control plane associated with a vPLC in the CSP-provided infrastructure in a region may provide access to both infrastructures. Furthermore, in other embodiments, a control plane associated with the vPLC but located in the remote infrastructure may provide access to both infrastructures.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a formal application under 35 U.S.C. §119(e) of and claims the benefit of and priority to U.S. Provisional Patent Application No. 63 / 407,571, "VIRTUAL PRIVATE LABEL CLOUD," filed September 16, 2022, the entire contents of which are incorporated herein by reference.

[0002] This application is also related to the following applications, the entire contents of each of which are incorporated herein by reference:

[0003] (1) U.S. Patent Application No. 18 / 368,881, "VIRTUAL PRIVATE LABEL CLOUDS," filed concurrently with this application (Attorney Docket No. 088325-1311130 (344800US)). (2) U.S. Patent Application No. 18 / 368,863, "CONSOLE CUSTOMIZATION FOR VIRTUAL PRIVATE LABEL CLOUDS," filed concurrently with this application (Attorney Docket No. 088325-1311132 (344900US)). (3) U.S. Patent Application No. 18 / 368,870, "RESOURCE ALLOCATION FOR VIRTUAL PRIVATE LABEL CLOUDS," filed concurrently with this application (Attorney Docket No. 088325-1311140 (345000US)). (4) U.S. Patent Application No. 18 / 368,877, "ENDPOINTS FOR VIRTUAL PRIVATE LABEL CLOUDS," filed concurrently with this application (Attorney Docket No. 088325-1311138 (345100US)). (5) U.S. Patent Application No. 18 / 368,884, "IDENTITY MANAGEMENT FOR VIRTUAL PRIVATE LABEL CLOUDS," filed concurrently with this application (Attorney Docket No. 088325-1311139 (345200US)). (6) U.S. Patent Application No. 18 / 468,024, "METADATA CUSTOMIZATION FOR VIRTUAL PRIVATE LABEL CLOUDS," filed concurrently with this application (Attorney Docket No. 088325-1311136 (345300US)). (7) U.S. Patent Application No. 18 / 468,238, "CONNECTIVITY FOR VIRTUAL PRIVATE LABEL CLOUDS," filed concurrently with this application (Attorney Docket No. 088325-1311142 (345500US)). (8) U.S. Patent Application No. 18 / 468,047, "RESOURCE USAGE MONITORING, BILLING AND ENFORCEMENT FOR VIRTUAL PRIVATE LABEL CLOUDS," filed concurrently with this application (Attorney Docket No. 088325-1315390 (346900US)). (9) U.S. Patent Application No. 18 / 468,044, "CLOUD INFRASTRUCTURE-BASED ONLINE PUBLISHING PLATFORMS FOR VIRTUAL PRIVATE LABEL CLOUDS," filed concurrently with this application (Attorney Docket No. 088325-1315384 (346800US)). Field The present disclosure generally relates to techniques for providing cloud services. More particularly, novel techniques are disclosed for accessing resources at both a CSP-provided infrastructure in a region and at remote sites through various control planes associated with a virtual private label cloud (vPLC). A vPLC can be created for a cloud service provider (CSP) reseller that uses the CSP-provided infrastructure in a region and used to offer one or more reseller-supplied cloud services to the reseller's customers. [Background technology]

[0004] background Due to its ubiquitous presence and ease of access to critical data, cloud computing has gradually become a part of modern life and plays a vital role in our daily activities. However, just like in the early days of telecommunications, only a few companies are in the cloud infrastructure business. Therefore, there is a need to make cloud infrastructure services more widely available. Summary of the Invention

[0005] overview The present disclosure generally relates to techniques for providing cloud services. More specifically, a novel technique is disclosed for accessing resources at both a CSP-provided infrastructure in a region and at remote sites through various control planes associated with a virtual private label cloud (vPLC). A vPLC can be created for a cloud service provider (CSP) reseller that uses the CSP-provided infrastructure in a region and used to provide one or more reseller-supplied cloud services to the reseller's customers. Various embodiments are described herein, including methods, systems, non-transitory computer-readable storage media that store programs, code, or instructions that can be executed by one or more processors, and the like.

[0006] A vPLC generated in accordance with the techniques described in this disclosure can be used for a variety of purposes. For example, in one use case, a vPLC can be generated for a reseller who is a customer of a CSP and used by the reseller to provide one or more reseller-supplied cloud services to the reseller's customers. In another use case, a vPLC can be used as a virtual data center and associated with a realm different from the realm associated with the cloud infrastructure in a region provided by the CSP.

[0007] In one embodiment, a vPLC represents a virtual cloud that includes a set of resources allocated to the vPLC from a CSP-provided infrastructure in a region. One or more vPLCs can be created using the CSP-provided infrastructure in a region.

[0008] In one embodiment, a CSP provides regional infrastructure in a region to enable both the provision of cloud services to its customers and the provision of reseller-supplied cloud services to its customers. This is achieved by creating one or more vPLCs for one or more resellers from the CSP-provided regional infrastructure, and the vPLCs created for a particular reseller can be used to provide reseller-supplied and reseller-branded cloud services to a particular reseller's customers. This allows the reseller to become a provider of reseller-branded cloud services to its customers without having to invest in the infrastructure required to provide those services. The reseller, in turn, provides the reseller-branded cloud services to the reseller's customers using the CSP-provided infrastructure. In this way, a CSP-supplied cloud service can be provided to customers of the CSP using a first portion of the infrastructure that the CSP provides in a region, a first reseller-supplied cloud service can be provided to customers of the first reseller using a second portion of the infrastructure that the CSP provides in the region and that is assigned to a first vPLC created for a first reseller, a second reseller-supplied cloud service can be provided to customers of the second reseller using a third portion of the infrastructure that the CSP provides in the region and that is assigned to a second vPLC created for a second reseller, and so on.

[0009] In one embodiment, a technology is provided that includes a method, the method including: providing one or more CSP-supplied cloud services to one or more customers of a cloud service provider (CSP) by using a first portion of a CSP-provided infrastructure in a first region; and generating a first virtual private label cloud (vPLC) for a first reseller based on the CSP-provided infrastructure, where generating the first vPLC includes allocating a second portion of the CSP-provided infrastructure to the first vPLC, the method further including: providing a first subset of the first reseller-supplied cloud services to one or more customers of the first reseller by using the first vPLC; connecting the first vPLC to a remote infrastructure outside the CSP-provided infrastructure in the first region through a communication channel; and providing a second subset of the first reseller-supplied cloud services to one or more customers of the first reseller by using the remote infrastructure.

[0010] In yet another embodiment, the method further includes providing access to a second subset of the first reseller-supplied cloud services by using a first control plane associated with and located within a first portion of the CSP-provided infrastructure.

[0011] In yet another embodiment, providing access to the second subset of the first reseller-supplied cloud services uses a communication channel.

[0012] In yet another embodiment, the method further includes providing access to a second subset of the first reseller-supplied cloud services by using a second control plane associated with and located within the first vPLC.

[0013] In yet another embodiment, the method further includes providing access to a first subset of the first reseller-supplied cloud services by using a third control plane associated with the first vPLC and located in the remote infrastructure.

[0014] In yet another embodiment, the communication channel is between a virtual cloud network (VCN) in the first vPLC and a VCN in the remote infrastructure.

[0015] In yet another embodiment, packets used for communication between the first vPLC and the remote infrastructure over the communication channel are tagged with vPLC-related information.

[0016] In yet another embodiment, the vPLC-related information includes information identifying the first vPLC.

[0017] In yet another embodiment, the underlying network of the CSP-provided infrastructure and the underlying network of the remote infrastructure are incompatible with each other.

[0018] In yet another embodiment, the method further includes mapping between a VLAN identifier and vPLC-related information.

[0019] In various embodiments, a system is provided that includes one or more data processors and a non-transitory computer-readable medium that includes instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more of the methods disclosed herein.

[0020] In various embodiments, a non-transitory computer-readable medium is provided that stores computer-executable instructions that, when executed by one or more processors of a computer system, cause the one or more processors to perform one or more methods disclosed herein.

[0021] In various embodiments, a computer program product is provided that includes computer programs / instructions that, when executed by a processor, cause the processor to perform any of the methods disclosed herein.

[0022] The techniques described above and below can be implemented in many ways and in many contexts. Several exemplary implementations and contexts are provided by reference to the drawings, as described in more detail below. However, the following implementations and contexts are only a few of many. [Brief explanation of the drawings]

[0023] [Figure 1] FIG. 1 is a high-level diagram of a distributed environment illustrating a virtual or overlay cloud network hosted by a cloud service provider infrastructure, according to one embodiment. [Figure 2] FIG. 2 is a simplified architectural diagram of physical components in a physical network within CSPI, according to one embodiment. [Figure 3] FIG. 1 illustrates an exemplary configuration within CSPI in which a host machine is connected to multiple network virtualization devices (NVDs), according to an embodiment. [Figure 4] FIG. 1 illustrates a connection between a host machine and an NVD to provide I / O virtualization that supports multi-tenancy, according to one embodiment. [Figure 5] FIG. 1 is a simplified block diagram of a physical network provided by CSPI, according to one embodiment. [Figure 6] 1 is a block diagram of a distributed environment illustrating an example of a virtual private label cloud (vPLC) hosted by a CSP-provided infrastructure in a region, according to one embodiment. [Figure 7] FIG. 1 illustrates relationships between a CSP, its direct customers, reseller customers and their associated users, and individual reseller customers and their associated users, according to one embodiment. [Figure 8]FIG. 1 illustrates the relationship between resources allocated to the tenancies of a CSP's direct customers, the tenancies of the CSP's resellers, and the tenancies of each of the resellers' customers, according to an embodiment. [Figure 9A] FIG. 1 illustrates a mechanism for storing information in the tenancies of customers of individual resellers of a CSP, according to an embodiment. [Figure 9B] FIG. 1 illustrates a mechanism for storing information in the tenancies of customers of individual resellers of a CSP, according to an embodiment. [Figure 10] 1 is a flowchart illustrating an example of a vPLC configuration process for a CSP reseller, according to an embodiment. [Figure 11] 1 is a block diagram of a distributed environment illustrating an example of a virtual private label cloud (vPLC) hosted by a CSP-provided infrastructure in a region, according to one embodiment. [Figure 12] FIG. 1 is a block diagram illustrating a vPLC segmentation model for resource partitioning, according to an embodiment. [Figure 13] FIG. 1 is a block diagram illustrating a vPLC nesting model for resource partitioning, according to an embodiment. [Figure 14] FIG. 10 is a flow diagram illustrating a sign-up process for a customer of a reseller associated with a vPLC, according to one embodiment. [Figure 15] FIG. 10 is a flow diagram illustrating a login process for a user associated with a customer of a reseller associated with a vPLC, according to an embodiment. [Figure 16] FIG. 1 is a flow diagram illustrating a resource allocation and provisioning process for a reseller's customer associated with a vPLC, according to an embodiment. [Figure 17] FIG. 1 is a flow diagram illustrating a resource allocation and provisioning process for a reseller's customer associated with a vPLC, according to an embodiment. [Figure 18]FIG. 1 is a simplified diagram illustrating a use case in which a vPLC is hosted by the infrastructure of a first realm while associated with a second, different realm, according to one embodiment. [Figure 19] 1 is a flowchart illustrating an exemplary method for creating a vPLC hosted by a CSP-provided infrastructure in a particular region in a particular realm, while virtually associated with a different region in a different realm, according to an embodiment. [Figure 20] 1 is a block diagram illustrating a CSP control plane model according to an embodiment. [Figure 21] FIG. 1 is a block diagram illustrating a local vPLC-CP model according to an embodiment. [Figure 22] FIG. 1 is a block diagram illustrating a remote vPLC-CP model according to an embodiment. [Figure 23] 1 is an exemplary flowchart illustrating a process for creating a communication channel between a CSP-provided infrastructure in a region and a remote site, according to an embodiment. [Figure 24] 1 is an exemplary flowchart illustrating a method for processing a request in a CSP-CP model, according to an embodiment. [Figure 25] 1 is an exemplary flowchart illustrating a method for processing a request in a local vPLC-CP model according to an embodiment. [Figure 26] 1 is an exemplary flowchart illustrating a method for processing a request in a remote vPLC-CP model according to an embodiment. [Figure 27] FIG. 1 is a block diagram illustrating a pattern for implementing a service-based cloud infrastructure system, according to at least one embodiment. [Figure 28] FIG. 1 is a block diagram illustrating another pattern for implementing a service-based cloud infrastructure system, according to at least one embodiment. [Figure 29]FIG. 1 is a block diagram illustrating another pattern for implementing a service-based cloud infrastructure system, according to at least one embodiment. [Figure 30] FIG. 1 is a block diagram illustrating another pattern for implementing a service-based cloud infrastructure system, according to at least one embodiment. [Figure 31] FIG. 1 is a block diagram illustrating an exemplary computer system according to at least one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0024] Detailed Description In the following description, for purposes of explanation, specific details are set forth in order to provide a thorough understanding of certain embodiments. However, it will be apparent that various embodiments may be practiced without these specific details. The drawings and this description are not intended to be limiting. Use of the word "exemplary" herein means "serving as an example, instance, or illustration." Any embodiment or design described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments or designs.

[0025]

[0001] The present disclosure generally relates to techniques for providing cloud services. More particularly, novel techniques are disclosed for providing cloud service provider (CSP)-supplied cloud services to CSP customers in a region provided by the CSP, enabling cloud infrastructure used to create one or more virtual private clouds (also referred to herein as virtual private label clouds or "vPLCs"). Various embodiments are described herein, including methods, systems, non-transitory computer-readable storage media storing programs, code, or instructions executable by one or more processors, and the like.

[0026] A vPLC generated in accordance with the techniques described in this disclosure can be used for a variety of purposes. For example, in one use case, a vPLC can be generated for a reseller who is a customer of a CSP and used by the reseller to provide one or more reseller-supplied cloud services to the reseller's customers. In another use case, a vPLC can be used as a virtual data center and associated with a realm different from the realm associated with the cloud infrastructure in a region provided by the CSP.

[0027] In one embodiment, a vPLC represents a virtual cloud that includes a set of resources allocated to the vPLC from a CSP-provided infrastructure in a region. One or more vPLCs can be created using the CSP-provided infrastructure in a region. In some embodiments, a virtual private label cloud (vPLC) represents a model and architecture that enables and facilitates CSP-provided, reseller-supplied, and reseller-branded cloud services within the CSP-provided infrastructure in a region.

[0028] Just as a CSP-provided infrastructure in a region provides cloud resources that can be accessed by the CSP's customers, a vPLC created for a reseller provides a set of resources that can be accessed by the reseller's customers. From the perspective of the reseller's customers, a vPLC is like a reseller-provided infrastructure in a region, providing resources that the reseller's customers can access. A vPLC is like a data center that delivers reseller-provided cloud services to the reseller's customers in the reseller-provided region.

[0029] In one embodiment, a CSP provides regional infrastructure in a region to enable both the provision of cloud services to its own customers and the provision of reseller-supplied cloud services to its customers. This is achieved by creating one or more vPLCs for one or more resellers from the CSP-provided regional infrastructure, and the vPLCs created for a particular reseller can be used to provide reseller-supplied and reseller-branded cloud services to a particular reseller's customers. This allows the reseller to become a provider of reseller-branded cloud services to its customers without having to invest in the infrastructure required to provide those services. Instead, the reseller uses the infrastructure provided by the CSP to provide reseller-branded cloud services to its customers.

[0030] In a reseller use case, a vPLC is created for the reseller by using cloud infrastructure in a region provided by a cloud service provider (CSP). The vPLC created for the reseller can be used by the reseller to provide one or more reseller-supplied cloud services to the reseller's customers. In one embodiment, the CSP-provided regional infrastructure providing the services used by the reseller to provide one or more reseller-supplied services is partitioned into securely isolated portions, and these isolated portions (or partitions) are assigned to different resellers, such that the partitions assigned to one reseller are isolated from other resellers. In this manner, a CSP-supplied cloud service can be provided to customers of the CSP using a first portion of the infrastructure provided by the CSP in a region, a first reseller-supplied cloud service can be provided to customers of the first reseller using a second portion of the infrastructure provided by the CSP in the region and assigned to a first vPLC created for a first reseller, a second reseller-supplied cloud service can be provided to customers of the second reseller using a third portion of the infrastructure provided by the CSP in the region and assigned to a second vPLC created for a second reseller, etc. Each portion of the infrastructure assigned to a reseller can also be further divided into second-level, securely isolated portions to provide multi-tenant cloud infrastructure services to customers of the resellers.

[0031] A reseller can be an entity such as a company, a business, or an individual. By using the technology described herein, a reseller entity can quickly become a cloud service provider while avoiding all of the barriers to entry discussed in the background section of this disclosure. For example, a reseller does not need to acquire the infrastructure used to provide cloud services because the provided cloud services use infrastructure provided by a CSP. To the reseller's customers, the reseller appears to be providing reseller-supplied cloud services. Thus, from the perspective of the reseller's customers, the reseller is the cloud service provider whose services the customer subscribes to. The reseller's customers may not even know or recognize the CSP whose infrastructure the reseller uses to provide reseller-supplied and reseller-branded cloud services.

[0032] The ability to provide a vPLC is offered as a service by a CSP, and one or more customers can sign up. For example, an entity that wants to provide cloud services to its customers but does not want to invest in the infrastructure required to provide the services can sign up for this CSP-supplied vPLC service. When an entity registers for a vPLC service, it becomes a customer of the CSP, but it is a "special" customer because the CSP-supplied infrastructure allocated to the entity in the form of a vPLC is used to provide the entity-supplied and branded services to the entity's customers. Thus, the customer is both a customer of the CSP and a reseller of cloud services that use the vPLC generated for the entity. For this reason, the entity is referred to as a "reseller" to distinguish it from the CSP's actual direct customers. A CSP's direct customers are entities that sign up for and consume one or more CSP-supplied and branded services, but do not use the CSP infrastructure to sell cloud services to their customers. For purposes of this disclosure, a CSP's direct customers are also referred to as "non-reseller customers" to distinguish them from resellers. With respect to the reseller, as part of the vPLC service, the CSP will provide infrastructure to the reseller in the form of vPLC, which the reseller will use to provide the reseller-branded cloud services to the reseller's customers.

[0033] A vPLC can also be used for a variety of other purposes that do not involve a reseller. As one example, a vPLC can be created and associated with a realm that is different from the realm associated with the CSP-provided regional infrastructure used to create the vPLC. For example, the CSP-provided regional infrastructure may be associated with a first realm, and the vPLC may be associated with a second realm that is different from the first realm. In such a scenario, the infrastructure corresponding to the vPLC is hosted by the first realm but is considered to belong to the second realm. This can be used for a variety of different purposes. For example, because the vPLC is associated with the second realm, it can communicate with other infrastructure in the second realm (e.g., other data centers). Thus, data centers in the second realm can communicate with the vPLC because they reside in the same realm and share the trust and identity profile of the second realm.

[0034] The creation and management of a vPLC is performed using infrastructure and services provided by a CSP, a highly complex technical task. When a vPLC is created for a reseller, the vPLC provides technical functionality that enables the reseller to offer reseller-supplied cloud services to the reseller's customers using the vPLC, including traffic isolation between resellers, traffic isolation between the reseller's different customers, dynamic management of CSP-provided resource allocation to vPLCs, management of resource allocation allocated to vPLCs among the vPLC's different customers, metering usage of vPLCs allocated to the reseller and performing related billing functions, metering usage of vPLCs allocated among the vPLC's different customers and performing related billing functions, enabling a marketplace for different resellers, and performing identity management functions for resellers and their respective customers. This disclosure describes various embodiments illustrating various architectures and corresponding methods for implementing and enabling vPLC-related functions.

[0035] A reseller may use the vPLC created for the reseller to provide reseller-supplied and reseller-branded cloud services that provide specialized services in the domain in which the reseller specializes. The reseller-supplied services may be tailored to various customer segments. In some embodiments, the reseller-supplied cloud services may be different from the CSP-supplied cloud services. In some other embodiments, the reseller-supplied cloud services may be based on the CSP-supplied cloud services. For example, the reseller-supplied services may be customized versions of the CSP-supplied cloud services.

[0036] For example, a first vPLC may be created in a first region from the CSP-provided infrastructure for a first reseller, the first vPLC being specialized for providing financial services. The first reseller's customers may be financial institutions, such as banks. The first reseller may then use the first vPLC to provide customized, first-reseller-branded financial cloud services to their customers. A second vPLC may be created in the first region from the same CSP-provided infrastructure for a second reseller, the second reseller's customers may be users of telecommunication services. The second reseller may then use the second vPLC to provide customized, second-reseller-branded telecommunication cloud services to their customers. In addition to the first and second vPLCs, the CSP may use the CSP-provided infrastructure in the first region to provide CSP-supplied and branded cloud services to one or more of the CSP's direct (non-reseller) customers. In this manner, use of the same CSP provisioning infrastructure in a region provides CSP-supplied and trademarked cloud services to one or more direct (non-reseller) customers of the CSP, provides customized first reseller-branded financial cloud services to customers of the first reseller, and provides customized second reseller-branded telecommunications cloud services to customers of the second reseller.

[0037] The present disclosure generally relates to techniques for providing cloud services. More particularly, novel techniques are disclosed for accessing resources at both a CSP-provided infrastructure in a region and at remote sites through various control planes associated with a virtual private label cloud (vPLC). A vPLC is created for a cloud service provider (CSP) reseller that uses the CSP-provided infrastructure in a region, enabling the reseller to offer one or more reseller-supplied cloud services to the reseller's customers.

[0038] In a CSP-provided infrastructure in a region, the CSP's control plane (CP) may reside in the same region or cluster as the data plane (DP) hosting resources (e.g., vPLC). In other words, the CSP's CP (or CSP CP) manages the resources owned by the CSP.

[0039] A CSP's reseller may have its own compute resources (e.g., on its premises or in a different cloud infrastructure) and want to integrate these resources with a CSP-provided regional cloud infrastructure (or resources) to provide reseller-supplied and rebranded services to its customers. However, the current CSP's CP cannot interact with the reseller's resources remotely because the CSP's CP and the remote resources reside in different environments. In this specification, such infrastructure or premises not owned by the CSP may be referred to as a remote site (or remote infrastructure or remote premises), which may include the reseller's premises (referred to as a remote customer site), such as a remote data plane.

[0040] However, issues remain regarding how a reseller may place data plane resources at a remote site, while the data plane resources are neither owned nor trusted by the CSP. The control plane that manages the compute devices may not be able to communicate with the devices it manages (i.e., the data plane devices) if they reside on a different network. Typically, the CSP CP and the managed compute devices reside on the same physical network (i.e., the infrastructure network, or infrastructure for short), and the CP and the compute devices trust each other. However, the remote data plane may not be trusted by the CSP. Therefore, concerns arise about directly connecting the CSP CP with devices in untrusted remote premises. Therefore, a different approach is needed to address these challenges and others.

[0041] The novel technology described in this disclosure creates an architecture (referred to as a remote data plane architecture) that allows a reseller to integrate its remote resources with a CSP-provided regional infrastructure so that the reseller's customers can manage and access both the reseller's resources at the remote site and the resources in the CSP-provided regional infrastructure, and both resources can be used to provide reseller-provisioned and reseller-rebranded services. For example, the remote site can provide a subset of reseller-provisioned and reseller-rebranded services, and a vPLC created by the CSP for the reseller can provide another subset of reseller-provisioned and reseller-rebranded services. Reseller-provisioned and reseller-rebranded services can include SaaS, PaaS, IaaS, etc.

[0042] In some embodiments, such technology allows for management of remote site infrastructure through the use of a vPLC-specific CP operating in a CSP-provided regional infrastructure virtual cloud network (VCN), which supports vPLC over a communications channel (e.g., a FastConnect or VPN connection). Because such a vPLC-specific CP operates within the VCP and not within the CSP's infrastructure, the CSP's infrastructure does not need to trust or have a direct connection to the remote data plane. In other words, the vPLC-specific CP and the remote sites it manages reside in an environment that is not part of the CSP's infrastructure.

[0043] In some embodiments, packet modification / rewriting by gateways in the CSP-provided regional infrastructure and gateways at the remote site can avoid direct connections between the CSP's infrastructure and the remote data plane.

[0044] The techniques described in this disclosure may provide three models for enabling management and access of reseller's remote resources and resources within CSP-provided regional infrastructure: (1) a CSP control plane (or CSP-CP) model (also known as the "relay model"), (2) a local vPLC-specific control plane (or vPLC-CP) model, and (3) a remote vPLC-CP model.

[0045] In the CSP-CP model, all vPLCs may use a CSP CP to manage and access both local and remote resources, whereas in the local vPLC-CP model, each vPLC may use its own vPLC-specific control plane to manage and access both local and remote resources. The term "local" in the local vPLC-CP model may refer to a vPLC-specific CP that is within (or local to) the CSP-provided regional infrastructure.

[0046] A third model, the remote vPLC-CP model, distinguishes from the local vPLC-CP model described above (i.e., the vPLC-specific CP is part of the CSP-provided regional infrastructure) and may have a vPLC control plane operating inside a remote site (i.e., outside the CSP-provided regional infrastructure). In this remote vPLC-CP model, a reseller may use its remote vPLC CP to control and access the DP of a local vPLC in the CSP-provided regional infrastructure, as well as to directly control and access the DP of a remote vPLC at the remote site (e.g., the reseller's premises).

[0047] The techniques described in this disclosure allow resellers to define the experience of their customers' users by using various control planes, providing customers with greater flexibility in accessing different cloud resources for the same user's workloads. As a result, these techniques can minimize vendor lock-in and provide a wider range of reseller-supplied and rebranded services to the reseller's customers.

[0048] Figures 1-5 and the associated description provided in the "Exemplary Virtual Networking Architecture" section below describe networking concepts including network virtualization, foundation networks, overlay networks, VNICs, etc., and provide example environments in which certain embodiments described herein may be implemented. Figures 6-26 describe examples and embodiments related to the vPLC architecture and vPLC remote data plane architecture described herein. Figures 27-30 illustrate example architectures for implementing a cloud infrastructure that provides one or more cloud services and may incorporate the teachings described herein. Figure 31 is a block diagram illustrating an example computer system or device according to at least one embodiment.

[0049] Exemplary Virtual Networking Architecture The term cloud service is generally used to describe services made available on demand (e.g., through a subscription model) by a cloud service provider (CSP) to users or customers through the use of systems and infrastructure (cloud infrastructure) provided by the CSP. Typically, the servers and systems that make up the CSP's infrastructure are separate from the customer's own on-premise servers and systems. This allows customers to use cloud services provided by CSPs without having to purchase separate hardware and software resources for the service. Cloud services are designed to provide subscribing customers with easy and scalable access to applications and computing resources without requiring the customer to invest in procuring the infrastructure used to deliver the service.

[0050] There are multiple cloud service providers offering different types of cloud services, which come in a variety of different types or models, such as Software as a Service (SaaS), Platform as a Service (PaaS), and Infrastructure as a Service (IaaS).

[0051] A customer may sign up for one or more cloud services offered by a CSP. A customer can be any entity, such as an individual, an organization, or a business. When a customer signs up or subscribes to a service offered by a CSP, a tenancy or account is created for the customer. This account allows the customer to access one or more registered cloud resources associated with the account.

[0052] As mentioned above, Infrastructure as a Service (IaaS) is a specific type of cloud computing service. In the IaaS model, a CSP provides infrastructure (called Cloud Service Provider Infrastructure, or CSPI) that customers can use to build their own customizable networks and deploy their resources. In this way, customer resources and networks are hosted in a distributed environment by the CSP-provided infrastructure. This differs from traditional computing, where customer resources and networks are hosted by customer-provided infrastructure.

[0053] CSPI may include interconnected high-performance compute resources, including various host machines, memory resources, and network resources, that make up a physical network, also referred to as an infrastructure network or base network. CSPI resources may be distributed across one or more data centers, which may be geographically distributed across one or more geographic regions. These physical resources may run virtualization software to provide a virtualized distributed environment. Virtualization creates an overlay network (also known as a software-based network, software-defined network, or virtual network) on the physical network. The CSPI physical network serves as the basis for creating one or more overlay or virtual networks on the physical network. The physical network (or infrastructure network or base network) comprises physical network devices such as physical switches, routers, computers, and host machines. An overlay network is a logical (or virtual) network that operates on the physical infrastructure network. A given physical network may support one or more overlay networks. Overlay networks typically use encapsulation techniques to distinguish traffic belonging to different overlay networks. A virtual or overlay network is also referred to as a virtual cloud network (VCN). Virtual networks are realized through the use of software virtualization technologies (e.g., hypervisors, virtualization functions realized by network virtualization devices (NVDs) (e.g., smart NICs), top-of-rack (TOR) switches, smart TORs that realize one or more functions performed by NVDs, and other mechanisms) to create a network abstraction layer that can operate over a physical network. Virtual networks can take many forms, including peer-to-peer networks, IP networks, etc. Virtual networks are typically Layer 3 IP networks or Layer 2 VLANs. This method of virtual or overlay networking is often referred to as virtual or overlay Layer 3 networking.Examples of protocols developed for virtual networks include IP-in-IP (or Generic Routing Encapsulation (GRE)), VXLAN-IETF RFC 7348 (Virtual Extensible LAN), VPN (Virtual Private Network) (e.g., MPLS Layer-3 Virtual Private Networks (RFC 4364)), VMware's NSX, and GENEVE (Generic Network Virtualization Encapsulation).

[0054] In IaaS, a CSP-provided infrastructure (CSPI) can be configured to provide virtualized computing resources over a public network (e.g., the Internet). In the IaaS model, a cloud computing service provider can host infrastructure components (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer), etc.). In some cases, the IaaS provider may also provide various services (e.g., billing, monitoring, logging, security, load balancing, clustering, etc.) associated with these infrastructure components. These services can be policy-driven, enabling IaaS users to maintain application availability and performance by implementing policies that drive load balancing. CSPI provides infrastructure and a set of complementary cloud services that enable customers to build and run a wide range of applications and services in a hosted, highly available, distributed environment. CSPI delivers high-performance compute resources and capabilities, as well as storage capacity, in a flexible virtual network that can be securely accessed from various network locations, such as the customer's on-premises network. When a customer registers or subscribes to an IaaS service offered by a CSP, a tenancy is created for that customer, which is an isolated and secure partition within CSP where the customer may create, organize, and manage their cloud resources.

[0055] Customers can build their own virtual networks using compute, memory, and network resources provided by CSPI. One or more customer resources or workloads, such as compute instances, can be deployed onto these virtual networks. For example, customers can build one or more customizable private virtual networks (referred to as virtual cloud networks (VCNs)) using resources provided by CSPI. Customers can deploy one or more customer resources, such as compute instances, into the customer VCN. Compute instances can take the form of virtual machines, bare metal instances, etc. In this way, CSPI provides infrastructure and a set of complementary cloud services that enable customers to build and run a wide range of applications and services in a hosted, highly available virtual environment. While customers do not manage or control the underlying physical resources provided by CSPI, they do control the operating system, storage, and deployed applications. Customers may have limited control over some networking components (e.g., firewalls).

[0056] The CSP may provide a console that enables customers and network administrators to use CSPI resources to configure, access, and manage resources deployed in the cloud. In some embodiments, the console provides a web-based user interface that can be used to access and manage the CSPI. In some implementations, the console is a web-based application provided by the CSP.

[0057] CSPI may support single-tenancy or multi-tenancy architectures. In a single-tenancy architecture, software (e.g., applications, databases) or hardware components (e.g., host machines or servers) serve a single customer or tenant. In a multi-tenancy architecture, software or hardware components serve multiple customers or tenants. Thus, in a multi-tenancy architecture, CSPI resources are shared among multiple customers or tenants. In a multi-tenancy situation, precautions and safeguards are taken within CSPI to ensure that each tenant's data is isolated and not visible to other tenants.

[0058] In a physical network, a network endpoint ("endpoint") represents a computing device or system that is connected to the physical network and communicates with the connected network. A network endpoint of a physical network may be connected to a local area network (LAN), a wide area network (WAN), or other types of physical networks. Examples of traditional endpoints in a physical network include modems, hubs, bridges, switches, routers, and other networking devices, physical computers (or host machines), etc. Each physical device in a physical network has a fixed network address that can be used to communicate with the device. This fixed network address can be a Layer 2 address (e.g., a MAC address), a fixed Layer 3 address (e.g., an IP address), etc. In a virtualized environment or virtual network, endpoints can include various virtual endpoints, such as virtual machines hosted by components of the physical network (e.g., hosted by a physical host machine). These virtual network endpoints are addressed by overlay addresses, such as overlay Layer 2 addresses (e.g., an overlay MAC address) and overlay Layer 3 addresses (e.g., an overlay IP address). Network overlays provide flexibility by allowing network administrators to move overlay addresses associated with network endpoints using software management (e.g., via software implementing the virtual network's control plane). Thus, unlike physical networks, in virtual networks, overlay addresses (e.g., overlay IP addresses) may be moved from one endpoint to another through the use of network management software. Because virtual networks are built on top of physical networks, communication between components of a virtual network involves both the virtual network and the underlying physical network.To facilitate such communications, CSPI components are configured to learn and store mappings from overlay addresses in the virtual network to actual physical addresses in the underlying network, and vice versa, and use these mappings to facilitate communications and encapsulate customer traffic to facilitate routing in the virtual network.

[0059] Thus, physical addresses (e.g., physical IP addresses) are associated with elements of a physical network, and overlay addresses (e.g., overlay IP addresses) are associated with entities of a virtual or overlay network. A physical IP address is an IP address associated with a physical device (e.g., a network device) of an underlying or physical network. For example, each NVD has an associated physical IP address. An overlay IP address is an overlay address associated with an entity of an overlay network, such as a compute instance in a customer's virtual cloud network (VCN). Two different customers or tenants, each with their own private VCN, can potentially use the same overlay IP address in their respective VCNs without any knowledge of each other. Both physical IP addresses and overlay IP addresses are types of real IP addresses. They are separate from virtual IP addresses. A virtual IP address is typically a single IP address that represents or maps to multiple real IP addresses. A virtual IP address provides a one-to-many mapping between the virtual IP address and multiple real IP addresses. For example, a load balancer may use a VIP to map to or represent multiple servers, each with its own real IP address.

[0060] Cloud infrastructure or CSPI is physically hosted in one or more data centers in one or more regions around the world. CSPI may include physical or underlying network components as well as virtualized components (e.g., virtual networks, compute instances, virtual machines, etc.) in a virtual network built on the physical network components. In one embodiment, CSPI is organized and hosted in realms, regions, and availability domains. A region is typically a local geographic area that includes one or more data centers. Regions are generally independent of each other and may be separated by large distances, for example, across countries or continents. For example, one region may be in Australia, another in Japan, and yet another in India. CSPI resources are divided among the regions so that each region has its own independent subset of CSPI resources. Each region may provide a set of core infrastructure services and resources, such as compute resources (e.g., bare metal servers, virtual machines, containers, and related infrastructure), storage resources (e.g., block volume storage, file storage, object storage, archive storage), networking resources (e.g., virtual cloud networks (VCNs), load balancing resources, connectivity to on-premises networks), database resources, edge networking resources (e.g., DNS), and access management and monitoring resources. Each region typically has multiple paths connecting it to other regions in the realm.

[0061] Typically, applications are deployed to the region where they are most heavily used (i.e., deployed to the infrastructure associated with that region) because using nearby resources is faster than using resources that are farther away. Applications may also be deployed to different regions for a variety of reasons, such as redundancy to mitigate the risk of region-wide events such as large-scale weather systems or earthquakes, or to meet various requirements such as jurisdictions, tax areas, and other business or societal standards.

[0062] Data centers within a region can be further organized and subdivided into availability domains (ADs). An availability domain can correspond to one or more data centers located within a region. A region can be composed of one or more availability domains. In such a distributed environment, CSPI resources are region-specific, such as a virtual cloud network (VCN), or availability domain-specific, such as a compute instance.

[0063] ADs within a region are isolated from each other, fault-tolerant, and designed to minimize the likelihood of simultaneous failures. This is achieved by ADs not sharing critical infrastructure resources such as networks, physical cables, cable paths, or cable entry points. Therefore, a failure in one AD within a region is unlikely to affect the availability of other ADs in the same region. ADs within the same region can be connected to each other via low-latency, high-bandwidth networks, enabling highly available connections to other networks (e.g., the Internet, customer on-premises networks, etc.). This allows for the creation of replica systems in multiple ADs for both high availability and disaster recovery. Cloud services use multiple ADs to ensure high availability and protect against resource failures. As the infrastructure offered by an IaaS provider grows, capacity can be added to more regions and ADs. Traffic between availability domains is typically encrypted.

[0064] In one embodiment, regions are grouped into realms. A realm is a logical collection of regions. Realms are isolated from each other and do not share any data. Regions in the same realm can communicate with each other, but regions in different realms cannot. A customer's tenancy or account using a CSP exists in a single realm and can be distributed across one or more regions belonging to that realm. Typically, when a customer signs up for an IaaS service, a tenancy or account for that customer is created in a customer-specific region (called the "home" region) within the realm. The customer can extend their tenancy across one or more other regions within the realm. A customer cannot access regions that are not in the realm in which their tenancy resides.

[0065] An IaaS provider may offer multiple realms, each corresponding to a particular set of customers or users. For example, a commercial realm may be offered for commercial customers. As another example, a realm may be offered for customers in a particular country. As yet another example, a government realm may be offered for governments, and so on. For example, a government realm may cater to a particular government and provide a higher level of security than a commercial realm. For example, OCI (Oracle Cloud Infrastructure) currently offers one realm for its commercial regions and two realms for its government cloud regions (e.g., FedRAMP and IL5 certified).

[0066] In one embodiment, an AD can be subdivided into one or more fault domains. A fault domain is a group of infrastructure resources within an AD that provides anti-affinity. Fault domains can distribute compute instances so that they do not reside on the same physical hardware within a single AD. This is known as anti-affinity. A fault domain represents a set of hardware components (computers, switches, etc.) that share a single point of failure. A compute pool is logically divided into fault domains. This ensures that a hardware failure or compute hardware maintenance event affecting one fault domain does not affect instances in other fault domains. Depending on the embodiment, the number of fault domains per AD can vary. For example, in one embodiment, each AD contains three fault domains. Fault domains act as logical data centers within an AD.

[0067] When a customer subscribes to an IaaS service, resources from CSPI are provisioned for the customer and associated with the customer's tenancy. The customer can use these provisioned resources to build private networks and deploy resources on these networks. A customer network hosted in the cloud by CSPI is called a virtual cloud network (VCN). A customer can configure one or more virtual cloud networks (VCNs) using the allocated CSPI resources. A VCN is a virtual or software-defined private network. Customer resources deployed in a customer's VCN may include compute instances (e.g., virtual machines, bare metal instances) and other resources. These compute instances may represent various customer workloads, such as applications, load balancers, databases, etc. Compute instances deployed in a VCN can communicate with publicly accessible endpoints (“public endpoints”) over a public network such as the Internet, other instances in the same VCN or other VCNs (e.g., other VCNs of the customer or VCNs not belonging to the customer), the customer's on-premises data center or network, and service endpoints and other types of endpoints.

[0068] CSPs can use CSPI to provide various services. In some cases, customers of a CSPI themselves can act as service providers and provide services using CSPI resources. Service providers may expose service endpoints characterized by identifying information (e.g., IP addresses, DNS names, and ports). Customer resources (e.g., compute instances) can consume a particular service by accessing the service endpoints exposed by that service. These service endpoints are generally publicly accessible to users over a public communications network, such as the Internet, using a public IP address associated with the endpoint. Publicly accessible network endpoints are also sometimes referred to as public endpoints.

[0069] In some embodiments, a service provider may expose a service via a service endpoint (sometimes referred to as a service endpoint). Customers of the service may then access the service using the service endpoint. In some embodiments, a given service endpoint for a service may be accessible by multiple customers intending to consume the service. In other embodiments, a customer may be given a dedicated service endpoint such that only that customer may access the service using the dedicated service endpoint.

[0070] In one embodiment, when a VCN is created, it is associated with a private overlay Classless Inter-Domain Routing (CIDR) address space (e.g., 10.0 / 16), which are broad private overlay IP addresses assigned to the VCN. A VCN includes associated subnets, route tables, and gateways. A VCN exists within a single region but can span one, more, or all availability domains in the region. A gateway is a virtual interface configured for a VCN that enables traffic to communicate between the VCN and one or more endpoints outside the VCN. One or more different types of gateways may be configured for a VCN to enable communication between different types of endpoints.

[0071] A VCN may be subdivided into one or more subnetworks, such as one or more subnets. A subnet is thus a configuration unit or subdivision that may be created within a VCN. A VCN may have one or more subnets. Each subnet within a VCN is associated with a contiguous range of overlay IP addresses (e.g., 10.0.0.0 / 24 and 10.0.1.0 / 24) that do not overlap with other subnets in the VCN and represent a subset of address space within the VCN's address space.

[0072] Each compute instance is associated with a virtual network interface card (VNIC) that enables the compute instance to participate in a subnet of a VCN. A VNIC is a logical representation of a physical network interface card (NIC). Generally, a VNIC is an interface between an entity (e.g., a compute instance, a service) and a virtual network. A VNIC resides in a subnet and has one or more associated IP addresses and associated security rules or policies. A VNIC corresponds to a Layer 2 port on a switch. A VNIC is associated with a compute instance and a subnet within a VCN. A VNIC associated with a compute instance enables the compute instance to be part of a subnet of a VCN and enables the compute instance to communicate (e.g., send and receive packets) with endpoints on the same subnet as the compute instance, endpoints in a different subnet of the VCN, or endpoints outside the VCN. Thus, a VNIC associated with a compute instance determines how the compute instance connects to endpoints inside and outside the VCN. A VNIC for a compute instance is created and associated with the compute instance when the compute instance is created and added to a subnet in a VCN. For a subnet containing a set of compute instances, the subnet contains VNICs corresponding to the set of compute instances, with each VNIC associated with one compute instance in the set of compute instances.

[0073] Each compute instance is assigned a private overlay IP address via the VNIC associated with that compute instance. This private overlay IP address is assigned to the VNIC associated with the compute instance when the compute instance is created and is used to route traffic to the compute instance. In a given subnet, all VNICs use the same route table, security list, and DHCP options. As described above, each subnet in a VCN is associated with a contiguous range of overlay IP addresses that do not overlap with other subnets in the VCN and represent a subset of address space within the VCN's address space (e.g., 10.0.0.0 / 24 and 10.0.1.0 / 24). For a VNIC on a particular subnet of a VCN, the private overlay IP address assigned to the VNIC is an address from the contiguous range of overlay IP addresses assigned to the subnet.

[0074] In some embodiments, a compute instance may be assigned a private overlay IP address and optionally additional overlay IP addresses, such as one or more public IP addresses in the case of a public subnet. These addresses are assigned to the same VNIC or multiple VNICs associated with the compute instance. However, each instance has a primary VNIC associated with an overlay private IP address that is created and assigned to the instance when the instance is launched (this primary VNIC cannot be deleted). Additional VNICs, called secondary VNICs, can be added to an existing instance in the same availability domain as the primary VNIC. All VNICs are in the same availability domain as the instance. Secondary VNICs can be in the same subnetwork as the primary VNIC, or in different subnetworks in the same or a different VCN.

[0075] If a compute instance resides in a public subnet, it may optionally be assigned a public IP address. A subnet can be designated as a public or private subnet at the time of creation. A private subnet means that resources (e.g., compute instances) and associated VNICs in the subnet cannot have public overlay IP addresses. A public subnet means that resources and associated VNICs in the subnet can have public IP addresses. Customers can specify a subnet to reside in a single availability domain or multiple availability domains of a region or realm.

[0076] As described above, a VCN can be subdivided into one or more subnets. In one embodiment, a virtual router (VR) (referred to as a VCN VR or simply VR) configured for a VCN enables communication between subnets in the VCN. For a subnet within a VCN, the VR represents the subnet's logical gateway, allowing that subnet (i.e., compute instances on that subnet) to communicate with endpoints on other subnets within the VCN and with other endpoints outside the VCN. A VCN VR is a logical entity configured to route traffic between VNICs in a VCN and a virtual gateway ("gateway") associated with the VCN. Gateways are described separately below with respect to FIG. 1. A VCN VR is a Layer 3 / IP layer concept. In one embodiment, there is one VCN VR for a VCN, and a VCN VR potentially has a limited number of ports addressed by IP addresses, one port per subnet in the VCN. As such, a VCN VR has a different IP address for each subnet in the VCN to which the VCN VR is attached. VRs are also connected to various gateways configured for the VCN. In some embodiments, a specific overlay IP address in an overlay IP address range for a subnet is reserved for a port in the VCN VR for that subnet. For example, consider two subnets in a VCN, each with an associated address range of 10.0 / 16 and 10.1 / 16. For the first subnet in the VCN with address range 10.0 / 16, an address in this range is reserved for a port in the VCN VR for that subnet. In some cases, the first IP address in this range may be reserved for a VCN VR. For example, for a subnet with overlay IP address range 10.0 / 16, IP address 10.0.0.1 may be reserved for a port in the VCN VR for that subnet. For a second subnet in the same VCN with address range 10.1 / 16, the VCN VR may have a port with IP address 10.1.0.1 for the second subnet.A VCN VR has a different IP address for each of the VCN's subnets.

[0077] In some other embodiments, each subnet within a VCN may have its own associated VR, which is addressable by the subnet through use of a reserved or default IP address associated with the VR. The reserved or default IP address may, for example, be the first IP address in a range of IP addresses associated with the subnet. VNICs in the subnet can communicate with (e.g., send and receive packets from) the VR associated with the subnet using this default or reserved IP address. In such embodiments, the VR is the ingress / egress point for the subnet. VRs associated with a subnet within a VCN can communicate with other VRs associated with other subnets in the VCN. VRs can also communicate with gateways associated with the VCN. The VR functions for a subnet are running on or performed by one or more NVDs that perform the VNIC functions for VNICs in the subnet.

[0078] A VCN can be configured with route tables, security rules, and DHCP options. Route tables are virtual route tables for a VCN and contain rules that route traffic from subnets within the VCN to destinations outside the VCN via gateways or specially configured instances. A VCN's route tables can be customized to control how packets are forwarded / routed to the VCN. DHCP options represent configuration information that is automatically provided to instances at launch.

[0079] Security rules configured for a VCN represent the overlay firewall rules for the VCN. Security rules include ingress and egress rules and can specify the type of traffic (e.g., based on protocol and port) that can enter and exit instances in the VCN. Customers can choose whether a given rule is stateful or stateless. For example, a customer can allow ingress SSH traffic from anywhere to a set of instances by configuring a stateful ingress rule with source CIDR 0.0.0.0 / 0 and destination TCP port 22. Security rules can be implemented using network security groups or security lists. A network security group consists of a set of security rules that apply only to the resources in that group. On the other hand, a security list contains rules that apply to all resources in any subnet that uses the security list. A VCN can be provided with a default security list that contains default security rules. DHCP options configured for a VCN provide configuration information that is automatically given to instances in the VCN at launch.

[0080] In one embodiment, configuration information for a VCN is determined and stored by a VCN control plane. The VCN configuration information may include, for example, information about address ranges associated with the VCN, subnets and related information within the VCN, one or more VRs associated with the VCN, compute instances and associated VNICs in the VCN, NVDs performing various virtualized network functions (e.g., VNICs, VRs, gateways) associated with the VCN, VCN state information, and other VCN-related information. In one embodiment, a VCN distribution service publishes the configuration information stored by the VCN control plane or portions thereof to the NVD. The distributed information is stored by the NVD and may be used to update information (e.g., forwarding tables, routing tables, etc.) used to forward packets to compute instances in the VCN.

[0081] In one embodiment, VCN and subnet creation is handled by a VCN control plane (CP), and compute instance launch is handled by the compute control plane. The compute control plane is responsible for allocating physical resources for the compute instance and then calls the VCN control plane to create and associate VNICs with the compute instance. The VCN CP also sends VCN data mappings to a VCN data plane, which is configured to perform packet forwarding and routing functions. In one embodiment, a VCN VP provides a distribution service responsible for providing updates to the VCN data plane. Examples of VCN control planes are also shown in Figures 1, 2, 3, and 4 (see reference numbers 116, 216, 316, and 416) and described below.

[0082] Customers can create one or more VCNs using resources hosted by CSPI. Compute instances deployed in a customer VCN can communicate with various endpoints. These endpoints can include CSPI-hosted endpoints and endpoints external to CSPI.

[0083] Various architectures for implementing cloud-based services using CSPI are shown in Figures 1, 2, 3, 4, and 5 and are described below. Figure 1 is a high-level diagram of a distributed environment 100 illustrating an overlay or customer VCN hosted by CSPI, according to one embodiment. The distributed environment shown in Figure 1 includes multiple components of an overlay network. The distributed environment 100 shown in Figure 1 is merely an example and is not intended to unduly limit the scope of the claimed embodiments. Many variations, alternatives, and modifications are possible. For example, in some embodiments, the distributed environment shown in Figure 1 may include more or fewer systems or components than those shown in Figure 1, may combine two or more systems, or may have different system configurations or arrangements.

[0084] As shown in the example of FIG. 1 , distributed environment 100 includes CSPI 101, which provides services and resources that customers can use to sign up and build their own virtual cloud networks (VCNs). In one embodiment, CSPI 101 provides IaaS services to its subscribing customers. Data centers within CSPI 101 may be organized as one or more regions. One exemplary region, “Region US” 102, is shown in FIG. 1 . A customer has configured a customer VCN 104 for region 102. The customer may deploy various compute instances into VCN 104, which may include virtual machines or bare metal instances. Example instances include applications, databases, load balancers, etc.

[0085] In the embodiment shown in FIG. 1 , customer VCN 104 includes two subnets, “Subnet 1” and “Subnet 2,” each with its own CIDR IP address range. In FIG. 1 , Subnet 1’s overlay IP address range is 10.0 / 16, and Subnet 2’s address range is 10.1 / 16. VCN virtual router 105 represents the VCN’s logical gateway, enabling communication between subnets in VCN 104 and with other endpoints outside the VCN. VCN VR 105 is configured to route traffic between VNICs in VCN 104 and the gateway associated with VCN 104. VCN VR 105 provides a port for each subnet in VCN 104. For example, VR 105 may provide a port with IP address 10.0.0.1 to Subnet 1 and a port with IP address 10.1.0.1 to Subnet 2.

[0086] Multiple compute instances may be deployed in each subnet, and the compute instances may be virtual machine instances and / or bare metal instances. The compute instances in a subnet may be hosted by one or more host machines within CSPI 101. A compute instance joins a subnet via a VNIC associated with the compute instance. For example, as shown in FIG. 1, compute instance C1 is part of subnet 1 via a VNIC associated with the compute instance. Similarly, compute instance C2 is part of subnet 1 via a VNIC associated with C2. Similarly, multiple compute instances (which may be virtual machine instances or bare metal instances) may be part of subnet 1. Each compute instance is assigned a private overlay IP address and a MAC address via its associated VNIC. For example, in Figure 1, compute instance C1 has an overlay IP address of 10.0.0.2 and a MAC address of M1, while compute instance C2 has a private overlay IP address of 10.0.0.3 and a MAC address of M2. Each compute instance in Subnet 1 (including compute instances C1 and C2) has a default route to VCN VR105 using IP address 10.0.0.1, which is the IP address of the port in VCN VR105 for Subnet 1.

[0087] Subnet2 may have multiple compute instances deployed, including virtual machine instances and / or bare metal instances. For example, as shown in FIG. 1, compute instances D1 and D2 are part of Subnet2 via VNICs associated with each of the compute instances. In the embodiment shown in FIG. 1, compute instance D1 has an overlay IP address of 10.1.0.2 and a MAC address of MM1, while compute instance D2 has a private overlay IP address of 10.1.0.3 and a MAC address of MM2. Each compute instance in Subnet2 (including compute instances D1 and D2) has a default route to VCN VR105 using IP address 10.1.0.1, which is the IP address of the port in VCN VR105 for Subnet2.

[0088] VCN A 104 may also include one or more load balancers. For example, a load balancer may be provided for a subnet and configured to load balance traffic among multiple compute instances on the subnet. A load balancer may also be provided to load balance traffic between subnets of the VCN.

[0089] A particular compute instance deployed in VCN 104 can communicate with a variety of different endpoints. These endpoints may include endpoints hosted by CSPI 200 and endpoints outside of CSPI 200. Endpoints hosted by CSPI 101 may include endpoints on the same subnet as the particular compute instance (e.g., communication between two compute instances in Subnet 1), endpoints on different subnets within the same VCN (e.g., communication between a compute instance in Subnet 1 and a compute instance in Subnet 2), endpoints in different VCNs in the same region (e.g., communication between a compute instance in Subnet 1 and an endpoint in a VCN in the same region 106 or 110, communication between a compute instance in Subnet 1 and an endpoint in the service network 110 in the same region), or endpoints in a VCN in a different region (e.g., communication between a compute instance in Subnet 1 and an endpoint in a VCN in a different region 108). Additionally, compute instances in a subnet hosted by CSPI 101 may communicate with endpoints not hosted by CSPI 101 (i.e., external to CSPI 101). These external endpoints include endpoints in a customer's on-premise network 116, endpoints in other hosted remote cloud networks 118, public endpoints 114 accessible over a public network such as the Internet, and other endpoints.

[0090] Communication between compute instances on the same subnet is facilitated through the use of VNICs associated with the source and destination compute instances. For example, compute instance C1 on Subnet 1 may want to send a packet to compute instance C2 on Subnet 1. For a packet originating from the source compute instance and destined for another compute instance in the same subnet, the packet is first processed by the VNIC associated with the source compute instance. The processing performed by the VNIC associated with the source compute instance may include determining the packet's destination information from the packet header, identifying any policies (e.g., security lists) configured for the VNIC associated with the source compute instance, determining the packet's next hop, performing encapsulation / decapsulation functions on the packet as needed, and forwarding / routing the packet to the next hop to facilitate communication of the packet to its intended destination. If the destination compute instance is in the same subnet as the source compute instance, the VNIC associated with the source compute instance is configured to identify the VNIC associated with the destination compute instance and forward the packet to that VNIC for processing, which then forwards the packet to the destination compute instance via the VNIC associated with the destination compute instance.

[0091] When a packet is sent from a compute instance in one subnet to an endpoint in a different subnet of the same VCN, this communication is facilitated by the VNICs and VCN VRs associated with the source and destination compute instances. For example, in Figure 1, if compute instance C1 in Subnet 1 wants to send a packet to compute instance D1 in Subnet 2, the packet is first processed by the VNIC associated with compute instance C1. The VNIC associated with compute instance C1 is configured to route the packet to VCN VR using the default route or port 10.0.0.1 of VCN VR105. VCN VR105 is configured to route the packet to Subnet 2 using port 10.1.0.1. Once the packet is received and processed by the VNIC associated with D1, this VNIC forwards the packet to compute instance D1.

[0092] When a packet is sent from a compute instance in VCN 104 to an endpoint outside VCN 104, the communication is facilitated by the VNIC associated with the source compute instance, VCN VR 105, and a gateway associated with VCN 104. VCN 104 may have one or more types of gateways associated with it. A gateway is an interface between a VCN and another endpoint, where the other endpoint is outside the VCN. A gateway is a Layer 3 / IP layer concept that allows a VCN to communicate with endpoints outside the VCN. In this way, a gateway facilitates the flow of traffic between a VCN and other VCNs or networks. A VCN may be configured with a variety of different types of gateways to facilitate different types of communication with different types of endpoints. Depending on the gateway, the communication may be over a public network (e.g., the Internet) or a private network. A variety of communication protocols may be used for these communications.

[0093] For example, compute instance C1 may want to communicate with an endpoint outside VCN 104. The packet may initially be processed by a VNIC associated with source compute instance C1. The VNIC's processing determines that the packet's destination is outside of Subnet 1 of C1. The VNIC associated with C1 may forward the packet to VCN VR105 in VCN 104. VCN VR105 then processes the packet and, as part of that processing, determines a particular gateway associated with VCN 104 as the packet's next hop based on the packet's destination. VCN VR105 may then forward the packet to the identified particular gateway. For example, if the destination is an endpoint in a customer's on-premises network, the packet may be forwarded by VCN VR105 to Dynamic Routing Gateway (DRG) gateway 122 configured for VCN 104. Forwarding the packet from the gateway to the next hop can then facilitate the packet's delivery to its ultimate destination.

[0094] Various different types of gateways may be configured for a VCN. Examples of gateways that may be configured for a VCN are shown in FIG. 1 and described below. Examples of gateways associated with VCNs are also shown in FIGS. 1, 2, 3, and 4 (e.g., gateways referenced by reference numbers 134, 136, 138, 234, 236, 238, 334, 336, 338, 434, 436, and 438) and described below. As shown in the embodiment of FIG. 1, a dynamic routing gateway (DRG) 122 may be added to or associated with customer VCN 104 to provide a path for private network traffic communication between customer VCN 104 and another endpoint, which may be the customer's on-premises network 116, a VCN 108 in a different region of CSPI 101, or another remote cloud network 118 not hosted by CSPI 101. The customer on-premises network 116 may be a customer network built using customer resources or may be a customer data center. Access to the customer on-premises network 116 is typically highly restricted. If a customer has both an on-premises network 116 and one or more VCNs 104 deployed or hosted in the cloud by CSPI 101, they may want to enable the on-premises network 116 and the cloud-based VCNs 104 to communicate with each other. This allows a customer to build an extended hybrid environment that includes the on-premises network 116 and the customer's VCN 104 hosted by CSPI 101. This communication is made possible by DRG 122. To enable such communication, a communication channel 124 is set up so that one endpoint of the channel resides in the customer on-premises network 116 and the other endpoint resides in CSPI 101 and is connected to the customer VCN 104. The communication channel 124 may traverse a public or private communication network, such as the Internet.A variety of different communication protocols may be used, such as IPsec VPN technology over a public communication network such as the Internet, or Oracle's FastConnect technology, which uses a private network instead of a public network. The device or equipment in the customer on-premises network 116 that forms one endpoint of the communication channel 124 is referred to as customer premises equipment (CPE), such as CPE 126 shown in Figure 1. On the CSPI 101 side, the endpoint may be a host machine running DRG 122.

[0095] In one embodiment, remote peering connections (RPCs) can be added to a DRG, allowing customers to peer one VCN with another VCN in a different region. Using such RPCs, a customer VCN 104 can connect to a VCN 108 in another region using a DRG 122. The DRG 122 can also be used to communicate with other remote cloud networks 118 not hosted by CSPI 101, such as the Microsoft Azure cloud or the Amazon AWS cloud.

[0096] 1, an Internet Gateway (IGW) 120 may be configured for customer VCN 104, allowing compute instances on VCN 104 to communicate with public endpoints 114 accessible over a public network, such as the Internet. IGW 120 is a gateway that connects a VCN to a public network, such as the Internet. IGW 120 allows a public subnet within a VCN, such as VCN 104, to directly access a public endpoint 112 on public network 114, such as the Internet, if the resources in that public subnet have public overlay IP addresses. Using IGW 120, connections can be initiated from a subnet within VCN 104 or from the Internet.

[0097] A network address translation (NAT) gateway 128 configured for a customer's VCN 104 may enable cloud resources in the customer's VCN that do not have dedicated public overlay IP addresses to access the Internet, but the NAT gateway 128 does not expose these resources to direct inbound Internet connections (e.g., L4-L7 connections). This allows private subnets within the VCN (such as Private Subnet 1 of VCN 104) to privately access public endpoints on the Internet. The NAT gateway only allows connections to be initiated from the private subnet to the public Internet, not from the Internet to the private subnet.

[0098] In some embodiments, a service gateway (SGW) 126 may be configured for customer VCN 104, providing a pathway for private network traffic between VCN 104 and service endpoints supported in service network 110. In some embodiments, service network 110 may be provided by a CSP and may offer a variety of services. One example of such a service network is Oracle's Services Network, which offers a variety of services available to customers. For example, compute instances (e.g., database systems) in a private subnet of customer VCN 104 can back up data to a service endpoint (e.g., Object Storage) without requiring a public IP address or access to the Internet. In some embodiments, a VCN may have only one SGW, and connections can be initiated only from subnets within the VCN, not from service network 110. When a VCN is peered with another VCN, resources in the other VCN typically cannot access the SGW. Additionally, resources in an on-premises network connected to a VCN with FastConnect or VPN Connect can use a service gateway configured for that VCN.

[0099] In one embodiment, SGW 126 uses the concept of a service CIDR (Classless Inter-Domain Routing) label, which is a string that represents all of the regional public IP address ranges for a service or services of interest. A customer uses the service CIDR label when configuring the SGW and associated routing rules to control traffic to the service. A customer can optionally use the service CIDR label when configuring security rules without having to adjust if the service's public IP address changes in the future.

[0100] A local peering gateway (LPG) 132 is a gateway that can be added to a customer VCN 104 to enable the VCN 104 to peer with another VCN in the same region. Peering means that the VCNs communicate using private IP addresses, but without traffic traversing a public network such as the Internet or routing traffic to the customer's on-premises network 116. In a preferred embodiment, a VCN has a separate LPG for each peering it establishes. Local peering or VCN peering is a common method used to establish network connectivity between different applications or infrastructure management functions.

[0101] Service providers (e.g., providers of services in service network 110) can provide access to services using various access models. In a public access model, services are exposed as public endpoints publicly accessible by compute instances in the customer VCN over a public network, such as the Internet, and / or they can be privately accessible through SGW 126. In a specific private access model, services are accessible as private IP endpoints in a private subnet of the customer VCN. This is called private endpoint (PE) access, and service providers can expose their services as instances in the customer's private network. Private endpoint resources represent services within the customer's VCN. Each PE appears as a VNIC in a customer-selected subnet in the customer's VCN (called a PE-VNIC and having one or more private IPs). Thus, the PE provides a way to present services in the private customer VCN subnet using VNICs. Because the endpoints are exposed as VNICs, all functionality associated with a VNIC, such as routing rules and security lists, is available to the PE VNIC.

[0102] Service providers may register their services to enable access through the PE. Providers can associate policies with services that restrict the visibility of the service to the customer tenancy. Providers can register multiple services under a single virtual IP address (VIP), especially for multi-tenant services. There may be multiple such private endpoints (in multiple VCNs) representing the same service.

[0103] Compute instances in the private subnet can then access the service using the private IP address or service DNS name of the PE VNIC. Compute instances in the customer VCN can access the service by sending traffic to the private IP address of the PE in the customer VCN. A private access gateway (PAGW) 130 is a gateway resource that can be attached to a service provider VCN (e.g., a VCN in service network 110) that acts as the ingress / egress point for all traffic to customer subnet private endpoints. The PAGW 130 allows providers to scale the number of PE connections without using their internal IP address resources. A provider need only configure one PAGW for any number of services registered in a single VCN. A provider can represent services as private endpoints in multiple VCNs for one or more customers. From the customer's perspective, the PE VNIC does not appear to be attached to the customer's instance, but rather to the service with which the customer wants to interact. Traffic destined for the private endpoint is routed to the service through the PAGW 130. These are called customer-to-service private connections (C2S connections).

[0104] The use of the PE concept also allows traffic to flow across FastConnect / IPsec links and private endpoints in the customer VCN, extending private access for services to the customer's on-premises network and data center, and allows traffic to flow between LPG 132 and PEs in the customer VCN, extending private access for services to the customer's peering VCN.

[0105] Customers can control VCN routing at the subnet level, allowing them to specify which subnets in their VCNs (such as VCN 104) use each gateway. The VCN's route tables are used to determine whether traffic is allowed from a VCN through a particular gateway. For example, in a particular instance, the route table for a public subnet in customer VCN 104 may direct non-local traffic through IGW 120. The route table for a private subnet in the same customer VCN 104 may direct traffic destined for CSP services through SGW 126. All other traffic may be directed through NAT gateway 128. Route tables only control traffic leaving a VCN.

[0106] Traffic entering the VCN via the gateway through an inbound connection is controlled using security lists associated with the VCN. All resources in a subnet use the same route table and security lists. Security lists can be used to control the specific types of traffic that can enter and exit instances in a VCN's subnets. Security list rules can include input (inbound) and output (outbound) rules. For example, an input rule can specify an allowed source address range, while an output rule can specify an allowed destination address range. Security rules can specify a specific protocol (e.g., TCP, ICMP), a specific port (e.g., 22 for SSH, 3389 for Windows RDP), etc. In some embodiments, the instance's operating system can enforce its own firewall rules in line with the security list rules. Rules can be stateful (e.g., connections are tracked and responses are automatically allowed without explicit security list rules for the response traffic) or stateless.

[0107] Access from a customer VCN (i.e., by resources or compute instances deployed in VCN 104) can be categorized as public access, private access, or dedicated access. Public access describes an access model in which public IP addresses or NATs are used to access public endpoints. Private access allows customer workloads in VCN 104 with private IP addresses (e.g., resources in a private subnet) to access services without traversing a public network such as the Internet. In one embodiment, CSPI 101 enables customer VCN workloads with private IP addresses to access (public service endpoints of) services using a service gateway. Thus, the service gateway provides a private access model by establishing a virtual link between the customer VCN and the public endpoints of the services that reside outside the customer's private network.

[0108] CSPI may also provide dedicated public access by using technologies such as FastConnect public peering, which allows customer on-premises instances to use a FastConnect connection to access one or more services in the customer VCN without traversing a public network such as the Internet. CSPI may also provide dedicated private access by using FastConnect private peering, which allows customer on-premises instances with private IP addresses to access workloads in the customer's VCN using a FastConnect connection. FastConnect is an alternative network connection using the public Internet to connect a customer's on-premises network to CSPI and its services. FastConnect provides an easy, flexible, and economical way to create a dedicated private connection with higher bandwidth options, higher reliability, and a consistent networking experience compared to Internet-based connections.

[0109] FIG. 1 and the accompanying discussion above describe various virtualized components in an exemplary virtual network. As noted above, a virtual network is built on an underlying physical or infrastructure network. FIG. 2 is a simplified architectural diagram of the physical components in a physical network within CSPI 200 that forms the basis of the virtual network, according to one embodiment. As shown, CSPI 200 provides a distributed environment including components and resources (e.g., compute, memory, and network resources) provided by a cloud service provider (CSP). These components and resources are used to provide cloud services (e.g., IaaS services) to subscribing customers, i.e., customers who have signed up for one or more services offered by the CSP. Customers are provisioned with a subset of CSPI 200's resources (e.g., compute, memory, and network resources) based on the services for which they subscribe. Customers can then build their own cloud-based (i.e., CSPI-hosted), customizable private virtual networks using the physical compute, memory, and network resources provided by CSPI 200. As previously mentioned, these customer networks are referred to as virtual cloud networks (VCNs). Customers can deploy one or more customer resources, such as compute instances, into these Customer VCNs. The compute instances can be in the form of virtual machines, bare metal instances, etc. CSPI 200 provides infrastructure and a set of complementary cloud services that enable customers to build and run a wide range of applications and services in a hosted, highly available environment.

[0110] In the exemplary embodiment shown in FIG. 2, the physical components of CSPI 200 include one or more physical host machines or servers (e.g., 202, 206, 208), network virtualization devices (NVDs) (e.g., 210, 212), top-of-rack (TOR) switches (e.g., 214, 216), and physical networks (e.g., 218) and switches of physical network 218. The physical host machines or servers can host and execute various compute instances that participate in one or more subnets of the VCN. The compute instances can include virtual machine instances and bare metal instances. For example, the various compute instances shown in FIG. 1 may be hosted by the physical host machines shown in FIG. 2. The virtual machine compute instances in the VCN may be executed by a single host machine or by multiple different host machines. Additionally, the physical host machines can host virtual host machines, container-based hosts or functions, etc. The VNICs and VCN VRs shown in FIG. 1 may be executed by the NVDs shown in FIG. 2. The gateway shown in FIG. 1 may be implemented by a host machine and / or NVD shown in FIG.

[0111] A host machine or server may run a hypervisor (also called a virtual machine monitor or VMM) that creates and makes available a virtualized environment on the host machine. Virtualization or a virtualized environment facilitates cloud-based computing. The hypervisor on a host machine may create, run, and manage one or more compute instances on the host machine. The hypervisor on a host machine enables the sharing of the host machine's physical computing resources (e.g., compute, memory, and network resources) among the various compute instances that the host machine runs.

[0112] For example, as shown in FIG. 2, host machines 202 and 208 execute hypervisors 260 and 266, respectively. These hypervisors may be implemented in software, firmware, hardware, or a combination thereof. Typically, a hypervisor is a process or software layer that resides on a host machine's operating system (OS) and runs on the host machine's hardware processor. A hypervisor provides a virtualized environment by enabling the sharing of the host machine's physical computing resources (e.g., processing resources such as processors / cores, memory resources, and networking resources) among various virtual machine compute instances that the host machine executes. For example, in FIG. 2, hypervisor 260 may reside on the OS of host machine 202 and enable the sharing of the host machine's computing resources (e.g., processing, memory, and networking resources) among the compute instances (e.g., virtual machines) that the host machine executes. A virtual machine may have its own operating system (referred to as a guest operating system), which may be the same as or different from the host machine's OS. The operating system of a virtual machine executed by a host machine may be the same as or different from the operating system of another virtual machine executed by the same host machine. In this manner, the hypervisor allows multiple operating systems to run side by side with each other while sharing the same computing resources of the host machine. The host machines shown in Figure 2 may have the same type of hypervisor or different types of hypervisors.

[0113] A compute instance can be a virtual machine instance or a bare metal instance. In Figure 2, compute instance 268 on host machine 202 and 274 on host machine 208 are examples of virtual machine instances. Host machine 206 is an example of a bare metal instance provided to a customer.

[0114] In some cases, an entire host machine is provisioned for a single customer, and the one or more compute instances (virtual machine or bare metal instances) that it hosts all belong to that same customer. In other cases, a host machine may be shared among multiple customers (i.e., multiple tenants). In such a multi-tenancy scenario, a host machine may host virtual machine compute instances that belong to different customers. These compute instances may be part of different VCNs for different customers. In some embodiments, bare metal compute instances are hosted on bare metal servers that do not include a hypervisor. When a bare metal compute instance is provisioned, a single customer or tenant maintains control of the physical CPU, memory, and network interfaces of the host machine that hosts the bare metal instance, and the host machine is not shared with other customers or tenants.

[0115] As previously mentioned, each compute instance that is part of a VCN is associated with a VNIC, which may make the compute instance a member of a subnet of the VCN. A VNIC associated with a compute instance facilitates transmission of packets or frames to the compute instance. A VNIC is associated with the compute instance when the compute instance is created. In one embodiment, for compute instances executed by a host machine, the VNIC associated with the compute instance is implemented by an NVD connected to the host machine. For example, in FIG. 2, host machine 202 executes virtual machine compute instance 268, which is associated with VNIC 276, which is implemented by NVD 210 connected to host machine 202. In another example, host machine 206 hosts bare metal instance 272, which is associated with VNIC 280, which is implemented by NVD 212 connected to host machine 206. In yet another example, the VNIC 284 is associated with a compute instance 274 that the host machine 208 executes, and the VNIC 284 is executed by an NVD 212 that is connected to the host machine 208 .

[0116] For compute instances hosted by a host machine, the NVD connected to that host machine also runs VCN VRs corresponding to the VCNs of which the compute instances are elements. For example, in the embodiment shown in Figure 2, NVD 210 runs VCN VR 277 corresponding to the VCN of which compute instance 268 is an element. NVD 212 may also run one or more VCN VRs 283 corresponding to the VCNs corresponding to the compute instances hosted by host machines 206 and 208.

[0117] A host machine may include one or more network interface cards (NICs) that allow the host machine to connect to other devices. A NIC on a host machine may provide one or more ports (or interfaces) that allow the host machine to communicate with other devices. For example, a host machine may be connected to an NVD by one or more ports (or interfaces) provided on the host machine and the NVD. A host machine may also be connected to other devices, such as another host machine.

[0118] 2, host machine 202 is connected to NVD 210 using link 220 extending between port 234 provided by NIC 232 of host machine 202 and port 236 of NVD 210. Host machine 206 is connected to NVD 212 using link 224 extending between port 246 provided by NIC 244 of host machine 206 and port 248 of NVD 212. Host machine 208 is connected to NVD 212 using link 226 extending between port 252 provided by NIC 250 of host machine 208 and port 254 of NVD 212.

[0119] The NVDs, in turn, are connected via communication links to top-of-rack (TOR) switches, which are connected to a physical network 218 (also referred to as a switch fabric). In one embodiment, the links between the host machines and the NVDs and between the NVDs and the TOR switches are Ethernet links. For example, in Figure 2, NVDs 210 and 212 are connected to TOR switches 214 and 216, respectively, using links 228 and 230. In one embodiment, links 220, 224, 226, 228, and 230 are Ethernet links. The collection of host machines and NVDs connected to a TOR may also be referred to as a rack.

[0120] The physical network 218 provides a communications fabric that allows the TOR switches to communicate with each other. The physical network 218 can be a multi-tier network. In one embodiment, the physical network 218 is a multi-tier Clos network of switches, with the TOR switches 214 and 216 representing leaf-level nodes of the multi-tier, multi-node physical switching network 218. Various Clos network configurations are possible, including, but not limited to, a 2-tier network, a 3-tier network, a 4-tier network, a 5-tier network, and generally an "n" tier network. An example of a Clos network is shown in FIG. 5 and described below.

[0121] A variety of different connection configurations are possible between host machines and NVDs, including one-to-one, many-to-one, and one-to-many configurations. In one embodiment of a one-to-one configuration, each host machine is connected to its own separate NVD. For example, in Figure 2, host machine 202 is connected to NVD 210 via NIC 232 on host machine 202. In a many-to-one configuration, multiple host machines are connected to a single NVD. For example, in Figure 2, host machines 206 and 208 are connected to the same NVD 212 via NICs 244 and 250, respectively.

[0122] In a one-to-multiple configuration, one host machine is connected to multiple NVDs. FIG. 3 shows an example of a CSPI 300 in which a host machine is connected to multiple NVDs. As shown in FIG. 3, a host machine 302 includes a network interface card (NIC) 304 including multiple ports 306 and 308. The host machine 300 is connected to a first NVD 310 via port 306 and link 320, and to a second NVD 312 via port 308 and link 322. Ports 306 and 308 may be Ethernet ports, and links 320 and 322 between the host machine 302 and the NVDs 310 and 312 may be Ethernet links. The NVD 310 is connected to a first TOR switch 314, and the NVD 312 is connected to a second TOR switch 316. The links between the NVDs 310 and 312 and the TOR switches 314 and 316 may be Ethernet links. TOR switches 314 and 316 represent layer 0 switching devices of a multi-tier physical network 318 .

[0123] 3 provides two separate physical network paths from the physical switch network 318 to the host machine 302: a first path through the TOR switch 314 to the NVD 310 and thus to the host machine 302, and a second path through the TOR switch 316 to the NVD 312 and thus to the host machine 302. These separate paths may increase the availability (referred to as high availability) of the host machine 302. If there is a problem with one of the paths (e.g., if a link on one of the paths goes down) or if there is a problem with one of the devices (e.g., if a particular NVD is not functioning), the other path may be used to communicate with the host machine 302.

[0124] In the configuration shown in Figure 3, the host machine is connected to two different NVDs using two different ports provided by the host machine's NIC. In other embodiments, the host machine may have multiple NICs allowing connection to multiple NVDs.

[0125] Referring again to Figure 2, an NVD is a physical device or component that performs one or more network and / or storage virtualization functions. An NVD may be any device that has one or more processing units (e.g., a CPU, a network processing unit (NPU), an FPGA, a packet processing pipeline, etc.), memory including cache, and ports. Various virtualization functions may be performed by software / firmware executed by one or more processing units of the NVD.

[0126] The NVD may be implemented in a variety of different forms. For example, in one embodiment, the NVD is implemented as an interface card with an embedded processor, referred to as a smart NIC or intelligent NIC. The smart NIC is a separate device from the NIC on the host machine. In Figure 2, NVDs 210 and 212 may be implemented as smart NICs connected to host machine 202 and host machines 206 and 208, respectively.

[0127] However, a smart NIC is merely one example of an implementation of an NVD. Various other implementations are possible. For example, in some other embodiments, the NVD or one or more functions performed by the NVD may be incorporated into or performed by one or more host machines, one or more TOR switches, and other components of CSPI200. For example, the NVD may be embodied in a host machine, and the functions performed by the NVD may be performed by the host machine. As another example, the NVD may be part of a TOR switch, or the TOR switch may be configured to perform the functions performed by the NVD, enabling the TOR switch to perform various complex packet transformations used in public clouds. A TOR performing the functions of an NVD may also be referred to as a smart TOR. In still other embodiments, if virtual machine (VM) instances rather than bare metal (BM) instances are provided to customers, the functions performed by the NVD may be implemented inside the hypervisor of the host machine. In some other embodiments, some of the NVD's functions may be offloaded to a centralized service running on a group of host machines.

[0128] In some embodiments, such as when implemented as a smart NIC as shown in FIG. 2, an NVD may have multiple physical ports that allow the NVD to connect to one or more host machines and one or more TOR switches. Ports on an NVD may be classified as host-side ports (also referred to as "south ports") or network-side or TOR-side ports (also referred to as "north ports"). A host-side port of an NVD is a port used to connect the NVD to a host machine. Examples of host-side ports in FIG. 2 include port 236 on NVD 210 and ports 248 and 254 on NVD 212. A network-side port of an NVD is a port used to connect the NVD to a TOR switch. Examples of network-side ports in FIG. 2 include port 256 on NVD 210 and port 258 on NVD 212. As shown in FIG. 2, the NVD 210 is connected to the TOR switch 214 by link 228, which extends from port 256 of the NVD 210 to the TOR switch 214. Similarly, the NVD 212 is connected to the TOR switch 216 by a link 230 extending from a port 258 of the NVD 212 to the TOR switch 216 .

[0129] An NVD can receive packets and frames from a host machine (e.g., packets and frames generated by a compute instance hosted by the host machine) via a host-side port, perform necessary packet processing, and then forward the packets and frames to a TOR switch via a network-side port of the NVD. An NVD can receive packets and frames from a TOR switch via a network-side port of the NVD, perform necessary packet processing, and then forward the packets and frames to a host machine via a host-side port of the NVD.

[0130] In some embodiments, there may be multiple ports and associated links between the NVD and the TOR switch. These ports and links may be aggregated to form a link aggregator group (LAG) consisting of multiple ports or links. Link aggregation allows multiple physical links between two endpoints (e.g., between the NVD and the TOR switch) to be treated as a single logical link. All physical links in a given LAG may operate in full-duplex mode at the same speed. LAGs help increase the bandwidth and reliability of the connection between two endpoints. If one of the physical links in the LAG goes down, traffic is dynamically and transparently reassigned to one of the other physical links in the LAG. The aggregated physical link provides higher bandwidth than each individual link. Multiple ports associated with a LAG are treated as a single logical port. Traffic may be load-balanced across the multiple physical links in the LAG. One or more LAGs may be configured between two endpoints. The two endpoints may be, for example, between the NVD and the TOR switch, or between a host machine and the NVD.

[0131] The NVD implements or performs network virtualization functions. These functions are performed by software / firmware executed by the NVD. Examples of network virtualization functions include, but are not limited to, packet encapsulation and decapsulation functions, functions for creating VCN networks, functions for enforcing network policies such as VCN security list (firewall) functions, and functions for facilitating routing and forwarding of packets to compute instances in the VCN. In one embodiment, upon receiving a packet, the NVD is configured to execute a packet processing pipeline for processing the packet and determining how to forward or route the packet. As part of this packet processing pipeline, the NVD may execute one or more virtual functions associated with the overlay network, such as running VNICs associated with compute instances in the VCN, running virtual routers (VRs) associated with the VCN, encapsulating and decapsulating packets to facilitate forwarding or routing in the virtual network, running a gateway (e.g., a local peering gateway), implementing security lists, network security groups, network address translation (NAT) functions (e.g., public IP to private IP translation on a per-host basis), throttling functions, and other functions.

[0132] In one embodiment, the NVD's packet processing data path may include multiple packet pipelines, each consisting of a series of packet transformation stages. In one embodiment, upon receipt of a packet, the packet is parsed and sorted into a single pipeline. The packet is then processed linearly, one stage at a time, until it is either discarded or sent out through the NVD's interface. These stages provide basic functional packet processing building blocks (e.g., header validation, throttling enforcement, new Layer 2 header insertion, L4 firewall enforcement, VCN encapsulation / decapsulation, etc.), such that new pipelines can be constructed by assembling existing stages, and new functionality can be added by creating and inserting new stages into existing pipelines.

[0133] The NVD may perform both control plane and data plane functions corresponding to the VCN's control and data planes. Examples of the VCN control plane are also shown in Figures 1, 2, 3, and 4 (see reference numbers 116, 216, 316, and 416) and described below. Examples of the VCN data plane are shown in Figures 1, 2, 3, and 4 (see reference numbers 118, 218, 318, and 418) and described below. Control plane functions include functions used to configure the network (e.g., routing and route table configuration, VNIC configuration, etc.) and control how data is forwarded. In one embodiment, a VCN control plane is provided that centrally computes and exposes all overlay-to-foundation mappings to the NVD and virtual network edge devices (e.g., various gateways such as DRGs, SGWs, and IGWs). Firewall rules may also be exposed by the same mechanism. In one embodiment, the NVD retrieves only mappings relevant to the NVD. The data plane functions include functionality for the actual routing / forwarding of packets based on the configuration set using the control plane. The VCN data plane is achieved by encapsulating customer network packets before they traverse the underlying network. The encapsulation / decapsulation functionality is achieved on the NVD. In one embodiment, the NVD is configured to intercept all network packets entering and leaving the host machine and perform network virtualization functions.

[0134] As described above, the NVD performs various virtualization functions, including VNICs and VCN VRs. The NVD may execute VNICs associated with compute instances hosted by one or more host machines connected to the VNIC. For example, as shown in FIG. 2, NVD 210 executes the functions of VNIC 276 associated with compute instance 268 hosted by host machine 202 connected to NVD 210. As another example, NVD 212 executes VNIC 280 associated with bare metal compute instance 272 hosted by host machine 206 and VNIC 284 associated with compute instance 274 hosted by host machine 208. The host machines can host compute instances that belong to different VCNs belonging to different customers, and the NVDs connected to the host machines can execute VNICs (i.e., perform VNIC-related functions) corresponding to these compute instances.

[0135] The NVDs also execute VCN virtual routers corresponding to the VCNs of the compute instances. For example, in the embodiment shown in FIG. 2, NVD 210 executes VCN VR 277 corresponding to the VCN to which compute instance 268 belongs. NVD 212 executes one or more VCN VRs 283 corresponding to one or more VCNs to which compute instances hosted by host machines 206 and 208 belong. In one embodiment, a VCN VR corresponding to a given VCN is executed by all NVDs connected to a host machine that hosts at least one compute instance belonging to that VCN. If a host machine hosts compute instances that belong to different VCNs, the NVDs connected to that host machine may execute VCN VRs corresponding to the different VCNs.

[0136] In addition to VNICs and VCN VRs, the NVD may run various software (e.g., daemons) and may include one or more hardware components that facilitate the various network virtualization functions performed by the NVD. For simplicity, these various components are grouped together as a “packet processing component” shown in FIG. 2 . For example, NVD 210 includes packet processing component 286, and NVD 212 includes packet processing component 288. For example, the packet processing component of the NVD may include a packet processor configured to interact with the NVD's ports and hardware interfaces to monitor all packets received and communicated using the NVD and to store network information. The network information may include, for example, network flow information and per-flow information (e.g., per-flow statistics) that identify the various network flows processed by the NVD. In one embodiment, network flow information may be stored per VNIC. The packet processor may perform per-packet operations as well as implement a stateful NAT and an L4 firewall (FW). As another example, the packet processing component may include a replication agent configured to replicate information stored by the NVD to one or more different replication target stores. As yet another example, the packet processing component may include a logging agent configured to perform logging functions for the NVD. The packet processing component may also include software for monitoring the performance and health of the NVD, and possibly the status and health of other components connected to the NVD.

[0137] FIG. 1 illustrates components of an exemplary virtual or overlay network, including a VCN, subnets within the VCN, compute instances deployed to the subnets, VNICs associated with the compute instances, VRs in the VCN, and a set of gateways configured for the VCN. The overlay components illustrated in FIG. 1 may be executed or hosted by one or more of the physical components illustrated in FIG. 2. For example, compute instances in a VCN may be executed or hosted by one or more host machines illustrated in FIG. 2. For compute instances hosted by a host machine, the VNICs associated with the compute instances are typically executed by an NVD connected to the host machine (i.e., the VNIC functionality is provided by an NVD connected to the host machine). The VCN VR functionality for a VCN is performed by all NVDs connected to host machines that host or execute compute instances that are part of the VCN. The gateways associated with a VCN may be executed by one or more different types of NVDs. For example, one gateway may be executed by a smart NIC, while other gateways may be executed by one or more host machines or other implementations of NVDs.

[0138] As mentioned above, a compute instance in a customer VCN can communicate with a variety of different endpoints, which can be in the same subnet as the source compute instance, in a different subnet but within the same VCN as the source compute instance, or outside the VCN of the source compute instance. These communications are facilitated through the use of VNICs associated with the compute instances, VCN VRs, and gateways associated with the VCN.

[0139] For communication between two compute instances on the same subnet in a VCN, this communication is facilitated through the use of VNICs associated with the source and destination compute instances. The source and destination compute instances may be hosted by the same host machine or different host machines. A packet originating from the source compute instance may be forwarded from the host machine hosting the source compute instance to an NVD connected to that host machine. On the NVD, the packet is processed using a packet processing pipeline, which may include running the VNIC associated with the source compute instance. Because the packet's destination endpoint is in the same subnet, the VNIC associated with the source compute instance forwards the packet to the NVD running the VNIC associated with the destination compute instance. The NVD then processes the packet and forwards it to the destination compute instance. The VNICs associated with the source and destination compute instances may be running on the same NVD (e.g., if both the source and destination compute instances are hosted by the same host machine) or on different NVDs (e.g., if the source and destination compute instances are hosted by different host machines connected to different NVDs). The VNICs may use routing / forwarding tables stored by the NVD to determine the next hop for a packet.

[0140] When a packet is sent from a compute instance in one subnet to an endpoint in a different subnet of the same VCN, the packet originating from the source compute instance is sent from the host machine hosting the source compute instance to the NVD connected to that host machine. The NVD processes the packet using a packet processing pipeline, which may include running one or more VNICs and VRs associated with the VCN. For example, as part of the packet processing pipeline, the NVD executes or invokes a function corresponding to the VNIC associated with the source compute instance (also referred to as VNIC execution). The function executed by the VNIC may include verifying the VLAN tag on the packet. Because the packet's destination is outside the subnet, the NVD then invokes a VCN VR function to execute it. The VCN VR then routes the packet to the NVD that executes the VNIC associated with the destination compute instance. The VNIC associated with the destination compute instance then processes the packet and forwards it to the destination compute instance. The VNICs associated with the source and destination compute instances may be running on the same NVD (e.g., if both the source and destination compute instances are hosted by the same host machine) or may be running on different NVDs (e.g., if the source and destination compute instances are hosted by different host machines connected to different NVDs).

[0141] If the packet's destination is outside the VCN of the source compute instance, the packet originating from the source compute instance is sent from the host machine hosting the source compute instance to the NVD connected to that host machine. The NVD runs the VNIC associated with the source compute instance. Because the packet's destination endpoint is outside the VCN, the packet is then processed by the VCN VR for that VCN. The NVD invokes a VCN VR function, which forwards the packet to an NVD running the appropriate gateway associated with the VCN. For example, if the destination is an endpoint in a customer's on-premises network, the packet may be forwarded by the VCN VR to an NVD running a DRG gateway configured for the VCN. The VCN VR may be running on the same NVD as the NVD running the VNIC associated with the source compute instance, or it may be running on a different NVD. The gateway may be running on the NVD, which may be a smart NIC, a host machine, or another implementation of the NVD. The packet is then processed by the gateway and forwarded to a next hop that facilitates delivery of the packet to the intended destination endpoint. For example, in the embodiment shown in FIG. 2, a packet originating from compute instance 268 may be sent from host machine 202 to NVD 210 over link 220 (using NIC 232). In NVD 210, VNIC 276 is called VNIC 276 because it is associated with source compute instance 268. VNIC 276 is configured to examine information encapsulated in the packet, determine a next hop to forward the packet to facilitate delivery of the packet to the intended destination endpoint, and then forward the packet to the determined next hop.

[0142] Compute instances deployed in a VCN can communicate with a variety of different endpoints. These endpoints can include endpoints hosted by CSPI 200 and endpoints external to CSPI 200. CSPI 200-hosted endpoints can include instances in the same VCN or other VCNs (which may be customer VCNs or VCNs not belonging to the customer). Communication between CSPI 200-hosted endpoints can be performed over physical network 218. Compute instances can also communicate with endpoints not hosted by CSPI 200 (i.e., external to CSPI 200). Examples of such endpoints include endpoints within a customer's on-premises network or data center or public endpoints accessible over a public network such as the Internet. CSPI 200's communication with external endpoints can be performed over a public network (e.g., the Internet) (not shown in FIG. 2) or over a private network (not shown in FIG. 2) using various communication protocols.

[0143] The architecture of CSPI 200 shown in FIG. 2 is illustrative only and is not intended to be limiting. Variations, substitutions, and improvements are possible in alternative embodiments. For example, in some implementations, CSPI 200 may include more or fewer systems or components than those shown in FIG. 2, may combine two or more systems, or may have different system configurations or arrangements. The systems, subsystems, and other components shown in FIG. 2 may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective systems, using hardware, or a combination thereof. Software may be stored in a non-transitory storage medium (e.g., a memory device).

[0144] FIG. 4 illustrates connections between host machines and an NVD to provide I / O virtualization to support multitenancy, according to one embodiment. As shown in FIG. 4, host machine 402 runs hypervisor 404, which provides a virtualized environment. Host machine 402 runs two virtual machine instances: VM1 406, which belongs to customer / tenant #1, and VM2 408, which belongs to customer / tenant #2. Host machine 402 includes a physical NIC 410 connected to NVD 412 via link 414. Each compute instance is associated with a VNIC that NVD 412 runs. In the embodiment of FIG. 4, VM1 406 is associated with VNIC-VM1 420, and VM2 408 is associated with VNIC-VM2 422.

[0145] 4, NIC 410 includes two logical NICs: logical NIC A 416 and logical NIC B 418. Each virtual machine is configured to associate with and work together on its own logical NIC. For example, VM1 406 is associated with logical NIC A 416, and VM2 408 is associated with logical NIC B 418. Although host machine 402 includes only one physical NIC 410 shared by multiple tenants, because of the logical NICs, each tenant's virtual machine sees itself as having its own host machine and NIC.

[0146] In one embodiment, each logical NIC is assigned its own VLAN ID. Thus, a particular VLAN ID is assigned to logical NIC A 416 for tenant #1, and a separate VLAN ID is assigned to logical NIC B 418 for tenant #2. When a packet is sent from VM1 406, the hypervisor attaches a tag assigned to tenant #1 to the packet, and the packet is then sent from host machine 402 to NVD 412 via link 414. Similarly, when a packet is sent from VM2 408, the hypervisor attaches a tag assigned to tenant #2 to the packet, and the packet is then sent from host machine 402 to NVD 412 via link 414. Thus, a packet 424 sent from host machine 402 to NVD 412 has an associated tag 426 that identifies the particular tenant and associated VM. On the NVD, for a packet 424 received from a host machine 402, a tag 426 associated with the packet is used to determine whether the packet should be processed by VNIC-VM1 420 or VNIC-VM2 422. The packet is then processed by the corresponding VNIC. According to the configuration shown in Figure 4, each tenant's compute instance can be recognized as having its own host machine and NIC. The configuration shown in Figure 4 provides I / O virtualization to support multi-tenancy.

[0147] FIG. 5 is a simplified block diagram of a physical network 500, according to one embodiment. The embodiment shown in FIG. 5 is structured as a Clos network. A Clos network is a specific type of network topology designed to provide connection redundancy while maintaining high bisection bandwidth and maximum resource utilization. A Clos network is a type of non-blocking multistage or multi-layer switching network, with possible numbers of stages or layers, such as two, three, four, or five. The embodiment shown in FIG. 5 is a three-layer network, including layers 1, 2, and 3. TOR switch 504 represents a layer 0 switch in a Clos network. One or more NVDs are connected to a TOR switch. A layer 0 switch is also referred to as an edge device of the physical network. A layer 0 switch is connected to a layer 1 switch, also referred to as a leaf switch. In the embodiment shown in FIG. 5, a set of “n” layer 0 TOR switches are connected to a set of “n” layer 1 switches, collectively forming a pod. Each layer 0 switch in a pod is interconnected to all layer 1 switches in that pod, but there are no switch connections between pods. In one embodiment, two pods are referred to as a block. Each block is served by or connected to a set of “n” layer 2 switches (sometimes referred to as spine switches). There may be multiple blocks in a physical network topology. The layer 2 switches are then connected to “n” layer 3 switches (sometimes referred to as super spine switches). Packet communication across the physical network 500 is typically performed using one or more layer 3 communication protocols. High availability is typically achieved by having n-way redundancy at all layers of the physical network except the TOR layer. To enable scaling of the physical network, policies may be specified for pods and blocks to control the visibility of switches to each other in the physical network.

[0148] A Clos network is characterized by a fixed maximum hop count for reaching another tier-0 switch (or from an NVD connected to a tier-0 switch to another NVD connected to a tier-0 switch). For example, in a three-tier Clos network, a packet requires at most seven hops to reach another NVD from one NVD, with the source and destination NVDs connected to the leaf layers of the Clos network. Similarly, in a four-tier Clos network, a packet requires at most nine hops to reach another NVD from one NVD, with the source and destination NVDs connected to the leaf layers of the Clos network. Therefore, the Clos network architecture maintains consistent latency throughout the network, which is important for intra- and inter-datacenter communications. Clos topologies are horizontally scalable and cost-effective. The network's bandwidth / throughput capacity can be easily increased by adding more switches (e.g., more leaf and spine switches) to various tiers and by increasing the number of links between switches in adjacent tiers.

[0149] In one embodiment, each resource in CSPI is assigned a unique identifier called a Cloud Identifier (CID). This identifier is included as part of the resource's information and can be used when managing the resource, for example, through a console or API. An exemplary syntax for a CID is as follows: ocid1.<RESOURCE TYPE> . <realm>.[REGION][.FUTURE USE].<UNIQUE ID> where: ocid1 is a string indicating the version of the CID, RESOURCE TYPE is the type of resource (for example, instance, volume, VCN, subnet, user, group, etc.), REALM is the realm in which the resource resides (example values ​​are "c1" for the commercial realm, "c2" for the government cloud realm, or "c3" for the federal cloud realm, etc.; each realm may have its own domain name); REGION is the region in which the resource resides (this part may be blank if region is not applicable to the resource), FUTURE USE is reserved for future use, UNIQUE ID is the unique part of the ID (the format can vary depending on the type of resource or service).

[0150] Virtual Private Label Cloud (vPLC) Described herein are techniques for enabling cloud infrastructure used to create one or more virtual private clouds (also referred to herein as virtual private label clouds or "vPLCs") in a region provided by a cloud service provider (CSP) to provide CSP-supplied cloud services to CSP customers. A vPLC created according to the techniques described in this disclosure can be used for a variety of purposes. For example, in one use case, a vPLC can be created for a reseller who is a customer of the CSP and used by the reseller to provide one or more reseller-supplied cloud services to the reseller's customers. In another use case, a vPLC can be used as a virtual data center and associated with a realm different from the realm associated with the cloud infrastructure in the CSP-supplied region.

[0151] A cloud service provider (CSP) provides infrastructure (referred to as cloud service provider infrastructure, or CSPI, or CSP-delivered infrastructure) used to deliver one or more cloud services to the CSP's customers. A CSP's customers may include the CSP's and / or its reseller's direct customers. CSPI may include physical and virtual infrastructure components. Physical CSPI may include a physical infrastructure or underlying network, network virtualization devices, and interconnected high-performance compute resources, including racks, servers and host machines, memory resources, and network resources used to configure other resources. Virtual CSPI components may include components in an overlay network, such as virtual machines, virtual routers and gateways, and other virtual components.

[0152] A CSP may offer various types of cloud services. These may include various types of services such as Software as a Service (SaaS), Platform as a Service (PaaS), and Infrastructure as a Service (IaaS). In some embodiments, as described in this disclosure, a CSP may offer a "vPLC cloud service." A customer of a CSP may subscribe to one or more cloud services offered by the CSP. A customer may be any entity, such as an individual, an organization, or a business.

[0153] Infrastructure as a Service (IaaS) is a specific type of cloud computing service. In the IaaS model, a CSP provides infrastructure that customers can use to build their own customizable networks and deploy their resources. In this way, customer resources and networks are hosted in a distributed environment by the CSP-provided infrastructure. This differs from traditional computing, where customer resources and networks are hosted by customer-provided infrastructure. In one embodiment, a vPLC as a Service is a type of IaaS service in which infrastructure is allocated to create a virtual private label cloud for the customer. For example, if a CSP customer is a reseller, the CSP can provide a reseller-supplied cloud service using a vPLC created for the reseller.

[0154] When a customer registers or subscribes to a cloud service offered by a CSP, a tenancy (or account) is created for the customer. This account / tenancy allows the customer to access one or more registered cloud resources associated with the account / tenancy. Each tenancy is identified by a tenancy identifier (T_ID) that is assigned to the tenancy and uniquely identifies the tenancy. The tenancy identifier is generated when the tenancy is created and associated with the tenancy. For example, when a direct customer of a CSP registers for a service offered by a CSP, a tenancy is created for the customer and a tenancy identifier that uniquely identifies the tenancy is generated. When a reseller registers for a vPLC service, a tenancy is created for the reseller and a tenancy identifier that uniquely identifies the tenancy is generated. A tenancy may be a logical and secure compartment that contains all the resources and services used by the customer associated with the tenancy.

[0155] Resources in a CSP-provided infrastructure are typically identified using a unique resource identifier (resource ID or RID). Because these resources are provided in the cloud and used to provide cloud services, resource IDs are also referred to as cloud resource identifiers (or cloud IDs or CIDs). For example, in the cloud environment provided by Oracle Corporation, resources are identified using an "ocid" or "Oracle cloud identifier." In some embodiments, a resource ID is a globally unique identifier string that identifies a cloud resource. In some embodiments, a tenancy ID is a type of cloud resource ID (or cloud identifier (CID)).

[0156] A resource or cloud ID (CID) may be represented in the following format, as an example:

[0157] RID_version. <resourcetype> . <realm>.[REGION][.FUTURE USE].<UNIQUE ID> where: RID_version indicates the version of the resource ID.

[0158] "Resource type" identifies the particular kind of resource that the resource ID refers to. Examples of different resource types include an instance, a virtual network subnet, a tenancy, etc. For example, an "instance RID" identifies a compute instance.

[0159] "Realm" and "region" indicate the location (realm and region) of the resource. The "unique ID" is the unique part of the ID (e.g., a unique string to identify a particular resource). Thus, the tenancy ID can identify a unique ID within a CID where the resource type is tenancy.

[0160] When resources are created, a CID is generated for them. Virtual machines, accounts, tenancies, permission policies, virtual network subnets, access control lists (ACLs), and load balancers are examples of cloud resources and can be referenced by their respective CIDs. A tenancy is a logical container used to group other cloud resources that belong to the same cloud customer account.

[0161] When an entity becomes a customer of a CSP and subscribes to the vPLC service, a vPLC is created for the customer and associated with the customer's tenancy. For example, if the customer is a reseller, a vPLC is created for the reseller and associated with the reseller's tenancy. A vPLC created for a reseller is treated as a resource and is assigned a unique resource identifier called a vPLC identifier (vPLC ID), which is associated with the reseller's tenancy. As described herein, for a vPLC created for a reseller, the vPLC ID for that vPLC is used to identify all resources and requests associated with the vPLC. Each vPLC ID uniquely identifies a vPLC.

[0162] For example, in one embodiment, all resources assigned to a vPLC are tagged with the vPLC identifier of that vPLC. In one embodiment, a vPLC ID field is inserted into the representation of the CID described above. If a resource is associated with a vPLC, the vPLC ID corresponding to that vPLC may be provided in this CID field; if a resource is not associated with a vPLC, the field may be left blank.

[0163] For the purposes of this disclosure, the following terminology will be used for the sake of clarity: CSP.Cn (or just Cn) identifies a direct (non-reseller) customer of the CSP. For example, "CSP.C1" represents the CSP's non-reseller direct customer C1, and "CSP.C2" represents the CSP's non-reseller direct customer C2.

[0164] CSP.Rn (or just Rn) identifies a reseller customer of the CSP. For example, "CSP.R1" represents CSP's reseller customer R1, and "CSP.R2" represents CSP's reseller customer R2.

[0165] In Rn.Cm, a reseller may have its own customers. This identifies a particular customer of a particular reseller. For example, "R1.C1" represents customer C1 of reseller R1, and "R2.C1" represents customer C1 of reseller R2.

[0166] T.Cn identifies the tenancy of a CSP's non-reseller customer. For example, "T.C1" represents the tenancy of non-reseller direct customer C1.

[0167] T.Rn identifies the reseller's tenancy. For example, "T.R1" represents the tenancy of reseller R1.

[0168] In T.Rn.Cm, each customer of a reseller has their own tenancy. This identifies the tenancy associated with a particular customer of a particular reseller. For example, "T.R1.C1" represents the tenancy associated with reseller R1's customer C1, and "T.R2.C1" represents the tenancy associated with reseller R2's customer C1.

[0169] T_ID.T.Cn identifies the tenancy identifier that identifies the tenancy of a CSP's non-reseller customer. For example, "T_ID.T.C1" represents the tenancy ID of non-reseller direct customer C1's tenancy.

[0170] T_ID.T.Rn identifies the tenancy identifier of the reseller's tenancy. For example, "T_ID.T.R1" represents the tenancy identifier of reseller R1's tenancy.

[0171] T_ID.Rn.Cm identifies the tenancy identifier of the tenancy associated with a particular customer of a particular reseller. For example, "T_ID.T.R1.C1" represents the tenancy identifier of the tenancy associated with customer C1 of reseller R1, and "T_ID.T.R2.C1" represents the tenancy identifier of the tenancy associated with customer C1 of reseller R2.

[0172] vPLC.Rn identifies a vPLC created for reseller Rn. For example, "vPLC.R1" identifies the tenancy created for and associated with reseller R1. This identifies a specific vPLC created for a reseller and associated with the reseller's tenancy.

[0173] vPLC_ID.vPLC.Rn identifies the vPLC identifier of a vPLC generated for a particular reseller. For example, "vPLC_ID.vPLC.R1" represents the vPLC identifier of a vPLC generated for reseller R1.

[0174] As described herein, a vPLC is created for a reseller, who is a customer of a CSP. The vPLC is associated with the reseller's tenancy. In some embodiments, a vPLC ID identifying a vPLC may be associated with a tenancy ID identifying the reseller's tenancy. This allows for the identification of a corresponding vPLC and the reseller for which the vPLC was created (including the reseller's tenancy and tenancy identifier), given a vPLC ID.

[0175] CSPI or CSP-provided infrastructure may be organized as realms, regions, and data centers. A region represents a local geographic area that includes one or more connected data centers. Regions are independent of other regions and may be separated by large distances, for example, across countries or continents. A set of application programming interfaces (APIs) is provided to the CSP-provided infrastructure in a region, and these APIs are used to perform operations using the infrastructure in the region. Operations that may be performed using the infrastructure in a region may include creating resources of the infrastructure in the region, accessing resources of the infrastructure in the region, performing operations involving resources of the infrastructure in the region, deleting resources from the infrastructure in the region, and performing other operations involving the infrastructure in the region.

[0176] A set of APIs is region-specific. Thus, a CSP-provided infrastructure in a region is characterized by a set of APIs usable with the infrastructure in that region. A CSP-provided infrastructure in a different region has a different set of APIs provided for the different region. Thus, for purposes of this application, the phrase "CSP-provided infrastructure in a region" or "CSP-provided infrastructure in a region" means that the infrastructure is characterized by a specific set of APIs provided for the region. Thus, a first set of APIs may be provided for a first region, a second set of APIs may be provided for a second region different from the first region, a third set of APIs may be provided for a third region different from the first and second regions, and so on. In this way, a specific set of APIs identifies a CSP-provided infrastructure in a particular region. In other words, a CSP-provided infrastructure in two separate regions would have two separate sets of APIs. In this manner, operations involving regional infrastructure are performed by characterizing a region with infrastructure in that region and a set of APIs associated with that region. CSP-provided infrastructure in a region is also referred to as "regional infrastructure" or "region-specific infrastructure."

[0177] For example, a CSP may provide a first infrastructure in Seattle and a second infrastructure in Portland. For the first infrastructure in Seattle, a first set of APIs may be provided, which may be characterized by the following format:

[0178] <service_identifier> .us-seattle.oci.oraclecloud.com For the second infrastructure in Portland, a second set of APIs may be provided, which may be characterized by the following format:

[0179] <service_identifier> .us-portland.oci.oraclecloud.com Note that Seattle and Portland are used here only as examples. The geographic locations of the regions can vary. For example, there can be more than one region in the same geographic city. For example, there can be a first region in a first building in a city and a second region in a second, separate building in the same city. As yet another example, there can be a first region on one floor of a building and a second region on another floor of the building. As yet another example, for infrastructure provided by a CSP on a floor, a first portion of the infrastructure can reside in a first region associated with a first set of APIs, and a second portion of the infrastructure can reside in a second region associated with a second set of APIs that is different from the first set of APIs.

[0180] A CSP's infrastructure in a region may be organized as one or more data centers. For example, in a particular region where a CSP may have two buildings each hosting infrastructure used to provide cloud services, the infrastructure in the first building may be organized and referred to as a first data center, and the infrastructure in the second building may be organized and referred to as a second data center. As another example, in a particular region where a CSP may have a single building with multiple floors each hosting infrastructure used to provide cloud services, the infrastructure on the first floor may be organized and referred to as a first data center, and the infrastructure on the second floor may be organized and referred to as a second data center. As yet another example, in a particular region where a CSP may have infrastructure used to provide cloud services, a first portion of the infrastructure may be organized and referred to as a first data center, and a second, separate portion of the infrastructure may be organized and referred to as a second data center.

[0181] A region may have one or more data centers. In some embodiments, all regional infrastructure is contained in a single data center, while in other embodiments, the infrastructure in a region may be organized as multiple data centers. Each data center may include infrastructure resources provided by a CSP, such as compute, storage, and networking resources. In a region, the data centers in the region may be organized as one or more availability domains (ADs). Availability domains are isolated from each other, fault-tolerant, and highly unlikely to fail simultaneously. ADs are configured so that a failure in one AD in a region is unlikely to affect the availability of other ADs in the same region.

[0182] A realm represents a logical collection of one or more regions. A realm may contain one or more regions. Realms are typically isolated from one another and do not share data. Each realm has its own identity and trust profile (e.g., password, authentication information, etc.). The realm's identity and trust profile are configured to prevent inter-realm communication. In this manner, each realm represents an isolated domain. For example, one realm may be created for government entities that are customers of a CSP, another realm may be created for commercial customers of the CSP, yet another realm may be created for private customers of the CSP, and so on. With respect to regions within a realm, regional infrastructures can communicate with each other, while regional infrastructure associated with a first realm cannot communicate with regional infrastructure associated with a second, different realm.

[0183] Resellers of cloud services can be entities such as businesses (eg, telecommunications companies), system integrators, government agencies, IT departments of large corporations, schools, and the like.

[0184] FIG. 6 is a simplified block diagram of a distributed environment 600 including a CSP-provided infrastructure in a region, illustrating an example of a virtual private label cloud (vPLC) hosted by the CSP-provided infrastructure, according to one embodiment. The distributed environment 600 illustrated in FIG. 6 is merely an example and is not intended to unduly limit the scope of the claimed embodiments. Many variations, alternatives, and improvements are possible. For example, in some implementations, the distributed environment 600 may include more or fewer systems or components than those illustrated in FIG. 6, may combine two or more systems, or may have different system configurations or arrangements. The systems, subsystems, and other components illustrated in FIG. 6 may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective systems, using hardware, or a combination thereof. Software may be stored in a non-transitory storage medium (e.g., a memory device).

[0185] Figure 6 illustrates a CSP-provided infrastructure 601 in a region. The infrastructure 601 may be organized as one or more data centers. The infrastructure 601 may include memory or storage resources 622, compute resources 620, networking resources (not shown), and other physical and logical resources provided by the CSP. The storage resources 622 may include one or more volatile or non-volatile memory resources. The compute resources 620 may include one or more racks, each with one or more servers hosting one or more operating systems.

[0186] A CSP can use CSP provisioning infrastructure 601 to offer one or more CSP-supplied cloud services to one or more of its direct customers and to create one or more vPLCs. In the example shown in Figure 6, two vPLCs (vPLC.R1 604 and vPLC.R2 606) have been created for resellers R1 and R2, respectively. vPLC.R1 604 is used to offer one or more reseller R1-supplied and R1-branded cloud services to one or more customers 642 of R1. vPLC.R2 606 is also used to offer one or more reseller R2-supplied and R2-branded cloud services to one or more customers 644 of R2.

[0187] As shown, infrastructure 601 can be communicatively coupled via communications network 652 to computing devices used by users associated with direct customer 640 and users associated with reseller customers (e.g., users associated with reseller R1's customer 642 and users associated with reseller R2's customer 644). Communications network 652 can be of various types and can include one or more communications networks. Examples of communications network 652 include, but are not limited to, the Internet, a wide area network (WAN), a local area network (LAN), an Ethernet network, a public or private network, a wired network, a wireless network, or the like, as well as combinations thereof. Additionally, various communications protocols can be used to facilitate communications, including both wired and wireless protocols, such as the IEEE 802.XX suite of protocols, TCP / IP, IPX, SAN, AppleTalk®, Bluetooth®, and other protocols. In general, communications network 652 can include any infrastructure that facilitates communications with infrastructure 601.

[0188] 6, infrastructure 601 is divided into three securely isolated portions: (1) a first portion 602 used for CSP provisioning and providing CSP-branded cloud services to one or more direct (or non-reseller) customers 640 of the CSP, (2) a second portion allocated to vPLC.R1 604 generated for reseller R1, and (3) a third portion allocated to vPLC.R2 606 generated for reseller R2. In one embodiment, these three portions are securely isolated from one another.

[0189] The partitioning of resources includes partitioning of storage resources 622 and compute resources 620. For example, as shown in Figure 6, storage resources 622 include a storage portion 622a included in first portion 602 used for providing CSP-supplied cloud services to the CSP's direct customers. Storage resources 622 also include a storage portion 622b allocated to vPLC.R1 604 and a storage portion 622c allocated to vPLC.R2 606. The storage or memory portion allocated to a vPLC may include one or more volatile and / or non-volatile memory resources.

[0190] 6, compute resources 620 are also partitioned. Compute resources 620 include a compute portion 620a in first portion 602 that is used to provide CSP-supplied cloud services to the CSP's direct customers. Compute resources 620 also include a portion 620b allocated to vPLC.R1 604 and a portion 620c allocated to vPLC.R2 606.

[0191] As described above, a set of APIs is provided for accessing and performing functions related to the CSP-provided infrastructure in a region. These APIs are accessible to users associated with the CSP's direct customers 640 by using a set of endpoints 610-1 provided by the CSP. For example, a user may use one or more of the endpoints 610-1 to access and / or use compute resources 620a or memory resources 622a allocated to the infrastructure used to provide CSP-provided cloud services to the CSP's customers. The endpoints in 610-1 may represent URLs with associated fully qualified domain names (FQDNs) used to access cloud services provided by the infrastructure 601. Examples of these endpoints include: createVM.compute.us-seattle.oci.oraclecloud.com (for creating VMs) deleteVM.compute.us-seattle.oci.oraclecloud.com (for erasing VMs) storeObject.storage.us-seattle.oci.oraclecloud.com (for data storage) etc. Users associated with the CSP's direct customers 640 and connected to the infrastructure 601 via a communications network can use one or more of the endpoints 610-1.

[0192] In one embodiment, a set of vPLC-specific endpoints is created for each vPLC. For example, as shown in FIG. 6, a set of endpoints 612-1 is created for vPLC.R1 and a set of endpoints 614-1 is created for vPLC.R2. A user associated with a customer 642 of reseller R1 can use endpoint 612-1 to access resources associated with vPLC.R1 and perform one or more functions or operations involving the resources assigned to vPLC.R1. A user associated with a customer 644 of reseller R2 can use endpoint 614-1 to access resources associated with vPLC.R2 and perform one or more functions or operations involving the resources assigned to vPLC.R2.

[0193] In one embodiment, a set of vPLC-specific endpoints for a vPLC is generated based on the endpoints that the CSP provides to its direct customers. For example, in FIG. 6, endpoint 612-1 for vPLC.R1 is generated based on endpoint 610-1. The endpoints generated for a vPLC may include an identifier that specifically identifies the vPLC for which the endpoint was generated. For example, given the exemplary endpoints that the CSP provides to its direct customers, endpoint 612-1 for vPLC.R1 is: createVM.compute.vplcR1.us-seattle.oci.oraclecloud.com (for creating a VM) deleteVM.compute.vplcR1.us-seattle.oci.oraclecloud.com (for erasing VMs) storeObject.storage.vplcR1.us-seattle.oci.oraclecloud.com (for data storage) etc., where "vplcR1" uniquely identifies the vPLC corresponding to the endpoint.

[0194] Also, an endpoint 614-1 for vPLC.R2 may be generated based on the endpoint 610-1. For example, the endpoint 614-1 for vPLC.R2 createVM.compute.vplcR2.us-seattle.oci.oraclecloud.com (for VM generation) deleteVM.compute.vplcR2.us-seattle.oci.oraclecloud.com (for erasing VMs) storeObject.storage.vplcR2.us-seattle.oci.oraclecloud.com (for data storage) etc., where "vplcR2" uniquely identifies the vPLC corresponding to the endpoint.

[0195] Because the endpoints associated with a vPLC include an identifier that identifies the particular vPLC, when the infrastructure 601 receives an API call, it can analyze the endpoint call to determine whether the call is for a direct customer of the CSP or for a reseller. Furthermore, because the endpoints use unique identifiers for different vPLCs, it is also possible for the endpoint call to identify the particular vPLC and associated reseller. Thus, based on the analysis of the endpoint call, the endpoint request can be appropriately routed and processed.

[0196] The CSP also provides a console 610-2 for use by its direct customers 640. Users associated with the CSP's direct customers 640 can use the console 610-2 to configure, access, and manage the first portion 602 of the CSP-provided regional infrastructure. In some embodiments, the console 610-2 may provide a set of web-based graphical user interfaces (GUIs) that can be used to access and manage the CSP-provided cloud infrastructure in the region. In some embodiments, the console is a web-based application.

[0197] Also, in one embodiment, a console is provided for each vPLC. For example, as shown in FIG. 6 , a console 612-2 is provided for vPLC.R1 and a console 614-2 is provided for vPLC.R2. A user associated with a customer 642 of reseller R1 can use the console 612-2 provided for vPLC.R1 to configure, access, and manage a second portion of the CSP-provided regional infrastructure assigned to vPLC.R1 604 created for reseller R1. A user associated with a customer 644 of reseller R2 can use the console 614-2 provided for vPLC.R2 to configure, access, and manage a third portion of the CSP-provided regional infrastructure assigned to vPLC.R2 606 created for reseller R2.

[0198] In some embodiments, a reseller may register with more than one vPLC, while each vPLC may still have a set of endpoints and a console. For example, if a reseller registers with two vPLCs (vPLC1 and vPLC2), vPLC1 may have a set of endpoints (e.g., 612-1) and a console (e.g., 612-2), and vPLC2 may have another set of endpoints (e.g., 614-1) and a console (e.g., 614-2). Both vPLC1 (e.g., 604) and vPLC2 (606) may be associated with the same reseller.

[0199] The infrastructure 601 may include networking resources that enable communication with the infrastructure 601. These network resources may include, for example, a network gateway 628 that enables communication of traffic from the CSP-provided regional infrastructure 601 to other destinations, which may be communicatively coupled to the infrastructure 601 via a public (e.g., the Internet) or private network, and may reside in an on-premise network, another cloud network, etc. The gateway 628 may be implemented as a physical network device (e.g., a router), as a logical networking entity (e.g., a virtual router), or as some combination of physical and logical entities.

[0200] The infrastructure 601 may include a control plane (CP) 624 and a management plane (MP) 626 provided by the CSP. In some embodiments, the CP 624 may be responsible for providing management, deployment, and orchestration functions for the cloud environment provided by the CSP. For example, the CP may receive requests from the CSP's direct customers, reseller R1's customers, and reseller R2's customers through endpoints and consoles, collaborate with other components in the infrastructure, such as resource managers and management planes, determine whether the requests can be fulfilled, and respond accordingly to the CSP's direct customers, reseller R1's customers, and reseller R2's customers. In other words, the CSP's CP 624 may be responsible for performing vPLC-related functions. In some embodiments, the CSP may also provide a vPLC-specific control plane configured specifically to perform vPLC operations. In some embodiments, there may be one vPLC control plane that can perform processing for multiple vPLCs. In other embodiments, each vPLC may have its own CP.

[0201] In some embodiments, the MP 626 may be a collection of software components responsible for provisioning all the processes and workflows required for a requested operation. For example, for a service request to access a cloud service, the MP may provision a workflow that includes checking the identity of the requester and checking DNS records to obtain an IP address to access the service backend. The MP ensures that the various tasks are performed correctly and in the correct order by the data plane.

[0202] In one embodiment, compute resources 620 may include one or more racks each including one or more servers hosting one or more operating systems. The division of computer resources 620 into resources 620a allocated to infrastructure for serving the CSP's direct customers 640 and resources 620b and 620c allocated to vPLCs may be performed along various levels or boundaries. The division of compute resources at a particular level may be referred to herein as a segmentation level. In one embodiment, the division may be performed at the rack level (i.e., rack-level segmentation), with one or more individual racks assigned to 620, 620b, and 620c. In this embodiment, a rack is exclusively assigned to a particular vPLC and is not used by other vPLCs or by the CSP to provide CSP-delivered services to its direct customers 640.

[0203] In another embodiment, the partitioning may be performed at the server level (i.e., segmented at the server level), with one or more individual servers in the same or different racks assigned to 620, 620b, and 620c. In this embodiment, a server is assigned exclusively to a particular vPLC and is not used by other vPLCs or by a CSP to provide CSP-delivered services to its direct customers 640. Because a rack may have multiple servers, one server on the rack may be assigned to 620a, another server to 620b, and a third server to 620c. Thus, a rack may be shared between two vPLCs or may be shared between a vPLC and the infrastructure used by the CSP to provide CSP-delivered services to its direct customers 640.

[0204] In some embodiments, the partitioning is performed at the hypervisor level (i.e., segmented at the hypervisor level), and one or more individual hypervisors on the same server or different servers on the same rack or different racks may be assigned to 620, 620b, and 620c. In this embodiment, a hypervisor is assigned exclusively to a particular vPLC and is not used by other vPLCs or by a CSP to provide CSP-delivered services to its direct customers 640. Because a server may host multiple hypervisors, one hypervisor on a server may be assigned to 620a, another hypervisor on the same server may be assigned to 620b, and a third hypervisor on the same server may be assigned to 620c. Thus, a server may be shared between two vPLCs or may be shared between a vPLC and the infrastructure used by a CSP to provide CSP-delivered services to its direct customers 640.

[0205] A hypervisor on a server on a rack can manage and run multiple virtual machines (VMs). In some embodiments, the partitioning is performed at the VM level (i.e., segmentation at the VM level), and VMs running with the same hypervisor can be assigned to 620, 620b, and 620c. In this embodiment, a VM is assigned exclusively to a particular vPLC and is not used by other vPLCs or by the CSP to provide CSP-delivered services to its direct customers 640. Because a hypervisor can support multiple VMs, one VM may be assigned to 620a, another to 620b, and a third VM on the same hypervisor may be assigned to 620c. Thus, a hypervisor may be shared between two vPLCs, or it may be shared between a vPLC and the infrastructure used by the CSP to provide CSP-delivered services to its direct customers 640.

[0206] In still other embodiments, a combination of the above partitioning techniques may be used. Regardless of the partitioning technique used, the partitioning is performed securely and safely so that traffic and storage destined for a particular vPLC for a particular reseller is not visible to or accessible by other resellers, their respective customers, or other direct customers of the CSP. Additionally, traffic and storage destined for a particular customer of a reseller is not visible to or accessible by other customers of that reseller.

[0207] Figure 7 illustrates the relationships between a CSP, its direct customers, its reseller customers, and their associated users, and each of the reseller's customers and their associated users, according to one embodiment. As shown in Figure 7, CSP 710 is at the top or root of the hierarchy. CSP 710 may provide one or more cloud services to which the CSP's customers may subscribe. In Figure 7, the CSP's customers are represented by 702. A tenancy, identified by a tenancy ID, is created for each registered customer of CSP 710.

[0208] Customers 702 of a CSP 710 can include direct or non-reseller customers and reseller customers. A reseller customer is an entity that subscribes to a vPLC service offered by a CSP, and a vPLC generated by the reseller as a result of subscription to the vPLC service is used to provide reseller-supplied and reseller-branded cloud services to the reseller's customer. A reseller's customer may be different from the CSP's customers. A direct or non-reseller customer of a CSP is an entity that subscribes to one or more CSP-supplied cloud services, but does not use the CSP's infrastructure to provide any cloud services of its own to its customers.

[0209] In the example shown in Figure 7, the direct or non-reseller customers of the CSP 710 include customer C1 726 with an associated tenancy T.CSP.C1 and customer C2 728 with an associated tenancy T.CSP.C2. There may be multiple such direct customers of the CSP 710. One or more users may be each direct customer of the CSP, and these users may use one or more of the CSP-supplied cloud services to which the corresponding direct customer subscribes. For example, users associated with a customer of the CSP may be employees or agents of the customer.

[0210] 7, reseller customers of a CSP 710 include reseller R1 720 with associated tenancy T.CSP.R1 and reseller R2 722 with associated tenancy T.CSP.R2. There may be multiple such reseller customers of a CSP 710. One or more users (not shown in FIG. 7) may be each reseller customer of the CSP.

[0211] Here, a reseller may have its own customers who subscribe to one or more reseller supply and reseller brand cloud services, which are provided to the reseller using a vPLC generated by the CSP. In Figure 7, customers of reseller R1 720 are indicated using reference number 704, and customers of reseller R2 722 are indicated using reference number 706.

[0212] A reseller's customers may each have their own tenancy and associated set of users. For example, as shown in FIG. 7 , reseller R1 may have multiple customers, including customer R1.C1 740 with tenancy T.R1.C1 and associated users, customer R1.C2 742 with tenancy T.R1.C2 and associated users, and others. Reseller R1 may supply a set of R1-supplied and branded cloud services, and R1's customers can register for one or more of the R1-supplied services. The set of one or more R1-supplied services that R1.C1 740 subscribes to may be the same as or different from the set of one or more R1-supplied services that R1.C2 742 subscribes to. The cloud services supplied by R1 722 may be different from the cloud services supplied by reseller R2 722 and the cloud services supplied by CSP 710.

[0213] Reseller R2 722 may have multiple customers subscribing to R2-supplied and branded cloud services provided using the vPLC created for R2. As shown in FIG. 7 , R2's 722 customers include customer R2.C1 744 with tenancy T.R2.C1 and associated users, customer R2.C2 746 with tenancy T.R2.C2 and associated users, and others. The cloud services supplied by R2 722 may be different from the cloud services supplied by reseller R1 720 and the cloud services supplied by CSP 710. R2's customers may subscribe to one or more of the R2-supplied cloud services. The set of one or more R2-supplied services that R2.C1 744 subscribes to may be the same as or different from the set of one or more R2-supplied services that R2.C2 746 subscribes to.

[0214] As shown in Figure 7, there are two levels of tenancies that come into play when a reseller uses vPLC to sell reseller-supplied cloud services to their customers. The first level of tenancies corresponds to tenancies associated with the CSP's 710 customers (including direct customers, i.e., non-reseller customers, and reseller customers). These tenancies may also be referred to as "CSP's Customer" tenancies (or CoCT). In Figure 7, the CoCT includes tenancies associated with reseller R1 720, reseller R2 722, direct customer C1 726, and direct customer C2 728. The second level of tenancies includes tenancies associated with the reseller's customers. These tenancies may also be referred to as reseller's customer tenancies (or CoRT). In Figure 7, CoRT includes tenancies associated with R1.C1 740, R1.C2 742, R2.C1 744, R2.C2 746, etc.

[0215] 8 illustrates the relationships between resources allocated to the tenancies of the CSP's direct customers, the CSP's reseller tenancies, and the tenancies of each reseller's customers, according to one embodiment. Referring to FIG. 8, in one embodiment, a CSP-provided infrastructure (e.g., data centers) 810 in a region may be divided into multiple portions that are securely isolated for the CSP's resellers, such as resources allocated to a vPLC.R1 820 created for reseller R1 and associated with reseller R1's tenancy (T.R1), and resources allocated to a vPLC.R2 822 created for reseller R2 and associated with reseller R2's tenancy (T.R2). The partitions of resources may also include partitions that are securely isolated for the CSP's non-reseller direct customers C1 and C2, such as resources 826 allocated to non-reseller customer CSP.C1's tenancy (T.CSP.C1) and resources 828 allocated to non-reseller customer CSP.C2's tenancy (T.CSP.C2). The partitions of CSP-provided regional infrastructure resources allocated to reseller R1's tenancy 820, reseller R2's tenancy 822, customer CSP.C1's tenancy, and customer CSP.C2's tenancy may be referred to as first-level resource partitions.

[0216] Each securely isolated portion of resources associated with a first-level resource partition or reseller's tenancy (i.e., the R1 tenancy and the R2 tenancy) may be assigned or tagged with a vPLC ID (i.e., resource ID), such as vPLC_ID.vPLC.R1 820 and vPLC_ID.vPLC.R2 822. Each of these CSP's non-reseller direct customer partitions may also be tagged with a resource ID. Each of the CSP's non-reseller direct customer tenancies is accessible to respective users, who can use one or more of the CSP-supplied cloud services to which each corresponding direct customer subscribes.

[0217] Within each first-level resource partition (or portion) of a reseller may be additional second-level resource partitions that are assigned to the tenancies of the reseller's customers. For example, the tenancy (T.R1.C1) associated with reseller R1's customer C1 may be assigned securely isolated resource partition (vPLC.R1.C1) 840, and the tenancy (T.R1.C2) associated with reseller R1's customer C2 may be assigned securely isolated resource partition (vPLC.R1.C2) 842. Isolation partition vPLC.R1.C1 may be tagged with a customer resource ID that identifies the resources assigned to T.R1.C1, and isolation partition vPLC.R1.C2 may be tagged with a customer resource ID that identifies the resources assigned to T.R1.C1.

[0218] Similarly, a tenancy (T.R2.C1) associated with reseller R2's customer C1 may be assigned securely isolated resource partition (vPLC.R2.C1) 844, and a tenancy (T.R2.C2) associated with reseller R2's customer C2 may be assigned securely isolated resource partition (vPLC.R2.C2) 846. The isolated partition vPLC.R2.C1 may be tagged with a customer resource ID that identifies the resources assigned to T.R2.C1, and the isolated partition vPLC.R2.C2 may be tagged with a customer resource ID that identifies the resources assigned to T.R2.C1. Each tenancy associated with a reseller's customer (e.g., R1.C1 840, R1.C2 842, R2.C1 844, or R2.C2 846) is accessible to one or more users associated with the customer who can use one or more of the reseller-supplied cloud services to which the corresponding customer subscribes.

[0219] 9A and 9B illustrate a mechanism for storing information in the tenancies of a CSP's individual reseller customers, according to one embodiment. In some embodiments, the vPLC IDs and tenancy IDs (sometimes referred to as customer tenancy IDs or CT IDs) of tenancies associated with a reseller's customers may be associated in two ways: 1) a distributed scheme (or record space per vPLC) as shown in FIG. 9A; and 7) a centralized scheme (or vPLC ID as the partition key) as shown in FIG. 9B. While FIGS. 9A and 9B show the information stored in a table format, in some embodiments the information may be stored in different formats, such as an array, a key-value store, XML, JSON, etc.

[0220] FIG. 9A illustrates a first approach, per-vPLC record space. In some embodiments, the per-vPLC record space may be an approach in which tenancy-related information for each reseller is stored in a vPLC-specific table under a vPLC ID. In other words, the information may be stored in a distributed manner. Each table is assigned a vPLC ID. In each table, one row may be designated for a customer of the reseller, another row may be designated for another customer of the reseller, and so on. For example, in FIG. 9A , there are two tables for storing information such as identification information (e.g., user logins, associated authentication information, and certificates) or billing information (e.g., resource usage, pricing). Table 910 is assigned the vPLC ID (vPLC_ID.vPLC.R1) of vPLC.R1 associated with reseller R1, and table 920 is assigned the vPLC ID (vPLC_ID.vPLC.R2) of vPLC.R2 associated with reseller R2. In table 910, there are three records for three customers C1, C2, and C3 of reseller R1. One record / row may include the tenancy ID (T_ID.T.R1.C1) of the tenancy associated with reseller R1's customer C1 and related information for customer C1 (e.g., identification information, billing, etc.). A second record / row may include the tenancy ID (T_ID.T.R1.C2) of the tenancy associated with reseller R1's customer C2 and related information for customer C2. A third record / row may include the tenancy ID (T_ID.T.R1.C3) of the tenancy associated with reseller R1's customer C3 and related information for customer C3.

[0221] The same structure can be used for table 920, storing three records for reseller R2's three customers C1, C2, and C3. One record / row may include the tenancy ID (T_ID.T.R2.C1) of the tenancy associated with reseller R2's customer C1 and related information for customer C1 (e.g., identification information, billing, etc.). A second record / row may include the tenancy ID (T_ID.T.R2.C2) of the tenancy associated with reseller R2's customer C2 and related information for customer C2. A third record / row may include the tenancy ID (T_ID.T.R2.C3) of the tenancy associated with reseller R2's customer C3 and related information for customer C3.

[0222] FIG. 9B illustrates a second approach using the vPLC ID as a partition key. In FIG. 9B, the second approach stores information in a table with two columns: a customer tenancy ID and a vPLC ID. Searches are performed using the vPLC ID as a partition key. In other words, information may be centrally stored in a table. For example, in FIG. 9B, a reseller R1 (vPLC_ID.vPLC.R1) associated with a vPLC ID has three customers C1, C2, and C3, and is associated with tenancy IDs T_ID.T.R1.C1, T_ID.T.R1.C2, and T_ID.T.R1.C3 that identify the tenancies of the three customers. In the vPLC ID column, the first three rows of table 930 may be marked as vPLC_ID.vPLC.R1, which is the vPLC ID of vPLC.R1 generated for reseller R1. The customer tenancy ID column in row 1 may be marked as the tenancy ID of the tenancy associated with reseller R1's customer C1 (T_ID.T.R1.C1), while the tenancy ID column in row 2 may be marked as the tenancy ID of the tenancy associated with reseller R1's customer C2 (T_ID.T.R1.C2), and the tenancy ID column in row 3 may be marked as the tenancy ID of the tenancy associated with reseller R1's customer C3 (T_ID.T.R1.C3). The information stored in the first three rows belongs to reseller R1's customers C1, C2, and C3.

[0223] The same structure can be used to store records for reseller R2's customers C1, C2, and C3 in the same table 930, e.g., the fourth, fifth, and sixth rows. For example, in the vPLC ID column, the last three rows of table 930 may be marked as vPLC_ID.vPLC.R2, which is the vPLC ID of vPLC.R2 generated for reseller R2. The customer tenancy ID column in row 4 may be marked as the tenancy ID (T_ID.T.R2.C1) of the tenancy associated with reseller R2's customer C1. Meanwhile, the tenancy ID column in row 5 may be marked as the tenancy ID (T_ID.T.R2.C2) of the tenancy associated with reseller R2's customer C2, and the tenancy ID column in row 6 may be marked as the tenancy ID (T_ID.T.R2.C3) of the tenancy associated with reseller R2's customer C3. The information stored in the last three rows belongs to reseller R2's customers C1, C2, and C3.

[0224] FIG. 10 is a flowchart illustrating an example of a vPLC configuration process, according to an embodiment. The process illustrated in FIG. 10 may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective systems, using hardware, or a combination thereof. The software may be stored in a non-transitory storage medium (e.g., a memory device). The method presented in FIG. 10 and described below is exemplary and not intended to be limiting. While FIG. 10 depicts various process steps occurring in a particular order or sequence, this is not intended to be limiting. In an alternative embodiment, the process may be performed in some different order, or some steps may be performed in parallel. Of course, in alternative embodiments, the process illustrated in FIG. 10 may include more or fewer steps than those depicted in FIG. 10.

[0225] For example, in one embodiment, the process illustrated in Figure 10 may be performed by a CSP, and some steps may be performed by components within a CSP-provided regional infrastructure (e.g., a control plane (CP), a resource manager (RM), and a management plane (MP)), or by one or more CSP-provided cloud services, such as an endpoint management service (EMS), an identity management service, etc.

[0226] As previously described, a vPLC may be created when a reseller registers for a CSP-provided vPLC service. In step 1002, the vPLC may be created by receiving a signal. In one embodiment, the signal may be a request received when a reseller registers for a CSP-provided vPLC service. The request may be an agreement between the CSP and the reseller, listing the virtual model (e.g., nested or segmented model), database, resource placement, customization details, etc. In another embodiment, a signal may be received from some component indicating that a vPLC is being created.

[0227] Step 1004 may include receiving configuration information for the vPLC to be created. In some embodiments, the configuration information may include information about the tenancy in which the vPLC is created (e.g., a tenancy identifier) ​​and information for identifying the realm associated with the vPLC. As described above, a tenancy is created for a reseller, and a tenancy identifier that uniquely identifies the tenancy is generated for the reseller associated with the vPLC.

[0228] As mentioned above, in use cases where a vPLC is used as a virtual data center and may be associated with a different realm than the realm associated with the cloud infrastructure in the CSP-provided region, the identity and trust profile information (e.g., passwords, credentials, etc.) of the different realm needs to be associated with the vPLC. The association of the identity and trust profile information of the different realm may be achieved by creating a mapping (or "shadow copy") of the identity and trust profile information in the vPLC. Further details are discussed in FIG. 18.

[0229] Step 1006, which includes steps 1008-1018, may involve performing a process to create a vPLC for a reseller R. Step 1008 may involve generating a vPLC identifier for the vPLC. The vPLC created for the reseller is treated as a resource and assigned a unique resource identifier, called a vPLC identifier (vPLC ID), which is associated with the reseller's tenancy.

[0230] In step 1010, a set of resources of a CSP-provided infrastructure in a region may be allocated to the vPLC. The set of resources allocated to the vPLC may be selected based on an agreement between the reseller and the CSP. For example, reseller R may subscribe to one or more cloud services offered by the CSP, such as compute services and object storage services. Thus, the set of resources may include compute resources and object storage resources.

[0231] In step 1012, a namespace may be reserved for the vPLC. The CSP's resource allocation service (or resource manager (RM)) may create and reserve a namespace that groups various types of resources that may support the vPLC. A namespace may represent a logical grouping or virtual boundary that helps organize and manage resources and prevent naming conflicts when multiple instances of similar resources are created. Thus, a namespace provides a structured way to organize and group related resources. In some embodiments, namespace 0 is reserved for the CSP. Namespaces 1 and beyond are used for the vPLC. In some embodiments, a wide range of Internet Protocol (IP) addresses may be reserved for the vPLC and associated with the namespace for traffic separation purposes.

[0232] In step 1014, a console may be generated for the vPLC that allows reseller R, the reseller's customers, and users associated with the reseller's customers to configure, access, and manage resources deployed in the vPLC, as well as visualize, access, and interact with reseller-supplied cloud services. Reseller R may also configure the console to provide different, customized experiences for different customers of the reseller.

[0233] In step 1016, a set of endpoints may be generated for the vPLC. In some embodiments, some of the set of endpoints may be default endpoints to be used (e.g., identity endpoints), and some endpoints may be determined based on the cloud services that the reseller registers. For example, the reseller may register for one or more CSP-supplied cloud services, such as compute, storage, virtual cloud network (VCN), database, etc. As a result, a default identity endpoint for accessing the identity services may be generated, as well as endpoints for accessing the registered services.

[0234] In step 1018, vPLC-related information such as the vPLC ID, the reseller's identification information, and other information facilitating the reseller-supplied cloud service may be stored in the vPLC namespace. In step 1020, the CSP may send a response indicating the created vPLC and other information to the reseller R. The reseller may then begin managing the vPLC for the reseller's customer, such as creating a customer account and configuring the reseller-supplied cloud service.

[0235] FIG. 11 is a block diagram of a distributed environment illustrating an example of a virtual private label cloud (vPLC) hosted by a CSP-provided infrastructure in a region, according to one embodiment. FIG. 11 may be similar to FIG. 6, and both figures include securely isolated portions (or partitions) of the CSP-provided regional infrastructure for the CSP's direct customers, vPLC.R1 associated with reseller R1, and vPLC.R2 associated with reseller R2 (e.g., first portion 602, second portion 604, and third portion 606 of FIG. 6). The CSP's direct customers, customers of reseller R1, and customers of reseller R2 may access their respective partitions through one or more endpoints and consoles. Also, as shown in FIG. 6, the CSP-provided regional infrastructure 602 may include a control plane, a management plane, and a gateway.

[0236] However, Figure 11 shows in more detail the interconnections between resources in the CSP-provided regional infrastructure, such as compute, storage, gateways, CPs, and the management plane. Additionally, the CSP's non-reseller direct customers, customers of reseller R1, and customers of reseller R2 may access the CSP-provided regional infrastructure from the Internet or a corporate network. For example, a gateway may be used to access the Internet or a corporate network from a Real Application Cluster (RAC) in the CSP-provided regional infrastructure.

[0237] FIG. 11 illustrates three cloud partitions running on the same CSP-provided regional infrastructure 1102: the CSP's native cloud 1120 and two vPLCs (vPLC.R1 1122 and vPLC.R2 1124) for resellers R1 and R2. The CSP-provided regional infrastructure 1102 is operated by the CSP. The CSP-provided regional infrastructure 1102 may include a control plane 1104 and a management plane 1105. As previously mentioned, the control plane 1104 may be responsible for providing management, deployment, and orchestration functions for the CSP-provided cloud environment. The management plane 1105 may be a collection of software components responsible for provisioning all processes and workflows required for requested operation. The software implementing the cloud service may originate from a deployment system 1106 and be integrated into data centers in the region through the management plane 1105. Software updates (e.g., new services) to the control plane 1104 may be delivered by a deployment system 1106 through the management plane 1105. The management plane 1105 also allows the CSP administrator or reseller operator 1107 deploying the software / services to access the CSP-provided regional infrastructure.

[0238] Compute resources, namely three Real Application Clusters (RACs) 1140, 1142, and 1144, sharing the same storage 1150, are partitioned and allocated to the CSP's native cloud 1120, vPLC.R1 1122, and vPLC.R2 1124, respectively. For example, RAC 1140 is dedicated to the CSP's direct customers, RAC 1142 is designated for vPLC.R1, and RAC 1144 is for vPLC.R2.

[0239] Network resources can also be divided or shared between the CSP's direct customers and resellers to connect their respective computing resources to the Internet or corporate network. For example, each RAC 1140, 1142, and 1144 uses its network gateway 1160, 1162, and 1164 to access the Internet 1108 or corporate network 1109. In some embodiments, two RACs may share the same network interface. For example, RAC 1142 and RAC 1144 share the same gateway 1162 to access the Internet 1108.

[0240] The storage in Figure 11 is simplified. Storage 1150 may include various types of storage, such as general storage, dedicated storage, block storage, and object storage. In Figure 11, vPLC.R1 and vPLC.R2 have dedicated storage 1152 and 1154, respectively. General storage 1150 belongs to the CSP.

[0241] In FIG. 11 , a CSP's native cloud 1120 may provide CSP-supplied cloud services to the CSP's non-reseller direct customers (e.g., CSP.C1-Cn) (e.g., businesses). Users associated with the reseller's customers can access the services through dedicated service endpoints that are part of a respective namespace, including a fully qualified domain name (FQDN), IP address, console experience, identity information, etc. The CSP may provide individual consoles for each reseller, and the consoles provide the services through the endpoints. The endpoints associated with the consoles may vary depending on the services provided to the reseller's customers based on the reseller's subscription. For example, individual reseller customers may submit access requests through dedicated endpoints or consoles.

[0242] Users can use service endpoints 1170 and 1172 (two endpoints are shown, but more are possible) published by the CSP, which have DNS names. The native cloud 1120 may also have a console 1174 with a look and feel. The CSP's direct customers on the Internet 1108 can request instance launches or terminations by accessing the CSP-provided regional infrastructure 1102 through public endpoints 1170 and 1172. Reseller R1's customers on the Internet 1108 can request instance launches or terminations by accessing the CSP-provided regional infrastructure 1102 through public endpoints 1180 and 1182. Reseller R2's customers on the corporate network 1109 can request instance launches or terminations by accessing the CSP-provided regional infrastructure 1102 through dedicated endpoints 1190 and 1192. The control plane 1104 can then launch instances, terminate instances, set up connections for these instances, and orchestrate services in the data center. Network interfaces 1160 and 1162 (e.g., Internet Gateways (IGWs)) of the CSP and vPLC.R1, respectively, can provide network interconnection to the Internet 1108. Network interface 1164 (e.g., Dynamic Routing Gateway (DRG)) of vPLC.R2 can provide network interconnection to the enterprise network 1109.

[0243] In FIG. 11 , vPLC.R1 1122 and vPLC.R2 1124 share the same physical infrastructure. Each endpoint for each reseller (e.g., endpoints 1180 and 1182 for reseller R1 and endpoints 1190 and 1192 for reseller R2) is unique. Consoles 1184 and 1194 are customized for reseller R1 and reseller R2, respectively. The vPLC endpoints and consoles can connect to the Internet or their respective corporate networks as needed. For example, in FIG. 11 , vPLC.R1 connects to the Internet 1108 through a network gateway 1162 (e.g., IGW). However, vPLC.R2 does not connect to the Internet; instead, it connects to the corporate network 1109 through a secure network 1164 (e.g., FastConnect). 11, gateway 1162 may be sliced ​​and shared by vPLC.R1 and vPLC.R2 and may be part of an orchestration implemented by a CSP. In this example, some of vPLC.R2's data may go to the Internet 1108 and some may go to its corporate network 1109.

[0244] If the request arrives through console 1194, it may go to control plane 604. The control plane may determine that the request arrived from vPLC.R2 1124, find the available resource (e.g., computer) from the RAC specified for vPLC.R2 (i.e., 1144 in this case), and provision and activate this resource. Control plane 604 may then provide an IP address for the network configuration (e.g., VCN) of reseller R2's customer through console 1194. Control plane 1104 can configure gateway 1164 to either the enterprise network 1109 or the Internet 1108, depending on the configuration. The control plane can recognize not only the request but also the requestor (either the CSP's direct customer, a customer of reseller R1, or a customer of reseller R2).

[0245] As mentioned above, partitioning of the CSP-provided regional infrastructure can be performed for different vPLCs at various predetermined infrastructure levels (e.g., rack level, server level, hypervisor level, or VM level). To further explain, in the case of partitioning at the rack level, a vPLC may have its own dedicated rack and its internal resources. In the case of partitioning at the server level, multiple vPLCs share a rack, while each vPLC has its own dedicated server. In yet another example, in the case of partitioning at the hypervisor level, multiple vPLCs share a server, while each vPLC has its own dedicated hypervisor. In the case of partitioning at the VM level, multiple vPLCs share a hypervisor, while each vPLC has its own dedicated VM. Thus, there can be two main vPLC infrastructure partitioning models (or two ends of a spectrum) for realizing a multi-vPLC scheme: a segmented model and a nested model. The segmented model can represent resource partitioning at the rack level. The nested model can represent resource partitioning at the VM level. However, other variations between the two are possible given operational efficiency (eg, control plane sharing) and fragmentation (ie, resource division) considerations.

[0246] FIG. 12 is a block diagram illustrating a vPLC segmentation model for resource partitioning, according to one embodiment. In this model, which represents one end of the partitioning spectrum, each vPLC may have its dedicated resources (including compute and network resources) at the rack level. In other words, a rack may be exclusively assigned to a particular vPLC and not used by other vPLCs or by a CSP to provide CSP-supplied services directly to customers. In the segmentation model, resource and usage tracking is performed at the physical resource level based on affinity tags (e.g., vPLC ID and customer tenancy ID). While there may be default behavior per segment, each segment can be customized by the resellers associated with the vPLCs in that segment.

[0247] In FIG. 12 , multiple vPLCs (e.g., vPLC.R1 1210, vPLC.R2 1220, and vPLC.Rn 1230) may exist in a CSP-provided infrastructure (e.g., a data center) in a certain region. In one embodiment, in a segmentation model, there may also be physical and logical partitions, as shown in FIG. 12 . For example, in the case of physical partitions, vPLCs may be assigned dedicated hardware resources (e.g., racks), each of which may include one or more servers hosting one or more operating systems. In FIG. 12 , a first rack is assigned to the vPLC.R1 segment 1210, a second rack is assigned to the vPLC.R2 segment 1220, and a third rack is assigned to the vPLC.Rn segment 1230. Each vPLC may also have a dedicated networking switch (also referred to as a gateway) that connects to the Internet / corporate network 1260. For example, vPLC.R1 1210 has gateway 1219, vPLC.R2 1220 has gateway 1229, and vPLC.Rn 1230 has gateway 1239. Thus, in FIG. 12 , vPLC.R1 1210 has dedicated servers 1214a and 1214b, a hypervisor 1216 running on server 1214b and hosting VMs 1218a-1218n, and a gateway (e.g., IGW) 1219. Similarly, vPLC.R2 1220 has dedicated servers 1224a and 1224b, a hypervisor 1226 running on server 1224b and hosting VMs 1228a-1228n, and a gateway 1229. The vPLC.Rn 1230 includes dedicated servers 1234a and 1234b, a hypervisor 1236 that runs on the server 1234b and hosts VMs 1238a to 1238n, and a gateway 1239.

[0248] In the case of logical partitioning, there is a common control plane 1250 and resource manager 1252 for all vPLCs that are managed and operated by the CSP. However, in a segmented model, a data plane may be dedicated to each vPLC. For example, each vPLC.R1, vPLC.R2, and vPLC.Rn has its own data plane 1212, 1222, and 1232, respectively. In some embodiments, the resource manager 1252 may be part of the management plane. In other embodiments, the resource manager 1252 may be separate from the management plane.

[0249] In some embodiments, a user associated with a customer 1280 of reseller R1 may launch a VM instance or a bare metal server by sending a request through one of vPLC.R1 endpoints 1270. Assume the request is to launch a new VM instance. In this case, control plane 1250 may work with an identity service to check the user's proper credentials for vPLC.R1, and resource manager 1252 may then identify all resources dedicated to vPLC.R1, such as an IP address range. This results in a VM instance (e.g., 1218a) being launched on hypervisor 1216 in vPLC.R1 segment 1210. Control plane 1250 then sends a response back to the requesting user through vPLC.R1 endpoint 1270. If the request is for a bare metal server (e.g., 1214a), resources for vPLC.R1 may be provisioned and allocated to vPLC.R1 with all the required settings by resource manager 1252. Similar processes may be performed such that requests from users associated with customers of reseller R2 1282 are fulfilled by vPLC.R2 segment 1220, and requests from users associated with customers of reseller Rn 1284 are fulfilled by vPLC.Rn segment 1230. Each process for a vPLC may be performed independently of and concurrently with other vPLCs.

[0250] The vPLC segmentation model offers several potential advantages, including physical separation and sharing of a common control plane 1250 operated by the CSP. Additionally, a resource manager 1252 manages resources across the vPLCs 1210-1230. However, scaling challenges may arise because resources, including networking and storage, are replicated as specified for each vPLC. This model also makes small footprints, such as a single rack or server, less feasible. However, the vPLC segmentation model still allows for sharing of power, cooling, and so on. As mentioned previously, the segmentation model is only one end of the spectrum, and there are various resource- and efficiency-based partitioning methods.

[0251] FIG. 13 is a block diagram illustrating a vPLC nesting model for resource partitioning, according to one embodiment. The nesting model of FIG. 13 illustrates the other end of the resource partitioning spectrum at the VM level. In other words, a VM may be exclusively assigned to a particular vPLC and not used by other vPLCs or by the CSP to provide CSP-delivered services directly to its customers. However, hypervisors, servers, or racks may be shared between the CSP's customers and the customers of one or more vPLCs. In some embodiments, the nesting model may have a base layer (such as a hypervisor) that is invisible to the customer as a shared foundation. Abstraction layers (e.g., VMs) may also be built on top of the base layer, allowing each reseller to clone and customize the abstraction layers for their customers.

[0252] In the embodiment of FIG. 13 , all vPLCs (e.g., vPLC.R1, vPLC.R2, and vPLC.Rn) are hosted on a single segment with a shared infrastructure represented as a “server CSP” 1320 (e.g., a bare metal server) operated by a CSP, which may have control plane functionality. The server CSP 1320 may be configured to host a hypervisor 1330 that manages virtual machines (VMs) 1332a-1332n running on the hypervisor. In some embodiments, a segment may include multiple racks. Here, two VM instances for vPLC.R1 1332a and 1332b, one VM instance for vPLC.R2 1332c, and one VM instance for vPLC.Rn 1332n reside on the shared hypervisor 1330. In segment 1310, all vPLCs share the same data plane 1312. In some embodiments, a segment may include multiple racks. Since hypervisors, servers, and racks can be shared by multiple vPLCs, the servers of these vPLCs may come from different racks.

[0253] 13 further illustrates a variation of the splitting scenario in which a customer of reseller R2 may request an additional bare metal server, server vPLC.R2 1322, that is provisioned as a dedicated server. In the vPLC nesting model, a shared server pool can be placed wherever space is available, thus providing a low-cost proposition to the reseller.

[0254] In FIG. 13 , a gateway 1340 (e.g., IGW or DRG) shared by the vPLCs has a connection to the Internet or a corporate network as needed. Incoming packets from the Internet / corporate network 1360 may be tagged (i.e., encapsulated in each packet header) with vPLC-related information (e.g., a vPLC ID and a customer tenancy ID) at the gateway 1340 for tracking purposes to reach the appropriate vPLC. Packet encapsulation helps separate traffic on a single segment 1310 shared by multiple vPLCs. As an example, a packet destined for VM 1332a, which belongs to a tenancy associated with customer C1 of reseller R1, which is associated with vPLC.R1, may have vPLC_ID.vPLC.R1 and customer tenancy ID (T_ID.R1.C1) encapsulated in the packet header. The packet may then be routed to VMs 1332a and 1332b separated for vPLC.R1 based on the vPLC ID (vPLC_ID.vPLC.R1). The customer tenancy ID (T_ID.R1.C1) associated with reseller R1's customer C1 in the packet header may further aid in routing the packet to tenant C1 running on VM 1332a, where the packet is decapsulated before delivery to VM 1332a.

[0255] Similarly, a packet destined for VM1332c, which belongs to a tenancy associated with customer C1 of reseller R2, which is associated with vPLC.R2, may have vPLC_ID.vPLC.R2 and a customer tenancy ID (T_ID.R2.C1) encapsulated in the packet header. The packet can then be routed to resources carved out for vPLC.R2 based on the vPLC ID (vPLC_ID.vPLC.R2). The customer tenancy ID (T_ID.R2.C1) associated with customer C1 of reseller R2 in the packet header can further aid in routing the packet to tenant C1 running on VM1332c in vPLC.R2. The packet is decapsulated before delivery to VM1332c.

[0256] Outgoing packets may perform the reverse process. For example, an outgoing packet originating from VM 1332a in vPLC.R1 and destined for the Internet 1360 may have vPLC_ID.vPLC.R1 and a customer tenancy ID (T_ID.R1.C1) associated with reseller R1's customer C1 encapsulated in the packet header. When the packet reaches gateway 1340, it may be decapsulated before proceeding to the Internet 1360. Similarly, an outgoing packet originating from VM 1332c in vPLC.R2 and destined for the Internet 1360 may have vPLC_ID.vPLC.R2 and a customer tenancy ID (T_ID.R2.C1) associated with reseller R2's customer C1 encapsulated in the packet header. When the packet reaches gateway 1340, it may be decapsulated before proceeding to the Internet 1360. As a result, both inbound and outbound traffic are separated in a single segment 1310 shared by multiple vPLCs.

[0257] In some embodiments, in a nested model, one or more VMs may belong to a direct customer of the CSP and share the same hypervisor as other vPLCs.

[0258] Reseller users accessing the vPLC infrastructure are authenticated in both the segmented and nested models. Users associated with a particular reseller's customer can access an endpoint and authenticate as a user of the vPLC associated with that particular reseller using the user's credentials (e.g., username and customer tenancy ID). For example, in some embodiments, in FIG. 13 , a user associated with reseller R1's customer C1 may launch a VM instance by sending a request through one of the vPLC.R1 endpoints 1302. The particular vPLC.R1 endpoint 1302 may augment the request with the vPLC ID of vPLC.R1. The control plane 1350 may work with an identity service to check the user's proper credentials for vPLC.R1 (e.g., the user's username and customer C1's tenancy ID). Resource manager 1352 can then identify all resources assigned to vPLC.R1, e.g., server 1320 and hypervisor 1330 based on the vPLC ID and VM 1332a based on the customer tenancy ID for customer C1, and VM instance 1332a is then launched on hypervisor 1330 accordingly.

[0259] 14 is a flow diagram illustrating a sign-up process for a customer of a reseller associated with a vPLC, according to one embodiment. When a CSP's direct customer registers or subscribes (i.e., signs up) for a cloud service offered by the CSP (i.e., a CSP-provided cloud service), a tenancy or account is created for that customer. Similarly, when a reseller's customer registers or subscribes for a cloud service offered by the individual reseller (i.e., a reseller-provided cloud service), a tenancy or account is created for that customer.

[0260] FIG. 14 illustrates the sign-up process for a particular customer C1 of reseller R1. However, this sign-up process is also applicable to customers of individual resellers. In FIG. 14, in step S1, reseller R1's customer C1 1401 may sign up using a sign-up portal on the console of vPLC.R1 1402 to create a new account by entering user ID credentials (user name or account name (e.g., John)). In step S2, vPLC.R1's console 1402 may enhance the sign-up request by adding a vPLC ID (e.g., vPLC_ID.vPLC.R1) that identifies the vPLC created for reseller R1 to the user credentials and forward the sign-up request to a CSP (control plane (CP) or management plane (MP)) 1404. In some embodiments, the account creation for reseller R1's customer C1 may be performed by reseller R1 with assistance from the CSP's CP or MP.

[0261] Then, in step S3, the CSP creates a customer tenancy (or account T.R1.C1) for reseller R1's customer C1 1401. The newly created tenancy (T.R1.C1) for customer C1 is assigned a tenancy identifier (T_ID.T.R1.C1). A vPLC ID (e.g., vPLC_ID.vPLC.R1) identifying vPLC.R1 for reseller R1 is associated with the newly created customer tenancy ID (e.g., T_ID.T.R1.C1) for customer C1 (shown in 1430) and may be stored in database 1420 along with other authentication information (e.g., account name and password). Other customers of reseller R1 and reseller R2 may also be stored in database 1420. For example, as shown at 1432, the customer tenancy ID of reseller R1's customer C2 (i.e., R1.C2) is associated with the vPLC ID of vPLC.R1 (T_ID.T.R1.C2 and vPLC_ID.vPLC.R1). Also, as shown at 1434, the customer tenancy ID of reseller R2's customer C2 (i.e., R2.C1) is associated with the vPLC ID of vPLC.R2 (T_ID.T.R2.C1 and vPLC_ID.vPLC.R2).

[0262] The association between the vPLC ID and the tenancy IDs of individual resellers' customers can enable isolation between customer tenancies within a vPLC. For example, customer tenancy R1.C1 can be securely isolated from customer tenancy R1.C2 by using different customer tenancy IDs (T_ID.T.R1.C1 for customer C1 and T_ID.T.R1.C2 for customer C2), but both customer tenancies reside in the same vPLC.R1. The tenancies associated with reseller R1 and reseller R2 can be securely isolated by using their respective vPLC IDs (vPLC_ID.vPLC.R1 for reseller R1 and vPLC_ID.vPLC.R2 for reseller R2).

[0263] In step S4, CSP 1404 may return a handle including the customer tenancy ID (e.g., T_ID.T.R1.C1), a temporary default password, and an account name to vPLC.R1 console 1402. In step S5, the vPLC.R1 console may receive the temporary authentication information and forward the customer tenancy ID and password to reseller R1's customer C1 1401. As shown in FIG. 14, reseller R1's customer C1 1401 can sign up for and register for one or more reseller-supplied cloud services by interacting directly with reseller R1 through vPLC.R1 console 1402. Similarly, a CSP's customer may register for a CSP-supplied cloud service by separate interaction and sign-up with the CSP at the CSP's console (not shown).

[0264] As described above with respect to creating a tenancy, a CSP's database (e.g., 1420 in FIG. 14) may store vPLC IDs, customer tenancy IDs, and their corresponding authentication information. In some embodiments, multiple customer tenancy IDs may be associated with the same vPLC ID, as shown in 1430 (T_ID.T.R1.C1 is associated with vPLC_ID.vPLC.R1) and 1432 (T_ID.T.R1.C2 is associated with vPLC_ID.vPLC.R1). Tenancies may be defined at the realm level and thus may have a realm identifier associated with a vPLC.

[0265] 15 is a flow diagram illustrating a login process for a user associated with a customer of a reseller associated with a vPLC, according to one embodiment. In FIG. 15, steps S1 and S2 include a login phase after an individual reseller customer completes the sign-up process described in FIG. 14.

[0266] In step S1, a user 1501 associated with a customer of reseller R1 can log in with a customer tenancy ID (or CT ID), username, and password at the vPLC.R1 endpoint 1510. In S2, the user 1501 and their customer tenancy may be verified at the vPLC.R1 endpoint 1510 by going through a verification process, including, but not limited to, user authentication (e.g., verifying valid credentials such as username and password), session management (e.g., session tokens or timeouts), and access control enforcement (based on access privileges, permissions, and policies).

[0267] In S3, the vPLC.R1 endpoint 1510 may return a session handle. The session handle may represent a token or unique identifier returned to a customer after successfully logging into their account. The session handle is used for subsequent requests to access cloud services and resources (e.g., launch an instance) without requiring re-authentication for each individual request during an active login session (or API session).

[0268] When resources are created, a cloud resource ID (CID) may be generated for those resources, which includes the customer tenancy ID. Virtual machines, accounts, tenancies, permission policies, virtual network subnets, ACLs, and load balancers are considered examples of cloud resources and can therefore be referenced by their respective CIDs.

[0269] 16 is a flow diagram illustrating a resource allocation and provisioning process for a reseller's customer associated with a vPLC, according to one embodiment. A resource manager (RM) manages the resources of a CSP-provided cloud infrastructure in a region and the resource partitions allocated to each vPLC.

[0270] As described above, a user associated with a customer of a reseller may use a vPLC-specific endpoint to access resources associated with the vPLC and perform one or more functions or operations involving resources allocated to the vPLC. In Figure 16, in step S1, a user 1601 associated with a customer C1 of reseller R1 may launch a compute instance (e.g., a VM instance with 1 GB of RAM) by issuing a request through a vPLC-specific endpoint, vPLC.R1 endpoint 1602.

[0271] In step S2, the vPLC.R1 endpoint 1602 may extend (or annotate) the request by tagging it with the vPLC.R1 endpoint's corresponding vPLC ID (e.g., vPLC_ID.vPLC.R1) to indicate that the request originates from a customer of reseller R1, and forwards the request to resource manager (RM) 1604. In other words, the request, once authenticated by the annotation information (e.g., vPLC ID), can be processed by all policies associated with vPLC.R1. In some embodiments, policies associated with a vPLC may include, for example, the type of resource, the amount of resource that can be allocated (i.e., capacity limits), permissions, etc.

[0272] In step S3, the RM 1604 determines whether the requested resources are available to vPLC.R1 under the policy by querying its database (i.e., resource database) 1606. Based on this query, in step S4, after confirming that the requested resources are available under the policy, the RM may perform resource allocation. In other words, the RM allocates a portion of the resources allocated to vPLC.R1 for reseller R1 to a customer tenancy (T.R1.C1) associated with reseller R1's customer C1 (based on the customer tenancy ID at user login). This resource allocation may be performed by allocating or designating a portion of the available cloud resources (allocated to vPLC.R1) to a user associated with customer C1 based on a policy agreed upon between customer C1 and reseller R1, such as capacity limits or permissions. For example, reseller R1 and its customer C1 may have a contract policy that allows for the use of up to 100 GB of memory. Currently, 80 GB is allocated (or used). If a user associated with customer C1 requests 10 GB, the RM can identify the 10 GB CID provisioned by the CP. If a user associated with customer C1 requests 30 GB, this exceeds the total 100 GB limit, and the request can be rejected.

[0273] In step S5, the RM 1604 retrieves from the resource database a cloud resource ID assigned to a resource (vPLC.R1.C1) allocated to the tenancy (T.R1.C1) of the customer C1 of the reseller R1. In step S6, the RM passes the resource ID to the vPLC.R1 endpoint 1602. In step S7, the vPLC.R1 endpoint 1602 requests a resource from the control plane (CP) 1608 for a resource having a particular resource ID assigned to the resource allocated to the tenancy (T.R1.C1) of the customer C1 of the reseller R1. In other words, the resource manager 1604 can augment the user's request (e.g., step S6) with information associated with the vPLC (e.g., the resource ID assigned to vPLC.R1.C1) that the control plane 1608 can use to provision the resource (e.g., step S8).

[0274] In step S8, the control plane may provision the requested resources (e.g., launch a specific VM instance and storage). The resources may include compute, networking, storage, etc. In step S9, the control plane 1608 may respond to the vPLC.R1 endpoint 1602 with a status and a resource handle. The resource handle may represent a unique resource identifier assigned to the provisioned resource (e.g., a cloud resource ID assigned to the provisioned compute instance (e.g., VM), storage volume, virtual cloud network (VCN or subnet), database, etc.). The resource handle may enable the customer to access and manage the provisioned resource through an API or command line tools.

[0275] In step S10, the status and resource handle are forwarded to user 1601 associated with customer C1 of reseller R1. In step S11, the control plane may notify the RM regarding the resource provisioning status (e.g., success or failure). In step S12, the RM may update a record in resource DB 1606 that tracks resource usage for customer C1 associated with the user. Continuing with the above example, the record may be updated to indicate that the total amount of memory used by customer C1 is 90 GB after successfully provisioning 10 GB in addition to the previously provisioned 80 GB, or that the total remains at 80 GB after the user's request was denied. Records maintained by the resource DB may be useful for enforcing policies associated with vPLCs.

[0276] Figure 17 is a flow diagram illustrating a vPLC resource allocation and provisioning process for a reseller's customer associated with a vPLC, according to one embodiment. Figure 17 is an alternative embodiment to Figure 16. Instead of using a Resource Manager (RM) as a central role for communicating with all subsystems and receiving requests to coordinate the resource allocation and provisioning process, a Control Plane (CP) may also be used for this central role.

[0277] For example, in FIG. 17 , in step S1, a user 1701 associated with a customer C1 of reseller R1 may issue a request to a vPLC-specific endpoint, vPLC.R1 endpoint 1702. In step S2, vPLC.R1 endpoint 1702 may extend the request by tagging it with a corresponding vPLC ID to indicate that the request originates from a customer of reseller R1, and forwards the request to control plane (CP) 1704. In step S3, CP 1704 forwards the request to resource manager (RM) 1706. In step S4, RM 1706 determines whether the requested resources are available for vPLC.R1 by querying its database (i.e., resource database) 1708. Based on this query, in step S5, if the requested resources are available, the RM performs resource allocation.

[0278] In step S6, the RM retrieves from the resource database the resource ID assigned to the resource (vPLC.R1.C1) allocated to the tenancy (T.R1.C1) of the customer C1 of the reseller R1. In step S7, the RM passes the resource ID to the CP 1704. In step S8, the control plane may provision the requested resource. In step S9, the control plane 1704 may respond to the vPLC.R1 endpoint 1702 with a status and a resource handle. In step S10, the status and resource handle may be forwarded to the user 1701. In step S11, the control plane may notify the RM regarding the provisioning status of the resource. In step S12, the RM may update a record in the resource DB that tracks resource usage for the customer associated with the user.

[0279] 17, the CP 1704 may play a central role by forwarding the resource request to the RM 1706 (e.g., step S3), receiving the resource ID from the RM (e.g., step S6), and provisioning the requested resources (e.g., step S8). Compared to FIG. 16, this embodiment avoids the need for the vPLC.R1 endpoint 1702 to perform extra steps, such as requesting resources from the CP as shown in steps S7 and S8 of FIG. 16.

[0280] FIG. 18 is a simplified diagram illustrating a use case in which a vPLC is hosted by the infrastructure of a first realm while associated with a second, different realm, according to one embodiment. In FIG. 18, there are two separate realms (CSP-provided realm A 1802 and realm B 1812). Each realm is a logical collection of regions. Realms are isolated from each other and do not share data. Thus, realm A 1802 is isolated from realm B 1812, and there is no data sharing or communication between the two realms. A tenancy created for a CSP customer exists only in one realm and cannot be used to access another realm or region in another realm. Realms span regions within a realm, allowing a CSP to provide a specified level of service that meets the needs of a particular organization. For example, for a high level of security and isolation, a CSP may create realms, such as a “Government” realm created for government entities that are customers of the CSP. A CSP may also create a second realm with a lower level of security and isolation, such as a "Commercial" realm for the CSP's non-government commercial customers. Examples of realms a CSP may create are (a) a Commercial realm, (b) a US Government FedRAMP Authorized realm, (c) a US Government IL5 Authorized realm, (d) a Country A Government realm, (e) a Country B Government realm, etc. For example, in FIG. 18, realm A 1802 may have a higher security posture than realm B 1812.

[0281] A realm can have one or more regions. Each realm has its own identity and trust profile (e.g., password, authentication information, etc.). This identity and trust profile information is shared between regions in the same realm, so regions within a realm can communicate with each other. The realm's identity and trust profile are configured to prevent inter-realm communication.

[0282] In the example shown in Figure 18, realm A 1802 includes region A 1804. The infrastructure provided by the CSP in region A 1804 is organized as datacenter 1806. In Figure 18, datacenter 1806 is shown as a dedicated regional cloud datacenter (DRCC) hosted on customer premises. A DRCC may represent a fully managed cloud region built with CSP-designed high-performance infrastructure to help customers bring cloud primitives and services closer to their existing data and applications. A DRCC may be dedicated to a single customer and operate within the customer's datacenter while being fully managed by the CSP.

[0283] In FIG. 18 , realm B 1812 includes region B 1814. The infrastructure provided by the CSP in region B 1814 is organized as data center 1816. As shown in FIG. 18 , data center 1816 hosts vPLC 1818. Thus, a portion of the infrastructure provided by the CSP in data center 1816 of region B 1812 is allocated to vPLC 1818. Thus, vPLC 1818 is physically located in region B 1814 of realm B 1812.

[0284] As previously discussed, when a vPLC is created, it is associated with a particular realm as part of its configuration. Typically, a vPLC is associated with the same realm associated with the region in which the CSP-provided regional infrastructure hosting the vPLC is located. In some embodiments, when a vPLC is configured, it may be associated with a realm different from the realm associated with the regional infrastructure hosting the vPLC. For example, in the example shown in FIG. 18 , vPLC 1818 may be associated with Realm A 1802 rather than Realm B 1812. The dashed box 1820 and associated arrow 1830 are meant to indicate that vPLC 1818 is physically hosted by the data center 1816 of Realm B 1812, but is logically or virtually associated with and belongs to Realm A 1802. Realm B 1812 is referred to as the hosting realm of vPLC 1818. Realm A 1802 is referred to as the logical or virtual realm of the vPLC 1818.

[0285] As a result, vPLC 1818 and datacenter 1806 exist in the same realm, i.e., realm A 1802. Because infrastructure in the same realm shares the same identity and trust profile, vPLC 1818 is configured with the identity and trust profile of realm A 1802. Configuring the identity and trust profile information of realm A 1802 can be achieved by creating a mapping in vPLC 1818. For example, the identity service API of the virtual realm (i.e., realm A 1802) can call the vPLC of the host realm (i.e., realm B 1812), and the host realm can accept the call and establish trust between the two realms (i.e., share identity and trust profile information). When an instance is created in a customer tenancy of the vPLC of the host realm, the resource ID and tenancy ID associated with the newly created instance in the host realm can be mapped to a new resource ID and tenancy ID in the virtual realm. Thus, a table of mappings (or "shadow tenancy") may be maintained in the host realm (i.e., realm B 1812). The same process and mapping occurs in the virtual realm (i.e., realm A 1802). Thus, both vPLC 1818 and realm A 1802 (i.e., the virtual realm) share the same identity and trust profile information. As a result, even though vPLC 1818 is physically hosted by a different realm B 1812 infrastructure, vPLC 1818 and datacenter 1806 can communicate with each other because they reside in realm A 1802. vPLC 1818 may not be able to communicate with other infrastructure in region B 1814 associated with realm B 1812.

[0286] Additionally, vPLC 1818 is physically hosted by region B 1814 but is logically associated with region A 1804. Region B 1814 is referred to as the hosting region or host region of vPLC 1818. Region A 1804 of realm A 1802 is referred to as the logical region or virtual region of vPLC 1818. When a vPLC is created, a virtual realm and virtual region for the vPLC may be configured.

[0287] As described above, a resource ID for a resource includes a portion that identifies the realm associated with the resource and a portion that identifies the region associated with the resource. A vPLC is considered a resource and is therefore assigned a resource ID. In one embodiment, a vPLC ID, which uniquely identifies a vPLC, is a type of resource ID. A vPLC ID includes a portion that identifies the virtual realm associated with the vPLC identified by the vPLC ID and a portion that identifies the virtual region associated with the vPLC. Thus, for purposes of identity management in Realm A 1802, a vPLC is considered part of the realm and region identified by the vPLC ID corresponding to the vPLC.

[0288] In one embodiment, when a vPLC is hosted by a region and a realm but associated with a different virtual region and a different virtual realm, the configuration of the vPLC as described above may be performed using Virtual Bootstrap Environment (ViBE) technology. A ViBE may represent a virtual cloud network (VCN) provisioned in an overlay of an existing region (e.g., a "host region"). The provisioned ViBE is connected to the new region using a communication channel (e.g., an IPSec Tunnel VPN). Certain essential core services (or "seed" services), such as a deployment orchestrator and public key infrastructure (PKI) services, may be provisioned in the ViBE. These services can provide the functionality necessary for bringing hardware online, establishing a chain of trust to the new region, and deploying other services in the new region. The virtual bootstrap environment allows for the use of resources in the host region to prevent circular dependencies between bootstrap resources. Services can be staged and tested in the ViBE before the physical region (e.g., the target region) is available.

[0289] For example, in some embodiments, Realm B 1812 may create a ViBE within its realm, which shares identity and trust profile information with Realm A 1802. The ViBE can then be used in Realm B 1812 to bootstrap new regions in Realm A 1802. Additional details related to ViBEs are described in the following applications, the contents of all of which are incorporated herein by reference:

[0290] (1) U.S. Patent Application No. 18 / 105,779, filed February 3, 2023, entitled "VIRTUAL BOOTSTRAP ENVIRONMENT FOR BUILDING REGIONAL DATA CENTERS" (2) U.S. Patent Application No. 18 / 105,768, filed February 3, 2023, entitled "TECHNIQUES FOR A VIRTUAL BOOTSTRAP ENVIRONMENT IN A DISTRIBUTED VIRTUAL PRIVATE NETWORK" (3) U.S. Patent Application No. 18 / 105,779, filed February 3, 2023, entitled "TECHNIQUES FOR MIGRATING SERVICES FROM A VIRTUAL BOOTSTRAP ENVIRONMENT" In the architecture shown in FIG. 18 , a vPLC is hosted by a region and a realm, but is associated with a different virtual region and a different virtual realm, which can be used for various purposes. For example, in the example shown in FIG. 18 , vPLC 1818 may function as a backup for data center 1806. This allows for disaster recovery (DR) backups in various regions. For example, if data center 1806 is a DRCC located on a customer's premises, the customer may request a CSP to provide backup services for DRCC 1806. In response, the CSP may select vPLC 1818 as a backup for DRCC 1806. vPLC 1818 is physically hosted in region B 1812, which is in a different region away from region A 1802, which hosts DRCC 1806. This may be a cost-effective solution because the CSP does not need to build a separate data center for the customer of realm A 1802 just for backup purposes. In some embodiments, the customer may not even be aware of where the backup is being performed, i.e., the presence and location of the vPLC 1818 may be transparent to the customer. The vPLC used for backup may be physically hosted in a CSP-provided cloud in a different realm and region.

[0291] Since the purpose of a DR site is to minimize the occurrence of correlated failures by introducing geographic separation between the primary and DR sites and avoiding simultaneous changes to both the primary and secondary locations, the DRCC 1806 can enjoy DR guarantees if the CSP can ensure that the DRCC and its paired vPLC 1818 (and therefore its hosting region in Realm B 1812) do not change simultaneously.

[0292] 19 is a flowchart 1900 illustrating an exemplary method for creating a vPLC hosted by a CSP provisioning infrastructure in a particular region in a particular realm, while virtually associated with a different region in a different realm, according to one embodiment. The process illustrated in FIG. 19 may be performed as part of the process performed for creating the vPLC. In one embodiment, the process illustrated in FIG. 19 may be performed primarily by the CSP provisioning infrastructure in the hosting region and realm.

[0293] Processing begins at 1902 with the receipt of information identifying a host region of a host realm in which a vPLC will be physically hosted. For example, in the embodiment shown in Figure 18, information may be received that a vPLC will be created in region B 1814 of realm A 1812. The region and realm identified at 1902 represent the host region and host realm of the vPLC to be created.

[0294] At 1904, information may be received that identifies a region of a particular realm. The particular realm is different from the host realm identified at 1902. Furthermore, the region and particular realm identified at 1904 will be associated as a virtual region and virtual realm of the vPLC being created. For example, in the embodiment shown in FIG. 18, information may be received that the vPLC is to be virtually associated with region A 1804 of realm A 1802.

[0295] At 1906, identity and trust profile information is obtained for the particular realm identified at 1904. The information obtained at 1906 may include identity authentication information (including passwords, certificates, etc.) associated with the realm that will be the virtual realm of the vPLC.

[0296] At 1908, a vPLC ID is generated for the vPLC being created. The vPLC ID identifies the region and particular realm identified in the information received at 1904.

[0297] At 1910, a vPLC is created in the host realm and resources are allocated to the vPLC. The allocated resources are selected from resources in the infrastructure provided by the CSP in the region and realm identified in 1902. These resources may include one or more physical resources (e.g., racks, servers, routers, memory, or storage resources) and / or logical resources (e.g., hypervisors, virtual machines, virtual routers). For example, in the embodiment shown in FIG. 18, resources are allocated to the vPLC from a data center 1816.

[0298] At 1912, a vPLC is configured using the identity and trust profile information obtained at 1906. For example, in the embodiment shown in Figure 18, the vPLC may be configured using the identity and trust profile information for Realm A 1802.

[0299] At 1914, the vPLC initiates communication with the CSP-provided infrastructure in the particular realm (the vPLC's virtual realm) identified at 1904. For example, in the embodiment shown in Figure 18, the vPLC 1818 communicates with the data center 1806. This communication may be initiated by the vPLC 1818 or by the data center 1806.

[0300] Virtual Private Label Cloud (vPLC) - Remote Data Plane FIG. 20 is a block diagram illustrating a CSP control plane model, according to one embodiment. The distributed environment 2000 illustrated in FIG. 20 is merely an example and is not intended to unduly limit the scope of the claimed embodiments. Many variations, alternatives, and improvements are possible. For example, in some embodiments, the distributed environment 2000 may include more or fewer systems or components than those illustrated in FIG. 20, may combine two or more systems, or may have different system configurations or arrangements. The systems, subsystems, and other components illustrated in FIG. 20 may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective systems, using hardware, or a combination thereof. Software may be stored in a non-transitory storage medium (e.g., a memory device).

[0301] As shown in Figure 20, there is a CSP-provided infrastructure 2002 in a region and a remote site 2060. The infrastructure network at the CSP-provided regional infrastructure 2002 (which may also be referred to as a local site) is referred to as Infrastructure 0, while the infrastructure network at the remote site 2060 is referred to as Infrastructure 1. In some embodiments, network Infrastructure 0 and Infrastructure 1 may be compatible with each other, such that each site may have a network virtualization device (NVD) and share the same address space. In some embodiments, network Infrastructure 0 and Infrastructure 1 may be incompatible with each other, such that communication between the local site 2002 and the remote site 2060 may require network address translation (NAT) and lookup tables.

[0302] The CSP-provided regional infrastructure may include CSP namespace 0 2010 and vPLC.R1 namespace 1 2030. CSP namespace 0 includes components such as a CSP CP (Primary Control Plane) 2012, a CSP Resource Manager (RM) 2014, a local Management Plane (MP) 2018, and a Management Plane Adapter (MPA) 2016. In some embodiments, the MPA may be a separate component from the local MP.

[0303] vPLC.R1 namespace 1 2030 includes a local vPLC.R1 data plane (DP) 2038 for performing data plane functions within vPLC.R1, a vPLC.R1 control plane (CP) 2032, a vPLC.R1 resource manager (RM) 2034, and a vPLC.R1 management plane (MP) 2036. As described above, vPLC.R1 may have a VCN 2032 (including the CP, RM, and MP) that enables the vPLC.R1 to communicate with a remote site 2060 over a communication channel 2050.

[0304] In some embodiments, CSP-provided regional infrastructure 2002 may include multiple vPLCs, such as vPLC.R1, vPLC.R2, etc. Each vPLC may have a vPLC-specific CP, a vPLC-specific RM, a vPLC-specific MP, and a vPLC-specific DP.

[0305] A set of endpoints 2004 may be used to receive requests from reseller R1's customers 2003 to access resources or perform functions involving resources, and may be located in vPLC.R1 namespace1 2030 or in a remote DP 2062. In this CSP-CP model, all URLs associated with the endpoints 2004 may point to the CSP-CP server 2012. The CSP-CP 2012 can determine whether a request through an endpoint is from a non-reseller customer of the CSP or a customer of reseller R1 based on the URL associated with the called endpoint. For example, the endpoint associated with the URL compute.CSPcloud.com may be called by a non-reseller customer of the CSP, while the endpoint associated with the URL compute.vplcR1cloud.com may be called by a customer of reseller R1. If the request originates from a reseller customer, the CSP CP 2012 may forward the request to a vPLC-specific CP 2032, if necessary.

[0306] 6, the control plane (CP) may be responsible for providing management, deployment, and orchestration functions, such as accepting requests from customers, working with other components in the cloud infrastructure (e.g., RMs and MPs) to determine whether the requests can be fulfilled, and responding to the customers. In this CSP-CP model, the main CP 2112 may be responsible for determining whether a customer 2003 request should go to the local DP 2038 via route 2080 or to the remote DP 2062 via route 2082. The vPLC-specific CP 2032 may be responsible for receiving requests from the main CP when a customer request is routed to the remote site 2060.

[0307] The management plane (MP) may be a collection of software components responsible for provisioning all the processes and workflows required for a requested operation. In other words, the MP monitors resource sufficiency by ensuring that the various tasks are performed properly and in the correct order by the data plane. In this CSP-CP model, the local MP 2018 may be responsible for monitoring resource sufficiency at the local DP 2038. The vPLC.R1 MP 2036 may be responsible for connecting to the remote site 2060 through the communication channel 2050.

[0308] A management plane adapter (MPA) may mediate communications between the CP and the local MP. In some embodiments, the MPA may be responsible for the local management plane while additionally routing requests for the remote management plane to the correct remote network. In this CSP-CP model, the local MPA 2016 is part of the local MP and may be responsible for mediating communications between the local MP 2018 and the primary CP 2012.

[0309] A resource manager (RM) may be responsible for tracking resources in the cloud infrastructure and determining whether requested resources are permitted and available for allocation based on resource allocation policies. In this CSP-CP model, the local RM 2014 may be responsible for tracking local resources (e.g., local DP 2038) in the CSP-provided regional infrastructure 2002. The vPLC.R1 RM 2034 may be responsible for tracking resources in the remote DP 2062.

[0310] 20 , in some embodiments, the remote site 2060 may include a remote DP 2062 that cooperates with a VCN 2064, a remote MP 2066, and a cache 2068. In some embodiments, a reseller-hosted remote site may include multiple VCNs, each associated with a customer of the reseller. Similar to the MP functionality described above, the remote MP 2066 may be responsible for monitoring resource availability at the remote DP 2062. The remote DP 2062 may include resources 2063 that the reseller R1 may use to provide reseller-provided cloud services (e.g., compute services for launching VMs). The cache 2068 may be used to improve performance of resource-intensive or bandwidth-intensive functions, such as remote communications between the CSP-provided regional infrastructure 2002 and the remote site 2060. For example, when a request is made to launch a VM, the remote DP may use a pre-prepared operating system (OS) image, disk image, or snapshot in its cache instead of accessing the information over communication channel 2050.

[0311] In Figure 20, a communication channel 2050 connects a CSP-provided regional infrastructure 2002 (sometimes referred to as a local site) and a remote site 2060. Specifically, the communication channel 2050 is established between a vPLC.R1 VCN 2031 and a remote VCN 2064. The communication channel 2050 may be an on-demand or persistent connection, such as an optical fiber or wavelength division multiplexing (WDM) link, FastConnect, etc. Two border devices, such as a gateway 2040 located at the local site (i.e., the CSP-provided regional infrastructure) 2002 and a gateway 2070 located at the remote site 2060, serve as ingress and egress points for traffic (e.g., packets) entering and exiting each site through the communication channel 2050. Gateway 2040 and gateway 2070 may include various types of gateways (e.g., a Dynamic Routing Gateway (DRG) for connecting to a remote site's on-premise network and an Internet Gateway (IGW) for connecting to the remote sit via a public network (e.g., the Internet)).

[0312] As shown in FIG. 20 , when a customer 2003 of reseller R1 requests access to a resource or use of a reseller-supplied cloud service, the request may pass through one of a set of endpoints 2004 to a primary CP 2012. The primary CP, on behalf of reseller R1, may determine whether a local DP 2038 or a remote DP 2062 is best suited for the customer's request. This determination may result in two routes (route 2080 and route 2082). In some embodiments, both the local DP 2038 and the remote DP 2062 may be used for the customer's request. For example, if the request involves different resources located at different DPs, one request may follow route 2080 and another may follow route 2082.

[0313] When a customer request is received, the primary CP 2012 may work with the local RM 2014 to determine whether the request to access the local resources of the local DP 2038 is permitted and whether resources are available to fulfill the request. The same check may be performed for the remote DP 2062. In some embodiments, the local RM 2014 may request the assistance of the vPLC.R1 RM 2034 to check the resource allocation history of the remote DP 2062. In some embodiments, the local RM 2014 may request the assistance of the vPLC.R1 RM 2034 to check the resource allocation history of the remote DP 2062. If the local DP can fulfill the request, the request may proceed along the route 2080. The local MP 2018 may then cause the local DP 2038 to perform the appropriate function or action to fulfill the request.

[0314] On the other hand, if the local DP cannot fulfill the request or if a remote DP is a better option, the request may proceed along route 2082. In this case, the primary CP 2012 may send the request to the vPLC-specific CP (or vPLC.R1 CP) 2032, which then requests the vPLC.R1 MP 2036 to send the request to the remote VCN 2064 at the remote site 2060 over communication channel 2050. The vPLC.R1 MP 2036 may cooperate with the remote MP 2066 to have the request fulfilled by the remote DP 2062. Once the request is fulfilled, a response may be sent from the remote MP 2066 back to the vPLC.R1 MP 2036 over communication channel 2050. The vPLC-specific CP 2032 can then communicate a completion response to the primary CP 2012 to notify the reseller R1's customer 2030 of the results.

[0315] Finally, for example, if both local DP 2038 and remote DP 2062 are used, a customer may have multiple requests, with some requests that can be fulfilled by local DP 2038 being routed to route 2080 and other requests that can be fulfilled by remote DP 2062 being routed to route 2082.

[0316] In some embodiments, a reseller's customer may indicate a preference for using resources of the local DP 2038 or the remote DP 2062. In this case, the primary CP 2012 may route requests to either the route 2080 for the local DP 2038 or the route 2082 for the remote DP 2062 depending on the preference.

[0317] To establish a communication channel 2050 between the vPLC.R1 VCN 2031 and the remote VCN 2064, the remote site 2060 may need to provide a web services API with primitives or functions to create VNCs and launch computing instances. For example, the primitives may create a virtual network, create a network tunnel, create a subnet in the virtual network, set DHCP options for the subnet, create a block volume, create a VM or bare metal instance, assign an IP address range to a host or load balancer, set instance metadata for a particular instance, terminate an instance, etc. Such primitives may enable hiding network differences between the local site 2002 (i.e., the CSP-provided regional infrastructure) and the remote site 2060.

[0318] The vPLC.R1 VCN 2031 and the remote VCN 2064 may communicate using packet encapsulation-decapsulation to enable transparent network packet flow from a subnet of the vPLC.R1 VCN 2031 for a customer of the reseller R1 to another subnet of the remote VCN 2064 belonging to the same customer.

[0319] A compatible infrastructure network may represent two physical networks that have the same characteristics (such as interfaces and protocols) so that they can communicate without encountering problems in the network infrastructure. If the infrastructure network is identical or compatible between both sites (i.e., local site 2002 and remote site 2060), network address translation is not required. Even if address translation is performed, firewall rules may not be required. For such a compatible infrastructure, in some embodiments, remote site 2060 may be implemented as a child site with a trusted extension of the infrastructure of local site 2002 and connected to the local site using, for example, a metro link.

[0320] For example, if the infrastructure (e.g., infrastructure 0) of the local site 2002 and the infrastructure (e.g., infrastructure 1) of the remote site 2060 are identical to or compatible with each other, vPLC-related information, such as the vPLC ID of vPLC.R1 and the tenancy ID (and / or VCN ID) of the requesting customer of reseller R1, may be encapsulated (or tagged) into packets originating from compute instances in vPLC.R1 VCN 2031 by a border device, such as gateway 2040, at the local site before being sent to the remote site via communication channel 2050.

[0321] After receiving the packet, a border device such as gateway 2070 at the remote site may decapsulate the packet, verify the vPLC ID, determine the destination VCN at the remote site 2060, and forward the decapsulated packet accordingly. In some embodiments, the encapsulation and decapsulation of the packet may be performed by a network virtualization device (NVD) such as a smart NIC at each site (i.e., local site 2030 and remote site 2060). The NVD may also be used to provide network virtualization primitives for creating VCNs.

[0322] Similarly, vPLC-related information, such as the vPLC ID of vPLC R1 and the tenancy ID (and / or VCN ID) of the requesting customer of reseller R1, may be encapsulated (or tagged) into packets originating from a compute instance in remote VCN 2064 by a border device, such as gateway 2070, at the remote site before sending the packets to the local site over communication channel 2050. After receiving the packets, a border device, such as gateway 2040 at local site 2002, may decapsulate the packets to determine the particular vPLC associated with the vPLC ID and the destination VCN for the requesting customer of reseller R1 at local site 2060, and forward the decapsulated packets accordingly.

[0323] If the infrastructure networks are not compatible between both sites (i.e., local site 2002 and remote site 2060), network address translation (NAT) and lookup tables may be required. For example, if the infrastructure (e.g., Infrastructure 0) of local site 2002 and the infrastructure (e.g., Infrastructure 1) of remote site 2060 are not compatible with each other, the remote site may not have an NVD, and a VLAN identifier (or VLAN ID or VLAN tag) may be used to encapsulate / decapsulate packets. In some embodiments, a lookup table containing VLAN ID and VCN ID mapping information 2072 at remote site 2060 may be used to determine the VCN associated with a VCN ID for a particular VLAN ID in a packet. Because VCNs are owned by customer tenancies and each customer tenancy belongs to a vPLC, the vPLC to which a packet belongs at local site 2002 may ultimately be determined by the packet's VLAN ID. VLAN ID / VCN ID mapping information 2072 is available for incompatible infrastructure networks between both sites and is shown as a dashed box in FIG.

[0324] For example, a packet originating from a compute instance in vPLC.R1 VCN 2031 may encapsulate similar vPLC-related information (e.g., vPLC ID, tenancy ID, and VCN ID) as described above, since the local site 2002 has all the information associated with the requesting customer of reseller R1. After receiving the packet, a border device, such as gateway 2070, at the remote site 2060 may decapsulate the packet, verify the vPLC ID, determine the destination VCN at the remote site 2060 based on the VCN ID, and forward the decapsulated packet accordingly. In some embodiments, if the remote site does not have knowledge of the VCN ID, a border device (e.g., gateway 2040) at the local site may perform a VCN ID-to-VLAN ID lookup and replace the VCN ID with the corresponding VLAN ID or add the looked-up VLAN ID to the packet header before sending the packet to the remote site over communication channel 2050.

[0325] However, for packets originating from a compute instance in the remote VCN 2064 in response to a requesting customer of reseller R1, a VLAN ID may be used to encapsulate the packet because the remote site does not have an NVD and does not know the vPLC ID. The packet may first be encapsulated with a VLAN ID and sent to a border device, such as gateway 2070 at the remote site 2060, which knows how to prepare the packet for transmission across the communication channel 2050 (e.g., a FastConnect connection). The gateway 2070 at the remote site 2060 can decapsulate the packet, remove the VLAN tag, and use the VLAN / VCN mapping information 2072 to look up the vPLC ID, tenancy ID, VCN ID, and IP address of a border device, such as gateway 2040, at the local site 2002. The gateway 2070 at the remote site 2060 can then re-encapsulate the discovered information into a packet and send it across the communication channel 2050 to the local site 2002.

[0326] Once the packet is received by a border device (e.g., gateway 2040) at local site 2002 and decapsulated to obtain vPLC-related information (e.g., vPLC ID), gateway 2040 may continue forwarding the packet via the next hop using encapsulation / decapsulation to reach the NVD of the vPLC (e.g., vPLC.R1) associated with the vPLC ID. The NVD then decapsulates the packet to find the destination VCN ID of the particular VCN (e.g., vPLC.R1 VCN) associated with the requesting customer of reseller R1 and forwards the packet accordingly.

[0327] Connectivity between resources allocated at the remote site 2060 and resources allocated at the local site 2002 is provided by packet encapsulation / decapsulation so that resources in one subnet at the remote site appear to exist on an extension of a subnet of a VCN hosted in a vPLC at the local site 2002. A CSP can achieve this result by publishing a vPLC-aware packet encapsulation / decapsulation protocol that allows allowed traffic to flow between the VCN implementation at the local site 2002 and the VCN implementation at the remote site 2060.

[0328] FIG. 21 is a block diagram illustrating a local vPLC-CP model, according to one embodiment. The distributed environment 2100 illustrated in FIG. 21 is merely an example and is not intended to unduly limit the scope of the claimed embodiments. Many variations, alternatives, and improvements are possible. For example, in some embodiments, the distributed environment 2100 may include more or fewer systems or components than those illustrated in FIG. 21, may combine two or more systems, or may have different system configurations or arrangements. The systems, subsystems, and other components illustrated in FIG. 21 may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective systems, using hardware, or a combination thereof. Software may be stored in a non-transitory storage medium (e.g., a memory device).

[0329] Figures 20 and 21 have the same architecture. Both have the same CSP provisioning infrastructure in region 2102 and remote site 2160, the same CSP namespace 0 2110 and vPLC.R1 namespace 1 2130 with their corresponding components (e.g., CP, RM, MP, and vPLC.R1 DP), and the same communication channel 2150. The difference between Figures 20 and 21 is the control plane used to control and provide access to the local vPLC.R1 DP 2138 and remote DP 2162. In Figure 20, the primary CP (i.e., CSP CP) 2012 fulfills this role. In contrast, in Figure 21, the vPLC.R1-specific CP 2132 fulfills this role.

[0330] As shown in FIG. 21 , a set of endpoints 2104 associated with vPLC.R1 may be used to receive requests from reseller R1's customers 2103 to access resources or perform functions involving resources, and may be located in vPLC.R1 namespace 1 2130 or in a remote DP 2162. All requests may be intercepted by a vPLC-specific CP (i.e., vPLC.R1-specific CP) 2132 before being sent to main CSP CP 2112. The vPLC.R1-specific CP may determine, on behalf of reseller R1, whether a local DP 2138 or a remote DP 2162 is best suited for the customer's request. This determination may result in two different routes: a route 2180 to the local DP 2138 and a route 2182 to the remote DP 2162. In some embodiments, both the local DP 2138 and the remote DP 2162 may be used for the customer's request.

[0331] Upon receiving a customer request, the vPLC.R1 specific CP 2132, in cooperation with the vPLC.R1 RM 2134, may determine whether local resources of the local vPLC.R1 DP 2138 are authorized and available to fulfill the request. If the local DP can fulfill the request, the request may proceed along route 2180. The vPLC.R1 specific CP 2132 may forward the request to the main CP (or CSP CP) 2112, and the request may then follow the process shown in route 2080 of FIG. 20 by passing through the local CSP MPA 2116, the local CSP MP 2118, and accessing the local vPLC.R1 DP 2138.

[0332] On the other hand, if the local vPLC.R1 DP 2138 cannot fulfill the request or if a remote DP is a better option, the request may proceed along route 2182. In this case, the vPLC.R1-specific CP 2132 may request the vPLC.R1 MP 2136 to send the request over communication channel 2150 to the remote VCN 2164 at the remote site 2160. The vPLC.R1 MP 2136 may then cooperate with the remote MP 2166 to have the request fulfilled by the remote DP 2162. Once the request is fulfilled, a response may be sent from the remote MP 2166 back to the vPLC.R1 MP 2136 over communication channel 2150. The vPLC-specific CP 2132 can then notify the reseller R1's customer 2103 of the results.

[0333] In this local vPLC-CP model, the main CSP CP 2112 is not involved if a request is determined to be destined for the remote site 2182. In other words, the vPLC.R1 specific CP 2132 has complete control over how the request is routed.

[0334] Communications between the CSP-provided regional infrastructure (i.e., local site) 2102 and the remote site 2160 are the same as those described in FIG. 20, with the use of packet encapsulation-decapsulation enabling transparent network packet flow through the communications channel 2150.

[0335] FIG. 22 is a block diagram illustrating a remote vPLC-CP model, according to one embodiment. The distributed environment 2200 illustrated in FIG. 22 is merely an example and is not intended to unduly limit the scope of the claimed embodiments. Many variations, alternatives, and improvements are possible. For example, in some embodiments, the distributed environment 2200 may include more or fewer systems or components than those illustrated in FIG. 22, may combine two or more systems, or may have different system configurations or arrangements. The systems, subsystems, and other components illustrated in FIG. 22 may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective systems, using hardware, or a combination thereof. Software may be stored in a non-transitory storage medium (e.g., a memory device).

[0336] FIG. 22 may be similar to FIG. 21 in that it includes a CSP-provided infrastructure in region 2202, a remote site 2260, and communication channel 2250. However, the components of vPLC.R1 namespace 1 2230 have moved to remote site 2260, except for local vPLC.R1 DP (i.e., local DP) 2238, which remains in CSP-provided regional infrastructure 2202. In other words, vPLC.R1 CP 2232, vPLC.R1 RM 2234, and vPLC.R1 MP 2236 reside at remote site 2260 and perform similar functions as these components in FIG. 21 (i.e., vPLC.R1 CP 2132, vPLC.R1 RM 2134, and vPLC.R1 MP 2136). The vPLC.R1 CP 2232 at the remote site 2260 (referred to as the remote vPLC.R1 CP) may be responsible for controlling and providing access to the local vPLC.R1 DP 2238 and the remote DP 2262, although the term "local" still comes from the perspective of the CSP-provided regional infrastructure 2202.

[0337] In some embodiments, the remote vPLC.R1 CP 2232, vPLC.R1 RM 2234, and vPLC.R1 MP 2236 may be created using resources within the remote site 2260, but are referred to as the remote vPLC.R1 CP because this CP is used to access the local DP 2238 of namespace 1 of vPLC.R1 2230 across communication channel 2250. In other embodiments, the remote vPLC.R1 CP 2232, vPLC.R1 RM 2234, and vPLC.R1 MP 2236 may be created using resources at the local site 2202, as long as there is network connectivity between the local network of the CSP at the local site 2202 and the remote vPLC.R1 CP 2232. This may be achieved by pointing the load balancer in the vPLC.R1 VCN 2268 to the remote address of the remote vPLC.R1 CP 2232.

[0338] In some embodiments, the remote vPLC-CP model may be used when the infrastructure networks of the local site 2202 and the remote site 2260 are compatible or incompatible. If the infrastructure networks of both sites are compatible, the remote site 2260 may be treated as a trusted extension of the local site 2202. If the networks of both sites are incompatible, the gateways of both sites (e.g., gateway 2240 and gateway 2270) may perform packet modification / rewriting for security purposes to prevent direct network access between infrastructure 0 of the local site 2202 and infrastructure 1 of the remote site 2260. Packet modification / rewriting may include modifying the packet's source or destination IP address, source or destination port, or other information. In some embodiments, such packet modification / rewriting may be performed for both compatible and incompatible infrastructures between the two sites. In FIG. 22 , the use of the vPLC.R1 VCN 2268 adds another security measure to prevent direct access by the remote site 2260 to the infrastructure of CSP namespace 0 2210.

[0339] 22, a set of endpoints 2204 associated with vPLC.R1 may be used to receive requests from reseller R1's customers 2203 to access resources or perform functions involving resources, and may be located at a local DP 2238 in vPLC.R1 namespace 1 2230 or at a remote DP 2262. However, access to resources in vPLC.R1 namespace 1 2230 may require traversing a communication channel 2250. In some embodiments, an additional set of endpoints 2206 associated with vPLC.R1 may be available to reseller R1's customers 2205 for accessing the local DP 2238 through a CSP CP 2212 at a local site (CSP-provided regional infrastructure) 2202.

[0340] Requests from reseller R1's customers 2203 may be directed to the remote vPLC.R1 CP, which, on behalf of reseller R1, determines whether the local vPLC.R1 DP 2238 or the remote DP 2262 is best suited for the customer's request. This determination may result in two different routes: route 2280 to the local vPLC.R1 DP 2238 and route 2282 to the remote DP 2262. In some embodiments, both the local DP 2238 and the remote DP 2262 may be used for the customer's request.

[0341] Upon receiving a customer request, the remote vPLC.R1 CP 2232, in cooperation with the vPLC.R1 RM 2234, may determine whether local resources of the local vPLC.R1 DP 2238 are authorized and available to fulfill the request. If the local DP can fulfill the request, the request may proceed along route 2280. In some embodiments, the remote vPLC.R1 CP 2232, with the assistance of the vPLC.R1 MP 2236, may establish a communication channel 2250 with the local vPLC.R1 VCN 2268 using the remote VCN 2264 to send the request to the CSP CP 2212. In other embodiments, if the remote site 2260 is a reseller's on-premises or enterprise network, a persistent communication channel such as FastConnect may exist between the remote VCN 2264 and the local vPLC.R1 VCN 2268, and the remote vPLC.R1 CP 2232 may send the request to the CSP CP 2212. When the remote vPLC.R1 CP 2232 sends the request to the CSP CP 2212, this is equivalent to delegating the access process of the local DP 2238 to the CSP CP 2212 at the local site 2210. In some embodiments, a new API at the local site may be used so that the CSP CP 2212 accesses the local vPLC.R1 DP 2238 when it receives a request from the remote site 2260 or performs some pre-processing and post-processing operations before executing the request.

[0342] CSP CP2212 may use local MP2218 to facilitate access to local vPLC.R1 DP2238. Once the request is fulfilled, a response may be sent from CSP CP2212 (or local MP2216) back to remote vPLC.R1 CP2232 (or vPLC.R1 MP2236) over communication channel 2250. Remote vPLC CP2232 can then notify reseller R1's customer 2203 of the results.

[0343] On the other hand, if the local vPLC.R1 DP 2238 cannot fulfill the request or if a remote DP is a better option, the request may proceed along route 2282. In this case, the remote vPLC.R1 CP 2232 may request the vPLC.R1 MP 2236 to enable / execute a workflow that enables the remote DP 2262 to perform the appropriate function or action to fulfill the request. The remote vPLC.R1 CP 2232 may then respond to the reseller R1 customer 2203 after the request is fulfilled.

[0344] The communication between the CSP-provided regional infrastructure (i.e., local site) 2202 and the remote site 2260 is the same as that described in FIG. 20, with the use of packet encapsulation-decapsulation enabling transparent network packet flow through the communication channel 2250 so that the reseller's customers are unaware of the processes in the backend.

[0345] FIG. 23 is an exemplary flowchart illustrating a process for creating a communication channel between a CSP-provided infrastructure in a region and a remote site, according to one embodiment. The process illustrated in FIG. 23 may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective systems, using hardware, or a combination thereof. The software may be stored in a non-transitory storage medium (e.g., a memory device). The method presented in FIG. 23 and described below is exemplary and not intended to be limiting. While FIG. 23 depicts various process steps occurring in a particular order or sequence, this is not intended to be limiting. In an alternative embodiment, the process may be performed in some different order, or some steps may be performed in parallel. Of course, in alternative embodiments, the process illustrated in FIG. 23 may include more or fewer steps than those depicted in FIG. 23.

[0346] For example, in one embodiment, the process illustrated in Figure 23 may be performed by a CSP, and some steps may be performed by components within a CSP-provided regional infrastructure (e.g., a control plane (CP), a resource manager (RM), and a management plane (MP), etc.), or by one or more CSP-provided cloud services, such as a VCN service, a vPLC service, etc.

[0347] In step 2302, a request to access resources at a remote site may be received. For example, in FIG. 20, a request from reseller R1's customer 2003 may be received by primary CSP CP 2080, which, with the assistance of local RM 2014, determines that the request can be fulfilled by remote site 2060. In step 2304, remote resources at the remote site may be identified. For example, continuing the example above, in FIG. 20, reseller R1's customer 2003 may be using local DP 2038 for compute services. However, the customer may desire a disaster recovery site that is not local to CSP-provided regional infrastructure 2002. As a result, remote DP 2062 at remote site 2060 may be considered a better candidate to fulfill the customer's request.

[0348] In step 2308, a remote virtual cloud network (VCN) at the remote site may be configured to make the remote resources part of the remote VCN. For example, as described above, the remote site may provide a web services API with primitives or functions for creating VNCs and launching computing instances. In FIG. 20 , the remote site 2060 may create a remote VCN 2064 and make the customer tenancy of the requesting customer of reseller R1 part of the remote VCN 2064. Gateways at both sites (e.g., gateway 2040 and gateway 2070) may also be configured.

[0349] Also, in the case of the remote vPLC-CP model shown in FIG. 22, the remote vPLC.R1 CP2232, vPLC.R1 RM2234, and vPLC.R1 MP2236 may be configured by the remote site 2260 or the local site 2202 as described with respect to FIG. 22.

[0350] In step 2310, a network communication channel may be created between the vPLC VCN and the remote VCN. For example, in FIG. 20, if the remote VCN 2064 is part of the reseller R's corporate network, a remote persistent communication channel (such as FastConnect) may be created between the vPLC.R1 VCN 2031 and the remote VCN 2064. Customers of the reseller R1 may use the network services of the reseller-supplied cloud service by providing necessary information such as a Dynamic Routing Gateway (DRG) associated with the VCN, an IP address, etc.

[0351] In step 2312, vPLC-specific CPs, RMs, and MPs may be configured to be part of the vPLC VCN. For example, in Figure 21, vPLC.R1-specific CP 2132, vPLC.R1 RM 2134, and vPLC.R1 MP 2136 are part of the vPLC.R1 VCN 2131 such that vPLC.R1 RM 2134 tracks remote resources of remote DP 2162 and vPLC.R1 MP 2136 communicates with remote MP 2166 to monitor the fulfillment of customer requests by remote DP 2162.

[0352] FIG. 24 is an exemplary flowchart illustrating a method for processing requests in the CSP-CP model, according to one embodiment. The process illustrated in FIG. 24 may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of a respective system, using hardware, or a combination thereof. The software may be stored in a non-transitory storage medium (e.g., a memory device). The method presented in FIG. 24 and described below is exemplary and not intended to be limiting. While FIG. 24 depicts various process steps occurring in a particular order or sequence, this is not intended to be limiting. In an alternative embodiment, the process may be performed in some different order, or some steps may be performed in parallel. Of course, in alternative embodiments, the process illustrated in FIG. 24 may include more or fewer steps than those depicted in FIG. 24.

[0353] For example, in one embodiment, the process shown in Figure 24 may be performed by a CSP, a vPLC, or a remote infrastructure (e.g., remote site 2060 of Figure 20). For example, some steps may be performed by components within a CSP-provided regional infrastructure (e.g., a control plane (CP), a resource manager (RM), and a management plane (MP), etc.) or by one or more CSP-provided cloud services. Other steps may be performed by components within the remote infrastructure.

[0354] In step 2410, receipt of the request by a primary control plane (CP) at the local site may cause an operation involving resources associated with the vPLC to be performed. For example, in Figure 20, a customer 2003 of reseller R1 may send a request for vPLC R1 (e.g., to deploy five VMs on a server) to primary CP 2012 through one of a set of endpoints 2004.

[0355] In step 2412, the primary CP determines whether resources in the local data plane (DP) of the vPLC or the remote DP at the remote site will be used to perform the requested operation. If the determination in step 2414 is that the local DP is used, the process proceeds to step 2430. If the determination is that the remote DP is used, the process proceeds to step 2450. For example, in FIG. 20 , the primary CP 2012 may query the local RM 2014 to check whether the local DP 2038 or the remote DP 2062 can fulfill a request to place five VMs on a server. In some embodiments, the local RM 2014 may request assistance from the vPLC.R1 RM 2034 to check the resource allocation history of the remote DP 2062. In other words, the primary CP, in cooperation with the local RM, performs a process to determine whether the requested operation is allowed and whether resources are available for the operation to be performed at the local DP or the remote DP.

[0356] For example, assume that the resource allocation policy for vPLC.R1 limits the maximum number of VMs per request to 10, and the local DP 2038 has three available VMs, while the remote DP 2062 has ten available VMs. The local RM may first check the resource allocation policy for vPLC.R1 and determine that the request is permitted / allowed. The local RM may then check availability and determine that the local DP does not have enough resources to fulfill the request (i.e., the three remaining VMs cannot fulfill the request for five VMs), but the remote DP has enough resources to fulfill the request (i.e., the ten remaining VMs can fulfill the request for five VMs). This allows the local RM 2014 to inform the primary CP that only the remote DP can perform the requested operation to place five VMs on the server.

[0357] In the case of a local DP, in step 2430, the primary CP requests the local MP to enable a workflow that performs the requested operation. For example, in FIG. 20 , after the primary CP 2012 receives a determination from the local RM 2014 that the request should proceed along route 2080, the primary CP 2012 may request the local MP 2018 to enable or provision a workflow that performs the requested operation (e.g., deploying five VMs on a server). In step 2432, the requested operation may be performed at the local DP of the vPLC. For example, in FIG. 20 , the local MP 2018 may obtain the resource ID of the identified server and the tenancy ID of the requesting customer of reseller R1 from the local RM and request the local DP 2038 to provision five VMs on a hypervisor running on the identified server. Once the requested operation is complete, the local MP 2018 may notify the primary CP 2012, which may then respond to the requesting customer 2003 of reseller R1.

[0358] In the case of a remote DP, in step 2450, the primary CP requests a connection to the remote site from a vPLC-specific CP in the vPLC's VCN. For example, in FIG. 20 , the primary CP 2012 may send a request for connection to the remote site 2060 to the vPLC.R1 CP 2032 to fulfill the request. In step 2452, the vPLC.R1 CP queries the vPLC.R1 RM 2036 for additional checks. For example, as described above, the vPLC.R1 RM may track the resource allocation history of the remote DP 2062. This allows the vPLC.R1 RM to match the resource allocation policy for vPLC.R1 and the resource availability of the remote DP for a particular customer 2003 of reseller R1. In some embodiments, the vPLC.R1 RM may check the vPLC.R1 resource DB, which contains the resource allocation and provisioning history of the remote DP 2062.

[0359] In step 2454, the vPLC-specific CP requests a vPLC-specific MP in the vPLC's VCN to communicate to a remote MP at a remote site. For example, in FIG. 20 , vPLC.R1 2032 may request vPLC.R1 MP 2036 to provision a workflow that can communicate the request to remote MP 2066 through communication channel 2050 established between vPLC.R1 VCN 2031 and remote VCN 2064. This communication may include packet encapsulation / decapsulation by border devices (e.g., gateways) at both local site 2002 and remote site 2060, as described above.

[0360] In step 2456, the remote MP performs the requested operation at the remote DP at the remote site. For example, in FIG. 20 , after receiving the request from vPLC.R1 MP 2036, the remote MP 2066 may provision a workflow to create five VMs on the server of the remote DP 2062. Once the requested operation is complete, the remote MP 2066 may notify the vPLC.R1 MP 2036 through communication channel 2050, which may relay the notification through the vPLC.R1 CP 2032 to the primary CP 2012. As a result, the primary CP 2012 is responsive to the requesting customer 2003 of reseller R1.

[0361] FIG. 25 is an exemplary flowchart illustrating a method for processing requests in a local vPLC-CP model, according to one embodiment. The process illustrated in FIG. 25 may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective systems, using hardware, or a combination thereof. The software may be stored in a non-transitory storage medium (e.g., a memory device). The method presented in FIG. 25 and described below is exemplary and not intended to be limiting. While FIG. 25 depicts various process steps occurring in a particular order or sequence, this is not intended to be limiting. In an alternative embodiment, the process may be performed in some different order, or some steps may be performed in parallel. Of course, in alternative embodiments, the process illustrated in FIG. 25 may include more or fewer steps than those depicted in FIG. 25.

[0362] For example, in one embodiment, the process shown in Figure 24 may be performed by a CSP, a vPLC, or a remote infrastructure (e.g., remote site 2160 of Figure 21). For example, some steps may be performed by components within a CSP-provided regional infrastructure (e.g., a control plane (CP), a resource manager (RM), and a management plane (MP), etc.) or by one or more CSP-provided cloud services. Other steps may be performed by components within the remote infrastructure.

[0363] In step 2510, receipt of the request by a vPLC-specific control plane (CP) at the local site may cause an operation involving resources associated with the vPLC. For example, in Figure 21, a customer 2103 of reseller R1 may send a request in vPLC.R1 (e.g., to place five VMs on a server) to vPLC.R1 CP 2132 through one of a set of endpoints 2104.

[0364] In step 2512, the vPLC-specific CP determines whether resources in the vPLC's local data plane (DP) or a remote DP at a remote site will be used to perform the requested operation. If the determination in step 2514 is that the local DP is used, the process proceeds to step 2530. If the determination is that the remote DP is used, the process proceeds to step 2550. For example, in FIG. 21 , the vPLC.R1 CP 2132 may query the vPLC.R1 RM 2134 to check whether the local DP 2138 or the remote DP 2162 can fulfill a request to place five VMs on a server. In some embodiments, the vPLC.R1 RM 2134 may request assistance from the local RM 2114 to check the resource allocation history of the local DP 2138. In other words, the vPLC.R1 CP, in cooperation with the vPLC.R1 RM, performs processing to determine whether the requested operation is allowed and whether resources are available for the operation to be performed at the local DP or the remote DP.

[0365] In the case of a local DP, the vPLC-specific CP forwards the request to the main CP of the local site in step 2530. For example, in FIG. 21, after the vPLC.R1 CP 2132 receives a determination from the vPLC.R1 RM 2134 that the request should proceed along route 2180, the vPLC.R1 CP 2132 may forward the request to the main CP (or CSP CP) 2112. The process proceeds to step 2532, where it executes steps 2430 and 2432 of FIG. 24 by requesting the local MP 2118 to provision a workflow that will perform the requested operation in the vPLC.R1's local DP 2138.

[0366] In the case of a remote DP, in step 2550, the vPLC-specific CP requests the vPLC-specific MP of the vPLC to communicate to the remote MP at the remote site. For example, in FIG. 21 , the vPLC.R1 CP 2132 may request the vPLC.R1 MP 2136 to provision a workflow that can communicate the request to the remote MP 2166 over the communication channel 2150 established between the vPLC.R1 VCN 2131 and the remote VCN 2164. This communication may include encapsulation / decapsulation of packets by border devices (e.g., gateways) at both the local site 2102 and the remote site 2160, as described above.

[0367] In step 2552, the remote MP performs the requested operation at the remote DP at the remote site. For example, in FIG. 21 , after receiving the request from vPLC.R1 MP 2136, the remote MP 2166 may provision a workflow to create five VMs on the server of the remote DP 2162. Once the requested operation is complete, the remote MP 2166 may notify the vPLC.R1 MP 2136 through communication channel 2150, which may relay the notification to the vPLC.R1 CP 2132 to respond to the requesting customer 2103 of reseller R1.

[0368] FIG. 26 is an exemplary flowchart illustrating a method for processing requests in a remote vPLC-CP model, according to one embodiment. The process illustrated in FIG. 26 may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective systems, using hardware, or a combination thereof. The software may be stored in a non-transitory storage medium (e.g., a memory device). The method presented in FIG. 26 and described below is exemplary and not intended to be limiting. While FIG. 26 depicts various process steps occurring in a particular order or sequence, this is not intended to be limiting. In an alternative embodiment, the process may be performed in some different order, or some steps may be performed in parallel. Of course, in alternative embodiments, the process illustrated in FIG. 26 may include more or fewer steps than those depicted in FIG. 26.

[0369] For example, in one embodiment, the process shown in Figure 25 may be performed by a CSP, a vPLC, or a remote infrastructure (e.g., remote site 2160 of Figure 21). For example, some steps may be performed by components within a CSP-provided regional infrastructure (e.g., a control plane (CP), a resource manager (RM), and a management plane (MP), etc.) or by one or more CSP-provided cloud services. Other steps may be performed by components within the remote infrastructure.

[0370] In step 2610, a request to perform an operation involving resources of a vPLC (e.g., deploying a VM on a server) is received at a remote site. For example, in Figure 22, a customer 2203 of reseller R1 may send a request in vPLC.R1 (e.g., deploying five VMs on a server) to remote vPLC.R1 CP 2232 through one of a set of endpoints 2204.

[0371] In step 2614, the CP at the remote site determines whether resources in the local data plane (DP) of the vPLC or the remote DP at the remote site will be used to perform the requested operation. If the determination in step 2616 is a remote DP, the process proceeds to step 2620. If the determination is a local DP, the process proceeds to step 2630. For example, in FIG. 22 , the remote vPLC.R1 CP 2232 may query the vPLC.R1 RM 2234 to check whether the local DP 2238 or the remote DP 2262 can fulfill a request to place five VMs on a server. In some embodiments, the vPLC.R1 RM 2234 may request assistance from the local RM 2214 to check the resource allocation history of the local DP 2238. In other words, the remote vPLC.R1 CP, in cooperation with the vPLC.R1 RM, performs processing to determine whether the requested operation is allowed and whether resources are available for the operation to be performed at the local DP or the remote DP.

[0372] In the case of a remote DP, in step 2620, the remote CP performs the operation at the remote DP by using the remote vPLC MP. For example, in FIG. 22, after the remote vPLC.R1 CP 2232 receives a determination from the vPLC.R1 RM 2234 that the request should proceed along route 2282, the remote vPLC.R1 CP 2232 may request the vPLC.R1 MP 2236 to provision a workflow that performs the requested operation (e.g., deploying five VMs on a server) at the remote DP 2262. Once the requested operation is complete, the vPLC.R1 MP 2236 may notify the remote vPLC.R1 CP 2232, which may respond to the requesting customer 2203 of reseller R1.

[0373] In the case of a local DP, in step 2630, the remote CP requests the remote vPLC MP to communicate to the CSP CP at the local site over a network communication channel. For example, in FIG. 22 , in some embodiments, the remote vPLC.R1 CP 2132 may request the vPLC.R1 MP 2236 to provision a workflow that can communicate the request to the CSP CP 2212 over the communication channel 2150 established between VCN 2268 and the remote VCN 2264. This communication may include packet encapsulation / decapsulation by border devices (e.g., gateways) at both the local site 2202 and the remote site 2260, as described above. As described above, in some embodiments, if a persistent communication channel, such as FastConnect, exists between the remote VCN 2264 and the local vPLC.R1 VCN 2268, the remote vPLC.R1 CP 2232 can send the request to the CSP CP 2212.

[0374] In step 2632, the requested operation is performed at the local DP of the vPLC at the local site. In Figure 22, CSP CP2212 may request local MP2218 to provision a workflow that causes five VMs to be created on the server of local DP2238. Once the requested operation is complete, CSP CP2212 (or local MP2218) may notify remote vPLC.R1 CP2232 (or vPLC.R1 MP2236) through communication channel 2250 so that the remote vPLC.R1 CP2232 can respond to the requesting customer 2203 of reseller R1.

[0375] Exemplary CSPI Architecture As mentioned above, infrastructure as a service (IaaS) is a specific type of cloud computing. IaaS can be configured to provide virtualized computing resources over a public network (e.g., the Internet). In the IaaS model, a cloud computing provider can host infrastructure components (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer), etc.). In some cases, the IaaS provider may also supply various services associated with these infrastructure components (example services include billing software, monitoring software, logging software, load balancing software, clustering software, etc.). Accordingly, because these services can be policy-driven, IaaS users can maintain application availability and performance by implementing policies that drive load balancing.

[0376] In some cases, IaaS customers can access resources and services over a wide area network (WAN) such as the Internet and use the cloud provider's services to install other elements of their application stack. For example, a user can log in to an IaaS platform to create virtual machines (VMs), install operating systems (OSs) on each VM, deploy middleware such as databases, create storage buckets for workloads and backups, and even install enterprise software on those VMs. The customer can then use the provider's services to perform a variety of functions, such as distributing network traffic, troubleshooting application issues, monitoring performance, and managing disaster recovery.

[0377] In most cases, the cloud computing model requires the participation of a cloud provider, which may be, but need not be, a third-party service that specializes in providing IaaS (e.g., providing, renting, selling). An entity may also deploy a private cloud and become its own infrastructure service provider.

[0378] In some examples, IaaS deployment is the process of putting a new application or a new version of an application onto a provisioned application server, etc. It may also include the process of provisioning the server (e.g., installing libraries, daemons, etc.), which is often managed below the hypervisor layer (e.g., server, storage, network hardware, and virtualization) by the cloud provider. As such, the customer may be responsible for handling (e.g., on self-service virtual machines (which can be spun up on demand)), middleware, and / or application deployment, etc.

[0379] In some instances, IaaS provisioning may refer to obtaining computers or virtual hosts for use and even installing the necessary libraries or services on them. In most cases, deployment does not include provisioning, which may need to be performed first.

[0380] IaaS provisioning sometimes presents two distinct challenges. First, there is the initial challenge of provisioning an initial set of infrastructure before anything is operational. Second, there is the challenge of evolving the existing infrastructure after all the provisioning has occurred (e.g., adding new services, modifying services, removing services, etc.). Sometimes these two challenges can be addressed by allowing the declarative definition of the infrastructure configuration. In other words, the infrastructure (e.g., the required components and how they interact) can be specified by one or more configuration files. Thus, the overall topology of the infrastructure (e.g., resource dependencies and how they work together) can be declaratively described. Sometimes, once the topology is specified, workflows can be generated to configure and / or manage the various components described in the configuration files.

[0381] In some examples, the infrastructure may have many interconnected elements. For example, there may be one or more virtual private clouds (VPCs) (e.g., potential on-demand pools of configurable and / or shared computing resources), also known as a core network. Also, in some examples, there may be one or more inbound / outbound traffic group rules provisioned to define how inbound and / or outbound traffic of the network is configured and one or more virtual machines (VMs). Other infrastructure elements, such as load balancers, databases, etc., may also be provisioned. The infrastructure may evolve incrementally as there is a demand and / or addition of more infrastructure elements.

[0382] In some cases, employing continuous deployment techniques may enable deployment of infrastructure code across various virtual computing environments. The described techniques may also enable infrastructure management within these environments. In some examples, a service team may write code that is desired to be deployed to one or more (but often many) different production environments (e.g., across various geographic locations, possibly across the globe). However, in some examples, the infrastructure into which the code will be deployed is configured first. In some cases, provisioning can be performed manually, provisioning tools can be used to provision resources, and / or deployment tools can be used to deploy the code after the infrastructure has been provisioned.

[0383] 27 is a block diagram 2700 illustrating an example pattern of an IaaS architecture according to at least one embodiment. A service operator 2702 may be communicatively coupled to a secure host tenancy 2704, which may include a virtual cloud network (VCN) 2706 and a secure host subnet 2708. In some examples, the service operator 2702 may employ one or more client computing devices, which may be portable handheld devices (e.g., iPhones, mobile phones, iPads, computing tablets, personal digital assistants (PDAs)) or wearable devices (e.g., Google Glass head-mounted displays, etc.) running software such as Microsoft Windows Mobile and / or various mobile operating systems such as iOS, Windows Phone, Android, BlackBerry 8, Palm OS, etc., and capable of using the Internet, email, short message service (SMS), BlackBerry, or other communication protocols. Alternatively, the client computing devices may be general-purpose personal computers, examples of which include personal computers and / or laptop computers running various versions of the Microsoft Windows, Apple Macintosh, and / or Linux operating systems. The client computing devices may also be workstation computers running any of a variety of commercially available UNIX or UNIX-like operating systems, including, but not limited to, various GNU / Linux operating systems such as Google Chrome OS.Alternatively or additionally, the client computing device may be any other electronic device, such as a thin client computer, an Internet-enabled gaming system (e.g., a Microsoft Xbox game console with or without a Kinect® gesture input device), and / or a personal messaging device, capable of communicating over a network that can access VCN 2706 and / or the Internet.

[0384] VCN 2706 may include a local peering gateway (LPG) 2710 that may be communicatively coupled to a secure shell (SSH) VCN 2712 via LPG 2710 included in SSH VCN 2712. SSH VCN 2712 may include an SSH subnet 2714, and SSH VCN 2712 may be communicatively coupled to a control plane VCN 2716 via LPG 2710 included in control plane VCN 2716. SSH VCN 2712 may also be communicatively coupled to a data plane VCN 2718 via LPG 2710. Control plane VCN 2716 and data plane VCN 2718 may be included in service tenancy 2719, which may be owned and / or operated by the IaaS provider.

[0385] The control plane VCN 2716 may include a control plane demilitarized zone (DMZ) tier 2720 that serves as a perimeter network (e.g., a portion of an enterprise network between the enterprise intranet and an external network). DMZ-based servers may limit liability and help mitigate breaches. DMZ tier 2720 may also include one or more load balancer (LB) subnets 2722, a control plane app tier 2724 that may include an app subnet 2726, and a control plane data tier 2728 that may include a database (DB) subnet 2730 (e.g., a front-end DB subnet and / or a back-end DB subnet). LB subnet 2722 included in control plane DMZ layer 2720 is communicatively coupled to app subnet 2726 included in control plane app layer 2724 and to Internet gateway 2734, which may be included in control plane VCN 2716, and app subnet 2726 may be communicatively coupled to DB subnet 2730, service gateway 2736, and network address translation (NAT) gateway 2738, which are included in control plane data layer 2728. Control plane VCN 2716 may include service gateway 2736 and NAT gateway 2738.

[0386] Control plane VCN 2716 can include data plane mirror app layer 2740, which can include app subnet 2726. App subnet 2726 included in data plane mirror app layer 2740 can include virtual network interface controller (VNIC) 2742, which can run compute instance 2744. Compute instance 2744 can communicatively couple app subnet 2726 of data plane mirror app layer 2740 to app subnet 2726, which can be included in data plane app layer 2746.

[0387] Data plane VCN 2718 can include data plane app layer 2746, data plane DMZ layer 2748, and data plane data layer 2750. Data plane DMZ layer 2748 can include LB subnet 2722, which can be communicatively coupled to app subnet 2726 of data plane app layer 2746 and internet gateway 2734 of data plane VCN 2718. App subnet 2726 can be communicatively coupled to service gateway 2736 of data plane VCN 2718 and NAT gateway 2738 of data plane VCN 2718. Data plane data layer 2750 can also include DB subnet 2730, which can be communicatively coupled to app subnet 2726 of data plane app layer 2746.

[0388] The internet gateways 2734 of the control plane VCNs 2716 and data plane VCNs 2718 may be communicatively coupled to a metadata management service 2752, which may be communicatively coupled to the public internet 2754. The public internet 2754 may be communicatively coupled to a NAT gateway 2738 of the control plane VCNs 2716 and data plane VCNs 2718. The service gateways 2736 of the control plane VCNs 2716 and data plane VCNs 2718 may be communicatively coupled to cloud services 2756.

[0389] In some examples, the service gateway 2736 of the control plane VCN 2716 or the data plane VCN 2718 can make application programming interface (API) calls to the cloud services 2756 without going over the public internet 2754. The API calls from the service gateway 2736 to the cloud services 2756 can be unidirectional. The service gateway 2736 can make the API calls to the cloud services 2756, and the cloud services 2756 can send the requested data to the service gateway 2736. However, the cloud services 2756 do not have to initiate the API calls to the service gateway 2736.

[0390] In some examples, secure host tenancy 2704 may be directly connected to an otherwise isolated service tenancy 2719. Secure host subnet 2708 may communicate with SSH subnet 2714 through LPG 2710, which may allow bidirectional communication through an otherwise isolated system. Connecting secure host subnet 2708 to SSH subnet 2714 allows secure host subnet 2708 to be accessible to other entities within service tenancy 2719.

[0391] Control plane VCN 2716 may enable users of service tenancy 2719 to configure or provision desired resources. The desired resources provisioned in control plane VCN 2716 may be deployed or used in data plane VCN 2718. In some examples, control plane VCN 2716 may be isolated from data plane VCN 2718, and data plane mirror app layer 2740 of control plane VCN 2716 may communicate with data plane app layer 2746 of data plane VCN 2718 via VNIC 2742, which may be included in data plane mirror app layer 2740 and data plane app layer 2746.

[0392] In some examples, a user or customer of the system may make a request (e.g., perform a create, read, update, or delete (CRUD) operation) over the public internet 2754, which may route the request to the metadata management service 2752. The metadata management service 2752 may route the request to the control plane VCN 2716 through the internet gateway 2734. The request may be received by the LB subnet 2722 included in the control plane DMZ tier 2720. The LB subnet 2722 may determine that the request is valid, and in response to this determination, the LB subnet 2722 may send the request to the app subnet 2726 included in the control plane app tier 2724. If the request is validated and a call to the public internet 2754 is required, the call to the public internet 2754 may be sent to the NAT gateway 2738, which may make the call to the public internet 2754. Metadata that may be desirable to store with the request may be stored in the DB subnet 2730.

[0393] In some examples, the data plane mirror app layer 2740 may facilitate direct communication between the control plane VCN 2716 and the data plane VCN 2718. For example, it may be desirable to apply configuration changes, updates, or other suitable modifications to resources included in the data plane VCN 2718. The control plane VCN 2716 may perform configuration changes, updates, or other suitable modifications to resources by communicating directly with the resources included in the data plane VCN 2718 via the VNIC 2742.

[0394] In some embodiments, control plane VCN 2716 and data plane VCN 2718 may be included in service tenancy 2719. In this case, a user or customer of the system may not own or operate either control plane VCN 2716 or data plane VCN 2718. Alternatively, an IaaS provider may own or operate both control plane VCN 2716 and data plane VCN 2718, and both may be included in service tenancy 2719. This embodiment may enable network isolation that may prevent users or customers from interacting with the resources of other users or customers. This embodiment may also enable private storage of databases by users or customers of the system without having to rely on the public internet 2754, which may not have the desired level of threat prevention for storage.

[0395] In another embodiment, LB subnet 2722 included in control plane VCN 2716 can be configured to receive signals from service gateway 2736. In this embodiment, control plane VCN 2716 and data plane VCN 2718 can be configured to be called by customers of the IaaS provider without calling the public internet 2754. Customers of the IaaS provider may desire this embodiment because the databases they use can be stored in service tenancy 2719, which is controlled by the IaaS provider and can be isolated from the public internet 2754.

[0396] 28 is a block diagram 2800 illustrating another example pattern of an IaaS architecture, according to at least one embodiment. A service operator 2802 (e.g., service operator 2702 of FIG. 27) may be communicatively coupled to a virtual cloud network (VCN) 2806 (e.g., VCN 2706 of FIG. 27) and a secure host tenancy 2804 (e.g., secure host tenancy 2704 of FIG. 27), which may include a secure host subnet 2808 (e.g., secure host subnet 2708 of FIG. 27). VCN 2806 may comprise a local peering gateway (LPG) 2810, which may be communicatively coupled to a secure shell (SSH) VCN 2812 via an LPG 2710 (e.g., LPG 2710 of FIG. 27) included in SSH VCN 2812. SSH VCN 2812 can include SSH subnet 2814 (e.g., SSH subnet 2714 in FIG. 27 ), and SSH VCN 2812 can be communicatively coupled to control plane VCN 2816 via LPG 2810, which is included in control plane VCN 2816 (e.g., control plane VCN 2716 in FIG. 27 ). Control plane VCN 2816 can be included in service tenancy 2819 (e.g., service tenancy 2719 in FIG. 27 ), and data plane VCN 2818 (e.g., data plane VCN 2718 in FIG. 27 ) can be included in customer tenancy 2821, which can be owned or operated by a user or customer of the system.

[0397] The control plane VCN 2816 may include a control plane DMZ layer 2820 (e.g., the control plane DMZ layer 2720 of FIG. 27 ) that may include a LB subnet 2822 (e.g., the LB subnet 2722 of FIG. 27 ), a control plane app layer 2824 (e.g., the control plane app layer 2724 of FIG. 27 ) that may include an app subnet 2826 (e.g., the app subnet 2726 of FIG. 27 ), and a control plane data layer 2828 (e.g., the control plane data layer 2728 of FIG. 27 ) that may include a database (DB) subnet 2830 (e.g., similar to the DB subnet 2730 of FIG. 27 ). LB subnet 2822 included in control plane DMZ layer 2820 is communicatively coupled to app subnet 2826 included in control plane app layer 2824 and to Internet gateway 2834 (e.g., Internet gateway 2734 in FIG. 27 ), which may be included in control plane VCN 2816, and app subnet 2826 may be communicatively coupled to DB subnet 2830, service gateway 2836 (e.g., service gateway 2736 in FIG. 27 ), and network address translation (NAT) gateway 2838 (e.g., NAT gateway 2738 in FIG. 27 ), which are included in control plane data layer 2828. Control plane VCN 2816 may comprise service gateway 2836 and NAT gateway 2838.

[0398] Control plane VCN 2816 may include a data plane mirror app layer 2840 (e.g., data plane mirror app layer 2740 of FIG. 27 ), which may include an app subnet 2826. App subnet 2826 included in data plane mirror app layer 2840 may include a virtual network interface controller (VNIC) 2842 (e.g., VNIC 2742 of FIG. 27 ), which may run a compute instance 2844 (e.g., similar to compute instance 2744 of FIG. 27 ). Compute instance 2844 may facilitate communication between app subnet 2826 of data plane mirror app layer 2840 and app subnet 2826, which may be included in data plane app layer 2846, via VNIC 2842 included in data plane mirror app layer 2840 and VNIC 2842 included in data plane app layer 2846 (e.g., data plane app layer 2746 of FIG. 27 ).

[0399] An internet gateway 2834 included in the control plane VCN 2816 may be communicatively coupled to a metadata management service 2852 (e.g., metadata management service 2752 of FIG. 27), which may be communicatively coupled to the public internet 2854 (e.g., public internet 2754 of FIG. 27). The public internet 2854 may be communicatively coupled to a NAT gateway 2838 included in the control plane VCN 2816. A service gateway 2836 included in the control plane VCN 2816 may be communicatively coupled to cloud services 2856 (e.g., cloud services 2756 of FIG. 27).

[0400] In some examples, data plane VCN 2818 may be included in customer tenancy 2821. In this case, the IaaS provider may provide a control plane VCN 2816 for each customer, and the IaaS provider may configure a unique compute instance 2844 for each customer, which may be included in service tenancy 2819. Each compute instance 2844 may enable communication between the control plane VCN 2816 in service tenancy 2819 and the data plane VCN 2818 in customer tenancy 2821. The compute instance 2844 may enable deployment or use of resources provisioned in the control plane VCN 2816 in service tenancy 2819 in the data plane VCN 2818 in customer tenancy 2821.

[0401] In another example, a customer of the IaaS provider may have a database that resides in customer tenancy 2821. In this example, control plane VCN 2816 may include data plane mirror app tier 2840, which may include app subnet 2826. Data plane mirror app tier 2840 may reside in data plane VCN 2818, but may not reside in data plane VCN 2818. That is, data plane mirror app tier 2840 may be accessible to customer tenancy 2821, but may not reside in data plane VCN 2818, and may not be owned or operated by the customer of the IaaS provider. Data plane mirror app tier 2840 may be configured to make calls to data plane VCN 2818, but may not be configured to make calls to any entities included in control plane VCN 2816. A customer may desire deployment or use of resources in data plane VCN 2818 that have been provisioned in control plane VCN 2816, and data plane mirror app layer 2840 may facilitate the deployment or other use of the resources desired by the customer.

[0402] In some embodiments, the IaaS provider's customer can apply filters to the data plane VCN 2818. In this embodiment, the customer can determine what the data plane VCN 2818 can access, and the customer may choose to restrict access from the data plane VCN 2818 to the public internet 2854. The IaaS provider may not be able to apply filters or control the data plane VCN 2818's access to any external networks or databases. The customer's application of filters and controls to the data plane VCN 2818 included in customer tenancy 2821 can help isolate the data plane VCN 2818 from other customers and the public internet 2854.

[0403] In some embodiments, calls made by service gateway 2836 allow cloud services 2856 to access services that may not reside on the public internet 2854, the control plane VCN 2816, or the data plane VCN 2818. The connection between cloud services 2856 and the control plane VCN 2816 or the data plane VCN 2818 may not be live or continuous. Cloud services 2856 may reside on different networks owned or operated by the IaaS provider. Cloud services 2856 may be configured to receive calls from service gateway 2836 or may not be configured to receive calls from the public internet 2854. Some cloud services 2856 may be isolated from other cloud services 2856, and control plane VCN 2816 may be isolated from cloud services 2856 that may not be in the same region as the control plane VCN 2816. For example, control plane VCN 2816 may be located in "Region 1," and cloud service "Deployment 27" may be located in Region 1 and Region 2. If a call to Deployment 27 is made by service gateway 2836 included in control plane VCN 2816 located in Region 1, the call may be sent to Deployment 27 in Region 1. In this example, Control Plane VCN 2816 or Deployment 27 in Region 1 may not be communicatively coupled to or in communication with Deployment 27 in Region 2.

[0404] 29 is a block diagram 2900 illustrating another example pattern of an IaaS architecture, according to at least one embodiment. A service operator 2902 (e.g., service operator 2702 of FIG. 27) may be communicatively coupled to a virtual cloud network (VCN) 2906 (e.g., VCN 2706 of FIG. 27) and a secure host tenancy 2904 (e.g., secure host tenancy 2704 of FIG. 27), which may include a secure host subnet 2908 (e.g., secure host subnet 2708 of FIG. 27). VCN 2906 may comprise an LPG 2910, which may be communicatively coupled to an SSH VCN 2912 via an LPG 2910 (e.g., LPG 2710 of FIG. 27) included in SSH VCN 2912 (e.g., SSH VCN 2712 of FIG. 27). SSH VCN 2912 may include SSH subnet 2914 (e.g., SSH subnet 2714 in FIG. 27 ), and SSH VCN 2912 may be communicatively coupled to control plane VCN 2916 via LPG 2910 included in control plane VCN 2916 (e.g., control plane VCN 2716 in FIG. 27 ), and may be communicatively coupled to data plane VCN 2918 via LPG 2910 included in data plane VCN 2918 (e.g., data plane VCN 2718 in FIG. 27 ). Control plane VCN 2916 and data plane VCN 2918 may be included in service tenancy 2919 (e.g., service tenancy 2719 in FIG. 27 ).

[0405] The control plane VCN 2916 may include a control plane DMZ layer 2920 (e.g., control plane DMZ layer 2720 of FIG. 27 ) that may include a load balancer (LB) subnet 2922 (e.g., LB subnet 2722 of FIG. 27 ), a control plane app layer 2924 (e.g., control plane app layer 2724 of FIG. 27 ) that may include an app subnet 2926 (e.g., similar to app subnet 2726 of FIG. 27 ), and a control plane data layer 2928 (e.g., control plane data layer 2728 of FIG. 27 ) that may include a DB subnet 2930. LB subnet 2922 included in control plane DMZ layer 2920 is communicatively coupled to app subnet 2926 included in control plane app layer 2924 and to Internet gateway 2934 (e.g., Internet gateway 2734 in FIG. 27 ), which may be included in control plane VCN 2916, and app subnet 2926 may be communicatively coupled to DB subnet 2930, service gateway 2936 (e.g., service gateway in FIG. 27 ), and network address translation (NAT) gateway 2938 (e.g., NAT gateway 2738 in FIG. 27 ), which are included in control plane data layer 2928. Control plane VCN 2916 may comprise service gateway 2936 and NAT gateway 2938.

[0406] Data plane VCN 2918 may include a data plane app layer 2946 (e.g., data plane app layer 2746 in FIG. 27 ), a data plane DMZ layer 2948 (e.g., data plane DMZ layer 2748 in FIG. 27 ), and a data plane data layer 2950 (e.g., data plane data layer 2750 in FIG. 27 ). Data plane DMZ layer 2948 may include a LB subnet 2922 that may be communicatively coupled to a trusted app subnet 2960 and an untrusted app subnet 2962 of data plane app layer 2946 and an internet gateway 2934 included in data plane VCN 2918. Trusted app subnet 2960 may be communicatively coupled to a service gateway 2936 included in data plane VCN 2918, a NAT gateway 2938 included in data plane VCN 2918, and a DB subnet 2930 included in data plane data layer 2950. Untrusted app subnet 2962 may be communicatively coupled to a service gateway 2936 included in data plane VCN 2918 and a DB subnet 2930 included in data plane data layer 2950. Data plane data layer 2950 may include DB subnet 2930, which may be communicatively coupled to a service gateway 2936 included in data plane VCN 2918.

[0407] Untrusted app subnet 2962 may include one or more primary VNICs 2964(1)-2964(N), which may be communicatively coupled to tenant virtual machines (VMs) 2966(1)-2966(N). Each tenant VM 2966(1)-2966(N) may be communicatively coupled to a respective app subnet 2967(1)-2967(N), which may be included in a respective container egress VCN 2968(1)-2968(N), which may be included in a respective customer tenancy 2970(1)-2970(N). Secondary VNICs 2972(1)-2972(N), respectively, may facilitate communication between untrusted app subnet 2962, which is included in data plane VCN 2918, and the app subnets included in the respective container egress VCNs 2968(1)-2968(N). Each container output VCN 2968(1)-2968(N) may include a NAT gateway 2938 that may be communicatively coupled to the public internet 2954 (eg, public internet 2754 of FIG. 27).

[0408] An internet gateway 2934 included in the control plane VCN 2916 and the data plane VCN 2918 may be communicatively coupled to a metadata management service 2952 (e.g., metadata management system 2752 of FIG. 27 ), which may be communicatively coupled to the public internet 2954. The public internet 2954 may be communicatively coupled to a NAT gateway 2938 included in the control plane VCN 2916 and the data plane VCN 2918. A service gateway 2936 included in the control plane VCN 2916 and the data plane VCN 2918 may be communicatively coupled to cloud services 2956.

[0409] In some embodiments, data plane VCN 2918 may be integrated with customer tenancy 2970. This integration may be useful or desirable for an IaaS provider's customers, such as those who may want support for running code. A customer may provide code to run, which may be destructive, communicate with other customer resources, or otherwise have undesirable effects. In response, the IaaS provider may determine whether to run the code provided to it by the customer.

[0410] In some examples, a customer of an IaaS provider may grant temporary network access to the IaaS provider to request functionality provided in data plane app layer 2946. The code that performs this functionality may be configured to run in VMs 2966(1)-2966(N) and may not be configured to run elsewhere on data plane VCN 2918. Each VM 2966(1)-2966(N) may be connected to a single customer tenancy 2970. Each container 2971(1)-2971(N) contained in VMs 2966(1)-2966(N) may be configured to run code. In this ...

Claims

1. It is a method, To provide one or more CSP-supplied cloud services to one or more customers of a CSP by using a first portion of the infrastructure provided by a Cloud Service Provider (CSP) in a first region, This includes generating a first virtual private label cloud (vPLC) for a first reseller based on the CSP provision infrastructure, Generating the first vPLC includes allocating a second portion of the CSP provisioning infrastructure to the first vPLC. This method is By using the first vPLC, a first subset of the first reseller supply cloud service is provided to one or more customers of the first reseller, Connecting the first vPLC to a remote infrastructure outside the CSP provision infrastructure in the first region via a communication channel, A method further comprising providing a second subset of the first reseller-supplied cloud service to one or more of the first reseller's customers by using the remote infrastructure.

2. The method according to claim 1, further comprising providing access to the second subset of the first reseller supply cloud service by using a first control plane associated with and located within the first portion of the CSP provision infrastructure.

3. The method of claim 2, wherein providing access to the second subset of the first reseller supply cloud service is done using the communication channel.

4. The method according to claim 1, further comprising providing access to the second subset of the first reseller supply cloud service by using a second control plane associated with and located within the first vPLC.

5. The method according to claim 1, further comprising providing access to the first subset of the first reseller supply cloud service by using a third control plane associated with the first vPLC and located within the remote infrastructure.

6. The method according to claim 1, wherein the communication channel is between the virtual cloud network (VCN) in the first vPLC and the VCN in the remote infrastructure.

7. The method according to claim 1, wherein packets used for communication between the first vPLC and the remote infrastructure via the communication channel are tagged with vPLC-related information.

8. The method according to claim 7, wherein the vPLC-related information includes information identifying the first vPLC.

9. The method according to claim 7, wherein the underlying network of the CSP provisioning infrastructure and the underlying network of the remote infrastructure are not compatible with each other.

10. The method according to claim 9, further comprising a mapping between a VLAN identifier and the vPLC-related information.

11. It is a system, One or more processors, The system comprises one or more non-temporary computer-readable media for storing computer-executable instructions, and when the computer-executable instructions are executed by the one or more processors of the computing system, the system To provide one or more CSP-supplied cloud services to one or more customers of a CSP by using a first portion of the infrastructure provided by a Cloud Service Provider (CSP) in a first region, Based on the aforementioned CSP provision infrastructure, a first virtual private label cloud (vPLC) is generated for the first reseller, and the following is performed: Generating the first vPLC includes allocating a second portion of the CSP provisioning infrastructure to the first vPLC. The aforementioned computer executable instructions are transmitted to the system. By using the first vPLC, a first subset of the first reseller supply cloud service is provided to one or more customers of the first reseller, Connecting the first vPLC to a remote infrastructure outside the CSP provision infrastructure in the first region via a communication channel, A system that further enables the provision of a second subset of the first reseller supply cloud service to one or more of the first reseller's customers by using the remote infrastructure.

12. The system according to claim 11, further causing the system to provide access to the second subset of the first reseller supply cloud service by using a first control plane associated with and located within the first portion of the CSP provision infrastructure.

13. The system according to claim 11 or 12, further causing the system to provide access to the second subset of the first reseller supply cloud service by using a second control plane associated with and located within the first vPLC.

14. The system according to claim 11 or 12, further causing the system to provide access to the first subset of the first reseller supply cloud services by using a third control plane associated with the first vPLC and located within the remote infrastructure.

15. The system according to claim 11 or 12, wherein packets used for communication between the first vPLC and the remote infrastructure via the communication channel are tagged with vPLC-related information.

16. A program for causing one or more processors to perform the method according to any one of claims 1 to 10.

Images