Virtual Private Label Cloud Metadata Customization

Abstract

Novel techniques are disclosed for providing a vPLC-specific metadata service that includes customized vPLC-specific metadata. In some embodiments, each vPLC may generate customized metadata using its corresponding vPLC-specific customization instructions. In some embodiments, the vPLC-specific metadata service may be performed using pre-generated customized vPLC-specific metadata, on-the-fly customized metadata, pre-generated CSP-formatted metadata, or a combination thereof.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a formal application under 35 U.S.C. §119(e) of and claims the benefit of and priority to U.S. Provisional Patent Application No. 63 / 407,571, "VIRTUAL PRIVATE LABEL CLOUD," filed September 16, 2022, the entire contents of which are incorporated herein by reference.

[0002] This application is also related to the following applications, the entire contents of each of which are incorporated herein by reference:

[0003] (1) U.S. Patent Application No. 18 / 368,881, "VIRTUAL PRIVATE LABEL CLOUDS," filed concurrently with this application (Attorney Docket No. 088325-1311130 (344800US)). (2) U.S. Patent Application No. 18 / 368,863, "CONSOLE CUSTOMIZATION FOR VIRTUAL PRIVATE LABEL CLOUDS," filed concurrently with this application (Attorney Docket No. 088325-1311132 (344900US)). (3) U.S. Patent Application No. 18 / 368,870, "RESOURCE ALLOCATION FOR VIRTUAL PRIVATE LABEL CLOUDS," filed concurrently with this application (Attorney Docket No. 088325-1311140 (345000US)). (4) U.S. Patent Application No. 18 / 368,877, "ENDPOINTS FOR VIRTUAL PRIVATE LABEL CLOUDS," filed concurrently with this application (Attorney Docket No. 088325-1311138 (345100US)). (5) U.S. Patent Application No. 18 / 368,884, "IDENTITY MANAGEMENT FOR VIRTUAL PRIVATE LABEL CLOUDS," filed concurrently with this application (Attorney Docket No. 088325-1311139 (345200US)). (6) U.S. Patent Application No. 18 / 468,037, "REMOTE DATA PLANES FOR VIRTUAL PRIVATE LABEL CLOUDS," filed concurrently with this application (Attorney Docket No. 088325-1311141 (345400US)). (7) U.S. Patent Application No. 18 / 468,238, "CONNECTIVITY FOR VIRTUAL PRIVATE LABEL CLOUDS," filed concurrently with this application (Attorney Docket No. 088325-1311142 (345500US)). (8) U.S. Patent Application No. 18 / 468,047, "RESOURCE USAGE MONITORING, BILLING AND ENFORCEMENT FOR VIRTUAL PRIVATE LABEL CLOUDS," filed concurrently with this application (Attorney Docket No. 088325-1315390 (346900US)). (9) U.S. Patent Application No. 18 / 468,044, "CLOUD INFRASTRUCTURE-BASED ONLINE PUBLISHING PLATFORMS FOR VIRTUAL PRIVATE LABEL CLOUDS," filed concurrently with this application (Attorney Docket No. 088325-1315384 (346800US)). Field The present disclosure generally relates to techniques for providing cloud services. More particularly, novel techniques are disclosed for providing vPLC-specific metadata services that include customized vPLC-specific metadata. A vPLC can be generated for a cloud service provider (CSP) reseller using a CSP-provided infrastructure in a region and used to provide one or more reseller-supplied cloud services to the reseller's customers. [Background technology]

[0004] background Due to its ubiquitous presence and ease of access to critical data, cloud computing has gradually become a part of modern life and plays a vital role in our daily activities. However, just like in the early days of telecommunications, only a few companies are in the cloud infrastructure business. Therefore, there is a need to make cloud infrastructure services more widely available. Summary of the Invention

[0005] overview The present disclosure generally relates to techniques for providing cloud services. More particularly, a novel technique is disclosed for providing a vPLC-specific metadata service including customized vPLC-specific metadata. A vPLC is generated for a cloud service provider (CSP) reseller using a CSP-provided infrastructure in a region, enabling the reseller to offer one or more reseller-supplied cloud services to the reseller's customers. Various embodiments are described herein, including methods, systems, non-transitory computer-readable storage media storing programs, code, or instructions executable by one or more processors, and the like.

[0006]

[0001] The present disclosure generally relates to techniques for providing cloud services. More particularly, novel techniques are disclosed for providing cloud service provider (CSP)-supplied cloud services to CSP customers in a region provided by the CSP, enabling cloud infrastructure used to create one or more virtual private clouds (also referred to herein as virtual private label clouds or "vPLCs"). Various embodiments are described herein, including methods, systems, non-transitory computer-readable storage media storing programs, code, or instructions executable by one or more processors, and the like.

[0007] A vPLC generated in accordance with the techniques described in this disclosure can be used for a variety of purposes. For example, in one use case, a vPLC can be generated for a reseller who is a customer of a CSP and used by the reseller to provide one or more reseller-supplied cloud services to the reseller's customers. In another use case, a vPLC can be used as a virtual data center and associated with a realm different from the realm associated with the cloud infrastructure in a region provided by the CSP.

[0008] In one embodiment, a vPLC represents a virtual cloud that includes a set of resources allocated to the vPLC from a CSP-provided infrastructure in a region. One or more vPLCs can be created using the CSP-provided infrastructure in a region.

[0009] In one embodiment, a CSP provides regional infrastructure in a region to enable both the provision of cloud services to its customers and the provision of reseller-supplied cloud services to its customers. This is achieved by creating one or more vPLCs for one or more resellers from the CSP-provided regional infrastructure, and the vPLCs created for a particular reseller can be used to provide reseller-supplied and reseller-branded cloud services to a particular reseller's customers. This allows the reseller to become a provider of reseller-branded cloud services to its customers without having to invest in the infrastructure required to provide those services. The reseller, in turn, provides the reseller-branded cloud services to the reseller's customers using the CSP-provided infrastructure. In this way, a CSP-supplied cloud service can be provided to customers of the CSP using a first portion of the infrastructure that the CSP provides in a region, a first reseller-supplied cloud service can be provided to customers of the first reseller using a second portion of the infrastructure that the CSP provides in the region and that is assigned to a first vPLC created for a first reseller, a second reseller-supplied cloud service can be provided to customers of the second reseller using a third portion of the infrastructure that the CSP provides in the region and that is assigned to a second vPLC created for a second reseller, and so on.

[0010] In an embodiment, a method includes generating a first virtual private label cloud (vPLC) for a first reseller based on a cloud service provider (CSP)-provided infrastructure in a region, where generating the first vPLC includes allocating a first portion of the CSP-provided infrastructure to the first vPLC, the method further includes generating a second vPLC for a second reseller based on the CSP-provided infrastructure, where generating the second vPLC includes allocating a second portion of the CSP-provided infrastructure to the second vPLC, the method further includes providing one or more first reseller-supplied cloud services to one or more of the first resellers using the first vPLC. and providing one or more second reseller-supplied cloud services to one or more customers of the second reseller by using a second vPLC; generating a first set of metadata customization instructions for the first vPLC; generating a second set of metadata customization instructions for the second vPLC; generating first customized metadata associated with the first vPLC based on at least the first set of metadata customization instructions; and generating second customized metadata associated with the second vPLC based on at least the second set of metadata customization instructions.

[0011] In yet another embodiment, the first set of metadata customization instructions for the first vPLC is different from the second set of metadata customization instructions for the second vPLC.

[0012] In yet another embodiment, generating the first customized metadata associated with the first vPLC is performed independently from generating the second customized metadata associated with the second vPLC.

[0013] In yet another embodiment, generating the first customized metadata associated with the first vPLC occurs before receiving a request for the first customized metadata associated with the first vPLC.

[0014] In yet another embodiment, the method further includes storing first customized metadata associated with the first vPLC in a repository prior to receiving the request.

[0015] In yet another embodiment, generating first customized metadata associated with the first vPLC includes obtaining raw metadata associated with the first vPLC before receiving the request, and applying a first set of metadata customization instructions to the obtained raw metadata associated with the first vPLC before receiving the request.

[0016] In yet another embodiment, generating the first customized metadata associated with the first vPLC occurs upon receiving a request for the first customized metadata associated with the first vPLC.

[0017] In yet another embodiment, generating first customized metadata associated with the first vPLC includes, upon receiving the request, obtaining raw metadata associated with the first vPLC; and applying a first set of metadata customization instructions to the obtained raw metadata associated with the first vPLC.

[0018] In yet another embodiment, generating first customized metadata associated with the first vPLC includes obtaining raw metadata associated with the first vPLC before receiving the request, and applying a first set of metadata customization instructions to the obtained raw metadata associated with the first vPLC upon receiving the request.

[0019] In yet another embodiment, the method further includes storing first customized metadata associated with the first vPLC in a repository prior to receiving the request.

[0020] In various embodiments, a system is provided that includes one or more data processors and a non-transitory computer-readable medium that includes instructions that, when executed on the one or more data processors, cause the one or more data processors to perform some or all of one or more of the methods disclosed herein.

[0021] In various embodiments, a non-transitory computer-readable medium is provided that stores computer-executable instructions that, when executed by one or more processors of a computer system, cause the one or more processors to perform one or more methods disclosed herein.

[0022] In various embodiments, a computer program product is provided that includes computer programs / instructions that, when executed by a processor, cause the processor to perform any of the methods disclosed herein.

[0023] The techniques described above and below can be implemented in many ways and in many contexts. Several exemplary implementations and contexts are provided by reference to the drawings, as described in more detail below. However, the following implementations and contexts are only a few of many. [Brief explanation of the drawings]

[0024] [Figure 1] FIG. 1 is a high-level diagram of a distributed environment illustrating a virtual or overlay cloud network hosted by a cloud service provider infrastructure, according to one embodiment. [Figure 2] FIG. 2 is a simplified architectural diagram of physical components in a physical network within CSPI, according to one embodiment. [Figure 3] FIG. 1 illustrates an exemplary configuration within CSPI in which a host machine is connected to multiple network virtualization devices (NVDs), according to an embodiment. [Figure 4] FIG. 1 illustrates a connection between a host machine and an NVD to provide I / O virtualization that supports multi-tenancy, according to one embodiment. [Figure 5] FIG. 1 is a simplified block diagram of a physical network provided by CSPI, according to one embodiment. [Figure 6] 1 is a block diagram of a distributed environment illustrating an example of a virtual private label cloud (vPLC) hosted by a CSP-provided infrastructure in a region, according to one embodiment. [Figure 7] FIG. 1 illustrates relationships between a CSP, its direct customers, reseller customers and their associated users, and individual reseller customers and their associated users, according to one embodiment. [Figure 8] FIG. 1 illustrates the relationship between resources allocated to the tenancies of a CSP's direct customers, the tenancies of the CSP's resellers, and the tenancies of each of the resellers' customers, according to an embodiment. [Figure 9A] FIG. 1 illustrates a mechanism for storing information in the tenancies of customers of individual resellers of a CSP, according to an embodiment. [Figure 9B] FIG. 1 illustrates a mechanism for storing information in the tenancies of customers of individual resellers of a CSP, according to an embodiment. [Figure 10] 1 is a flowchart illustrating an example of a vPLC configuration process for a CSP reseller, according to an embodiment. [Figure 11] 1 is a block diagram of a distributed environment illustrating an example of a virtual private label cloud (vPLC) hosted by a CSP-provided infrastructure in a region, according to one embodiment. [Figure 12] FIG. 1 is a block diagram illustrating a vPLC segmentation model for resource partitioning, according to an embodiment. [Figure 13]FIG. 1 is a block diagram illustrating a vPLC nesting model for resource partitioning, according to an embodiment. [Figure 14] FIG. 10 is a flow diagram illustrating a sign-up process for a customer of a reseller associated with a vPLC, according to one embodiment. [Figure 15] FIG. 10 is a flow diagram illustrating a login process for a user associated with a customer of a reseller associated with a vPLC, according to an embodiment. [Figure 16] FIG. 1 is a flow diagram illustrating a resource allocation and provisioning process for a reseller's customer associated with a vPLC, according to an embodiment. [Figure 17] FIG. 1 is a flow diagram illustrating a resource allocation and provisioning process for a reseller's customer associated with a vPLC, according to an embodiment. [Figure 18] FIG. 1 is a simplified diagram illustrating a use case in which a vPLC is hosted by the infrastructure of a first realm while associated with a second, different realm, according to one embodiment. [Figure 19] 1 is a flowchart illustrating an exemplary method for creating a vPLC hosted by a CSP-provided infrastructure in a particular region in a particular realm, while virtually associated with a different region in a different realm, according to an embodiment. [Figure 20] 1 is a block diagram illustrating a distributed environment for providing metadata services in accordance with an embodiment. [Figure 21] 1 is an exemplary flowchart illustrating a metadata service configuration process according to an embodiment. [Figure 22] 1 is an exemplary flowchart illustrating a metadata collection process for a metadata service according to an embodiment. [Figure 23] 10 is an exemplary flowchart illustrating a vPLC-specific metadata response process of a metadata service according to an embodiment. [Figure 24]FIG. 1 is a block diagram illustrating a pattern for implementing a service-based cloud infrastructure system, according to at least one embodiment. [Figure 25] FIG. 1 is a block diagram illustrating another pattern for implementing a service-based cloud infrastructure system, according to at least one embodiment. [Figure 26] FIG. 1 is a block diagram illustrating another pattern for implementing a service-based cloud infrastructure system, according to at least one embodiment. [Figure 27] FIG. 1 is a block diagram illustrating another pattern for implementing a service-based cloud infrastructure system, according to at least one embodiment. [Figure 28] FIG. 1 is a block diagram illustrating an exemplary computer system according to at least one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0025] Detailed Description In the following description, for purposes of explanation, specific details are set forth in order to provide a thorough understanding of certain embodiments. However, it will be apparent that various embodiments may be practiced without these specific details. The drawings and this description are not intended to be limiting. Use of the word "exemplary" herein means "serving as an example, instance, or illustration." Any embodiment or design described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments or designs.

[0026]

[0001] The present disclosure generally relates to techniques for providing cloud services. More particularly, novel techniques are disclosed for providing cloud service provider (CSP)-supplied cloud services to CSP customers in a region provided by the CSP, enabling cloud infrastructure used to create one or more virtual private clouds (also referred to herein as virtual private label clouds or "vPLCs"). Various embodiments are described herein, including methods, systems, non-transitory computer-readable storage media storing programs, code, or instructions executable by one or more processors, and the like.

[0027] A vPLC generated in accordance with the techniques described in this disclosure can be used for a variety of purposes. For example, in one use case, a vPLC can be generated for a reseller who is a customer of a CSP and used by the reseller to provide one or more reseller-supplied cloud services to the reseller's customers. In another use case, a vPLC can be used as a virtual data center and associated with a realm different from the realm associated with the cloud infrastructure in a region provided by the CSP.

[0028] In one embodiment, a vPLC represents a virtual cloud that includes a set of resources allocated to the vPLC from a CSP-provided infrastructure in a region. One or more vPLCs can be created using the CSP-provided infrastructure in a region. In some embodiments, a virtual private label cloud (vPLC) represents a model and architecture that enables and facilitates CSP-provided, reseller-supplied, and reseller-branded cloud services within the CSP-provided infrastructure in a region.

[0029] Just as a CSP-provided infrastructure in a region provides cloud resources that can be accessed by the CSP's customers, a vPLC created for a reseller provides a set of resources that can be accessed by the reseller's customers. From the perspective of the reseller's customers, a vPLC is like a reseller-provided infrastructure in a region, providing resources that the reseller's customers can access. A vPLC is like a data center that delivers reseller-provided cloud services to the reseller's customers in the reseller-provided region.

[0030] In one embodiment, a CSP provides regional infrastructure in a region to enable both the provision of cloud services to its own customers and the provision of reseller-supplied cloud services to its customers. This is achieved by creating one or more vPLCs for one or more resellers from the CSP-provided regional infrastructure, and the vPLCs created for a particular reseller can be used to provide reseller-supplied and reseller-branded cloud services to a particular reseller's customers. This allows the reseller to become a provider of reseller-branded cloud services to its customers without having to invest in the infrastructure required to provide those services. Instead, the reseller uses the infrastructure provided by the CSP to provide reseller-branded cloud services to its customers.

[0031] In a reseller use case, a vPLC is created for the reseller by using cloud infrastructure in a region provided by a cloud service provider (CSP). The vPLC created for the reseller can be used by the reseller to provide one or more reseller-supplied cloud services to the reseller's customers. In one embodiment, the CSP-provided regional infrastructure providing the services used by the reseller to provide one or more reseller-supplied services is partitioned into securely isolated portions, and these isolated portions (or partitions) are assigned to different resellers, such that the partitions assigned to one reseller are isolated from other resellers. In this manner, a CSP-supplied cloud service can be provided to customers of the CSP using a first portion of the infrastructure provided by the CSP in a region, a first reseller-supplied cloud service can be provided to customers of the first reseller using a second portion of the infrastructure provided by the CSP in the region and assigned to a first vPLC created for a first reseller, a second reseller-supplied cloud service can be provided to customers of the second reseller using a third portion of the infrastructure provided by the CSP in the region and assigned to a second vPLC created for a second reseller, etc. Each portion of the infrastructure assigned to a reseller can also be further divided into second-level, securely isolated portions to provide multi-tenant cloud infrastructure services to customers of the resellers.

[0032] A reseller can be an entity such as a company, a business, or an individual. By using the technology described herein, a reseller entity can quickly become a cloud service provider while avoiding all of the barriers to entry discussed in the background section of this disclosure. For example, a reseller does not need to acquire the infrastructure used to provide cloud services because the provided cloud services use infrastructure provided by a CSP. To the reseller's customers, the reseller appears to be providing reseller-supplied cloud services. Thus, from the perspective of the reseller's customers, the reseller is the cloud service provider whose services the customer subscribes to. The reseller's customers may not even know or recognize the CSP whose infrastructure the reseller uses to provide reseller-supplied and reseller-branded cloud services.

[0033] The ability to provide a vPLC is offered as a service by a CSP, and one or more customers can sign up. For example, an entity that wants to provide cloud services to its customers but does not want to invest in the infrastructure required to provide the services can sign up for this CSP-supplied vPLC service. When an entity registers for a vPLC service, it becomes a customer of the CSP, but it is a "special" customer because the CSP-supplied infrastructure allocated to the entity in the form of a vPLC is used to provide the entity-supplied and branded services to the entity's customers. Thus, the customer is both a customer of the CSP and a reseller of cloud services that use the vPLC generated for the entity. For this reason, the entity is referred to as a "reseller" to distinguish it from the CSP's actual direct customers. A CSP's direct customers are entities that sign up for and consume one or more CSP-supplied and branded services, but do not use the CSP infrastructure to sell cloud services to their customers. For purposes of this disclosure, a CSP's direct customers are also referred to as "non-reseller customers" to distinguish them from resellers. With respect to the reseller, as part of the vPLC service, the CSP will provide infrastructure to the reseller in the form of vPLC, which the reseller will use to provide the reseller-branded cloud services to the reseller's customers.

[0034] A vPLC can also be used for a variety of other purposes that do not involve a reseller. As one example, a vPLC can be created and associated with a realm that is different from the realm associated with the CSP-provided regional infrastructure used to create the vPLC. For example, the CSP-provided regional infrastructure may be associated with a first realm, and the vPLC may be associated with a second realm that is different from the first realm. In such a scenario, the infrastructure corresponding to the vPLC is hosted by the first realm but is considered to belong to the second realm. This can be used for a variety of different purposes. For example, because the vPLC is associated with the second realm, it can communicate with other infrastructure in the second realm (e.g., other data centers). Thus, data centers in the second realm can communicate with the vPLC because they reside in the same realm and share the trust and identity profile of the second realm.

[0035] The creation and management of a vPLC is performed using infrastructure and services provided by a CSP, a highly complex technical task. When a vPLC is created for a reseller, the vPLC provides technical functionality that enables the reseller to offer reseller-supplied cloud services to the reseller's customers using the vPLC, including traffic isolation between resellers, traffic isolation between the reseller's different customers, dynamic management of CSP-provided resource allocation to vPLCs, management of resource allocation allocated to vPLCs among the vPLC's different customers, metering usage of vPLCs allocated to the reseller and performing related billing functions, metering usage of vPLCs allocated among the vPLC's different customers and performing related billing functions, enabling a marketplace for different resellers, and performing identity management functions for resellers and their respective customers. This disclosure describes various embodiments illustrating various architectures and corresponding methods for implementing and enabling vPLC-related functions.

[0036] A reseller may use the vPLC created for the reseller to provide reseller-supplied and reseller-branded cloud services that provide specialized services in the reseller's specialized domain. The reseller-supplied services may be tailored to various customer segments. In some embodiments, the reseller-supplied cloud services may be different from the CSP-supplied cloud services. In some other embodiments, the reseller-supplied cloud services may be based on the CSP-supplied cloud services. For example, the reseller-supplied services may be customized versions of the CSP-supplied cloud services.

[0037] For example, a first vPLC may be created in a first region from the CSP-provided infrastructure for a first reseller, the first vPLC being specialized for providing financial services. The first reseller's customers may be financial institutions, such as banks. The first reseller may then use the first vPLC to provide customized, first-reseller-branded financial cloud services to their customers. A second vPLC may be created in the first region from the same CSP-provided infrastructure for a second reseller, the second reseller's customers may be users of telecommunication services. The second reseller may then use the second vPLC to provide customized, second-reseller-branded telecommunication cloud services to their customers. In addition to the first and second vPLCs, the CSP may use the CSP-provided infrastructure in the first region to provide CSP-supplied and branded cloud services to one or more of the CSP's direct (non-reseller) customers. In this manner, use of the same CSP provisioning infrastructure in a region provides CSP-supplied and trademarked cloud services to one or more direct (non-reseller) customers of the CSP, provides customized first reseller-branded financial cloud services to customers of the first reseller, and provides customized second reseller-branded telecommunications cloud services to customers of the second reseller.

[0038] The Metadata Service (MDS) provides information about running instances in a cloud infrastructure (e.g., running compute instances, whether virtual machines or bare metal instances). The metadata information includes various details about the instance (e.g., its associated virtual network interface cards (VNICs), its associated multipath-enabled volume attachments, and any custom metadata specified by the customer). The MDS can also provide information that a customer can use for various system initialization tasks. For example, after launching a compute instance, a user can call specific APIs to request specific data associated with the launched instance in order to complete the initialization of the instance and perform the specific tasks that the instance was intended to perform.

[0039] As demand for cloud services increases, more and more businesses and enterprises are becoming CSPs and wanting to provide cloud services to their respective customers. However, if CSPs want to provide cloud services to different segments of customers, they may need to diversify their cloud services. As a result, a one-size-fits-all approach cannot meet today's cloud service needs.

[0040] The present disclosure generally relates to techniques for providing cloud services. More particularly, novel techniques are disclosed for providing a vPLC-specific metadata service that includes customized vPLC-specific metadata. A vPLC is generated for a cloud service provider (CSP) reseller using a CSP-provided infrastructure in a region, enabling the reseller to offer one or more reseller-supplied cloud services to the reseller's customers.

[0041] Using the techniques described in this disclosure, each reseller can offer a customized version of the CSP-supplied metadata service. For example, a first reseller that specializes in providing financial services can offer a reseller-supplied metadata service customized for customers such as banks and other financial institutions. A second reseller that specializes in providing telecommunications services can offer a reseller-supplied metadata service customized for customers such as telecommunications companies. Such customized metadata services can facilitate efficient system initialization and deployment for various segments of customers.

[0042] According to the techniques described in this disclosure, a vPLC-specific metadata service for multiple resellers may enable each reseller associated with a vPLC to provide customized metadata through the use of vPLC-specific customization instructions. For example, a metadata request from a user associated with a customer of the reseller may be received by one of a set of endpoints associated with the vPLC and processed by the metadata service system by collecting raw metadata from instances of the vPLC, customizing the collected raw data using vPLC-specific customization instructions to generate customized vPLC-specific metadata, and preparing a response using vPLC-specific response instructions. As a result, the requesting user associated with the customer of the reseller may receive a response that includes the customized vPLC-specific metadata.

[0043] In some embodiments, the vPLC-specific metadata services for each vPLC may be configured to run independently and concurrently by using its corresponding vPLC-specific customization commands and response commands.

[0044] In some embodiments, a vPLC-specific metadata service for a vPLC may be performed using pre-generated customized vPLC-specific metadata, where the customized vPLC-specific metadata is generated before the metadata service receives a metadata request from a user associated with a customer of a reseller associated with the vPLC.

[0045] In other embodiments, on-the-fly customized metadata may be used to perform a vPLC-specific metadata service for a vPLC, such that upon receiving a request from a user associated with a customer of the reseller, the metadata service system may collect raw metadata from the vPLC, customize the raw data using vPLC-specific customization instructions to generate customized vPLC-specific metadata, and then send it directly to the requesting user.

[0046] In other embodiments, raw metadata from a vPLC may be collected and stored in a repository before the metadata service receives a metadata request from a user associated with a customer of a reseller associated with the vPLC.

[0047] Figures 1-5 and the associated description provided in the "Exemplary Virtual Networking Architecture" section below describe networking concepts including network virtualization, foundation networks, overlay networks, VNICs, etc., and provide example environments in which certain embodiments described herein may be implemented. Figures 6-23 describe examples and embodiments related to the vPLC architecture and vPLC-specific metadata services described herein. Figures 24-27 illustrate example architectures for implementing a cloud infrastructure that provides one or more cloud services and may incorporate the teachings described herein. Figure 28 is a block diagram illustrating an example computer system or device according to at least one embodiment.

[0048] Exemplary Virtual Networking Architecture The term cloud service is generally used to describe services made available on demand (e.g., through a subscription model) by a cloud service provider (CSP) to users or customers through the use of systems and infrastructure (cloud infrastructure) provided by the CSP. Typically, the servers and systems that make up the CSP's infrastructure are separate from the customer's own on-premise servers and systems. This allows customers to use cloud services provided by CSPs without having to purchase separate hardware and software resources for the service. Cloud services are designed to provide subscribing customers with easy and scalable access to applications and computing resources without requiring the customer to invest in procuring the infrastructure used to deliver the service.

[0049] There are multiple cloud service providers offering different types of cloud services, which come in a variety of different types or models, such as Software as a Service (SaaS), Platform as a Service (PaaS), and Infrastructure as a Service (IaaS).

[0050] A customer may sign up for one or more cloud services offered by a CSP. A customer can be any entity, such as an individual, an organization, or a business. When a customer signs up or subscribes to a service offered by a CSP, a tenancy or account is created for the customer. This account allows the customer to access one or more registered cloud resources associated with the account.

[0051] As mentioned above, Infrastructure as a Service (IaaS) is a specific type of cloud computing service. In the IaaS model, a CSP provides infrastructure (called Cloud Service Provider Infrastructure, or CSPI) that customers can use to build their own customizable networks and deploy their resources. In this way, customer resources and networks are hosted in a distributed environment by the CSP-provided infrastructure. This differs from traditional computing, where customer resources and networks are hosted by customer-provided infrastructure.

[0052] CSPI may include interconnected high-performance compute resources, including various host machines, memory resources, and network resources, that make up a physical network, also referred to as an infrastructure network or base network. CSPI resources may be distributed across one or more data centers, which may be geographically distributed across one or more geographic regions. These physical resources may run virtualization software to provide a virtualized distributed environment. Virtualization creates an overlay network (also known as a software-based network, software-defined network, or virtual network) on the physical network. The CSPI physical network serves as the basis for creating one or more overlay or virtual networks on the physical network. The physical network (or infrastructure network or base network) comprises physical network devices such as physical switches, routers, computers, and host machines. An overlay network is a logical (or virtual) network that operates on the physical infrastructure network. A given physical network may support one or more overlay networks. Overlay networks typically use encapsulation techniques to distinguish traffic belonging to different overlay networks. A virtual or overlay network is also referred to as a virtual cloud network (VCN). Virtual networks are realized through the use of software virtualization technologies (e.g., hypervisors, virtualization functions realized by network virtualization devices (NVDs) (e.g., smart NICs), top-of-rack (TOR) switches, smart TORs that realize one or more functions performed by NVDs, and other mechanisms) to create a network abstraction layer that can operate over a physical network. Virtual networks can take many forms, including peer-to-peer networks, IP networks, etc. Virtual networks are typically Layer 3 IP networks or Layer 2 VLANs. This method of virtual or overlay networking is often referred to as virtual or overlay Layer 3 networking.Examples of protocols developed for virtual networks include IP-in-IP (or Generic Routing Encapsulation (GRE)), VXLAN-IETF RFC 7348 (Virtual Extensible LAN), VPN (Virtual Private Network) (e.g., MPLS Layer-3 Virtual Private Networks (RFC 4364)), VMware's NSX, and GENEVE (Generic Network Virtualization Encapsulation).

[0053] In IaaS, a CSP-provided infrastructure (CSPI) can be configured to provide virtualized computing resources over a public network (e.g., the Internet). In the IaaS model, a cloud computing service provider can host infrastructure components (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer), etc.). In some cases, the IaaS provider may also provide various services (e.g., billing, monitoring, logging, security, load balancing, clustering, etc.) associated with these infrastructure components. These services can be policy-driven, enabling IaaS users to maintain application availability and performance by implementing policies that drive load balancing. CSPI provides infrastructure and a set of complementary cloud services that enable customers to build and run a wide range of applications and services in a hosted, highly available, distributed environment. CSPI delivers high-performance compute resources and capabilities, as well as storage capacity, in a flexible virtual network that can be securely accessed from various network locations, such as the customer's on-premises network. When a customer registers or subscribes to an IaaS service offered by a CSP, a tenancy is created for that customer, which is an isolated and secure partition within CSP where the customer may create, organize, and manage their cloud resources.

[0054] Customers can build their own virtual networks using compute, memory, and network resources provided by CSPI. One or more customer resources or workloads, such as compute instances, can be deployed onto these virtual networks. For example, customers can build one or more customizable private virtual networks (referred to as virtual cloud networks (VCNs)) using resources provided by CSPI. Customers can deploy one or more customer resources, such as compute instances, into the customer VCN. Compute instances can take the form of virtual machines, bare metal instances, etc. In this way, CSPI provides infrastructure and a set of complementary cloud services that enable customers to build and run a wide range of applications and services in a hosted, highly available virtual environment. While customers do not manage or control the underlying physical resources provided by CSPI, they do control the operating system, storage, and deployed applications. Customers may have limited control over some networking components (e.g., firewalls).

[0055] The CSP may provide a console that enables customers and network administrators to use CSPI resources to configure, access, and manage resources deployed in the cloud. In some embodiments, the console provides a web-based user interface that can be used to access and manage the CSPI. In some implementations, the console is a web-based application provided by the CSP.

[0056] CSPI may support single-tenancy or multi-tenancy architectures. In a single-tenancy architecture, software (e.g., applications, databases) or hardware components (e.g., host machines or servers) serve a single customer or tenant. In a multi-tenancy architecture, software or hardware components serve multiple customers or tenants. Thus, in a multi-tenancy architecture, CSPI resources are shared among multiple customers or tenants. In a multi-tenancy situation, precautions and safeguards are taken within CSPI to ensure that each tenant's data is isolated and not visible to other tenants.

[0057] In a physical network, a network endpoint ("endpoint") represents a computing device or system that is connected to the physical network and communicates with the connected network. A network endpoint of a physical network may be connected to a local area network (LAN), a wide area network (WAN), or other types of physical networks. Examples of traditional endpoints in a physical network include modems, hubs, bridges, switches, routers, and other networking devices, physical computers (or host machines), etc. Each physical device in a physical network has a fixed network address that can be used to communicate with the device. This fixed network address can be a Layer 2 address (e.g., a MAC address), a fixed Layer 3 address (e.g., an IP address), etc. In a virtualized environment or virtual network, endpoints can include various virtual endpoints, such as virtual machines hosted by components of the physical network (e.g., hosted by a physical host machine). These virtual network endpoints are addressed by overlay addresses, such as overlay Layer 2 addresses (e.g., an overlay MAC address) and overlay Layer 3 addresses (e.g., an overlay IP address). Network overlays provide flexibility by allowing network administrators to move overlay addresses associated with network endpoints using software management (e.g., via software implementing the virtual network's control plane). Thus, unlike physical networks, in virtual networks, overlay addresses (e.g., overlay IP addresses) may be moved from one endpoint to another through the use of network management software. Because virtual networks are built on top of physical networks, communication between components of a virtual network involves both the virtual network and the underlying physical network.To facilitate such communications, CSPI components are configured to learn and store mappings from overlay addresses in the virtual network to actual physical addresses in the underlying network, and vice versa, and use these mappings to facilitate communications and encapsulate customer traffic to facilitate routing in the virtual network.

[0058] Thus, physical addresses (e.g., physical IP addresses) are associated with elements of a physical network, and overlay addresses (e.g., overlay IP addresses) are associated with entities of a virtual or overlay network. A physical IP address is an IP address associated with a physical device (e.g., a network device) of an underlying or physical network. For example, each NVD has an associated physical IP address. An overlay IP address is an overlay address associated with an entity of an overlay network, such as a compute instance in a customer's virtual cloud network (VCN). Two different customers or tenants, each with their own private VCN, can potentially use the same overlay IP address in their respective VCNs without any knowledge of each other. Both physical IP addresses and overlay IP addresses are types of real IP addresses. They are separate from virtual IP addresses. A virtual IP address is typically a single IP address that represents or maps to multiple real IP addresses. A virtual IP address provides a one-to-many mapping between the virtual IP address and multiple real IP addresses. For example, a load balancer may use a VIP to map to or represent multiple servers, each with its own real IP address.

[0059] Cloud infrastructure or CSPI is physically hosted in one or more data centers in one or more regions around the world. CSPI may include physical or underlying network components as well as virtualized components (e.g., virtual networks, compute instances, virtual machines, etc.) in a virtual network built on the physical network components. In one embodiment, CSPI is organized and hosted in realms, regions, and availability domains. A region is typically a local geographic area that includes one or more data centers. Regions are generally independent of each other and may be separated by large distances, for example, across countries or continents. For example, one region may be in Australia, another in Japan, and yet another in India. CSPI resources are divided among the regions so that each region has its own independent subset of CSPI resources. Each region may provide a set of core infrastructure services and resources, such as compute resources (e.g., bare metal servers, virtual machines, containers, and related infrastructure), storage resources (e.g., block volume storage, file storage, object storage, archive storage), networking resources (e.g., virtual cloud networks (VCNs), load balancing resources, connectivity to on-premises networks), database resources, edge networking resources (e.g., DNS), and access management and monitoring resources. Each region typically has multiple paths connecting it to other regions in the realm.

[0060] Typically, applications are deployed to the region where they are most heavily used (i.e., deployed to the infrastructure associated with that region) because using nearby resources is faster than using resources that are farther away. Applications may also be deployed to different regions for a variety of reasons, such as redundancy to mitigate the risk of region-wide events such as large-scale weather systems or earthquakes, or to meet various requirements such as jurisdictions, tax areas, and other business or societal standards.

[0061] Data centers within a region can be further organized and subdivided into availability domains (ADs). An availability domain can correspond to one or more data centers located within a region. A region can be composed of one or more availability domains. In such a distributed environment, CSPI resources are region-specific, such as a virtual cloud network (VCN), or availability domain-specific, such as a compute instance.

[0062] ADs within a region are isolated from each other, fault-tolerant, and designed to minimize the likelihood of simultaneous failures. This is achieved by ADs not sharing critical infrastructure resources such as networks, physical cables, cable paths, or cable entry points. Therefore, a failure in one AD within a region is unlikely to affect the availability of other ADs in the same region. ADs within the same region can be connected to each other via low-latency, high-bandwidth networks, enabling highly available connections to other networks (e.g., the Internet, customer on-premises networks, etc.). This allows for the creation of replica systems in multiple ADs for both high availability and disaster recovery. Cloud services use multiple ADs to ensure high availability and protect against resource failures. As the infrastructure offered by an IaaS provider grows, capacity can be added to more regions and ADs. Traffic between availability domains is typically encrypted.

[0063] In one embodiment, regions are grouped into realms. A realm is a logical collection of regions. Realms are isolated from each other and do not share any data. Regions in the same realm can communicate with each other, but regions in different realms cannot. A customer's tenancy or account using a CSP exists in a single realm and can be distributed across one or more regions belonging to that realm. Typically, when a customer signs up for an IaaS service, a tenancy or account for that customer is created in a customer-specific region (called the "home" region) within the realm. The customer can extend their tenancy across one or more other regions within the realm. A customer cannot access regions that are not in the realm in which their tenancy resides.

[0064] An IaaS provider may offer multiple realms, each corresponding to a particular set of customers or users. For example, a commercial realm may be offered for commercial customers. As another example, a realm may be offered for customers in a particular country. As yet another example, a government realm may be offered for governments, and so on. For example, a government realm may cater to a particular government and provide a higher level of security than a commercial realm. For example, OCI (Oracle Cloud Infrastructure) currently offers one realm for its commercial regions and two realms for its government cloud regions (e.g., FedRAMP and IL5 certified).

[0065] In one embodiment, an AD can be subdivided into one or more fault domains. A fault domain is a group of infrastructure resources within an AD that provides anti-affinity. Fault domains can distribute compute instances so that they do not reside on the same physical hardware within a single AD. This is known as anti-affinity. A fault domain represents a set of hardware components (computers, switches, etc.) that share a single point of failure. A compute pool is logically divided into fault domains. This ensures that a hardware failure or compute hardware maintenance event affecting one fault domain does not affect instances in other fault domains. Depending on the embodiment, the number of fault domains per AD can vary. For example, in one embodiment, each AD contains three fault domains. Fault domains act as logical data centers within an AD.

[0066] When a customer subscribes to an IaaS service, resources from CSPI are provisioned for the customer and associated with the customer's tenancy. The customer can use these provisioned resources to build private networks and deploy resources on these networks. A customer network hosted in the cloud by CSPI is called a virtual cloud network (VCN). A customer can configure one or more virtual cloud networks (VCNs) using the allocated CSPI resources. A VCN is a virtual or software-defined private network. Customer resources deployed in a customer's VCN may include compute instances (e.g., virtual machines, bare metal instances) and other resources. These compute instances may represent various customer workloads, such as applications, load balancers, databases, etc. Compute instances deployed in a VCN can communicate with publicly accessible endpoints (“public endpoints”) over a public network such as the Internet, other instances in the same VCN or other VCNs (e.g., other VCNs of the customer or VCNs not belonging to the customer), the customer's on-premises data center or network, and service endpoints and other types of endpoints.

[0067] CSPs can use CSPI to provide various services. In some cases, customers of a CSPI themselves can act as service providers and provide services using CSPI resources. Service providers may expose service endpoints characterized by identifying information (e.g., IP addresses, DNS names, and ports). Customer resources (e.g., compute instances) can consume a particular service by accessing the service endpoints exposed by that service. These service endpoints are generally publicly accessible to users over a public communications network, such as the Internet, using a public IP address associated with the endpoint. Publicly accessible network endpoints are also sometimes referred to as public endpoints.

[0068] In some embodiments, a service provider may expose a service via a service endpoint (sometimes referred to as a service endpoint). Customers of the service may then access the service using the service endpoint. In some embodiments, a given service endpoint for a service may be accessible by multiple customers intending to consume the service. In other embodiments, a customer may be given a dedicated service endpoint such that only that customer may access the service using the dedicated service endpoint.

[0069] In one embodiment, when a VCN is created, it is associated with a private overlay Classless Inter-Domain Routing (CIDR) address space (e.g., 10.0 / 16), which are broad private overlay IP addresses assigned to the VCN. A VCN includes associated subnets, route tables, and gateways. A VCN exists within a single region but can span one, more, or all availability domains in the region. A gateway is a virtual interface configured for a VCN that enables traffic to communicate between the VCN and one or more endpoints outside the VCN. One or more different types of gateways may be configured for a VCN to enable communication between different types of endpoints.

[0070] A VCN may be subdivided into one or more subnetworks, such as one or more subnets. A subnet is thus a configuration unit or subdivision that may be created within a VCN. A VCN may have one or more subnets. Each subnet within a VCN is associated with a contiguous range of overlay IP addresses (e.g., 10.0.0.0 / 24 and 10.0.1.0 / 24) that do not overlap with other subnets in the VCN and represent a subset of address space within the VCN's address space.

[0071] Each compute instance is associated with a virtual network interface card (VNIC) that enables the compute instance to participate in a subnet of a VCN. A VNIC is a logical representation of a physical network interface card (NIC). Generally, a VNIC is an interface between an entity (e.g., a compute instance, a service) and a virtual network. A VNIC resides in a subnet and has one or more associated IP addresses and associated security rules or policies. A VNIC corresponds to a Layer 2 port on a switch. A VNIC is associated with a compute instance and a subnet within a VCN. A VNIC associated with a compute instance enables the compute instance to be part of a subnet of a VCN and enables the compute instance to communicate (e.g., send and receive packets) with endpoints on the same subnet as the compute instance, endpoints in a different subnet of the VCN, or endpoints outside the VCN. Thus, a VNIC associated with a compute instance determines how the compute instance connects to endpoints inside and outside the VCN. A VNIC for a compute instance is created and associated with the compute instance when the compute instance is created and added to a subnet in a VCN. For a subnet containing a set of compute instances, the subnet contains VNICs corresponding to the set of compute instances, with each VNIC associated with one compute instance in the set of compute instances.

[0072] Each compute instance is assigned a private overlay IP address via the VNIC associated with that compute instance. This private overlay IP address is assigned to the VNIC associated with the compute instance when the compute instance is created and is used to route traffic to the compute instance. In a given subnet, all VNICs use the same route table, security list, and DHCP options. As described above, each subnet in a VCN is associated with a contiguous range of overlay IP addresses that do not overlap with other subnets in the VCN and represent a subset of address space within the VCN's address space (e.g., 10.0.0.0 / 24 and 10.0.1.0 / 24). For a VNIC on a particular subnet of a VCN, the private overlay IP address assigned to the VNIC is an address from the contiguous range of overlay IP addresses assigned to the subnet.

[0073] In some embodiments, a compute instance may be assigned a private overlay IP address and optionally additional overlay IP addresses, such as one or more public IP addresses in the case of a public subnet. These addresses are assigned to the same VNIC or multiple VNICs associated with the compute instance. However, each instance has a primary VNIC associated with an overlay private IP address that is created and assigned to the instance when the instance is launched (this primary VNIC cannot be deleted). Additional VNICs, called secondary VNICs, can be added to an existing instance in the same availability domain as the primary VNIC. All VNICs are in the same availability domain as the instance. Secondary VNICs can be in the same subnetwork as the primary VNIC, or in different subnetworks in the same or a different VCN.

[0074] If a compute instance resides in a public subnet, it may optionally be assigned a public IP address. A subnet can be designated as a public or private subnet at the time of creation. A private subnet means that resources (e.g., compute instances) and associated VNICs in the subnet cannot have public overlay IP addresses. A public subnet means that resources and associated VNICs in the subnet can have public IP addresses. Customers can specify a subnet to reside in a single availability domain or multiple availability domains of a region or realm.

[0075] As described above, a VCN can be subdivided into one or more subnets. In one embodiment, a virtual router (VR) (referred to as a VCN VR or simply VR) configured for a VCN enables communication between subnets in the VCN. For a subnet within a VCN, the VR represents the subnet's logical gateway, allowing that subnet (i.e., compute instances on that subnet) to communicate with endpoints on other subnets within the VCN and with other endpoints outside the VCN. A VCN VR is a logical entity configured to route traffic between VNICs in a VCN and a virtual gateway ("gateway") associated with the VCN. Gateways are described separately below with respect to FIG. 1. A VCN VR is a Layer 3 / IP layer concept. In one embodiment, there is one VCN VR for a VCN, and a VCN VR potentially has a limited number of ports addressed by IP addresses, one port per subnet in the VCN. As such, a VCN VR has a different IP address for each subnet in the VCN to which the VCN VR is attached. VRs are also connected to various gateways configured for the VCN. In some embodiments, a specific overlay IP address in an overlay IP address range for a subnet is reserved for a port in the VCN VR for that subnet. For example, consider two subnets in a VCN, each with an associated address range of 10.0 / 16 and 10.1 / 16. For the first subnet in the VCN with address range 10.0 / 16, an address in this range is reserved for a port in the VCN VR for that subnet. In some cases, the first IP address in this range may be reserved for a VCN VR. For example, for a subnet with overlay IP address range 10.0 / 16, IP address 10.0.0.1 may be reserved for a port in the VCN VR for that subnet. For a second subnet in the same VCN with address range 10.1 / 16, the VCN VR may have a port with IP address 10.1.0.1 for the second subnet.A VCN VR has a different IP address for each of the VCN's subnets.

[0076] In some other embodiments, each subnet within a VCN may have its own associated VR, which is addressable by the subnet through use of a reserved or default IP address associated with the VR. The reserved or default IP address may, for example, be the first IP address in a range of IP addresses associated with the subnet. VNICs in the subnet can communicate with (e.g., send and receive packets from) the VR associated with the subnet using this default or reserved IP address. In such embodiments, the VR is the ingress / egress point for the subnet. VRs associated with a subnet within a VCN can communicate with other VRs associated with other subnets in the VCN. VRs can also communicate with gateways associated with the VCN. The VR functions for a subnet are running on or performed by one or more NVDs that perform the VNIC functions for VNICs in the subnet.

[0077] A VCN can be configured with route tables, security rules, and DHCP options. Route tables are virtual route tables for a VCN and contain rules that route traffic from subnets within the VCN to destinations outside the VCN via gateways or specially configured instances. A VCN's route tables can be customized to control how packets are forwarded / routed to the VCN. DHCP options represent configuration information that is automatically provided to instances at launch.

[0078] Security rules configured for a VCN represent the overlay firewall rules for the VCN. Security rules include ingress and egress rules and can specify the type of traffic (e.g., based on protocol and port) that can enter and exit instances in the VCN. Customers can choose whether a given rule is stateful or stateless. For example, a customer can allow ingress SSH traffic from anywhere to a set of instances by configuring a stateful ingress rule with source CIDR 0.0.0.0 / 0 and destination TCP port 22. Security rules can be implemented using network security groups or security lists. A network security group consists of a set of security rules that apply only to the resources in that group. On the other hand, a security list contains rules that apply to all resources in any subnet that uses the security list. A VCN can be provided with a default security list that contains default security rules. DHCP options configured for a VCN provide configuration information that is automatically given to instances in the VCN at launch.

[0079] In one embodiment, configuration information for a VCN is determined and stored by a VCN control plane. The VCN configuration information may include, for example, information about address ranges associated with the VCN, subnets and related information within the VCN, one or more VRs associated with the VCN, compute instances and associated VNICs in the VCN, NVDs performing various virtualized network functions (e.g., VNICs, VRs, gateways) associated with the VCN, VCN state information, and other VCN-related information. In one embodiment, a VCN distribution service publishes the configuration information stored by the VCN control plane or portions thereof to the NVD. The distributed information is stored by the NVD and may be used to update information (e.g., forwarding tables, routing tables, etc.) used to forward packets to compute instances in the VCN.

[0080] In one embodiment, VCN and subnet creation is handled by a VCN control plane (CP), and compute instance launch is handled by the compute control plane. The compute control plane is responsible for allocating physical resources for the compute instance and then calls the VCN control plane to create and associate VNICs with the compute instance. The VCN CP also sends VCN data mappings to a VCN data plane, which is configured to perform packet forwarding and routing functions. In one embodiment, a VCN VP provides a distribution service responsible for providing updates to the VCN data plane. Examples of VCN control planes are also shown in Figures 1, 2, 3, and 4 (see reference numbers 116, 216, 316, and 416) and described below.

[0081] Customers can create one or more VCNs using resources hosted by CSPI. Compute instances deployed in a customer VCN can communicate with various endpoints. These endpoints can include CSPI-hosted endpoints and endpoints external to CSPI.

[0082] Various architectures for implementing cloud-based services using CSPI are shown in Figures 1, 2, 3, 4, and 5 and are described below. Figure 1 is a high-level diagram of a distributed environment 100 illustrating an overlay or customer VCN hosted by CSPI, according to one embodiment. The distributed environment shown in Figure 1 includes multiple components of an overlay network. The distributed environment 100 shown in Figure 1 is merely an example and is not intended to unduly limit the scope of the claimed embodiments. Many variations, alternatives, and modifications are possible. For example, in some embodiments, the distributed environment shown in Figure 1 may include more or fewer systems or components than those shown in Figure 1, may combine two or more systems, or may have different system configurations or arrangements.

[0083] As shown in the example of FIG. 1 , distributed environment 100 includes CSPI 101, which provides services and resources that customers can use to sign up and build their own virtual cloud networks (VCNs). In one embodiment, CSPI 101 provides IaaS services to its subscribing customers. Data centers within CSPI 101 may be organized as one or more regions. One exemplary region, “Region US” 102, is shown in FIG. 1 . A customer has configured a customer VCN 104 for region 102. The customer may deploy various compute instances into VCN 104, which may include virtual machines or bare metal instances. Example instances include applications, databases, load balancers, etc.

[0084] In the embodiment shown in FIG. 1 , customer VCN 104 includes two subnets, “Subnet 1” and “Subnet 2,” each with its own CIDR IP address range. In FIG. 1 , Subnet 1’s overlay IP address range is 10.0 / 16, and Subnet 2’s address range is 10.1 / 16. VCN virtual router 105 represents the VCN’s logical gateway, enabling communication between subnets in VCN 104 and with other endpoints outside the VCN. VCN VR 105 is configured to route traffic between VNICs in VCN 104 and the gateway associated with VCN 104. VCN VR 105 provides a port for each subnet in VCN 104. For example, VR 105 may provide a port with IP address 10.0.0.1 to Subnet 1 and a port with IP address 10.1.0.1 to Subnet 2.

[0085] Multiple compute instances may be deployed in each subnet, and the compute instances may be virtual machine instances and / or bare metal instances. The compute instances in a subnet may be hosted by one or more host machines within CSPI 101. A compute instance joins a subnet via a VNIC associated with the compute instance. For example, as shown in FIG. 1, compute instance C1 is part of subnet 1 via a VNIC associated with the compute instance. Similarly, compute instance C2 is part of subnet 1 via a VNIC associated with C2. Similarly, multiple compute instances (which may be virtual machine instances or bare metal instances) may be part of subnet 1. Each compute instance is assigned a private overlay IP address and a MAC address via its associated VNIC. For example, in Figure 1, compute instance C1 has an overlay IP address of 10.0.0.2 and a MAC address of M1, while compute instance C2 has a private overlay IP address of 10.0.0.3 and a MAC address of M2. Each compute instance in Subnet 1 (including compute instances C1 and C2) has a default route to VCN VR105 using IP address 10.0.0.1, which is the IP address of the port in VCN VR105 for Subnet 1.

[0086] Subnet2 may have multiple compute instances deployed, including virtual machine instances and / or bare metal instances. For example, as shown in FIG. 1, compute instances D1 and D2 are part of Subnet2 via VNICs associated with each of the compute instances. In the embodiment shown in FIG. 1, compute instance D1 has an overlay IP address of 10.1.0.2 and a MAC address of MM1, while compute instance D2 has a private overlay IP address of 10.1.0.3 and a MAC address of MM2. Each compute instance in Subnet2 (including compute instances D1 and D2) has a default route to VCN VR105 using IP address 10.1.0.1, which is the IP address of the port in VCN VR105 for Subnet2.

[0087] VCN A 104 may also include one or more load balancers. For example, a load balancer may be provided for a subnet and configured to load balance traffic among multiple compute instances on the subnet. A load balancer may also be provided to load balance traffic between subnets of the VCN.

[0088] A particular compute instance deployed in VCN 104 can communicate with a variety of different endpoints. These endpoints may include endpoints hosted by CSPI 200 and endpoints outside of CSPI 200. Endpoints hosted by CSPI 101 may include endpoints on the same subnet as the particular compute instance (e.g., communication between two compute instances in Subnet 1), endpoints on different subnets within the same VCN (e.g., communication between a compute instance in Subnet 1 and a compute instance in Subnet 2), endpoints in different VCNs in the same region (e.g., communication between a compute instance in Subnet 1 and an endpoint in a VCN in the same region 106 or 110, communication between a compute instance in Subnet 1 and an endpoint in the service network 110 in the same region), or endpoints in a VCN in a different region (e.g., communication between a compute instance in Subnet 1 and an endpoint in a VCN in a different region 108). Additionally, compute instances in a subnet hosted by CSPI 101 may communicate with endpoints not hosted by CSPI 101 (i.e., external to CSPI 101). These external endpoints include endpoints in a customer's on-premise network 116, endpoints in other hosted remote cloud networks 118, public endpoints 114 accessible over a public network such as the Internet, and other endpoints.

[0089] Communication between compute instances on the same subnet is facilitated through the use of VNICs associated with the source and destination compute instances. For example, compute instance C1 on Subnet 1 may want to send a packet to compute instance C2 on Subnet 1. For a packet originating from the source compute instance and destined for another compute instance in the same subnet, the packet is first processed by the VNIC associated with the source compute instance. The processing performed by the VNIC associated with the source compute instance may include determining the packet's destination information from the packet header, identifying any policies (e.g., security lists) configured for the VNIC associated with the source compute instance, determining the packet's next hop, performing encapsulation / decapsulation functions on the packet as needed, and forwarding / routing the packet to the next hop to facilitate communication of the packet to its intended destination. If the destination compute instance is in the same subnet as the source compute instance, the VNIC associated with the source compute instance is configured to identify the VNIC associated with the destination compute instance and forward the packet to that VNIC for processing, which then forwards the packet to the destination compute instance via the VNIC associated with the destination compute instance.

[0090] When a packet is sent from a compute instance in one subnet to an endpoint in a different subnet of the same VCN, this communication is facilitated by the VNICs and VCN VRs associated with the source and destination compute instances. For example, in Figure 1, if compute instance C1 in Subnet 1 wants to send a packet to compute instance D1 in Subnet 2, the packet is first processed by the VNIC associated with compute instance C1. The VNIC associated with compute instance C1 is configured to route the packet to VCN VR using the default route or port 10.0.0.1 of VCN VR105. VCN VR105 is configured to route the packet to Subnet 2 using port 10.1.0.1. Once the packet is received and processed by the VNIC associated with D1, this VNIC forwards the packet to compute instance D1.

[0091] When a packet is sent from a compute instance in VCN 104 to an endpoint outside VCN 104, the communication is facilitated by the VNIC associated with the source compute instance, VCN VR 105, and a gateway associated with VCN 104. VCN 104 may have one or more types of gateways associated with it. A gateway is an interface between a VCN and another endpoint, where the other endpoint is outside the VCN. A gateway is a Layer 3 / IP layer concept that allows a VCN to communicate with endpoints outside the VCN. In this way, a gateway facilitates the flow of traffic between a VCN and other VCNs or networks. A VCN may be configured with a variety of different types of gateways to facilitate different types of communication with different types of endpoints. Depending on the gateway, the communication may be over a public network (e.g., the Internet) or a private network. A variety of communication protocols may be used for these communications.

[0092] For example, compute instance C1 may want to communicate with an endpoint outside VCN 104. The packet may initially be processed by a VNIC associated with source compute instance C1. The VNIC's processing determines that the packet's destination is outside of Subnet 1 of C1. The VNIC associated with C1 may forward the packet to VCN VR105 in VCN 104. VCN VR105 then processes the packet and, as part of that processing, determines a particular gateway associated with VCN 104 as the packet's next hop based on the packet's destination. VCN VR105 may then forward the packet to the identified particular gateway. For example, if the destination is an endpoint in a customer's on-premises network, the packet may be forwarded by VCN VR105 to Dynamic Routing Gateway (DRG) gateway 122 configured for VCN 104. Forwarding the packet from the gateway to the next hop can then facilitate the packet's delivery to its ultimate destination.

[0093] Various different types of gateways may be configured for a VCN. Examples of gateways that may be configured for a VCN are shown in FIG. 1 and described below. Examples of gateways associated with VCNs are also shown in FIGS. 1, 2, 3, and 4 (e.g., gateways referenced by reference numbers 134, 136, 138, 234, 236, 238, 334, 336, 338, 434, 436, and 438) and described below. As shown in the embodiment of FIG. 1, a dynamic routing gateway (DRG) 122 may be added to or associated with customer VCN 104 to provide a path for private network traffic communication between customer VCN 104 and another endpoint, which may be the customer's on-premises network 116, a VCN 108 in a different region of CSPI 101, or another remote cloud network 118 not hosted by CSPI 101. The customer on-premises network 116 may be a customer network built using customer resources or may be a customer data center. Access to the customer on-premises network 116 is typically highly restricted. If a customer has both an on-premises network 116 and one or more VCNs 104 deployed or hosted in the cloud by CSPI 101, they may want to enable the on-premises network 116 and the cloud-based VCNs 104 to communicate with each other. This allows a customer to build an extended hybrid environment that includes the on-premises network 116 and the customer's VCN 104 hosted by CSPI 101. This communication is made possible by DRG 122. To enable such communication, a communication channel 124 is set up so that one endpoint of the channel resides in the customer on-premises network 116 and the other endpoint resides in CSPI 101 and is connected to the customer VCN 104. The communication channel 124 may traverse a public or private communication network, such as the Internet.A variety of different communication protocols may be used, such as IPsec VPN technology over a public communication network such as the Internet, or Oracle's FastConnect technology, which uses a private network instead of a public network. The device or equipment in the customer on-premises network 116 that forms one endpoint of the communication channel 124 is referred to as customer premises equipment (CPE), such as CPE 126 shown in Figure 1. On the CSPI 101 side, the endpoint may be a host machine running DRG 122.

[0094] In one embodiment, remote peering connections (RPCs) can be added to a DRG, allowing customers to peer one VCN with another VCN in a different region. Using such RPCs, a customer VCN 104 can connect to a VCN 108 in another region using a DRG 122. The DRG 122 can also be used to communicate with other remote cloud networks 118 not hosted by CSPI 101, such as the Microsoft Azure cloud or the Amazon AWS cloud.

[0095] 1, an Internet Gateway (IGW) 120 may be configured for customer VCN 104, allowing compute instances on VCN 104 to communicate with public endpoints 114 accessible over a public network, such as the Internet. IGW 120 is a gateway that connects a VCN to a public network, such as the Internet. IGW 120 allows a public subnet within a VCN, such as VCN 104, to directly access a public endpoint 112 on public network 114, such as the Internet, if the resources in that public subnet have public overlay IP addresses. Using IGW 120, connections can be initiated from a subnet within VCN 104 or from the Internet.

[0096] A network address translation (NAT) gateway 128 configured for a customer's VCN 104 may enable cloud resources in the customer's VCN that do not have dedicated public overlay IP addresses to access the Internet, but the NAT gateway 128 does not expose these resources to direct inbound Internet connections (e.g., L4-L7 connections). This allows private subnets within the VCN (such as Private Subnet 1 of VCN 104) to privately access public endpoints on the Internet. The NAT gateway only allows connections to be initiated from the private subnet to the public Internet, not from the Internet to the private subnet.

[0097] In some embodiments, a service gateway (SGW) 126 may be configured for customer VCN 104, providing a pathway for private network traffic between VCN 104 and service endpoints supported in service network 110. In some embodiments, service network 110 may be provided by a CSP and may offer a variety of services. One example of such a service network is Oracle's Services Network, which offers a variety of services available to customers. For example, compute instances (e.g., database systems) in a private subnet of customer VCN 104 can back up data to a service endpoint (e.g., Object Storage) without requiring a public IP address or access to the Internet. In some embodiments, a VCN may have only one SGW, and connections can be initiated only from subnets within the VCN, not from service network 110. When a VCN is peered with another VCN, resources in the other VCN typically cannot access the SGW. Additionally, resources in an on-premises network connected to a VCN with FastConnect or VPN Connect can use a service gateway configured for that VCN.

[0098] In one embodiment, SGW 126 uses the concept of a service CIDR (Classless Inter-Domain Routing) label, which is a string that represents all of the regional public IP address ranges for a service or services of interest. A customer uses the service CIDR label when configuring the SGW and associated routing rules to control traffic to the service. A customer can optionally use the service CIDR label when configuring security rules without having to adjust if the service's public IP address changes in the future.

[0099] A local peering gateway (LPG) 132 is a gateway that can be added to a customer VCN 104 to enable the VCN 104 to peer with another VCN in the same region. Peering means that the VCNs communicate using private IP addresses, but without traffic traversing a public network such as the Internet or routing traffic to the customer's on-premises network 116. In a preferred embodiment, a VCN has a separate LPG for each peering it establishes. Local peering or VCN peering is a common method used to establish network connectivity between different applications or infrastructure management functions.

[0100] Service providers (e.g., providers of services in service network 110) can provide access to services using various access models. In a public access model, services are exposed as public endpoints publicly accessible by compute instances in the customer VCN over a public network, such as the Internet, and / or they can be privately accessible through SGW 126. In a specific private access model, services are accessible as private IP endpoints in a private subnet of the customer VCN. This is called private endpoint (PE) access, and service providers can expose their services as instances in the customer's private network. Private endpoint resources represent services within the customer's VCN. Each PE appears as a VNIC in a customer-selected subnet in the customer's VCN (called a PE-VNIC and having one or more private IPs). Thus, the PE provides a way to present services in the private customer VCN subnet using VNICs. Because the endpoints are exposed as VNICs, all functionality associated with a VNIC, such as routing rules and security lists, is available to the PE VNIC.

[0101] Service providers may register their services to enable access through the PE. Providers can associate policies with services that restrict the visibility of the service to the customer tenancy. Providers can register multiple services under a single virtual IP address (VIP), especially for multi-tenant services. There may be multiple such private endpoints (in multiple VCNs) representing the same service.

[0102] Compute instances in the private subnet can then access the service using the private IP address or service DNS name of the PE VNIC. Compute instances in the customer VCN can access the service by sending traffic to the private IP address of the PE in the customer VCN. A private access gateway (PAGW) 130 is a gateway resource that can be attached to a service provider VCN (e.g., a VCN in service network 110) that acts as the ingress / egress point for all traffic to customer subnet private endpoints. The PAGW 130 allows providers to scale the number of PE connections without using their internal IP address resources. A provider need only configure one PAGW for any number of services registered in a single VCN. A provider can represent services as private endpoints in multiple VCNs for one or more customers. From the customer's perspective, the PE VNIC does not appear to be attached to the customer's instance, but rather to the service with which the customer wants to interact. Traffic destined for the private endpoint is routed to the service through the PAGW 130. These are called customer-to-service private connections (C2S connections).

[0103] The use of the PE concept also allows traffic to flow across FastConnect / IPsec links and private endpoints in the customer VCN, extending private access for services to the customer's on-premises network and data center, and allows traffic to flow between LPG 132 and PEs in the customer VCN, extending private access for services to the customer's peering VCN.

[0104] Customers can control VCN routing at the subnet level, allowing them to specify which subnets in their VCNs (such as VCN 104) use each gateway. The VCN's route tables are used to determine whether traffic is allowed from a VCN through a particular gateway. For example, in a particular instance, the route table for a public subnet in customer VCN 104 may direct non-local traffic through IGW 120. The route table for a private subnet in the same customer VCN 104 may direct traffic destined for CSP services through SGW 126. All other traffic may be directed through NAT gateway 128. Route tables only control traffic leaving a VCN.

[0105] Traffic entering the VCN via the gateway through an inbound connection is controlled using security lists associated with the VCN. All resources in a subnet use the same route table and security lists. Security lists can be used to control the specific types of traffic that can enter and exit instances in a VCN's subnets. Security list rules can include input (inbound) and output (outbound) rules. For example, an input rule can specify an allowed source address range, while an output rule can specify an allowed destination address range. Security rules can specify a specific protocol (e.g., TCP, ICMP), a specific port (e.g., 22 for SSH, 3389 for Windows RDP), etc. In some embodiments, the instance's operating system can enforce its own firewall rules in line with the security list rules. Rules can be stateful (e.g., connections are tracked and responses are automatically allowed without explicit security list rules for the response traffic) or stateless.

[0106] Access from a customer VCN (i.e., by resources or compute instances deployed in VCN 104) can be categorized as public access, private access, or dedicated access. Public access describes an access model in which public IP addresses or NATs are used to access public endpoints. Private access allows customer workloads in VCN 104 with private IP addresses (e.g., resources in a private subnet) to access services without traversing a public network such as the Internet. In one embodiment, CSPI 101 enables customer VCN workloads with private IP addresses to access (public service endpoints of) services using a service gateway. Thus, the service gateway provides a private access model by establishing a virtual link between the customer VCN and the public endpoints of the services that reside outside the customer's private network.

[0107] CSPI may also provide dedicated public access by using technologies such as FastConnect public peering, which allows customer on-premises instances to use a FastConnect connection to access one or more services in the customer VCN without traversing a public network such as the Internet. CSPI may also provide dedicated private access by using FastConnect private peering, which allows customer on-premises instances with private IP addresses to access workloads in the customer's VCN using a FastConnect connection. FastConnect is an alternative network connection using the public Internet to connect a customer's on-premises network to CSPI and its services. FastConnect provides an easy, flexible, and economical way to create a dedicated private connection with higher bandwidth options, higher reliability, and a consistent networking experience compared to Internet-based connections.

[0108] FIG. 1 and the accompanying discussion above describe various virtualized components in an exemplary virtual network. As noted above, a virtual network is built on an underlying physical or infrastructure network. FIG. 2 is a simplified architectural diagram of the physical components in a physical network within CSPI 200 that forms the basis of the virtual network, according to one embodiment. As shown, CSPI 200 provides a distributed environment including components and resources (e.g., compute, memory, and network resources) provided by a cloud service provider (CSP). These components and resources are used to provide cloud services (e.g., IaaS services) to subscribing customers, i.e., customers who have signed up for one or more services offered by the CSP. Customers are provisioned with a subset of CSPI 200's resources (e.g., compute, memory, and network resources) based on the services for which they subscribe. Customers can then build their own cloud-based (i.e., CSPI-hosted), customizable private virtual networks using the physical compute, memory, and network resources provided by CSPI 200. As previously mentioned, these customer networks are referred to as virtual cloud networks (VCNs). Customers can deploy one or more customer resources, such as compute instances, into these Customer VCNs. The compute instances can be in the form of virtual machines, bare metal instances, etc. CSPI 200 provides infrastructure and a set of complementary cloud services that enable customers to build and run a wide range of applications and services in a hosted, highly available environment.

[0109] In the exemplary embodiment shown in FIG. 2, the physical components of CSPI 200 include one or more physical host machines or servers (e.g., 202, 206, 208), network virtualization devices (NVDs) (e.g., 210, 212), top-of-rack (TOR) switches (e.g., 214, 216), and physical networks (e.g., 218) and switches of physical network 218. The physical host machines or servers can host and execute various compute instances that participate in one or more subnets of the VCN. The compute instances can include virtual machine instances and bare metal instances. For example, the various compute instances shown in FIG. 1 may be hosted by the physical host machines shown in FIG. 2. The virtual machine compute instances in the VCN may be executed by a single host machine or by multiple different host machines. Additionally, the physical host machines can host virtual host machines, container-based hosts or functions, etc. The VNICs and VCN VRs shown in FIG. 1 may be executed by the NVDs shown in FIG. 2. The gateway shown in FIG. 1 may be implemented by a host machine and / or NVD shown in FIG.

[0110] A host machine or server may run a hypervisor (also called a virtual machine monitor or VMM) that creates and makes available a virtualized environment on the host machine. Virtualization or a virtualized environment facilitates cloud-based computing. The hypervisor on a host machine may create, run, and manage one or more compute instances on the host machine. The hypervisor on a host machine enables the sharing of the host machine's physical computing resources (e.g., compute, memory, and network resources) among the various compute instances that the host machine runs.

[0111] For example, as shown in FIG. 2, host machines 202 and 208 execute hypervisors 260 and 266, respectively. These hypervisors may be implemented in software, firmware, hardware, or a combination thereof. Typically, a hypervisor is a process or software layer that resides on a host machine's operating system (OS) and runs on the host machine's hardware processor. A hypervisor provides a virtualized environment by enabling the sharing of the host machine's physical computing resources (e.g., processing resources such as processors / cores, memory resources, and networking resources) among various virtual machine compute instances that the host machine executes. For example, in FIG. 2, hypervisor 260 may reside on the OS of host machine 202 and enable the sharing of the host machine's computing resources (e.g., processing, memory, and networking resources) among the compute instances (e.g., virtual machines) that the host machine executes. A virtual machine may have its own operating system (referred to as a guest operating system), which may be the same as or different from the host machine's OS. The operating system of a virtual machine executed by a host machine may be the same as or different from the operating system of another virtual machine executed by the same host machine. In this manner, the hypervisor allows multiple operating systems to run side by side with each other while sharing the same computing resources of the host machine. The host machines shown in Figure 2 may have the same type of hypervisor or different types of hypervisors.

[0112] A compute instance can be a virtual machine instance or a bare metal instance. In Figure 2, compute instance 268 on host machine 202 and 274 on host machine 208 are examples of virtual machine instances. Host machine 206 is an example of a bare metal instance provided to a customer.

[0113] In some cases, an entire host machine is provisioned for a single customer, and the one or more compute instances (virtual machine or bare metal instances) that it hosts all belong to that same customer. In other cases, a host machine may be shared among multiple customers (i.e., multiple tenants). In such a multi-tenancy scenario, a host machine may host virtual machine compute instances that belong to different customers. These compute instances may be part of different VCNs for different customers. In some embodiments, bare metal compute instances are hosted on bare metal servers that do not include a hypervisor. When a bare metal compute instance is provisioned, a single customer or tenant maintains control of the physical CPU, memory, and network interfaces of the host machine that hosts the bare metal instance, and the host machine is not shared with other customers or tenants.

[0114] As previously mentioned, each compute instance that is part of a VCN is associated with a VNIC, which may make the compute instance a member of a subnet of the VCN. A VNIC associated with a compute instance facilitates transmission of packets or frames to the compute instance. A VNIC is associated with the compute instance when the compute instance is created. In one embodiment, for compute instances executed by a host machine, the VNIC associated with the compute instance is implemented by an NVD connected to the host machine. For example, in FIG. 2, host machine 202 executes virtual machine compute instance 268, which is associated with VNIC 276, which is implemented by NVD 210 connected to host machine 202. In another example, host machine 206 hosts bare metal instance 272, which is associated with VNIC 280, which is implemented by NVD 212 connected to host machine 206. In yet another example, the VNIC 284 is associated with a compute instance 274 that the host machine 208 executes, and the VNIC 284 is executed by an NVD 212 that is connected to the host machine 208 .

[0115] For compute instances hosted by a host machine, the NVD connected to that host machine also runs VCN VRs corresponding to the VCNs of which the compute instances are elements. For example, in the embodiment shown in Figure 2, NVD 210 runs VCN VR 277 corresponding to the VCN of which compute instance 268 is an element. NVD 212 may also run one or more VCN VRs 283 corresponding to the VCNs corresponding to the compute instances hosted by host machines 206 and 208.

[0116] A host machine may include one or more network interface cards (NICs) that allow the host machine to connect to other devices. A NIC on a host machine may provide one or more ports (or interfaces) that allow the host machine to communicate with other devices. For example, a host machine may be connected to an NVD by one or more ports (or interfaces) provided on the host machine and the NVD. A host machine may also be connected to other devices, such as another host machine.

[0117] 2, host machine 202 is connected to NVD 210 using link 220 extending between port 234 provided by NIC 232 of host machine 202 and port 236 of NVD 210. Host machine 206 is connected to NVD 212 using link 224 extending between port 246 provided by NIC 244 of host machine 206 and port 248 of NVD 212. Host machine 208 is connected to NVD 212 using link 226 extending between port 252 provided by NIC 250 of host machine 208 and port 254 of NVD 212.

[0118] The NVDs, in turn, are connected via communication links to top-of-rack (TOR) switches, which are connected to a physical network 218 (also referred to as a switch fabric). In one embodiment, the links between the host machines and the NVDs and between the NVDs and the TOR switches are Ethernet links. For example, in Figure 2, NVDs 210 and 212 are connected to TOR switches 214 and 216, respectively, using links 228 and 230. In one embodiment, links 220, 224, 226, 228, and 230 are Ethernet links. The collection of host machines and NVDs connected to a TOR may also be referred to as a rack.

[0119] The physical network 218 provides a communications fabric that allows the TOR switches to communicate with each other. The physical network 218 can be a multi-tier network. In one embodiment, the physical network 218 is a multi-tier Clos network of switches, with the TOR switches 214 and 216 representing leaf-level nodes of the multi-tier, multi-node physical switching network 218. Various Clos network configurations are possible, including, but not limited to, a 2-tier network, a 3-tier network, a 4-tier network, a 5-tier network, and generally an "n" tier network. An example of a Clos network is shown in FIG. 5 and described below.

[0120] A variety of different connection configurations are possible between host machines and NVDs, including one-to-one, many-to-one, and one-to-many configurations. In one embodiment of a one-to-one configuration, each host machine is connected to its own separate NVD. For example, in Figure 2, host machine 202 is connected to NVD 210 via NIC 232 on host machine 202. In a many-to-one configuration, multiple host machines are connected to a single NVD. For example, in Figure 2, host machines 206 and 208 are connected to the same NVD 212 via NICs 244 and 250, respectively.

[0121] In a one-to-multiple configuration, one host machine is connected to multiple NVDs. FIG. 3 shows an example of a CSPI 300 in which a host machine is connected to multiple NVDs. As shown in FIG. 3, a host machine 302 includes a network interface card (NIC) 304 including multiple ports 306 and 308. The host machine 300 is connected to a first NVD 310 via port 306 and link 320, and to a second NVD 312 via port 308 and link 322. Ports 306 and 308 may be Ethernet ports, and links 320 and 322 between the host machine 302 and the NVDs 310 and 312 may be Ethernet links. The NVD 310 is connected to a first TOR switch 314, and the NVD 312 is connected to a second TOR switch 316. The links between the NVDs 310 and 312 and the TOR switches 314 and 316 may be Ethernet links. TOR switches 314 and 316 represent layer 0 switching devices of a multi-tier physical network 318 .

[0122] 3 provides two separate physical network paths from the physical switch network 318 to the host machine 302: a first path through the TOR switch 314 to the NVD 310 and thus to the host machine 302, and a second path through the TOR switch 316 to the NVD 312 and thus to the host machine 302. These separate paths may increase the availability (referred to as high availability) of the host machine 302. If there is a problem with one of the paths (e.g., if a link on one of the paths goes down) or if there is a problem with one of the devices (e.g., if a particular NVD is not functioning), the other path may be used to communicate with the host machine 302.

[0123] In the configuration shown in Figure 3, the host machine is connected to two different NVDs using two different ports provided by the host machine's NIC. In other embodiments, the host machine may have multiple NICs allowing connection to multiple NVDs.

[0124] Referring again to Figure 2, an NVD is a physical device or component that performs one or more network and / or storage virtualization functions. An NVD may be any device that has one or more processing units (e.g., a CPU, a network processing unit (NPU), an FPGA, a packet processing pipeline, etc.), memory including cache, and ports. Various virtualization functions may be performed by software / firmware executed by one or more processing units of the NVD.

[0125] The NVD may be implemented in a variety of different forms. For example, in one embodiment, the NVD is implemented as an interface card with an embedded processor, referred to as a smart NIC or intelligent NIC. The smart NIC is a separate device from the NIC on the host machine. In Figure 2, NVDs 210 and 212 may be implemented as smart NICs connected to host machine 202 and host machines 206 and 208, respectively.

[0126] However, a smart NIC is merely one example of an implementation of an NVD. Various other implementations are possible. For example, in some other embodiments, the NVD or one or more functions performed by the NVD may be incorporated into or performed by one or more host machines, one or more TOR switches, and other components of CSPI200. For example, the NVD may be embodied in a host machine, and the functions performed by the NVD may be performed by the host machine. As another example, the NVD may be part of a TOR switch, or the TOR switch may be configured to perform the functions performed by the NVD, enabling the TOR switch to perform various complex packet transformations used in public clouds. A TOR performing the functions of an NVD may also be referred to as a smart TOR. In still other embodiments, if virtual machine (VM) instances rather than bare metal (BM) instances are provided to customers, the functions performed by the NVD may be implemented inside the hypervisor of the host machine. In some other embodiments, some of the NVD's functions may be offloaded to a centralized service running on a group of host machines.

[0127] In some embodiments, such as when implemented as a smart NIC as shown in FIG. 2, an NVD may have multiple physical ports that allow the NVD to connect to one or more host machines and one or more TOR switches. Ports on an NVD may be classified as host-side ports (also referred to as "south ports") or network-side or TOR-side ports (also referred to as "north ports"). A host-side port of an NVD is a port used to connect the NVD to a host machine. Examples of host-side ports in FIG. 2 include port 236 on NVD 210 and ports 248 and 254 on NVD 212. A network-side port of an NVD is a port used to connect the NVD to a TOR switch. Examples of network-side ports in FIG. 2 include port 256 on NVD 210 and port 258 on NVD 212. As shown in FIG. 2, the NVD 210 is connected to the TOR switch 214 by link 228, which extends from port 256 of the NVD 210 to the TOR switch 214. Similarly, the NVD 212 is connected to the TOR switch 216 by a link 230 extending from a port 258 of the NVD 212 to the TOR switch 216 .

[0128] An NVD can receive packets and frames from a host machine (e.g., packets and frames generated by a compute instance hosted by the host machine) via a host-side port, perform necessary packet processing, and then forward the packets and frames to a TOR switch via a network-side port of the NVD. An NVD can receive packets and frames from a TOR switch via a network-side port of the NVD, perform necessary packet processing, and then forward the packets and frames to a host machine via a host-side port of the NVD.

[0129] In some embodiments, there may be multiple ports and associated links between the NVD and the TOR switch. These ports and links may be aggregated to form a link aggregator group (LAG) consisting of multiple ports or links. Link aggregation allows multiple physical links between two endpoints (e.g., between the NVD and the TOR switch) to be treated as a single logical link. All physical links in a given LAG may operate in full-duplex mode at the same speed. LAGs help increase the bandwidth and reliability of the connection between two endpoints. If one of the physical links in the LAG goes down, traffic is dynamically and transparently reassigned to one of the other physical links in the LAG. The aggregated physical link provides higher bandwidth than each individual link. Multiple ports associated with a LAG are treated as a single logical port. Traffic may be load-balanced across the multiple physical links in the LAG. One or more LAGs may be configured between two endpoints. The two endpoints may be, for example, between the NVD and the TOR switch, or between a host machine and the NVD.

[0130] The NVD implements or performs network virtualization functions. These functions are performed by software / firmware executed by the NVD. Examples of network virtualization functions include, but are not limited to, packet encapsulation and decapsulation functions, functions for creating VCN networks, functions for enforcing network policies such as VCN security list (firewall) functions, and functions for facilitating routing and forwarding of packets to compute instances in the VCN. In one embodiment, upon receiving a packet, the NVD is configured to execute a packet processing pipeline for processing the packet and determining how to forward or route the packet. As part of this packet processing pipeline, the NVD may execute one or more virtual functions associated with the overlay network, such as running VNICs associated with compute instances in the VCN, running virtual routers (VRs) associated with the VCN, encapsulating and decapsulating packets to facilitate forwarding or routing in the virtual network, running a gateway (e.g., a local peering gateway), implementing security lists, network security groups, network address translation (NAT) functions (e.g., public IP to private IP translation on a per-host basis), throttling functions, and other functions.

[0131] In one embodiment, the NVD's packet processing data path may include multiple packet pipelines, each consisting of a series of packet transformation stages. In one embodiment, upon receipt of a packet, the packet is parsed and sorted into a single pipeline. The packet is then processed linearly, one stage at a time, until it is either discarded or sent out through the NVD's interface. These stages provide basic functional packet processing building blocks (e.g., header validation, throttling enforcement, new Layer 2 header insertion, L4 firewall enforcement, VCN encapsulation / decapsulation, etc.), such that new pipelines can be constructed by assembling existing stages, and new functionality can be added by creating and inserting new stages into existing pipelines.

[0132] The NVD may perform both control plane and data plane functions corresponding to the VCN's control and data planes. Examples of the VCN control plane are also shown in Figures 1, 2, 3, and 4 (see reference numbers 116, 216, 316, and 416) and described below. Examples of the VCN data plane are shown in Figures 1, 2, 3, and 4 (see reference numbers 118, 218, 318, and 418) and described below. Control plane functions include functions used to configure the network (e.g., routing and route table configuration, VNIC configuration, etc.) and control how data is forwarded. In one embodiment, a VCN control plane is provided that centrally computes and exposes all overlay-to-foundation mappings to the NVD and virtual network edge devices (e.g., various gateways such as DRGs, SGWs, and IGWs). Firewall rules may also be exposed by the same mechanism. In one embodiment, the NVD retrieves only mappings relevant to the NVD. The data plane functions include functionality for the actual routing / forwarding of packets based on the configuration set using the control plane. The VCN data plane is achieved by encapsulating customer network packets before they traverse the underlying network. The encapsulation / decapsulation functionality is achieved on the NVD. In one embodiment, the NVD is configured to intercept all network packets entering and leaving the host machine and perform network virtualization functions.

[0133] As described above, the NVD performs various virtualization functions, including VNICs and VCN VRs. The NVD may execute VNICs associated with compute instances hosted by one or more host machines connected to the VNIC. For example, as shown in FIG. 2, NVD 210 executes the functions of VNIC 276 associated with compute instance 268 hosted by host machine 202 connected to NVD 210. As another example, NVD 212 executes VNIC 280 associated with bare metal compute instance 272 hosted by host machine 206 and VNIC 284 associated with compute instance 274 hosted by host machine 208. The host machines can host compute instances that belong to different VCNs belonging to different customers, and the NVDs connected to the host machines can execute VNICs (i.e., perform VNIC-related functions) corresponding to these compute instances.

[0134] The NVDs also execute VCN virtual routers corresponding to the VCNs of the compute instances. For example, in the embodiment shown in FIG. 2, NVD 210 executes VCN VR 277 corresponding to the VCN to which compute instance 268 belongs. NVD 212 executes one or more VCN VRs 283 corresponding to one or more VCNs to which compute instances hosted by host machines 206 and 208 belong. In one embodiment, a VCN VR corresponding to a given VCN is executed by all NVDs connected to a host machine that hosts at least one compute instance belonging to that VCN. If a host machine hosts compute instances that belong to different VCNs, the NVDs connected to that host machine may execute VCN VRs corresponding to the different VCNs.

[0135] In addition to VNICs and VCN VRs, the NVD may run various software (e.g., daemons) and may include one or more hardware components that facilitate the various network virtualization functions performed by the NVD. For simplicity, these various components are grouped together as a “packet processing component” shown in FIG. 2 . For example, NVD 210 includes packet processing component 286, and NVD 212 includes packet processing component 288. For example, the packet processing component of the NVD may include a packet processor configured to interact with the NVD's ports and hardware interfaces to monitor all packets received and communicated using the NVD and to store network information. The network information may include, for example, network flow information and per-flow information (e.g., per-flow statistics) that identify the various network flows processed by the NVD. In one embodiment, network flow information may be stored per VNIC. The packet processor may perform per-packet operations as well as implement a stateful NAT and an L4 firewall (FW). As another example, the packet processing component may include a replication agent configured to replicate information stored by the NVD to one or more different replication target stores. As yet another example, the packet processing component may include a logging agent configured to perform logging functions for the NVD. The packet processing component may also include software for monitoring the performance and health of the NVD, and possibly the status and health of other components connected to the NVD.

[0136] FIG. 1 illustrates components of an exemplary virtual or overlay network, including a VCN, subnets within the VCN, compute instances deployed to the subnets, VNICs associated with the compute instances, VRs in the VCN, and a set of gateways configured for the VCN. The overlay components illustrated in FIG. 1 may be executed or hosted by one or more of the physical components illustrated in FIG. 2. For example, compute instances in a VCN may be executed or hosted by one or more host machines illustrated in FIG. 2. For compute instances hosted by a host machine, the VNICs associated with the compute instances are typically executed by an NVD connected to the host machine (i.e., the VNIC functionality is provided by an NVD connected to the host machine). The VCN VR functionality for a VCN is performed by all NVDs connected to host machines that host or execute compute instances that are part of the VCN. The gateways associated with a VCN may be executed by one or more different types of NVDs. For example, one gateway may be executed by a smart NIC, while other gateways may be executed by one or more host machines or other implementations of NVDs.

[0137] As mentioned above, a compute instance in a customer VCN can communicate with a variety of different endpoints, which can be in the same subnet as the source compute instance, in a different subnet but within the same VCN as the source compute instance, or outside the VCN of the source compute instance. These communications are facilitated through the use of VNICs associated with the compute instances, VCN VRs, and gateways associated with the VCN.

[0138] For communication between two compute instances on the same subnet in a VCN, this communication is facilitated through the use of VNICs associated with the source and destination compute instances. The source and destination compute instances may be hosted by the same host machine or different host machines. A packet originating from the source compute instance may be forwarded from the host machine hosting the source compute instance to an NVD connected to that host machine. On the NVD, the packet is processed using a packet processing pipeline, which may include running the VNIC associated with the source compute instance. Because the packet's destination endpoint is in the same subnet, the VNIC associated with the source compute instance forwards the packet to the NVD running the VNIC associated with the destination compute instance. The NVD then processes the packet and forwards it to the destination compute instance. The VNICs associated with the source and destination compute instances may be running on the same NVD (e.g., if both the source and destination compute instances are hosted by the same host machine) or on different NVDs (e.g., if the source and destination compute instances are hosted by different host machines connected to different NVDs). The VNICs may use routing / forwarding tables stored by the NVD to determine the next hop for a packet.

[0139] When a packet is sent from a compute instance in one subnet to an endpoint in a different subnet of the same VCN, the packet originating from the source compute instance is sent from the host machine hosting the source compute instance to the NVD connected to that host machine. The NVD processes the packet using a packet processing pipeline, which may include running one or more VNICs and VRs associated with the VCN. For example, as part of the packet processing pipeline, the NVD executes or invokes a function corresponding to the VNIC associated with the source compute instance (also referred to as VNIC execution). The function executed by the VNIC may include verifying the VLAN tag on the packet. Because the packet's destination is outside the subnet, the NVD then invokes a VCN VR function to execute it. The VCN VR then routes the packet to the NVD that executes the VNIC associated with the destination compute instance. The VNIC associated with the destination compute instance then processes the packet and forwards it to the destination compute instance. The VNICs associated with the source and destination compute instances may be running on the same NVD (e.g., if both the source and destination compute instances are hosted by the same host machine) or may be running on different NVDs (e.g., if the source and destination compute instances are hosted by different host machines connected to different NVDs).

[0140] If the packet's destination is outside the VCN of the source compute instance, the packet originating from the source compute instance is sent from the host machine hosting the source compute instance to the NVD connected to that host machine. The NVD runs the VNIC associated with the source compute instance. Because the packet's destination endpoint is outside the VCN, the packet is then processed by the VCN VR for that VCN. The NVD invokes a VCN VR function, which forwards the packet to an NVD running the appropriate gateway associated with the VCN. For example, if the destination is an endpoint in a customer's on-premises network, the packet may be forwarded by the VCN VR to an NVD running a DRG gateway configured for the VCN. The VCN VR may be running on the same NVD as the NVD running the VNIC associated with the source compute instance, or it may be running on a different NVD. The gateway may be running on the NVD, which may be a smart NIC, a host machine, or another implementation of the NVD. The packet is then processed by the gateway and forwarded to a next hop that facilitates delivery of the packet to the intended destination endpoint. For example, in the embodiment shown in FIG. 2, a packet originating from compute instance 268 may be sent from host machine 202 to NVD 210 over link 220 (using NIC 232). In NVD 210, VNIC 276 is called VNIC 276 because it is associated with source compute instance 268. VNIC 276 is configured to examine information encapsulated in the packet, determine a next hop to forward the packet to facilitate delivery of the packet to the intended destination endpoint, and then forward the packet to the determined next hop.

[0141] Compute instances deployed in a VCN can communicate with a variety of different endpoints. These endpoints can include endpoints hosted by CSPI 200 and endpoints external to CSPI 200. CSPI 200-hosted endpoints can include instances in the same VCN or other VCNs (which may be customer VCNs or VCNs not belonging to the customer). Communication between CSPI 200-hosted endpoints can be performed over physical network 218. Compute instances can also communicate with endpoints not hosted by CSPI 200 (i.e., external to CSPI 200). Examples of such endpoints include endpoints within a customer's on-premises network or data center or public endpoints accessible over a public network such as the Internet. CSPI 200's communication with external endpoints can be performed over a public network (e.g., the Internet) (not shown in FIG. 2) or over a private network (not shown in FIG. 2) using various communication protocols.

[0142] The architecture of CSPI 200 shown in FIG. 2 is illustrative only and is not intended to be limiting. Variations, substitutions, and improvements are possible in alternative embodiments. For example, in some implementations, CSPI 200 may include more or fewer systems or components than those shown in FIG. 2, may combine two or more systems, or may have different system configurations or arrangements. The systems, subsystems, and other components shown in FIG. 2 may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective systems, using hardware, or a combination thereof. Software may be stored in a non-transitory storage medium (e.g., a memory device).

[0143] FIG. 4 illustrates connections between host machines and an NVD to provide I / O virtualization to support multitenancy, according to one embodiment. As shown in FIG. 4, host machine 402 runs hypervisor 404, which provides a virtualized environment. Host machine 402 runs two virtual machine instances: VM1 406, which belongs to customer / tenant #1, and VM2 408, which belongs to customer / tenant #2. Host machine 402 includes a physical NIC 410 connected to NVD 412 via link 414. Each compute instance is associated with a VNIC that NVD 412 runs. In the embodiment of FIG. 4, VM1 406 is associated with VNIC-VM1 420, and VM2 408 is associated with VNIC-VM2 422.

[0144] 4, NIC 410 includes two logical NICs: logical NIC A 416 and logical NIC B 418. Each virtual machine is configured to associate with and work together on its own logical NIC. For example, VM1 406 is associated with logical NIC A 416, and VM2 408 is associated with logical NIC B 418. Although host machine 402 includes only one physical NIC 410 shared by multiple tenants, because of the logical NICs, each tenant's virtual machine sees itself as having its own host machine and NIC.

[0145] In one embodiment, each logical NIC is assigned its own VLAN ID. Thus, a particular VLAN ID is assigned to logical NIC A 416 for tenant #1, and a separate VLAN ID is assigned to logical NIC B 418 for tenant #2. When a packet is sent from VM1 406, the hypervisor attaches a tag assigned to tenant #1 to the packet, and the packet is then sent from host machine 402 to NVD 412 via link 414. Similarly, when a packet is sent from VM2 408, the hypervisor attaches a tag assigned to tenant #2 to the packet, and the packet is then sent from host machine 402 to NVD 412 via link 414. Thus, a packet 424 sent from host machine 402 to NVD 412 has an associated tag 426 that identifies the particular tenant and associated VM. On the NVD, for a packet 424 received from a host machine 402, a tag 426 associated with the packet is used to determine whether the packet should be processed by VNIC-VM1 420 or VNIC-VM2 422. The packet is then processed by the corresponding VNIC. According to the configuration shown in Figure 4, each tenant's compute instance can be recognized as having its own host machine and NIC. The configuration shown in Figure 4 provides I / O virtualization to support multi-tenancy.

[0146] FIG. 5 is a simplified block diagram of a physical network 500, according to one embodiment. The embodiment shown in FIG. 5 is structured as a Clos network. A Clos network is a specific type of network topology designed to provide connection redundancy while maintaining high bisection bandwidth and maximum resource utilization. A Clos network is a type of non-blocking multistage or multi-layer switching network, with possible numbers of stages or layers, such as two, three, four, or five. The embodiment shown in FIG. 5 is a three-layer network, including layers 1, 2, and 3. TOR switch 504 represents a layer 0 switch in a Clos network. One or more NVDs are connected to a TOR switch. A layer 0 switch is also referred to as an edge device of the physical network. A layer 0 switch is connected to a layer 1 switch, also referred to as a leaf switch. In the embodiment shown in FIG. 5, a set of “n” layer 0 TOR switches are connected to a set of “n” layer 1 switches, collectively forming a pod. Each layer 0 switch in a pod is interconnected to all layer 1 switches in that pod, but there are no switch connections between pods. In one embodiment, two pods are referred to as a block. Each block is served by or connected to a set of “n” layer 2 switches (sometimes referred to as spine switches). There may be multiple blocks in a physical network topology. The layer 2 switches are then connected to “n” layer 3 switches (sometimes referred to as super spine switches). Packet communication across the physical network 500 is typically performed using one or more layer 3 communication protocols. High availability is typically achieved by having n-way redundancy at all layers of the physical network except the TOR layer. To enable scaling of the physical network, policies may be specified for pods and blocks to control the visibility of switches to each other in the physical network.

[0147] A Clos network is characterized by a fixed maximum hop count for reaching another tier-0 switch (or from an NVD connected to a tier-0 switch to another NVD connected to a tier-0 switch). For example, in a three-tier Clos network, a packet requires at most seven hops to reach another NVD from one NVD, with the source and destination NVDs connected to the leaf layers of the Clos network. Similarly, in a four-tier Clos network, a packet requires at most nine hops to reach another NVD from one NVD, with the source and destination NVDs connected to the leaf layers of the Clos network. Therefore, the Clos network architecture maintains consistent latency throughout the network, which is important for intra- and inter-datacenter communications. Clos topologies are horizontally scalable and cost-effective. The network's bandwidth / throughput capacity can be easily increased by adding more switches (e.g., more leaf and spine switches) to various tiers and by increasing the number of links between switches in adjacent tiers.

[0148] In one embodiment, each resource in CSPI is assigned a unique identifier called a Cloud Identifier (CID). This identifier is included as part of the resource's information and can be used when managing the resource, for example, through a console or API. An exemplary syntax for a CID is as follows: ocid1.<RESOURCE TYPE> . <realm>.[REGION][.FUTURE USE].<UNIQUE ID> where: ocid1 is a string indicating the version of the CID, RESOURCE TYPE is the type of resource (for example, instance, volume, VCN, subnet, user, group, etc.), REALM is the realm in which the resource resides (example values ​​are "c1" for the commercial realm, "c2" for the government cloud realm, or "c3" for the federal cloud realm, etc.; each realm may have its own domain name); REGION is the region in which the resource resides (this part may be blank if region is not applicable to the resource), FUTURE USE is reserved for future use, UNIQUE ID is the unique part of the ID (the format can vary depending on the type of resource or service).

[0149] Virtual Private Label Cloud (vPLC) Described herein are techniques for enabling cloud infrastructure used to create one or more virtual private clouds (also referred to herein as virtual private label clouds or "vPLCs") in a region provided by a cloud service provider (CSP) to provide CSP-supplied cloud services to CSP customers. A vPLC created according to the techniques described in this disclosure can be used for a variety of purposes. For example, in one use case, a vPLC can be created for a reseller who is a customer of the CSP and used by the reseller to provide one or more reseller-supplied cloud services to the reseller's customers. In another use case, a vPLC can be used as a virtual data center and associated with a realm different from the realm associated with the cloud infrastructure in the CSP-supplied region.

[0150] A cloud service provider (CSP) provides infrastructure (referred to as cloud service provider infrastructure, or CSPI, or CSP-delivered infrastructure) used to deliver one or more cloud services to the CSP's customers. A CSP's customers may include the CSP's and / or its reseller's direct customers. CSPI may include physical and virtual infrastructure components. Physical CSPI may include a physical infrastructure or underlying network, network virtualization devices, and interconnected high-performance compute resources, including racks, servers and host machines, memory resources, and network resources used to configure other resources. Virtual CSPI components may include components in an overlay network, such as virtual machines, virtual routers and gateways, and other virtual components.

[0151] A CSP may offer various types of cloud services. These may include various types of services such as Software as a Service (SaaS), Platform as a Service (PaaS), and Infrastructure as a Service (IaaS). In some embodiments, as described in this disclosure, a CSP may offer a "vPLC cloud service." A customer of a CSP may subscribe to one or more cloud services offered by the CSP. A customer may be any entity, such as an individual, an organization, or a business.

[0152] Infrastructure as a Service (IaaS) is a specific type of cloud computing service. In the IaaS model, a CSP provides infrastructure that customers can use to build their own customizable networks and deploy their resources. In this way, customer resources and networks are hosted in a distributed environment by the CSP-provided infrastructure. This differs from traditional computing, where customer resources and networks are hosted by customer-provided infrastructure. In one embodiment, a vPLC as a Service is a type of IaaS service in which infrastructure is allocated to create a virtual private label cloud for the customer. For example, if a CSP customer is a reseller, the CSP can provide a reseller-supplied cloud service using a vPLC created for the reseller.

[0153] When a customer registers or subscribes to a cloud service offered by a CSP, a tenancy (or account) is created for the customer. This account / tenancy allows the customer to access one or more registered cloud resources associated with the account / tenancy. Each tenancy is identified by a tenancy identifier (T_ID) that is assigned to the tenancy and uniquely identifies the tenancy. The tenancy identifier is generated when the tenancy is created and associated with the tenancy. For example, when a direct customer of a CSP registers for a service offered by a CSP, a tenancy is created for the customer and a tenancy identifier that uniquely identifies the tenancy is generated. When a reseller registers for a vPLC service, a tenancy is created for the reseller and a tenancy identifier that uniquely identifies the tenancy is generated. A tenancy may be a logical and secure compartment that contains all the resources and services used by the customer associated with the tenancy.

[0154] Resources in a CSP-provided infrastructure are typically identified using a unique resource identifier (resource ID or RID). Because these resources are provided in the cloud and used to provide cloud services, resource IDs are also referred to as cloud resource identifiers (or cloud IDs or CIDs). For example, in a cloud environment provided by Oracle Corporation, resources are identified using an "ocid" or "Oracle cloud identifier." In some embodiments, a resource ID is a globally unique identifier string that identifies a cloud resource. In some embodiments, a tenancy ID is a type of cloud resource ID (or cloud identifier (CID)).

[0155] A resource or cloud ID (CID) may be represented in the following format, as an example:

[0156] RID_version. <resourcetype> . <realm>.[REGION][.FUTURE USE].<UNIQUE ID> where: RID_version indicates the version of the resource ID.

[0157] "Resource type" identifies the particular kind of resource that the resource ID refers to. Examples of different resource types include an instance, a virtual network subnet, a tenancy, etc. For example, an "instance RID" identifies a compute instance.

[0158] "Realm" and "region" indicate the location (realm and region) of the resource. The "unique ID" is the unique part of the ID (e.g., a unique string to identify a particular resource). Thus, the tenancy ID can identify a unique ID within a CID where the resource type is tenancy.

[0159] When resources are created, a CID is generated for them. Virtual machines, accounts, tenancies, permission policies, virtual network subnets, access control lists (ACLs), and load balancers are examples of cloud resources and can be referenced by their respective CIDs. A tenancy is a logical container used to group other cloud resources that belong to the same cloud customer account.

[0160] When an entity becomes a customer of a CSP and subscribes to the vPLC service, a vPLC is created for the customer and associated with the customer's tenancy. For example, if the customer is a reseller, a vPLC is created for the reseller and associated with the reseller's tenancy. A vPLC created for a reseller is treated as a resource and is assigned a unique resource identifier called a vPLC identifier (vPLC ID), which is associated with the reseller's tenancy. As described herein, for a vPLC created for a reseller, the vPLC ID for that vPLC is used to identify all resources and requests associated with the vPLC. Each vPLC ID uniquely identifies a vPLC.

[0161] For example, in one embodiment, all resources assigned to a vPLC are tagged with the vPLC identifier of that vPLC. In one embodiment, a vPLC ID field is inserted into the representation of the CID described above. If a resource is associated with a vPLC, the vPLC ID corresponding to that vPLC may be provided in this CID field; if a resource is not associated with a vPLC, the field may be left blank.

[0162] For the purposes of this disclosure, the following terminology will be used for the sake of clarity: CSP.Cn (or just Cn) identifies a direct (non-reseller) customer of the CSP. For example, "CSP.C1" represents the CSP's non-reseller direct customer C1, and "CSP.C2" represents the CSP's non-reseller direct customer C2.

[0163] CSP.Rn (or just Rn) identifies a reseller customer of the CSP. For example, "CSP.R1" represents CSP's reseller customer R1, and "CSP.R2" represents CSP's reseller customer R2.

[0164] In Rn.Cm, a reseller may have its own customers. This identifies a particular customer of a particular reseller. For example, "R1.C1" represents customer C1 of reseller R1, and "R2.C1" represents customer C1 of reseller R2.

[0165] T.Cn identifies the tenancy of a CSP's non-reseller customer. For example, "T.C1" represents the tenancy of non-reseller direct customer C1.

[0166] T.Rn identifies the reseller's tenancy. For example, "T.R1" represents the tenancy of reseller R1.

[0167] In T.Rn.Cm, each customer of a reseller has their own tenancy. This identifies the tenancy associated with a particular customer of a particular reseller. For example, "T.R1.C1" represents the tenancy associated with reseller R1's customer C1, and "T.R2.C1" represents the tenancy associated with reseller R2's customer C1.

[0168] T_ID.T.Cn identifies the tenancy identifier that identifies the tenancy of a CSP's non-reseller customer. For example, "T_ID.T.C1" represents the tenancy ID of non-reseller direct customer C1's tenancy.

[0169] T_ID.T.Rn identifies the tenancy identifier of the reseller's tenancy. For example, "T_ID.T.R1" represents the tenancy identifier of reseller R1's tenancy.

[0170] T_ID.Rn.Cm identifies the tenancy identifier of the tenancy associated with a particular customer of a particular reseller. For example, "T_ID.T.R1.C1" represents the tenancy identifier of the tenancy associated with customer C1 of reseller R1, and "T_ID.T.R2.C1" represents the tenancy identifier of the tenancy associated with customer C1 of reseller R2.

[0171] vPLC.Rn identifies a vPLC created for reseller Rn. For example, "vPLC.R1" identifies the tenancy created for and associated with reseller R1. This identifies a specific vPLC created for a reseller and associated with the reseller's tenancy.

[0172] vPLC_ID.vPLC.Rn identifies the vPLC identifier of a vPLC generated for a particular reseller. For example, "vPLC_ID.vPLC.R1" represents the vPLC identifier of a vPLC generated for reseller R1.

[0173] As described herein, a vPLC is created for a reseller, who is a customer of a CSP. The vPLC is associated with the reseller's tenancy. In some embodiments, a vPLC ID identifying a vPLC may be associated with a tenancy ID identifying the reseller's tenancy. This allows for the identification of a corresponding vPLC and the reseller for which the vPLC was created (including the reseller's tenancy and tenancy identifier), given a vPLC ID.

[0174] CSPI or CSP-provided infrastructure may be organized as realms, regions, and data centers. A region represents a local geographic area that includes one or more connected data centers. Regions are independent of other regions and may be separated by large distances, for example, across countries or continents. A set of application programming interfaces (APIs) is provided to the CSP-provided infrastructure in a region, and these APIs are used to perform operations using the infrastructure in the region. Operations that may be performed using the infrastructure in a region may include creating resources of the infrastructure in the region, accessing resources of the infrastructure in the region, performing operations involving resources of the infrastructure in the region, deleting resources from the infrastructure in the region, and performing other operations involving the infrastructure in the region.

[0175] A set of APIs is region-specific. Thus, a CSP-provided infrastructure in a region is characterized by a set of APIs usable with the infrastructure in that region. A CSP-provided infrastructure in a different region has a different set of APIs provided for the different region. Thus, for purposes of this application, the phrase "CSP-provided infrastructure in a region" or "CSP-provided infrastructure in a region" means that the infrastructure is characterized by a specific set of APIs provided for the region. Thus, a first set of APIs may be provided for a first region, a second set of APIs may be provided for a second region different from the first region, a third set of APIs may be provided for a third region different from the first and second regions, and so on. In this way, a specific set of APIs identifies a CSP-provided infrastructure in a particular region. In other words, a CSP-provided infrastructure in two separate regions would have two separate sets of APIs. In this manner, operations involving regional infrastructure are performed by characterizing a region with infrastructure in that region and a set of APIs associated with that region. CSP-provided infrastructure in a region is also referred to as "regional infrastructure" or "region-specific infrastructure."

[0176] For example, a CSP may provide a first infrastructure in Seattle and a second infrastructure in Portland. For the first infrastructure in Seattle, a first set of APIs may be provided, which may be characterized by the following format:

[0177] <service_identifier> .us-seattle.oci.oraclecloud.com For the second infrastructure in Portland, a second set of APIs may be provided, which may be characterized by the following format:

[0178] <service_identifier> .us-portland.oci.oraclecloud.com Note that Seattle and Portland are used here only as examples. The geographic locations of the regions can vary. For example, there can be more than one region in the same geographic city. For example, there can be a first region in a first building in a city and a second region in a second, separate building in the same city. As yet another example, there can be a first region on one floor of a building and a second region on another floor of the building. As yet another example, for infrastructure provided by a CSP on a floor, a first portion of the infrastructure can reside in a first region associated with a first set of APIs, and a second portion of the infrastructure can reside in a second region associated with a second set of APIs that is different from the first set of APIs.

[0179] A CSP's infrastructure in a region may be organized as one or more data centers. For example, in a particular region where a CSP may have two buildings each hosting infrastructure used to provide cloud services, the infrastructure in the first building may be organized and referred to as a first data center, and the infrastructure in the second building may be organized and referred to as a second data center. As another example, in a particular region where a CSP may have a single building with multiple floors each hosting infrastructure used to provide cloud services, the infrastructure on the first floor may be organized and referred to as a first data center, and the infrastructure on the second floor may be organized and referred to as a second data center. As yet another example, in a particular region where a CSP may have infrastructure used to provide cloud services, a first portion of the infrastructure may be organized and referred to as a first data center, and a second, separate portion of the infrastructure may be organized and referred to as a second data center.

[0180] A region may have one or more data centers. In some embodiments, all regional infrastructure is contained in a single data center, while in other embodiments, the infrastructure in a region may be organized as multiple data centers. Each data center may include infrastructure resources provided by a CSP, such as compute, storage, and networking resources. In a region, the data centers in the region may be organized as one or more availability domains (ADs). Availability domains are isolated from each other, fault-tolerant, and highly unlikely to fail simultaneously. ADs are configured so that a failure in one AD in a region is unlikely to affect the availability of other ADs in the same region.

[0181] A realm represents a logical collection of one or more regions. A realm may contain one or more regions. Realms are typically isolated from one another and do not share data. Each realm has its own identity and trust profile (e.g., password, authentication information, etc.). The realm's identity and trust profile are configured to prevent inter-realm communication. In this manner, each realm represents an isolated domain. For example, one realm may be created for government entities that are customers of a CSP, another realm may be created for commercial customers of the CSP, yet another realm may be created for private customers of the CSP, and so on. With respect to regions within a realm, regional infrastructures can communicate with each other, while regional infrastructure associated with a first realm cannot communicate with regional infrastructure associated with a second, different realm.

[0182] Resellers of cloud services can be entities such as businesses (eg, telecommunications companies), system integrators, government agencies, IT departments of large corporations, schools, and the like.

[0183] FIG. 6 is a simplified block diagram of a distributed environment 600 including a CSP-provided infrastructure in a region, illustrating an example of a virtual private label cloud (vPLC) hosted by the CSP-provided infrastructure, according to one embodiment. The distributed environment 600 illustrated in FIG. 6 is merely an example and is not intended to unduly limit the scope of the claimed embodiments. Many variations, alternatives, and improvements are possible. For example, in some implementations, the distributed environment 600 may include more or fewer systems or components than those illustrated in FIG. 6, may combine two or more systems, or may have different system configurations or arrangements. The systems, subsystems, and other components illustrated in FIG. 6 may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective systems, using hardware, or a combination thereof. Software may be stored in a non-transitory storage medium (e.g., a memory device).

[0184] Figure 6 illustrates a CSP-provided infrastructure 601 in a region. The infrastructure 601 may be organized as one or more data centers. The infrastructure 601 may include memory or storage resources 622, compute resources 620, networking resources (not shown), and other physical and logical resources provided by the CSP. The storage resources 622 may include one or more volatile or non-volatile memory resources. The compute resources 620 may include one or more racks, each with one or more servers hosting one or more operating systems.

[0185] A CSP can use CSP provisioning infrastructure 601 to offer one or more CSP-supplied cloud services to one or more of its direct customers and to create one or more vPLCs. In the example shown in Figure 6, two vPLCs (vPLC.R1 604 and vPLC.R2 606) have been created for resellers R1 and R2, respectively. vPLC.R1 604 is used to offer one or more reseller R1-supplied and R1-branded cloud services to one or more customers 642 of R1. vPLC.R2 606 is also used to offer one or more reseller R2-supplied and R2-branded cloud services to one or more customers 644 of R2.

[0186] As shown, infrastructure 601 can be communicatively coupled via communications network 652 to computing devices used by users associated with direct customer 640 and users associated with reseller customers (e.g., users associated with reseller R1's customer 642 and users associated with reseller R2's customer 644). Communications network 652 can be of various types and can include one or more communications networks. Examples of communications network 652 include, but are not limited to, the Internet, a wide area network (WAN), a local area network (LAN), an Ethernet network, a public or private network, a wired network, a wireless network, or the like, as well as combinations thereof. Additionally, various communications protocols can be used to facilitate communications, including both wired and wireless protocols, such as the IEEE 802.XX suite of protocols, TCP / IP, IPX, SAN, AppleTalk®, Bluetooth®, and other protocols. In general, communications network 652 can include any infrastructure that facilitates communications with infrastructure 601.

[0187] 6, infrastructure 601 is divided into three securely isolated portions: (1) a first portion 602 used for CSP provisioning and providing CSP-branded cloud services to one or more direct (or non-reseller) customers 640 of the CSP, (2) a second portion allocated to vPLC.R1 604 generated for reseller R1, and (3) a third portion allocated to vPLC.R2 606 generated for reseller R2. In one embodiment, these three portions are securely isolated from one another.

[0188] The partitioning of resources includes partitioning of storage resources 622 and compute resources 620. For example, as shown in Figure 6, storage resources 622 include a storage portion 622a included in first portion 602 used for providing CSP-supplied cloud services to the CSP's direct customers. Storage resources 622 also include a storage portion 622b allocated to vPLC.R1 604 and a storage portion 622c allocated to vPLC.R2 606. The storage or memory portion allocated to a vPLC may include one or more volatile and / or non-volatile memory resources.

[0189] 6, compute resources 620 are also partitioned. Compute resources 620 include a compute portion 620a in first portion 602 that is used to provide CSP-supplied cloud services to the CSP's direct customers. Compute resources 620 also include a portion 620b allocated to vPLC.R1 604 and a portion 620c allocated to vPLC.R2 606.

[0190] As described above, a set of APIs is provided for accessing and performing functions related to the CSP-provided infrastructure in a region. These APIs are accessible to users associated with the CSP's direct customers 640 by using a set of endpoints 610-1 provided by the CSP. For example, a user may use one or more of the endpoints 610-1 to access and / or use compute resources 620a or memory resources 622a allocated to the infrastructure used to provide CSP-provided cloud services to the CSP's customers. The endpoints in 610-1 may represent URLs with associated fully qualified domain names (FQDNs) used to access cloud services provided by the infrastructure 601. Examples of these endpoints include: createVM.compute.us-seattle.oci.oraclecloud.com (for creating VMs) deleteVM.compute.us-seattle.oci.oraclecloud.com (for erasing VMs) storeObject.storage.us-seattle.oci.oraclecloud.com (for data storage) etc. Users associated with the CSP's direct customers 640 and connected to the infrastructure 601 via a communications network can use one or more of the endpoints 610-1.

[0191] In one embodiment, a set of vPLC-specific endpoints is created for each vPLC. For example, as shown in FIG. 6, a set of endpoints 612-1 is created for vPLC.R1 and a set of endpoints 614-1 is created for vPLC.R2. A user associated with a customer 642 of reseller R1 can use endpoint 612-1 to access resources associated with vPLC.R1 and perform one or more functions or operations involving the resources assigned to vPLC.R1. A user associated with a customer 644 of reseller R2 can use endpoint 614-1 to access resources associated with vPLC.R2 and perform one or more functions or operations involving the resources assigned to vPLC.R2.

[0192] In one embodiment, a set of vPLC-specific endpoints for a vPLC is generated based on the endpoints that the CSP provides to its direct customers. For example, in FIG. 6, endpoint 612-1 for vPLC.R1 is generated based on endpoint 610-1. The endpoints generated for a vPLC may include an identifier that specifically identifies the vPLC for which the endpoint was generated. For example, given the exemplary endpoints that the CSP provides to its direct customers, endpoint 612-1 for vPLC.R1 is: createVM.compute.vplcR1.us-seattle.oci.oraclecloud.com (for creating a VM) deleteVM.compute.vplcR1.us-seattle.oci.oraclecloud.com (for erasing VMs) storeObject.storage.vplcR1.us-seattle.oci.oraclecloud.com (for data storage) etc., where "vplcR1" uniquely identifies the vPLC corresponding to the endpoint.

[0193] Also, an endpoint 614-1 for vPLC.R2 may be generated based on the endpoint 610-1. For example, the endpoint 614-1 for vPLC.R2 createVM.compute.vplcR2.us-seattle.oci.oraclecloud.com (for VM generation) deleteVM.compute.vplcR2.us-seattle.oci.oraclecloud.com (for erasing VMs) storeObject.storage.vplcR2.us-seattle.oci.oraclecloud.com (for data storage) etc., where "vplcR2" uniquely identifies the vPLC corresponding to the endpoint.

[0194] Because the endpoints associated with a vPLC include an identifier that identifies the particular vPLC, when the infrastructure 601 receives an API call, it can analyze the endpoint call to determine whether the call is for a direct customer of the CSP or for a reseller. Furthermore, because the endpoints use unique identifiers for different vPLCs, it is also possible for the endpoint call to identify the particular vPLC and associated reseller. Thus, based on the analysis of the endpoint call, the endpoint request can be appropriately routed and processed.

[0195] The CSP also provides a console 610-2 for use by its direct customers 640. Users associated with the CSP's direct customers 640 can use the console 610-2 to configure, access, and manage the first portion 602 of the CSP-provided regional infrastructure. In some embodiments, the console 610-2 may provide a set of web-based graphical user interfaces (GUIs) that can be used to access and manage the CSP-provided cloud infrastructure in the region. In some embodiments, the console is a web-based application.

[0196] Also, in one embodiment, a console is provided for each vPLC. For example, as shown in FIG. 6 , a console 612-2 is provided for vPLC.R1 and a console 614-2 is provided for vPLC.R2. A user associated with a customer 642 of reseller R1 can use the console 612-2 provided for vPLC.R1 to configure, access, and manage a second portion of the CSP-provided regional infrastructure assigned to vPLC.R1 604 created for reseller R1. A user associated with a customer 644 of reseller R2 can use the console 614-2 provided for vPLC.R2 to configure, access, and manage a third portion of the CSP-provided regional infrastructure assigned to vPLC.R2 606 created for reseller R2.

[0197] In some embodiments, a reseller may register with more than one vPLC, while each vPLC may still have a set of endpoints and a console. For example, if a reseller registers with two vPLCs (vPLC1 and vPLC2), vPLC1 may have a set of endpoints (e.g., 612-1) and a console (e.g., 612-2), and vPLC2 may have another set of endpoints (e.g., 614-1) and a console (e.g., 614-2). Both vPLC1 (e.g., 604) and vPLC2 (606) may be associated with the same reseller.

[0198] The infrastructure 601 may include networking resources that enable communication with the infrastructure 601. These network resources may include, for example, a network gateway 628 that enables communication of traffic from the CSP-provided regional infrastructure 601 to other destinations, which may be communicatively coupled to the infrastructure 601 via a public (e.g., the Internet) or private network, and may reside in an on-premise network, another cloud network, etc. The gateway 628 may be implemented as a physical network device (e.g., a router), as a logical networking entity (e.g., a virtual router), or as some combination of physical and logical entities.

[0199] The infrastructure 601 may include a control plane (CP) 624 and a management plane (MP) 626 provided by the CSP. In some embodiments, the CP 624 may be responsible for providing management, deployment, and orchestration functions for the cloud environment provided by the CSP. For example, the CP may receive requests from the CSP's direct customers, reseller R1's customers, and reseller R2's customers through endpoints and consoles, collaborate with other components in the infrastructure, such as resource managers and management planes, determine whether the requests can be fulfilled, and respond accordingly to the CSP's direct customers, reseller R1's customers, and reseller R2's customers. In other words, the CSP's CP 624 may be responsible for performing vPLC-related functions. In some embodiments, the CSP may also provide a vPLC-specific control plane configured specifically to perform vPLC operations. In some embodiments, there may be one vPLC control plane that can perform processing for multiple vPLCs. In other embodiments, each vPLC may have its own CP.

[0200] In some embodiments, the MP 626 may be a collection of software components responsible for provisioning all the processes and workflows required for a requested operation. For example, for a service request to access a cloud service, the MP may provision a workflow that includes checking the identity of the requester and checking DNS records to obtain an IP address to access the service backend. The MP ensures that the various tasks are performed correctly and in the correct order by the data plane.

[0201] In one embodiment, compute resources 620 may include one or more racks each including one or more servers hosting one or more operating systems. The division of computer resources 620 into resources 620a allocated to infrastructure for serving the CSP's direct customers 640 and resources 620b and 620c allocated to vPLCs may be performed along various levels or boundaries. The division of compute resources at a particular level may be referred to herein as a segmentation level. In one embodiment, the division may be performed at the rack level (i.e., rack-level segmentation), with one or more individual racks assigned to 620, 620b, and 620c. In this embodiment, a rack is exclusively assigned to a particular vPLC and is not used by other vPLCs or by the CSP to provide CSP-delivered services to its direct customers 640.

[0202] In another embodiment, the partitioning may be performed at the server level (i.e., segmented at the server level), with one or more individual servers in the same or different racks assigned to 620, 620b, and 620c. In this embodiment, a server is assigned exclusively to a particular vPLC and is not used by other vPLCs or by a CSP to provide CSP-delivered services to its direct customers 640. Because a rack may have multiple servers, one server on the rack may be assigned to 620a, another server to 620b, and a third server to 620c. Thus, a rack may be shared between two vPLCs or may be shared between a vPLC and the infrastructure used by the CSP to provide CSP-delivered services to its direct customers 640.

[0203] In some embodiments, the partitioning is performed at the hypervisor level (i.e., segmented at the hypervisor level), and one or more individual hypervisors on the same server or different servers on the same rack or different racks may be assigned to 620, 620b, and 620c. In this embodiment, a hypervisor is assigned exclusively to a particular vPLC and is not used by other vPLCs or by a CSP to provide CSP-delivered services to its direct customers 640. Because a server may host multiple hypervisors, one hypervisor on a server may be assigned to 620a, another hypervisor on the same server may be assigned to 620b, and a third hypervisor on the same server may be assigned to 620c. Thus, a server may be shared between two vPLCs or may be shared between a vPLC and the infrastructure used by a CSP to provide CSP-delivered services to its direct customers 640.

[0204] A hypervisor on a server on a rack can manage and run multiple virtual machines (VMs). In some embodiments, the partitioning is performed at the VM level (i.e., segmentation at the VM level), and VMs running with the same hypervisor can be assigned to 620, 620b, and 620c. In this embodiment, a VM is assigned exclusively to a particular vPLC and is not used by other vPLCs or by the CSP to provide CSP-delivered services to its direct customers 640. Because a hypervisor can support multiple VMs, one VM may be assigned to 620a, another to 620b, and a third VM on the same hypervisor may be assigned to 620c. Thus, a hypervisor may be shared between two vPLCs, or it may be shared between a vPLC and the infrastructure used by the CSP to provide CSP-delivered services to its direct customers 640.

[0205] In still other embodiments, a combination of the above partitioning techniques may be used. Regardless of the partitioning technique used, the partitioning is performed securely and safely so that traffic and storage destined for a particular vPLC for a particular reseller is not visible to or accessible by other resellers, their respective customers, or other direct customers of the CSP. Additionally, traffic and storage destined for a particular customer of a reseller is not visible to or accessible by other customers of that reseller.

[0206] Figure 7 illustrates the relationships between a CSP, its direct customers, its reseller customers, and their associated users, and each of the reseller's customers and their associated users, according to one embodiment. As shown in Figure 7, CSP 710 is at the top or root of the hierarchy. CSP 710 may provide one or more cloud services to which the CSP's customers may subscribe. In Figure 7, the CSP's customers are represented by 702. A tenancy, identified by a tenancy ID, is created for each registered customer of CSP 710.

[0207] Customers 702 of a CSP 710 can include direct or non-reseller customers and reseller customers. A reseller customer is an entity that subscribes to a vPLC service offered by a CSP, and a vPLC generated by the reseller as a result of subscription to the vPLC service is used to provide reseller-supplied and reseller-branded cloud services to the reseller's customer. A reseller's customer may be different from the CSP's customers. A direct or non-reseller customer of a CSP is an entity that subscribes to one or more CSP-supplied cloud services, but does not use the CSP's infrastructure to provide any cloud services of its own to its customers.

[0208] In the example shown in Figure 7, the direct or non-reseller customers of the CSP 710 include customer C1 726 with an associated tenancy T.CSP.C1 and customer C2 728 with an associated tenancy T.CSP.C2. There may be multiple such direct customers of the CSP 710. One or more users may be each direct customer of the CSP, and these users may use one or more of the CSP-supplied cloud services to which the corresponding direct customer subscribes. For example, users associated with a customer of the CSP may be employees or agents of the customer.

[0209] 7, reseller customers of a CSP 710 include reseller R1 720 with associated tenancy T.CSP.R1 and reseller R2 722 with associated tenancy T.CSP.R2. There may be multiple such reseller customers of a CSP 710. One or more users (not shown in FIG. 7) may be each reseller customer of the CSP.

[0210] Here, a reseller may have its own customers who subscribe to one or more reseller supply and reseller brand cloud services, which are provided to the reseller using a vPLC generated by the CSP. In Figure 7, customers of reseller R1 720 are indicated using reference number 704, and customers of reseller R2 722 are indicated using reference number 706.

[0211] A reseller's customers may each have their own tenancy and associated set of users. For example, as shown in FIG. 7 , reseller R1 may have multiple customers, including customer R1.C1 740 with tenancy T.R1.C1 and associated users, customer R1.C2 742 with tenancy T.R1.C2 and associated users, and others. Reseller R1 may supply a set of R1-supplied and branded cloud services, and R1's customers can register for one or more of the R1-supplied services. The set of one or more R1-supplied services that R1.C1 740 subscribes to may be the same as or different from the set of one or more R1-supplied services that R1.C2 742 subscribes to. The cloud services supplied by R1 722 may be different from the cloud services supplied by reseller R2 722 and the cloud services supplied by CSP 710.

[0212] Reseller R2 722 may have multiple customers subscribing to R2-supplied and branded cloud services provided using the vPLC created for R2. As shown in FIG. 7 , R2's 722 customers include customer R2.C1 744 with tenancy T.R2.C1 and associated users, customer R2.C2 746 with tenancy T.R2.C2 and associated users, and others. The cloud services supplied by R2 722 may be different from the cloud services supplied by reseller R1 720 and the cloud services supplied by CSP 710. R2's customers may subscribe to one or more of the R2-supplied cloud services. The set of one or more R2-supplied services that R2.C1 744 subscribes to may be the same as or different from the set of one or more R2-supplied services that R2.C2 746 subscribes to.

[0213] As shown in Figure 7, there are two levels of tenancies that come into play when a reseller uses vPLC to sell reseller-supplied cloud services to their customers. The first level of tenancies corresponds to tenancies associated with the CSP's 710 customers (including direct customers, i.e., non-reseller customers, and reseller customers). These tenancies may also be referred to as "CSP's Customer" tenancies (or CoCT). In Figure 7, the CoCT includes tenancies associated with reseller R1 720, reseller R2 722, direct customer C1 726, and direct customer C2 728. The second level of tenancies includes tenancies associated with the reseller's customers. These tenancies may also be referred to as reseller's customer tenancies (or CoRT). In Figure 7, CoRT includes tenancies associated with R1.C1 740, R1.C2 742, R2.C1 744, R2.C2 746, etc.

[0214] 8 illustrates the relationships between resources allocated to the tenancies of the CSP's direct customers, the CSP's reseller tenancies, and the tenancies of each reseller's customers, according to one embodiment. Referring to FIG. 8, in one embodiment, a CSP-provided infrastructure (e.g., data centers) 810 in a region may be divided into multiple portions that are securely isolated for the CSP's resellers, such as resources allocated to a vPLC.R1 820 created for reseller R1 and associated with reseller R1's tenancy (T.R1), and resources allocated to a vPLC.R2 822 created for reseller R2 and associated with reseller R2's tenancy (T.R2). The partitions of resources may also include partitions that are securely isolated for the CSP's non-reseller direct customers C1 and C2, such as resources 826 allocated to non-reseller customer CSP.C1's tenancy (T.CSP.C1) and resources 828 allocated to non-reseller customer CSP.C2's tenancy (T.CSP.C2). The partitions of CSP-provided regional infrastructure resources allocated to reseller R1's tenancy 820, reseller R2's tenancy 822, customer CSP.C1's tenancy, and customer CSP.C2's tenancy may be referred to as first-level resource partitions.

[0215] Each securely isolated portion of resources associated with a first-level resource partition or reseller's tenancy (i.e., the R1 tenancy and the R2 tenancy) may be assigned or tagged with a vPLC ID (i.e., resource ID), such as vPLC_ID.vPLC.R1 820 and vPLC_ID.vPLC.R2 822. Each of these CSP's non-reseller direct customer partitions may also be tagged with a resource ID. Each of the CSP's non-reseller direct customer tenancies is accessible to respective users, who can use one or more of the CSP-supplied cloud services to which each corresponding direct customer subscribes.

[0216] Within each first-level resource partition (or portion) of a reseller may be additional second-level resource partitions that are assigned to the tenancies of the reseller's customers. For example, the tenancy (T.R1.C1) associated with reseller R1's customer C1 may be assigned securely isolated resource partition (vPLC.R1.C1) 840, and the tenancy (T.R1.C2) associated with reseller R1's customer C2 may be assigned securely isolated resource partition (vPLC.R1.C2) 842. Isolation partition vPLC.R1.C1 may be tagged with a customer resource ID that identifies the resources assigned to T.R1.C1, and isolation partition vPLC.R1.C2 may be tagged with a customer resource ID that identifies the resources assigned to T.R1.C1.

[0217] Similarly, a tenancy (T.R2.C1) associated with reseller R2's customer C1 may be assigned securely isolated resource partition (vPLC.R2.C1) 844, and a tenancy (T.R2.C2) associated with reseller R2's customer C2 may be assigned securely isolated resource partition (vPLC.R2.C2) 846. The isolated partition vPLC.R2.C1 may be tagged with a customer resource ID that identifies the resources assigned to T.R2.C1, and the isolated partition vPLC.R2.C2 may be tagged with a customer resource ID that identifies the resources assigned to T.R2.C1. Each tenancy associated with a reseller's customer (e.g., R1.C1 840, R1.C2 842, R2.C1 844, or R2.C2 846) is accessible to one or more users associated with the customer who can use one or more of the reseller-supplied cloud services to which the corresponding customer subscribes.

[0218] 9A and 9B illustrate a mechanism for storing information in the tenancies of a CSP's individual reseller customers, according to one embodiment. In some embodiments, the vPLC IDs and tenancy IDs (sometimes referred to as customer tenancy IDs or CT IDs) of tenancies associated with a reseller's customers may be associated in two ways: 1) a distributed scheme (or record space per vPLC) as shown in FIG. 9A; and 7) a centralized scheme (or vPLC ID as the partition key) as shown in FIG. 9B. While FIGS. 9A and 9B show the information stored in a table format, in some embodiments the information may be stored in different formats, such as an array, a key-value store, XML, JSON, etc.

[0219] FIG. 9A illustrates a first approach, per-vPLC record space. In some embodiments, the per-vPLC record space may be an approach in which tenancy-related information for each reseller is stored in a vPLC-specific table under a vPLC ID. In other words, the information may be stored in a distributed manner. Each table is assigned a vPLC ID. In each table, one row may be designated for a customer of the reseller, another row may be designated for another customer of the reseller, and so on. For example, in FIG. 9A , there are two tables for storing information such as identification information (e.g., user logins, associated authentication information, and certificates) or billing information (e.g., resource usage, pricing). Table 910 is assigned the vPLC ID (vPLC_ID.vPLC.R1) of vPLC.R1 associated with reseller R1, and table 920 is assigned the vPLC ID (vPLC_ID.vPLC.R2) of vPLC.R2 associated with reseller R2. In table 910, there are three records for three customers C1, C2, and C3 of reseller R1. One record / row may include the tenancy ID (T_ID.T.R1.C1) of the tenancy associated with reseller R1's customer C1 and related information for customer C1 (e.g., identification information, billing, etc.). A second record / row may include the tenancy ID (T_ID.T.R1.C2) of the tenancy associated with reseller R1's customer C2 and related information for customer C2. A third record / row may include the tenancy ID (T_ID.T.R1.C3) of the tenancy associated with reseller R1's customer C3 and related information for customer C3.

[0220] The same structure can be used for table 920, storing three records for reseller R2's three customers C1, C2, and C3. One record / row may include the tenancy ID (T_ID.T.R2.C1) of the tenancy associated with reseller R2's customer C1 and related information for customer C1 (e.g., identification information, billing, etc.). A second record / row may include the tenancy ID (T_ID.T.R2.C2) of the tenancy associated with reseller R2's customer C2 and related information for customer C2. A third record / row may include the tenancy ID (T_ID.T.R2.C3) of the tenancy associated with reseller R2's customer C3 and related information for customer C3.

[0221] FIG. 9B illustrates a second approach using the vPLC ID as a partition key. In FIG. 9B, the second approach stores information in a table with two columns: a customer tenancy ID and a vPLC ID. Searches are performed using the vPLC ID as a partition key. In other words, information may be centrally stored in a table. For example, in FIG. 9B, a reseller R1 (vPLC_ID.vPLC.R1) associated with a vPLC ID has three customers C1, C2, and C3, and is associated with tenancy IDs T_ID.T.R1.C1, T_ID.T.R1.C2, and T_ID.T.R1.C3 that identify the tenancies of the three customers. In the vPLC ID column, the first three rows of table 930 may be marked as vPLC_ID.vPLC.R1, which is the vPLC ID of vPLC.R1 generated for reseller R1. The customer tenancy ID column in row 1 may be marked as the tenancy ID of the tenancy associated with reseller R1's customer C1 (T_ID.T.R1.C1), while the tenancy ID column in row 2 may be marked as the tenancy ID of the tenancy associated with reseller R1's customer C2 (T_ID.T.R1.C2), and the tenancy ID column in row 3 may be marked as the tenancy ID of the tenancy associated with reseller R1's customer C3 (T_ID.T.R1.C3). The information stored in the first three rows belongs to reseller R1's customers C1, C2, and C3.

[0222] The same structure can be used to store records for reseller R2's customers C1, C2, and C3 in the same table 930, e.g., the fourth, fifth, and sixth rows. For example, in the vPLC ID column, the last three rows of table 930 may be marked as vPLC_ID.vPLC.R2, which is the vPLC ID of vPLC.R2 generated for reseller R2. The customer tenancy ID column in row 4 may be marked as the tenancy ID (T_ID.T.R2.C1) of the tenancy associated with reseller R2's customer C1. Meanwhile, the tenancy ID column in row 5 may be marked as the tenancy ID (T_ID.T.R2.C2) of the tenancy associated with reseller R2's customer C2, and the tenancy ID column in row 6 may be marked as the tenancy ID (T_ID.T.R2.C3) of the tenancy associated with reseller R2's customer C3. The information stored in the last three rows belongs to reseller R2's customers C1, C2, and C3.

[0223] FIG. 10 is a flowchart illustrating an example of a vPLC configuration process, according to an embodiment. The process illustrated in FIG. 10 may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective systems, using hardware, or a combination thereof. The software may be stored in a non-transitory storage medium (e.g., a memory device). The method presented in FIG. 10 and described below is exemplary and not intended to be limiting. While FIG. 10 depicts various process steps occurring in a particular order or sequence, this is not intended to be limiting. In an alternative embodiment, the process may be performed in some different order, or some steps may be performed in parallel. Of course, in alternative embodiments, the process illustrated in FIG. 10 may include more or fewer steps than those depicted in FIG. 10.

[0224] For example, in one embodiment, the process illustrated in Figure 10 may be performed by a CSP, and some steps may be performed by components within a CSP-provided regional infrastructure (e.g., a control plane (CP), a resource manager (RM), and a management plane (MP)), or by one or more CSP-provided cloud services, such as an endpoint management service (EMS), an identity management service, etc.

[0225] As previously described, a vPLC may be created when a reseller registers for a CSP-provided vPLC service. In step 1002, the vPLC may be created by receiving a signal. In one embodiment, the signal may be a request received when a reseller registers for a CSP-provided vPLC service. The request may be an agreement between the CSP and the reseller, listing the virtual model (e.g., nested or segmented model), database, resource placement, customization details, etc. In another embodiment, a signal may be received from some component indicating that a vPLC is being created.

[0226] Step 1004 may include receiving configuration information for the vPLC to be created. In some embodiments, the configuration information may include information about the tenancy in which the vPLC is created (e.g., a tenancy identifier) ​​and information for identifying the realm associated with the vPLC. As described above, a tenancy is created for a reseller, and a tenancy identifier that uniquely identifies the tenancy is generated for the reseller associated with the vPLC.

[0227] As mentioned above, in use cases where a vPLC is used as a virtual data center and may be associated with a different realm than the realm associated with the cloud infrastructure in the CSP-provided region, the identity and trust profile information (e.g., passwords, credentials, etc.) of the different realm needs to be associated with the vPLC. The association of the identity and trust profile information of the different realm may be achieved by creating a mapping (or "shadow copy") of the identity and trust profile information in the vPLC. Further details are discussed in FIG. 18.

[0228] Step 1006, which includes steps 1008-1018, may involve performing a process to create a vPLC for a reseller R. Step 1008 may involve generating a vPLC identifier for the vPLC. The vPLC created for the reseller is treated as a resource and assigned a unique resource identifier, called a vPLC identifier (vPLC ID), which is associated with the reseller's tenancy.

[0229] In step 1010, a set of resources of a CSP-provided infrastructure in a region may be allocated to the vPLC. The set of resources allocated to the vPLC may be selected based on an agreement between the reseller and the CSP. For example, reseller R may subscribe to one or more cloud services offered by the CSP, such as compute services and object storage services. Thus, the set of resources may include compute resources and object storage resources.

[0230] In step 1012, a namespace may be reserved for the vPLC. The CSP's resource allocation service (or resource manager (RM)) may create and reserve a namespace that groups various types of resources that may support the vPLC. A namespace may represent a logical grouping or virtual boundary that helps organize and manage resources and prevent naming conflicts when multiple instances of similar resources are created. Thus, a namespace provides a structured way to organize and group related resources. In some embodiments, namespace 0 is reserved for the CSP. Namespaces 1 and beyond are used for the vPLC. In some embodiments, a wide range of Internet Protocol (IP) addresses may be reserved for the vPLC and associated with the namespace for traffic separation purposes.

[0231] In step 1014, a console may be generated for the vPLC that allows reseller R, the reseller's customers, and users associated with the reseller's customers to configure, access, and manage resources deployed in the vPLC, as well as visualize, access, and interact with reseller-supplied cloud services. Reseller R may also configure the console to provide different, customized experiences for different customers of the reseller.

[0232] In step 1016, a set of endpoints may be generated for the vPLC. In some embodiments, some of the set of endpoints may be default endpoints to be used (e.g., identity endpoints), and some endpoints may be determined based on the cloud services that the reseller registers. For example, the reseller may register for one or more CSP-supplied cloud services, such as compute, storage, virtual cloud network (VCN), database, etc. As a result, a default identity endpoint for accessing the identity services may be generated, as well as endpoints for accessing the registered services.

[0233] In step 1018, vPLC-related information such as the vPLC ID, the reseller's identification information, and other information facilitating the reseller-supplied cloud service may be stored in the vPLC namespace. In step 1020, the CSP may send a response indicating the created vPLC and other information to the reseller R. The reseller may then begin managing the vPLC for the reseller's customer, such as creating a customer account and configuring the reseller-supplied cloud service.

[0234] FIG. 11 is a block diagram of a distributed environment illustrating an example of a virtual private label cloud (vPLC) hosted by a CSP-provided infrastructure in a region, according to one embodiment. FIG. 11 may be similar to FIG. 6, and both figures include securely isolated portions (or partitions) of the CSP-provided regional infrastructure for the CSP's direct customers, vPLC.R1 associated with reseller R1, and vPLC.R2 associated with reseller R2 (e.g., first portion 602, second portion 604, and third portion 606 of FIG. 6). The CSP's direct customers, customers of reseller R1, and customers of reseller R2 may access their respective partitions through one or more endpoints and consoles. Also, as shown in FIG. 6, the CSP-provided regional infrastructure 602 may include a control plane, a management plane, and a gateway.

[0235] However, Figure 11 shows in more detail the interconnections between resources in the CSP-provided regional infrastructure, such as compute, storage, gateways, CPs, and the management plane. Additionally, the CSP's non-reseller direct customers, customers of reseller R1, and customers of reseller R2 may access the CSP-provided regional infrastructure from the Internet or a corporate network. For example, a gateway may be used to access the Internet or a corporate network from a Real Application Cluster (RAC) in the CSP-provided regional infrastructure.

[0236] FIG. 11 illustrates three cloud partitions running on the same CSP-provided regional infrastructure 1102: the CSP's native cloud 1120 and two vPLCs (vPLC.R1 1122 and vPLC.R2 1124) for resellers R1 and R2. The CSP-provided regional infrastructure 1102 is operated by the CSP. The CSP-provided regional infrastructure 1102 may include a control plane 1104 and a management plane 1105. As previously mentioned, the control plane 1104 may be responsible for providing management, deployment, and orchestration functions for the CSP-provided cloud environment. The management plane 1105 may be a collection of software components responsible for provisioning all processes and workflows required for requested operation. The software implementing the cloud service may originate from a deployment system 1106 and be integrated into data centers in the region through the management plane 1105. Software updates (e.g., new services) to the control plane 1104 may be delivered by a deployment system 1106 through the management plane 1105. The management plane 1105 also allows the CSP administrator or reseller operator 1107 deploying the software / services to access the CSP-provided regional infrastructure.

[0237] Compute resources, namely three Real Application Clusters (RACs) 1140, 1142, and 1144, sharing the same storage 1150, are partitioned and allocated to the CSP's native cloud 1120, vPLC.R1 1122, and vPLC.R2 1124, respectively. For example, RAC 1140 is dedicated to the CSP's direct customers, RAC 1142 is designated for vPLC.R1, and RAC 1144 is for vPLC.R2.

[0238] Network resources can also be divided or shared between the CSP's direct customers and resellers to connect their respective computing resources to the Internet or corporate network. For example, each RAC 1140, 1142, and 1144 uses its network gateway 1160, 1162, and 1164 to access the Internet 1108 or corporate network 1109. In some embodiments, two RACs may share the same network interface. For example, RAC 1142 and RAC 1144 share the same gateway 1162 to access the Internet 1108.

[0239] The storage in Figure 11 is simplified. Storage 1150 may include various types of storage, such as general storage, dedicated storage, block storage, and object storage. In Figure 11, vPLC.R1 and vPLC.R2 have dedicated storage 1152 and 1154, respectively. General storage 1150 belongs to the CSP.

[0240] In FIG. 11 , a CSP's native cloud 1120 may provide CSP-supplied cloud services to the CSP's non-reseller direct customers (e.g., CSP.C1-Cn) (e.g., businesses). Users associated with the reseller's customers can access the services through dedicated service endpoints that are part of a respective namespace, including a fully qualified domain name (FQDN), IP address, console experience, identity information, etc. The CSP may provide individual consoles for each reseller, and the consoles provide the services through the endpoints. The endpoints associated with the consoles may vary depending on the services provided to the reseller's customers based on the reseller's subscription. For example, individual reseller customers may submit access requests through dedicated endpoints or consoles.

[0241] Users can use service endpoints 1170 and 1172 (two endpoints are shown, but more are possible) published by the CSP, which have DNS names. The native cloud 1120 may also have a console 1174 with a look and feel. The CSP's direct customers on the Internet 1108 can request instance launches or terminations by accessing the CSP-provided regional infrastructure 1102 through public endpoints 1170 and 1172. Reseller R1's customers on the Internet 1108 can request instance launches or terminations by accessing the CSP-provided regional infrastructure 1102 through public endpoints 1180 and 1182. Reseller R2's customers on the corporate network 1109 can request instance launches or terminations by accessing the CSP-provided regional infrastructure 1102 through dedicated endpoints 1190 and 1192. The control plane 1104 can then launch instances, terminate instances, set up connections for these instances, and orchestrate services in the data center. Network interfaces 1160 and 1162 (e.g., Internet Gateways (IGWs)) of the CSP and vPLC.R1, respectively, can provide network interconnection to the Internet 1108. Network interface 1164 (e.g., Dynamic Routing Gateway (DRG)) of vPLC.R2 can provide network interconnection to the enterprise network 1109.

[0242] In FIG. 11 , vPLC.R1 1122 and vPLC.R2 1124 share the same physical infrastructure. Each endpoint for each reseller (e.g., endpoints 1180 and 1182 for reseller R1 and endpoints 1190 and 1192 for reseller R2) is unique. Consoles 1184 and 1194 are customized for reseller R1 and reseller R2, respectively. The vPLC endpoints and consoles can connect to the Internet or their respective corporate networks as needed. For example, in FIG. 11 , vPLC.R1 connects to the Internet 1108 through a network gateway 1162 (e.g., IGW). However, vPLC.R2 does not connect to the Internet; instead, it connects to the corporate network 1109 through a secure network 1164 (e.g., FastConnect). 11, gateway 1162 may be sliced ​​and shared by vPLC.R1 and vPLC.R2 and may be part of an orchestration implemented by a CSP. In this example, some of vPLC.R2's data may go to the Internet 1108 and some may go to its corporate network 1109.

[0243] If the request arrives through console 1194, it may go to control plane 604. The control plane may determine that the request arrived from vPLC.R2 1124, find the available resource (e.g., computer) from the RAC specified for vPLC.R2 (i.e., 1144 in this case), and provision and activate this resource. Control plane 604 may then provide an IP address for the network configuration (e.g., VCN) of reseller R2's customer through console 1194. Control plane 1104 can configure gateway 1164 to either the enterprise network 1109 or the Internet 1108, depending on the configuration. The control plane can recognize not only the request but also the requestor (either the CSP's direct customer, a customer of reseller R1, or a customer of reseller R2).

[0244] As mentioned above, partitioning of the CSP-provided regional infrastructure can be performed for different vPLCs at various predetermined infrastructure levels (e.g., rack level, server level, hypervisor level, or VM level). To further explain, in the case of partitioning at the rack level, a vPLC may have its own dedicated rack and its internal resources. In the case of partitioning at the server level, multiple vPLCs share a rack, while each vPLC has its own dedicated server. In yet another example, in the case of partitioning at the hypervisor level, multiple vPLCs share a server, while each vPLC has its own dedicated hypervisor. In the case of partitioning at the VM level, multiple vPLCs share a hypervisor, while each vPLC has its own dedicated VM. Thus, there can be two main vPLC infrastructure partitioning models (or two ends of a spectrum) for realizing a multi-vPLC scheme: a segmented model and a nested model. The segmented model can represent resource partitioning at the rack level. The nested model can represent resource partitioning at the VM level. However, other variations between the two are possible given operational efficiency (eg, control plane sharing) and fragmentation (ie, resource division) considerations.

[0245] FIG. 12 is a block diagram illustrating a vPLC segmentation model for resource partitioning, according to one embodiment. In this model, which represents one end of the partitioning spectrum, each vPLC may have its dedicated resources (including compute and network resources) at the rack level. In other words, a rack may be exclusively assigned to a particular vPLC and not used by other vPLCs or by a CSP to provide CSP-supplied services directly to customers. In the segmentation model, resource and usage tracking is performed at the physical resource level based on affinity tags (e.g., vPLC ID and customer tenancy ID). While there may be default behavior per segment, each segment can be customized by the resellers associated with the vPLCs in that segment.

[0246] In FIG. 12 , multiple vPLCs (e.g., vPLC.R1 1210, vPLC.R2 1220, and vPLC.Rn 1230) may exist in a CSP-provided infrastructure (e.g., a data center) in a certain region. In one embodiment, in a segmentation model, there may also be physical and logical partitions, as shown in FIG. 12 . For example, in the case of physical partitions, vPLCs may be assigned dedicated hardware resources (e.g., racks), each of which may include one or more servers hosting one or more operating systems. In FIG. 12 , a first rack is assigned to the vPLC.R1 segment 1210, a second rack is assigned to the vPLC.R2 segment 1220, and a third rack is assigned to the vPLC.Rn segment 1230. Each vPLC may also have a dedicated networking switch (also referred to as a gateway) that connects to the Internet / corporate network 1260. For example, vPLC.R1 1210 has gateway 1219, vPLC.R2 1220 has gateway 1229, and vPLC.Rn 1230 has gateway 1239. Thus, in FIG. 12 , vPLC.R1 1210 has dedicated servers 1214a and 1214b, a hypervisor 1216 running on server 1214b and hosting VMs 1218a-1218n, and a gateway (e.g., IGW) 1219. Similarly, vPLC.R2 1220 has dedicated servers 1224a and 1224b, a hypervisor 1226 running on server 1224b and hosting VMs 1228a-1228n, and a gateway 1229. The vPLC.Rn 1230 includes dedicated servers 1234a and 1234b, a hypervisor 1236 that runs on the server 1234b and hosts VMs 1238a to 1238n, and a gateway 1239.

[0247] In the case of logical partitioning, there is a common control plane 1250 and resource manager 1252 for all vPLCs that are managed and operated by the CSP. However, in a segmented model, a data plane may be dedicated to each vPLC. For example, each vPLC.R1, vPLC.R2, and vPLC.Rn has its own data plane 1212, 1222, and 1232, respectively. In some embodiments, the resource manager 1252 may be part of the management plane. In other embodiments, the resource manager 1252 may be separate from the management plane.

[0248] In some embodiments, a user associated with a customer 1280 of reseller R1 may launch a VM instance or a bare metal server by sending a request through one of vPLC.R1 endpoints 1270. Assume the request is to launch a new VM instance. In this case, control plane 1250 may work with an identity service to check the user's proper credentials for vPLC.R1, and resource manager 1252 may then identify all resources dedicated to vPLC.R1, such as an IP address range. This results in a VM instance (e.g., 1218a) being launched on hypervisor 1216 in vPLC.R1 segment 1210. Control plane 1250 then sends a response back to the requesting user through vPLC.R1 endpoint 1270. If the request is for a bare metal server (e.g., 1214a), resources for vPLC.R1 may be provisioned and allocated to vPLC.R1 with all the required settings by resource manager 1252. Similar processes may be performed such that requests from users associated with customers of reseller R2 1282 are fulfilled by vPLC.R2 segment 1220, and requests from users associated with customers of reseller Rn 1284 are fulfilled by vPLC.Rn segment 1230. Each process for a vPLC may be performed independently of and concurrently with other vPLCs.

[0249] The vPLC segmentation model offers several potential advantages, including physical separation and sharing of a common control plane 1250 operated by the CSP. Additionally, a resource manager 1252 manages resources across the vPLCs 1210-1230. However, scaling challenges may arise because resources, including networking and storage, are replicated as specified for each vPLC. This model also makes small footprints, such as a single rack or server, less feasible. However, the vPLC segmentation model still allows for sharing of power, cooling, and so on. As mentioned previously, the segmentation model is only one end of the spectrum, and there are various resource- and efficiency-based partitioning methods.

[0250] FIG. 13 is a block diagram illustrating a vPLC nesting model for resource partitioning, according to one embodiment. The nesting model of FIG. 13 illustrates the other end of the resource partitioning spectrum at the VM level. In other words, a VM may be exclusively assigned to a particular vPLC and not used by other vPLCs or by the CSP to provide CSP-delivered services directly to its customers. However, hypervisors, servers, or racks may be shared between the CSP's customers and the customers of one or more vPLCs. In some embodiments, the nesting model may have a base layer (such as a hypervisor) that is invisible to the customer as a shared foundation. Abstraction layers (e.g., VMs) may also be built on top of the base layer, allowing each reseller to clone and customize the abstraction layers for their customers.

[0251] In the embodiment of FIG. 13 , all vPLCs (e.g., vPLC.R1, vPLC.R2, and vPLC.Rn) are hosted on a single segment with a shared infrastructure represented as a “server CSP” 1320 (e.g., a bare metal server) operated by a CSP, which may have control plane functionality. The server CSP 1320 may be configured to host a hypervisor 1330 that manages virtual machines (VMs) 1332a-1332n running on the hypervisor. In some embodiments, a segment may include multiple racks. Here, two VM instances for vPLC.R1 1332a and 1332b, one VM instance for vPLC.R2 1332c, and one VM instance for vPLC.Rn 1332n reside on the shared hypervisor 1330. In segment 1310, all vPLCs share the same data plane 1312. In some embodiments, a segment may include multiple racks. Since hypervisors, servers, and racks can be shared by multiple vPLCs, the servers of these vPLCs may come from different racks.

[0252] 13 further illustrates a variation of the splitting scenario in which a customer of reseller R2 may request an additional bare metal server, server vPLC.R2 1322, that is provisioned as a dedicated server. In the vPLC nesting model, a shared server pool can be placed wherever space is available, thus providing a low-cost proposition to the reseller.

[0253] In FIG. 13 , a gateway 1340 (e.g., IGW or DRG) shared by the vPLCs has a connection to the Internet or a corporate network as needed. Incoming packets from the Internet / corporate network 1360 may be tagged (i.e., encapsulated in each packet header) with vPLC-related information (e.g., a vPLC ID and a customer tenancy ID) at the gateway 1340 for tracking purposes to reach the appropriate vPLC. Packet encapsulation helps separate traffic on a single segment 1310 shared by multiple vPLCs. As an example, a packet destined for VM 1332a, which belongs to a tenancy associated with customer C1 of reseller R1, which is associated with vPLC.R1, may have vPLC_ID.vPLC.R1 and customer tenancy ID (T_ID.R1.C1) encapsulated in the packet header. The packet may then be routed to VMs 1332a and 1332b separated for vPLC.R1 based on the vPLC ID (vPLC_ID.vPLC.R1). The customer tenancy ID (T_ID.R1.C1) associated with reseller R1's customer C1 in the packet header may further aid in routing the packet to tenant C1 running on VM 1332a, where the packet is decapsulated before delivery to VM 1332a.

[0254] Similarly, a packet destined for VM1332c, which belongs to a tenancy associated with customer C1 of reseller R2, which is associated with vPLC.R2, may have vPLC_ID.vPLC.R2 and a customer tenancy ID (T_ID.R2.C1) encapsulated in the packet header. The packet can then be routed to resources carved out for vPLC.R2 based on the vPLC ID (vPLC_ID.vPLC.R2). The customer tenancy ID (T_ID.R2.C1) associated with customer C1 of reseller R2 in the packet header can further aid in routing the packet to tenant C1 running on VM1332c in vPLC.R2. The packet is decapsulated before delivery to VM1332c.

[0255] Outgoing packets may perform the reverse process. For example, an outgoing packet originating from VM 1332a in vPLC.R1 and destined for the Internet 1360 may have vPLC_ID.vPLC.R1 and a customer tenancy ID (T_ID.R1.C1) associated with reseller R1's customer C1 encapsulated in the packet header. When the packet reaches gateway 1340, it may be decapsulated before proceeding to the Internet 1360. Similarly, an outgoing packet originating from VM 1332c in vPLC.R2 and destined for the Internet 1360 may have vPLC_ID.vPLC.R2 and a customer tenancy ID (T_ID.R2.C1) associated with reseller R2's customer C1 encapsulated in the packet header. When the packet reaches gateway 1340, it may be decapsulated before proceeding to the Internet 1360. As a result, both inbound and outbound traffic are separated in a single segment 1310 shared by multiple vPLCs.

[0256] In some embodiments, in a nested model, one or more VMs may belong to a direct customer of the CSP and share the same hypervisor as other vPLCs.

[0257] Reseller users accessing the vPLC infrastructure are authenticated in both the segmented and nested models. Users associated with a particular reseller's customer can access an endpoint and authenticate as a user of the vPLC associated with that particular reseller using the user's credentials (e.g., username and customer tenancy ID). For example, in some embodiments, in FIG. 13 , a user associated with reseller R1's customer C1 may launch a VM instance by sending a request through one of the vPLC.R1 endpoints 1302. The particular vPLC.R1 endpoint 1302 may augment the request with the vPLC ID of vPLC.R1. The control plane 1350 may work with an identity service to check the user's proper credentials for vPLC.R1 (e.g., the user's username and customer C1's tenancy ID). Resource manager 1352 can then identify all resources assigned to vPLC.R1, e.g., server 1320 and hypervisor 1330 based on the vPLC ID and VM 1332a based on the customer tenancy ID for customer C1, and VM instance 1332a is then launched on hypervisor 1330 accordingly.

[0258] 14 is a flow diagram illustrating a sign-up process for a customer of a reseller associated with a vPLC, according to one embodiment. When a CSP's direct customer registers or subscribes (i.e., signs up) for a cloud service offered by the CSP (i.e., a CSP-provided cloud service), a tenancy or account is created for that customer. Similarly, when a reseller's customer registers or subscribes for a cloud service offered by the individual reseller (i.e., a reseller-provided cloud service), a tenancy or account is created for that customer.

[0259] FIG. 14 illustrates the sign-up process for a particular customer C1 of reseller R1. However, this sign-up process is also applicable to customers of individual resellers. In FIG. 14, in step S1, reseller R1's customer C1 1401 may sign up using a sign-up portal on the console of vPLC.R1 1402 to create a new account by entering user ID credentials (user name or account name (e.g., John)). In step S2, vPLC.R1's console 1402 may enhance the sign-up request by adding a vPLC ID (e.g., vPLC_ID.vPLC.R1) that identifies the vPLC created for reseller R1 to the user credentials and forward the sign-up request to a CSP (control plane (CP) or management plane (MP)) 1404. In some embodiments, the account creation for reseller R1's customer C1 may be performed by reseller R1 with assistance from the CSP's CP or MP.

[0260] Then, in step S3, the CSP creates a customer tenancy (or account T.R1.C1) for reseller R1's customer C1 1401. The newly created tenancy (T.R1.C1) for customer C1 is assigned a tenancy identifier (T_ID.T.R1.C1). A vPLC ID (e.g., vPLC_ID.vPLC.R1) identifying vPLC.R1 for reseller R1 is associated with the newly created customer tenancy ID (e.g., T_ID.T.R1.C1) for customer C1 (shown in 1430) and may be stored in database 1420 along with other authentication information (e.g., account name and password). Other customers of reseller R1 and reseller R2 may also be stored in database 1420. For example, as shown at 1432, the customer tenancy ID of reseller R1's customer C2 (i.e., R1.C2) is associated with the vPLC ID of vPLC.R1 (T_ID.T.R1.C2 and vPLC_ID.vPLC.R1). Also, as shown at 1434, the customer tenancy ID of reseller R2's customer C2 (i.e., R2.C1) is associated with the vPLC ID of vPLC.R2 (T_ID.T.R2.C1 and vPLC_ID.vPLC.R2).

[0261] The association between the vPLC ID and the tenancy IDs of individual resellers' customers can enable isolation between customer tenancies within a vPLC. For example, customer tenancy R1.C1 can be securely isolated from customer tenancy R1.C2 by using different customer tenancy IDs (T_ID.T.R1.C1 for customer C1 and T_ID.T.R1.C2 for customer C2), but both customer tenancies reside in the same vPLC.R1. The tenancies associated with reseller R1 and reseller R2 can be securely isolated by using their respective vPLC IDs (vPLC_ID.vPLC.R1 for reseller R1 and vPLC_ID.vPLC.R2 for reseller R2).

[0262] In step S4, CSP 1404 may return a handle including the customer tenancy ID (e.g., T_ID.T.R1.C1), a temporary default password, and an account name to vPLC.R1 console 1402. In step S5, the vPLC.R1 console may receive the temporary authentication information and forward the customer tenancy ID and password to reseller R1's customer C1 1401. As shown in FIG. 14, reseller R1's customer C1 1401 can sign up for and register for one or more reseller-supplied cloud services by interacting directly with reseller R1 through vPLC.R1 console 1402. Similarly, a CSP's customer may register for a CSP-supplied cloud service by separate interaction and sign-up with the CSP at the CSP's console (not shown).

[0263] As described above with respect to creating a tenancy, a CSP's database (e.g., 1420 in FIG. 14) may store vPLC IDs, customer tenancy IDs, and their corresponding authentication information. In some embodiments, multiple customer tenancy IDs may be associated with the same vPLC ID, as shown in 1430 (T_ID.T.R1.C1 is associated with vPLC_ID.vPLC.R1) and 1432 (T_ID.T.R1.C2 is associated with vPLC_ID.vPLC.R1). Tenancies may be defined at the realm level and thus may have a realm identifier associated with a vPLC.

[0264] 15 is a flow diagram illustrating a login process for a user associated with a customer of a reseller associated with a vPLC, according to one embodiment. In FIG. 15, steps S1 and S2 include a login phase after an individual reseller customer completes the sign-up process described in FIG. 14.

[0265] In step S1, a user 1501 associated with a customer of reseller R1 can log in with a customer tenancy ID (or CT ID), username, and password at the vPLC.R1 endpoint 1510. In S2, the user 1501 and their customer tenancy may be verified at the vPLC.R1 endpoint 1510 by going through a verification process, including, but not limited to, user authentication (e.g., verifying valid credentials such as username and password), session management (e.g., session tokens or timeouts), and access control enforcement (based on access privileges, permissions, and policies).

[0266] In S3, the vPLC.R1 endpoint 1510 may return a session handle. The session handle may represent a token or unique identifier returned to a customer after successfully logging into their account. The session handle is used for subsequent requests to access cloud services and resources (e.g., launch an instance) without requiring re-authentication for each individual request during an active login session (or API session).

[0267] When resources are created, a cloud resource ID (CID) may be generated for those resources, which includes the customer tenancy ID. Virtual machines, accounts, tenancies, permission policies, virtual network subnets, ACLs, and load balancers are considered examples of cloud resources and can therefore be referenced by their respective CIDs.

[0268] 16 is a flow diagram illustrating a resource allocation and provisioning process for a reseller's customer associated with a vPLC, according to one embodiment. A resource manager (RM) manages the resources of a CSP-provided cloud infrastructure in a region and the resource partitions allocated to each vPLC.

[0269] As described above, a user associated with a customer of a reseller may use a vPLC-specific endpoint to access resources associated with the vPLC and perform one or more functions or operations involving resources allocated to the vPLC. In Figure 16, in step S1, a user 1601 associated with a customer C1 of reseller R1 may launch a compute instance (e.g., a VM instance with 1 GB of RAM) by issuing a request through a vPLC-specific endpoint, vPLC.R1 endpoint 1602.

[0270] In step S2, the vPLC.R1 endpoint 1602 may extend (or annotate) the request by tagging it with the vPLC.R1 endpoint's corresponding vPLC ID (e.g., vPLC_ID.vPLC.R1) to indicate that the request originates from a customer of reseller R1, and forwards the request to resource manager (RM) 1604. In other words, the request, once authenticated by the annotation information (e.g., vPLC ID), can be processed by all policies associated with vPLC.R1. In some embodiments, policies associated with a vPLC may include, for example, the type of resource, the amount of resource that can be allocated (i.e., capacity limits), permissions, etc.

[0271] In step S3, the RM 1604 determines whether the requested resources are available to vPLC.R1 under the policy by querying its database (i.e., resource database) 1606. Based on this query, in step S4, after confirming that the requested resources are available under the policy, the RM may perform resource allocation. In other words, the RM allocates a portion of the resources allocated to vPLC.R1 for reseller R1 to a customer tenancy (T.R1.C1) associated with reseller R1's customer C1 (based on the customer tenancy ID at user login). This resource allocation may be performed by allocating or designating a portion of the available cloud resources (allocated to vPLC.R1) to a user associated with customer C1 based on a policy agreed upon between customer C1 and reseller R1, such as capacity limits or permissions. For example, reseller R1 and its customer C1 may have a contract policy that allows for the use of up to 100 GB of memory. Currently, 80 GB is allocated (or used). If a user associated with customer C1 requests 10 GB, the RM can identify the 10 GB CID provisioned by the CP. If a user associated with customer C1 requests 30 GB, this exceeds the total 100 GB limit, and the request can be rejected.

[0272] In step S5, the RM 1604 retrieves from the resource database a cloud resource ID assigned to a resource (vPLC.R1.C1) allocated to the tenancy (T.R1.C1) of the customer C1 of the reseller R1. In step S6, the RM passes the resource ID to the vPLC.R1 endpoint 1602. In step S7, the vPLC.R1 endpoint 1602 requests a resource from the control plane (CP) 1608 for a resource having a particular resource ID assigned to the resource allocated to the tenancy (T.R1.C1) of the customer C1 of the reseller R1. In other words, the resource manager 1604 can augment the user's request (e.g., step S6) with information associated with the vPLC (e.g., the resource ID assigned to vPLC.R1.C1) that the control plane 1608 can use to provision the resource (e.g., step S8).

[0273] In step S8, the control plane may provision the requested resources (e.g., launch a specific VM instance and storage). The resources may include compute, networking, storage, etc. In step S9, the control plane 1608 may respond to the vPLC.R1 endpoint 1602 with a status and a resource handle. The resource handle may represent a unique resource identifier assigned to the provisioned resource (e.g., a cloud resource ID assigned to the provisioned compute instance (e.g., VM), storage volume, virtual cloud network (VCN or subnet), database, etc.). The resource handle may enable the customer to access and manage the provisioned resource through an API or command line tools.

[0274] In step S10, the status and resource handle are forwarded to user 1601 associated with customer C1 of reseller R1. In step S11, the control plane may notify the RM regarding the resource provisioning status (e.g., success or failure). In step S12, the RM may update a record in resource DB 1606 that tracks resource usage for customer C1 associated with the user. Continuing with the above example, the record may be updated to indicate that the total amount of memory used by customer C1 is 90 GB after successfully provisioning 10 GB in addition to the previously provisioned 80 GB, or that the total remains at 80 GB after the user's request was denied. Records maintained by the resource DB may be useful for enforcing policies associated with vPLCs.

[0275] Figure 17 is a flow diagram illustrating a vPLC resource allocation and provisioning process for a reseller's customer associated with a vPLC, according to one embodiment. Figure 17 is an alternative embodiment to Figure 16. Instead of using a Resource Manager (RM) as a central role for communicating with all subsystems and receiving requests to coordinate the resource allocation and provisioning process, a Control Plane (CP) may also be used for this central role.

[0276] For example, in FIG. 17 , in step S1, a user 1701 associated with a customer C1 of reseller R1 may issue a request to a vPLC-specific endpoint, vPLC.R1 endpoint 1702. In step S2, vPLC.R1 endpoint 1702 may extend the request by tagging it with a corresponding vPLC ID to indicate that the request originates from a customer of reseller R1, and forwards the request to control plane (CP) 1704. In step S3, CP 1704 forwards the request to resource manager (RM) 1706. In step S4, RM 1706 determines whether the requested resources are available for vPLC.R1 by querying its database (i.e., resource database) 1708. Based on this query, in step S5, if the requested resources are available, the RM performs resource allocation.

[0277] In step S6, the RM retrieves from the resource database the resource ID assigned to the resource (vPLC.R1.C1) allocated to the tenancy (T.R1.C1) of the customer C1 of the reseller R1. In step S7, the RM passes the resource ID to the CP 1704. In step S8, the control plane may provision the requested resource. In step S9, the control plane 1704 may respond to the vPLC.R1 endpoint 1702 with a status and a resource handle. In step S10, the status and resource handle may be forwarded to the user 1701. In step S11, the control plane may notify the RM regarding the provisioning status of the resource. In step S12, the RM may update a record in the resource DB that tracks resource usage for the customer associated with the user.

[0278] 17, the CP 1704 may play a central role by forwarding the resource request to the RM 1706 (e.g., step S3), receiving the resource ID from the RM (e.g., step S6), and provisioning the requested resources (e.g., step S8). Compared to FIG. 16, this embodiment avoids the need for the vPLC.R1 endpoint 1702 to perform extra steps, such as requesting resources from the CP as shown in steps S7 and S8 of FIG. 16.

[0279] FIG. 18 is a simplified diagram illustrating a use case in which a vPLC is hosted by the infrastructure of a first realm while associated with a second, different realm, according to one embodiment. In FIG. 18, there are two separate realms (CSP-provided realm A 1802 and realm B 1812). Each realm is a logical collection of regions. Realms are isolated from each other and do not share data. Thus, realm A 1802 is isolated from realm B 1812, and there is no data sharing or communication between the two realms. A tenancy created for a CSP customer exists only in one realm and cannot be used to access another realm or region in another realm. Realms span regions within a realm, allowing a CSP to provide a specified level of service that meets the needs of a particular organization. For example, for a high level of security and isolation, a CSP may create realms, such as a “Government” realm created for government entities that are customers of the CSP. A CSP may also create a second realm with a lower level of security and isolation, such as a "Commercial" realm for the CSP's non-government commercial customers. Examples of realms a CSP may create are (a) a Commercial realm, (b) a US Government FedRAMP Authorized realm, (c) a US Government IL5 Authorized realm, (d) a Country A Government realm, (e) a Country B Government realm, etc. For example, in FIG. 18, realm A 1802 may have a higher security posture than realm B 1812.

[0280] A realm can have one or more regions. Each realm has its own identity and trust profile (e.g., password, authentication information, etc.). This identity and trust profile information is shared between regions in the same realm, so regions within a realm can communicate with each other. The realm's identity and trust profile are configured to prevent inter-realm communication.

[0281] In the example shown in Figure 18, realm A 1802 includes region A 1804. The infrastructure provided by the CSP in region A 1804 is organized as datacenter 1806. In Figure 18, datacenter 1806 is shown as a dedicated regional cloud datacenter (DRCC) hosted on customer premises. A DRCC may represent a fully managed cloud region built with CSP-designed high-performance infrastructure to help customers bring cloud primitives and services closer to their existing data and applications. A DRCC may be dedicated to a single customer and operate within the customer's datacenter while being fully managed by the CSP.

[0282] In FIG. 18 , realm B 1812 includes region B 1814. The infrastructure provided by the CSP in region B 1814 is organized as data center 1816. As shown in FIG. 18 , data center 1816 hosts vPLC 1818. Thus, a portion of the infrastructure provided by the CSP in data center 1816 of region B 1812 is allocated to vPLC 1818. Thus, vPLC 1818 is physically located in region B 1814 of realm B 1812.

[0283] As previously discussed, when a vPLC is created, it is associated with a particular realm as part of its configuration. Typically, a vPLC is associated with the same realm associated with the region in which the CSP-provided regional infrastructure hosting the vPLC is located. In some embodiments, when a vPLC is configured, it may be associated with a realm different from the realm associated with the regional infrastructure hosting the vPLC. For example, in the example shown in FIG. 18 , vPLC 1818 may be associated with Realm A 1802 rather than Realm B 1812. The dashed box 1820 and associated arrow 1830 are meant to indicate that vPLC 1818 is physically hosted by the data center 1816 of Realm B 1812, but is logically or virtually associated with and belongs to Realm A 1802. Realm B 1812 is referred to as the hosting realm of vPLC 1818. Realm A 1802 is referred to as the logical or virtual realm of the vPLC 1818.

[0284] As a result, vPLC 1818 and datacenter 1806 exist in the same realm, i.e., realm A 1802. Because infrastructure in the same realm shares the same identity and trust profile, vPLC 1818 is configured with the identity and trust profile of realm A 1802. Configuring the identity and trust profile information of realm A 1802 can be achieved by creating a mapping in vPLC 1818. For example, the identity service API of the virtual realm (i.e., realm A 1802) can call the vPLC of the host realm (i.e., realm B 1812), and the host realm can accept the call and establish trust between the two realms (i.e., share identity and trust profile information). When an instance is created in a customer tenancy of the vPLC of the host realm, the resource ID and tenancy ID associated with the newly created instance in the host realm can be mapped to a new resource ID and tenancy ID in the virtual realm. Thus, a table of mappings (or "shadow tenancy") may be maintained in the host realm (i.e., realm B 1812). The same process and mapping occurs in the virtual realm (i.e., realm A 1802). Thus, both vPLC 1818 and realm A 1802 (i.e., the virtual realm) share the same identity and trust profile information. As a result, even though vPLC 1818 is physically hosted by a different realm B 1812 infrastructure, vPLC 1818 and datacenter 1806 can communicate with each other because they reside in realm A 1802. vPLC 1818 may not be able to communicate with other infrastructure in region B 1814 associated with realm B 1812.

[0285] Additionally, vPLC 1818 is physically hosted by region B 1814 but is logically associated with region A 1804. Region B 1814 is referred to as the hosting region or host region of vPLC 1818. Region A 1804 of realm A 1802 is referred to as the logical region or virtual region of vPLC 1818. When a vPLC is created, a virtual realm and virtual region for the vPLC may be configured.

[0286] As described above, a resource ID for a resource includes a portion that identifies the realm associated with the resource and a portion that identifies the region associated with the resource. A vPLC is considered a resource and is therefore assigned a resource ID. In one embodiment, a vPLC ID, which uniquely identifies a vPLC, is a type of resource ID. A vPLC ID includes a portion that identifies the virtual realm associated with the vPLC identified by the vPLC ID and a portion that identifies the virtual region associated with the vPLC. Thus, for purposes of identity management in Realm A 1802, a vPLC is considered part of the realm and region identified by the vPLC ID corresponding to the vPLC.

[0287] In one embodiment, when a vPLC is hosted by a region and a realm but associated with a different virtual region and a different virtual realm, the configuration of the vPLC as described above may be performed using Virtual Bootstrap Environment (ViBE) technology. A ViBE may represent a virtual cloud network (VCN) provisioned in an overlay of an existing region (e.g., a "host region"). The provisioned ViBE is connected to the new region using a communication channel (e.g., an IPSec Tunnel VPN). Certain essential core services (or "seed" services), such as a deployment orchestrator and public key infrastructure (PKI) services, may be provisioned in the ViBE. These services can provide the functionality necessary for bringing hardware online, establishing a chain of trust to the new region, and deploying other services in the new region. The virtual bootstrap environment allows for the use of resources in the host region to prevent circular dependencies between bootstrap resources. Services can be staged and tested in the ViBE before the physical region (e.g., the target region) is available.

[0288] For example, in some embodiments, Realm B 1812 may create a ViBE within its realm, which shares identity and trust profile information with Realm A 1802. The ViBE can then be used in Realm B 1812 to bootstrap new regions in Realm A 1802. Additional details related to ViBEs are described in the following applications, the contents of all of which are incorporated herein by reference:

[0289] (1) U.S. Patent Application No. 18 / 105,779, filed February 3, 2023, entitled "VIRTUAL BOOTSTRAP ENVIRONMENT FOR BUILDING REGIONAL DATA CENTERS" (2) U.S. Patent Application No. 18 / 105,768, filed February 3, 2023, entitled "TECHNIQUES FOR A VIRTUAL BOOTSTRAP ENVIRONMENT IN A DISTRIBUTED VIRTUAL PRIVATE NETWORK" (3) U.S. Patent Application No. 18 / 105,779, filed February 3, 2023, entitled "TECHNIQUES FOR MIGRATING SERVICES FROM A VIRTUAL BOOTSTRAP ENVIRONMENT" In the architecture shown in FIG. 18 , a vPLC is hosted by a region and a realm, but is associated with a different virtual region and a different virtual realm, which can be used for various purposes. For example, in the example shown in FIG. 18 , vPLC 1818 may function as a backup for data center 1806. This allows for disaster recovery (DR) backups in various regions. For example, if data center 1806 is a DRCC located on a customer's premises, the customer may request a CSP to provide backup services for DRCC 1806. In response, the CSP may select vPLC 1818 as a backup for DRCC 1806. vPLC 1818 is physically hosted in region B 1812, which is in a different region away from region A 1802, which hosts DRCC 1806. This may be a cost-effective solution because the CSP does not need to build a separate data center for the customer of realm A 1802 just for backup purposes. In some embodiments, the customer may not even be aware of where the backup is being performed, i.e., the presence and location of the vPLC 1818 may be transparent to the customer. The vPLC used for backup may be physically hosted in a CSP-provided cloud in a different realm and region.

[0290] Since the purpose of a DR site is to minimize the occurrence of correlated failures by introducing geographic separation between the primary and DR sites and avoiding simultaneous changes to both the primary and secondary locations, the DRCC 1806 can enjoy DR guarantees if the CSP can ensure that the DRCC and its paired vPLC 1818 (and therefore its hosting region in Realm B 1812) do not change simultaneously.

[0291] 19 is a flowchart 1900 illustrating an exemplary method for creating a vPLC hosted by a CSP provisioning infrastructure in a particular region in a particular realm, while virtually associated with a different region in a different realm, according to one embodiment. The process illustrated in FIG. 19 may be performed as part of the process performed for creating the vPLC. In one embodiment, the process illustrated in FIG. 19 may be performed primarily by the CSP provisioning infrastructure in the hosting region and realm.

[0292] Processing begins at 1902 with the receipt of information identifying a host region of a host realm in which a vPLC will be physically hosted. For example, in the embodiment shown in Figure 18, information may be received that a vPLC will be created in region B 1814 of realm A 1812. The region and realm identified at 1902 represent the host region and host realm of the vPLC to be created.

[0293] At 1904, information may be received that identifies a region of a particular realm. The particular realm is different from the host realm identified at 1902. Furthermore, the region and particular realm identified at 1904 will be associated as a virtual region and virtual realm of the vPLC being created. For example, in the embodiment shown in FIG. 18, information may be received that the vPLC is to be virtually associated with region A 1804 of realm A 1802.

[0294] At 1906, identity and trust profile information is obtained for the particular realm identified at 1904. The information obtained at 1906 may include identity authentication information (including passwords, certificates, etc.) associated with the realm that will be the virtual realm of the vPLC.

[0295] At 1908, a vPLC ID is generated for the vPLC being created. The vPLC ID identifies the region and particular realm identified in the information received at 1904.

[0296] At 1910, a vPLC is created in the host realm and resources are allocated to the vPLC. The allocated resources are selected from resources in the infrastructure provided by the CSP in the region and realm identified in 1902. These resources may include one or more physical resources (e.g., racks, servers, routers, memory, or storage resources) and / or logical resources (e.g., hypervisors, virtual machines, virtual routers). For example, in the embodiment shown in FIG. 18, resources are allocated to the vPLC from a data center 1816.

[0297] At 1912, a vPLC is configured using the identity and trust profile information obtained at 1906. For example, in the embodiment shown in Figure 18, the vPLC may be configured using the identity and trust profile information for Realm A 1802.

[0298] At 1914, the vPLC initiates communication with the CSP-provided infrastructure in the particular realm (the vPLC's virtual realm) identified at 1904. For example, in the embodiment shown in Figure 18, the vPLC 1818 communicates with the data center 1806. This communication may be initiated by the vPLC 1818 or by the data center 1806.

[0299] Virtual Private Label Cloud (vPLC) - Metadata Customization FIG. 20 is a block diagram illustrating a distributed environment that provides metadata services, according to one embodiment. The distributed environment 2000 illustrated in FIG. 20 is merely an example and is not intended to unduly limit the scope of the claimed embodiments. Many variations, alternatives, and improvements are possible. For example, in some implementations, the distributed environment 2000 may include more or fewer systems or components than those illustrated in FIG. 20, may combine two or more systems, or may have different system configurations or arrangements. The systems, subsystems, and other components illustrated in FIG. 20 may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective systems, or may be implemented using hardware, or a combination thereof. The software may be stored in a non-transitory storage medium (e.g., a memory device).

[0300] 20, a distributed environment 2000 may include a vPLC instance 2020, a metadata service (MD) system 2030, a global repository / memory 2050, and a set of endpoints (2002, 2004, and 2006). As shown in FIG. 20, the metadata service system 2030 may include, but is not limited to, a metadata collection subsystem (referred to as a metadata collector) 2032 for collecting metadata information, a metadata customizer 2034 for customizing metadata, a metadata responder 2036 for responding to requests for metadata, and an interface subsystem 2038 for providing an interface to the metadata service system. Additionally, a set of vPLC instances 2020, including vPLC.R1 2024, vPLC.R2 2026, etc., may be communicatively coupled to the metadata service system 2030.

[0301] The overall repository 2050 may be a non-volatile memory (NV), a database, a disk, etc., and may include three repositories used to store information from or provide information to the metadata service system: (1) a first repository (CSP format metadata repository) 2052 for storing CSP format metadata; (2) a second repository (vPLC specific information repository) 2054 for storing vPLC specific information for customization commands and response commands; and (3) a third repository (vPLC specific metadata repository) 2056 for vPLC specific metadata.

[0302] A set of vPLC-specific endpoints and consoles (vPLC.R1 2002, vPLC.R1 2004, and vPLC.R1 2006) may be provided to customers 2082 of reseller R1, customers 2084 of reseller R2, and customers 2086 of reseller Rn to access and perform functions related to the metadata service.

[0303] As shown in FIG. 20 , vPLC instances 2020 may include vPLCs allocated to resellers from a CSP-provided infrastructure in a region. For example, vPLC.R1 2024 is an isolated portion of a CSP-provided infrastructure in a region and may be allocated to reseller R1 to provide reseller-provided cloud services, such as metadata services. vPLC.R2 2026 is an isolated portion of a CSP-provided infrastructure in a region and may be allocated to reseller R2 to provide reseller-provided cloud services, such as metadata services. In some embodiments, each vPLC instance may include resources such as compute, storage, databases, and networks. The resources of each vPLC instance may have associated metadata, such as a region, fault domain, and domain name.

[0304] In FIG. 20 , a metadata collector 2032 of a metadata service system 2030 may collect metadata associated with resources of each vPLC instance. The collected metadata associated with resources of a vPLC instance (referred to herein as raw metadata (i.e., uncustomized)) may be in CSP format and stored in a first repository (CSP format metadata repository) 2052. The raw metadata is in CSP format (e.g., JSON blob) because vPLC resources are allocated from a CSP-provided infrastructure in a certain region. In one embodiment, the CSP format metadata 2052 collected from a vPLC instance 2020 may include static metadata and dynamic metadata. Examples of static metadata include resource identifiers (resource IDs), console names, IP addresses, etc. Dynamic metadata may include information that may change over time, such as performance information (e.g., CPU usage or the number of VMs running at a particular time).

[0305] In some embodiments, collection of raw metadata may be performed periodically from the vPLC instance 2020. For example, some metadata may be dynamic and change over time, and therefore, for efficiency, may be collected periodically as needed. In other embodiments, collection of raw metadata may be performed in response to a request, for example, from a reseller's customer (e.g., 2082-2086). As an example, when a metadata request is received, the metadata service (MDS) system 2030 may check the CSP-formatted metadata repository 2052. If the CSP-formatted metadata is stored in the repository 2052, the MDS system may retrieve the CSP-formatted metadata and perform customization based on the vPLC-specific customization instructions. If the CSP-formatted metadata is not present in the repository 2052, the MDS system may need to retrieve the requested metadata from the particular vPLC instance (e.g., vPLC.R1 2024 or vPLC.R2 2026) via the metadata collector 2032 and perform customization based on the vPLC-specific customization instructions.

[0306] 20 , in some embodiments, a metadata customizer 2034 may interact with a metadata collector 2032 to perform metadata customization by customizing raw metadata (i.e., CSP-formatted metadata) into vPLC-specific metadata using vPLC-specific customization instructions stored in a vPLC-specific information repository 2054. For example, the CSP-formatted metadata for a top-level domain name may be "oraclecloud.com." After customization of vPLC.R1, the vPLC-specific metadata for the top-level domain name may be "vplcR1cloud.com."

[0307] The vPLC-specific information stored in the second repository (vPLC-specific information repository) 2054 may include metadata customization instructions and response instructions. The metadata customization and response instructions 2054 may be vPLC-specific and stored in separate partitions for vPLC.R1 and vPLC.R2 (e.g., namespace 2054a associated with vPLC.R1 and namespace 2054b associated with vPLC.R2).

[0308] The metadata customizer 2034 may use the vPLC-specific customization instructions to customize the CSP format metadata 2052 (or raw metadata) into vPLC-specific metadata 2056 (described below). The metadata responder 2036 may also use the vPLC-specific metadata response instructions to determine how to respond to metadata requests. In some embodiments, the customization and response instructions may be provided by the individual reseller associated with each vPLC and may be different for different customers of the individual reseller. For example, metadata customization for the reseller's customer C1 may display one subset of customized vPLC-specific metadata at the reseller level, while metadata customization for the reseller's customer C2 may display a different subset of customized vPLC-specific metadata at the reseller level.

[0309] The vPLC-specific customization instructions may provide instructions (e.g., settings and rules) for how to convert CSP format metadata into vPLC-specific metadata and how to display the vPLC-specific metadata on a vPLC-specific console (e.g., console 2002 for vPLC.R1 or console 2004 for vPLC.R2). For example, in some embodiments, the vPLC-specific metadata customization instructions provided by a reseller associated with a vPLC may include, but are not limited to, the type of metadata to be customized, the format to be customized for display, the metadata to be displayed to customers of the reseller associated with the particular vPLC, and how the customized metadata is displayed (i.e., aspects of the user interface (UI)).

[0310] For example, with regard to metadata types, region or availability domain metadata may not be customized and may still appear in CSP format, such as "us-phoenix-1" for the first AD in the US West region (Phoenix). However, a reseller may prefer to customize top-level domain name metadata to indicate that the reseller's customer is accessing reseller-supplied services. As an example, the CSP format top-level domain "oraclecloud.com" may be customized to appear as "vplcR1cloud.com."

[0311] Regarding the format of customized metadata, in some embodiments, the customization instructions may indicate the type of criteria to be used for a particular metadata. For example, virtual machine (VM) load (i.e., resources in use) may be expressed in many different ways, such as CPU usage, total memory consumption (in gigabytes), or storage capacity in use (in terabytes). As another example, network performance may be expressed in latency (in ns) or throughput (in GB / sec). As yet another example, time criteria for initiating live migration or terminating cloud resources (e.g., VMs) may be expressed in different formats, such as a UTC timestamp (e.g., 2023-09-16T15:00:00Z) or a day-of-week and time (e.g., Saturday at 01:00 AM).

[0312] With respect to the metadata to display, these instructions may wish to display static metadata in its entirety (such as the operating system (OS) on which the VM is running) and only some, but not all, dynamic metadata (performance information for a particular VM (e.g., VM load) rather than the total number of VMs available for use at any given time).

[0313] In some embodiments, a reseller may aggregate and modify raw metadata in a CSP format. For example, if the raw metadata includes CPU cycle information for a hardware resource, the reseller may customize the raw metadata to generate an average CPU cycle usage over a range. As another example, a reseller may add cost savings information to metadata provided by a CSP.

[0314] Regarding how customized metadata is displayed, the customization instructions may want to provide a graph of VM load over time, as well as a pull-down menu and a command line for metadata requests. Resellers may want to display their logo if the reseller is an organization.

[0315] vPLC-specific metadata (i.e., metadata customized from CSP-formatted metadata using customization instructions) may be stored in a third repository (vPLC-specific metadata repository) 2056 that includes separate partitions for different vPLCs. Each vPLC has its own customized metadata. For example, vPLC-specific metadata for vPLC.R1 may be customized metadata based on the CSP-formatted metadata collected from vPLC.R1 2024 and stored in namespace 2056a associated with vPLC.R1. vPLC-specific metadata for vPLC.R2 may be customized metadata based on the CSP-formatted metadata collected from vPLC.R2 2026 and stored in namespace 2056b associated with vPLC.R2.

[0316] In some embodiments, metadata may be pre-generated in two forms: (1) pre-generated CSP-formatted metadata and (2) pre-generated customized vPLC-specific metadata. Pre-generated CSP-formatted metadata may represent raw metadata collected by metadata collector 2032 from vPLC instances 2020 and stored in CSP-formatted metadata repository 2052 before the MDS system receives any requests from users associated with the reseller's customers. Pre-generated customized vPLC-specific metadata may represent metadata customized based on raw metadata (i.e., CSP-formatted metadata) obtained directly from vPLC instances 2020 or from CSP-formatted metadata repository 2052 and stored in vPLC-specific metadata repository 2056 before the MDS system receives any requests from users associated with the reseller's customers.

[0317] In other embodiments, vPLC-specific metadata may be customized on the fly without accessing either the CSP-formatted metadata repository 2052 or the vPLC-specific metadata repository 2056. For example, when the MDS system 2030 receives a request from a user associated with a reseller's customer, it may collect raw metadata from the vPLC instance 2020, customize the raw metadata using vPLC-specific customization instructions 2054, and send the vPLC-specific metadata directly to the requesting user. Metadata generated by such an on-the-fly customization process may be referred to as on-the-fly customized metadata. In some embodiments, the vPLC-specific metadata generated during the on-the-fly customization process may be cached in the repository 2056 for future use.

[0318] The metadata responder 2036 may be responsible for obtaining metadata requests from the reseller's customers and responding to the requesting customer. For example, when the MDS system 2030 receives a metadata request, the MDS system's metadata responder 2036 may check the vPLC-specific response instructions 2054 to determine whether (1) pre-generated customized vPLC-specific metadata stored in 2056 can satisfy the request, (2) the CSP-formatted metadata stored in 2052 should be used to perform the customization and generate a response, or (3) on-the-fly customized metadata should be generated. In some embodiments, a combination of these three options may be used based on the request.

[0319] As one example, some raw metadata (e.g., regional metadata as described above) does not need to be customized according to vPLC-specific customization instructions and can simply be collected from the vPLC instance 2020 and stored directly in the vPLC-specific metadata repository 2056. Thus, a metadata responder may only need to access the vPLC-specific metadata repository 2056 to create a response. As another example, metadata requests (e.g., network performance) may change more frequently, possibly faster than the frequency of collection by the metadata collector 2032. Thus, on-the-fly customized metadata may be used.

[0320] As yet another example, fully qualified domain name (FQDN) metadata may change infrequently because users associated with a reseller's customers typically access a particular cloud service (e.g., compute) for a period of time before switching to another cloud service (e.g., storage). For metadata requests involving FQDN metadata, for efficiency reasons, pre-generated CSP-formatted metadata in repository 2052 (e.g., compute.oraclecloud.com for the compute service and storage.oraclecloud.com for the storage service) may be used to perform customization and generate a response. Accessing pre-generated CSP-formatted metadata (i.e., simply accessing storage) may be more efficient than collecting raw metadata from vPLC instance 2020, as collecting raw metadata from vPLC instance 2020 may involve multiple processing steps, such as identifying a particular vPLC instance and a particular resource within that vPLC, executing a metadata collection function, and reading the raw metadata into metadata collector 2032.

[0321] Finally, for requests involving multiple metadata, a combination of the above three options may be used, for example, some metadata may require on-the-fly customized metadata and other metadata may require pre-generated customized vPLC-specific metadata.

[0322] As shown in FIG. 20 , interface subsystem 2038 may be communicatively coupled to a set of vPLC-specific endpoints and consoles (for convenience, referred to only as endpoints) vPLC.R1 2002, vPLC.R1 2004, and vPLC.R1 2006, and may interface with customers of resellers. For example, in FIG. 20 , customer 2082 of reseller R1 can request and receive vPLC-specific metadata for vPLC.R1 2024 through endpoint 2002 and interface subsystem 2038 associated with vPLC.R1. Similarly, customer 2084 of reseller R2 can request and receive vPLC-specific metadata for vPLC.R2 2026 through endpoint 2004 and interface subsystem 2038 associated with vPLC.R2. In some embodiments, metadata requests may be internally originated. For example, the metadata service may generate pre-generated customized vPLC-specific metadata rather than waiting for a customer to request it. In such a case, raw metadata may be collected from a vPLC instance (e.g., vPLC.R1 2024) by the metadata collector 2032, customized by the metadata customizer 2034 using vPLC-specific customization instructions 2054a of vPLC.R1, and then stored in vPLC.R1 2056a of the vPLC-specific metadata repository 2056.

[0323] As shown in FIG. 20 , the metadata service executed by the distributed environment 2000 can support multiple vPLCs simultaneously. In some embodiments, the metadata service for each vPLC may be executed independently. For example, a metadata request from a user associated with a customer of reseller R1 may be received by one of the set of endpoints 2002 for vPLC.R1 and processed by the MDS system 2030 by collecting raw metadata from the vPLC.R1 instance 2024, customizing the collected raw data using vPLC.R1-specific customization instructions 2054a to generate customized vPLC.R1-specific metadata, and preparing a response using vPLC.R1-specific response instructions 2054a. As a result, the requesting user associated with the customer of reseller R1 may receive a response including the customized vPLC.R1-specific metadata.

[0324] Similarly, a metadata request from a user associated with a customer of reseller R2 may be received by one of the set of endpoints 2004 of vPLC.R2 and processed by MDS system 2030 by collecting raw metadata from vPLC.R2 instance 2026, customizing the collected raw data using vPLC.R2-specific customization instructions 2054b to generate customized vPLC.R2-specific metadata, and preparing a response using vPLC.R2-specific response instructions 2054b. As a result, the requesting user associated with the customer of reseller R2 can receive a response that includes the customized vPLC.R2-specific metadata. The processing of metadata requests from users associated with customers of reseller R1 may be performed independently of processing of metadata requests from users associated with customers of reseller R2.

[0325] In some embodiments, the metadata service may enable a reseller to manage (e.g., add, modify, or remove) metadata and control the use of certain resources. For example, a user associated with a reseller's customer (e.g., 2082) may create a spot (or lead) instance running on vPLC.R1 2024 through a console, API, or other means. A spot (or lead) instance is a compute instance that utilizes spare capacity at a significant discount, but is reusable after short notice by the service provider (e.g., CSP or reseller) if capacity is needed elsewhere. Through the metadata service, the instance can automatically obtain metadata about the impending re-use of the spot (or lead) instance. For example, a process on a running instance can periodically or event-drivenly query metadata related to environmental information, such as static metadata (e.g., domain name) or dynamic metadata (e.g., upcoming live migration, cost changes). The instance can then be gracefully shut down based on the obtained information (e.g., environmental changes). This metadata service may provide customized vPLC-specific metadata (pre-generated or on-the-fly) on a vPLC-specific console (e.g., 2002) that allows a user to interact with the metadata service (e.g., initiate a smooth shutdown of a specific instance).

[0326] FIG. 21 is an exemplary flowchart illustrating a metadata service configuration process, according to one embodiment. The process illustrated in FIG. 21 may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective systems, using hardware, or a combination thereof. The software may be stored in a non-transitory storage medium (e.g., a memory device). The method presented in FIG. 21 and described below is exemplary and not intended to be limiting. While FIG. 21 depicts various process steps occurring in a particular order or sequence, this is not intended to be limiting. In an alternative embodiment, the process may be performed in some different order, or some steps may be performed in parallel. Of course, in alternative embodiments, the process illustrated in FIG. 21 may include more or fewer steps than those depicted in FIG. 21.

[0327] 21 may be performed by one or more components responsible for configuring the metadata service. For example, obtaining the metadata customization instructions may be performed by the CSP control plane (CP) and one or more subsystems of the metadata service system 2030.

[0328] In step 2110, a request to configure a metadata service (MDS) for the vPLC may be received. For example, reseller R1 may register for a vPLC service provided by a CSP. As a result, a vPLC is created for reseller R1 that provides reseller-supplied and reseller-branded cloud services, including the metadata service, to reseller R1's customers. In some embodiments, this request may originate from a component of the CSP-provided regional infrastructure upon creation of the vPLC for reseller R1.

[0329] Metadata customization instructions for vPLC-specific metadata customization and metadata response instructions for vPLC-specific metadata responses may be obtained in step 2112. For example, in Figure 20, vPLC-specific metadata customization instructions and response instructions may be obtained and stored separately for each vPLC in vPLC-specific information repository 2054 (vPLC.R1 2054 for vPLC.R1 specific instructions and vPLC.R2 2054 for vPLC.R2 specific instructions).

[0330] In step 2114, the metadata service (MDS) system 2030 may initiate metadata service-related functions for the vPLC. For example, in FIG. 20, the MDS system 2030 may perform metadata services for vPLC.R1 by obtaining a metadata request from a user associated with customer 2082 of reseller R1, collecting raw metadata from vPLC.R1 2024, and generating vPLC-specific metadata to respond to the request by customizing the raw metadata into a CSP format using vPLC-specific customization instructions obtained from vPLC.R1 2054a.

[0331] FIG. 22 is an exemplary flowchart illustrating a metadata collection process for a metadata service, according to one embodiment. The process illustrated in FIG. 22 may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of a respective system, using hardware, or a combination thereof. The software may be stored in a non-transitory storage medium (e.g., a memory device). The method presented in FIG. 22 and described below is exemplary and not intended to be limiting. While FIG. 22 depicts various process steps occurring in a particular order or sequence, this is not intended to be limiting. In an alternative embodiment, the process may be performed in some different order, or some steps may be performed in parallel. Of course, in alternative embodiments, the process illustrated in FIG. 22 may include more or fewer steps than those depicted in FIG. 22.

[0332] 22 may be performed by one or more components responsible for performing the metadata collection process of a metadata service. For example, step 2210 may be performed by the metadata collector 2032 to collect raw metadata, while step 2220 may be performed by the metadata customizer 2034 to access customization instructions.

[0333] Metadata associated with the vPLC instance may be collected in step 2210. For example, in FIG. 20 , raw metadata for vPLC R1 may be collected by metadata collector 2032 upon request by another component in the distributed environment based on an internal request, such as a metadata request from a customer 2082 of reseller R1 or a periodic metadata collection process for efficiency (e.g., pre-generated CSP format metadata or pre-generated customized vPLC-specific metadata).

[0334] In step 2212, the metadata information (CSP format) collected in 2212 may be stored in the CSP format metadata repository 2052. As described above, the raw metadata collected from the vPLC instance 2020 and stored in the CSP format metadata repository 2052 may occur prior to any metadata requests from users associated with the reseller's customers. This early collected raw metadata is referred to as pre-generated CSP format metadata. Because the raw metadata is in CSP format and common to all vPLC instances, it is stored in the repository 2052 used by all vPLCs.

[0335] In step 2220, vPLC-specific customization instructions may be accessed to determine how to customize (or convert) the collected raw metadata (i.e., CSP-formatted metadata) into vPLC-specific metadata. For example, in FIG. 20, the customization instructions may indicate whether to customize the metadata, the format or criteria to use for the customization, the metadata to display to the requesting customer, and how to display the customized metadata. For example, the customization instructions may indicate that metadata for a geographic region is not customized and can be collected from the vPLC instance 2020 and stored directly in the vPLC-specific metadata repository 2056. VM load may be expressed as the number of CPUs used for one customer of the reseller and the total memory consumption (in gigabytes) for another customer of the reseller.

[0336] In step 2222, customized vPLC-specific metadata is generated from the CSP-formatted metadata based on the vPLC-specific customization instructions of 2220. For example, in FIG. 20 , the CSP-formatted top-level domain metadata “oraclecloud.com” may be customized as “vplcR1cloud.com” based on the vPLC-specific customization instructions stored in vPLC.R1 2054a of vPLC-specific information repository 2054. In some embodiments, two or more customized vPLC-specific metadata may be generated from the CSP-formatted metadata at one time rather than sequentially.

[0337] The customized vPLC-specific metadata may be stored in step 2224. For example, in FIG. 20 , the customized vPLC-specific metadata may be generated and stored in vPLC-specific metadata repository 2056 before MDS system 2030 receives any metadata requests from users associated with the reseller's customers. In some embodiments, the customized vPLC-specific metadata generated during the on-the-fly customization process may be cached in vPLC-specific metadata repository 2056 for use if another user associated with the same customer requests the same vPLC-specific metadata.

[0338] FIG. 23 is an exemplary flowchart illustrating a vPLC-specific metadata response process of a metadata service, according to one embodiment. The process illustrated in FIG. 23 may be implemented in software (e.g., code, instructions, programs) executed by one or more processing units (e.g., processors, cores) of the respective systems, using hardware, or a combination thereof. The software may be stored in a non-transitory storage medium (e.g., a memory device). The method presented in FIG. 23 and described below is exemplary and not intended to be limiting. While FIG. 23 depicts various process steps occurring in a particular order or sequence, this is not intended to be limiting. In an alternative embodiment, the process may be performed in some different order, or some steps may be performed in parallel. Of course, in alternative embodiments, the process illustrated in FIG. 23 may include more or fewer steps than those depicted in FIG. 23.

[0339] 23 may be performed by one or more components responsible for performing the vPLC-specific metadata response process of the metadata service. For example, step 2310 may be performed by the vPLC-specific endpoints (2002-2006) and interface subsystem 2038 to receive and communicate the request to the MDS system 2030. As another example, step 2350 may be performed by the metadata responder 2036 to prepare the response.

[0340] In step 2310, a request for metadata for a vPLC may be received. For example, in FIG. 20 , a user associated with customer C1 of reseller R1 may request metadata associated with a VM instance through a compute service endpoint in set of endpoints vPLC.R1 2002. The request may be received by interface subsystem 2038 of MDS system 2030. In some embodiments, the request may originate internally to MDS system 2030.

[0341] In step 2312, one or more processes / flows (2320, 2322, and 2324) for generating a response to the request may be used depending on a determination based on the request and vPLC-specific response instructions of 2310. In some embodiments, the MDS system may determine that only one of the following flows needs to be used: pre-generated customized vPLC-specific metadata flow 2320, pre-generated CSP-formatted metadata flow 2322, and on-the-fly customized metadata flow 2324. For example, the request indicates that a requesting user associated with customer C1 of reseller R1 wants only the latest performance information about the VM. Therefore, based on the request and response instructions, only on-the-fly customized metadata flow 2324 may be used.

[0342] In other embodiments, the MDS system may determine that two processes (or flows) should be used to fulfill a request. For example, a metadata request for both the FQDN of a compute service and the region in which the compute service runs may use both pre-generated customized vPLC-specific metadata flow 2320 and pre-generated CSP-formatted metadata flow 2322. The pre-generated customized metadata flow may access vPLC-specific metadata repository 2056 to obtain region metadata (e.g., us-phoenix-1), which does not require customization via the customization instructions described above. The pre-generated CSP-formatted metadata flow may access CSP-formatted metadata repository 2052 to obtain an FQDN that changes infrequently (e.g., compute.oraclecloud.com) to generate vPLC-specific metadata (e.g., compute.vplcR1cloud.com).

[0343] In FIG. 23 , pre-generated customized metadata flow 2320 includes steps 2330, 2332, 2350, and 2352. In step 2330, pre-generated customized vPLC-specific metadata stored in vPLC-specific metadata repository 2056 may be accessed. For example, in FIG. 20 , if the metadata request is for vPLC.R1, vPLC.R1 2054a may be accessed. If the metadata request is for vPLC.R2, vPLC.R2 2054a may be accessed. In step 2332, customized vPLC-specific metadata to be included in the response may be identified. As shown in the example above, region metadata (e.g., us-phoenix-1) to be included in the response may be identified. In step 2350, a response including the pre-generated customized vPLC-specific metadata may be prepared and generated. In step 2352, the generated response may be sent to the requestor (a user associated with the reseller's customer or another component of the MDS system 2030).

[0344] Referring again to FIG. 23 , pre-generated CSP format metadata flow 2322 may include steps 2340, 2342, 2344, 2350, and 2352. In step 2340, pre-generated CSP format metadata may be accessed. The pre-generated CSP format metadata may be stored in CSP format metadata repository 2052 and may be common to all vPLCs. In step 2342, vPLC-specific customization instructions may be obtained. For example, in FIG. 20 , vPLC-specific customization instructions for vPLC.R1 may be obtained from namespace 2054a associated with vPLC.R1 in vPLC-specific information repository 2054 to determine how to customize the pre-generated CSP format metadata obtained in 2340. In addition, vPLC-specific customization instructions for vPLC.R2 may be obtained from the namespace 2054b associated with vPLC.R2 in the vPLC-specific information repository 2054, and a method for customizing the pre-generated CSP format metadata obtained in 2340 may be determined.

[0345] In step 2344, customized vPLC-specific metadata may be generated based on the vPLC-specific customization instructions. For example, in Figure 20, customized vPLC-specific metadata may be generated for requests from users associated with customers of reseller R1 based on customization instructions obtained from vPLC.R1 2054a. Customized vPLC-specific metadata may be generated for requests from users associated with customers of reseller R2 based on customization instructions obtained from vPLC.R2 2054b.

[0346] A response including the customized vPLC-specific metadata may be prepared and generated in step 2350. In step 2352, the generated response may be sent to the requestor (a user associated with the reseller's customer or another component of the MDS system 2030).

[0347] 23 , on-the-fly customized metadata flow 2324 may include steps 2341, 2342, 2344, 2350, and 2352. In step 2341, raw metadata information associated with a vPLC instance may be collected. For example, in FIG. 20 , raw metadata in CSP format associated with vPLC.R1 may be collected from vPLC.R1 instance 2024, and raw metadata in CSP format associated with vPLC.R2 may be collected from vPLC.R2 instance 2026. In step 2342, vPLC-specific customization instructions may be obtained. For example, in FIG. 20 , vPLC-specific customization instructions for vPLC.R1 may be obtained from namespace 2054 a associated with vPLC.R1 in vPLC-specific information repository 2054 to determine how to customize the raw metadata collected from vPLC.R1 instance 2024. Additionally, vPLC-specific customization instructions for vPLC.R2 may be obtained from namespace 2054b associated with vPLC.R2 in vPLC-specific information repository 2054 to determine how to customize the raw metadata collected from vPLC.R2 instance 2026. Steps 2344, 2350, and 2352 are the same for pre-generated CSP format metadata flow 2322 and on-the-fly customized metadata flow 2324 and may therefore be performed as discussed with respect to on-the-fly customized metadata flow 2324.

[0348] Exemplary CSPI Architecture As mentioned above, infrastructure as a service (IaaS) is a specific type of cloud computing. IaaS can be configured to provide virtualized computing resources over a public network (e.g., the Internet). In the IaaS model, a cloud computing provider can host infrastructure components (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer), etc.). In some cases, the IaaS provider may also supply various services associated with these infrastructure components (example services include billing software, monitoring software, logging software, load balancing software, clustering software, etc.). Accordingly, because these services can be policy-driven, IaaS users can maintain application availability and performance by implementing policies that drive load balancing.

[0349] In some cases, IaaS customers can access resources and services over a wide area network (WAN) such as the Internet and use the cloud provider's services to install other elements of their application stack. For example, a user can log in to an IaaS platform to create virtual machines (VMs), install operating systems (OSs) on each VM, deploy middleware such as databases, create storage buckets for workloads and backups, and even install enterprise software on those VMs. The customer can then use the provider's services to perform a variety of functions, such as distributing network traffic, troubleshooting application issues, monitoring performance, and managing disaster recovery.

[0350] In most cases, the cloud computing model requires the participation of a cloud provider, which may be, but need not be, a third-party service that specializes in providing IaaS (e.g., providing, renting, selling). An entity may also deploy a private cloud and become its own infrastructure service provider.

[0351] In some examples, IaaS deployment is the process of putting a new application or a new version of an application onto a provisioned application server, etc. It may also include the process of provisioning the server (e.g., installing libraries, daemons, etc.), which is often managed below the hypervisor layer (e.g., server, storage, network hardware, and virtualization) by the cloud provider. As such, the customer may be responsible for handling (e.g., on self-service virtual machines (which can be spun up on demand)), middleware, and / or application deployment, etc.

[0352] In some instances, IaaS provisioning may refer to obtaining computers or virtual hosts for use and even installing the necessary libraries or services on them. In most cases, deployment does not include provisioning, which may need to be performed first.

[0353] IaaS provisioning sometimes presents two distinct challenges. First, there is the initial challenge of provisioning an initial set of infrastructure before anything is operational. Second, there is the challenge of evolving the existing infrastructure after all the provisioning has occurred (e.g., adding new services, modifying services, removing services, etc.). Sometimes these two challenges can be addressed by allowing the declarative definition of the infrastructure configuration. In other words, the infrastructure (e.g., the required components and how they interact) can be specified by one or more configuration files. Thus, the overall topology of the infrastructure (e.g., resource dependencies and how they work together) can be declaratively described. Sometimes, once the topology is specified, workflows can be generated to configure and / or manage the various components described in the configuration files.

[0354] In some examples, the infrastructure may have many interconnected elements. For example, there may be one or more virtual private clouds (VPCs) (e.g., potential on-demand pools of configurable and / or shared computing resources), also known as a core network. Also, in some examples, there may be one or more inbound / outbound traffic group rules provisioned to define how inbound and / or outbound traffic of the network is configured and one or more virtual machines (VMs). Other infrastructure elements, such as load balancers, databases, etc., may also be provisioned. The infrastructure may evolve incrementally as there is a demand and / or addition of more infrastructure elements.

[0355] In some cases, employing continuous deployment techniques may enable deployment of infrastructure code across various virtual computing environments. The described techniques may also enable infrastructure management within these environments. In some examples, a service team may write code that is desired to be deployed to one or more (but often many) different production environments (e.g., across various geographic locations, possibly across the globe). However, in some examples, the infrastructure into which the code will be deployed is configured first. In some cases, provisioning can be performed manually, a provisioning tool can be used to provision resources, and / or a deployment tool can be used to deploy the code after the infrastructure has been provisioned.

[0356] 24 is a block diagram 2400 illustrating an example pattern of an IaaS architecture according to at least one embodiment. A service operator 2402 may be communicatively coupled to a secure host tenancy 2404, which may include a virtual cloud network (VCN) 2406 and a secure host subnet 2408. In some examples, the service operator 2402 may employ one or more client computing devices, which may be portable handheld devices (e.g., iPhones, mobile phones, iPads, computing tablets, personal digital assistants (PDAs)) or wearable devices (e.g., Google Glass head-mounted displays, etc.) running software such as Microsoft Windows Mobile and / or various mobile operating systems such as iOS, Windows Phone, Android, BlackBerry 8, Palm OS, etc., and capable of using the Internet, email, short message service (SMS), BlackBerry, or other communication protocols. Alternatively, the client computing devices may be general-purpose personal computers, examples of which include personal computers and / or laptop computers running various versions of the Microsoft Windows, Apple Macintosh, and / or Linux operating systems. The client computing devices may also be workstation computers running any of a variety of commercially available UNIX or UNIX-like operating systems, including, but not limited to, various GNU / Linux operating systems such as Google Chrome OS.Alternatively or additionally, the client computing device may be any other electronic device, such as a thin client computer, an Internet-enabled gaming system (e.g., a Microsoft Xbox game console with or without a Kinect® gesture input device), and / or a personal messaging device, capable of communicating over a network that can access VCN 2406 and / or the Internet.

[0357] VCN 2406 may include a local peering gateway (LPG) 2410 that may be communicatively coupled to a secure shell (SSH) VCN 2412 via LPG 2410 included in SSH VCN 2412. SSH VCN 2412 may include an SSH subnet 2414, and SSH VCN 2412 may be communicatively coupled to a control plane VCN 2416 via LPG 2410 included in control plane VCN 2416. SSH VCN 2412 may also be communicatively coupled to a data plane VCN 2418 via LPG 2410. Control plane VCN 2416 and data plane VCN 2418 may be included in service tenancy 2419, which may be owned and / or operated by the IaaS provider.

[0358] The control plane VCN 2416 may include a control plane demilitarized zone (DMZ) tier 2420 that serves as a perimeter network (e.g., a portion of an enterprise network between the enterprise intranet and an external network). DMZ-based servers may limit liability and help mitigate breaches. DMZ tier 2420 may also include one or more load balancer (LB) subnets 2422, a control plane app tier 2424 that may include app subnets 2426, and a control plane data tier 2428 that may include database (DB) subnets 2430 (e.g., a front-end DB subnet and / or a back-end DB subnet). LB subnet 2422 included in control plane DMZ layer 2420 is communicatively coupled to app subnet 2426 included in control plane app layer 2424 and to Internet gateway 2434, which may be included in control plane VCN 2416, and app subnet 2426 may be communicatively coupled to DB subnet 2430, service gateway 2436, and network address translation (NAT) gateway 2438, which are included in control plane data layer 2428. Control plane VCN 2416 may include service gateway 2436 and NAT gateway 2438.

[0359] Control plane VCN 2416 can include data plane mirror app layer 2440, which can include app subnet 2426. App subnet 2426 included in data plane mirror app layer 2440 can include virtual network interface controller (VNIC) 2442, which can run compute instance 2444. Compute instance 2444 can communicatively couple app subnet 2426 of data plane mirror app layer 2440 to app subnet 2426, which can be included in data plane app layer 2446.

[0360] Data plane VCN 2418 can include data plane app layer 2446, data plane DMZ layer 2448, and data plane data layer 2450. Data plane DMZ layer 2448 can include LB subnet 2422, which can be communicatively coupled to app subnet 2426 of data plane app layer 2446 and internet gateway 2434 of data plane VCN 2418. App subnet 2426 can be communicatively coupled to service gateway 2436 of data plane VCN 2418 and NAT gateway 2438 of data plane VCN 2418. Data plane data layer 2450 can also include DB subnet 2430, which can be communicatively coupled to app subnet 2426 of data plane app layer 2446.

[0361] The internet gateways 2434 of the control plane VCNs 2416 and data plane VCNs 2418 may be communicatively coupled to metadata management services 2452, which may be communicatively coupled to the public internet 2454. The public internet 2454 may be communicatively coupled to NAT gateways 2438 of the control plane VCNs 2416 and data plane VCNs 2418. The service gateways 2436 of the control plane VCNs 2416 and data plane VCNs 2418 may be communicatively coupled to cloud services 2456.

[0362] In some examples, the service gateways 2436 of the control plane VCN 2416 and the data plane VCN 2418 can make application programming interface (API) calls to the cloud services 2456 without going over the public internet 2454. API calls from the service gateway 2436 to the cloud services 2456 can be unidirectional. The service gateway 2436 can make API calls to the cloud services 2456, and the cloud services 2456 can send the requested data to the service gateway 2436. However, the cloud services 2456 do not have to initiate the API calls to the service gateway 2436.

[0363] In some examples, secure host tenancy 2404 may be directly connected to an otherwise isolated service tenancy 2419. Secure host subnet 2408 may communicate with SSH subnet 2414 through LPG 2410, which may allow bidirectional communication through an otherwise isolated system. Connecting secure host subnet 2408 to SSH subnet 2414 allows secure host subnet 2408 to be accessible to other entities within service tenancy 2419.

[0364] The control plane VCN 2416 may enable users of the service tenancy 2419 to configure or provision desired resources. The desired resources provisioned in the control plane VCN 2416 may be deployed or used in the data plane VCN 2418. In some examples, the control plane VCN 2416 may be isolated from the data plane VCN 2418, and the data plane mirror app layer 2440 of the control plane VCN 2416 may communicate with the data plane app layer 2446 of the data plane VCN 2418 via a VNIC 2442 that may be included in the data plane mirror app layer 2440 and the data plane app layer 2446.

[0365] In some examples, a user or customer of the system may make a request (e.g., perform a create, read, update, or delete (CRUD) operation) over the public internet 2454, which may route the request to the metadata management service 2452. The metadata management service 2452 may route the request to the control plane VCN 2416 through the internet gateway 2434. The request may be received by the LB subnet 2422 included in the control plane DMZ tier 2420. The LB subnet 2422 may determine that the request is valid, and in response to this determination, the LB subnet 2422 may send the request to the app subnet 2426 included in the control plane app tier 2424. If the request is validated and a call to the public internet 2454 is required, the call to the public internet 2454 may be sent to the NAT gateway 2438, which may make the call to the public internet 2454. Metadata that may be desirable to store with the request may be stored in the DB subnet 2430.

[0366] In some examples, the data plane mirror app layer 2440 may facilitate direct communication between the control plane VCN 2416 and the data plane VCN 2418. For example, it may be desirable to apply configuration changes, updates, or other suitable modifications to resources included in the data plane VCN 2418. The control plane VCN 2416 may perform configuration changes, updates, or other suitable modifications to resources by communicating directly with the resources included in the data plane VCN 2418 via the VNIC 2442.

[0367] In some embodiments, the control plane VCN 2416 and the data plane VCN 2418 may be included in the service tenancy 2419. In this case, a user or customer of the system may not own or operate either the control plane VCN 2416 or the data plane VCN 2418. Alternatively, an IaaS provider may own or operate both the control plane VCN 2416 and the data plane VCN 2418, and both may be included in the service tenancy 2419. This embodiment may enable network isolation that may prevent users or customers from interacting with the resources of other users or customers. This embodiment may also enable private storage of databases by users or customers of the system without having to rely on the public internet 2454, which may not have the desired level of threat protection for storage.

[0368] In another embodiment, LB subnet 2422 included in control plane VCN 2416 can be configured to receive signals from service gateway 2436. In this embodiment, control plane VCN 2416 and data plane VCN 2418 can be configured to be called by customers of the IaaS provider without calling the public internet 2454. Customers of the IaaS provider may desire this embodiment because the databases they use can be stored in service tenancy 2419, which is controlled by the IaaS provider and can be isolated from the public internet 2454.

[0369] 25 is a block diagram 2500 illustrating another example pattern of an IaaS architecture, according to at least one embodiment. A service operator 2502 (e.g., service operator 2402 of FIG. 24) may be communicatively coupled to a virtual cloud network (VCN) 2506 (e.g., VCN 2406 of FIG. 24) and a secure host tenancy 2504 (e.g., secure host tenancy 2404 of FIG. 24), which may include a virtual cloud network (VCN) 2506 (e.g., VCN 2406 of FIG. 24) and a secure host subnet 2508 (e.g., secure host subnet 2408 of FIG. 24). VCN 2506 may include a local peering gateway (LPG) 2510 (e.g., LPG 2410 of FIG. 24), which may be communicatively coupled to a secure shell (SSH) VCN 2512 via an LPG 2410 included in the SSH VCN 2512 (e.g., SSH VCN 2412 of FIG. 24). SSH VCN 2512 can include SSH subnet 2514 (e.g., SSH subnet 2414 in FIG. 24 ), and SSH VCN 2512 can be communicatively coupled to control plane VCN 2516 via LPG 2510, which is included in control plane VCN 2516 (e.g., control plane VCN 2416 in FIG. 24 ). Control plane VCN 2516 can be included in service tenancy 2519 (e.g., service tenancy 2419 in FIG. 24 ), and data plane VCN 2518 (e.g., data plane VCN 2418 in FIG. 24 ) can be included in customer tenancy 2521, which can be owned or operated by a user or customer of the system.

[0370] The control plane VCN 2516 may include a control plane DMZ layer 2520 (e.g., control plane DMZ layer 2420 of FIG. 24 ) that may include a LB subnet 2522 (e.g., LB subnet 2422 of FIG. 24 ), a control plane app layer 2524 (e.g., control plane app layer 2424 of FIG. 24 ) that may include an app subnet 2526 (e.g., app subnet 2426 of FIG. 24 ), and a control plane data layer 2528 (e.g., control plane data layer 2428 of FIG. 24 ) that may include a database (DB) subnet 2530 (e.g., similar to database subnet 2430 of FIG. 24 ). LB subnet 2522 included in control plane DMZ layer 2520 is communicatively coupled to app subnet 2526 included in control plane app layer 2524 and to Internet gateway 2534 (e.g., Internet gateway 2434 in FIG. 24 ), which may be included in control plane VCN 2516, and app subnet 2526 may be communicatively coupled to DB subnet 2530, service gateway 2536 (e.g., service gateway 2436 in FIG. 24 ), and network address translation (NAT) gateway 2538 (e.g., NAT gateway 2438 in FIG. 24 ), which are included in control plane data layer 2528. Control plane VCN 2516 may comprise service gateway 2536 and NAT gateway 2538.

[0371] Control plane VCN 2516 may include data plane mirror app layer 2540 (e.g., data plane mirror app layer 2440 of FIG. 24 ), which may include app subnet 2526. App subnet 2526 included in data plane mirror app layer 2540 may include virtual network interface controller (VNIC) 2542 (e.g., VNIC 2442), which may run compute instance 2544 (e.g., similar to compute instance 2444 of FIG. 24 ). Compute instance 2544 may facilitate communication between app subnet 2526 of data plane mirror app layer 2540 and app subnet 2526, which may be included in data plane app layer 2546, via VNIC 2542 included in data plane mirror app layer 2540 and VNIC 2542 included in data plane app layer 2546 (e.g., data plane app layer 2446 of FIG. 24 ).

[0372] An internet gateway 2534 included in the control plane VCN 2516 may be communicatively coupled to a metadata management service 2552 (e.g., metadata management service 2452 of FIG. 24), which may be communicatively coupled to the public internet 2554 (e.g., public internet 2454 of FIG. 24). The public internet 2554 may be communicatively coupled to a NAT gateway 2538 included in the control plane VCN 2516. A service gateway 2536 included in the control plane VCN 2516 may be communicatively coupled to cloud services 2556 (e.g., cloud services 2456 of FIG. 24).

[0373] In some examples, the data plane VCN 2518 may be included in the customer tenancy 2521. In this case, the IaaS provider may provide a control plane VCN 2516 for each customer, and the IaaS provider may configure a unique compute instance 2544 for each customer, which may be included in the service tenancy 2519. Each compute instance 2544 may enable communication between the control plane VCN 2516 in the service tenancy 2519 and the data plane VCN 2518 in the customer tenancy 2521. The compute instance 2544 may enable deployment or use of resources provisioned in the control plane VCN 2516 in the service tenancy 2519 in the data plane VCN 2518 in the customer tenancy 2521.

[0374] In another example, a customer of the IaaS provider may have a database that resides in customer tenancy 2521. In this example, control plane VCN 2516 may include data plane mirror app tier 2540, which may include app subnet 2526. Data plane mirror app tier 2540 may reside in data plane VCN 2518, but may not reside in data plane VCN 2518. That is, data plane mirror app tier 2540 may be accessible to customer tenancy 2521, but may not reside in data plane VCN 2518, and may not be owned or operated by the IaaS provider's customer. Data plane mirror app tier 2540 may be configured to make calls to data plane VCN 2518, but may not be configured to make calls to any entities included in control plane VCN 2516. A customer may desire deployment or use of resources in data plane VCN 2518 that have been provisioned in control plane VCN 2516, and data plane mirror app layer 2540 may facilitate the deployment or other use of the resources desired by the customer.

[0375] In some embodiments, the IaaS provider's customer can apply filters to the data plane VCN 2518. In this embodiment, the customer can determine what the data plane VCN 2518 can access, and the customer may choose to restrict access from the data plane VCN 2518 to the public internet 2554. The IaaS provider may not be able to apply filters or control the data plane VCN 2518's access to any external networks or databases. The customer's application of filters and controls to the data plane VCN 2518 in the customer tenancy 2521 can help isolate the data plane VCN 2518 from other customers and the public internet 2554.

[0376] In some embodiments, calls made by the service gateway 2536 allow cloud services 2556 to access services that may not reside on the public internet 2554, the control plane VCN 2516, or the data plane VCN 2518. The connection between the cloud services 2556 and the control plane VCN 2516 or the data plane VCN 2518 may not be live or continuous. The cloud services 2556 may reside on different networks owned or operated by the IaaS provider. The cloud services 2556 may be configured to receive calls from the service gateway 2536 or may not be configured to receive calls from the public internet 2554. Some cloud services 2556 may be isolated from other cloud services 2556, and the control plane VCN 2516 may be isolated from cloud services 2556 that may not be in the same region as the control plane VCN 2516. For example, control plane VCN 2516 may be located in "Region 1," and cloud service "deployment 24" may be located in Region 1 and Region 2. If a call to deployment 24 is made by service gateway 2536 included in control plane VCN 2516 located in Region 1, the call may be sent to deployment 24 in Region 1. In this example, control plane VCN 2516 or deployment 24 in Region 1 may not be communicatively coupled to or in communication with deployment 24 in Region 2.

[0377] 26 is a block diagram 2600 illustrating another example pattern of an IaaS architecture, according to at least one embodiment. A service operator 2602 (e.g., service operator 2402 of FIG. 24) may be communicatively coupled to a secure host tenancy 2604 (e.g., secure host tenancy 2404 of FIG. 24), which may include a virtual cloud network (VCN) 2606 (e.g., VCN 2406 of FIG. 24) and a secure host subnet 2608 (e.g., secure host subnet 2408 of FIG. 24). VCN 2606 may comprise an LPG 2610, which may be communicatively coupled to an SSH VCN 2612 via an LPG 2610 (e.g., LPG 2410 of FIG. 24) included in SSH VCN 2612 (e.g., SSH VCN 2412 of FIG. 24). SSH VCN 2612 can include SSH subnet 2614 (e.g., SSH subnet 2414 in FIG. 24 ), and SSH VCN 2612 can be communicatively coupled to control plane VCN 2616 via LPG 2610 included in control plane VCN 2616 (e.g., control plane VCN 2416 in FIG. 24 ), and can be communicatively coupled to data plane VCN 2618 via LPG 2610 included in data plane VCN 2618 (e.g., data plane VCN 2418 in FIG. 24 ). Control plane VCN 2616 and data plane VCN 2618 can be included in service tenancy 2619 (e.g., service tenancy 2419 in FIG. 24 ).

[0378] The control plane VCN 2616 may include a control plane DMZ layer 2620 (e.g., control plane DMZ layer 2420 of FIG. 24 ) that may include a load balancer (LB) subnet 2622 (e.g., LB subnet 2422 of FIG. 24 ), a control plane app layer 2624 (e.g., control plane app layer 2424 of FIG. 24 ) that may include an app subnet 2626 (e.g., similar to app subnet 2426 of FIG. 24 ), and a control plane data layer 2628 (e.g., control plane data layer 2428 of FIG. 24 ) that may include a DB subnet 2630. LB subnet 2622 included in control plane DMZ layer 2620 is communicatively coupled to app subnet 2626 included in control plane app layer 2624 and to Internet gateway 2634 (e.g., Internet gateway 2434 in FIG. 24 ), which may be included in control plane VCN 2616, and app subnet 2626 may be communicatively coupled to DB subnet 2630, service gateway 2636 (e.g., service gateway in FIG. 24 ), and network address translation (NAT) gateway 2638 (e.g., NAT gateway 2438 in FIG. 24 ), which are included in control plane data layer 2628. Control plane VCN 2616 may comprise service gateway 2636 and NAT gateway 2638.

[0379] Data plane VCN 2618 may include a data plane app layer 2646 (e.g., data plane app layer 2446 in FIG. 24 ), a data plane DMZ layer 2648 (e.g., data plane DMZ layer 2448 in FIG. 24 ), and a data plane data layer 2650 (e.g., data plane data layer 2450 in FIG. 24 ). Data plane DMZ layer 2648 may include a LB subnet 2622 that may be communicatively coupled to trusted app subnet 2660 and untrusted app subnet 2662 of data plane app layer 2646 and to an internet gateway 2634 included in data plane VCN 2618. Trusted app subnet 2660 may be communicatively coupled to a service gateway 2636 included in data plane VCN 2618, a NAT gateway 2638 included in data plane VCN 2618, and a DB subnet 2630 included in data plane data layer 2650. Untrusted app subnet 2662 may be communicatively coupled to a service gateway 2636 included in data plane VCN 2618 and a DB subnet 2630 included in data plane data layer 2650. Data plane data layer 2650 may include DB subnet 2630, which may be communicatively coupled to a service gateway 2636 included in data plane VCN 2618.

[0380] Untrusted app subnet 2662 may include one or more primary VNICs 2664(1)-2664(N), which may be communicatively coupled to tenant virtual machines (VMs) 2666(1)-2666(N). Each tenant VM 2666(1)-2666(N) may be communicatively coupled to a respective app subnet 2667(1)-2667(N), which may be included in a respective container egress VCN 2668(1)-2668(N), which may be included in a respective customer tenancy 2670(1)-2670(N). Secondary VNICs 2672(1)-2672(N), respectively, may facilitate communication between untrusted app subnet 2662, which is included in data plane VCN 2618, and the app subnets included in the respective container egress VCNs 2668(1)-2668(N). Each container output VCN 2668(1)-2668(N) may include a NAT gateway 2638 that may be communicatively coupled to the public internet 2654 (e.g., public internet 2454 in FIG. 24).

[0381] An internet gateway 2634 included in the control plane VCN 2616 and the data plane VCN 2618 may be communicatively coupled to a metadata management service 2652 (e.g., metadata management system 2452 of FIG. 24 ), which may be communicatively coupled to the public internet 2654. The public internet 2654 may be communicatively coupled to a NAT gateway 2638 included in the control plane VCN 2616 and the data plane VCN 2618. A service gateway 2636 included in the control plane VCN 2616 and the data plane VCN 2618 may be communicatively coupled to cloud services 2656.

[0382] In some embodiments, data plane VCN 2618 can be integrated with customer tenancy 2670. This integration can be useful or desirable for customers of an IaaS provider, such as when they may want support for running code. A customer may provide code to run, which may be destructive, communicate with other customer resources, or otherwise have undesirable effects. In response, the IaaS provider may determine whether to run the code provided to it by the customer.

[0383] In some examples, a customer of an IaaS provider may grant temporary network access to the IaaS provider to request functionality provided in data plane app layer 2646. The code that performs this functionality may be configured to run in VMs 2666(1)-2666(N) and may not be configured to run elsewhere on data plane VCN 2618. Each VM 2666(1)-2666(N) may be connected to one customer tenancy 2670. Each container 2671(1)-2671(N) contained in VMs 2666(1)-2666(N) may be configured to run code. In this case, double isolation may exist (e.g., containers 2671(1)-2671(N) executing code may be contained in VMs 2666(1)-2666(N) included in at least untrusted app subnet 2662), which may help prevent erroneous or unwanted code from damaging the IaaS provider's network or a different customer's network. Containers 2671(1)-2671(N) may be communicatively coupled to customer tenancy 2670 and configured to send or receive data from customer tenancy 2670. Containers 2671(1)-2671(N) may be configured not to send or receive data from any other entity in data plane VCN 2618. Upon completion of code execution, the IaaS provider may disable or discard containers 2671(1)-2671(N).

[0384] In some embodiments, trusted app subnet 2660 may execute code that may be owned or operated by the IaaS provider. In this embodiment, trusted app subnet 2660 may be communicatively coupled to DB subnet 2630 and may be configured to perform CRUD operations on DB subnet 2630. Untrusted app subnet 2662 may be communicatively coupled to DB subnet 2630, but in this embodiment, may be configured to perform read operations on DB subnet 2630. Containers 2671(1)-2671(N) included in each customer's VM 2666(1)-2666(N) and that may execute code from the customer may not be communicatively coupled to DB subnet 2630.

[0385] In other embodiments, the control plane VCN 2616 and the data plane VCN 2618 may not be directly communicatively coupled. In this embodiment, there may not be direct communication between the control plane VCN 2616 and the data plane VCN 2618. However, communication may occur indirectly in at least one manner. An IaaS provider may establish an LPG 2610 that may facilitate communication between the control plane VCN 2616 and the data plane VCN 2618. In another example, the control plane VCN 2616 or the data plane VCN 2618 may make a call to a cloud service 2656 via a service gateway 2636. For example, a call from the control plane VCN 2616 to the cloud service 2656 may include a request for a service that may communicate with the data plane VCN 2618.

[0386] 27 is a block diagram 2700 illustrating another example pattern of an IaaS architecture, according to at least one embodiment. A service operator 2702 (e.g., service operator 2402 of FIG. 24) may be communicatively coupled to a secure host tenancy 2704 (e.g., secure host tenancy 2404 of FIG. 24), which may include a virtual cloud network (VCN) 2706 (e.g., VCN 2406 of FIG. 24) and a secure host subnet 2708 (e.g., secure host subnet 2408 of FIG. 24). VCN 2706 may comprise an LPG 2710, which may be communicatively coupled to an SSH VCN 2712 via an LPG 2710 (e.g., LPG 2410 of FIG. 24) included in SSH VCN 2712 (e.g., SSH VCN 2412 of FIG. 24). SSH VCN 2712 can include SSH subnet 2714 (e.g., SSH subnet 2414 in FIG. 24 ), and SSH VCN 2712 can be communicatively coupled to control plane VCN 2716 via LPG 2710 included in control plane VCN 2716 (e.g., control plane VCN 2416 in FIG. 24 ), and can be communicatively coupled to data plane VCN 2718 via LPG 2710 included in data plane VCN 2718 (e.g., data plane VCN 2418 in FIG. 24 ). Control plane VCN 2716 and data plane VCN 2718 can be included in service tenancy 2719 (e.g., service tenancy 2419 in FIG. 24 ).

[0387] The control plane VCN 2716 may include a control plane DMZ layer 2720 (e.g., control plane DMZ layer 2420 of FIG. 24 ) that may include a LB subnet 2722 (e.g., LB subnet 2422 of FIG. 24 ), a control plane app layer 2724 (e.g., control plane app layer 2424 of FIG. 24 ) that may include an app subnet 2726 (e.g., app subnet 2426 of FIG. 24 ), and a control plane data layer 2728 (e.g., control plane data layer 2428 of FIG. 24 ) that may include a DB subnet 2730 (e.g., DB subnet 2630 of FIG. 26 ). LB subnet 2722 included in control plane DMZ layer 2720 is communicatively coupled to app subnet 2726 included in control plane app layer 2724 and to Internet gateway 2734 (e.g., Internet gateway 2434 in FIG. 24 ), which may be included in control plane VCN 2716, and app subnet 2726 may be communicatively coupled to DB subnet 2730, service gateway 2736 (e.g., service gateway in FIG. 24 ), and network address translation (NAT) gateway 2738 (e.g., NAT gateway 2438 in FIG. 24 ), which are included in control plane data layer 2728. Control plane VCN 2716 may comprise service gateway 2736 and NAT gateway 2738.

[0388] Data plane VCN 2718 can include a data plane app layer 2746 (e.g., data plane app layer 2446 in FIG. 24 ), a data plane DMZ layer 2748 (e.g., data plane DMZ layer 2448 in FIG. 24 ), and a data plane data layer 2750 (e.g., data plane data layer 2450 in FIG. 24 ). Data plane DMZ layer 2748 can include trusted app subnet 2760 (e.g., trusted app subnet 2660 in FIG. 26 ) and non-trusted app subnet 2762 (e.g., non-trusted app subnet 2662 in FIG. 26 ) of data plane app layer 2746 and LB subnet 2722, which can be communicatively coupled to Internet gateway 2734 included in data plane VCN 2718. Trusted app subnet 2760 may be communicatively coupled to service gateway 2736 included in data plane VCN 2718, NAT gateway 2738 included in data plane VCN 2718, and DB subnet 2730 included in data plane data layer 2750. Untrusted app subnet 2762 may be communicatively coupled to service gateway 2736 included in data plane VCN 2718 and DB subnet 2730 included in data plane data layer 2750. Data plane data layer 2750 may comprise DB subnet 2730, which may be communicatively coupled to service gateway 2736 included in data plane VCN 2718.

[0389] Untrusted app subnet 2762 may include primary VNICs 2764(1)-2764(N), which may be communicatively coupled to tenant virtual machines (VMs) 2766(1)-2766(N) residing within untrusted app subnet 2762. Each tenant VM 2766(1)-2766(N) may execute code in a respective container 2767(1)-2767(N), and may be communicatively coupled to app subnet 2726, which may be included in data plane app layer 2746, which may be included in container egress VCN 2768. Secondary VNICs 2772(1)-2772(N), respectively, may facilitate communication between untrusted app subnet 2762, which may be included in data plane VCN 2718, and the app subnet, which may be included in container egress VCN 2768. The container egress VCN may include a NAT gateway 2738 that may be communicatively coupled to the public internet 2754 (e.g., public internet 2454 in FIG. 24).

[0390] An internet gateway 2734 included in the control plane VCN 2716 and the data plane VCN 2718 may be communicatively coupled to a metadata management service 2752 (e.g., metadata management system 2452 of FIG. 24 ), which may be communicatively coupled to the public internet 2754. The public internet 2754 may be communicatively coupled to a NAT gateway 2738 included in the control plane VCN 2716 and the data plane VCN 2718. A service gateway 2736 included in the control plane VCN 2716 and the data plane VCN 2718 may be communicatively coupled to cloud services 2756.

[0391] In some examples, the pattern illustrated by the architecture of block diagram 2700 in FIG. 27 may be considered an exception to the pattern illustrated by the architecture of block diagram 2600 in FIG. 26 and may be desirable for customers of an IaaS provider when the IaaS provider cannot communicate directly with the customer (e.g., in an unconnected region). Containers 2767(1)-2767(N) contained in VMs 2766(1)-2766(N) for each customer may be accessible in real time by the customer. Containers 2767(1)-2767(N) may be configured to make calls to secondary VNICs 2772(1)-2772(N), respectively, contained in app subnet 2726 of data plane app tier 2746, which may be included in container egress VCN 2768. Secondary VNICs 2772(1)-2772(N) can send the calls to NAT gateway 2738, which may send the calls to public Internet 2754. In this example, containers 2767(1)-2767(N) that are accessible in real time by customers may be isolated from control plane VCN 2716 and may be isolated from other entities included in data plane VCN 2718. Containers 2767(1)-2767(N) may also be isolated from resources of other customers.

[0392] In another example, a customer can use containers 2767(1)-2767(N) to invoke cloud service 2756. In this example, the customer can execute code in containers 2767(1)-2767(N) that requests services from cloud service 2756. Containers 2767(1)-2767(N) can send the request to secondary VNICs 2772(1)-2772(N), which can send the request to a NAT gateway, which can send the request to public internet 2754. Public internet 2754 can send the request to LB subnet 2722, which is included in control plane VCN 2716, via internet gateway 2734. In response to determining that the request is valid, the LB subnet can send the request to the app subnet 2726, which can send the request to the cloud service 2756 via the service gateway 2736.

[0393] It should be understood that the IaaS architectures 2400, 2500, 2600, and 2700 depicted in the figures may include components other than those shown. Furthermore, the illustrated embodiment is merely an example of a cloud infrastructure system that may incorporate an embodiment of the present disclosure. In other embodiments, an IaaS system may include more or fewer components than those depicted, may combine two or more components, or may have a different configuration or arrangement of components.

[0394] In some embodiments, the IaaS system described herein may include a suite of application, middleware, and database services delivered to customers in a self-service, subscription-based, elastically scalable, reliable, highly available, and secure manner. Oracle Cloud Infrastructure (OCI), offered by the present assignee, is an example of such an IaaS system.

[0395] 28 illustrates an exemplary computer system 2800 upon which various embodiments may be implemented. System 2800 may be adapted to implement any of the computer systems described above. As shown in the figure, computer system 2800 includes a processing unit 2804 that communicates with a number of peripheral subsystems via a bus subsystem 2802. The peripheral subsystems may include a processing acceleration unit 2806, an I / O subsystem 2808, a storage subsystem 2818, and a communication subsystem 2824. The storage subsystem 2818 includes a tangible computer-readable storage medium 2822 and a system memory 2810.

[0396] Bus subsystem 2802 provides a mechanism for allowing the various components and subsystems of computer system 2800 to communicate with each other as desired. While bus subsystem 2802 is shown schematically as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses. Bus subsystem 2802 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. For example, such architectures may include an Industry Standard Architecture (ISA) bus, which may be implemented as a mezzanine bus manufactured in accordance with the IEEE P1386.1 standard, a MicroChannel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0397] Processing unit 2804, which may be implemented as one or more integrated circuits (e.g., conventional microprocessors or microcontrollers), controls the operation of computer system 2800. Processing unit 2804 may include one or more processors. These processors may include single-core or multi-core processors. In some embodiments, processing unit 2804 may be implemented as one or more independent processing units 2832 and / or 2834, each including a single-core or multi-core processor. In other embodiments, processing unit 2804 may be implemented as a quad-core processing unit formed by integrating two dual-core processors onto a single chip.

[0398] In various embodiments, processing unit 2804 executes various programs in response to program code and can maintain multiple simultaneously executing programs or processes. At any given time, some or all of the program code being executed may reside in processor 2804 and / or in storage subsystem 2818. With suitable programming, processor 2804 can provide the various functions described above. Computer system 2800 may also include a processing acceleration unit 2806, which may include a digital signal processor (DSP), a special purpose processor, and / or the like.

[0399] The I / O subsystem 2808 may include user interface input devices and user interface output devices. User interface input devices may include a keyboard, a pointing device such as a mouse or trackball, a touchpad or touchscreen integrated into a display, a scroll wheel, a click wheel, a dial, buttons, switches, a keypad, an audio input device with a voice command recognition system, a microphone, and other types of input devices. User interface input devices may include a motion-sensing and / or gesture-recognition device such as a Microsoft Kinect® motion sensor that enables a user to control and interact with an input device such as a Microsoft Xbox® 360 game controller through a natural user interface using gestures and voice commands. User interface input devices may also include an eye-gesture recognition device such as a Google Glass® blink detector that detects a user's eye activity (e.g., "blinking" during photography and / or menu selection) and translates the eye gesture as input to an input device (e.g., Google Glass®). The user interface input devices may also include a voice recognition sensing device that allows a user to interact with a voice recognition system (e.g., the Siri® navigator) via voice commands.

[0400] User interface input devices may also include, but are not limited to, three-dimensional (3D) mice, joysticks or pointing sticks, gamepads, and graphic tablets, as well as audio / visual devices such as speakers, digital cameras, digital video cameras, portable media players, webcams, image scanners, fingerprint scanners, barcode reader 3D scanners, 3D printers, laser range finders, and eye-tracking devices. User interface input devices may also include medical imaging input devices such as computed tomography, magnetic resonance imaging, positron emission tomography, and medical ultrasound devices. User interface input devices may also include audio input devices such as MIDI keyboards and digital musical instruments.

[0401] User interface output devices may include non-visual displays such as a display subsystem, indicator lights, or audio output devices. The display subsystem may be a flat panel device such as one using a cathode ray tube (CRT), a liquid crystal display (LCD), or a plasma display, a projection device, a touch screen, etc. In general, use of the term "output device" is intended to include all conceivable types of devices and mechanisms for outputting information from computer system 2800 to a user or to another computer. For example, user interface output devices may include, but are not limited to, a variety of display devices that visually convey text, graphics, and audio / video information, such as monitors, printers, speakers, headphones, automobile navigation systems, plotters, sound output devices, and modems.

[0402] Computer system 2800 may include a storage subsystem 2818 that provides a tangible, transitory, computer-readable storage medium for storing software and data constructs that provide the functionality of embodiments described in this disclosure. The software may include programs, code modules, instructions, scripts, etc. that, when executed by one or more cores or processors of processing unit 2804, provide the functionality described above. Storage subsystem 2818 may also provide a repository for storing data used in accordance with the present disclosure.

[0403] 28 , storage subsystem 2818 may include various components including system memory 2810, computer-readable storage medium 2822, and computer-readable storage medium reader 2820. System memory 2810 may store program instructions that may be loaded and executed by processing unit 2804. System memory 2810 may also store data used during execution of the instructions and / or data generated during ...

Claims

1. It is a method, This includes generating a first virtual private label cloud (vPLC) for a first reseller based on the infrastructure provided by a cloud service provider (CSP) in a certain region. Generating the first vPLC includes allocating a first portion of the CSP provisioning infrastructure to the first vPLC. The aforementioned method, The method further includes generating a second vPLC for a second reseller based on the aforementioned CSP provision infrastructure. Generating the second vPLC includes allocating the second portion of the CSP provisioning infrastructure to the second vPLC. The aforementioned method, By using the first vPLC, one or more first reseller supply cloud services are provided to one or more customers of the first reseller, By using the second vPLC, one or more second reseller supply cloud services are provided to one or more customers of the second reseller, To generate a first set of metadata customization instructions for the first vPLC, To generate a second set of metadata customization instructions for the second vPLC, To generate first customized metadata associated with the first vPLC based on at least one set of first metadata customization instructions, To generate a second customized metadata associated with the second vPLC based on at least the second set of metadata customization instructions, Methods that further include the above.

2. The method according to claim 1, wherein the first set of metadata customization instructions for the first vPLC is different from the second set of metadata customization instructions for the second vPLC.

3. The method according to claim 1, wherein generating the first customized metadata associated with the first vPLC is performed independently of generating the second customized metadata associated with the second vPLC.

4. The method according to claim 1, wherein generating the first customized metadata associated with the first vPLC occurs before receiving a request for the first customized metadata associated with the first vPLC.

5. The method according to claim 4, further comprising storing the first customized metadata associated with the first vPLC in a repository before receiving the request.

6. Generating the first customized metadata associated with the first vPLC is: Before receiving the aforementioned request, the raw metadata associated with the first vPLC is obtained, Before receiving the aforementioned request, the acquired unacquired data associated with the first vPLC Applying the first set of metadata customization instructions to the processed metadata, The method according to claim 1, including the method described in claim 1.

7. The method according to claim 1, wherein generating the first customized metadata associated with the first vPLC occurs upon receiving a request for the first customized metadata associated with the first vPLC.

8. Generating the first customized metadata associated with the first vPLC is: Upon receiving the aforementioned request, the raw metadata associated with the first vPLC is obtained, Applying the first set of metadata customization instructions to the acquired raw metadata associated with the first vPLC, The method according to claim 7, including the method described in claim 7.

9. Generating the first customized metadata associated with the first vPLC is: Before receiving the aforementioned request, the raw metadata associated with the first vPLC is obtained, Upon receiving the aforementioned request, the first set of metadata customization instructions is applied to the acquired raw metadata associated with the first vPLC, The method according to claim 1, including the method described in claim 1.

10. The method according to claim 9, further comprising storing the first customized metadata associated with the first vPLC in a repository before receiving the request.

11. A program for causing one or more processors to perform the method described in any one of Claims 1 to 10.

12. It is a system, One or more processors, It comprises one or more memories for storing computer executable instructions, When the aforementioned computer executable instruction is executed by one or more processors, Based on the infrastructure provided by a Cloud Service Provider (CSP) in a certain region, a first virtual private label cloud (vPLC) is created for a first reseller. Having the system perform the task and generate the first vPLC includes allocating the first portion of the CSP provision infrastructure to the first vPLC. The system is further made to generate a second vPLC for a second reseller based on the CSP provision infrastructure, and generating the second vPLC includes allocating a second portion of the CSP provision infrastructure to the second vPLC. By using the first vPLC, one or more first reseller supply cloud services are provided to one or more customers of the first reseller, By using the second vPLC, one or more second reseller supply cloud services are provided to one or more customers of the second reseller, To generate a first set of metadata customization instructions for the first vPLC, To generate a second set of metadata customization instructions for the second vPLC, To generate first customized metadata associated with the first vPLC based on at least one set of first metadata customization instructions, A system that further causes the system to generate a second customized metadata associated with the second vPLC based on at least the second set of metadata customization instructions.

13. The system according to claim 12, wherein generating the first customized metadata associated with the first vPLC is performed independently of generating the second customized metadata associated with the second vPLC.

14. The generation of the first customized metadata associated with the first vPLC occurs before the receipt of a request for the first customized metadata associated with the first vPLC. Generating the first customized metadata associated with the first vPLC is: Before receiving the aforementioned request, the raw metadata associated with the first vPLC is obtained, Before receiving the aforementioned request, the first set of metadata customization instructions is applied to the acquired raw metadata associated with the first vPLC, The system according to claim 12 or claim 13, including the above.

15. The generation of the first customized metadata associated with the first vPLC occurs when a request for the first customized metadata associated with the first vPLC is received. Generating the first customized metadata associated with the first vPLC is: Upon receiving the aforementioned request, the raw metadata associated with the first vPLC is obtained, Applying the first set of metadata customization instructions to the acquired raw metadata associated with the first vPLC, The system according to claim 12 or claim 13, including the above.

16. The generation of the first customized metadata associated with the first vPLC occurs when a request for the first customized metadata associated with the first vPLC is received. Generating the first customized metadata associated with the first vPLC is: Before receiving the aforementioned request, the raw metadata associated with the first vPLC is obtained, Upon receiving the aforementioned request, the first set of metadata customization instructions is applied to the acquired raw metadata associated with the first vPLC, The system according to claim 12 or claim 13, including the above.

Images